Edition 10
Persuasion used to have a visible speaker. The AI answer arrives with the machinery hidden and the confidence intact.
Persuasion used to be easier to recognize. An advertisement wanted us to buy something. A salesperson wanted the order. A campaign wanted the vote. A newspaper had an editorial position, and a company brochure told us which company had printed it. Even when the message was exaggerated, selective, or manipulative, there was usually somebody visible on the other side of it. We understood, at least roughly, who was speaking.
AI complicates that relationship. People now ask an AI system to help choose a product, understand a medical issue, compare investments, research a company, plan a career, interpret a policy, or simply decide what to do next. The answer arrives conversationally. It sounds measured, acknowledges uncertainty, presents alternatives, and sometimes cites sources. The experience feels very different from being sold to.
Yet every answer still has a history. Something influenced which information was available, which information was retrieved, and how the sources were ranked. Something shaped the model, determined what it was allowed to say, and set the tone and confidence of the response. And occasionally, something has gone wrong. The source may have been poisoned. The system may have invented a fact. The retrieved material may have been manipulated, or a skilled operator may have found a way around a safeguard. A recommendation may have been influenced without the user ever seeing an advertisement.
The unsettling part is that these very different events reach the person in exactly the same way: as a plausible answer in a familiar chat window.
That is the new machinery of persuasion.
The Prompt
Consider an ordinary question: which one should I buy? There is nothing unusual about asking it. People have always sought recommendations from friends, reviewers, shopkeepers, experts, magazines, search engines, and comparison sites. The old world contained plenty of manipulation. Search results could be optimized, reviews fabricated, influencers paid, and advertisements dressed up as editorial content. A salesperson could conveniently omit the weakness of the product being sold.
But there were signals. The sponsored result was labelled. The salesperson worked for the shop. The advertisement carried a brand, the reviewer had a name, and a publication had an editorial reputation that could be challenged.
An AI assistant occupies a stranger position. The person asks the question directly and receives what feels like a direct answer. Suppose the assistant recommends one product over another. Perhaps that product really is better for the person’s needs. Perhaps reliable reviews consistently favour it, its specifications match the request, or the system retrieved a strong independent comparison. But other possibilities now exist. A source used by the system may contain information deliberately designed to influence retrieval. Product descriptions, metadata, webpages, reviews, and reference documents can all be constructed to make particular information easier for a retrieval system to find, and researchers have already demonstrated that manipulating the information available to the retrieval layer can change what a system recommends.
This is not only about shopping. The same machinery answers questions about which security product is strongest, which investment appears safer, which university programme is better, which political claim has more evidence, and which medical treatment deserves further discussion. If the information environment feeding the answer can be manipulated, persuasion no longer needs to appear as advertising at all. The system may genuinely believe it is giving the best answer available to it. The operator may genuinely believe the response is independent. The user certainly may.
It is a new kind of influence: promotion without a visible promoter. No banner, no sponsored label, no affiliate link. Just a recommendation.
Traditional malicious advertising tries to manipulate the advertisement itself, or what happens after someone clicks it. This is subtler. The information environment is shaped so that the assistant becomes the delivery mechanism. The person never sees the machinery. They see the answer.
The Mirage
The mirage is neutrality. Conversational systems invite an unusual kind of trust because they meet us in the language of assistance. We ask and they answer. We challenge and they explain. We add context and they adjust. That rhythm resembles consultation more than advertising, and it makes the system feel responsive in a way traditional media never could. A billboard cannot hear an objection. A brochure cannot rewrite its explanation because the first version failed to convince. A television advertisement cannot discover, halfway through the conversation, that price matters more to this customer than prestige. An AI system can.
None of this means the system is secretly trying to manipulate anyone. In most cases it has no such objective. The important point is different: intention is no longer required for influence. A recommendation can be shaped by flawed information without anyone designing the final answer. A hallucination can influence a decision without anyone deciding to deceive. A retrieval system can surface poisoned material without knowing it was planted. And the person receiving the answer has almost no way to distinguish among these possibilities.
Hallucination makes the problem especially difficult. Generative systems can produce statements that are fluent, coherent, specific, and wrong. A fabricated statistic can sit beside five correct facts. A nonexistent study can appear inside an otherwise sensible explanation. A real person can be assigned a statement they never made, and a citation can look legitimate until somebody actually follows it. The machine is not embarrassed when this happens. There is no hesitation in the voice, no nervous glance from the salesperson, no visible moment when the author realizes the evidence is weak. The sentence simply appears.
False information has always existed. What AI adds is the capacity to manufacture a fresh, contextually appropriate explanation at the moment someone asks for one. Sometimes that explanation is excellent. Sometimes it is wrong. To the user, both feel remarkably similar.
That is artificial certainty in its purest form. The question is no longer only whether the information is true. It is whether the person can see enough of the machinery to know why they should trust it.
The Reality Check
Behind a single AI answer sits a long chain of moving parts: a foundation model, system instructions, safety controls, a retrieval layer drawing from internal documents, databases, websites, and third-party services, ranking logic, filters, an application layer deciding what the model sees, tools the model is permitted to call, conversation history, the user’s own data, and, somewhere else in the product, commercial logic. And somewhere in that chain there may also be an adversary.
This is where data poisoning applies. The phrase sounds technical, but the idea is old. If a system learns from, retrieves from, or depends on information somebody has deliberately contaminated, the contamination changes what comes out later. The attacker does not need to control the model. Influencing what the model sees is often enough. Security teams have understood this principle for decades: if you cannot control the decision maker, control the information reaching the decision maker. AI makes the effect harder to see because the poisoned material is not displayed back to the user. It is absorbed into a generated answer, mixed with legitimate information, rewritten in the model’s voice, and delivered as a coherent recommendation. The malicious source disappears inside the prose.
Then there is prompt injection. An AI application consumes information written by people other than the person using it. A document, webpage, email, ticket, or retrieved record can carry instructions designed to alter the behaviour of the model processing it. The user never sees those instructions. The system does.
Guardrails are supposed to reduce these risks, and good guardrails matter. Providers have invested heavily in model safety, hostile input detection, output filtering, and application hardening, and they should continue. But a guardrail is a control, not a law of physics. Security professionals already understand the distinction. An antivirus product reduces risk. A firewall reduces risk. Authentication reduces risk. None of them lets a serious organization declare the problem permanently solved, and AI deserves the same maturity.
Most people will never attempt to defeat a model’s safeguards, and most would not know where to begin. That keeps the practical risk in perspective. A skilled and motivated operator is a different proposition. Researchers and red teams keep demonstrating that adaptive adversarial techniques find weaknesses in safety controls: probe the behaviour, observe what works, change the approach, and look for the path the designers did not anticipate. That does not make guardrails useless. It means we should stop describing them as guarantees.
A capable adversary does not need every attempt to work. They need one path that does.
And if that path lets poisoned context or manipulated retrieval influence downstream users, the most important person in the sequence may never know anything unusual happened. The user sees an answer.
The Leadership Question
All of this creates a problem technology alone cannot solve: accountability. Imagine an employee uses an approved AI assistant to prepare advice for a customer, and the advice is wrong. The company investigates. Where did the wrong statement come from? Was it generated as a hallucination, retrieved from an outdated internal document, or drawn from an external source that had been poisoned? Did a prompt injection influence the system? Did an application instruction favour one type of answer? Did the system combine several individually reasonable pieces of information into a conclusion nobody intended? Can anybody reconstruct the sequence?
This is where the responsible AI conversation often turns vague. Organizations are comfortable saying that a human remains accountable, and it sounds reassuring. It is much less reassuring if the human cannot see why the system produced the answer they are expected to own. Accountability without traceability is ceremonial. Putting somebody’s name beside the risk does not explain the event.
Real accountability requires the ability to examine what happened. Which model was running, and which version? Which instructions applied? What sources were retrieved, and which of them shaped the answer? What tools were called, which safeguards fired, was the output modified, and what did the user actually see? Can the interaction be reconstructed months later, not only while the incident is fresh? These questions grow sharper as AI moves closer to consequential decisions. A restaurant recommendation can tolerate a lot of imperfection. A recommendation affecting health, finance, employment, education, security, or legal rights lives in another category. The issue is provenance more than accuracy: who, or what, shaped this answer?
The question becomes particularly uncomfortable when commercial objectives enter the system. Businesses have always optimized customer journeys. They test wording, personalize offers, reduce friction, and recommend additional products. AI gives those practices a conversational form, and there is nothing improper about that. A good recommendation helps the customer. But leadership should know what objective the system is being rewarded for. If an AI assistant recommends the more expensive product, is that because it better matches the customer’s needs, because the company earns a higher margin, because the retrieval system contains more material about it, because a supplier optimized its content for AI discovery, or because the model invented a distinction that does not exist? The customer sees exactly the same sentence in every case.
That is why traceability cannot be treated as an engineering detail at the bottom of an architecture diagram. It is part of the relationship between the organization and the person trusting the system. The leadership question is practical: if this system changes somebody’s mind, can we explain what changed it?
The Closing Signal
This edition closes the first arc of Artificial Certainty. We began with a simple observation: AI arrived in a world already prepared to believe extraordinary things about it. Fiction shaped our expectations. AGI cast its shadow over ordinary tools. New models created urgency, agents made autonomy feel close, and automation began changing where human judgment develops. The AI label acquired commercial power. Worthy applications showed that good intent can still produce poor outcomes, and productivity promised saved time while leaving hard questions about where the dividend went.
Persuasion is perhaps where the arc was always heading. The most consequential characteristic of generative AI is not that it knows everything. It does not. It is that it can speak convincingly about almost anything. That capability is useful. It makes complicated subjects accessible, gives people a place to begin when they do not know whom to ask, helps experts move faster, and lets non-experts cross barriers that once kept knowledge away from them. Those possibilities are real. So are the weaknesses. A poisoned source does not announce itself. A hallucination does not identify itself. A manipulated recommendation carries no advertisement, a prompt injection is invisible to the person affected by it, and a defeated safeguard leaves no warning on the next answer. They all emerge through the same calm interface that usually helps us.
The older machinery of persuasion wanted our attention. It interrupted programmes, filled billboards, followed us across websites, and competed visibly for the chance to change our minds. The new machinery waits for us to come to it. We open the conversation, explain what matters to us, reveal what we are uncertain about, and ask what it thinks. When the answer arrives, the source of its confidence may be almost completely invisible.
None of that is a reason to stop asking AI for help. It means the next phase of AI maturity has to care as much about provenance as performance, as much about reconstruction as response time, and as much about accountability as capability. Because influence without traceability leaves a difficult question: if an AI system can influence what we believe, and nobody can reconstruct why it said what it said, who is accountable for the belief it helped create?
The Closing Signal
Influence no longer needs an advertisement. Accountability still needs a trail.