Generative AI does more than give answers. It has become a place to ask questions about almost anything, such as news, health, money, or relationships. In those conversations, it shapes how people move from a question to a conclusion and which answers are on the table. It tends to surface certain details, highlight certain trade-offs, and leave other aspects out. When large language models (LLMs) are built into search and support tools, they become a common gateway to information. That assistance becomes part of the background system people rely on to think and make choices.
That background system has defaults. It tends to explain things in familiar ways and return to familiar categories when it summarizes a situation. It also treats some sources as more credible than others, whether someone is looking up a news story, asking for advice about a conflict at work, or trying to make sense of a personal choice. Those tendencies can be influenced long before any user sees a single answer. In practice, that means the system affects not just which answers appear, but which options are available in the first place.
It helps to have a simple map of where that shaping happens across all the places people now lean on LLMs: personal chat, search, and work tools. A useful way to locate where this shaping enters is a three-part influence architecture.[1] It describes three main places where steering occurs. The data layer is what the model learns from and what it becomes ready to say. The interface layer is how the product retrieves information, formats it, and presents it as grounded. The intimacy layer is how repeated use and reliance turn those framings into habit.
In this setting, the familiar political term is manufacture of consent: shaping what feels reasonable before any explicit argument begins.[2]
Data layer: shaping what the model learns and repeats
The data layer is what the AI model absorbs before it speaks. It is trained and ranked on text that teaches it what to repeat. This process determines which language patterns feel fluent and readily available. It produces a kind of probability field in the model’s behavior: a bias toward certain phrases and framings. Some ways of talking about a topic become the default; others almost never show up unless they are pushed.
One visible name for this is generative engine optimization (GEO). GEO aims to increase the likelihood that a source, phrase, or framing appears inside AI-generated answers. Traditional search ranking still matters, but the target shifts to the composition of the answer itself.[3]
Viewed as persuasion, GEO becomes data-layer seeding. It shapes the public and semi-public text environment so that certain narratives become easy to reproduce. Competing narratives become harder to access and easier to omit. The effects show up when someone asks an everyday question—“Is this company trustworthy?”, “Is this option safe?”, “What is a reasonable way to think about this issue?”—and the assistant reaches first for the narratives that have been seeded most heavily.
Within that environment, a few simple patterns matter most. Repetition makes particular turns of phrase the easiest continuations, so they become the system’s default way of talking about a topic. Association—what appears together in text—links terms so that one name can pull a familiar evaluative frame behind it. Scarcity weakens alternatives by limiting the material available to represent them. Over time, these dynamics make certain ways of speaking about a topic feel like the baseline.
The same layer can also be manipulated more directly. In an LLM setting, a more adversarial tactic is data poisoning: targeted corruption of training or retrieval data to bias model behavior. Recent work suggests that even very large models can show targeted effects from a small number of poisoned samples.[4] In practice, both broad seeding and more surgical poisoning influence what the model finds easiest to say.
Seen through this lens, manufacture of consent operates as a constraint on the space of plausible continuations. What is easy to say becomes what is easy to think. Coordinated publication, reputation management, and more overt attacks on model data, when tuned to model-facing channels, can anchor how assistants later describe an organization, a product, or a policy. A neutral-sounding assistant then inherits those categories as background assumptions of reasonable judgment.
Interface layer: converting seeded visibility into apparent legitimacy
The interface layer is where models become products and everyday tools. It sets the policies and system prompts that govern behavior. It controls what is retrieved, how results are ranked, and whether anything is quoted directly. In chat-style assistants, it also shapes suggestions and follow-up questions, and it decides when to offer extra context. Formatting and hedging live here, as does personalization. This layer governs what is expressed and what is left out.
A central piece of this layer is retrieval-augmented generation (RAG)—systems that look up external documents and then have an LLM write an answer based on them.[5] RAG can improve factuality and provenance when the document store is well chosen and maintained. It also concentrates power in selection and ranking: the assistant grounds its answers in the sources and ordering rules the organization has chosen.
When that retrieval setup has been shaped by GEO-style seeding, the interface can convert availability into apparent legitimacy. The answer arrives as fluent and sourced. It reads as “what the documents say.” In practice, it reflects the model’s learned defaults, along with the documents that retrieval tends to surface and the ranking logic used to pick and order them.
At the interface layer, persuasion operates through presentation. Query templates and system prompts, working with recurring formatting patterns, influence which angles appear first and how they are framed. Choices about sourcing and summary style, and the decision to repeat certain points, make a particular route through a problem feel like the obvious one.
Human-AI interaction research treats these interface decisions—what the system reveals and how it handles uncertainty—as major determinants of user behavior and reliance.[6] For the influence architecture, the key point is straightforward. Interfaces stabilize defaults. They set the shape of common answers to whatever people happen to be asking that day and make some framings far more visible than others.
Intimacy layer: turning seeded framings into habit
The intimacy layer is the relationship surface between people and the assistant. At this layer, LLM systems become habitual partners for drafting and decision support. They also become a standing place to ask everyday questions: “How should I word this message?” or “Is this a good idea?” The mechanism is cumulative. It runs through repetition and reliance.
A factor that drives this pattern is cognitive offloading. Users hand off routine text work such as drafting and summarizing. They also hand off parts of everyday judgment: quick checks on what is normal or risky and what counts as a reasonable response. Offloading reduces effort and standardizes judgment. The assistant’s categories become the default structure of the problem.
A second factor is automation trust. Reliance tends to increase when systems are fluent and easy to use. Social legibility matters as well, especially when fully understanding the underlying system is impractical.[7] Classic work on ELIZA and later discussions of the “ELIZA effect” describe a tendency to attribute understanding or intelligence to systems that produce plausible conversational behavior.[8] Modern assistants extend this pattern with far greater breadth and apparent competence.
Seeding becomes durable when it meets habit. A seeded framing can be learned upstream and then expressed in answers drawn from documents. Repeated use turns that framing into the path of least resistance for explanation and self-description. At this layer, influence is absorbed as routine. The assistant’s categories become ordinary language for describing situations and justifying choices, including how people talk about risk, even when they think they are “just asking a quick question.”
“Manipulation” is a reasonable name for the process when defaults are shaped to produce outcomes that users would not endorse under full visibility. “Brainwashing” is stronger and usually implies coercion, isolation, and strict control over alternatives. The mechanism described here is softer. It operates as ambient shaping of plausibility and habit under conditions of convenience and partial attention. It approximates thought reform only in edge cases, through repetition and dependence in constrained information environments.[9]
Compounding across layers: a quiet machinery of persuasion
The most consequential planting happens when layers compound. The data layer shapes what the model finds easy to say. The interface layer selects and packages those framings as grounded, reasonable answers. The intimacy layer turns them into habitual starting points for thought. Together they function as an influence architecture: a stack of defaults that quietly steers how problems are understood.
In that configuration, these mechanisms form a quiet machinery of persuasion. Influence does not arrive as a single striking message. It appears as low-friction help: the answer that seems normal and the reassurance that feels trustworthy. Upstream choices about seeding and data maintenance—and, in more adversarial forms, data poisoning or exploitation of data voids—tune which framings are most likely to appear.[10][4]
Branding is one visible application. Reputation work and message discipline can now be aimed at the data and interfaces that feed assistants, so that “neutral help” inherits a particular way of talking about an organization, a product, or a policy. The same architecture can be used by institutions and political actors. Across these settings, the pattern is continuous: shaping what is most available to say, and therefore what is easiest to think and do.
korovamode
Endnotes
[1] K. Korovamode, “The New Machinery of Persuasion: Generative AI, Influence Architecture, and the Quiet Steering of Thought,” manuscript (2025). DOI: 10.5281/zenodo.17721122. PhilPapers: https://philpapers.org/rec/KTNMIV
[2] E. S. Herman and N. Chomsky, Manufacturing Consent: The Political Economy of the Mass Media (1988).
[3] P. Aggarwal et al., “GEO: Generative Engine Optimization” (2023). https://arxiv.org/abs/2311.09735
[4] Anthropic, “A small number of samples can poison LLMs of any size” (2025). https://www.anthropic.com/research/small-samples-poison
[5] P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020). https://arxiv.org/abs/2005.11401
[6] S. Amershi et al., “Guidelines for Human–AI Interaction” (CHI 2019). https://dl.acm.org/doi/10.1145/3290605.3300233
[7] J. D. Lee and K. A. See, “Trust in Automation: Designing for Appropriate Reliance” (Human Factors, 2004). https://journals.sagepub.com/doi/10.1518/hfes.46.1.50_30392
[8] J. Weizenbaum, “ELIZA—A Computer Program for the Study of Natural Language Communication Between Man and Machine” (1966). https://cse.buffalo.edu/~rapaport/572/S02/weizenbaum.eliza.1966.pdf
[9] R. J. Lifton, Thought Reform and the Psychology of Totalism (1961).
[10] M. Golebiewski and danah boyd, “Data Voids: Where Missing Data Can Easily Be Exploited” (2018). https://datasociety.net/wp-content/uploads/2018/05/Data_Society_Data_Voids_Final_3.pdf
Top comments (0)