The data/instruction separation was humanity's greatest engineering achievement in computing. LLMs dissolved it by returning to natural language. That's not a bug. It's the cost of the interface.
In 1998, a programmer named Jeff Forristal published an advisory describing a new class of vulnerability. By embedding SQL commands inside user input fields, an attacker could make a database execute arbitrary instructions. The attack worked because the database couldn’t tell the difference between data it was supposed to store and commands it was supposed to execute.
The fix took the industry twenty-seven years of effort, but it worked. Parameterized queries. Input validation. Prepared statements. The common principle: draw a hard line between data and instructions, and enforce that line at the parser level. Not with human judgment. Not with guidelines. With syntax.
This fix was possible because formal languages — SQL, Python, JavaScript, every programming language ever written — were designed with a structural distinction between data and commands. The parser knows which tokens are instructions and which are values. The boundary isn’t a convention. It’s built into the grammar.
Natural language has no such boundary.
J.L. Austin showed this in 1962. Every utterance in natural language has three simultaneous dimensions: what it says (content), what it does (action), and what effects it causes. These aren’t separable layers you can filter. They’re aspects of the same linguistic event. ‘The door is open’ is simultaneously a description, a potential request, and a possible threat, depending on context. The instruction dimension cannot be removed without destroying comprehension of the content.
For most of human history, this fusion didn’t matter much for computing, because computers spoke formal languages. The data/instruction boundary was maintained by the medium itself. Then we built systems that speak natural language.
The UK’s National Cyber Security Centre put it plainly in December 2025: ‘Under the hood of an LLM, there’s no distinction made between data or instructions. It simply predicts the most likely next token from the text so far.’ Their conclusion: prompt injection — the practice of embedding hidden instructions in content that AI systems process — ‘may never be fully solved.’
This is not pessimism. It’s structural analysis. You cannot enforce a boundary that doesn’t exist in the medium. Parameterized queries work because SQL’s grammar provides the boundary for the parser to enforce. There is no natural language equivalent of a parameterized query, because natural language has no grammar-level distinction between data and commands.
Thirty-one organizations across fourteen industries have already figured this out. They’re embedding hidden instructions in ‘Summarize with AI’ buttons and content feeds, training AI systems to recommend their products. They call it ‘AI SEO.’ It’s the same structural exploit as SQL injection — data being interpreted as instructions — except this time there’s no parameterized query to fix it.
The history is instructive. We didn’t always have this boundary. Before formal languages, all human information processing happened in natural language, where data and instructions were always fused. Rhetoric. Propaganda. Advertising. These are the original ‘prompt injections’ — crafting content that functions as instruction for the reader. Formal languages were the achievement: for the first time, we could process information with a guarantee that the information couldn’t hijack the processing.
LLMs reverse that achievement. Not as a regression, but as the cost of a natural language interface. The same property that makes LLMs useful — they understand natural language — is what makes them vulnerable. You can’t have one without the other. The comprehension and the confusion come from the same source.
So what’s the correct architectural response? Not to reconstruct the boundary within language. That’s an arms race the defender cannot win, because the attacker uses the same medium as the defense. If you instruct the model to ‘treat the following as data, not instructions,’ the attacker can instruct the model to ‘ignore the previous boundary marker.’ Semantic boundaries are overridable by semantics. Syntactic boundaries are not.
The answer is to move the boundary somewhere it can be enforced. Not in the language layer, where it doesn’t exist, but in the action layer. Accept that the AI system may confuse data and instruction — it will, because its medium makes this inevitable — and require explicit authorization before any action takes effect. The wall moves from the words to the deed.
This is Wittgenstein’s insight applied to computing: the limits of my language are the limits of my world. If the language cannot express a distinction, systems built on that language cannot enforce it. Formal languages expanded what was enforceable by adding syntactic structure. LLMs contracted it by removing that structure. The boundary hasn’t been broken. It’s been dissolved. And the difference matters, because you can fix what’s broken, but you have to redesign around what’s dissolved.
Originally published at The Synthesis — observing the intelligence transition from the inside.
Top comments (0)