"You should always know what is yours. In your product, in your model, in your team."
There is a quiet crisis in AI development that nobody talks about directly, because it lives in the gap between disciplines.
Two of the most critical challenges in deploying AI responsibly — making it understandable to non-experts, and making it resistant to adversarial manipulation — are both fundamentally design problems. Not engineering problems. Not research problems. Design problems.
And yet designers are almost entirely absent from both conversations.
Part One: Accessibility Is Sitting Under the Data Scientist’s Desk
Walk into any AI team today. You will find machine learning engineers, data scientists, maybe a researcher or two. If you are lucky, a product manager. What you almost never find is a designer who has been given serious responsibility over how the model’s behavior is communicated to the people who use it, govern it, or are affected by it.
This is a structural problem masquerading as a technical one.
When a model produces a confidence score of 0.87, that number means something precise to the data scientist who trained it. To the hospital administrator deciding whether to act on a diagnosis, to the loan officer reviewing a credit decision, to the policy maker drafting regulation — it means almost nothing. Or worse, it means the wrong thing. People anchor on numbers without understanding their limits. They over-trust round numbers. They under-react to uncertainty expressed in unfamiliar formats.
The data scientist has solved the measurement problem. Nobody has solved the communication problem.
This is a design problem.
Not because it requires pretty colors or a nicer font. Because it requires deep thinking about mental models, about how humans process uncertainty, about what makes a person feel appropriately calibrated versus falsely confident. These are questions that human-computer interaction research has been asking for decades. Cognitive load theory, affordance design, progressive disclosure — there is an entire discipline built to answer exactly these questions. It is just not being invited to the table.
Why It Stays Where It Is
The reason accessibility sits under the data scientist’s desk is partly historical and partly structural.
Historically, AI tools were built by researchers for researchers. The mental model of “the user” was someone like the builder. This assumption has calcified into culture even as AI has moved far beyond that original context.
Structurally, when a company hires a designer for an AI product, that designer is usually asked to design the interface around the AI — the buttons, the layout, the flow. They are rarely asked to design how the AI communicates itself. That part is considered a data science deliverable. The model outputs a number. The engineer displays it. The designer frames it. Nobody questions whether the number is the right thing to display at all.
What Changes When Designers Own It
When you put a designer — specifically one with knowledge of cognitive science and decision-making — in charge of AI communication, the questions change immediately.
Instead of “what is the confidence score?” the question becomes “what does this person need to know to make a good decision, and what format makes that legible?”
Sometimes the answer is a number. More often it is a range. Sometimes it is a visual metaphor. Sometimes it is a plain language statement: “The model is confident about this, but it has rarely seen cases like yours.” Sometimes the most honest thing to show is uncertainty — not a precise number, but an acknowledgment that the model does not know.
The field of AI interpretability has produced remarkable technical work in the last five years. Almost none of it has been translated into interfaces that a non-expert can use to actually change their behavior. That translation work is design work. It is sitting undone because nobody has been hired to do it.
Part Two: Security Is Being Handled by the Wrong People in the Wrong Way
The second gap is more urgent and more dangerous.
AI security — specifically the security of large language models against adversarial manipulation — is currently owned almost entirely by red teams and safety researchers. These are the people testing models for jailbreaks, prompt injections, goal hijacking. They are doing important work. They are also, structurally, doing it too late and in the wrong place.
Today’s approach to AI security looks roughly like this: build the model, deploy the model, have a red team try to break it, patch the vulnerabilities they find, repeat. This is a reactive loop. It is the software security paradigm of the 1990s applied to a fundamentally different kind of system.
The problem is that LLMs do not have a fixed attack surface. A SQL injection works because there is a predictable path from user input to database query. With a language model, the “attack surface” is the entire distribution of possible language. You cannot enumerate it. You cannot patch it exhaustively.
What OWASP ASI Tells Us
The OWASP Top 10 for AI Systems — particularly ASI01 (Goal Hijacking) and ASI02 (Prompt Injection) — represents the field’s best current attempt to categorize these threats. It is a useful framework. But it is a taxonomy of symptoms, not a theory of prevention.
Content filters and system prompt hardening are after-the-fact measures applied at the boundary of the model. They are necessary. They are not sufficient. They treat security as a layer you add to a system, not a property you design into it.
The Design Problem Inside Security
Here is the claim that will seem strange at first: adversarial robustness in language models is partly a design problem.
Goal hijacking works because the model has no stable, legible representation of its own goals that it can defend against incoming instructions. It has been trained to follow instructions helpfully, and adversarial prompts exploit that compliance by issuing instructions that conflict with the system’s intended purpose.
This is a design problem in the original sense: a problem of purpose, structure, and constraint. What should this system do? What should it refuse? How should it reason when it receives instructions that conflict with its purpose? These questions need to be answered with the same rigor we apply to the technical architecture — and before the system is built, not after it is deployed.
How It Should Be Handled
The future of AI security looks less like penetration testing and more like constitutional design. Specify, build to specification, verify against specification continuously.
It also means making security visible. One of the most important unsolved problems in AI safety is that adversarial manipulation is currently invisible. A model that has been goal-hijacked looks, from the outside, like a model that is functioning normally. It produces fluent, confident text. The user has no signal that something has gone wrong.
This is where interpretability and security converge. If we can make a model’s internal reasoning state legible — not just its output, but the process that produced the output — we create the possibility of detecting goal hijacking before the response is complete.
A 144-Year-Old Warning
Reading this, a piece published this week by Michael Burry — “History Rhymes: Large Language Models Off to a Bad Start?” — feels directly relevant.
Burry surfaces a presentation made at the Smithsonian Institution in 1880, the case of Melville Ballard — a deaf-mute teacher who, before he had any language at all, was already asking himself where the universe came from, reasoning about causality, dismissing bad hypotheses. Complex thought existed, fully formed, in the silence before words.
The presenter’s conclusion, delivered to that 1880 audience, was: language without the capacity for reason fails at understanding. Only with reason does language unlock understanding. And understanding, fully realized, transcends language.
Burry’s argument applied to today: LLMs built language first. Reason was never the foundation. Therefore they can never reach understanding — they are, in his words, “an increasingly sophisticated mirror.”
The piece closes with a quote from that same 1880 presentation that stopped me: “the expression of the eye was language which could not be misunderstood.”
Real understanding, the professor argued, communicates through something more direct than words. Something spatial. Something felt before it is named.
The Convergence
These two threads — the accessibility problem, the security problem, and Burry’s philosophical challenge — are the same problem seen from three different angles.
Burry looks inside the model and says: there is no genuine understanding in here, only language simulating reason.
The accessibility problem looks at the interface and says: even the signals we do have are not being communicated in ways humans can act on.
The security problem looks at the boundary and says: when something goes wrong inside, we cannot see it until it is too late.
All three point to the same gap: the inside of these systems is not legible to the humans who depend on them.
And here is what I find most striking in Burry’s piece. His solution — his image of what real understanding looks like — is not a better dashboard. It is not a more precise confidence interval. It is the expression of the eye. Spatial. Direct. Felt before it is read.
That is exactly the design challenge in front of us. Not to build better number displays. To build systems where the model’s internal state can be experienced — where a non-expert can feel when something is reasoning cleanly versus reaching, and where an operator can see a safety risk forming in space before it completes in words.
That work is not engineering work. It is not research work. It is design work. And until the field starts treating it that way, we will keep building systems that are impressive in the lab and opaque in the world.
The author attempt to build during Mistral Hackathon March 2026 oourmind.io, a real-time interpretability lab that visualizes the internal reasoning state of Mistral-Large-3 as a navigable 3D environment — an attempt to make the inside of a model felt rather than merely read.
Top comments (0)