By Jinav Shah. Views are personal.
We are at an inflection point.
AI systems are moving from answering questions on a screen to taking actions in the world. Booking appointments. Approving transactions. Navigating physical environments. Writing and executing code.
With that shift comes a natural question. Not whether AI is capable enough, but whether we understand it well enough to deploy it confidently in high stakes situations.
This is not a question about AI being good or bad. It is a question about how well we understand what is happening inside these systems when they get something wrong.
The honest answer is that we are still early. Not because researchers are not trying. But because the nature of how AI stores and processes information is fundamentally different from anything we have debugged before.
Every major technology shift built infrastructure alongside capability
Cars brought crash testing. Electricity brought circuit breakers. Medicine brought clinical trials.
Nobody called these anti-progress movements. They were the responsible infrastructure that allowed those technologies to scale into everyday life.
AI needs the same. Not resistance. Infrastructure.
The specific infrastructure AI needs is different from anything we have built before. Because the failure modes of AI are different from anything we have debugged before.
In software, a bug has an address
A line number. A variable. A function that received the wrong input. You set a breakpoint, step through execution, and find exactly where reality diverged from expectation.
The source code is human written, which means human readable. The logic is explicit. The intent is traceable.
When an AI model gets something wrong, none of this exists. There are billions of floating point numbers whose collective meaning emerged from training on human text. No line numbers. No variables. No explicit logic.
You can only observe what goes in and what comes out.
So what do we do when AI gets something wrong
We change the model. Upgrade versions. Tweak the prompt. Add a guardrail.
All reasonable. None of them root cause.
It is like treating recurring headaches with stronger painkillers each visit without ever investigating why they started. The reason we rely on external interventions is not laziness. It is that root cause investigation requires understanding what is happening inside, and that is one of the hardest unsolved problems in the field.
The reason goes deeper than tooling
Think of a hex color code. Three primary colors. Six digits. Yet 16 million possible combinations, most indistinguishable to the naked eye.
A transformer represents each word as 256 numbers. Not 256 separate meanings in 256 separate boxes. 256 numbers that combine, overlap, and interact to represent potentially thousands of concepts simultaneously.
This is called superposition. And it is not a design flaw. It is an inevitable consequence of matrix multiplication, the core operation of every transformer. Mixing happens by default.
The word bank does not have a dedicated slot for financial institution and another for riverbank. Both meanings, and dozens of others, are folded into the same 256 numbers, in overlapping directions, sharing the same space.
When bank is interpreted wrongly, there is no single number to point to. The error is distributed across all 256 numbers, in combinations we have not named, emerging from interactions we cannot enumerate.
You cannot fix what you cannot locate.
And that is just one token, in one layer
Large models have 96 layers. Each with its own Q, K and V transformations mixing information differently. Then non-linear FFN layers introducing combinations no linear operation can express. Then MoE, Mixture of Experts, where the same input takes different computational paths depending on context.
There is no single moment where the wrong answer was decided. It emerged gradually, collectively, across the entire network.
That is what you are trying to debug.
Now we are giving these systems tools to act
In software, a function call is deterministic. Same input, same output, traceable every time.
In AI with tool calling, the model decides whether to call a tool, which one to call, what parameters to pass, how to interpret the result, and whether to call another tool based on that result. Each decision is a probabilistic inference emerging from the same unlocatable internal state.
Tool outputs feed back into context. Errors compound across steps. Actions happen in the real world.
In software, a multi-step process has a call stack. You can inspect every frame, every variable, every state transition.
In an AI agent, there is no call stack. There is a sequence of probabilistic decisions producing actions that can be irreversible.
We have taken a system we cannot fully debug and given it the ability to act. The debugging problem did not just get harder. It got consequential.
This is solvable. But it requires the right framing.
Chris Olah at Anthropic has spent years trying to reverse engineer what concepts a model has learned internally, a field called mechanistic interpretability. His team has found recognisable features, grammar rules, factual associations, sentiment.
But the honest truth is that we find features by looking for things we already suspect exist. Nobody has a complete list. An accuracy failure caused by a feature you have not named is one you cannot systematically prevent.
The hex code has 16 million combinations and we can see them all.
AI has infinite combinations and we have named a fraction.
The path forward is not slowing AI down. It is building the interpretability tools, accountability frameworks, and testing infrastructure that let us deploy it confidently as the stakes rise.
Cars did not wait for perfect safety before scaling. They built safety infrastructure alongside scale.
That is exactly where we are with AI. And the work, by researchers, engineers, and product teams, is already underway.
The better news is that we now at least know what we are up against.
Connect on LinkedIn: https://www.linkedin.com/in/jinav-shah-27b3a255/
Top comments (0)