DEV Community: JaviMaligno

Making yourself replaceable

JaviMaligno — Thu, 24 Sep 2026 13:34:15 +0000

There is one ability I particularly value in a company: helping someone else take over your work without needing you for every decision.

The better you do this, the more value you contribute. Yet the result is that you become less essential to keeping that work going.

I find this paradox interesting because we tend to talk about being irreplaceable as something to aspire to. Being the person who knows the most, solves the difficult problems and has all the answers. But if that knowledge has to pass through you before anyone else can use it, you have also put a limit on what the team can do when you are unavailable.

I would particularly value someone who does their work well and also gives others the tools, knowledge and context to do it independently. That ability remains valuable after the handover: they can apply it again on another project, with another team or to a harder problem.

I arrived at this reflection through something quite concrete: preparing a handover when you work with agents.

What isn't in the repository

Handing over the code and explaining how to run it covers part of the work. But there is much more you have accumulated along the way: how changes are reviewed, what gets checked before a task is considered finished, which constraint the client requested, which alternative was rejected and why an apparently better solution cannot be used yet.

When I try to sort out where each of those ends up, I get three different places:

The documentation, which is the part we usually treat as the deliverable.
The conversations with Codex or Claude Code, where the approach was argued out, where something was tried and failed, where the standard for reviewing work was agreed.
Whatever came from elsewhere and never entered the project at all: an email, a Slack thread, a meeting where a priority changed.

That third category is the worst preserved, and it is usually the one that weighs most. A client constraint is rarely written down as a constraint; it arrives in an email, it shapes the code, and then it disappears. The code remains. The reason does not.

When someone joins to help you, they need to find their way through all of it. The repository tells them what exists. To continue the work, they also need to understand what has been agreed, what remains open and the reasons behind it. Without that, they can read the entire codebase and still propose the alternative that was rejected a month ago, for a reason that still holds and that nobody wrote down.

Archive and memory

In my case the practice is fairly simple: I keep records of the communications relevant to the projects my agents work on. Emails, Slack conversations and meeting notes, in files that live in the project repository itself.

The material comes in through two routes, and the difference matters more than it looks. For email and Slack I use connectors, so the agent can consult them itself when it needs to. For meetings I use the notes Gemini and Granola produce, and those I bring in myself. Some conversations have no connector at all and I simply paste them.

That asymmetry is why the local files are not redundant once you have connectors. A connector solves whatever is connected; the files are the only place everything else can land. They are also what makes the context outlive the tool: if I switch note-taking systems tomorrow, whatever has already been captured is still there.

The part I care about most is separating the current state from its history while preserving the reasons and decisions. They are two different artefacts and it pays not to merge them:

	Archive	Maintained memory
What it holds	every communication, in full	only what still applies
How it's ordered	by date	by topic
Question it answers	what happened, and when?	what do we know today?
When something changes	one more entry is added	it's rewritten, the old marked superseded
What it's for	investigating, checking, citing	getting to work

I keep that up at two moments. When new material arrives, it is archived as it is and whatever it changes in the current state gets updated. And when a working session ends, whatever was decided during that session goes down into the memory too.

Both are needed. The first captures what happens outside; the second, what happens while you work. With only the first, the memory never records the decisions you made yourself. With only the second, the memory has no idea the client changed their mind on Tuesday.

Right now I supervise both distillations. Automating them is the next step, not something I have already solved.

Imagine that in the meeting on the 17th we agree to deliver an integration on Friday the 18th. That meeting note is archived with its date, and the memory now says delivery is Friday. Two days later an email arrives: there is a dependency on the client and delivery moves to the following Tuesday. The email is archived with its date, exactly as the note was. And the earlier memory entry is not deleted:

## Integration delivery

Date: Tuesday 22                      [decision · 19 Sep]
Blocked by: client dependency
Outstanding: confirmation of the staging endpoint
Source: email 19 Sep — client

~~Date: Friday 18~~                   [superseded · 19 Sep]
Source: meeting record, 17 Sep

Six lines doing three things at once: they say what currently applies, they let you see what came before, and they point at the source of both. Whoever picks up the project finds the live date first, along with what is still needed to meet it, which is what they need in order to start working. If they need to understand the change, the earlier agreement is one step away. Merging the two forces you to read everything just to work out what still stands.

This is why the date of a note, where it came from and whether it records a proposal or a decision matter a great deal. Someone floating a date is not the same as the team agreeing to one, and in a summary those two look far too similar. A summary can be useful and wrong. Being able to return to the email or the meeting record helps prevent an agent's interpretation from turning into an agreement nobody made.

There is one last decision that looks administrative and isn't: whether those files get committed. I don't always commit them. While they sit uncommitted they are my memory, and they work just as well for my own work. The moment they enter the repository they stop being mine and become the team's context, available to anyone who opens the project. Same technical gesture, completely different purpose.

What holds that gesture back isn't laziness, it's that it forces you to decide what may go in. A Slack thread carries names, an email may carry client data, a meeting record captures things people said without expecting them to be written down. Preparing that material to be shared is real work, and it is precisely the work that makes the handover possible. Until it is done, what I have is a very comfortable personal practice that is no use to anyone else.

This helps me too

None of this requires anyone to be joining. I forget things too. I also return to projects after several weeks and need to recover why we decided something. Keeping that record reduces the work of getting my bearings again.

That is the part I didn't expect: when the current state is written down somewhere, changes become visible. If the memory says delivery is Friday and an email turns up assuming a different date, the contradiction surfaces. When everything lives in your head, that same contradiction resolves itself quietly, usually in favour of whatever you read most recently.

For someone new, the difference can be greater. We are asking them to continue conversations they were never part of. If they also have to discover where those conversations happened, who remembers what and which parts still matter, much of their onboarding becomes a search for context.

Access to that material does not instantly bring anyone to your level. Experience, practice and guidance still matter. But it lets that guidance focus on developing judgement instead of reconstructing what has already happened.

This is where preparing a handover starts to look like a daily practice rather than a farewell task. Every important decision that remains easy to find and up to date is something you will not have to explain from scratch later. It helps when you take a holiday, bring in support or simply want a colleague to make progress while you are busy.

The company has to make room for it

If sharing knowledge always happens after the "important work" is finished, it will be the first thing dropped under pressure. And there is always pressure.

If recognition also goes only to whoever personally unblocks every problem, preparing others to solve it will receive little credit, because its effect shows up exactly where nobody is looking: in the problems that stop escalating.

It should count as a contribution when a colleague can take on a responsibility that previously depended on you. The same applies when someone can recover a decision without calling you, or when the team keeps going while you are away. These are outcomes worth considering when evaluating someone's work, and they are hard to see if you only look at the problems that were solved and not at the ones that stopped arriving.

Where each kind of context belongs

The next technical step would be to move this memory into a shared environment: several agents consulting the same context, and updates reaching the project even when its lead did not attend a meeting or write that code.

Before that there is a more basic question, and it is the one I have open: where each kind of context belongs.

For what belongs to the project — dates, agreements, client constraints, what was ruled out — the repository itself is a reasonable home. It sits where the work sits, it is versioned along with it, and it reaches only the people who already have access.

But there is another half that belongs to no single project: how a change gets reviewed, what is checked before calling something finished, which technologies we have tried and which we dropped, the skills and tooling I have been refining. That applies across every project at once. Copying it into each repository guarantees the copies drift apart, and that the good version ends up being the one in the head of whoever wrote it — which is exactly the starting point I was trying to get away from.

That material needs a home of its own that can be shared, and there the questions change:

How the current version is maintained when nobody owns the file.
How a mistake gets corrected once a practice becomes obsolete.
Which information is appropriate to share with each person.

There is enough engineering behind that for another article. But you can start long before solving any of it: capturing what matters, distinguishing agreements from proposals, and making clear what still applies and where to check it. Files in a repository take you a long way.

I want my contribution to show in what others can do afterwards, too. If someone can carry on my work with good judgement because I prepared the context and helped them learn, that independence is part of a job I have done well.

Making yourself replaceable is an ability worth keeping on the team.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

When AI chooses the questions

JaviMaligno — Wed, 23 Sep 2026 13:39:52 +0000

On September 21, OpenAI announced that an internal model had solved more than a hundred open mathematical problems. Two weeks earlier, it had published a proof of the Navier–Stokes problem, accompanied by a Lean formalization. The announcements offer different kinds of evidence: the latter gives us an argument to study; the statement about a hundred problems includes neither a list nor their proofs. The Clay Mathematics Institute, meanwhile, is continuing its evaluation process for the Navier–Stokes result.

I have already written about what the Navier–Stokes result means. What interests me now is what comes next. If AI can answer questions we have spent decades trying to solve, what new questions will we be able to ask? And if it also learns to choose them better than we do, what will participating in mathematics mean?

To get there, it helps to start with something the headlines tend to take for granted: why open problems exist, and why some of them matter to us.

Where the questions come from

A problem is open when we do not know a solution that answers its formulation with the required guarantees. That may be because we lack the techniques, because we have not found the right way to frame it, or because hardly anyone has worked on it. The age of a question tells us how long it has been around; on its own, it does not measure how much intelligence answering it requires. And a pattern observed across millions of examples may still fall short of a proof that it always holds.

There is no final inventory of everything left to discover. Every definition makes questions possible; every theorem invites us to examine its assumptions; every connection between two fields gives us things we previously did not even know how to ask. Solving problems changes the conditions under which the next ones arise.

We can see this without reaching for a famous conjecture. If a proof uses a symmetry assumption, we can investigate which parts of the result survive when we remove it. If a counterexample appears, we can try to identify what makes it fail and which cases remain valid. If the argument works on seemingly different objects, we can look for the structure they share. A specific answer can grow into a theory.

Counting open questions and solved questions would therefore be a rather poor measure of progress. Variations on a statement are easy to manufacture. The difficult part is finding a question whose answer changes what we are able to understand.

Who decides what is worth studying

No central authority makes that judgment. Researchers propose problems; others decide to spend time on them; seminars, journals, PhD supervisors and funding amplify some directions more than others. Famous lists make a selection visible. Clay itself explains that it chose its problems for their depth and their potential to drive new structures and methods, with consequences extending beyond the original question.

There are criteria we can discuss: how much a problem unifies, which obstacles it helps us understand, what techniques it might unlock, what applications it suggests. Beauty, surprise and the appeal of exploring something with no recognizable use also play a part. These criteria can conflict. A problem may be fruitful for one field and peripheral to another. And the prestige of the person proposing it may attract attention that an equally good question elsewhere never receives.

Mathematical judgment develops through work: seeing which attempts fail, which assumptions do the real work, and which ideas survive a change of example. Recognizing famous names is only a small part of it.

AI can participate throughout this process. It can search for counterexamples, compare cases, suggest a generalization and help us notice that two results express something similar. There are precedents that predate today's models: a 2021 study by Davies and colleagues in Nature used machine learning to detect relationships that guided new conjectures and results in knot theory and representation theory. Mathematicians interpreted and developed those clues. The collaboration helped formulate new mathematics.

My expectation is that more capable tools will expand the questions we can tackle. A direction that once seemed impractical can become a viable research project if exploring examples or proving preliminary lemmas costs much less. But there is no guarantee that this capability will be spent on the most fruitful questions. A system rewarded for accumulating solutions has an incentive to select problems it can close. A system rewarded for making an impression has an incentive to select recognizable names. Neither objective necessarily coincides with producing understanding.

What if AI develops its own judgment?

Here comes the reassuring answer: we will supply the judgment, and the machine will do the work.

I do not think we can assume that division will last.

Part of judgment is anticipating consequences: which idea will connect results, which experiment will distinguish two explanations, which question will open a line of research. I see no sufficient reason to declare that AI can never learn to do this. Nor is a model proposing ten sophisticated-sounding questions enough to establish that it already can. The evidence would come from following its proposals: checking whether they produce reusable methods, unexpected connections and worthwhile subsequent work.

Two transitions are worth distinguishing. One is learning to choose well by the criteria we already use. Another is proposing a direction those criteria initially reject, then giving us reason to revise our judgment. The second looks more like developing its own mathematical taste. Recognizing it would require time and results; asking the model whether it feels curious would not settle anything.

The fact that a criterion was learned does not automatically disqualify it, either. Our own taste develops through reading, teachers, examples and institutional rewards. The practical question is whether the system can revise what it has learned in light of its discoveries, and whether its choices remain valuable beyond the evaluation it was trained for.

Even then, identifying a mathematically promising direction would differ from deciding how much we want to invest in it, who should have access to its results, or which human needs to prioritize. The technical ability to recommend a course does not, by itself, confer authority to set the goals. Who controls the system also matters: an agenda chosen by AI may be responding to the incentives of the company training it.

Consumers of truths

Would that turn us into consumers of truths?

We already consume many truths we did not discover. Almost everything a mathematician learns was first thought through by someone else. Yet studying a proof can profoundly change what that person is able to do. Authorship has never been a necessary condition for acquiring intuition. A machine author does not, by itself, make a result impossible to understand.

What matters is how the result reaches us. A certificate of correctness, an explanation and a technique I can reuse offer different things. Formal verification checks that a conclusion follows from definitions and assumptions within a system. Interpreting what has been formalized and why it matters requires further work. Learning to recognize when the idea is useful requires more still.

A correct proof does not, by itself, establish that someone understands the idea or that choosing the problem was a good decision.

In his 1994 essay On Proof and Progress in Mathematics, Thurston argued that mathematical progress should be examined through what it enables people to understand. His essay describes the distance between a written proof and the different ways of understanding it, as well as the effort involved in communicating those ideas. This concern predates generative models.

AI could also help with that communication: finding a simple case, explaining where an intuition fails, constructing a counterexample or searching for another proof. Human understanding could grow through results nobody would have obtained unaided. To know whether that is happening, we would need to look at what the reader can do afterwards: recognize a new situation, adapt the argument, identify a limit. Feeling that an explanation is clear is not enough.

A harder scenario is possible too: correct results whose methods we can barely absorb, or production advancing much faster than our ability to study it. We might use some consequences without mastering the entire mechanism. If AI also became better at finding applications, human understanding could cease to be necessary for certain parts of technical progress.

That would not make understanding worthless. It enables participation in decisions, teaching, debate and intellectual independence. Understanding something can be valuable to the person who understands it even if another intelligence could do it better. We do not have to prove that learning makes us economically irreplaceable in order to want to keep learning. But we should not promise that preserving this value will, by itself, solve the problem of academic employment.

What academia would have to reward

This is where universities enter the picture. Part of research training involves learning through work that leads to an original result. If that result can be obtained with much less human involvement, we will need to assess more directly what the researcher has learned and contributed. Increasing publication requirements would preserve the metric while its meaning deteriorates.

The scale already deserves attention. arXiv went from 185,692 new submissions in 2022 to 284,486 in 2025: an increase of approximately 53%. The trend continues in 2026: adding up arXiv's monthly statistics gives 230,322 submissions from January through August, compared with 181,595 in the same eight months of 2025, a 26.8% increase. September is still incomplete as of the access date, September 21, 2026.

These are submissions to the repository across its disciplines, not peer-reviewed mathematics articles. The series does not isolate AI's effect; the 2025 report identifies the rise in AI-generated manuscripts as a challenge for the platform.

Author’s visualization using arXiv's monthly CSV, accessed September 21, 2026. January–August is compared across all years: new submissions, not revisions or peer-reviewed publications.

In my own case, I have one paper from before I used AI and four since. That is a sample of one, with different projects and time periods. What I can describe more precisely is in my experience doing research with AI: models participate in planning, execution and scientific review. The number of completed documents does not, on its own, tell us which questions I chose well, which errors I learned to detect, or how valuable the results are.

That is why I think the academic model that uses paper volume as a substitute for intellectual contribution is rapidly losing its justification. Reform would need to give more weight to contributions that can be examined: a well-motivated question, a reusable tool, an independent check or an explanation that enables others to work with an idea. It would also need to recognize the time spent reviewing, organizing and teaching what is discovered.

Training would need real opportunities to practise reasoning, alongside opportunities to use these tools with judgment. Asking someone to adapt a proof when an assumption changes tells us something different from asking them to hand in a correct text. When AI participates, it matters how the question was chosen, how the answer was verified and what the researcher can explain about its limits.

The mathematicians' letter published on September 11 raises a related concern: using open problems as a benchmark may favour answer production while weakening the understanding and training those problems helped develop. I think that warning deserves serious attention. An abundance of results should come with resources to study and communicate them, and access to the tools used to produce them.

What interests me about AI in mathematics is how far it can extend our ability to ask questions. First, by helping us explore what we currently cannot. Later, perhaps, by proposing directions we would not have known how to value in advance.

If that second moment arrives, our role will not be guaranteed by some permanent inability of the machine. It will also depend on what we choose to build around it: institutions that make learning possible, ways to turn results into shared ideas, and the ability to influence the goals of research. Having more truths available will be an achievement. Making them our own will remain a task.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Jev: what survives the hype

JaviMaligno — Tue, 22 Sep 2026 13:25:54 +0000

In one of the systems I work on, a model reads invoices and extracts prices, amounts and other fields. I then need to check something much smaller: is that value actually in the invoice? I do not need another model to write a report. I need a decision.

That is the gap Jev, from TypeSafe, fits into. I give it information and a question with bounded answers; it returns a category, a score or a probability. The promise is to make those decisions much faster and more cheaply than a large generative model.

Jev has just launched and attracted plenty of hype. I wanted to see how much survived testing it on things I actually do. I have used it at three levels: my work with agents, the products I develop, and the environments where I would have to deploy it. Each has exposed a different boundary.

Personal: helping me work with agents

When I use Claude Code or Codex, different tasks need different models. Finding a file is different from reviewing an authentication change. There are also decisions before running commands: reading a file, deleting data and deploying an application have different consequences.

In Claude Code, I have connected Jev to both points. It recommends a model tier for a task and assesses command risk. I run it in shadow mode: it records what it would recommend, while the actual decision stays with the agent and me.

In an initial test with invented tasks, it distinguished clear examples well: searches and mechanical edits on one side, architecture or security work on the other. The queries cost a fraction of a cent. That lets me collect recommendations cheaply; finding out whether cheaper models are worthwhile also requires measuring whether they finish the work correctly and how often it needs to be repeated.

In Codex, I ran a comparison using actual agent work on a test project: fixing a pagination bug and a discount calculation. I ran each task with the same model, once with Jev and once without it. For the Jev version, I connected it as a tool that Codex consulted before using the terminal.

Codex solved both tasks correctly in both conditions. But in these runs, consulting Jev took roughly twice as long. The Jev requests cost a fraction of a cent; the additional work came from the exchanges the agent needed to request and read each assessment.

It is a small test, but it gave me a concrete result: adding an opinion before every read or test did not improve those fixes, and it made them slower. That integration is useful for experimentation; it has not yet improved that workflow for me.

The distinction between those uses matters to me. Choosing the right model can save an expensive task. Consulting a model before every step adds work even when the answer was obvious. For now, I see more value in reserving Jev for decisions that change what I am about to do.

Product: detecting deception and checking data

SMS: telling a legitimate notification from a scam

One of the systems I work on analyzes SMS messages for fraud: messages impersonating an organization to get someone to open a link, disclose credentials or make a payment.

I tested the Jev integration with 80 Spanish messages, mixing legitimate messages and scams. The complete system correctly classified 78 of those 80. Both mistakes were scams it missed; it did not flag any legitimate message in the test as fraud.

Reviewing the two misses showed me where to improve. One message fell just below the alert threshold when the scores were combined. The other went back to the previous detector because Jev's answer was not certain enough, and that detector missed it. The test pointed me toward two concrete changes: how I combine Jev's assessment with the other signals, and how I review uncertain messages.

Invoices: checking is different from asking “are you sure?”

For invoices, I wanted to know whether Jev could find errors in another model's extracted data. I supplied the invoice text and the values I wanted checked.

That check was much more useful than the confidence reported by the extractor itself. I found incorrect values, or values absent from the document, that the extractor had assigned very high confidence. Comparing them against the text, Jev flagged many of those problems.

The limit was what I meant by an “error.” A field can be correct even if the extractor normalizes its format or combines information from several parts of the document. In the initial test, roughly half of Jev's alerts were false alarms. Refining what it should check made the review more useful.

I see a useful component here: an inexpensive second reader that flags suspicious fields for review. I would not treat every flagged field as incorrect.

Email: deciding when expensive analysis is necessary

For email, I already had a second review with Sonnet for suspicious messages. I tried placing Jev before it: let Jev settle clear cases and leave uncertain ones to Sonnet.

On the evaluation set, Jev resolved 56% of emails without needing that second call. I observed no errors in the cases it handled on its own. The estimated cost of that stage fell from about \11.90 to \5.30 per thousand emails.

I can connect that saving to a specific decision: avoiding expensive calls that are unnecessary. The test included reconstructed attacks, so it supports an evaluation on live traffic rather than a claim that it will catch every new campaign.

I also tested Jev for reviewing suspicious transactions. In a test with twenty simulated frauds, it detected seventeen, the same ones as Haiku, and produced the same false alarms on legitimate transactions. It did so faster and more cheaply. That is promising for a second opinion, with the obvious limitation that the frauds were simulated.

Where it did not add enough

Not everything that looks like classification improves when I add Jev.

I tested it for selecting the correct material among catalog candidates, and it recovered none of the human corrections I was looking for. On fraudulent domains with a single changed letter, it missed cases the previous model detected. When selecting chatbot tools, ambiguous tool descriptions still led to wrong choices.

I also evaluated whether it could decide which user feedback should become a ticket. Creating an unnecessary ticket is annoying; discarding a real problem can make it invisible. The test did not contain enough examples of that second risk. I am interested in using it to propose tickets, but I lack evidence to let it discard reports automatically.

My reading is concrete: it works better when I provide the necessary evidence and well-defined options. If the catalog is ambiguous or information is missing, an inexpensive model still faces a poorly specified problem.

Infrastructure: cases that work but I still cannot deploy

My industry classification service investigates what a company does and assigns it a category. That process contains several classification and source-verification steps where I tested Jev. My experience was that it produced results equivalent to GPT‑5.6 Luna, faster and more cheaply.

Data policy nevertheless limited adoption. I encountered something similar in the industrial pilots: the client's environment required an approved route through Azure Foundry, and the integration I was testing did not meet that requirement.

That boundary is easy to forget while looking at a results table. I can establish that a model performs a task well and still be unable to send it the data it would need.

TypeSafe's privacy policy says the service is hosted in the United States. It offers a data processing agreement, commits to not training on customer data without consent, and provides zero retention for enterprise customers. None replaces a requirement to process data in a particular region or through an approved provider.

Using OpenRouter does not resolve that by itself either: I need to check the terms of the provider processing the request.

Other limits can be addressed in the design. If I need to add amounts or compare dates, I do it in code. Jev is intended for judgments about text, and its own documentation warns about difficulties with arithmetic, irrelevant context and adversarial instructions. I give it only what it needs and keep an alternative for failures or answers that are not clear enough.

What remains of the hype

Jev has not changed my life, but it has made parts of my work faster and cheaper.

I am left with a useful tool for some small decisions I was paying a large model to make. Document verification and filtering before an expensive analysis are the most convincing cases for me. In my work with agents, I am still finding where the extra query pays off.

My bet is that OpenAI, Anthropic, Google or others will eventually offer comparable one-shot classification models. It would make sense to me: many applications need to choose among a few options quickly and cheaply.

If they appear, I want to compare them on these same decisions. What I want to keep is the questions, the tests and the criteria for deciding when to trust an answer. Jev has passed some of those tests. In others, I prefer what I already had, and in some the limit comes from the environment I work in.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Frontend, Backend, and the Agentic Engine

JaviMaligno — Mon, 21 Sep 2026 15:07:30 +0000

A separation keeps appearing in projects I work on. There is the frontend. There is the backend that keeps the application running. And there is the agentic engine: workflows, prompts, tools, context management, models, and the evaluations that tell us whether all of that does its job well.

I could call the last two pieces the backend and remain technically correct. But the distinction is increasingly useful when working on them. Changing how an agent investigates a data source is different work from changing how a user reviews and accepts its results. Both need server code; their responsibilities and reasons to change are different.

These are three blocks of responsibility. They can live in a monorepo, span repositories, or share a process. Giving the engine an identity of its own does not require making it a microservice.

The application around the agent

One example is a data source automation pipeline. Its work includes investigating sources, proposing extraction methods, generating specifications, and producing services. There are several stages, tools, decisions, and review points. There is also an evaluation set that compares stage outputs against corrected references.

The application from which that work is managed has its own frontend and backend. The backend submits jobs to the engine, checks their status, and retrieves results. The logic that makes those results part of an application has enough substance to remain separate from the logic that produces them.

This is the distribution of responsibilities I am interested in:

Two possible arrangements. Boxes express responsibilities; arrows express exchanges. They specify neither the number of repositories or processes nor every communication path.

The application backend can have plenty to do. In a document application, for example, it determines who can open a case, which review it needs, and when a result becomes accepted. The engine extracts information and supplies evidence. A completed run does not necessarily mean a resolved case.

In another document processing project I work on, extraction runs in a worker, and the application backend manages cases and their artefacts. For questions about previously processed documents, that same backend imports the AI service as a library. The separation of responsibilities accommodates both integration styles within one product.

A chat can hide an entire system

In the conversational AI projects I wrote about when discussing interfaces built inside a chat, the visible surface is a conversation. Behind it are document capture, tools, verification, persistent state, and processes that need human participation.

Some of those engines share a library with common capabilities: persistence during execution, activity events, context management, model connections, and protection against loops or repeated failures. Each engine keeps the tools, instructions, and structures specific to its domain.

That introduces another reason to give the agentic block an identity: several experiences can reuse the same execution capabilities. The shared core is a library; each consumer incorporates it into its service. A recognisable boundary in the code already provides value.

This also calls for more precision about what we mean by “a simple chatbot.” Chat describes how the user interacts. It says little about the system they are using. In I Had Built an Expensive Form, I described how a conversation could rest on a graph of phases, validations, and decisions, yet still offer a worse experience than a form. Engine complexity and interface usefulness are separate questions.

The agent can also operate on the application

An assistant embedded in a case review platform introduces another relationship. The analyst is already working on a case and opens the assistant within the application. They can ask it to look up information, update details, or propose a status change. The agent uses backend capabilities to work on that same case.

In the integrated implementation, tools are adapters over product operations. For example, the tool that updates a case validates the data and calls the existing update service. The tool that changes status checks that the transition is allowed from the current state. Writes go through a confirmation card and leave an audit record.

The assistant provides another way to operate on the product. This is a map of responsibilities: conversation and tools pass through server code, and writes require confirmation in the interface.

Here the backend provides capabilities consumed by both the screens and the assistant's tools. The useful boundary separates interpreting the request from executing the business operation. In this case, the assistant module lives within the application's own backend: that distinction exists without a separate agent service.

This broadens the initial picture. An application can delegate work to the engine, and an agent can use application operations as tools. Both relationships can coexist. The three blocks help assign responsibilities, but they do not impose a single chain of calls.

The case where two blocks were enough

The old frontend for a business activity classifier was a way into the agent: submit a query and see the classification. There was little intermediate logic beyond authenticating access.

That case worked well as frontend plus agent service. A separate application backend would have needed a concrete responsibility to justify maintaining it.

“Direct access” here means the frontend communicates with the service running the agent. Provider credentials and tool execution remain on the server. The interface client presents results and events; the engine prepares context and executes the work.

I find this example as useful as the others because it prevents a practical observation from becoming a universal recipe. Even a complex engine can have a very thin access interface.

What is new about this

The separation between an application and a processing engine has clear precedents. The Web–Queue–Worker pattern already separates request handling from long or intensive work. Applications with inference services also know that boundary.

There are explicit references in the agent ecosystem. LangGraph Cloud launched in June 2024 with persistence, background jobs, streaming, and human collaboration. CopilotKit's architecture describes a frontend, a runtime inside the application server, and an agent backend. That runtime covers integration with the interface; the business backend can have broader responsibilities.

What I see in my projects is AI logic acquiring enough substance to need an engineering cycle of its own. Changing a model or prompt calls for evaluating the quality of results as well as checking that requests work. A flow can finish without errors while choosing the wrong tool, omitting a fact, or consuming too much budget.

That cycle combines software tests, evaluations on representative cases, and inspection of runs. It is a reason to recognise the engine as a block, even when it shares infrastructure with the application. The separation still needs tracing across the whole system: a run must be traceable to the user's work that initiated it.

I have not found one established name for this exact arrangement. “Frontend, application backend, and agentic engine” describes what I want to point to. Also, part of that engine may be a workflow whose path is fixed by code. Anthropic's distinction between workflows and agents is useful here: this boundary can make sense for both.

Which boundary is worth drawing

How central AI is to the product and how complex it is both help with the decision, but neither is sufficient alone. An application can depend on a single, simple AI operation. Another can offer automated research as a secondary feature and need several complex workflows to deliver it.

I would look for concrete signals:

The AI logic has tools, context, and evaluations that change independently of the rest of the product.
Runs need to last, resume, or wait for human input beyond a web request.
Several processes or experiences reuse the same engine or its common capabilities.
The application has permissions, reviews, and business states that retain their meaning when the agent's approach changes.

The first three give the engine substance. The last gives the application backend substance. The classifier example illustrates why both questions matter.

Technology does not define the boundary either. The engine can have APIs, queues, and plenty of deterministic code. The backend can administer configurations and documents consumed by agents. A tool can invoke a business operation whose permissions and rules are enforced by the service responsible for that operation.

What needs to be clear is who decides what, which state each part maintains, and which contract lets them work together. That contract covers inputs and results, but also progress, errors, cancellation, and review when the product needs them.

Separating processes adds costs: communication failures, compatible versions, and synchronisation. If a retry creates two jobs, three well-drawn boxes will not solve the problem. I would therefore start with a boundary between modules and separate deployments when there is an operational reason.

For a small feature, an AI module inside the backend may be enough. For an interface whose only job is to provide access to the agent, its service may be enough. When both the product and the engine accumulate responsibilities of their own, recognising the three blocks helps us work on each without unnecessarily pulling the others along.

The question I find useful when reviewing these projects is: if we change how the agent works tomorrow, what would need to change in the application, and why? The answer says much more about the architecture than counting repositories.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

The Bug Nobody Can Reach

JaviMaligno — Thu, 17 Sep 2026 13:29:24 +0000

Suppose the map your system plans on is missing a room. Not "slightly off about the room" — the room is not on the map at all. What does that cost you?

I have spent a few months making that question precise, and the answer turned out to be narrower and stranger than I expected. It depends on exactly one thing, and it is not the size of the error, not how confident the model was, not even whether the missing thing is dangerous. It is whether anything that plans on that map can get to the room.

This is the short version of a preprint (arXiv:2608.28541); the long post has the same story with the numbers, the proofs and the parts that went wrong. Here I want just the one idea, because it is the one I would actually use.

The setup, in one paragraph

A small robot on a plane. Somewhere on that plane there is a band it must not cross — a fence around a high-value spot it would otherwise drive straight at. A language model is handed the physics and asked to write the simulator the planner will use, and the description it receives simply leaves the fence out. Then the model's simulator is tested: run the real system a few dozen times, check that the written code predicts every step exactly. If it does, the code is accepted. That is all a "test suite" is here, and it is exactly what one is in practice.

Fences, containment shells, geofenced no-go zones: that is the shape safety-critical omissions actually take, and it is why I stopped using walls and started using rings.

The setup. The robot wants the high-value spot; the fence around it is real but absent from the description the model was given, so the code the model writes says the way is clear.

When being wrong is free

Close the band fully — a complete ring around the spot — and here is what the model writes: not a ring, but a filled disc. The whole interior marked as forbidden, when in truth only the rim is. Wrong about the shape of the world, not by a millimetre but categorically.

Two things are true about that wrong model, and they are the reason I wrote the paper.

No test can catch it. Not "we got unlucky", not "you would need more samples". There is a proof. The fence stops the robot on contact, so no run that starts outside can ever end up inside; therefore no observation that any test could ever make distinguishes the filled disc from the truth. You can run a million samples at any tolerance you like. They agree, always, because the place where they disagree is a place nothing can reach.

It costs nothing. The planner trusting the filled disc picks the same action at every step as a planner holding the true map: same route, same result, same contacts, run for run, seed for seed. Not approximately — identically.

So: certified, wrong, and free. Those three usually travel together in our heads; here they come apart cleanly.

A word on how I measure the cost, because it makes the rest readable. I compare what the planner earns against two references: what it would earn holding the truth, and what it would earn acting at random. Zero means the wrong model costs nothing. One means you might as well have acted at random. Above one means the model actively steered you somewhere worse than random.

The same blindness, a different world

That was one kind of wrong model: one that invents a forbidden region where nothing can go. Here is the other, and the one that actually hurts — a model that simply does not know the fence is there at all. It is the common case: the description omitted the fence, the test runs never happened to touch it, so the code came back without it.

Now the fence is on the route. The planner drives confidently at the high-value spot, the real fence stops it dead, and it replans the same doomed route every step. Cost: 1.116 — worse than acting at random, because the model is not merely uninformative, it is actively promising a route that does not exist.

Now change one thing, and it is a thing about the world, not about the model: cut a gap in the fence wide enough to drive through, and put that gap in front of the robot, on the way it already wanted to go. Same model. Same blindness. Same missing clause in the code. Cost: 0.029. Almost nothing — because the confident wrong route now goes somewhere the truth actually allows.

And to be sure the gap itself is not what did it, put the same gap — identical width, and in both cases the fence is equally not-a-closed-ring — round the back, where no route ever goes. Cost: 1.116 again, to four decimals the same as the fully closed fence.

Same model, same mistake, same shape of hole. One number is 0.029 and the other is 1.116, and the only thing separating them is whether the robot's own path crosses the gap.

The same blind model in two worlds that differ by a rotation. On the left the gap sits where the robot was already going, so its confident wrong route turns out to be allowed. On the right the identical gap sits behind the goal, the fence still blocks the route, and the model costs more than acting at random.

That is the whole finding, and the reason the slogan is reach, not shape: you cannot look at what your model got wrong — not its size, not its geometry, not even a robust structural property like "is there a hole in it" — and conclude anything at all about what it will cost you. You have to ask where the thing planning against it can go.

"But in the real world I could go around it"

That was the first objection I got, and it is the right one. In two dimensions a ring is a wall: of course nothing gets in. Maybe the whole result is an artefact of a toy where the mistake happens to be sealed off.

So the paper runs the case where going around is genuinely possible: a doughnut-shaped region floating in three-dimensional space, between the robot and its target. Nothing is sealed off — there is an explicit route that goes around it without touching it at all.

The doughnut in three dimensions. On the left the route passes through its hole and the wrong model costs almost nothing; on the right the same object has been moved so the route runs into it, and it costs nearly as much as acting at random. The dashed path arcing over the top is the one that matters for the second half of the story: it reaches the goal without touching anything, which is what makes the error catchable in principle again.

The result splits in two, and this is the version I would carry into a real system.

The danger survives. Put the doughnut so the planned route runs into its solid part and the cost is 0.898. Move it so the route threads the hole instead and the cost is 0.019 — same object, same rarity of contact, same trivial shape. Being on the path is what costs you, whether or not the thing encloses anything.

The guarantee does not. Once you can go around, there is no region a competent planner provably cannot query, so there is no longer any proof that a test could not have caught the error. It becomes merely unlikely to be caught, which is a much weaker and much more familiar situation.

Two different questions, then, and they had been fused together in my head until this experiment pulled them apart:

Does a plan cross the place where my model is wrong? This decides what the error costs.
Is that place walled off from everything I can run? This decides whether any test could ever have found it.

What I would ask of my own system

Three questions, and none of them needs the mathematics:

Where is my model wrong in a way nothing I run ever visits? That part is free today — and it is also invisible to every test I have, so I will not be told when it stops being free.
What would put a plan through it? A new feature, a new goal, a shortcut someone adds next quarter. Reach is not a property of the model; it is a property of the model plus whatever is planning with it, and the second half changes far more often than the first.
Do my tests sample where my system acts, or where it is easy to sample? A passing suite certifies the reachable part and says nothing whatsoever about the rest.

The uncomfortable version of all this: a model can be exactly right on everything you can check and arbitrarily wrong beyond it, and the difference between "free" and "catastrophic" is not a property of the error. It is a property of the plans you happen to run today.

The long version, with the danger curves, the repair experiments and a pre-registered test that came out null: Being Wrong Can Be Free. The formal version: arXiv:2608.28541, with code and every result artifact open.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Benchmaxing: Winning the Exam Is Not Doing Better Work

JaviMaligno — Wed, 16 Sep 2026 16:36:16 +0000

With Opus 5, I have experienced something that people I speak to directly have also described: better benchmark results do not necessarily feel like a more intelligent model, or one that does better work. Some tables even put it above Fable, which clashes with my experience.

That perception deserves investigation. It also deserves a test that can contradict it. If an article starts by treating it as established that Anthropic optimized the exam at the expense of real work, I have chosen the answer before examining the evidence.

So I did two things: review what is documented about benchmaxing, and run a small set of probes with Opus 5 and Fable 5. The result was less convenient than an accusation: I found interesting failures, but not the general superiority of Fable that my intuition might have suggested.

What it means to optimize for the exam

I use benchmaxing to mean directing model optimization, or the selection of reported results, towards maximizing evaluation scores. The problem arises when improving that score stops being a useful signal of improving what I need. This is an application of the Goodhart problems studied by Manheim and Garrabrant: a useful measure can become a poorer guide under intense optimization.

Preparing for an exam can teach the subject. It can also teach recognition of questions, mastery of a format, or how to please the grader. The question is what transfers to new problems. Training capabilities that benchmarks evaluate can produce real advances; a higher score alone demonstrates neither fraud nor a lack of intelligence.

At least three different phenomena need separating.

Familiarity with the test. GSM1k introduced new problems comparable to GSM8k and found accuracy drops and signs of overfitting in several model families. But it also found little evidence of overfitting in many frontier models, and generalization in all the models evaluated. This does not establish that models only memorize. It does invite the question of how much of a score depends on having seen something too similar.

Selective reporting. When introducing Llama 4, Meta highlighted an LMArena Elo of 1417 and specified that it belonged to an experimental chat version. That qualification matters: a score for one variant does not automatically transfer to another. The Leaderboard Illusion documented private testing and selective disclosure, identifying 27 private Meta variants before Llama 4. A ranking can be distorted when only the selected result is visible without seeing all the attempts.

The gap between the metric and the work. In a randomized METR study, 16 experienced developers completed 246 tasks: allowing early-2025 AI tools increased completion time by 19%, although participants believed they had saved time. This does not demonstrate benchmaxing or describe all AI-assisted programming. It shows why productivity needs direct measurement. The February 2026 update also identified selection biases in the follow-up study and considered its signal unreliable. Repeating the 19% as a description of current models would misread that evidence.

What I can say about Opus 5

Anthropic presents Opus 5 as close to Fable 5 and stronger on certain evaluations, including OSWorld 2.0. Its Frontier-Bench note specifies an internal run, a particular environment, mean reward over five attempts per task, and Opus 4.8 as a fallback for safety blocks. A score describes those conditions; it does not automatically describe an everyday conversation.

There are also public accounts resembling my perception: one user reports incorrect diagnoses and confusion between code comments and actual behavior, preferring Fable for investigation. The same thread contains favorable opinions of Opus. My experience, direct conversations and that thread are sources of hypotheses, not a representative survey.

Where I notice the difference most is not in a bounded answer. It is in multitasking and in managing flows of agents: a Claude Code session with several tasks in flight, subagents to launch and wait for, results to fold back into one deliverable, and a decision about what to do while a slow step finishes. None of the benchmarks in the announcement measures a model managing other agents, so the table and my perception are looking at different work.

Public instruments for that work exist, and they are recent. ClawArena-Team scores a text-only conductor that creates, empowers and schedules a pool of subagents across 41 multi-turn scenarios, and must integrate their returns into a correct deliverable rather than relay them. Fable 5 leads its twelve models with a subagent-management score of 60.0%, ahead of Gemini 3.5 Flash at 53.8% and GPT-5.5 at 51.0%, evaluated as shipped with the vendor-recommended fallback to Opus 4.8 on refusals. Opus 5 is absent: the paper was submitted on 30 June 2026 and Opus 5 shipped on 24 July. AsyncTool is closer to multitasking proper: concurrent tasks with delayed, out-of-order tool feedback, where the question is what the agent does with the idle time. It includes no Claude model, and its overall leader is GPT-4.1 at 38.06, ahead of GPT-5 at 31.32. Neither benchmark tests my perception. They show that the dimension where I feel the gap can be measured, and I found no published run that puts both models on it.

Opus winning some tests while Fable proves more useful for other work can be entirely coherent. Solving a bounded assignment and correctly discovering what needs solving make different demands. I wanted to see whether that distinction appeared in concrete cases.

A pilot allowed to contradict me

I compared claude-opus-5 and claude-fable-5 through Claude Code on a Max subscription, at high effort and with the same 8192-output-token limit. The cases were synthetic, supplied entirely in the prompt, and the models had no tools.

The set contains five cases per model: an initial double-charge incident, two variants involving external effects and paused workers, a written sequence of permission changes, and numerical analysis with different task mixes. There was one valid run per model and case. The two worker variants explore the same mechanism; they are not independent replications of the whole user experience.

The expansion's criteria were saved before execution. Those cases were written after seeing the first one, so the entire exercise is exploratory. The prompts, answers, criteria and counterexample check are available for inspection.

Qualitative assessment of the core criteria. One run per model and case; D2b is a variant, not an identical repeat. “Covered” does not mean perfection or general performance.

The initial case already gave me a reason to question the intuition. Opus proposed recording a durable intent before charging, although it left the handling of uncertain states incomplete. Fable retained a gap between the external effect and the record, treating a duplicate charge after deduplication expired as a risk to accept. Avoiding a second attempt, even while leaving work pending, was an option that answer did not develop.

The failure they shared

For the next two cases I made the priority explicit: never produce the external effect twice, even if an uncertain operation remains blocked. The provider remembers a key for a few hours; workers may crash or remain paused indefinitely. The provider offers no mechanism to invalidate an old worker's authority.

Both models recognized much of the problem. They proposed stable keys, persistent states and stopping retries after a deadline. But both retained authorized retries while the old worker could still be alive, relying on a local deadline check and a time margin.

The race that breaks this proposal takes four steps.

A local check cannot prevent A from pausing immediately afterwards. If B has already produced the effect and the key has expired, A’s late send produces a second effect. Rejecting its database write is too late.

A checks that it may still send, then pauses immediately afterwards. B takes over, sends, and finishes. The provider's record of the key expires. A resumes and sends what it had already decided to send. The provider produces the effect again. Preventing A from updating the database does not undo a printed letter or a prepared package.

Fable acknowledged this residual window; in one answer it said the margin made it “negligible, not impossible.” But the case allowed indefinite pauses and required at most one effect. There was no distribution of pause durations that justified calling it negligible. Opus left the same gap: in one variant it called the check best-effort, then described prevention as guaranteed.

A conservative alternative exists under those rules: durably grant a single emission permit, never transfer it, and never retry an operation that might have been emitted. If the process crashes before sending, nothing may happen; the case explicitly allows that loss of automatic progress. My executable check produces two effects with the described takeover and one with a non-transferable permit. It simulates the logic of their answers; it is not code implemented or executed by the models.

The interesting gap is between identifying the right concepts and closing the guarantee being promised. An answer can mention all the expected patterns and still require a substantial correction.

Ties count too

On permissions, both resolved the eight core decisions: tenant and owner restrictions, bounded exceptions, the excluded time endpoint, and amount redaction. Both also detected that a cache shared by tenant could leak data and that permissions and later data changes needed revalidation.

On data analysis, both performed the requested calculations and rejected the misleading comparison: one system looked better in aggregate because it had received many more easy tasks. Both included the cost of repairing failures and distinguished a projection from a causal conclusion.

Those cases did not separate the models on the core criteria. I did not discard them or keep increasing the difficulty until I found a winner. They mark a limit of the instrument: tasks that looked demanding proved insufficient to distinguish the models in this sample.

What remains of the suspicion

My initial perception remains a valid experience. These tests do not turn it into a demonstration that Opus 5 has been benchmaxed, and they do not establish that Fable is generally better. I did not measure long sessions, repository investigation, orchestration of subagents, or minutes of human supervision, which is where that perception lives. The pilot tested bounded reasoning. I did not inspect either model's training.

I did find something concrete: two capable systems can diagnose part of a problem correctly and promise more than their proposed solution guarantees. In earlier work on verified world models, I explored a different mismatch between passing a check and being adequate for the intended use. The mechanism differs, but the question is again what the metric actually licenses me to conclude.

The literature gives good reasons to take benchmaxing seriously. The pilot requires greater precision when applying it to a particular model. To choose a tool, I want to know whether it reaches the right diagnosis, preserves constraints, and reduces the corrections I have to make. A table can provide evidence about those abilities. The further its evaluation sits from my work, the more that transfer needs checking.

Method note: cases and evaluation prepared with Codex assistance; qualitative review by the same assistant, neither independent nor blind. The ten compared answers and limitations are in the evidence repository. Diagnostic attempts and the truncated calibration run are retained separately and excluded from the comparable answers.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Being Wrong Can Be Free — Until the Planner Can Reach It

JaviMaligno — Tue, 15 Sep 2026 13:35:01 +0000

Last week I wrote about a model that infers the rule you forgot, but only in one dimension. The practical rule I ended on was that coverage of the boundary is the whole game: your sampling gate certifies your model where your samples land, and a rule with a shape you still have to specify.

That leaves a question I could not answer with the instruments in that paper. All of those wrong models were wrong somewhere a planner could get to. What happens when the part the model gets wrong encloses something nothing can ever reach? The answer turns out to be sharper than "it's probably fine", in both directions: the error becomes provably uncatchable and provably costless — and then a door 0.1 radians wide, in the right place, undoes the second half while leaving the topology untouched. That is a preprint, An Enclosed Mode Is a Gauge Choice (arXiv:2608.28541), with the code, result artifacts and Lean proofs open.

A mode with an inside

The instrument is deliberately minimal: a thrust-and-drag mover on a plane, and an annular band — inner radius 3.5, outer radius 5.0 — that freezes the mover the instant it touches. Inside the ring's hole sits a high-reward "lode" the planner would love to visit. The spec handed to the language model pins the physics exactly and simply omits the band, exactly as before.

The reason for a ring rather than another wall is that this is the shape safety-critical omissions actually take. Fences, containment shells, geofenced no-go regions: a boundary drawn around something, with an inside. And a pipeline that cannot tell a fenced void from a fenced hazard — or a fence from a filled wall — is certifying less than it looks like it is.

Three knobs, all fixed before any run: the width \gamma of an angular channel cut through the band (at \gamma = 0 the ring is closed), whether that channel faces the start or hides behind the lode, and whether the mover starts outside the ring or inside its hole. Everything in the paper follows from what those knobs do to one object: the set of state-action pairs a rollout can actually query.

The instrument, drawn to scale, under the knobs that matter. Closed: the planner is stopped at the rim and the inside is unreachable. Facing: the same band with a channel the planner can drive through. Hidden: the same channel, same width, rotated behind the lode — topologically identical to the facing case, and unreachable. Inside: the mover starts in the hole, so the interior is sampled and the omission becomes falsifiable.

Beyond reach is gauge

Here is the theory in one sentence. If a gate accepts every candidate whose sampled transitions match, then acceptance-with-certainty pins the model down exactly on the reachable query set — and everything beyond reach is gauge, in the physicist's sense: a free choice that changes no observable. Two models that differ only out there are the same model as far as any sampling gate can ever be.

On the closed ring that has a limiting case you can hold in your hand. The natural wrong artifact is a filled disc: no hole at all, the whole interior frozen. It is wrong about the topology, not just the parameters. And it is:

unfalsifiable by any sampling gate. Not "unlikely to be caught" — there is a proof, and it needs no assumption about sample size or tolerance. Because the band freezes the mover on contact, no rollout that starts outside can ever be inside the hole, so no possible transition distinguishes the filled disc from the truth.
bitwise harmless at play. The planner that trusts the filled disc plans identically to the planner that knows the truth: same action at every step, same return, same final state, same contacts, seed for seed. Paired-seed MPC episodes confirm it exactly, not approximately.

Why no sample can separate them. Every rollout that starts outside is stopped at the rim, so the sampled transitions are identical under both models; the two differ only inside the hole, which is exactly where no rollout can be. This is the proof, not a sampling accident.

So certification, correctness and consequence come apart three ways rather than two. This artifact is certified, wrong, and free. My previous two papers had shown certified-and-wrong-and-costly, and certified-and-wrong-and-unfalsifiable; the ring is where "wrong" and "expensive" fully decouple, with a theorem rather than a measurement.

Two identical holes, opposite danger

That is the calm half. Now cut a channel of angular width \gamma through the band, and put the same channel in two different places.

Facing the start, where the planner drives: at \gamma = 0.6 the blind artifact's play cost is 0.029. Hidden behind the lode, where no plan ever goes: at the same \gamma = 0.6, the same gap, the same first Betti number of zero, it is 1.116 — which is also, to four decimals, what the band scores fully closed. Same hole, same topology, forty times the cost.

It is worth being precise about what does and does not change here, because the slogan invites a misreading. Opening the gap at all does change the topology: a closed band separates the plane and a band with a gap does not. That is exactly why the comparison carrying the claim is not closed-versus-open, but these two open cases against each other. Between them nothing topological differs — same \beta_1, same non-separating band, same width — and the only thing that moves is whether the planner's own path crosses the gap.

Sweeping \gamma shows where the switch happens. A dense sweep of the scripted blind model — 16 paired MPC episodes per point — puts play_cost (how much return the planner loses by trusting the wrong model, normalised against the truth planner) at 0.999 with the ring closed, 0.139 at \gamma = 0.1, and essentially zero from \gamma = 0.15 on. There is a knee, and it sits exactly where the channel becomes wide enough for a step to fit through: at \gamma = 0.1 the gap's arc is about 0.35 world-units, comparable to the planner's own step. The synthesis arm reproduces the collapse on its own exploited blind artifacts, in both model sizes and in the Claude relay: 0.348 at the knee, 0.029 by \gamma = 0.6.

The hidden channel does none of that. At \gamma = 0.6 the blind artifact's play cost is 1.116; at \gamma = 1.2 it is 1.116 again; with the band fully closed, 1.116 once more. Not "comparable to" the closed ring: its number to four decimals, because it is the same blind program facing the same reachable world.

Blind-model play cost against channel width, 16 paired MPC episodes per point. Both series collapse once the facing channel admits the planner's step: the dense scripted sweep from 0.999 to 0.139 at γ = 0.1 and to ~0 beyond, and the synthesized artifacts from 1.116 to 0.348 at the knee and 0.029 by γ = 0.6. Rotating that same channel behind the lode (rose) holds 1.116 at γ = 0.6 and γ = 1.2 — the closed band's own value, to four decimals.

Same hole, same Betti number, opposite danger. Which tells you that the property doing the work is not topological at all. Danger is topology relative to reach. And the mechanism is the gate quotient showing up on the play side: as the channel opens where the planner actually drives, the phantom stops being phantom — the blind plan (straight at the lode) becomes executable in the truth, so the blind model and the truth agree along the operative path, which is the only path that gets to bill you.

I like this result because it kills a tempting shortcut. If you are auditing a synthesized model, you cannot look at the geometry of what it got wrong — not even at an invariant as robust as "is there a hole" — and conclude anything about consequence. You have to ask where the thing planning against it can go.

But what if you can simply go around it?

The ring is a two-dimensional instrument, and in two dimensions an enclosing band is a wall: if it blocks the path, nothing gets in. That is a fair thing to be suspicious of, because it makes "the planner cannot reach it" look like a property of the drawing rather than a finding. So the paper runs the case where going around is possible: a solid torus in \mathbbR^3 placed between start and lode, which does not separate space at all. An explicit path goes around it and reaches the far side without ever touching it.

The three-dimensional control, where going around is possible. Left: the planned route threads the torus's hole and the blind model costs 0.019. Right: the same object, moved so the route runs into the tube, and it costs 0.898 — at the same contact rarity, 0.0033, and the same trivial topology. The contact-free path around it is why nothing here is reach-null, and so why the unfalsifiability theorem has nothing to act on.

Two things come apart there, and it is the cleanest decomposition in the paper.

The gauge disappears. Nothing is reach-null any more — there is no region a competent planner provably cannot query — so there is no exact unfalsifiability to be had. Certification-wise this mode drops back to merely rare, which is where the companion papers live.

The danger does not. It is governed by one thing: where the torus sits relative to the optimal path. Put the hole on the start–lode axis, so the plan threads it, and the blind model's play cost is 0.019. Move the tube onto that axis, so the plan clips it, and it is 0.898 — at the same rarity (0.0033) and the same trivial topology.

So the slogan splits in two, and this is the version I would actually carry around. Danger is path-relative: an omission on the path is exploited whether or not it encloses anything. Exact unfalsifiability is separation-relative: only an enclosing boundary manufactures a region no sampling gate can ever query. The ring conflates the two because there the enclosing boundary and the blocked path are the same object. The torus is what pulls them apart.

Can the loop repair a ring?

Same question as last time, harder shape. Three model families (GPT-5.x at two sizes, Qwen, Claude), the same synthesize-refine-accept loop, 903 artifacts across 39 conditions in the end.

From outside the ring, nothing recovers the region. Not one artifact encodes the band. What they write instead are superstitious point fits: an integrator plus a comment hypothesising a tiny localised trap, freezing on exact float equality with the single contact state their sample happened to contain. One of those is my favourite specimen in the whole series — it passed its own gate at 1.000, and the hardcoded coordinate sits two floating-point ulps away from the same trajectory computed with a different maths library. Its certificate was a property of the last bit of sin on the machine that generated it. An independent gate rejects it on every platform.

That is not the models being careless; it is the theory being obeyed. From outside, ring evidence and disc evidence are pathwise identical — there is no observation that separates them — so an honest summary of the evidence can only report the reachable arc.

From inside the hole, they pose the right topology and cannot pin it. Start the mover inside and the interior becomes reachable, so the omission is falsifiable now. Artifacts do pose hollow structures, loops, annuli — the right shape — and gate-pass rates stay at essentially zero anyway, because the band's radii are not round numbers and the gate wants 10^-9. The single gate-certified recovery out of twenty used the one form whose only free parameter is anchored in the reward spec: the complement of a disc whose radius is the lode's own. The strongest cross-family repairer wrote exactly the same form.

The held-out audit is the part I would want to see in someone else's paper. Re-scoring every artifact on a disjoint gate block: acceptance coincides with "that independent gate's own sample also missed the band" in 156 of 156 cases — an exact identity, per artifact, zero off-diagonal. And of 214 in-sample gate passes, 121 fail an independent gate, every single one of them at a contact with the band. An in-sample pass is training-set consistency, and what it omits is exactly the mode.

The sensor that guides the loop has a resolution limit

To give repair its best shot I fed each attempt an honest topological summary of its own evidence — cluster counts, bounding box, and a persistent-homology estimate \hat\beta_1 of how many holes the contact cloud has. Wording frozen before any run, no shape family ever named.

A summary like that is a sensor, and sensors have resolution. This one reports \hat\beta_1 = 1 — a closed loop — for every channel narrower than about two arc-units, even though the true \beta_1 is 0 for every \gamma > 0: a ring with a gap is not a loop. The flip happens around \gamma = 1.8.

That limit is geometric rather than budgetary, and the paper proves the two-sided version: below a scale set by the largest angular gap in the sample, the gap is invisible to the detector, and above another explicit scale the loop cannot survive. A factorial over the detector's point budget (30, 90, 270) and the evidence dose (40 and 160 rollouts) does not move the flip at all.

Worse, at the boundary more evidence makes it more confident in the wrong topology: quadruple the dose and the false-loop rate rises from 1 of 5 seeds to 3 of 5, because the denser sample fills the shells adjacent to the channel and the spurious bar's persistence grows from 0.05 to 0.50 while the detector's own threshold grows only modestly. Resolving a narrow channel takes a different filtration, not a bigger sample.

And the posed topology of the artifacts tracks the summary, not the truth: closed structures dominate wherever the summary says "closed loop" and all but vanish where it honestly says "arc" — 1 closed against 26 arc-shaped at \gamma = 2.4.

Now the part I have to report against myself. That crossover is cross-gap: the environment and the summary change together, so it does not isolate the summary's claim as the cause. So I pre-registered an intervention — design, analysis script and stopping rule committed before any outcome existed — that flips only the claim line: same 60 seeds, bit-identical evidence, every other byte of the prompt held fixed, the \hat\beta_1 line and its one interpretive sentence negated, scored against a freshly generated honest control.

Result: of 11 discordant pairs, 9 moved in the direction the claim predicted and 2 against. Exact two-sided binomial p = 0.065, with the registered interval on the effect spanning one half. Directionally consistent, and short of the level I committed to. So the paper reports the association and refuses the causal sentence, and the pre-registration is what makes that refusal cost me nothing to write.

Mitigation has to match the dimension and the direction

The companion paper's defense against this failure mode was a distrust fence: when the model's prediction is contradicted at some state, mark a ball around it and make the planner treat imagined paths through it as untrustworthy. On the patch instrument, that worked.

On the ring, at its calibrated radius, it does nothing at all — and the reason is a counting argument, not an implementation detail. A point fence is a zero-dimensional object; the ring's reachable boundary is a one-dimensional curve roughly 16 world-units long. Sealing a curve with \varepsilon-balls needs covering-number-many of them, boundary length over fence radius, and the planner concedes two to four contacts per episode while re-routing through the unfenced arc.

The covering argument, drawn. The band is faint because the planner cannot see it — it only sees the fences it has been given. Point fences leave arcs uncovered and the imagined path simply re-routes through one of them; a fence built along the boundary's own dimension leaves none, which is why it works after two lessons rather than never.

Defense	play cost	Why
Point fences, calibrated radius	0.999 → 1.003	0-dimensional cover of a 1-dimensional boundary
Point fences, 4× radius	0.999 → 0.742	partial relief once the balls reach the geometry's scale
Dimension-matched fence, per episode	0.999 → 0.957	right shape, forgotten every episode
Dimension-matched, persisted	0.999 → 0.058	two lessons total, truth-equal returns from episode 2
Any distrust fence, invented mode	1.769 → 1.769	fires 18.6 times per episode, changes nothing
Freedom patch, invented mode	1.769 → 0.029	the dual certificate: un-freeze where the model was over-pessimistic

The last two rows are the ones I did not see coming. Point the same machinery at the opposite error — a model that hallucinates an obstacle where the world is empty, which costs more than the phantom-freedom case at 1.769 — and every distrust variant is inert by construction: it fires constantly, because the model is being contradicted everywhere, and it has nothing to offer, because distrust cannot manufacture the freedom the planner needs. The defense that works is the dual: mark the states where the model was refuted as too pessimistic and let imagination run free there. That collapses it at once.

Two wrongnesses, opposite defenses, and each defense's cost is set by how often its failure lies to you. A false obstruction refutes itself at every single step, so one episode teaches the planner everything. A false freedom refutes itself only at the rare boundary, so you have to pay for coverage. Same geometry as the rest of the paper, seen from the planner's side.

In n dimensions both knobs max out

One extension, because it separates two things that look like one. Replace the ring with an enclosing shell in n dimensions and sweep n.

The rarity of contact collapses geometrically — a measured factor of 0.411 per dimension, with the exponential rate proved for an isotropic action interface and an explicit bound for the instrument's own. That factor comes from a 10,000-rollout sweep of the cone event, because the cheaper calibration runs out of resolution first: contacts fall to 1 in 600 rollouts by n = 4, and past it 600 rollouts can no longer separate the cells (0 in 600 at n = 5, 1 in 600 again at n = 6). Either way mis-synthesis becomes near-certain: the gate's sample almost never contains the thing the spec omitted.

Meanwhile the danger does not decay at all. A competent planner with a vector action interface is exploited at play_cost ≈ 1.0 at every n \le 6: it drives straight at the lode and gets pinned. Rarity lives on the synthesis axis, reachability on the play axis, and they are independent knobs. A high-dimensional enclosed mode maxes out both — the omission is almost certain to happen and fully exploitable when it does.

(One methods note that cost me a day: the same sweep with the scalar planner's candidate set measures zero danger at every n, and that is a property of the planner, not the geometry. Its candidates lack the axial sequences that drive straight at the shell. Competence is a property of the action interface, and an incidental planner weakness can hide a fully exploitable model.)

What I take from this

A sampling gate certifies the reachable restriction of your model and nothing else. That is the whole series in one sentence, and the ring is where it stops being a slogan: beyond reach, the model's content is a free choice that no test can pin and no planner can bill you for — a gauge, and the wrong-topology artifact that exploits it is both uncatchable and harmless, by theorem.

Which flips the question you should be asking. Not "is the model right?" but "does the place where it is wrong intersect the operative reach of whatever is planning against it?" Three consequences I would carry into a real system:

Reach, not shape — and path before separation. The geometry and even the topology of an omission tell you nothing about consequence on their own. The same hole, moved from in front of the goal to behind it, went from harmless to fully exploited without changing a single invariant. The torus sharpens it into two questions worth asking separately: does a plan cross it? decides the cost, and does it enclose something? decides whether any test could ever have caught it. An audit that classifies model errors by kind, rather than by those two, is measuring the wrong thing.
Your evidence summary is a sensor with a resolution. If anything in the loop — a monitor, a report, a retrieval step, a topological or statistical summary — decides what shape the evidence has, its blind spot propagates into what gets certified. Ours reports a closed loop for every gap narrower than two arc-units, and the artifacts follow the report. More data made it worse, not better.
Fences pay for dimension and direction. A defense built from points cannot seal a curve, and a defense built from distrust cannot repair over-pessimism. Match the boundary's dimension, persist what you learn across episodes, and know which of the two errors you are defending against — they need opposite certificates.

If you want the formal version — the gate quotient, the unfalsifiable-and-harmless theorem, the two-sided resolution sandwich, the n-dimensional rate, and which parts are machine-checked in Lean — it is in the preprint, and the code and every result artifact are open.

Preprint: "An Enclosed Mode Is a Gauge Choice" (arXiv:2608.28541) · code. Companion papers: An Omitted Mode Is a Rare Rule and the post about it — An LLM Can Infer the Rule You Forgot — and When a Verified World Model Still Loses.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

It Takes a Product to Send Messages

JaviMaligno — Mon, 14 Sep 2026 15:02:10 +0000

A hundred and fifty support messages, each needing one call: billing problem or login problem. Ninety free-text fields to rewrite in the house voice without inventing a claim. Forty markets to size: the same question about software in Spain, then about biotech in Portugal, then logistics in France, forty times over — same wording every time, two words swapped. The work repeats, and every row needs a judgement, which is why it is already being done inside a chat window — one message at a time, by a person who reads each answer as it lands and would notice at row thirty if the wording had drifted.

Almost nobody automates that. Not because it's hard.

Two things called hard

Two different difficulties share a word, and the argument dies if you let them merge. One is intellectual: the problem you don't yet know how to solve. That is the one that genuinely resists a script, because you cannot write down a procedure you don't have — and it's also the one that gets attention, because it's interesting to work on.

The work above is the other thing entirely. Each row is easy: one decision, two seconds, obvious to anyone who knows the domain. What makes it unbearable is that there are a hundred and fifty of them and a person has to be present for all of them. Nothing about it is difficult. It's repetitive and it needs judgement, and it's the pair that strands it — remove either half and it would have been dealt with years ago.

I've argued before that when a team copies context into a chat and carries the answer back, the person is the integration layer. The fix there is usually a connector nobody remembered to ask permission for. Here there's nothing to install, because the thing being repeated isn't a system call. It's a message.

The disproportion

So why does it stay manual? Because of what you're asked to build in exchange. The work is: paste a row into a chat, read the answer, paste the next one. The moment you want to stop doing that with your hands, every available exit walks you out of the chat and hands you a product to build. A provider account, an API key and a bill metered by the token. A prompt that now has to be tested the way code is tested, except each test is a paid call and there's no green tick waiting at the end. And an output that a person has to read, which means a presentation, which means an interface with rows, pagination and a login in front of it.

That's a product. To send messages in a chat.

The arithmetic most of us carry around is xkcd's table, Is It Worth the Time? — how long you can spend automating something before you spend more than you save. It's a good table and nothing here contradicts it. It just prices the one term that has already collapsed. Writing the loop isn't the problem; a coding agent writes it faster than you can specify it, which is most of what I mean when I say building is no longer the bottleneck. The three costs above survive that untouched, and not one of them is code.

What I built instead

Prompt Scripter — on the Chrome Web Store — takes one prompt with placeholders where those two words go — Analyse the {{ sector }} market in {{ country }} — and a list of rows, and sends one message per row into the conversation you already have open, in ChatGPT, Claude or Gemini. It waits for each answer to finish before sending the next.

The template comes from a message you already sent. There is no separate editor to learn: the prompt you spent six rounds getting right is sitting in the thread, and a button on it turns that message into the template.

The answers then arrive as answers, in the thread, with the platform's own formatting and its own citations — which is the image at the top of this page, and the part that decides whether any of this is worth doing. The answers land in the thread, which is where you were reading them anyway.

The rest of this piece is the three costs it exists to avoid. None of them is the twenty lines of Python.

The bill you didn't have

To push a row through a model from a script, you need a credential of your own. Pick a provider, add a payment method, keep a key somewhere that isn't the repository — and from then on every row and every retry has a unit price. An afternoon of office work becomes a budget line with somebody watching the spend. At home that's an annoyance; at work it's a procurement conversation, and if you don't control what you're allowed to use, it's one you may not win.

The obvious way to make that bill tolerable is to drop to a cheaper model, and it's the move I'd warn against hardest. I took the same task through progressively weaker models and measured that the capability was the expensive part: the weakest one never once went and looked, and filed fourteen reports about a release that didn't exist. Cheap tokens on work that needs judgement buy you confident answers nobody checked.

The bill isn't the only thing you take on by leaving. The prompt you refined was refined against a particular model, in a particular conversation, and neither of those is incidental. The platforms have spent two years making that conversation load-bearing: a project holding the documents the task depends on, memory that already knows the house voice you keep asking for, a thread where the last forty answers are still on screen. Go out to an API and all of it drops. What the chat gave you as a setting becomes something you have to engineer — retrieval, a system prompt that reconstructs what the project already knew, and some way of keeping the two in step as the work changes. You left to save an afternoon and inherited a context problem.

The extension sidesteps the line item entirely, because the model call isn't a network call of its own. It types into the page and presses send: the inference is ChatGPT, Claude or Gemini, in your tab, on the subscription you already pay for — the same model, in the same conversation, with whatever project and memory that conversation already has. No key to obtain, no provider to choose, and no context to rebuild. It does have an account of its own, and I'd rather say what that costs: a flat plan with caps on templates, runs and rows rather than a token meter, a loop that runs on a chat page without signing in at all, and — when you are signed in — rows travelling over HTTPS to a server of mine, each row's input and the model's answer stored as that run's results.

A prompt is code you can't read

Committing a prompt to a file is not the problem. A prompt lives in a repository like anything else and gets edited like anything else; version control is not the part that hurts.

The cost is that a prompt is non-deterministic code, and non-deterministic code isn't checked by reading it. You check it by running it, and a run doesn't come back as an assertion that passed. It comes back as a text somebody has to judge. No red, no green — a person reading outputs and deciding whether they're right, which is the same activity the automation was supposed to remove, relocated into the test suite. And the loop that makes a prompt work in the first place — reword it, cut the sentence that made it verbose, add one example, run it again — is N runs over M cases, every one billed, before the first useful row comes out.

The tempting shortcut is to have another model judge, so the loop closes without you in it. I measured that one: the same 45 blinded comparisons, three judges, three different rankings, each judge preferring its own answers, and on the subjective tasks agreement at roughly chance. On exactly the kind of question this article is about, the judge is a participant rather than an instrument — and the prompt was always the last 10% anyway. The first 90% is deciding what you're measuring, on what, with what in view. That is the product you didn't want to build.

What the extension avoids isn't the testing. It's having to build the test bench somewhere else and pay for its rounds separately. The prompt you automate is the one you already tuned by hand in that same chat, watching real answers, on a flat subscription — there's a button on your own messages that turns one of them into a template. The trial and error happens where it was already happening; the tool picks it up at the end instead of opening a second place to do it.

Somebody has to read this

Serialising JSON is trivial, and has been for twenty years. The problem is who the JSON is for. The person who has to read a hundred and fifty answers is not going to open an array of objects. They want the input beside the output, to jump to row 90, to reread a long answer without stepping over escaped quotes, and to find it still there tomorrow. That's a view: pairing, pagination, legible text. It's an application. And an application holding other people's rows drags the rest along with it — accounts, permissions, retention.

I've paid that bill. Building a conversational KYC flow, we ended up maintaining our own interrupt format and a widget registry by hand, until a standard turned up to do it for us. That's the honest price of presenting a model's output to somebody who isn't you.

The extension builds none of it, because the output appears where the reading was already happening: answers arrive in the thread as ordinary messages, with the platform's own formatting — headings, lists, code blocks, copy button. No screen had to be designed, because the screen was already there. There's a CSV export on the server for when you want the file. There is no screen, because the screen was already there.

Where the script is still right

None of this abolishes the threshold; it just isn't the point, so here it is briefly. Write the pipeline when the job runs on a schedule forever and the build cost gets divided by every future run. Write it when the output feeds a system rather than a person, because then you genuinely do want schemas, validation and a retry policy. Write it when the judgement turns out to be deterministic after all — a surprising amount of "the model should decide" is a rule nobody has written down yet, and the regex is faster and free. And write it when N is large enough that checking becomes sampling rather than reading.

There's a floor as well as a ceiling, and the floor gets forgotten more often: below some number of rows, describing the job is the job. If you'd have finished the list before you finished explaining it, you should have finished the list.

The direction

What I keep noticing is that all the available exits point outward. The work happens in a chat, and every way to stop doing it by hand proposes a new destination: a console, a dashboard, a platform with its own login, its own bill and its own tab in somebody's browser. I've made the opposite argument about products in general — put your app inside the agent your users already have instead of asking them to come to you — and this is that argument turned inward, at your own work. If the job lives in the chat, the automation belongs in the chat.

Prompt Scripter is that argument with a build attached. The disproportion above is the whole reason it exists; if the disproportion isn't real, neither is the tool, and I'd rather be told.

Here's the test I'd apply to your own version of that pile of work. Write down what you'd have to build to stop doing it by hand. If the list comes out as a provider account, a test bench and an interface — and the job is sending messages in a chat — the list is the argument.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

When the Fact Stops Being True

JaviMaligno — Sun, 13 Sep 2026 13:19:41 +0000

Correction. The headline count of this article — Haiku at 3/44 against 44/44, and "93 of 93 dependent steps" with explicit state — came from an instrument that could not register every miss: it read its dependent steps off a simulated trajectory instead of the agent's real one, and applied the correct action whenever a runtime returned none. Re-measured per episode on the real trajectory, across three models, explicit state applies the correction in 70 of 72 episodes and the full transcript in 10 of 72 (Haiku alone: 24/24 against 1/24). The direction stands; the perfection does not. The re-measurement is in the replication paper.

A replication of SKILL.state: Scalable Long-Horizon Agent Skills (Badhe, Tiwari and Chung, accepted at EMNLP) across two models and 500+ episodes. Every headline number here is a count of decisions, not an average of episodes.

An agent reads an event at step 10: the pallet you filed at step 3 was never actually put away; that shelf is empty. Twenty steps later it has to decide where to store the next pallet. The correct answer is the shelf the correction freed.

With the entire transcript in its context — the original record, the correction, everything in between — Claude Haiku 4.5 gets that decision right 3 times out of 44. Given a 200-character JSON state object instead, and no transcript at all, the same model gets it right 44 out of 44.

That is the strongest effect in this replication. It is also the paper's own claim: its Experiment 3 reports that history-based runtimes hallucinate for five to eight turns after a correction while explicit state recovers in zero. What is new below is the unit of measurement, not the sign.

What the paper proposes

SKILL.state replaces the append-only conversation history of a ReAct-style agent with an explicit, mutable execution state. At each step the model receives the procedure P, the current state Σ_t, and the latest observation O_t. It replies with a JSON patch and an action. The patch is validated and merged, Σ_{t+1} = Σ_t ⊕ ΔΣ_t, and the reasoning that produced it is discarded. Nothing accumulates.

The claim has two halves: better accuracy on long procedures, and a prompt that stays O(1) instead of growing O(T).

The four arms differ only in what sits between the procedure and the latest observation. Stateful and ReAct carry nearly the same content; the order they carry it in turns out to cost 5.7x.

SkillExecBench has no public code, so the environment here is a reimplementation from the paper's §4.1 description — a 500-shelf warehouse with dense pipe-separated event records, matched on context density rather than literal content, and running 1.2–1.5x heavier than theirs by average prompt size.

One difference is deliberate and worth stating up front: their Table 1 runs on Gemini-3-Flash, with Gemma-4-31B-it and Qwen-3-8B-it elsewhere in the paper. This replication runs Claude Haiku 4.5 and Claude Sonnet 5. Where a result here disagrees with theirs, the first candidate explanation is the model family, not the method — and saying which of the two it is turns out to be most of the work.

Half of it replicates

The cost curve reproduces exactly. Average prompt at T=50, in the same unit the paper reports — characters: SKILL.state 2,157, flat from T=10 (2,136) to T=50, against their 1,773. ReAct: 16,437 and growing linearly, against their 11,931. O(1) against O(T), as advertised, at 1.2–1.4x their density.

The accuracy half does not, and the interesting part is that it does not at any horizon they tested. Their degradation is a scaling effect — ReAct falls from 0.90 at T=10 to 0.74 at T=200 — so the only honest way to check it is to run their whole range.

Their transcript arm degrades with the horizon exactly as they report. On a different model family, on an environment 1.2–1.4x denser than theirs, run out to the same T=200, it does not.

runtime	T=10	T=25	T=50	T=100	T=200
ReAct	1.00	1.00	1.00	1.00	0.99 ±0.02
SKILL.state	1.00	1.00	1.00	1.00	1.00 ±0.00
Stateful	1.00	1.00	1.00	0.99 ±0.02	—
Memory	1.00	0.96	0.75	0.72	—

At T=200 the transcript arm is holding a 48,000-character prompt and 690 actionable events, and it misses one decision out of roughly 600. On Gemini-3-Flash the same arm misses a quarter of them. Re-measuring the T=50 SKILL.state cell with 3 seeds × 6 repetitions gives 18/18 at exactly 1.000, zero deviation, so this is not a lucky run either.

The Memory row looked at first like an artefact rather than a result: its runtime is the only one that makes a second call per step, and at T=100 it lost 23, 14 and 2 replies out of 100 to the output cap on the three seeds, scoring 0.58, 0.67 and 0.91 in that order. So it was re-measured with double the output budget:

Memory	output cap 600	output cap 1,200	truncated replies
T=50	0.75	0.79	3–10 of 50
T=100	0.72	0.71	12–37 of 100

Doubling the budget raised the truncation count — 23 replies cut off became 34 on the same seed — and left the score where it was. The model fills whatever budget it is given, and the low score is not what the cut-off replies were costing. Memory really does degrade, and it degrades further than in the paper.

The reason the other three hold is worth naming, because it governs everything after: in this task the load-bearing information is never far away. The freed shelf an agent has to reuse sits 1.9 positions from the top of the stack on average, at most 7. Making the horizon longer adds steps without moving information further from its use. If you want to measure whether a runtime remembers, T is the wrong knob — which is what the rest of this article is about.

The token count is not the bill

The paper compares tokens. Anyone running this compares money, and the moment prompt caching is on those are different quantities. An append-only transcript is the ideal cacheable prefix: every step re-sends exactly what it sent before, plus a suffix. A block that mutates invalidates the cache from the point it mutates.

There is one number you have to measure before any of this means anything: the minimum cacheable prefix. Below it, nothing caches at all. On Claude Haiku 4.5 it is 4,096 tokens exactly — a 3,984-token system block, sent twice, reads nothing from cache; a 4,116-token one reads all of it. So the answer depends on how long your procedure is, and both cases are worth having.

Raw tokens are the dashed outline; the filled bar is what you are billed. Every method that compresses rewrites its prefix, and rewriting the prefix kills the cache — until the procedure itself is long enough to cache on its own.

Short procedure — 1,491 tokens. Below the threshold, so no arm's procedure caches. Only ReAct caches, and only because its accumulated transcript pushes the prefix past 4,096 on its own. SKILL.state's advantage over ReAct falls from 7.54x in raw tokens to 1.39x in money. The orderings disagree too: by tokens it is SKILL.state < Memory < ReAct < Stateful; by money, SKILL.state < ReAct < Memory < Stateful.

Realistic procedure — 5,243 tokens. A field reference for the 112 fields the events actually carry, six exception rules, five worked examples. Real operating procedures look like this. Now every arm's static half caches, and two things happen:

SKILL.state gets 41% cheaper: 109 →64 per thousand episodes. The procedure got three and a half times longer and the bill went down, because it crossed the threshold. Memory goes from saving 0% to 48%, Stateful from 0% to 22%.
The advantage in money widens to 2.51x while the advantage in raw tokens narrows to 3.43x. In both conditions the raw token count is the wrong number to quote: it says 7.54x or 3.43x where the invoice says 1.39x or 2.51x.

The row to actually act on is Stateful. It sends almost exactly what ReAct sends. It puts a mutating state block in front of the transcript instead of behind it — which is where the paper's own Appendix A.3 template puts it — and is billed 849 against162 per thousand episodes. Same content, same task, same score of 1.00. A 5.2x difference, and it holds in both conditions.

On Sonnet 5, at 3x the input price, those become 2,546 against486. Per thousand episodes, prompt order is a four-figure line item.

Where explicit state actually wins

The paper's Experiment 3 tests what happens when the world changes underneath the agent, and reports it as a yes/no over four hand-built scenarios: history-based runtimes recover after five to eight turns, SKILL.state after zero. It is the one result of theirs that reproduces here, and the probe below measures it per decision instead of per scenario: the agent files a pallet at step t; at t+10 a correction says that put-away never completed and the shelf is empty. From then on that shelf is the lowest free one, and every subsequent decision depends on having applied the correction.

Two design choices make this measurable at all. First, the unit of counting is not the episode or the seed but the dependent step: each step after the notice whose correct action changes because of it. Second, the seeds are chosen by measured range before spending anything — a perfect-but-deaf agent, one that executes everything correctly and simply never applies the correction, defines the floor, and seeds differ enormously in how much room there is above it:

seed	perfect-but-deaf floor	dependent steps
0	0.931	2
1	0.893	3
2	1.000	0
4	0.522	11
6	0.846	4
10	0.759	7

Seeds 4, 10 and 6 carry 22 dependent steps per repetition. Seeds 0, 1 and 2 carry five between them, and one of them carries none at all. Computing that table costs zero API calls.

Explicit state does not miss a single correction in 93 dependent steps across two models. The full transcript, which physically contains the correction, applies it on 18 of 82.

Three things only visible when you count decisions rather than average episodes:

The transcript arm fails all-or-nothing per scenario. In Haiku, errors of any other kind are exactly zero across all six episodes: its only mistakes are the correction steps, and it misses them in blocks — 11 of 11, 7 of 7, 4 of 4. This is not an agent drifting. It is an agent executing a 50-step procedure flawlessly while never updating one fact. In Sonnet the same pattern appears bimodally: one episode applies all 11, the next repetition on the same scenario misses all 11.

More capability helps and does not solve it. Sonnet with the full transcript goes from 6.8% to 39.5%. It reconciles the contradiction far more often, and still loses three decisions in five.

Explicit state buys it, and in Sonnet charges for it elsewhere. Sonnet emitted 66 out-of-schema patches across 8 episodes, exhausting the retry budget on 10 steps that then produced no action at all, which cost it 21 errors of other kinds. Haiku emitted zero. Accuracy on the correction is 100% in both; the runtime's reliability is model-dependent, and that is a property of the method rather than of the task.

The mechanism is unglamorous. Explicit state has exactly one place where the fact lives, and correcting it is the operation the runtime already performs every step. The transcript erases nothing: it holds the original record and its retraction simultaneously, and every subsequent step has to resolve the contradiction again from scratch.

Having a single place where the truth lives is an advantage precisely when the truth changes. That is the paper's claim, not a counter-result. What this adds is a number under it — every dependent decision rather than four scenarios — and a replication in two models it never tested.

Where explicit state does nothing

The complementary probe: at step t the environment announces a shelf is quarantined. At t + k that shelf is the lowest free one and the correct action is to skip it. The dependent step and the shelf are identical for every k; only the distance between the information and its use moves. Nothing gets contradicted — the fact simply has to survive.

Here explicit state, by itself, does nothing at all. At k=40, Haiku with a state object and no field for the notice: 0 out of 24. What changes the outcome is not the runtime but where the fact is allowed to live:

Availability accounts for two thirds of the effect; distillation accounts for the rest, and it is the third that separates "usually" from "always".

Three interventions, measured paired on the same seeds:

A schema field that names the fact (quarantined_shelves) takes Haiku from 0/24 to 100% and Sonnet from 12% to 75%. It works because the schema's designer anticipated exactly this fact — which is one of the three settings the paper's Limitations section declares and does not measure (a state update that depends on an observation whose relevance was not recognised when it was read), now with a number attached.
A generic free-text field (notes, no indication of what to put in it) scores 5/24 = 21% on Sonnet, confidence interval 9–40%, statistically indistinguishable from having no field at all. On one seed it scores 0/8, worse than nothing.
Re-injecting the standing fact into every observation takes Haiku from 0/24 to 24/24 and Sonnet from 12% to 83%, and requires anticipating nothing.

And the last one splits further. The re-injected banner carries three of the original notice's fifteen fields, hoisted to the top. Repeating the entire original notice verbatim instead, same position, same header, 981 characters instead of 67, scores 16/24 = 67%. All three Wilson intervals are disjoint, and the distilled version wins on all three seeds.

So: making the fact available recovers two thirds of the failure. The remaining third is distillation — with the same fact present at every single step, buried among fourteen fields of operational metadata, the agent overlooks it one time in three.

The practical version, scoped to what was measured (one environment, k=40, the three-level comparison on Haiku only): if a fact stays true across many steps, re-inject the field, not the record. A system that replays the whole document into context leaves a third of the failures on the table. This is the same shape as the finding in The Scaffolding You Pay For — the intervention that survives is the one that changes what the model is looking at, not the one that adds structure around it.

Four checks this kind of experiment needs

An experiment on agents produces numbers whether or not it is measuring anything, and when it goes wrong the output is not noise — it is a clean result. Two findings here were fully written up, tables and non-overlapping intervals included, before these four removed them.

Compute the floor before the effect. Simulate an agent that is perfect except that it ignores the thing you are measuring. On seeds 0, 1 and 2 that agent scores 0.931, 0.893 and 1.000, so the largest possible effect there averages 0.06 and one seed carries no information at all. A separation of +0.199 measured on those seeds was arithmetically impossible before anyone asked what caused it.
Count steps with no action separately from steps with a wrong action. That +0.199 was truncation: the transcript arm lost 8 to 19 replies out of 50 to the output cap, and a reply cut off before its Action: line scores as an error. Raising the cap does not fix it — 600 tokens gave 19 truncated replies, 1,500 gave 11, 4,000 gave 18. A fixed output budget penalises the arm whose prompt grows, and the paper reports ReAct degrading with T without reporting truncation.
Decompose the score by the type of step it averages. A second environment reported a sign-flipping interaction between models, with non-overlapping intervals on both separations. Instrumented, all four cells got its load-bearing rule right; only 5 of 34 actionable steps tested that rule, a blind policy scored 0.853, and the whole reported effect lived in routine steps. The section was withdrawn.
Know your noise floor. Same seed, byte-identical prompt, eight repetitions: one seed alternated hit and miss eight times running. Single-step accuracy carries tens of points of sampling noise; an average over ~170 events carries almost none; token accounting carries none.

What I would take from this
The advertised axis is the wrong axis. How much context you keep barely moves accuracy on a task where the information is nearby. What moves it is whether the load-bearing fact is present, current, and distilled at the moment of the decision.
Explicit state earns its place when facts get invalidated. One location to correct beats a transcript holding a claim and its retraction — 93/93 against 18/82, in both models. If your agent's world only ever accumulates, this buys you much less.
Compressing context and caching context are in conflict. Every method that rewrites its prefix pays full price for it. Measure the bill, not the token count, and put your mutating block after whatever you want cached.
A schema protects only against what its designer anticipated. A generic field to "write things down in" measured indistinguishable from no field at all.

The claims here are scoped to two models, one environment that discriminates, and a procedure short enough to sit below the caching threshold. What generalises is not any single number but the arithmetic: check that your effect fits inside the range your effect can have.

Replication of SKILL.state: Scalable Long-Horizon Agent Skills (Badhe, Tiwari and Chung, accepted at EMNLP). Related: The Scaffolding You Pay For on interventions that cost more than they buy, and The Forgetting You Don't Measure on what a single benchmark number hides.

Correction, 13 September 2026: an earlier version of this article framed the retroactive-invalidation result as contradicting a limitation stated in the paper. It does not. The paper reserves that case for tasks whose objective is the history itself — auditing, explaining past actions — which this probe does not test, and its own Experiment 3 already reports the same direction qualitatively. The contribution is the metric, not the sign.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

They knew it wasn't the model. They patched it anyway.

JaviMaligno — Sat, 12 Sep 2026 12:11:49 +0000

Two articles ago I described a habit: when a system has a language model inside it and the output wobbles, the explanation drifts to the model. Then I measured it, and the interesting result wasn't the blame — it was the behaviour. Without access to the code, nineteen of twenty agents set about damping the output instead of looking for the cause.

This piece was supposed to answer why. I had two hypotheses and they made different predictions, which is the good kind of problem to have. Instead I ran the control first, and the control made both of them pointless.

The control

The design is almost embarrassingly simple. Take the same classifier, the same planted fault, the same corpus, the same five passes, the same trace format. Change one thing: the head that does the classifying.

In one arm it's a language model. In the other it's a random forest — trained by distilling the model's own labels, frozen into a pickle, and given the identical interface so that nothing else in the system differs by a single byte.

The whole control. The fault sits in retrieval, upstream of the head, so it is the same fault in both arms — and the head, whichever it is, classifies correctly whatever it is handed.

Both briefs carry the same measured certification, and it is the piece that makes the comparison fair: "re-running the classification over the same context reproduced the output in 260 of 260 cases" — the real number, identical in both arms. Without it, an agent facing the forest could reason, entirely correctly, that a trained forest is deterministic and therefore the cause must be upstream, and would patch less for a good reason rather than a revealing one.

Forty agents saw the table without the code, twenty per arm. And then another forty, to find out whether what came back was the effect or the sample.

What happened

	language model	random forest	p
Patches the symptom	19/20	19/20	0.76
Accuses it of randomness of its own	14/20	4/20	0.0018
Proposes taking the randomness out	17/20	11/20	0.041
Blames the head	13/20	6/20	0.028
Uses the determinism argument	9/20	16/20	0.024
Places the cause upstream	16/20	17/20	0.50
Finds the real cause	2/20	8/20	0.032
Asks for the data it lacks	0/20	0/20	—

Thirty-eight of forty patched the symptom. Nineteen in each arm, the same exact figure on both sides. Wilson interval [0.84, 0.99].

That's the row that breaks the frame I brought in. Take the language model out of the loop, put a frozen forest in its place — a pure function, and the brief says so — and the patching doesn't move by a single unit. Whatever drives an engineer to smooth an output rather than trace it, a language model is not a prerequisite. A closed component is.

And then, having exonerated the box, they patch it anyway.

What it gets accused of

The second row is the one that does separate the arms, and it's the one that had to be measured properly: not how much they blame the head, but what they blame it for.

There are two ways to accuse a component of an output that wobbles. One is that it's random by nature: it samples, it has noise, it rolls the dice. The other is that it's deterministic and the system does something to it: retrains it, parallelises it, changes its configuration. Both are formulable against both heads — a forest can vote with randomness, an inference server can batch requests — and the criterion was fixed in writing, with examples of both accusations for both heads, before a single response was read.

Fourteen of twenty against four of twenty. Two independent coders, agreement 0.95, and the two disagreements settled by a third who didn't know how either had voted.

The vocabulary is no longer a measure, it's simply what's on the page. The model arm gives you "it re-samples every night", "temperature > 0 with no seed", "sampling noise on every call", "the nightly re-roll". The four in the forest arm who also accuse their head say nothing of the kind: they say it retrains without random_state, that predict_proba runs over the whole batch, that floating point moves under parallelism.

The forest gets accused of what the system does to it. The model, of what it is.

The first row is the same bar twice: patching doesn't care what's in the box. The second is the one that separates the arms, and it drags the other two along — the head believed to be random gets its randomness taken away, and the head known to be deterministic gets exonerated with exactly that.

The remedies follow the diagnosis: proposing to take the randomness out of the head, seventeen of twenty against eleven. temperature=0 and seed on one side; random_state and n_jobs=1 on the other. Worth noting that the first is a dial most current reasoning models no longer expose: it proposes switching off something that isn't on, on a component that in this setup reads from disk.

And six responses do both at once: they accuse the model of sampling and cite the certification that contradicts it, in the same document.

The dissociation

Of the twenty-five responses that exonerated the head — across both arms, on the same argument and the same certification — twenty-three proposed a patch anyway.

One writes: "a random forest is a pure function: same feature vector, same vote. The 260/260 confirms it. The classifier is ruled out." Its fourth recommendation is to publish by margin instead of top-1, with a threshold on the confidence gap and human review below it — "this cuts the symptom the user sees, whatever the root cause".

That last clause is the whole article. Cutting the symptom the user sees, whatever the root cause, is a perfectly sensible operational instinct. It is also what you do instead of finding the cause, and for that the thing in the box doesn't need to be a language model — it needs to be closed.

Measured twice

Forty responses are forty responses. Before these forty there are another forty, over the same two packages without a byte of change, and they're what tells effect from sample.

Gap between the two arms across two independent samples of twenty per arm. Where only one dot shows, both landed on the same number and overlap. The red row is the only one that really moves: looking upstream came out six apart the first time and one apart the second, so nothing can be claimed from that row.

Patching comes out 19 and 19 both times. The determinism argument, nine against sixteen both times, to the digit. Blaming the head, twelve against six and thirteen against six. Finding the cause, zero against four and two against eight.

And one doesn't replicate: placing the cause upstream. Ten against sixteen in the first sample, sixteen against seventeen in the second. With a language model in front of them, agents look upstream as often as with a forest; the first figure was the sample. I mention it because it's exactly the kind of row you build a beautiful thesis on if you only measure it once.

What the model does and doesn't change

The honest reading splits the thing I'd been calling one behaviour into two, and only one of them is generic.

Patching is generic. Nineteen of twenty on each side, in both samples, head exonerated or not.

Investigating barely moves. They look upstream equally. What changes is that they get less far: two of twenty against eight find the cause, and the same two against eight name the real mechanism. A small, consistent difference — not the one I expected.

What does change, and it's the only large thing, is the nature of the suspicion. With the same certification in front of them and the same argument available to clear it, one head gets accused of being random and the other doesn't.

That is the LLM-specific claim, and it turns out to be the oldest and simplest of the ones I brought in: it isn't that people investigate less, it's that the model gets charged with a class of fault that the thing standing in its place does not.

The question that lost half its object

I came into this piece with two hypotheses about why the reflex exists. One said it was a fossil of the training corpus — a habit from an era when temperature really was the main dial and treating output variance as a property of the model really was correct. The other said it was the model's personality, some being more inclined than others to look outward before looking at their own work.

For the patching, both are moot: there's no model-specific behaviour there to explain, because it shows up unchanged with no model in the loop.

For the accusation they're both still live, and the first now has a hint in its favour it didn't have before: what shows up in the model arm isn't reasoning about this system, it's a vocabulary — temperature, seed, roll, sampling — applied to a component that here reads from disk. That is what a habit looks like. But still live isn't separated: a habit learned from a corpus that steers tokens without passing through any consultable belief is indistinguishable from a disposition, for any experiment that only observes behaviour. Three independent reviewers of the design converged on that before a single response was collected.

What doesn't hold

Looking upstream doesn't replicate, so I claim nothing about where the search stops. It's the row I'd most have liked to keep.
The difficulty band broke in the second sample. The pre-committed criterion allowed up to four of twenty of difference in finding the cause; it came out exactly four the first time and six the second. The forest arm is somewhat easier, and that has to sit alongside that row — not alongside the patching row, which is identical in both arms.
The coder cannot be blinded. The text says "the model" or "the random forest" in every paragraph, and pretending otherwise would be a lie. What stands in its place: a symmetric criterion fixed in writing before a single response was read, two independent coders agreeing at 0.95, an arbiter for the disagreements, and the quotes published so anyone can argue with each call.
The two samples are not pooled. They're reported separately with an explicit statement of what replicates and what doesn't. Summing them for power would be exactly the shortcut this experiment measures in others.
One fault, one corpus, two heads. That the behaviour is generic against this failure under this opacity doesn't make it generic against any.

And a process note

The first version of this control didn't certify the two arms alike: 260/260 for the forest and 240/260 for the model. Since the argument for ruling out the head is one of the things being measured, making it more available on one side contaminated exactly what mattered. It was rebuilt — storing the output by context equalises the two certifications, and takes the network call out of the model's package on the way — and no figure from that version appears here.

It earns a line because it's the same error the experiment measures, committed by me on the experiment: I attributed to the head an effect that was in large part my own scaffolding.

Two hundred and eighty

Across five scenarios, two kinds of head, passive permission and explicit permission, not one of two hundred and eighty responses has asked for the information it was missing before concluding.

That number has survived every manipulation I've thrown at it, including the one designed to break it, the one that took the language model out of the loop, and the one that repeated the whole measurement from scratch. It is the most robust thing in the whole series, and I still don't have a good explanation for it.

The best I have is the shape of what replaces it: they build their own measurement instead — a script, a sweep, a synthetic reproduction. They want the data. They just don't ask.

Code, data and the script that recomputes every number: blaming-the-model. The series: the observation, the measurement, and this control.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Nineteen of twenty patched the symptom

JaviMaligno — Fri, 11 Sep 2026 12:50:19 +0000

In the previous article I described a habit I kept running into: when a system has a language model inside it and something goes wrong, the explanation drifts towards the model. The sampling. The nondeterminism. Something nobody wrote and therefore nobody has to fix.

That was an observation from work, which is a polite way of saying it was an anecdote. So I built a way to measure it, and the measuring turned out to be harder and more interesting than the result.

The setup

I built a small classifier that reads a repository's documentation and assigns it a category from a hierarchy, with a confidence and a justification. Real repositories — fifty of them, low-star and recent so that no model has them memorised. It runs on a real model. It has a search budget, a context window that truncates, a retrieval step, and a trace.

Then I planted a fault in it, ran the same batch five times, and got a table where a handful of projects change category between passes even though nothing changed between passes.

That table is what the agent sees. The only thing I varied is whether it also gets the code.

The whole design. Two groups see the identical failure; one of them can open the system. Nothing else differs, including the prompt asking them to investigate.

Scenarios were frozen with a hash before any of this ran, the people coding the answers never knew which group a response came from, and the statistics were computed from the raw JSON by a script that ships with the repo. I'll come back to why all that ceremony mattered.

The first fault: the order of retrieval

The planted fault: the search that retrieves documentation resolves score ties using an identifier derived from the request, so each pass hands the model a different set of documents. The variability comes entirely from the input. The model's sampling has nothing to do with it.

Twenty agents saw the table without the code, twenty with it.

	without the code	with the code	p
Found the cause	3/20	20/20	<0,0001
Proposed voting or retries	18/20	5/20	<0,0001
Patched the symptom (voting or temperature)	19/20	9/20	0,0006
Blamed sampling as the primary cause	4/20	0/20	0,053
Asked for instrumentation before concluding	0/20	0/20	—

The headline isn't the attribution. Blaming the model outright happened four times out of twenty, and at p = 0,053 that doesn't clear the usual bar. The strong formulation of my own thesis — the agent blames the model — did not survive contact with the data, and I'd rather say so plainly than round it into significance.

What did survive is the behaviour. Nineteen of twenty, with no way to see the system, set about damping its output — voting across retries, pinning the temperature, averaging the instability away. With the code in front of them, nine. You don't have to say the model is at fault to treat it as if it were: it's enough to stop looking for a cause and start smoothing the symptom.

The second fault: where blaming the model is half right

The first scenario has a weakness I could see from the start. Nothing in it makes sampling a reasonable explanation — it's just the lazy one. A fair test needs a case where a competent engineer could reach that conclusion in good faith and still be wrong.

So I built a second one, and this is the part I'd defend hardest.

A cache shared across the runs of one batch, keyed by project and section. Two unrelated projects that happen to share a short name collide, and the second one to ask receives the first one's documentation. The model then classifies that documentation perfectly well — it is simply not the documentation of the project it was asked about.

Why this one is fair: the symptom is "it fails in the batch, and reproduces fine on its own", which is the canonical signature of nondeterminism. The affected projects wander from pass to pass. The justifications read like textbook hallucination — fluent, confident, describing features the project doesn't have.

And here's the part that makes it honest. Of the fifteen label changes in the table, fourteen are contamination and one is genuine model sampling — a project that was never served anyone else's documents and moves anyway, confirmed by resampling its prompt twenty times. For that one, "it's the model" is the correct answer.

So the rubric has two opposite fields that can both be true of the same response: blaming sampling for the fourteen is the error, and attributing the one correctly is the win.

	without the code	with the code	p
Blamed sampling for the systematic changes	6/20	0/20	0,0101
Found the cache	13/20	20/20	0,0042
Correctly attributed the sampling tail	12/20	15/20	0,25
Proposed temperature to reduce variance	2/20	8/20	0,032
Asked for instrumentation before concluding	0/20	0/20	—
Built its own measurement instead	18/20	16/20	0,33

Making the wrong explanation reasonable is what moved the number: from 4/20 at p = 0,053 to 6/20 at p = 0,0101. The reflex isn't summoned by laziness. It's summoned by a situation where it half fits.

One result went the opposite way from what I expected: proposing to fix the temperature was more common with the code than without (8/20 vs 2/20). Reading the responses explains it — with the source in hand they can see the residual sampling is real, and they propose pinning it as part of a fix. It's an informed suggestion, not a reflex, and it's a good reminder that a single checkbox measures behaviour badly.

Nobody asked. A hundred and forty times.

Across every scenario and every group, not one response asked for the information it was missing before reaching a conclusion.

Not once. Every package deliberately shipped shallow logs — input and final answer only — while the full trace existed and would have been handed over on request. The brief said so: you can ask for whatever you're missing. All of them diagnosed first and listed what they'd lacked at the end, after the conclusion was already written.

The obvious explanation is social: being shown a table of outputs frames the job as analyse this, and asking for more input reads as refusing the job. That's testable with one sentence, so I tested it. Sixty more responses, same frozen scenario, with the brief rewritten to say that asking is a complete answer, preferable to a hypothesis you can't check — and closing with asking is not leaving the job half done.

	passive permission	explicit permission	p
Asked for instrumentation before concluding	0/20	0/20	—
Asked and stopped there	0/20	0/20	—
Presented the conclusion as provisional	6/20	14/20	0,013

A hundred and forty out of a hundred and forty. Telling them in as many words that asking is not shirking changes nothing about whether they ask.

What it does change is the shape of the answer. Hedging doubles. Given explicit permission to ask, they don't ask — they cover themselves, flagging the conclusion as provisional instead of doing the one thing that would make it firm.

And the other half of the explanation is what they do instead. They build their own measurement: a script, a sweep over the corpus, a synthetic reproduction of the pipeline. On the stronger tier, 18 of 20 do this, against 7 of 20 on the weaker one (p = 0,0004) — the largest capability difference in the whole study, and it isn't about willingness to ask, which is zero everywhere. It's about being able to manufacture the answer without asking.

So it isn't incuriosity about data. Several of these responses did genuinely rigorous work to get data. Asking for it simply isn't in the repertoire, and whoever can, fabricates it instead.

The first scenario, twenty responses per group. Access to the code multiplies the rate of finding the cause by nearly seven, and roughly halves the rate of damping the symptom. It does nothing at all to the willingness to ask for missing information, which is zero either way.

A finding that didn't survive

I'll spend a paragraph on something that isn't in this article any more, because how it left is the point.

Half of each group ran on a stronger model tier and half on a weaker one. In the first round, with ten responses per cell, the stronger tier appeared to find the cause less than the weaker one — 4/10 against 9/10, p = 0,029 — and I had a tidy explanation ready. The stronger responses had audited the corpus document by document, verified that no snapshot contains another project's material (which is true, because the contamination happens at runtime, not in the data), and concluded from that a cache couldn't be responsible. Correct reasoning, correct check, false conclusion.

It was a good story. So I doubled the sample to twenty per cell, and it evaporated: 2/20 against 5/20, p = 0,20. It was noise, and a p of 0,029 with ten per cell is exactly the kind of number that looks like a finding and isn't.

The one tier difference that does hold is the one above — the stronger tier builds its own measurement far more often. That one I'd defend.

What doesn't hold

With the code, 20/20 found the first fault. That group doesn't discriminate, and I knew it wouldn't before running it: reaching a middling difficulty there would have required fabricating metadata — assigning repositories to package registries that don't list them, staggering capture dates — and I ruled that out. A study about agents cutting corners can't cut that one. The with-code group is the control, not the measurement.
The literal form of the thesis remains unmeasured. Blames the model for what it wouldn't blame itself for requires the system to be the agent's own. In all four groups it audits someone else's code. What I measured is an asymmetry of material, not of authorship.
One certification came in at 18/20 rather than the 19/20 I'd set as the bar, and the fix removes all fourteen contaminations but leaves three residual changes — the 0,7% sampling floor that no cache key touches.
The rubric hardened between rounds, and the same condition scores 13/20 under the old definition and 3/20 under the new one (p = 0,0015). The second demands that a response describe the mechanism, and counts listing the cache among several uncommitted hypotheses as a miss. Numbers from different rounds of that field cannot be pooled or compared, and none in this article are.

The part that took longest

Three attempts failed before one worked, and the failures are more instructive than the result.

The first gave agents the code with a planted bug and asked what went wrong. All eight found it. A scenario nobody fails measures nothing.

The second showed a batch of bad classifications with no variability at all — and the thesis is about variability, so there was nothing for anyone to attribute to sampling. I'd built a case that couldn't contain the phenomenon.

The third — a rewrite that tried to make the fault hard by using concurrency — was killed in review before it ran, on a point I hadn't seen: if the entropy comes from network jitter against the inference service, then "the variability comes from the model layer" is true, and the rubric would have scored a correct answer as an error. Hard and deterministic pull in opposite directions, and you have to resolve that on purpose.

What survives all three is a design rule I'd now apply to anything of this shape: the fault has to be something the model provably cannot have caused. In the final scenario that's guaranteed by one fact — the same project, asking for the same sections, receives different prompt bytes across passes. Sampling cannot change the bytes of a prompt. It isn't a judgement call by whoever grades the answers; it's arithmetic.

And the ceremony earns its keep. Scenarios hashed before the run so they can't be tuned after seeing results. Coders blind to which group a response came from. Calibration data kept separate from confirmation data rather than merged to inflate the sample — which would have been free, and would have been exactly the shortcut this experiment exists to detect in others.

Code, data, and the script that recomputes every number in this article: blaming-the-model. The observation this started from is in the previous article.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

It blames the model for what it wouldn't blame itself for

JaviMaligno — Thu, 10 Sep 2026 12:56:17 +0000

I spend most of my time building systems that have a language model somewhere inside them, and most of that time I am not writing the code myself — an agent is. Which means I spend a lot of my day reading an agent's explanation of why something didn't work.

There is a pattern in those explanations that took me a while to name. When the system under discussion is ordinary software, the agent reasons about it well: it looks for the bug, it finds the bug. When the system under discussion has a model inside it, the reasoning changes. The suspicion drifts somewhere else, and it lands on the model.

What makes this odd, rather than merely wrong, is that the agent doing the reasoning is itself a model — frequently the same one that is running inside the thing it's diagnosing.

Five times this came up. None of them is a disaster story; they're all small, and that's rather the point.

The trace nobody asked for

We had a classifier: it reads documents about something and assigns it a category, with a confidence and a justification. Results were uneven, and I asked the agents working on it to figure out why.

What came back were hypotheses. Good ones, in the sense that they were plausible and well written. The category boundaries might be ambiguous. The documents might be too short. The model might be over-indexing on certain words. Each of these could have been true. None of them could be checked with what we had, because what we had was the input and the final answer, and nothing in between.

Not once did anyone propose the obvious thing: let's log what actually happened.

So I decided it myself, and it took three passes to get right — first what was searched and what came back, then which documents actually reached the model rather than merely what the search returned, then every tool call rather than only the searches. Each pass I thought we had enough and each pass we didn't.

And then something I didn't expect. Once the trace was complete, the analysis got better on its own. I had also written a rule — look at the reasoning, look at the confidences, look at every intermediate step — and I'd assumed the rule was doing the work. It wasn't, or not most of it. Handing over the information turned out to be more effective than handing over the instruction about how to analyse it.

That's worth sitting with, because the current reflex for fixing an agent's behaviour is to write it a better rule: a skill, a section in CLAUDE.md, a checklist. Sometimes what it needs is not a better instruction but a wider window.

"It's just stochastic"

The second one is the most common, and it's the one that named the whole thing.

Results vary between runs, and the explanation offered is the sampling of the model. Sometimes this is said outright; more often it arrives as a shrug — these systems are nondeterministic, you can't expect stability.

In my experience that has almost never been the actual cause. The inputs changed. The retrieval returned documents in a different order. A tool omitted an optional field. Two rules in the prompt both applied and nobody had declared which one wins. The context got truncated and the decisive document fell outside the window. These are design problems, and they have the useful property of being fixable.

The variability almost always comes from one of the top four layers, all of which are design decisions somebody made. The explanation reaches for the fifth, which is the only one nobody can be blamed for.

Notice what the bottom layer has that the others don't: nobody wrote it. Attributing a problem to the sampling of the model is the one diagnosis that ends the investigation without implicating any decision anyone made. That is not a small part of its appeal.

The switch

The second story has a reflex attached to it, and the reflex is its own small story.

Once "it's stochastic" is on the table, the remedy that follows is: lower the temperature. It is the first thing proposed, often before anything has been measured.

It fails on two levels at once. It isn't the cause, so it doesn't fix anything — at best it freezes the wrong answer instead of making it vary. And on a good number of current models it isn't even available: several reasoning deployments reject the parameter outright. I checked this while building the experiment that follows this article, because I wanted to be precise rather than rhetorical, and it isn't uniform even within one family — some gpt-5.x deployments refuse any value of temperature at all while others accept it without complaint. So the reflex is always a diagnostic error, and on some models it's also an API error you discover by hitting it.

What I find telling is not the mistake. It's that reaching for a knob is easier than reaching for the design, and there is always a knob.

There's a second explanation I find plausible and can't settle from here. These models were trained on a corpus where ordinary software vastly outnumbers software with a model inside it, and the part that does cover language models skews old: posts and notebooks from a time when temperature was the main dial anyone had, and when treating output variance as a property of the model was simply correct. On that reading the reflex isn't carelessness, it's a fossil — advice that was good practice for the models of a few years ago, applied to systems that no longer work that way. It would explain the specific shape of the error: the knob reached for is always the one that used to be the right knob. I can't establish that without seeing the training data, but a narrower version is testable — whether an agent's beliefs about model APIs track the era it was trained on rather than the models it's actually calling — and that's probably its own article.

Nobody audited the ground truth

We were moving from one Claude version to the next, and the feedback was that the new one was worse.

I didn't believe it, and not out of loyalty to the new model — I just knew how the comparison had been built. It turned out the prompt used for the new model wasn't equivalent to the one used for the old one. And underneath that, the gold set had errors of its own, and those errors happened to reward the answers the old model gave.

The measurement was wrong in the direction that made the conclusion look obvious. And it took someone asking is this ground truth actually correct? for anyone to look — which is a question about one's own method, not about the model.

That asymmetry is the clearest version of the whole pattern. Suspicion of the model came first. Suspicion of the setup that produced the number came only when prompted.

It happened again later, bigger, and I wrote that one up on its own: the grader knew less than the system it graded. An external report put a classifier at 54% accuracy over 500 companies; the gold set behind that number had been generated by a general-purpose model in one go, and its labels could be guessed from the company name alone. The defensible figure was twenty points higher. Same shape, again: the number was doubted only after someone doubted it out loud.

A regex where judgement was needed

The fifth is what happens after the agent finally accepts that the problem is one of design.

The remedy it proposes is a regex. Or a list of keywords, or a hard threshold on a score. Something that works perfectly on the three examples in front of it and falls over on the fourth. In practice I have to stop almost every determinism proposal I'm offered, because I can usually see where it breaks before it's written.

And the motive behind it is the other half of the same problem. The agent reaches for brittle rules because it doesn't trust the model to discriminate — which is precisely the thing a model is for. When the task genuinely has no enumerable rule, when the set of cases isn't bounded and the phrasing is open-ended, the correct answer is to give the model better scaffolding and let it judge. Reaching for a keyword list there isn't caution; it's substituting the one component that could have handled the case.

So the two halves point in opposite directions and share a cause. It suspects the model where the fault is in the design, and it distrusts the model where the model is the design. Both are failures of calibration about what a system with a language model in it actually is — and I'd argue the second is the more expensive one, because the brittle rule tends to survive in the codebase long after the person who added it has moved on.

The same model on both sides

None of these are exotic failures. They're the ordinary texture of building this kind of software with agents, and I want to be careful not to make them sound worse than they are: in most of these cases the agent's work was good and the fix, once pointed at, was competent.

But the pattern holds, and what makes it strange is the symmetry it breaks. An agent debugging a web server does not conclude that the CPU is unreliable. It looks for the mistake, and it looks for it in the code, because the code is where mistakes live. Put a language model in the system and the same agent starts treating one layer as weather — something that happens to you rather than something someone built.

The layer it excuses is running the agent that's doing the excusing. Which is a strange enough thing to notice that I stopped writing about it and went to measure it instead: whether the suspicion really does move depending on what you show the agent, and whether it moves for the reason it looks like. That's the next piece.

Related: coding agents and teamwork, where a similar question — is this a capability problem or a structure problem? — turned out to have the same answer; and what practices actually help an agent, measured over 750 runs.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

DEV Community: JaviMaligno

Making yourself replaceable

What isn't in the repository

Archive and memory

This helps me too

The company has to make room for it

Where each kind of context belongs

When AI chooses the questions

Where the questions come from

Who decides what is worth studying

What if AI develops its own judgment?

Consumers of truths

What academia would have to reward

Jev: what survives the hype

Personal: helping me work with agents

Product: detecting deception and checking data

SMS: telling a legitimate notification from a scam

Invoices: checking is different from asking “are you sure?”

Email: deciding when expensive analysis is necessary

Where it did not add enough

Infrastructure: cases that work but I still cannot deploy

What remains of the hype

Frontend, Backend, and the Agentic Engine

The application around the agent

A chat can hide an entire system

The agent can also operate on the application

The case where two blocks were enough

What is new about this

Which boundary is worth drawing

The Bug Nobody Can Reach

The setup, in one paragraph

When being wrong is free

The same blindness, a different world

"But in the real world I could go around it"

What I would ask of my own system

Benchmaxing: Winning the Exam Is Not Doing Better Work

What it means to optimize for the exam

What I can say about Opus 5

A pilot allowed to contradict me

The failure they shared

Ties count too

What remains of the suspicion

Being Wrong Can Be Free — Until the Planner Can Reach It

A mode with an inside

Beyond reach is gauge

Two identical holes, opposite danger

But what if you can simply go around it?

Can the loop repair a ring?

The sensor that guides the loop has a resolution limit

Mitigation has to match the dimension and the direction

In n dimensions both knobs max out

What I take from this

It Takes a Product to Send Messages

Two things called hard

The disproportion

What I built instead

The bill you didn't have

A prompt is code you can't read

Somebody has to read this

Where the script is still right

The direction

When the Fact Stops Being True

What the paper proposes

Half of it replicates

The token count is not the bill

Where explicit state actually wins

Where explicit state does nothing

Four checks this kind of experiment needs

What I would take from this

They knew it wasn't the model. They patched it anyway.

The control

What happened

What it gets accused of

The dissociation

Measured twice

What the model does and doesn't change

The question that lost half its object

What doesn't hold

And a process note

Two hundred and eighty