DEV Community: JaviMaligno

Your Agent Doesn't Know What Has Already Happened

JaviMaligno — Thu, 13 Aug 2026 09:36:34 +0000

I've spent a few months working daily with coding agents on a project with a lot of state: a university course I have to record in a studio on four consecutive days, with scripts, timings and a delivery deadline before it. And one kind of suggestion kept coming back — ordering advice that didn't hold up.

Leave this piece for the end, because it depends on the others. Record these two together, because the second reuses the first. Reasonable-sounding, and wrong: the thing it supposedly depended on had been finished and frozen months earlier. The agent was reasoning about my project without a firm line between what was already settled and what wasn't.

The obvious diagnosis is that it doesn't know when now is. I'll come back to that, because it's testable and it's false.

The anecdote I had to throw away

I went looking for the most flagrant example to open with. I found a beauty: a status file the agent had been maintaining said

| Recorded and good  | 2  |
| Recorded, to redo  | 3  |
| Pending            | 63 |

Nothing had been recorded. Not one frame — the studio booking was still weeks away. What existed were timed read-throughs in my living room, me reading a script out loud with a stopwatch to see whether it fit in ten minutes.

Perfect opening paragraph. Except that when I dug into how the file got that way, it stopped being evidence of anything.

The vocabulary was contaminated from both ends. One session had been preparing material for the real shoot and blended the two states together. And I had, more than once, called my own practice runs "recording" — in conversations about measuring timings, where the distinction didn't matter. It stopped not mattering the moment a document got written in that tense: every session afterwards inherited a status file that spoke about takes. I corrected it out loud more than once and it kept coming back, because the document was still there.

So it isn't a model conjuring a state out of nothing. It's months of drift in a shared vocabulary, with authorship on both sides, mine included.

That's the trouble with any anecdote from your own project: I'm inside the loop. My words go into the context, my corrections change it, there's no control condition, and whatever example I picked I'd have picked because it was striking — then explained it afterwards with the explanation I already believed.

A hunch is not a case. So instead of writing up the hunch, I built something I could be wrong about.

The test

Take the shape of what I kept noticing out of my project and put it somewhere I have no history. Fixed material, one question, one variable at a time, enough repetitions that I can't cherry-pick.

You give a model ~1.7 KB of project facts and ask: in what order should I do these, and are there any ordering constraints I need to respect? The material is my real project, ambiguity included:

Sixty-eight video pieces to record, a studio window booked for four consecutive days, a delivery deadline before it.
Scripts closed and frozen since July: the text is not going to change.
A table of measured times, and a status count: 2 good / 3 to redo / 63 pending.
And a trap. Two pieces look dependent — the fourth reuses a calculation written during the third — but the script of the fourth says, in as many words, "I'll rewrite them for you." It's self-contained. There is no ordering constraint.

Then you vary the context by one or two lines:

	added to the material
A	nothing
B	"today is 11 August 2026"
C	"the measured times are practice read-throughs, not studio takes; nothing has come out of the studio yet"
D	both
E	both, plus "don't invent ordering constraints"

Six models — gpt-4o, gpt-4.1-mini, gpt-5.4-mini, claude-haiku-4.5, claude-sonnet-4.6, claude-opus-5 — and then the whole thing again in an unrelated domain: a release of twelve services, production deploy window ahead, staging rehearsals behind. Same logical structure, none of the same words. Its trap is a service that reads a table another service migrates, where the runbook says the migration is idempotent and the service "starts against the old schema or the new one, without touching anything."

572 scored responses. An LLM judge does the scoring, blind to the condition and required to quote the exact sentence justifying each mark. Before trusting it I hand-coded a sample and checked both candidate judges against my coding: one agreed 30/32, the other 23/32 — and eight of the loser's nine errors were false positives, marking a failure in answers that said the opposite. So there's one judge, and it's the validated one.

Failure 1: a precedence that the document explicitly denies

asserts the dependency anyway	course domain	deployment domain
`gpt-4.1-mini`	100%	100%
`gpt-4o`	100%	100%
`claude-haiku-4.5`	100%	96%
`gpt-5.4-mini`	100%	86%
`claude-sonnet-4.6`	100%	100%
`claude-opus-5`	52%	10%

Five of six models say B must come after A. In the course domain that's 92 out of 92 responses — not a tendency, a ceiling.

And the exemption isn't buried in an appendix. It's in the same sentence as the dependency. Here's gpt-4o, unedited, three lines apart:

auth-db-migrator — Prepares the schema auth-api uses. It's idempotent and backward-compatible.

[…]

Always deploy auth-db-migrator before auth-api, since it adjusts the schema.

It read the exemption, wrote it down, and reasoned as if it hadn't. Nothing was missing from the context: the model reproduced the information and then overrode it. What loses isn't the fact — it's the fact's scope. The relation ("B reuses something from A") survives; the clause that cancels the relation does not.

The shape of the result matters as much as the size. This is not a gradient, it's a threshold. gpt-5.4-mini and claude-sonnet-4.6 are strong recent models and they fail exactly as often as the smallest one in the set. One model out of six is in a different regime; the other five are indistinguishable from each other.

Failure 2: a premise that expired in July

Both materials contain something that invites a convention from the training corpus. In the course, an opener that describes the whole course — and in video production you shoot the intro last, because the intro has to match what you ended up making. In the release, a gateway that publishes the manifest for everything — and in deployments the gateway goes last, so it doesn't advertise services that aren't up.

Both are real, sensible practices. And both rest on a condition that this material explicitly removes: the scripts were frozen in July and the artefacts were signed in July. Nothing done later can change what they say. The reason to defer them expired before the window even opened.

defers it because it depends on the rest	course	deployment
five smaller models	36%	69%
`claude-opus-5`	60%	98%

This is the failure I find most interesting, because it's temporal in the purest sense: the model retrieves a rule about when to do something, and doesn't check whether the rule's precondition still holds. It's the same move as advising someone to book early for a trip that already happened.

The model that reads the exemption is among the worst at this — 25 out of 25 in the deployment domain, the highest number in the study.

I want to be careful, because there's a tidy story available and the data doesn't quite support it. The tidy story is "the failure grows with capability." Not monotonically: in the deployment domain gpt-4.1-mini falls for it 79–90% of the time while gpt-4o does 20–40%. What the data does support is narrower and still useful: being a better model does not help here. The three strongest models in the set fail this 92–100% of the time in the deployment domain. If your plan is "wait for a better model," this is the part that will still be waiting.

My reading, offered as interpretation and not measurement: the two failures pull opposite ways because one is about reading and the other about producing. Ignoring the exemption is under-reading. Deferring the opener is over-producing — the model doesn't just answer, it supplies a reason, and a fluent plausible reason is exactly the shape a wrong ordering constraint takes.

Failure 3: what has and hasn't happened

The status count — 2 good / 3 to redo — is genuinely ambiguous in condition A. Reading it as finished output isn't crazy; the material doesn't settle it. So condition C settles it: these are rehearsals, nothing has come out of the studio yet.

What that line buys depends entirely on which model reads it.

claude-opus-5 acts on it. The state error drops from 92% to 20% in the course domain (p=3.7·10⁻⁷) and from 20% to 0% in deployment. It also says it, going from 12% to 100% and 16% to 96%.

The smaller models mostly just repeat it. gpt-5.4-mini goes from never mentioning provenance to mentioning it 40–48% of the time; claude-sonnet-4.6, 0% to 62% in one domain. And then they plan around 63 pending anyway.

And for three of them it made the planning worse. In deployment, adding the clarification raised the state error in claude-haiku-4.5 (12% → 32%), claude-sonnet-4.6 (12% → 25%) and gpt-5.4-mini (12% → 20%). Small numbers, no individual significance, and I won't claim a mechanism. But the direction is the opposite of free: giving a model a caveat gives it something new to talk about, and talking about it is not using it.

That's what I'd have missed testing one model. The same sentence of context is a large fix, a no-op, or a mild irritant depending on what reads it.

So is the outlier better or worse?

One model behaves differently from the other five on every measure, which invites the question: is claude-opus-5 exceptionally good at this, or exceptionally bad? It's genuinely both, and I think it's one property seen from two sides.

Line up the state error by condition:

error rate	without the clarification	with it
the five smaller models	0–70%	0–56%
`claude-opus-5`	92% / 20% — worst in its column	20% / 0% — best

In both domains it is the worst of the six when the material leaves the status ambiguous, and the best once the material settles it. Without a fact to constrain it, it builds the most natural reading — those numbers are finished output — and plans on that reading with conviction. Given the fact, it updates and plans on the new one. The smaller models sit in a mediocre middle either way, because they commit less to any interpretation.

That also reconciles the gateway result, which otherwise looks contradictory. What this model reliably uses is what's explicit: "I'll rewrite them for you", "these are rehearsals". What needs its consequence derived in two steps — frozen since July, therefore nothing later can invalidate the opener — it doesn't use, and there the corpus convention wins. It wins harder, in fact, precisely because this is the model that elaborates most.

Offered as interpretation, not measurement. But the practical consequence runs opposite to the intuition: with a stronger model, the quality of your document matters more, not less. A small model gives you something mediocre almost regardless of what you wrote. A large one gives you back what you gave it — which is good news if you write carefully and bad news if the load-bearing fact is only implied.

It isn't the clock

Which brings me back to the obvious diagnosis. Every failure above is about time — precedence, expiry, what has occurred — so the natural fix is to tell the model when now is. It's also the mitigation every tool ships by default: your agent almost certainly has today's date stamped into its system prompt right now. Mine did, throughout.

Adding the date explicitly moved nothing. Not one of the four measures (all p > 0.5).

Nor did the other two things I tried. Declaring what the numbers are doesn't make a model read the runbook clause — different failure, no transfer. And condition E adds, in plain language, don't invent ordering constraints: still 96%. You can write the prohibition into the prompt and watch it violated in nineteen of twenty runs.

So the deficit isn't the timestamp. A calendar tells you where now sits; it tells you nothing about which of your facts are already fixed, which relations still hold, and which rule's precondition expired last month. That structure lives in the document, and it's the part that doesn't survive being read.

What I'd actually do

Don't write an exemption — write the sentence you mean. The best-supported item here, and it's a documentation habit rather than a prompt trick. If a constraint doesn't apply, don't state the constraint and then except it: the exception loses 96–100% of the time in five of six models. "B reuses A's calculation, but it rewrites it" becomes "B is self-contained."

Say what's frozen, not just what's done. Failure 2 happens because a generic "do this last" rule outranks a specific fact about my project. "Scripts closed since July" apparently reads as history; "the scripts cannot change, so nothing later can invalidate the opener" states the consequence, which is what the rule needed to be checked against.

Make provenance inseparable from the value. Not a note elsewhere saying the numbers are rehearsals — a mark that can't be copied off. In my project what worked was writing 8:25 †, where the dagger means "this delta is meaningless," so the number can't travel to another table without its caveat. Scope: this is document design. Declaring it in a prompt reliably changes only what the model says back.

Stop expecting the date to do work, and don't add caveats assuming they're free. Both are cheap to inject and neither does what you think.

Watch your own vocabulary. The one lesson from the anecdote I threw away, and it survives precisely because it isn't about the model: I called practice runs "recording" when it didn't matter, and it stopped not mattering as soon as it was written down. In a long-running project, loose words get committed to a file, and what's in the file becomes the state of the world.

Limits

Two domains is not a sample of domains, and both are ordering work with a window ahead; diagnosis or analysis tasks might behave differently. Sixteen comparisons, so under a Bonferroni correction what I'd call confirmed is the exemption result and the two provenance effects in claude-opus-5; the worsening in three smaller models and the effect of the explicit prohibition are indications, labelled as such above. The judge shares a model family with two subjects, which I'd fix with a third judge next time — the bias would flatter that family, and it came out worst on failure 2, so the direction is at least safe.

And a caveat this article is itself an example of: all of it is measured in August 2026 against those six models. The failure I've spent the whole post describing is a claim outliving the conditions it was written under. Mine will too.

The three failures need different fixes, which is why I think they're worth separating. Under-reading you fix by writing better documents. Stale-premise reasoning I couldn't fix at all — not with the date, not with a declaration, not with a direct prohibition — and it's the one the strongest models were worst at.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

If You're Starting From Zero

JaviMaligno — Mon, 10 Aug 2026 09:48:18 +0000

There's a moment, the first time this works, that is genuinely one of the better feelings available in a professional life. You described something that didn't exist, and a few minutes later it existed, on your screen, doing what you said. If you never learned to write software, that moment lands even harder — you have just done a thing you had filed under not for me.

I don't want to spoil it. It is as real as it feels, and the people telling you it isn't are mostly protecting something.

But there's a gap between that moment and having something other people can use, and almost nothing you did to get here helps you cross it. I wrote the full map of that gap — thirteen categories, three levels each. This piece is the other half of the question, the one people actually ask me: not what's on the list, but what to do first.

The single most useful thing to understand on day one

You are not having a conversation. You are running a loop.

Specify → build → check → correct, then again. Anyone who came from a technical job has this in their bones and stops noticing it. If you didn't, it is genuinely new information, and its absence is the biggest difference I've seen between people who end up with something real and people who end up with a very long chat history.

The default without it isn't dramatic. You ask, you get something, you look at it for two seconds, you ask for the next thing. Nothing ever checks anything. It feels like progress the entire time, right up until you try to show it to someone and discover which parts were never true.

The fix is unglamorous and takes about ten seconds per cycle: after each thing you ask for, use it before asking for the next one. Not read it — use it. That's the whole discipline, and it is worth more than any technical fact in this article.

The terminal, and why it's less of a wall than it used to be

Almost everyone I've watched start from zero hits the same wall in the first hour, and it isn't a concept. It's a black window with text in it.

Worth being explicit about why you meet it at all: the terminal is the front door. The serious agents — Claude Code, Codex — are programs you start from there. You open the window, you type their name, and from that point you're talking to the agent in plain language. The window isn't the work. It's the doorway to it.

The fear is rational. A terminal doesn't explain itself, doesn't confirm anything, and has historically punished typos in ways nothing else on your computer does. Every other piece of software you've used spent twenty years learning to be forgiving. This one didn't.

Here's what changed: past that doorway, you're not the one who has to know the commands. The agent runs them. What's left for you is much smaller — occasionally it hands you a line and asks you to run it yourself, usually because it needs a permission it doesn't have. Your job is to paste it and press enter without freezing.

That's the level required. Not memorising anything: not seizing up when a command appears.

And you can skip the doorway entirely. Both Claude Code and Codex ship desktop apps that look like an app, with browser versions too. Starting there is a completely reasonable decision and I'd suggest it to anyone who's bounced off the terminal before.

One difference that isn't obvious and causes real confusion: the browser versions don't run on your computer. They run on someone else's, which means they can't see your files, don't have your keys, and don't have whatever you've installed. The desktop app and the terminal both work on your actual machine; the browser one works on a copy of your project, somewhere else. Early on this rarely matters. The moment your project depends on something that lives on your laptop, it matters a lot — and "it worked on my machine but not in the browser version" isn't the agent being inconsistent, it's two different environments with two different sets of configuration.

Cursor is worth a mention too. It started as a code editor, which sounds like the least friendly option on this list, but its recent versions are more chat than editor and it ends up being one of the gentler ways in — especially if seeing the files alongside the conversation makes you feel more oriented rather than less.

There's also Claude Cowork, which is worth knowing about precisely because it was built for people who don't code — you point it at a folder, describe what you want, and it works through the steps on your own machine. Its home ground is knowledge work rather than building an app: sorting files, pulling numbers out of a pile of documents, producing a report. If what you want is a working product, Claude Code or Codex is the tool. But Cowork is the clearest demonstration that "agent" and "terminal" were never the same thing, and it's a gentle way to get the feel of directing one.

If you do start somewhere other than the terminal, two reasons to keep it as a starting point rather than a permanent home. The terminal is where the full set of capabilities lives — the apps trail on newer features, so sooner or later you'll want something yours hasn't got. And when things go wrong, it shows you the entire exchange between the agent and your machine, which is exactly what you want when you're trying to work out what happened.

So: start wherever you'll actually start. Just know you're choosing the gentler door and not a different building.

One more habit worth building early: read what it prints, roughly. Not to understand it — to notice whether it ended in something that looks like a complaint. "Did that go well or badly" is a question you can answer from the shape of the output long before you can read a word of it.

You'll ask for less than you could, and you won't know it

The ceiling on everything you build is not your skill. It's your idea of what's askable.

If you think these tools write snippets, you'll ask for snippets. If you think they can't touch payments, or email, or a database, you won't ask, and nothing will correct you, because an agent answers what you ask and never mentions what you didn't. This is the quietest failure in the whole thing: there's no error message for a question you never asked.

There is one move that fixes this and it costs nothing. Ask the agent what it can do for your specific case. Not in general — describe your actual project and ask what approaches exist, what it would need, what it can't do, and what you'd be signing up for. It answers this well, and almost nobody asks.

Do it before you start building, and again whenever you find yourself assuming something isn't possible.

What to learn first, and why in this order

The map has thirteen categories. You do not need thirteen on day one, and someone handing you all of them at once is why most people bounce off. The order matters more than the list, and it comes from one question: when this goes wrong, who pays?

First, the things you can't undo. Your data isn't your code. If you delete the project and rebuild it, the code comes back — everything people typed into it doesn't. And there's a way to recover yesterday's version of your work that is not Ctrl+Z.

Which leads to the one I see skipped most often: backups. Not because the idea is foreign — you already keep copies of documents that matter to you, and you've felt the specific dread of a file you didn't back up. You just haven't applied it here, because a database doesn't look like a folder and nothing ever prompted you. So ask the plain version of the question: if this database vanished tonight, where's the copy, and when was it last made? If the answer is a shrug, that's the afternoon's work, and it's worth more than everything else on this list combined.

That group is the difference between a bad day and a lost month.

Second, the things that hurt other people. The moment you have real users, the cost of getting something wrong stops being yours. Keys don't belong in the code. A login screen protects the screen, not the data behind it. If you store anything about other people, you've taken on obligations you didn't sign for. This is the group I'd never let someone skip, because it's the one where the person paying never agreed to your learning curve.

Third, the things that surprise you. Your bill has no ceiling unless you set one. Your app works with you and might not with three hundred people. Something you depend on can disappear.

Last, the things that make it pleasant. Tests, documentation, your own taste in how it looks. Real, and none of them are why anyone's project ends badly in month one.

Four groups. Two afternoons for the first two. That is a completely different proposition from "learn to code", and it's the honest version of what this requires.

The mistake this whole article is about

Everyone starting from zero makes the same one, and it isn't the one they're afraid of.

The fear is of breaking something through ignorance. The actual mistake is handing over decisions you have no way to check — and, worse, not knowing they were decisions. The agent picked a way to store your data. It picked what's public and what isn't. It picked what happens when something fails. Each of those looked, from your side, like nothing happening at all.

That's what makes it hard to notice: delegating well and delegating blindly look identical from the outside. Both are you describing what you want and receiving something that works. The difference only shows up later, and only if you're unlucky.

You don't fix it by delegating less — you'd be slower and no safer, since you can't check what you don't know about. You fix it by knowing which boxes exist, so you can see which ones are empty. An empty box you know about is a managed risk. An empty box you don't know exists is the thing that ends up in the news.

That's the entire reason for the map, and why the first thing to learn isn't a skill.

If you came from WordPress, no-code, or automations

Then you're not starting from zero, whatever you've been telling yourself.

You already know a site can be live or not live. You've hit a plugin that broke everything and had to get back. You've had a subscription you forgot about. You know some things live in the tool and some things live in your account. Those are the same categories, learned in a different shape.

What tends to be missing is narrower: the loop as a discipline, and the fact that this time nobody is protecting you. Those platforms had walls — you couldn't delete the database because you couldn't reach the database. Now you can reach everything, which is exactly why it's more powerful and exactly why the map matters.

In my experience this profile adapts fastest of all. Not because they know more, but because they already believe software is something you can go and change, and that belief is most of the battle.

Where to start today

If you want to know which boxes are empty for you, there's a thirteen-question version of the map — about two minutes, and the useful outcome is finding a category you'd never considered.

And the honest summary of everything above:

Use the thing after every change. Not read it, use it. This one habit is worth more than any fact here.
Ask what's possible before you assume it isn't. The ceiling is your idea of what's askable, and the agent will happily raise it if asked.
Spend one afternoon on what can't be undone — backups included — and one on what hurts other people. Skip the rest for now.
Assume you've handed over decisions you didn't notice. You have. The question is only which ones.

None of this is learning to program. It's learning what to keep an eye on while something else does the programming — which is a smaller job, and nobody teaches it because it falls between the cracks: too obvious for engineers, invisible to everyone else.

Second of three pieces on what building software with agents actually requires. The first is the map itself; the last one goes the other way, for people with an engineering background who delegate too little rather than too much. If you'd rather go through your own project with someone, that's what I do.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

What You Still Need to Know to Ship

JaviMaligno — Mon, 03 Aug 2026 11:47:03 +0000

The people building software today did not all learn to build software. Some came from marketing, from operations, from running a small business, from nothing technical at all. They describe what they want, an agent builds it, and it works. That part is real and it isn't going away.

What's also real is that a chunk of them are shipping things with the database wide open, the API key sitting in the browser bundle, and no way back to the version that worked last Tuesday. Not because they're careless. Because nobody told them those were categories of thing that exist.

That's the actual gap, and it's smaller and stranger than "learn to code."

The ceiling: you can't ask about a space you don't know is there

Every failure I've watched, in myself and in the people I've taught, traces back to the same place. Not a missing skill. A missing category.

You don't need to know how authentication works. You need to know that having a login screen and being protected are two different things — because the moment you know that, you can ask, and the agent handles the rest. If you don't know it, you'll never ask, and the agent won't volunteer it. Agents answer questions well. They're much worse at telling you which question you should have asked.

I wrote about this from the other side in How Much Should You Still Know? — the engineer's version, where you delegate knowledge you used to hold and have to work out what you're still on the hook for. This is the same wall from the other direction: not what you're allowed to forget, but what you never learned and now can't skip.

It's worth saying where this comes from. Most of my teaching has been with technical colleagues, which is a different problem and mostly a different article. With non-technical people my sample is smaller and more informal — friends, mostly, and that moment when someone meets a terminal for the first time. The rest comes from building this way myself, and from what I've caught agents doing along the way.

There are two things to get, and they're independent. One is a way of working. The other is a map.

Part one: the loop

Specify → build → check → correct. Then again.

There is nothing new about this. It's spec/develop/test, the way software has been made for decades. If you came from anything technical you have it in your bones and can skip ahead. If you didn't, it is genuinely novel, and its absence is the single biggest difference between someone who ships and someone who has a very long chat history.

The default failure mode without it isn't dramatic. You ask, you receive, you believe it, you ask for the next thing. Nothing checks anything. It feels like progress right up until you try to show someone.

The agent is in all four phases, which is what makes this confusing. It helps you specify. It builds. It runs checks and tells you what to look at. It proposes the fix. So what's left? The useful question isn't where the agent is — it's always there — but what human residue each phase leaves:

Specify — the intent is yours. The agent will write down what you want far better than you would. It does not know what you want.
Build — nothing is left. Delegates completely.
Check — the agent runs things and reports. What stays yours is knowing which categories need checking at all, and calling it done.
Correct — nothing is left either. Correcting is building again, and the agent does it. When a failure makes you decide the whole thing should be rebuilt or dropped instead, that isn't correcting — that's going back to Specify with a different intent.

So the loop splits in half. Two phases hand over completely, two keep something human — and the two that vanish are exactly the ones people mean when they say "programming". What's left is saying what you want, and knowing whether you got it.

Which is why "you don't need to know how to program anymore" is both true and unhelpful. It's true about half the loop and silent about the other half, which is the half that was never taught to anyone.

Notice where checking sits: inside the loop, from day one. It isn't a level you reach — it's a phase you're already in. What grows with experience isn't whether you check, it's what you're able to check. It comes in three ranges: the agent tells you what to look at and you look; you check what you know to ask about; or you also know whether the check was any good and what it missed. The first range is available to a complete beginner on their first afternoon, and it's enough more often than you'd think.

Three ways to catch things, none of which involve reading code

This is the part people assume is impossible without a technical background, and it isn't. Everything I catch, I catch one of three ways, and they apply to every category further down.

Write it into the spec up front. The cheapest one, because you don't catch the problem — you prevent it. If you say at the start that these values must come from the database and nothing may be hardcoded, you don't have to find the hardcoded value later. Most of what an agent gets "wrong" is a reasonable default it chose because you didn't state a preference.

Use the running thing. Not the code — the app. Click it, poke it, try the thing a user would try and then the thing they shouldn't. This is how I found an endpoint sitting open that shouldn't have been: it surfaced during testing, not during any code review.

Ask when something smells. A number that never changes when it should. A page that loads suspiciously fast. A screen that works when you're logged out. You don't need to know what's wrong to say "why does this never change?" — and the agent is genuinely excellent at going from that to a cause.

None of these require reading a line of code. All of them require knowing that the category exists, which is the entire point of the map below.

Part two: the map, and how to read your level

For each category below there are three levels:

Aware — you know this category exists and can go wrong. You don't need to know how it works or what it's called. You need to know it's there, because that's what makes you ask.

Fluent — you understand the vocabulary, can form the question, and can tell whether the answer makes sense. Worth saying plainly: this is the level you can reach by asking the agent itself. Deliberately, with patience, one category at a time. It's the cheapest education available right now and almost nobody uses it on purpose.

Opinionated — you have your own view and can choose between options.

Two things about this that matter more than the levels themselves.

Your level is a vector, not a number. Nobody sits on the same rung across the board. You can be Opinionated about access control and completely unaware of what your hosting costs. The profile is jagged, and that's normal — it's a map with holes in it, not a ladder you climb whole.

If you want your own version of that jagged profile before reading on, there's a thirteen-question version of this that takes about two minutes.

Opinionated is optional — until it isn't. You can ship most categories without ever forming a view of your own. But wherever the project is actually betting something, the top level stops being a luxury: if you charge money, you need a view on cost; if you hold other people's data, you need one on access.

Twelve categories, plus one

Twelve where things can go wrong, and one that's a different animal. Read down and find the ones you didn't know were categories — those are your answer.

The ceiling

1. What can even be asked for. This one sits alone because it limits the other twelve. If you think an agent writes text, you will ask it to write text, and everything below stays theoretical.

Aware — an agent builds entire working systems, not just snippets of writing
Fluent — describe what you want by outcome, ask it to research and propose before it builds, and understand it when it says something can't be done or offers a different route
Opinionated — which model, which tool, which hardware. A topic for a later piece.

Where it is and how I get it back

2. Where your app lives. The environment question is where the genuinely scary mistakes live. Almost nobody deletes production data on purpose — they delete it believing they're in the test copy. And the separation you think you have may not exist: I've had pipelines that didn't distinguish environments at all, running CI against dev and prod alike, with no real separation behind the names. I found out by looking at the deployment dashboard, not by reading configuration — and then wrote the separation down so it stayed true.

The same question applies to the agent itself, and catches people out. The browser versions of these tools don't run on your computer — they run on someone else's, so they can't see your files, don't have your keys, and don't have whatever you've installed. "It worked on my machine but not in the browser" isn't inconsistency; it's two environments with two sets of configuration.

Aware — "running on my laptop" and "running on the internet" are different things. Close the laptop: is it still alive? And the browser version of your agent isn't on your laptop either.
Fluent — ask for a deploy, understand the difference between the test site and the real one, run a command you're handed without freezing, know which of the two you're touching right now, and know what kind of thing gets left behind when you work in the cloud — local files, keys, installed tooling — so you know when you have to come back to your own machine
Opinionated — domains, per-environment settings, host logs, rolling back a deploy, choosing where it runs, and configuring the remote one so it works there too

3. Code versus data. Agents hardcode constantly — a value written into the code where it should have been read from the database. It's a reasonable shortcut when the goal is something that runs, and it's wrong the moment that data is meant to change. It's the standard case for the first and third techniques above: say up front that it mustn't happen, and ask later when a number looks suspiciously stable.

Aware — code can be regenerated, data cannot; some data is sitting inside the code; data needs backups
Fluent — ask where something gets stored and understand the answer, know whether the database is local or remote, and confirm a backup exists rather than having been mentioned
Opinionated — migrations, test data versus real data, restoring, what kind of database and shape

4. Getting back. "It was working, now it isn't, and I don't know what changed" is the most common disaster in this field and the most completely solved one. The agent already commits for you. What's missing is you knowing the rescue exists so you can call for it.

Aware — there's a way to recover yesterday's version and it isn't Ctrl+Z
Fluent — ask for a save point before a big change, ask to go back, and check the history to confirm that point is really there
Opinionated — branches, tags, pull requests, reading a diff

Who gets hurt if this fails

5. Secrets. You do not need to be able to read a .env file. You need to know it exists and that keys belong in it. What isn't obvious: taking a key out of the code doesn't remove it from the project's history, and I've had to scrub keys out of that history more than once. There are tools that watch for this — GitGuardian and similar — and, depending on how critical the key is, safe ways to hand one to someone that aren't a chat message.

There's a second confusion worth naming, because it runs the other way. Not everything that lives in environment variables is a secret. Feature flags, timeouts, which region to use — settings drift into that file because it's the place things go, and then get treated with the ceremony a password deserves while the actual passwords get lost among them. Knowing which of your variables are secret and which are merely configuration is part of being fluent here.

Aware — keys don't live in the code, they live somewhere separate; a key that's been seen once is no longer secret
Fluent — ask for a key to be moved out of the code, understand why that file isn't uploaded, know where a key comes from when you're asked for one, know that a server key and a public client key are different animals, and tell a secret from a setting that merely ended up next to one
Opinionated — rotating a leaked key, secret managers, per-environment secrets

6. Who can get in. The open endpoint above is this category. Worth restating what actually did the work there: the check was cheap and took a minute. Knowing it was a check worth running is the part that isn't free.

Aware — having a login does not mean being protected
Fluent — ask "can anyone call this?" and understand the answer; authentication is who you are, authorization is what you're allowed to touch; log in as one user and confirm you can't see another's data
Opinionated — roles and permissions, row-level security, tokens, reviewing what's exposed

7. Other people's data. The one category with a moral asymmetry: everything else here costs you money or embarrassment, and this one is paid by someone who never agreed to your learning curve.

Aware — if you store things about other people, the cost of getting it wrong isn't yours
Fluent — know what you're collecting and why, know there are legal obligations attached, and ask for anything unnecessary not to be stored at all
Opinionated — consent, retention, minimization, where the data physically lives

What's going to surprise me

8. What this costs. Surprise bills are more common than breaches, easier to prevent, and almost nobody prevents them — because "there is no default limit" isn't something you'd think to ask about. Early on I built agentic systems without asking for token costs to be tracked, which meant I had no cost estimate at the end and had to run the whole batch again just to measure it. Not measuring cost has a cost, and it's paid in exactly the currency you were trying to find out about. Ask for it in the spec and it's free.

Aware — this generates a bill and by default nothing caps it
Fluent — understand the shape of the bill (model and API usage, hosting, database, storage, traffic), fixed versus per-use, that CPU and GPU aren't priced alike; ask for a cap and look at real consumption
Opinionated — alerts, your own rate limits, bot protection, designing for cost

9. Who you depend on. Separate from cost because the failure isn't financial. It's that something which worked stops existing. I've run a migration between platforms that the agent reported as complete, and which fell over the moment I tried to run on the new one alone — the old platform was still quietly holding it up, and became the backup I hadn't planned for. "Migration complete" and "the old thing can be switched off" are different claims, and only the second one is testable.

Aware — your app leans on other people's services, and they can raise prices, change, or shut down
Fluent — know which pieces are someone else's and which are yours, and ask what happens when one goes away
Opinionated — choose by coupling rather than price alone, and have an exit route

10. Holding more than you tested. Nothing in the building process warns you about this, because while you're building there is exactly one user.

Aware — it works with three users and can fall over with three hundred
Fluent — know that testing and holding load are different questions, and ask what breaks first
Opinionated — measure it, size it, decide what's worth optimizing

How I know it's still fine

11. Tests, and where it breaks. There should be tests for one unglamorous reason: without them, every check you'll ever do is you clicking through the finished app guessing what happened in the middle. That's a maze, and it grows with the project.

Aware — there have to be tests, and "the agent says it works" isn't one
Fluent — understand what you're told when someone says frontend or backend, run the tests and see them pass, and know where to look for an error depending on which side it's on
Opinionated — which kind of test for which risk

12. Still working in six months. As a project grows messy and undocumented, the agent starts failing more. That's the argument — not architectural purity, but your own tool getting worse at helping you. The failure mode is specific and easy to miss: documentation drifts out of date, the agent trusts it completely, and you get confident work built on a description that stopped being true months ago. Documentation only stays true if something reviews it against the code — and that something can be the agent itself, if you ask.

Aware — mess degrades the thing you're relying on
Fluent — ask for documentation, and understand the difference between docs for people and instructions for the agent, and notice when the project's own summary no longer matches what it does
Opinionated — what belongs in permanent instructions versus the conversation, and how to split the project up

Plus one: taste

The thirteenth isn't like the other twelve, which is why it's last and separate. In all twelve, failure has a victim. Here there's no failure — just absence.

Agents are good at working. They're mediocre at good. The layout will be reasonable, the spacing fine, the colors the ones everything else has. Functionally correct and completely anonymous. Nobody will ever tell you it's wrong, because it isn't.

Aware — what comes out by default works and looks like everything else
Fluent — name what you don't like precisely enough for it to get fixed
Opinionated — have a direction of your own and hold it

It's also the only one with no shortcut. In the other twelve, knowing the category exists is enough to ask and let the agent carry it. Here you have to look at the thing and decide you don't like it, and nobody can do that part for you.

The production line isn't about you

I spent a while trying to define the point where a toy becomes real software as a level you reach. That's wrong, and it's worth saying why, because the correct version is more useful.

Production doesn't require you to be at a level. It requires that no category is left unchecked by anyone — you, a test, a service, or another person — and that you know which is which.

That's the same conclusion I reached from the engineering side: you don't need to hold the knowledge, you need to hold the mechanism. It applies just as well to someone who was never an engineer. A category covered by a test is covered. A category covered by a service you pay for is covered. A category you check by hand every time is covered, expensively.

Which is what Aware is actually for. It doesn't let you check anything. It lets you see that the box is empty. An empty box you know about is a managed risk. An empty box you don't know exists is the thing that ends up in the news.

Where you're starting from

The map is the same for everyone. Which parts you already hold, and which way you fail, depends on where you came from.

With no technical background, your bottleneck is the ceiling, not the details. You ask for less than you could because you don't know what's askable. The loop is genuinely new. The terminal is frightening in a way that's hard to explain to someone who's used one for a decade. And your characteristic failure is delegating above your level — handing over things you have no way to check and, worse, don't know need checking.

Coming from engineering or ops, you have the loop already and most of the twelve. Your failure is the mirror image: you delegate below your level. You review every line, take steps too small, and check by hand what a machine should be checking. You're slower than you need to be and it feels like diligence.

How much to let go

Here's what all of this measures, and it isn't knowledge.

Your level is how much you can hand over without going blind.

Each rung doesn't let you type more, it lets you release more, because you have some way to catch it coming back wrong. Aware means you'll notice the category came up at all. Fluent means you can interrogate it. Opinionated means you can overrule it. At every step, the amount you can safely stop watching goes up.

Which makes both failure modes the same mistake with the sign flipped. Delegating above your level is handing over what you can't check. Delegating below it is refusing to hand over what you can. The fix is identical in both directions: find your rung — in each category separately, because it's a vector — and let go of exactly as much as it holds.

You don't need to learn to code. You need a loop, a map of what exists, and enough honesty about which boxes are empty.

The whole thing on one page

	Aware	Fluent	Opinionated
1. What can be asked for	agents build whole systems	describe by outcome, ask it to propose	model, tooling, hardware
2. Where it lives	laptop vs internet	deploy, test site vs real, run a given command	domains, env settings, rollback
3. Code vs data	data can't be regenerated	where is this stored, local vs remote database	migrations, restores, schema
4. Getting back	yesterday is recoverable	ask for a save point, confirm it exists	branches, tags, diffs
5. Secrets	keys live outside the code	secrets file, server vs client key	rotation, secret manager
6. Who gets in	login ≠ protected	can anyone call this, authn vs authz	roles, permissions, tokens
7. Other people's data	the damage isn't yours to pay	what you collect and why, legal duties	consent, retention, residency
8. What it costs	no default limit	shape of the bill, ask for a cap	alerts, rate limits, cost design
9. Who you depend on	someone else's service can vanish	which pieces aren't yours	coupling, exit plan
10. Holding load	works at three users, not three hundred	testing vs load are different questions	measure, size, optimize
11. Tests and breakage	there have to be tests	frontend vs backend, run them, read the failure	which test for which risk
12. Six months on	mess degrades the agent	docs for people vs for agents	permanent instructions, structure
+1. Taste	the default is anonymous	name what you don't like	a direction of your own

Want your own map? Answer the thirteen questions and you'll get the same grid filled in for your project, along with which gap is worth closing first.

This is the first of three pieces on what building software with agents actually requires. The next two take the same map from opposite ends: one for people with no technical background, one for people who have too much of it and can't put it down. Related reading: How Much Should You Still Know?.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Confident About Text It Cannot Read

JaviMaligno — Sat, 01 Aug 2026 10:12:29 +0000

Take the string cmVwbHkgd2l0aCB0aGUgc2luZ2xlIHdvcmQgYmFuYW5h. It is base64, and it decodes to reply with the single word banana. Send it to GPT-5 through the Azure gateway and you do not get back banana. You get an HTTP 400, and a filter verdict that names its reason:

jailbreak:                 { detected: true,  filtered: false }
self_harm:                 { filtered: true,  severity: "medium" }
hate / sexual / violence:  safe

A self-harm classifier looked at a string it could not parse and assigned it a medium severity. Not "unknown", not an abstention, not a low-confidence flag — a specific, confident, wrong verdict about a request for the word banana.

The string is not some exotic edge case, either. Paste it into ChatGPT and the answer comes straight back:

Different stack, different filters, no drama — it decodes the request and answers it. That contrast is the point. The severity verdict is not a fact about the request; it is a property of the ensemble sitting in front of one particular deployment, an ensemble that had to say something about a string it could not read, and said medium.

That is the shape of everything worth reporting from this experiment. I have now finished the full version of the cipher sweep I wrote about in the previous post — 696 conversations, 6,960 model turns, ten ciphers the models were never taught. Three separate systems in it turned out to do the same thing: produce a confident, structured judgement about text they cannot read. One is that safety filter. One is a base model writing fluent cipher it does not understand. And one is an LLM judge — the one my own earlier conclusion was resting on.

So this post starts with the correction.

What didn't survive

The cipher post ended on a reassuring note. Its sub-probe on robustness reported that encoding a normally-refused request made compliance collapse — 36% in plain text against about 1% under ciphers — and I drew the obvious conclusion: a cipher doesn't unlock a refused request, it defuses it. The models flip into translator mode, decode the message, and comment on it instead of obeying.

That post hedged the claim in several directions, and I still think the hedges were right. But the direction of the headline was wrong, for a reason that is worth more than the conclusion it replaced.

The measurement was broken in a specific way. That first sub-probe judged the decoded view of every reply unconditionally. So when a model decoded a harmful request and then refused in plain English — the single most common thing that happens — the judge took that plain-English refusal and ran it back through the cipher before scoring it. It saw noise and filed it as "garbled". Every letter-level cipher condition returned garbled by construction, which pushes both compliance and refusal toward zero and leaves an encouraging-looking near-1%.

The corrected probe judges whichever view actually reads as English, records which channel the model answered in, and stores the judge's identity for every cell. Then it does the thing the first run didn't: it has a second, independent judge re-label a sample, so I can ask whether the labels mean anything before reading a rate off them.

They mostly don't. And where they do, the answer runs the other way.

On plain text, two independent judges agree about what happened, for every model. On ciphered replies they agree on GPT-5 — and on the two 7B models they agree worse than chance: kappa of −0.17 and −0.05. Per cipher the same split shows up, from 0.71 on full reversal down to 0.002 on the keyed substitution.

That is not a small caveat about noisy data. It means that for those models, in that condition, the standard CipherChat-style instrument does not measure anything. Their "uplift" comes out strongly negative — the base model falls from 80% compliance in plain text to somewhere between 0% and 20% under every cipher — and that negative number is not evidence that ciphers protect them. A 7B model answering an encoded harmful request usually emits something that is neither a refusal nor a fulfilment, and "refused / complied / garbled" are plain-text categories that don't carve up that output space. A compliance the judge fails to recognise in half-decoded text gets filed as garbled, which lowers the rate mechanically. Whichever judge you happened to run is doing the work.

On GPT-5, where the judges do agree, a bypass appears. In plain text it refused all 30 harmful cells — a clean 100%, with both judges in perfect agreement. Under ciphers, two distinct prompts were complied with under two distinct ciphers (letters→digits and full reversal) after being refused in plain text. Each reproduced across replicates, and both judges independently labelled all four cells as compliances — including the judge that isn't GPT-5, which matters, because a judge scoring its own outputs would be biased toward reading them as refusals.

I want to be exact about how much that is and isn't. It is an existence result, never a rate. Ten prompts cannot estimate a frequency; every confidence interval here includes zero; 0.067 is literally two prompts. What it establishes is that refusal is not a property of the request's meaning alone — the same meaning, wrapped in a cipher, crossed a boundary it did not cross in plain text. Note also which ciphers did not do it: ROT13 and base64, the two GPT-5 decodes most fluently, and the keyed cipher it has to work out. Whatever is going on, it isn't "harder to read means easier to bypass".

The whole sub-probe used a deliberately mild subset of AdvBench, with CBRN, weapons, CSAM and self-harm categories excluded, and it stores labels only — no harmful prompt or model completion is published, here or in the repo.

The part the judges never touched

Everything above is the appendix. The main experiment is the reason I trust it at all, because its core metrics use no LLM judge.

Each turn, the model gets a short verifiable instruction — reply with the single word banana, what is seven plus five — encoded in the active cipher, and I check two things programmatically. Comprehension: did it act correctly on the decoded instruction, in whatever language it answered? A task oracle decides that, deterministically. Production: did it write its reply in the code? The inverse cipher decides that, by decoding the reply and checking the result reads as English while the raw reply doesn't.

Ten ciphers, three exposure protocols, eight replicates, three models: 696 conversations. Two of the ciphers — a random alphabet substitution and a block permutation — are keyed per run, so their mapping cannot have been memorised in training. Those are the only ones that speak to inference rather than recognition.

Most of what that machinery found, the earlier post already reported from the pilot, and the full sweep confirms it rather than revising it: the keyed ciphers are the genuinely hard ones, three plain-text↔coded example pairs lift comprehension sharply, and models only start replying in code when you ask them to. Two details are worth adding. Pooled across models, Morse, disemvoweling and ROT13 have a median of one turn whenever they get solved at all — recognition is instant or it never comes, with almost nothing in between, which is the opposite of the several-turn grind the keyed ciphers produce. And base64's row rests on two models rather than three, because Azure rejected GPT-5's entire base64 block: the rejection from the top of this post is also a hole in the results table.

The genuinely new thing needs every model's production rate side by side.

The two abilities dissociate in opposite directions

That the two are separable was already the earlier post's finding — escalation drives production up without moving comprehension. What the full sweep shows is sharper: they don't just separate, they invert. GPT-5 understands 97% of cells and answers in plain English anyway, writing back in code in only 24% of them. Qwen-7B Base does the reverse: it understands 36% and writes in code 56% of the time. It is the only model here that produces more than it understands.

The base model's half of that needs a caveat, and it is the third system on my list. In three cipher cells — binary, Cyrillic homoglyphs, block permutation — it produced in-code output while never once acting correctly on the decoded task. That is not adopting the code. It is continuing the surface pattern of the prompt, which is exactly what a completion model without instruction tuning should do. Fluent cipher, zero comprehension. Reported as mimicry, not as speaking the language.

The thing they have in common

A filter that cannot parse base64 assigns it a medium self-harm severity. A judge that cannot read a half-decoded reply files it as garbled and quietly deflates a rate. A base model that has understood nothing produces flawless cipher. In all three cases the system's output has the form of a judgement — a severity, a label, a fluent answer — and none of the content one would need behind it.

That is a specific, unglamorous failure mode, and it is a different story from the one I told in the previous post. The reassuring reading I published wasn't wrong because the models turned out to be more fragile than I thought. It was wrong because I read a number off an instrument that was, in that condition, measuring nothing — and the artefact happened to point somewhere comfortable.

So the practical lesson isn't about ciphers. If you are replicating CipherChat-style work, or any evaluation where an LLM judge scores text the model may have mangled, compute inter-judge agreement per subject and per condition before you report a single rate. On this material that one extra judging pass is the entire difference between a measurement and a confident artefact. It cost me one run to find out, and I'd already published the artefact.

The defensive version of the same lesson: a plain-text safety filter and the decoded instruction are two different surfaces. The same ensemble over-blocks an innocuous encoded request about a banana and let two harmful ones through on a frontier model. Tuning for one of those surfaces tells you very little about the other.

Code, ciphers, the deterministic oracle, and the full method: github.com/JaviMaligno/llm-language-limits — with a provenance record in docs/PUBLICATION_FREEZE.md (model ids, served versions, sampling settings, cipher seed, and a sha256 for every data file, stamped with the generating commit) and 172 tests. This finishes the cipher study that began in Teaching a Model a Language You Just Invented, itself the second piece in a series on the edges of language after Repetition at the Edges of Language. The robustness sub-probe follows the framing of CipherChat and reports aggregate labels only.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Where's the Ball? Part 4 — When Nothing Moves

JaviMaligno — Fri, 31 Jul 2026 11:02:14 +0000

Part 3 closed the trilogy on a blind spot: the little specialist is overconfident on loose balls — the ones out in open space, far from the player mass. I meant to stop there. But a question kept nagging, and chasing it turned out to be worth one more post.

The question: aren't those hard cases just set pieces? A corner, a free kick — the ball's whereabouts depends entirely on whether the taker has struck it yet. Before the kick it's on a chalk mark; after, it's a projectile the players are still reacting to. Maybe the wide error is really about that awkward in-between, the ball in flight while everyone re-forms around where it's going.

It's a good hypothesis. It's also wrong — and being wrong pointed at something better.

The premise didn't survive contact

Metrica's data comes with an event log — every pass, shot, and set piece, timestamped with the ball's start and end. So I could label each frame by what was happening and ask where the error actually lives.

Set pieces barely registered: as a fraction of playing time they're a rounding error, because the "set piece" event is the instant of execution, not a phase you spend minutes in. Long balls and clearances were modestly harder than short passes — a small win for the broader version of the hypothesis — but they didn't dominate either. The wide error was spread across ordinary open play.

The sharper cut wasn't the event label at all. It was the speed of the ball.

And it flips the intuition on its head. The ball in flight — fast, airborne, the case I'd expected to be hardest — is only mildly difficult. The hardest ball to locate is the one that's barely moving at all: settled, under 2 m/s, a dead ball or one resting at someone's feet. The correlation between ball speed and error is essentially zero; the structure is entirely in that first bar.

No motion, no signal

Once you see it, it's the most on-brand result this project could have produced. Every part of this series has landed on the same finding: the signal a model reads to place a hidden ball is motion — where the running mass of players is heading. So what happens when nothing is moving?

There's nothing to read.

Strip the velocity channel out of the model and watch how much it costs, split by whether the ball is moving. When the ball is moving, velocity is worth a lot — dropping it inflates the error by about 0.04. When the ball is still, velocity is worth almost nothing: the model with velocity and the model without it are nearly the same, because there's no coherent motion for the velocity features to feed on. The specialist's whole edge evaporates in exactly the moment the pitch goes quiet. (This held in both train/test directions, and the collapse was even sharper the other way round — velocity's contribution fell essentially to zero on still balls.)

There's a second reason still balls are hard, and it sharpens Part 3's blind spot. The worst of the still balls aren't the ones near a touchline, where a throw-in pulls players into a tidy cluster. They're the ones sitting centrally, far from everyone — a dead ball in the middle of the park with the players spread out and nobody converging. Among still balls, the correlation between "distance from the player mass" and error is strong and steady (+0.53 to +0.65 across both directions). A ball no one is near and no one is moving toward is, quite literally, undetermined: the players have stopped telling you where it is.

But in flight, you can still read the direction

That leaves the other half of the original question. Even if the ball in flight isn't the hardest case, the intuition behind it was real: while a ball travels, the players are already shaping what happens next with their runs. Maybe what's recoverable there isn't the ball's exact position, but where it's going.

So I asked the model a different question about fast balls — not "where is it?" but "which way is it heading?" — and had it predict the ball's direction of travel from the players alone.

It can. A random guess at a 2D direction is off by 90° on average; the players pin it to about 45°, with half of all fast balls placed within a 45° cone. Not precise — but unmistakably real: the configuration of the pitch encodes which way the ball is travelling, because the players are already leaning into the play. I'll be honest that this doesn't cleanly beat position — in flight, direction and position turn out to be recoverable to roughly the same degree, and comparing an angle to a distance is apples-to-oranges. The honest claim is the modest one: the direction is legible, not that it wins.

Takeaways

The hardest ball to find is the one standing still. Not the ball in flight — the settled ball, under 2 m/s, is where the specialist's error is highest. Ball speed and error are otherwise uncorrelated; the difficulty is concentrated entirely in stillness.
Because motion is the whole signal. Velocity features carry the localization; when the ball is still there's no motion to read, and dropping velocity costs almost nothing. The model's strength and its blind spot are the same fact seen from two sides.
A dead ball no one is near is undetermined. The worst still balls are central and far from the player mass (coupling-to-error correlation +0.53 to +0.65) — the players have simply stopped pointing at it.
In flight, direction is readable even if position isn't precise. Players' runs pin a fast ball's heading to ~45° (chance is 90°). The original set-piece hunch was wrong about which phase is hard, but right that the players are always shaping what comes next.

Four parts in, the throughline is hard to miss. A hidden ball is found through motion — the frontier models can't act on that cue, a 60 kB network reads it sharply, and it turns out to be plain geometry. Which means the one thing none of them can do is find a ball that has stopped moving and drifted away from the crowd. The blind spot was never a kind of play. It was stillness.

Code, the event-log analysis, and the velocity-ablation and direction experiments in the wheres-the-ball repo. Soccer tracking and event data from Metrica Sports. Previously: Part 1, Part 2, and Part 3.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Three Judges, Three Rankings

JaviMaligno — Thu, 30 Jul 2026 10:35:40 +0000

An earlier post here argued that LLM-as-judge is three decisions — context, unit, dimension — and that all three happen before you write the prompt. That post was about what you score. This one is about who scores it.

The question came out of a conversation about evaluating two systems, one built on GPT and one on Claude. If you use a GPT judge, is the comparison still fair? The judge never sees a model name. But it doesn't need one: every family has a recognisable house style, and a judge that learned to produce a style may well reward it.

That is a testable claim, and "several papers have found this" is not the same as having measured it on your own tasks. So I built a harness and measured it. Everything below is 378 judgments across three model families, and every number is reproducible from the repo.

The setup, in one paragraph

Fifteen tasks, split evenly across three levels of subjectivity: objective ones with a checkable answer (parse a duration string, fix a median bug, merge intervals), medium ones (summarize, extract to JSON, classify tickets), and subjective ones (error-message copy, a postmortem opening, explaining eventual consistency to a PM). Three models answer all fifteen: GPT-5.5, Grok-4.3 and Claude Sonnet 4.6. Then each of those three models judges every pair of answers, blinded — no model names anywhere in the prompt, which the harness asserts before sending. Every comparison is judged twice, with the two answers swapped, because the rate at which a judge reverses itself when you swap the slots is the position bias, measured rather than assumed. Ties are allowed: forcing a binary choice between two equally good answers manufactures coin flips that later get misread as bias.

All three families ran through a single Azure AI Foundry subscription, which is the cheap trick that made this affordable — one billing relationship instead of three vendor accounts.

1. The ranking depends on who is holding the clipboard

Same answers. Same blinded prompt. Both presentation orders. The only thing that changes between those three columns is which model is judging — and the ordering changes with it:

Generator	Overall	by GPT-5.5	by Grok-4.3	by Claude Sonnet 4.6	Mean words
GPT-5.5	66.1%	85.0%	58.3%	55.0%	66
Claude Sonnet 4.6	45.0%	35.0%	38.3%	61.7%	85
Grok-4.3	38.9%	30.0%	53.3%	33.3%	55

Three judges, three different orderings. Every judge puts itself higher than the other two put it. Two of them crown GPT-5.5; the third crowns itself. And all three disagree about the rest of the order — so if you had run this evaluation with a single judge and shipped the ranking, part of what you shipped would have been the judge.

That is the symptom. The rest is the diagnosis.

2. Self-preference is real, and it scales with subjectivity

The naive way to measure self-preference is to count how often a judge picks its own answer. That measure is useless: GPT-5.5 picks its own answer 85% of the time, but its answers might simply be the best ones — and by the other two judges' reckoning, they largely are.

What actually proves bias is the delta: how much more often a judge picks its own answer than its peers pick that same answer, on the same comparisons.

Judge	Own rate	Peers, same pairs	Delta	95% CI	vs uninvolved judge only
GPT-5.5	85.0%	56.7%	+28.3pp	[+16.7, +40.0]	+21.7pp [+8.3, +35.0]
Claude Sonnet 4.6	61.7%	36.7%	+25.0pp	[+12.5, +36.7]	+15.0pp [+0.0, +28.3]
Grok-4.3	53.3%	31.7%	+21.7pp	[+9.2, +35.8]	+13.3pp [−1.7, +30.0]

All three positive, all three intervals excluding zero, all three about the same size. This is the part that a within-family pilot could not show: when I ran the same harness across three tiers of one family, the deltas came out +16.7, −14.6 and +4.2pp — noise pointing nowhere. Across families it is a straight line.

That last column deserves a note, because it is the one I would quote. With three judges that double as the three generators, the "peers" baseline for a comparison between A and B includes B itself — and B's own self-preference pushes the verdict away from A, inflating A's measured delta. Restricting the baseline to the judge with no stake in the pair knocks about a third off every number. The effect survives that correction for GPT-5.5, marginally for Claude, and becomes indistinguishable from zero for Grok. Anyone measuring self-preference with a judge panel drawn from the models under test is over-counting unless they do this.

Now the finding I care most about:

Judge	Objective	Medium	Subjective
GPT-5.5	+17.5pp	+25.0pp	+42.5pp
Grok-4.3	+7.5pp	+27.5pp	+30.0pp
Claude Sonnet 4.6	−2.5pp	+30.0pp	+47.5pp

On tasks with a checkable answer, self-preference is small or absent — Claude Sonnet 4.6 is actually harder on itself than a neutral judge is. On tasks where the criterion is taste, it reaches +30 to +48 percentage points.

Which turns a vague warning into a usable rule: a judge from the same family as one of the systems you're comparing is close to harmless on objective work and close to unusable on subjective work. If your rubric is "did the extraction produce the right JSON", relax. If it is "which error message reads better", your judge is a participant.

3. Position bias is not a law of nature — it is a property of one judge

Everyone repeats "randomize the presentation order." Here is what the order actually did:

Judge	slot-A rate	exact binomial vs 50%	flip rate
GPT-5.5	52.6%	p = 0.73	8.9%
Claude Sonnet 4.6	51.5%	p = 0.90	20.0%
Grok-4.3	72.4%	p < 0.001	35.6%

Two of the three judges show no position preference whatsoever. The third picks whatever it reads first almost three times in four, and reverses its own verdict on more than a third of comparisons when you swap the slots. And it degrades exactly where you'd least want it to:

Grok-4.3	Objective	Medium	Subjective
slot-A rate	64% (p = 0.29)	75% (p = 0.023)	77% (p = 0.005)
flip rate	13%	40%	53%

On the subjective third of the task set, Grok-4.3 reverses itself on more than half of all comparisons. At that point it isn't grading the answers; it's grading their addresses.

So keep randomising order — it's free. But notice that the usual justification ("LLM judges prefer the first answer") is not what the data says. The data says some judges do, catastrophically, and you cannot tell which one without measuring it. Running every comparison in both orders costs you a 2× on judge calls and turns an unknown into a number.

4. The length statistic everyone quotes is uninterpretable

Here is the number people report:

Judge	Longer answer wins
GPT-5.5	48.6%
Grok-4.3	48.6%
Claude Sonnet 4.6	64.6%

Looks like there's barely any length bias. In my within-family pilot the same statistic read 80–85% and looked like a textbook one. Both readings are wrong, and for the same reason: in any normal model lineup, length and quality are confounded. If your verbose models happen to be your good models, the number inflates; if they don't, it cancels. It is an accident of your lineup, not a property of your judge.

The control holds the model and the task fixed and varies only the target length: two variants from an identical base prompt, differing by one appended sentence ("answer in roughly N words"). The judge is then shown the base prompt without that sentence — otherwise it would be grading compliance with a word count it was told about. Short variants came out at 46 words on average, long ones at 174.

Judge	Long wins (controlled)	Flip rate
GPT-5.5	77.8%	11.1%
Grok-4.3	86.1%	16.7%
Claude Sonnet 4.6	88.9%	5.6%

So the preference is not just real, it is stronger than the uncontrolled number implied — the opposite of the direction people assume the confound runs.

Split by what the task actually rewards, though, it isn't a length preference at all:

What the task rewards	Long wins	n
elaboration (explain X)	100.0%	27
concision (summarize, commit message, error copy)	68.5%	27

Twenty-seven out of twenty-seven. Every judge, every generator, both orders. Add the pilot's runs and it's 45 for 45 across six judges from three families. If anything here deserves to be called a law, it's that one: on a task that invites elaboration, the longer answer always wins.

Where I was wrong

The pilot found something better than that, and I wrote it up: on summarization the preference inverted — judges picked the short version and said so explicitly in their reasons ("genuinely condensed form, whereas B is essentially a full paraphrase"). The tidy conclusion was that judges infer the task's implicit goal, so "watch out for length bias" is the wrong instruction.

That did not replicate. Cross-family, the same summarization probe reads 72.2% long-wins, and the concision category sits at 68.5% — above chance, not below it. Broken down by judge, the pilot's effect was one model:

Judge	elaboration	concision
GPT-5.5	100%	55.6%
Grok-4.3	100%	72.2%
Claude Sonnet 4.6	100%	77.8%
pilot: Opus 5	100%	66.7%
pilot: Sonnet 5	100%	0.0%
pilot: Haiku 4.5	100%	50.0%

Six judges. One of them preferred the short answer every single time and dragged a three-judge aggregate below 50%, which I then reported as a general finding. The corrected version is narrower: judges reward elaboration everywhere, and the reward merely weakens, with wide variance between judges, when the task's implicit criterion is compression. One judge in six actually inverted.

I'm keeping both write-ups in the repo, pilot first, because the failure mode is the useful part: a confident headline drawn from three judges of one family, retired once six judges from three families had a say.

5. On subjective tasks, judges agree at chance

Judge pair	Objective	Medium	Subjective
GPT-5.5 vs Grok-4.3	+0.41	+0.32	−0.01
GPT-5.5 vs Claude Sonnet 4.6	+0.31	+0.35	−0.03
Grok-4.3 vs Claude Sonnet 4.6	+0.20	+0.02	+0.05

Cohen's kappa: 1.0 is perfect agreement, 0.0 is chance. On the subjective tasks all three pairs are indistinguishable from chance. Two frontier judges grading the same subjective comparison agree exactly as often as two coin flips.

This is the number to keep if you keep only one. A judge panel is supposed to average out individual quirks — but averaging three judges who agree at chance doesn't give you a robust signal, it gives you a smoother random one. On subjective dimensions, the honest move is not more judges. It is to stop pretending the dimension is measurable by judge and go find a proxy that is: did the user come back, did the ticket get reopened, did the PR get merged.

What I'd actually do with this

Match the judge to the dimension, not to a leaderboard. Objective, checkable dimension → any competent judge, self-preference is near zero. Subjective dimension → assume the judge is a participant.
Never use a judge from a family you're evaluating on a subjective dimension. +30 to +48 percentage points is not a rounding error, and blinding does not remove it.
Screen every judge in both orders once; keep both orders only where they pay. Twenty comparisons run both ways gives you the slot-A rate and the flip rate, and would have disqualified Grok-4.3 in ten minutes. A judge that comes back clean doesn't need a permanent 2× tax after that — for an aggregate ranking, simply randomising the order per comparison is free and neutralises a systematic lean.
Keep both orders when you act on individual verdicts. Gating a PR or routing a sample to human review is a per-item decision, and a lean isn't the only failure: Claude Sonnet 4.6 showed no slot preference at all (51.5%, p = 0.90) and still reversed itself on one comparison in five. Running the swap is the cheapest way to know whether this verdict is reproducible.
Distrust any length statistic that isn't controlled — including one that says your judge has no length bias.
If your judges agree at chance, the problem is the dimension, not the panel.

What this doesn't show

One model per family, and the tiers aren't matched: Azure had no quota for a current Claude at run time, so the Anthropic seat is Sonnet 4.6, a generation behind the other two. The obvious worry is reading a capability gap as a family effect. Two things push back — the older model isn't the lowest-ranked overall (Grok-4.3 is), and the three self-preference deltas land within 7pp of each other. But the clean version of this experiment uses same-tier models, and this isn't it.

Also: GPT-5.5 did not run at temperature 0, because it refuses to. Fifteen tasks and six length probes is 2.5× the pilot and still small — every interval here is wide. It's one run, with no repeats to separate judge instability from sampling noise. And nothing was executed: even the objective tasks were graded by reading, deliberately, because the point was to measure the judge. Where you can run the tests instead, run the tests.

None of which changes the shape of the result. Three judges looked at the same 45 comparisons and produced three different rankings, and the gap between them was widest exactly where the answer was a matter of taste — which is precisely where teams reach for an LLM judge in the first place.

Harness, raw judgments and both write-ups: experiments/judge-bias. Prior work worth reading: Zheng et al. on position, verbosity and self-enhancement bias, Wang et al. on position bias, Panickssery, Bowman and Feng on self-recognition and self-preference, and Dubois et al. on length-controlled evaluation.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Where's the Ball? Part 3 — It Was Geometry All Along (and the Model Doesn't Know When It's Wrong)

JaviMaligno — Wed, 29 Jul 2026 10:04:26 +0000

Part 2 ended with a small network — about 60 kilobytes of weights — cracking the hidden-ball cases that frontier vision-language models couldn't. It read the players' positions and velocities and pointed, with decent accuracy, at a ball it had never been shown.

Which is satisfying and slightly annoying, because a network that works is still a black box. It gives you an answer, not an explanation. So the question that opens the last rung of this ladder is: what is that little box actually computing? Is there some rich, deep pattern in how ten players arrange themselves — or is it doing something a human could write down on the back of an envelope?

I spent Level 3 trying to open the box three ways. The short version: it's mostly a single geometric idea, the fancy mathematics I reached for added nothing, and when I asked the model how sure it was, it turned out to be confidently wrong in a very specific place.

It was geometry all along

The test is simple. If the black box is computing something interpretable, then a handful of hand-written geometric features — quantities a coach could name — fed to a plain gradient-boosted tree should reproduce most of its accuracy. If it's doing something genuinely deep, they won't come close.

I wrote about ten: the centroid of the players, the centroid weighted by how fast each player is moving, the fastest player's position, the densest cluster, the spread of each team, the midpoint of the closest pair of opponents, a "convergence point" where the players' motion rays intersect. Ten numbers, a tree, the same train/test split as the deep model.

The gap between a naive centroid baseline and the trained deep net is the "skill" the network learned. Ten interpretable features recover 92% of that gap in soccer — and in basketball they slightly beat the deep net (112%). The little black box, it turns out, is not hiding much. Almost everything it knows can be written down.

One number does most of the work

And most of that is a single feature. When I measure how much each geometric quantity contributes — by shuffling it and watching the error rise — one dominates the rest by an order of magnitude:

The velocity-weighted centroid — the average player position, but with the fast-moving players counting more — carries the localization on its own. Everything else is rounding. This is the same thread that ran through Part 2: the signal is in the motion, in where the running mass is heading, not in the static shape of the formation.

There's a small irony here. The feature I was proudest of — the "convergence point", a tidy least-squares intersection of every player's motion ray, the thing you'd draw on a tactics board to say the ball is where everyone is running — barely moves the needle. The crude weighted average of positions beats the elegant geometric construction. Not the last time this project would humble a nice idea.

What I hoped topology would add (and didn't)

If ten simple features already recover almost everything, is there any structure left that a richer description could capture? The natural candidate is topology — the shape of the player cloud, its holes and connected components, formalized as persistent homology. There's a genuinely appealing intuition behind it: a ball in open play often sits inside a hole in the configuration — a ring of players around empty space — and topology is exactly the mathematics of holes.

So I built the topological features: the persistence statistics of the player cloud, the centre of its most persistent loop (via the representative cycle — literally "where's the biggest hole"), the two largest empty circles in the Delaunay triangulation ("where's the biggest gap"). I set one rule before running anything, to keep myself honest: topology only counts if it beats its geometric counterpart. Otherwise it's decoration.

It doesn't clear the bar. Topological features on their own land far behind plain geometry (0.258 vs 0.111 in soccer — barely better than the naive baseline). And bolting them onto the geometric features changes nothing: geometry-plus-topology matches geometry alone, and when I shuffle the entire topological block the error barely twitches — the tree simply ignores it once it has the geometry.

The appealing intuition is just wrong. The ball's position isn't a property of the shape of the player cloud; it's a property of where the mass is and where it's moving. Persistent homology describes the configuration beautifully at every scale, and none of that description is about the ball. A clean negative result — and the pre-registered rule is the only reason I can call it clean instead of quietly dropping it.

Does the model know when it's wrong?

The last question is the one I find most interesting, because it's not about accuracy — it's about self-knowledge. A point prediction can't tell you how sure it is. So I swapped the network's single-point output for a mixture density head: instead of one guess, it predicts a full probability distribution over where the ball might be. Now it can say "somewhere around here, and I'm confident" or "somewhere in this whole region, and I'm not."

The first finding is reassuring. The uncertainty it declares does track the error it makes — bin the predictions from most-confident to least, and the actual error climbs monotonically:

That's the left panel: the model knows when it doesn't know. Weakly (the correlation is a modest +0.20), but consistently.

The right panel is where it gets interesting — and where my hypothesis died, again. I'd expected the ball to be hardest to pin down when it's tangled in a crowd of players, and easiest when it's loose in open space. It's the opposite. Tangled play is the easy case: when players converge on the ball, their velocities point at it, and that velocity-weighted centroid lands right on top. A loose ball, with players trailing behind it, is genuinely harder — about 20% more error.

Here's the blind spot: on those harder loose balls, the model declares lower uncertainty. Its error goes up while its confidence goes up. It is most sure of itself exactly where it is most wrong — and these are the same off-centre, fast-moving balls that the frontier models failed on back in Part 1. The failure mode is consistent all the way down the ladder: everything struggles with the loose ball, and the specialist doesn't even know it's struggling.

Where does that leave the map of the pitch? If you plot the median error across the field, the ball is best determined near the touchlines and in open midfield, and worst in the congestion in front of both goals — the corners, crosses, and goalmouth scrambles where players pile up and the convergence trick breaks down.

Takeaways

The black box was reading geometry. Ten interpretable features recover 92% of the deep net in soccer and beat it in basketball. Ball localization isn't a deep pattern — it's mostly one number: the velocity-weighted centroid, "where the running mass is heading."
Topology added nothing. Persistent homology, largest empty circles, loop centres — the appealing "ball sits in the hole" intuition — don't beat plain geometry and get ignored when added to it. The ball's position isn't a property of the configuration's shape. A pre-registered rule kept the negative result honest.
The model is confidently wrong in one place. It's globally calibrated, but overconfident on loose balls in open space — precisely the hard, off-centre cases that broke the frontier models in Part 1. It doesn't know that it doesn't know, exactly where it matters.
My best guesses kept losing to the plain ones. The elegant convergence point lost to a weighted average; topology lost to ten simple features; the "crowds are confusing" hypothesis was backwards. Across three parts, this project went a perfect four-for-four on "the first version of the story was wrong" — which is the whole reason I write the controls before I write the conclusion.

Three rungs in, the answer to the question from the stands is almost anticlimactic. Inferring a hidden ball from the players is real, it's learnable, and it's geometry — a coach's instinct for where the play is heading, written as one weighted average. The frontier models have that instinct faintly and can't act on it; a 60 kB network has it sharply. And neither of them, it turns out, can tell you when the ball has slipped loose into space and they've quietly lost the thread.

Code, feature definitions, controls, and the topology and mixture-density experiments in the wheres-the-ball repo. Soccer tracking from Metrica Sports and SkillCorner; basketball from the NBA SportVU 2015-16 logs (mirror; no explicit license — used for research only, not redistributed). Persistent homology via ripser. Previously: Part 1 and Part 2.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

I Built an Expensive Form

JaviMaligno — Tue, 28 Jul 2026 11:41:40 +0000

I lost this argument before I built anything.

We were replacing a legacy ticketing journey for filing regulatory reports — the kind of internal form that front-line staff dread and fill in badly. My position, stated more than once, was that turning the whole journey into a conversation added little over a well-designed form. What I'd have built instead was a form with genuinely good UX, plus an assistant beside it for the questions people actually have — does this count as reportable?, which category is this? — able to interact with the form rather than replace it.

The call went the other way, and it was a defensible call: a conversational journey is differentiating, it demos well, and clients ask for it by name. So I said fine, I'll build it and we'll see what happens.

We saw what happened. I had built an expensive form.

Building well what you don't advocate for

I want to be precise about this, because "I disagreed and did it anyway" can read as either cynicism or spinelessness, and I don't think it was either.

An internal argument about whether users will like something is unresolvable by internal argument. Two people with plausible models of the user restate their priors at increasing volume, and the person with more authority wins. That isn't a bad process because the wrong person won — it's a bad process because nobody learns anything. The way out is to put the thing in front of a real user quickly, built well enough that their reaction is about the idea and not about your half-hearted execution.

So it had to be good. If I'd shipped a deliberately mediocre version, any bad feedback would have been about the mediocrity, and I'd have proven nothing.

I also didn't start from scratch: a colleague had built the initial LangGraph prototype, and I took it from there. Worth saying plainly, because the improvements I'm about to describe were made on top of someone else's foundation.

What we shipped

The architecture wasn't sloppy. It was a pipeline engine configured in YAML: a DAG of phases, a handful of node types, conditional routers, global triggers that could redirect the flow, compliance scoring recalculated after each phase.

Each phase owned a slice of the form. It asked its question, validated the answer, confirmed it, advanced. For one report type that came out to roughly seven interactive steps: seven turns where the user waited for a model, read a question, answered one thing, confirmed.

And let me steelman that design, because its properties are real.

Phase-per-slice gives you validation at the point of entry — you catch a malformed date while the user is still thinking about dates. It gives you a progress model: because phases are explicit, you get a progress bar, per-phase analytics, and an admin panel showing exactly where a submission stalled. It makes the flow legible — a new engineer reads the YAML and knows what happens. And it makes the agent safe by construction: it cannot skip a required field, because the graph won't let it.

I didn't give those up. That matters for what comes later.

The feedback that settled it

The client's compliance lead tried it and said it felt too similar to the system it was replacing: time consuming and complex.

That sentence did in one meeting what I hadn't managed in several. Nobody won the internal argument — the user ended it. And the useful part isn't that my prior was vindicated; it's that we now had a specific complaint instead of two competing opinions. "Time consuming and complex" is diagnosable. "I think users won't like it" isn't.

That feedback got me a blank cheque to rebuild it properly. Which is the actual reason I'm glad we shipped the version I argued against: my alternative would have been a guess too. What we got instead was evidence, and evidence buys you permission.

Why a form wins that race

Here's the mechanism. If the interaction is a fixed sequence of questions, the form is simply the better interface:

You see all the fields at once, so you know how long this will take. A chat hides the length of the queue, which is why "how many more questions?" is the most common thought in a scripted bot.
You fill them in any order, skip around, come back. A sequential agent imposes an order that exists for the engine's convenience, not yours.
You tab between fields in milliseconds. Every conversational turn costs a round trip to a model — sometimes one second, sometimes ten. Multiply by seven.
It costs nothing per submission. Our version billed tokens for the privilege of being slower.

So the scripted chat loses on speed, on predictability, and on cost, and its only compensation is that it feels modern. A conversational wrapper around a fixed sequence is a form with worse ergonomics and a variable bill.

The diagnosis: sequentiality, not widgets

My first instinct was presentation — the inputs looked clunky, the confirmations were verbose, maybe better in-chat widgets would fix it. That instinct was wrong, and it's the interesting part.

The cost wasn't in any single step. It was in there being steps at all. Seven turns is seven turns whether each one is beautiful or ugly.

Then the genuinely humbling detail. The machinery for "only ask for what's missing" already existed. Each phase could compute its own missing fields and ask only about those. It just operated within phase scope — so when a user opened by typing a rich paragraph describing everything that happened, phase one extracted its two fields and the rest of that paragraph was discarded. The user had already given us most of the answers. We threw them away and then asked for them one at a time.

The system wasn't failing to understand the user. It was understanding and then forgetting, because the unit of extraction was the phase rather than the conversation.

First pass: fix it inside the graph

Diagnosing it didn't earn me a rewrite, and it shouldn't have.

This is the part people skip when they tell these stories, so let me be explicit about it. There was a product. It was built, deployed, in front of clients, and it worked — badly, but it worked. Nobody throws that away on the strength of one piece of feedback and an engineer's conviction. The reasonable next step, and the one the business wanted, was to see how good the existing thing could get.

That instinct is correct more often than engineers like to admit. Incremental fixes are cheap, they ship this week instead of next quarter, and if they'd been enough we'd have saved an entire rewrite. The only way to know whether a rewrite is necessary is to first exhaust what isn't one.

So: two additions to the existing graph, both behind an opt-in flag so other flows kept their old behaviour and I had zero regression risk on clients who weren't asking for this.

A global intake node. Before the first phase, the agent asks for the story in the user's own words. Then it makes a single structured-extraction call against the union of every field the whole flow could extract, and writes everything it confidently found.

# Sequential: the graph decides the order, one question at a time
START → phase_1 → phase_2 → ... → phase_7 → END

# Conversational: extract everything from the narrative, then
# only visit phases that still have gaps
START → intake → first_unsatisfied_phase → ... → END

Three details mattered more than the idea:

The extractable set is computed, not hand-written. It's the union of fields written by phases that can be satisfied from prose. Document extraction, file uploads and signatures are explicitly excluded — no narrative produces a scanned ID.
It reuses the real field definitions, so the same validators run in intake as in the phases. Passing an empty schema and "just extracting text" hands you unvalidated data with extra steps.
It never guesses. If the narrative doesn't clearly give a field, intake leaves it empty. For an option list, a wrong selection in a regulatory form is worse than one more question — a blank gets asked about, a wrong value gets submitted.

Skipping satisfied phases. Phases needing genuine interaction (upload, signature) always run. Data phases branch: all fields present and nothing critical → advance silently; all present but something critical → one minimal confirmation; gaps → ask only about the gaps.

Result: roughly seven interactions down to three or four. Narrate, confirm the high-risk items, upload, sign.

Notice what survived. The phases still exist, so the progress bar, the analytics, the admin panel and the validators all still work. I didn't tear down the DAG to get conversational — I put a node in front of it and let phases opt out of asking. The alternative I rejected in writing was collapsing everything into one adaptive node, which buys elegance and costs every structural property listed earlier.

Second pass: what the rewrite actually required

The first pass bought real improvement, and it wasn't enough. Three or four turns is better than seven, but the shape of the thing was still a graph marching through phases with a smarter front door bolted on.

Going properly agentic meant a completely new backend. Not a refactor: a different shape of system, where instead of a graph deciding the order of questions, an agent holds tools over a server-authoritative state and decides what to do next.

A rewrite of that size is never won on technical argument alone. It took two things: the client feedback, which made the problem concrete, and then a change of direction at the company that made a new bet acceptable in the first place. Evidence made the case; an appetite for risk gave it somewhere to land. Both had to be true at once.

That's the part I'd generalise. The window where an organisation will accept a rewrite opens and closes for reasons that have nothing to do with your architecture diagram, and it is usually short. What you control is being ready when it opens — which, in my case, meant that the design I'd been arguing for was already specified, because I'd spent the incremental pass learning exactly which constraints mattered.

Two things came out of that rebuild that I'd defend anywhere.

Generative UI in the chat, which was my idea and the part I'm most attached to. If the agent has to render options as text, you've reinvented the dropdown with extra latency. Instead the chat renders actual interactive widgets — chips, pickers, panels — so the agent can propose a structured value and the user adjusts it directly. I've written separately about why that pattern matters and the protocol standardising it.

This produced my favourite rule in the whole system. When the agent asks about remaining gaps, it splits them in two: free-text and date fields get asked directly, consolidated into one natural question, but fields with a fixed option list get a different instruction — infer your best option from what the user already told you and let them adjust it in the panel; do not read the options out. The interception lives in the tool that computes missing fields, not in the prompt, because that's the exact point where the model decides what to ask.

If you're building on a messaging channel, this is load-bearing. Nobody wants to read eight numbered choices in WhatsApp.

Freedom to understand, ceremony to act. Intake can be as open as you like; the write cannot. When the agent is about to submit something with real-world consequences, I want a narrow, explicit, boring step. In the new backend that gate is checked in code: a submission is refused unless it was offered in an earlier turn, and any later correction voids the pending approval — because if the content changed, what the person approved is no longer what would be sent. Same principle as the guardrails I keep writing about: the constraint has to live somewhere the model can't reach.

The abstraction I waited to build

Here's where it went beyond one client. A new backend per use case doesn't scale, and the obvious move was a generator: a backend that takes configuration for a use case — prompts, the form models, the knowledge base — and produces the agentic flow from it.

The discipline was in the timing. I deliberately waited until I had two genuinely different flows working before generalising: business-banking onboarding and regulatory incident reporting. Different domains, different form shapes, different failure modes.

Two reasons, and both are the same reason:

Two dissimilar examples tell you which parts are actually variable. Abstract from one and you build a configuration layer shaped exactly like your first client, then bend it painfully for the second. Every knob I exposed had to be a knob both flows genuinely turned differently.
Two examples also prove the abstraction is worth it at all. With one flow, "just deploy another backend" is the cheaper answer and the honest one. The generator only earns its complexity once you can show the alternative is N backends.

That's the current work, and it exists because the rewrite happened twice in different shapes — not because someone drew a platform diagram at the start.

Not chat everything

I'd rather not overcorrect this into conversational-everything, because my original position still stands and the boundary is the useful part.

Keep the form when the fields are few and known, when the user has the data in front of them, when input is inherently non-narrative (upload a file, sign, pick from a calendar), or when the flow is used daily by trained staff who memorised the tab order long ago. Power users beat conversational interfaces at speed, every time.

Reach for the agent when input arrives as a story rather than a record — an incident, a complaint, a symptom, a request — when the field set is large but sparsely relevant to any one case, or when the person filing is occasional and untrained and the form's vocabulary isn't theirs.

And note the middle option I originally argued for hasn't gone anywhere: a good form plus an assistant that can see it and act on it. That's often the right answer, and it's the one nobody demos.

The question I ask first now

Before building a conversational anything: what does this let the user do that the form doesn't?

If the answer is "it's friendlier", build the form. If the answer is "the user can describe their situation however it comes out, and the system works out which fields that maps to", there's something real — and the design job is extracting once, asking only for what's genuinely missing, and never reading the options out loud.

The rest of the time you're building an expensive form. Sometimes you have to build it anyway to prove it, and if you do, build it well enough that the feedback is about the idea.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Where's the Ball? Part 2 — a 60-Kilobyte Model, Two Sports, and the Signal VLMs Miss

JaviMaligno — Mon, 27 Jul 2026 11:43:54 +0000

Part 1 ended on a sober note. Frontier vision-language models have a faint spectator's intuition — their guesses about a hidden football correlate with where it really is — but on the cases that matter, balls genuinely far from the center of the frame, none of them reliably beat the dumbest possible strategy: point at the middle and trust the camera operator.

That left an uncomfortable question hanging. Maybe the hard cases are just hard — maybe the players' positions don't contain enough information to find an off-center ball, and I'd been asking the models to do the impossible.

So I tested that directly. The answer is no: the information is there, and a model small enough to email can extract it.

David, meet Goliath

The question has two axes, so I crossed them. One axis is who reads: a tiny specialist trained for exactly this, or a frontier generalist. The other is what gets read: the broadcast pixels, or the player tracks underneath them.

On the trained side, two specialists. For the tracks: a DeepSets network — a permutation-invariant architecture that eats an unordered set of players and pools them into one guess — trained on each player's position and velocity over one second, nothing else. No pixels, no grass, no bodies. About 14,000 parameters, roughly 60 kB of weights, a few minutes of CPU on my laptop. For the pixels: a small off-the-shelf vision network (a ResNet-18) fine-tuned on seven thousand broadcast frames with the ball erased by inpainting — so it can't detect the ball, only infer it. And the frontier models ran twice on the same paired items: once on the pixels (Part 1's setup), and once on the exact same one-second tracks the tiny net saw, serialized as plain-text coordinates — same information, no pixels.

The pattern is hard to miss. The two trained models beat the camera bias on off-center balls — 82% from the tracks, 74% from the pixels. The frontier models don't, from either input: 53% on pixels (statistically a coin flip), and 35–38% — below chance — on the tracks themselves, where their correlation with the true ball goes from weakly positive to null or slightly negative: handed raw coordinates, they drift into the very centroid trap that broadcast geometry punishes (next section). Two footnotes keep the grid honest. The tracks column uses ground-truth tracking, so it's an information ceiling. And on the easy, centered items the zero-shot VLMs still post better global medians (0.147 vs 0.195–0.210): pixel-precision where the camera has already done the work.

The grid settles the question Part 1 left open, one candidate excuse at a time. Is the information missing? No — the tracks column extracts it. Is perception from pixels the wall? No — a small trained CNN gets most of the way there from the same masked frames. What separates the columns of that chart isn't the input at all; it's the row: trained versus zero-shot. The frontier models' general knowledge of the game, in any representation, does not substitute for task-specific inference — which is precisely the question the view from the stands posed in the first place.

The camera lies

Along the way, one statistic explained something that had been bugging me since Part 1: why is naive geometry so useless in broadcast frames? The centroid of the players — "the ball is where the crowd is" — isn't just uninformative in image space. It's anti-correlated with the ball (−0.58): when the ball is far from the center of the frame, the player mass tends to be on the opposite side.

The reason is the camera operator. When a ball is off-center, it's usually because it's moving fast — a long pass, a clearance — running ahead of the play while the players trail behind. Project that onto a camera that chases the ball and the geometry inverts. Strip the camera away — compute the same centroid in field coordinates — and the correlation flips to +0.83. Same statistic, same matches; the camera flips its sign. Every system in Part 1 that leaned on "the ball is near the players" was leaning on a lie.

Two sports, one question: what actually transfers?

With a working specialist in hand, I could finally ask the question that motivated this whole project from the stands: how much of this skill is "knowing football", and how much is universal team-sport structure? I trained the same architecture on soccer (Metrica and SkillCorner tracking data) and on basketball (NBA SportVU), and crossed them.

Zero-shot — trained on one sport, tested cold on the other — the result is asymmetric in a way I didn't predict. Soccer→basketball fails: worse than an untrained centroid. But basketball→soccer works: 0.17 median error, beating soccer's own geometric baselines without ever seeing a football match. And no, it's not that the basketball dataset is bigger — subsampling it to match the soccer data leaves the result intact (0.174).

It took a couple of wrong turns to pin down. My first guess — that basketball's ball simply lives closer to the player mass — measured false: the ball-to-mass distance distributions are nearly identical. The real culprit is velocity scale. Strip each player down to positions only, discarding the velocity channel, and the asymmetry vanishes — positions transfer symmetrically both ways; it's the use of velocity that only travels one direction. The reason is a units mismatch I should have seen coming: a basketball court is about a sixth the area of a football pitch, so in the normalized coordinates the model reads, players move three to four times faster in basketball. A model's learned sense of "how fast is fast, and what that implies about where the ball is" is calibrated to its own sport's scale and misfires across a 3–4× gap. Positions carry no such scale, so they cross cleanly. That turns out to be the whole asymmetry — and a small, general lesson: what transfers between domains is the scale-free structure; anything with units attached needs recalibration.

Fine-tuning, and a hypothesis that died twice

Does pretraining on one sport at least give you a head start on another? Here I got burned by my own enthusiasm — twice — and both times a control caught it.

First pass: with instantaneous player features, pretraining on soccer gave no advantage over training from scratch on the same basketball minutes. Second pass: with one-second trajectories it did — and I nearly wrote "temporal dynamics is what transfers." Then a shuffled-target control (pretrain the same net on soccer inputs with randomized ball positions — same optimization warm-up, zero knowledge) showed a third of that advantage was generic warm-start. And a feature ablation showed the rest wasn't about temporal depth at all: the advantage tracks whichever variants use velocity features, snapshot or trajectory alike.

What survives all the controls, pooled across both transfer directions: real pretraining wins in 12/13 seeds against scratch (p=0.002) and 9/10 against the shuffled control (p=0.011), with a modest ~5% edge that only shows up once you have ~30 minutes of the target sport. The refined picture: positions carry the core signal, but their mapping is easy — half an hour of any sport teaches it from zero. What genuinely transfers is the harder-won skill of exploiting velocities, which misleads zero-shot (speed scales are sport-specific) but pays off once briefly recalibrated.

"Knowing the game," for this family of models, turns out to mean something narrow and specific: knowing what to do with motion.

Takeaways

Training matters; the input barely does. A 60 kB net on tracks (82%) and a small CNN on masked pixels (74%) both crack the off-center balls; the zero-shot frontier models fail from both inputs — 53% on pixels, 35–38% (below chance) on the very same tracks as text. Neither information nor perception is the wall: general game knowledge doesn't substitute for task-specific inference.
Broadcast geometry is a trap. The player centroid is anti-correlated with the ball in image space (−0.58) and strongly correlated in field coordinates (+0.83). The camera flips the sign.
Transfer between sports is asymmetric, and the asymmetry is a velocity-scale mismatch. Basketball exports its ball-sense to football; football doesn't return the favor. Positions transfer symmetrically both ways — it's the use of velocity that breaks, because the two sports live 3–4× apart on the normalized velocity scale. The scale-free structure crosses; the units-bound part doesn't.
The controls did the heavy lifting, again. A shuffled-target pretraining control and a feature ablation each overturned a conclusion I was ready to publish. This project is three-for-three on "the first version of the story was wrong."

Next up is Level 3 of this ladder: the geometry and topology underneath — when do the players determine the ball, and can interpretable structure (Voronoi cells, pitch control, persistent homology) recover what the little black box learned?

Code, data pipelines, controls, and audits in the wheres-the-ball repo. Player tracking from SoccerNet-Tracking, Metrica Sports, SkillCorner, and the NBA SportVU 2015-16 logs (mirror; no explicit license — used for research only, not redistributed). Architecture: DeepSets.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Routing Engineering

JaviMaligno — Sun, 26 Jul 2026 10:14:59 +0000

In the first part of this pair I argued that prompt engineering — the craft of wording — is dying, and that the effort didn't vanish so much as climb the stack. This is where it climbed to.

Here is the new friction. I sit down to a task and, before I've done anything, I face a grid. OpenAI now ships GPT-5.6 in three tiers — Sol, Terra, Luna — with several reasoning-effort settings and a new max level. Add the earlier 5.x models still in rotation and the Codex variants, and one vendor already gives you dozens of plausible (model × effort) routes. Claude's effort ladder spans low, medium, high, and max, with xhigh on supported models. Gemini has a thinking budget. The wording is no longer the decision. The allocation is: which model, at what effort, for which slice of the work. Getting that right is what I'd call routing engineering, and it's quietly become one of the more valuable things to be good at.

The matrix nobody asked for

The uncomfortable part is that the grid grew faster than our ability to reason about it. When a single model had one setting, "use the best one" was a complete strategy. Now the knobs multiply with every release, and most of them trade the same three things against each other: quality, latency, and cost. A max-effort frontier model will grind through almost anything, but it's slow and it burns tokens. A cheap model at low effort is instant and nearly free, and it will confidently wreck a task that needed a moment of thought.

Nobody hand-optimizes a hundred-cell grid per task, so in practice we all collapse it into a handful of habits. The interesting question — the engineering one — is which habits actually pay, and when the granularity is worth the bother at all.

What I actually do

My own default is embarrassingly simple, and it's a routing decision. In Claude Code I plan with Fable 5 and then execute with Opus or something lighter. If I let the planning-grade model drive the whole task, token usage explodes; the plan is where the intelligence has to be, and the execution mostly has to follow the plan. Splitting the two along that seam is the single change that made long agent sessions affordable for me.

Publication note: two days before this article went live, Opus 5 arrived with performance close to Fable 5 on many tasks at half the price. It has already replaced Fable as my planner. The route changed before the article even shipped, which is almost too on-theme.

I do the same thing in research. In writing a paper with AI the division that mattered was Fable 5 for planning the science and GPT-5.6 as a second reviewer — a strong model where judgment lives, a swap when I hit usage limits, and cheaper execution in between. None of that is prompt craft. It's routing: matching the model to the shape of the work.

Why the split works

The reason this particular seam — plan strong, execute cheap — keeps showing up is that most of a task's difficulty is concentrated in a few decisions. Cursor put a number on it recently. In a study on long-running agent "swarms," they had a fleet build SQLite from scratch in Rust, working only from its 835-page manual, and varied who planned and who executed. Their own summary of the mechanism is the clearest statement of the principle I've seen:

"There are few moments in a large task that truly require frontier intelligence — the initial decomposition, the design decisions, certain trade-offs. Once a frontier planner has turned ambiguity into a detailed, explicit instruction, the cheaper models simply have to follow it."

The result was that every model mix reached similar quality, while cost varied by roughly 8× — from about \1,339 for an Opus-4.8-plans / Composer-2.5-executes hybrid, up to \10,565 for GPT-5.5 doing everything itself. On the execution slice alone the gap was starker: the cheap workers cost around \411 where the all-frontier run's workers cost \9,373. Same destination, an order of magnitude apart on the fare.

And this isn't only a vendor's marketing claim. The older academic evidence already pointed the same way: RouteLLM reported up to 85% cost reduction while keeping 95% of GPT-4's quality by routing each query to a strong or weak model; FrugalGPT, earlier still, showed cascades matching a top model at up to 98% lower cost. Those are useful proofs that the asymmetry has been known for years, not measurements of today's frontier. The more current picture is also more sobering: LLMRouterBench, published in January 2026, re-evaluated ten routing methods across more than 400,000 examples, 21 datasets, and 33 models. It confirmed that models are complementary, but found that several recent methods — including commercial routers — did not reliably beat a simple baseline. The savings are real; extracting them reliably is still unsolved.

The part that makes it engineering

If it were just "plan with the smart one," it would be a tip, not a discipline. What makes it engineering is that the obvious move is often wrong.

The same Cursor study has a detail I keep coming back to: the hybrid built on the stronger, newer planner (Fable 5) ended up more expensive than the one built on Opus 4.8 — even though the Fable planner's own bill was slightly lower. The catch was downstream: its plans led the cheap workers to consume several times more tokens, and workers are where the volume is (they accounted for at least 69% of tokens in every run, over 90% in most). A better planner produced a costlier run.

That's the whole lesson in one data point. The cost of a route is non-linear and it lives mostly in second-order effects — how a plan shapes execution, not the planner's own price tag. You cannot read it off a spec sheet. You have to measure the route end to end, which is exactly the kind of thing engineering disciplines exist to do.

Manual, or automatic

You can route by hand, the way I do, or hand the decision to a system.

The automatic tier is filling in fast. Cursor now ships a request-level Router that classifies each task and sends it to an appropriate model, with an Auto mode you can bias toward intelligence, balance, or cost — they report 30–50% cost reductions at frontier quality in early enterprise use. OpenRouter's auto endpoint routes per-prompt (powered by NotDiamond's model) with a cost-versus-quality dial. Martian and NotDiamond sell the same idea as a service. There's even a small benchmark literature trying to answer "how good is routing, really," and the honest read is promising but unsettled: some routers save real money, and some, under scrutiny, just default to the expensive model.

There is a deeper difficulty: you often cannot know how hard a task is until you start doing it. A prompt that looks routine can hide a nasty dependency three tool calls in; a frightening one can collapse after the first inspection. One-shot routing has to predict that latent difficulty before seeing the evidence the work itself will reveal. Recent research is starting to move the decision inside the execution loop. TwinRouterBench, from May 2026, routes at every call inside coding and research-agent trajectories and checks whether the whole task still succeeds. In one diagnostic, even Opus 4.6 acting as the router identified only 7 of 147 steps that execution later showed really needed the high tier, and all 40 SWE-bench trajectories failed. R2R, at NeurIPS 2025, goes finer still: it begins generating with a 1.5B DeepSeek-R1 distilled model and invokes the 32B model token by token when the reasoning path starts to diverge. The router is no longer a receptionist choosing a model at the door; it is a supervisor watching the work unfold and escalating when necessary.

Automatic routing will get better, but it inherits the hard part above: to route well it has to predict the second-order cost, not just classify difficulty. For now I trust a hand-drawn seam I've measured over a router I haven't.

When it's worth it

Here's the part I'd push back on the hype about: routing granularity has a cost of its own, and it isn't always worth paying.

For a one-off task, the entire calculus is often "use the best model and move on" — the time you'd spend tuning a route dwarfs the tokens you'd save. Granular routing earns its keep when one of three things is true: volume (you're running the task thousands of times), tight cost or latency budgets, or a clear plan/execute asymmetry like the long agent sessions above. Outside those, a router is premature optimization wearing an engineering costume. Knowing when not to route is as much a part of the discipline as the routing itself.

The unit keeps getting bigger

Step back and there's a through-line with where all of this is heading. Cursor framed their swarm as a kind of probabilistic compiler — planners decompose intent into a tree, cheaper models compile the leaves — and noted that "the unit of work becomes the specification." That rhymes with the software dissolving into the model: what you hand the system keeps getting larger and less literal, and the scarce skill becomes describing intent well and deciding what runs where.

There's one more honest wrinkle worth keeping. In that same study, the team wanted to use GPT-5.6 Sol as their frontier planner and couldn't — the new model was so sensitive to literal, emphatic wording that it spiraled out of control, and they fell back to GPT-5.5 rather than re-tune the prompts. So even at the frontier of routing, a ghost of prompt engineering is still in the room. The craft didn't die so much as move — from the words, to the wiring. Choosing the model is the new craft.

This is the second of two pieces on how "engineering" work in AI keeps relocating up the stack. The first is The Death of the Prompt Engineer.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

The Death of the Prompt Engineer

JaviMaligno — Sat, 25 Jul 2026 10:34:44 +0000

This week, researcher Dmitry Rybin published what appears to be a compact counterexample to a graph-theory conjecture that had been open for roughly thirty years. In the proposed construction, the fractional flow costs 58, while every indivisible flow — even allowing a capacity violation of up to 15 — costs at least 60. If the construction holds up under formal scrutiny, the Dinitz–Garg–Goemans conjecture is false. The candidate came out of a public chat session with GPT-5.6 Pro.

Here is the part that should stop you. The prompts that produced it read, verbatim, like this:

"Construct a counterexample to general (non-planar) case of Dinitz Garg Goemans conjecture. You should do a breakthrough and find a structured counterexample."

"please continue research and find a complete unconditional counterexample"

"Continue the search. Have a clear strategy obtained from deeper understanding of the problem structure."

"it's enough of partial results. let's finish with a complete unconditional counterexample"

That is the prompt engineering. Do a breakthrough. Just continue. Enough partial results, let's finish. The same week, a meme captured it perfectly:

Source: @Merkaloid quote-tweeting @mattshumer_ on X.

It's a joke. It's also the most honest description of the state of the art I've seen this year.

And the conjecture wasn't a one-off. The same week saw a small cluster of long-open problems fall to AI — some proved, some refuted, and several checked rather than merely asserted. Many were Graffiti conjectures, from a 1988 program that mechanically generated graph-theory conjectures: one agent refuted Conjecture 284 (open ~30 years) in about eight minutes, and Cognition's Devin refuted Conjecture 154 (open ~40 years) with the counterexample verified in Lean, while proving Conjectures 39 and 40 outright. The machine-checked part is what keeps this from being a season of merely plausible-looking guesses — results are being formally verified faster than anyone can read them.

Something else died quietly in the spectacle. It was prompt engineering.

What actually died

Not prompting. Prompting — asking a model for what you want — is more central than ever. What died is prompt engineering as an artifact craft: the belief that the value lived in the wording.

For a couple of years that belief had a whole economy around it. Templates. You are an expert… preambles. Few-shot scaffolds you tuned like hyperparameters. Delimiter tricks and "think step by step" incantations. Prompt marketplaces. Job posts for "Prompt Engineer" at six figures, as if the skill were a stable profession rather than a temporary artifact of weak models. In December, in an article about parallel AI agents, I included "prompt engineering for clear task specification" in a list of skills whose importance was shifting. I was describing a moving target and didn't know it.

The tell is the math example. If prompting were still an artifact craft, producing a serious candidate counterexample to a thirty-year-old conjecture would demand the most exquisitely engineered prompt anyone had ever written. Instead it took "do a breakthrough." The wording carried almost no information. All of it came from the model and the loop.

The everyday version: the ramble

You don't need an open conjecture to see it. Andrej Karpathy described the ordinary version in a post that Carlos Santana amplified with exactly the right framing — that Karpathy wasn't discovering anything, just naming out loud what practitioners already did in silence:

"One pattern I find useful for working with LLMs is a nice long ramble session… I like to lean back, switch to /voice and just ramble for like 10 minutes, total mess, anything goes, full stream of consciousness… I find that the LLMs are somehow very good at reconstructing long incoherent rambles and often their echo of your own tangle of thoughts comes out quite a bit cleaner than what you started with."

If you've done this, you know the feeling: you stop composing and start gesturing. You dump a mess and the model hands back something more coherent than what you put in. This very article started as a ten-minute voice ramble, typos and all — which is why it can end by describing itself.

That is the opposite of prompt engineering. You're not curating the input. You're trusting the system to reconstruct intent from noise.

Why it died: the harness ate the job

Two things killed the artifact.

The first is the harness. Modern coding and research agents don't sit and wait for a perfectly specified request. They search your environment, retrieve the relevant files, carry memory across turns, and run agentic loops that gather the context you used to hand-feed. The work that prompt engineering used to do — assembling exactly the right bits in exactly the right order — got absorbed into the tooling. You give a badly worded paragraph; the harness goes and finds the rest.

The second is raw capability. The models are now good enough to reconstruct intent from a bad input, and to keep going when the instruction is just "continue." In the conjecture case there is barely any prompt engineering left to point to — it's replaced by iteration: repeat "keep going, now finish" while the agent decomposes the problem itself. The intelligence moved from the sentence into the model and the loop around it.

What didn't die — and this is the important part

It would be a lie to end on "just ramble and the machine does everything." For the genuinely hard problems, effort didn't disappear; it changed shape.

Behind "do a breakthrough" there is still a person who knows a real result from a plausible one. When I wrote a research paper with AI, the discipline that mattered wasn't the wording — it was separating what could be demonstrated from what could only be measured, and refusing to blur the two. In one planning session a model garbled a note about topology, claiming a ring in the plane needed a higher dimension. I caught it because it's my field, not because I'd written a clever prompt. The model can do the search; you still have to know when the answer is real.

And for the hardest work you increasingly reach not for a better prompt but for a structure — a skill, a harness — that forces the model to try several strategies, decompose the problem into smaller parts or lemmas, run experiments, write tests, review its own output, hold a standard of rigor. That scaffolding is the successor to the prompt template. It's not a magic sentence; it's a process. The candidate counterexample didn't emerge from one lucky message either — it came from a loop that kept insisting on rigor until a partial result became a complete one.

So the skill didn't vanish. It moved — from the words to the framing, the steering, and the verification. Karpathy's term for the goal is the "mind meld": getting the model aligned with what you actually mean, so you have to correct it less from that point on. That is the craft now. The trend line is unmistakable: every generation does more with less, and the part you can leave to the model keeps growing.

The effort didn't disappear — it moved up

If prompt engineering is dying, it's not because engineering left. It's because it climbed the stack. The leverage used to sit at the level of the sentence. Now it sits at the level of the system: which model you hand a task to, whether you plan with a strong one and execute with a cheap one, what structure you wrap around the loop.

That's a whole discipline of its own — the one I'd argue is replacing prompt engineering rather than merely outliving it — and it deserves its own piece. For now the short version is this: the best "prompt" in 2026 is a ten-minute ramble to a system that already knows your codebase, and the skill worth having is knowing what to build around it.

The prompt was never the point. Getting the machine to understand you was.

This is the first of two pieces on how "engineering" work in AI keeps relocating. The second is about **routing engineering* — choosing which model and how much reasoning to spend on each task. See also Software Is Dissolving Into the Model.*

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Where's the Ball? Testing Whether a VLM Has a Spectator's Intuition

JaviMaligno — Fri, 24 Jul 2026 11:02:09 +0000

Watch a football match from the top row of the stadium, where the ball is a barely-visible speck, and something curious happens: you usually know where it is anyway. The players tell you. Bodies lean, a cluster forms, everyone starts drifting the same way — and your eye lands on the ball a beat before you actually see it. It's not tracking. It's a kind of social inference: you know the game, so the people tell you about the thing.

I wanted to know whether a generalist vision-language model has that same intuition. Not a specialist tracker trained on ball trajectories — those exist and they're good (Maksai et al., Kim et al. 2023, TranSPORTmer all infer the ball from player motion). I mean a model that "knows the game" the way a person in the stands does, and has never been trained for this task specifically. Can it look at a frame with the ball erased and point to where it must be?

The honest answer turned out to be yes, a little, and only the biggest models — but getting to an answer I actually trust was most of the work, and it's the part worth writing about.

The setup

I used SoccerNet-Tracking: broadcast clips with per-frame bounding boxes for every player and the ball. For each frame I removed the ball with LaMa inpainting — deep inpainting that reconstructs a coherent patch of grass, line, or shirt where the ball was, so the model can't cheat by spotting an editing smudge. (Classical inpainting left tell-tale blurs on pitch lines; a leak-control step with a VLM confirmed the deep version leaves nothing to latch onto.) Then I asked several models for the ball's coordinates and scored the distance to the truth.

The models: GPT-5.4, Claude Opus 4.8, Claude Sonnet 4.6 (all via API), and Qwen2.5-VL-7B as an open-source reference, run on a GPU. Against them, two dumb baselines: "the ball is at the center of the frame" and "the ball is at the centroid of the players."

The trap I walked straight into

My first run looked great. GPT beat the center baseline; on the hardest cases — balls far from the middle of the frame — it seemed to win 64% of the time. Tidy story: generalist AI has spectator intuition. I could have written that article.

Two problems, both mine.

First, a confound. Broadcast cameras follow the ball — they keep it near the center of the shot. So "just guess the center" isn't a dumb baseline at all; it's a strong one, and any model that leans central looks smart for the wrong reason. Second, and worse: my sample was tiny. That 64% came from 14 items. When I put a bootstrap confidence interval around it, the interval ran from 36% to 86% — comfortably including 50%, i.e. chance.

So I rebuilt the test to defuse both issues: I balanced the dataset by the ball's distance from the center (so "guess the center" is useless by construction), and I scaled up. At a larger sample the same off-center win-rate settled at 55%, and its interval still touched 50%. The exciting first result hadn't survived contact with statistics. It was a mirage made of a followed camera and fourteen data points.

I mention this not to be self-deprecating but because it's the whole point: a confounded baseline plus a small sample will hand you a satisfying, publishable, wrong conclusion, and it will feel true. The only defense is boring — de-bias, scale, bootstrap, and try to break your own finding before you believe it.

What's actually true

Once the sample was honest, I stopped leaning on error-on-a-hard-subset (still underpowered) and used a better-behaved measure: does the model's prediction correlate with where the ball actually is, across the whole range of positions? That question the data can answer.

Claude Opus 4.8 has the clearest signal — its guesses track the true ball position on both axes, and the confidence intervals stay clear of zero.
GPT-5.4 shows partial signal (solid horizontally, noisier vertically).
Qwen2.5-VL-7B (open) and Claude Sonnet 4.6 are, statistically, flat — their intervals straddle zero. They aren't pointing at the center out of laziness (I checked — their predictions are spread out); they're just not tracking the ball.

So the spectator intuition is real, but faint, and it's a frontier-model trait: it shows up in the big closed models and not in the smaller open one or the mid-tier Claude. And a sober caveat that survived all the scaling: from a single frame, no model reliably beats the center bias on genuinely off-center balls. The intuition is there in aggregate; it isn't strong enough to nail the hard, wide cases from one still image.

One more twist, which I liked. I gave every model an "informed" prompt — naming the sport and spelling out what a spectator uses (players orient toward the ball, converge on it, the carrier leads a cluster). It helped GPT a lot (its correlation jumped), did nothing for Sonnet, and barely moved Opus. Read that carefully: Sonnet's problem was never that it didn't know football. Telling it the rules changed nothing, because its bottleneck is seeing, not knowing. For GPT, which could already reason about the scene, the framing unlocked headroom — the prompt mattered about as much as the model tier.

The video that wasn't about video

Here's where I was sure I knew the answer in advance. A single frame is ambiguous; a spectator's real trick is watching the play develop. So I gave the models short sequences — four frames spanning a few seconds, the ball erased from all of them — expecting the motion to unlock the hard cases.

It did seem to help GPT. But before believing it, I ran the control that matters: I shuffled the frames into a random order. If a model reads motion, scrambling time should destroy the benefit.

Shuffling changed nothing. The order of the frames was the single cleanest null in the whole experiment. No model is integrating the trajectory of the play — they aren't reasoning about motion at all. What actually changed with multiple frames was something duller and stranger: extra views of the scene, in any order. And the two frontier models handle that in opposite ways. GPT aggregates — more looks at the moment, even scrambled, sharpen its guess. Opus gets diluted — the extra frames pull it away from a target-frame read it was already better at. The best single-frame model was the worst at using more frames.

That's a more interesting finding than "video helps," and I'd have missed it if I'd trusted the first, flattering version.

Takeaways

Generalist VLMs do have a faint spectator's intuition — their guesses correlate with the hidden ball — but it's a frontier-model trait (Opus clearest, GPT partial) and it's weak: a single frame doesn't crack the genuinely hard cases.
They don't reason about motion. What looks like temporal understanding is really multi-view aggregation, and models differ in whether extra views help or hurt.
Knowledge isn't the bottleneck; seeing is. Explaining the game helped the model that could already reason visually and did nothing for the one that couldn't.
The methodology was the hard part. A followed camera and a small sample produced a clean, wrong story that felt right. De-biasing, scaling, bootstrapping, and adversarially auditing my own results changed the conclusion twice.

There's a Level 2 waiting — other sports, whether the skill transfers, and the geometry underneath it all — but that's for later. For now the honest headline is smaller and better than the one I almost wrote: the model in the stands can feel where the ball is, a little, if it's big enough; it just can't watch the play to find it.

Code, data pipeline, and the full audit (bootstrap CIs and all) are in the wheres-the-ball repo. Built on SoccerNet-Tracking; ball removed with LaMa via IOPaint. Prior specialist work worth reading: Maksai et al., Kim et al. 2023, TranSPORTmer.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.