DEV Community: Ernesto Herrera Salinas

Binding members to a persona without magic

Ernesto Herrera Salinas — Sun, 12 Jul 2026 02:34:17 +0000

Automatic coherence sounds like magic, and magic in a library is a liability.
The entire job in this part of the design was making it trustworthy: it has to fire when it should, stay silent when it should not, and be explainable when you ask. Several decisions, one theme.

Binding: automatic, with an explicit escape

First, how does a member get attached to its object's persona? I shipped two paths on purpose.

The primary path is automatic. Semantic inference, which already recognizes FirstName and Email, binds those members to the object's persona, so Lie<Customer>.Generate() comes out coherent with zero configuration. That is the seed/demo win, and it has to be the default because it is the only thing that reaches the headline one-liner.

The escape path is explicit: a d.Person accessor you can read inside a With or Derive rule when you want precision. It is a complement, not a replacement, and an explicit rule always overrides the automatic binding.

Where it lives: a stage, not a patch

Coherence is not bolted onto the semantic matcher. It is its own resolver stage on the ordered pipeline I pre-paid for earlier in the series, driven by an entity descriptor that maps roles to facets. That choice is what allows the same machinery to later serve Address and Company by adding a descriptor instead of rewriting the matcher. It also gives Explain() a real, first-class thing to report.

Detection: borrow the truth you already have

Now the part that decides whether the whole feature is trustworthy. How does the engine know FirstName is a person facet and Name on a Product is not?

The tempting move is to give the persona its own table of names it owns. I did not do that, because then two tables can drift. Instead, the persona piggybacks the existing semantic candidate table: a member binds to a facet when the semantic candidate it already wins is one the persona claims. One source of truth, no drift. And the existing model-hint disambiguation comes along for free:
Product.Name already resolves to a product-name candidate, never a person one, so coherence inherits that false-positive avoidance without writing a line of new
guarding.

Activation: gate on evidence

The last question is: when does an object get a persona at all? Always-on would be reckless. So activation reuses the inference mode the library already exposes (Conservative, Balanced, Aggressive) as an evidence dial, and counts corroborating role members:

Conservative: needs a name anchor (FirstName/LastName/FullName)
Balanced:     a name anchor, or two correlated members
Aggressive:   one matched member is enough
Disabled:     never

It is a build-time pre-pass, not a per-member check, and it self-limits the two-people collapse from the last post: prefixed, ambiguous members score low, so the mistaken merge only happens under Aggressive, where you asked for reach.

(Two smaller decisions ride along here: the explicit d.Person is a fixed snapshot of properties, not a generator, so reads inside a rule line up with the
auto-bound members; and reading it on an object with no activated persona materializes one lazily. That lazy draw is the one order-dependent corner in an otherwise eager design, and it is documented as such.)

Takeaway

Automatic behavior earns trust by reusing the system's existing truth and gating on evidence, not by inventing a parallel guesser that can drift or misfire. Magic
you can explain is just a feature. Magic you cannot is a support ticket.

What's next

I have been hand-waving one contractual detail: when exactly does a persona draw its random values from the stream? In a deterministic library, that is not a detail at all. Next post: the PRNG draw-tape, eager versus lazy.

Fake data that is not obviously fake

Ernesto Herrera Salinas — Thu, 09 Jul 2026 00:38:35 +0000

Back in the audit, I flagged a tension I had created for myself: I called seed and
demo data a co-equal job, then pointed the whole roadmap at composition, which
barely serves it. This post is where I start paying that off. It begins with an
ugly little object.

Here is what zero-config inference produces today, for a customer:

{
  FirstName: "Maria",
  LastName:  "Garcia",
  FullName:  "John Smith",          // unrelated to First/Last
  Email:     "k7zp@example.net",    // unrelated to the name
  Username:  "blue-tiger-441"       // unrelated again
}

For a unit test, nobody cares. Any plausible string is fine. But for seed data in
a demo database, this looks broken. A human reads it in two seconds and knows it
is fake. And this is precisely the job composition cannot help with, because
composition is about structure (which definition builds a nested object), and this
is about correlation between flat scalar fields. Different problem, different
machinery.

The idea: a persona

The fix is an idea Bogus already has, called Person: a coherent bundle generated
once, where the fields agree because they all come from the same underlying
identity. Munchausen's version has to be deterministic and inferred rather than
hand-wired, but the core is the same. There is a hidden Person for the object,
and FirstName, Email, and FullName are facets of it, not independent
guesses.

The load-bearing fork: what is one identity?

The first real decision is the one that sets the mental model for everything after
it: in an object graph, which members share a single persona?

class Customer { string FirstName; string LastName; string Email; SalesRep Rep; }
class SalesRep { string Name; string Email; }

A: object-scoped. Persona boundary equals object boundary.
   Customer.FirstName/LastName/Email -> Customer's persona (one identity)
   Rep.Name/Rep.Email                -> Rep's own persona  (a different identity)

B: role-scoped. Group by a member-name prefix, so two people flat on one object
   stay distinct (CustomerName vs SalesRepName).

I chose object-scoped. It is predictable, it needs no name-parsing heuristics, and
the boundary is something the user can already see in their own type. A nested
object simply gets its own identity, which is almost always what you want.

Object-scoping has a known sharp edge. If someone flattens two people onto one
type, Order { CustomerName, SalesRepName }, object-scoping makes them the same
person. That is wrong, and I did not pretend otherwise. I deferred role grouping
as a later refinement and made the collapse diagnosable through Explain(), which
will happily report "four members bound to one Person" so you can see it.

The reason I was comfortable deferring it: object-scoping already beats AutoBogus
for the common case, a single identity per object, which is most of the win. The
flat-two-people shape is a denormalized DTO that no tool handles automatically
today. So the principled default ships now, and the differentiator can come later
without repainting the model.

Takeaway

Coherence is not "better random values." It is a shared source of truth that
several fields read from. And the first decision in any system like that is the
boundary question: what counts as one identity? Pick the boundary your users can
already see, and be honest about the case it gets wrong.

What's next

A persona is only useful if members actually attach to it, and the attaching has
to be trustworthy rather than magic. Next post: binding members to a persona
without magic, including how the engine decides a field is a person at all.

Paying for the future on purpose

Ernesto Herrera Salinas — Tue, 07 Jul 2026 19:13:24 +0000

There is a category of design decision where the cheap option for the version you are shipping quietly mortgages the version after it. With a solo project and no deadline, I kept choosing to pay now. Here are the two clearest ases, and the principle that connected them.

Pre-pay one: structure over shortcuts

Composition adds rules to members. A collection member, for instance, might use a child definition for its elements, hold between two and five of them, and be null
ten percent of the time. v1.1 only needs the first of those. v1.2 wants all three stacked on one member. So how should a member hold its rules?

A: one rule per member, conflicts unless whitelisted.
   Stacking N features needs an O(N^2) compatibility matrix.

B: a member is a record of independent facets (element, size, null-probability).
   Stacking is free; only two rules of the SAME facet conflict.

Option A is less code for v1.1 today. Option B introduces a facet record now, for a feature that only uses one facet, which feels like over-building. I took B
anyway. The same logic applied one layer up: resolution became an ordered pipeline of stages rather than a stack of hardcoded if branches, so v1.2 can insert a stage by position instead of rewriting the branch.

The reason is the brief, not taste. Feature growth is an explicit goal, and there is no deadline. The design doc itself had concluded that the cheap options are the
only thing that would force v1.2 to reopen composition. When you can see the roadmap that clearly and you are not racing a clock, paying the structural cost
once is strictly better than paying interest on it every release. (This pipeline, it turns out, becomes the host for the entire coherence feature later in the series. The pre-pay compounded in my favor faster than I expected.)

Pre-pay two: a default that fails loudly

The second case is about composition boundaries. If you compose a reusable customer definition into an order, and that customer is silent about how money is generated, should a money rule you set on the order reach inside the customer?

Two pure answers, and the choice is really about how each one fails.

hermetic: the rule seals at the boundary. The customer generates its own way.
ambient:  the rule flows in. The customer's money becomes yours.

Ambient is convenient right up until it is not. The failure mode is that a distant ancestor silently mutates a fixture you wrote and tested, and you find out by
auditing the whole chain. Hermetic fails the other way: your rule under-reaches, you see the wrong value at your own call site, and the fix is local. Local and diagnosable beats non-local and silent. So the default is hermetic: a composed definition is a sealed contract.

But hermetic under-reaches on purpose, and sometimes you really do mean "every money value in this whole graph is in euros, no exceptions." For that I reserved
an escape valve, UseDeep, that a consumer writes explicitly and that always wins, penetrating any boundary. The fork there was whether reach should be the
consumer's call (UseDeep) or the author's (Sealed/Open on the definition). I took a hybrid: sealed by default so authors get safe contracts for free, plus a consumer UseDeep that overrides anything. Two concepts, but each is in the hands of the person who knows the intent. UseDeep is greppable and visible at the call
site, which keeps even the ambient-style power loud rather than silent. Its shape is reserved now and ships in a later point release, so the default can never paint
it out.

Takeaway

When you can see the feature roadmap and you are not under deadline, pay your structural debt down before it compounds. The brief is what tells you whether you
can afford to. And when you pick a default, choose the one whose failure mode is local and visible, not the one whose convenience hides the bug several layers up.

What's next

That is the composition thread settled. Now the threads switch. The next question is not how objects compose, it is whether the data inside them looks real at all.

Next post: fake data that is not obviously fake.

Reconciling a public API that drifted

Ernesto Herrera Salinas — Sun, 05 Jul 2026 21:11:47 +0000

The audit found that my two binding documents disagreed about the v1.1 composition API. Closing that gap was not paperwork. A public method is a promise you can almost never take back, so reconciling the surface meant making real choices and living with them.

Some quick context on what composition is. It lets a definition control how the nested objects inside a generated graph are produced: "every Address in this order graph uses this address definition," or "this specific member uses that child definition." The surface in flux covered type-scoped, member-scoped, and element-scoped binding, plus substituting a concrete type for an abstract member.

Three forks had to be settled.

Fork one: the name

The element-scoped method was UseEach in the binding doc and UseElements in the design doc. Tiny, but public, so it gets a real decision. I kept UseEach. It was the name in the top-authority document already, and it reads cleanly next to
its siblings: Use for a type, UseEach for the elements of a collection. Not every fork is deep. Some are just "pick one on purpose and stop relitigating it."

Fork two: the shape

This one mattered. The binding doc had a single-generic member binding plus a separate UseAs method for substituting a concrete type for an abstraction. The design doc had folded both into one two-generic method.

// before: two verbs
Use<TChild>(expr, def);
UseAs<TAbstraction, TImplementation>(def);
UseAs<TAbstraction, TImplementation>(expr, def);

// after: one verb, abstract-capable
Use<TMember, TConcrete>(expr, def) where TConcrete : TMember;

I took the folded shape. The generic constraint carries abstract substitution for free: for a concrete member, TMember and TConcrete infer to the same type, so existing calls are unchanged, and for an interface-typed member the second generic
supplies the concrete definition. One verb instead of three, and the type system does the work that a separate method used to. UseAs disappeared.

Fork three: how much to ship

The folded Use handles abstract members one at a time. The binding doc also promised a type-level version: "every IParty in this graph is a Person," one registration, graph-wide. I deferred it.

The reasons were concrete. The compiler cannot infer both type arguments for the type-level form, so every call would spell them out. It needs a conflict rule for two concretes registered against one abstraction. And it quietly nudges users toward expecting random polymorphic selection, which is a different feature
entirely. It is cheap to add later and expensive to walk back, so it waits. In v1.1, abstract substitution is member-scoped and element-scoped only.

I also added one new diagnostic, LIE012, for an unsupported composition target (binding to a collection shape the library cannot materialize), since the existing diagnostic registry had no fitting code.

Writing the why into the contract

The part I am most glad I did: I recorded all of this as a "Revision 2.2 amendment" inside API_DESIGN.md itself. Not in a side note, not in a commit message, in the binding document, next to the surface it changed. The amendment says what folded into what, what was deferred and why, and that the type-level form is an additive follow-up.

The contract and the design doc finally describe the same API, and the next person to read either one will find the reasoning attached to the decision instead of having to reconstruct it, or worse, rediscover the conflict I just closed.

Takeaway

A public API is a promise, and drift between your contract and your design is a quiet way to break it. Reconcile deliberately, prefer the change that is additive and lets the type system carry the load, defer the expensive-to-reverse pieces, and write the why into the contract so it does not drift again.

What's next

With the surface honest, the deeper composition decisions were waiting, the ones that were not about taste at all. They were decided by the brief: solo, no deadline, quality over speed. Next post, paying for the future on purpose.

Auditing my own frozen design

Ernesto Herrera Salinas — Sat, 04 Jul 2026 01:33:00 +0000

With the rigor lane chosen, the disciplined next move was not to design anything.

It was to reread my own v1.0 design documents, the ones I had stamped "binding," as if a slightly hostile consultant were reading them for the first time. I am glad I did, because the audit found a contradiction sitting inside my own contracts.

The contradiction

The trouble is that API_DESIGN.md had sketched the v1.1 composition surface months ago, and the active composition design doc had kept evolving past it, into a genuinely better shape. So two documents I had both called authoritative now described different public APIs for the same feature.

API_DESIGN.md (binding):   UseEach<TChild, TCollection>(...) + a separate UseAs(...)
Composition design doc:    UseEach<TElement, TConcrete>(...), abstract folded into Use, no UseAs

This is the kind of bug that does not crash anything and does not show up in a test. It just sits there until a contributor follows the authority order, builds the surface the top document describes, and ships the wrong API. Documentation that drifts is worse than no documentation, because it still carries the badge of truth.

That was finding one, and it became the next post: reconcile the surface, deliberately pick the winner, and write down why.

Two more things the audit surfaced

While I was reading hostile, two strategic gaps emerged that had nothing to do with the surface conflict.

The first is a tension between my own answers. I had said the two jobs, test fixtures and seed/demo data, are co-equal. But the planned v1.1 anchor, composition, almost entirely serves the test-fixture and rigor job. It does very little for the seed/demo job. The thing that makes seed/demo data compelling without breadth is in-locale realism and coherence: a customer whose name, email,
and date of birth actually agree. My own positioning note listed exactly those as current gaps. So I had declared a job co-equal, and then pointed the whole roadmap elsewhere. That gap is the reason the back half of this series exists.

The second is a ceiling I had built on purpose and then half forgotten. The zero-config path, Lie<Customer>.Generate(), is permanently non-configurable by contract. That is what makes the process-wide plan cache safe. But every extension mechanism- datasets, locales, data packs, providers- is definition-scoped. The consequence: a third-party data package can only enrich the defined path, never the headline one-liner. And the headline one-liner is the adoption hook. So the extensibility story, the thing that is supposed to scale the ecosystem, structurally cannot touch the feature people would actually adopt me for. That is not necessarily wrong, but it is a thing to decide on purpose, not to discover later.

The lesson

None of these three were design mistakes exactly. The surface conflict was drift.

The coherence gap was a roadmap that had quietly diverged from the stated goals.

The extensibility ceiling was a correct tradeoff I needed to hold consciously. But all three were invisible until I read my own contracts adversarially.

The takeaway I keep coming back to: a "binding" document is only binding if it is periodically tested against the design that is actually happening. An authority that is never audited is just a stale opinion wearing a badge. Reread your own contracts as a stranger before you build on them, because the most expensive bugs
are the ones your documents agree to.

What's next

The first finding was a live contradiction between two binding documents, and it was the cheapest to close. Next post: reconciling a public API that drifted, which meant making real, permanent surface choices, and recording the why inside the contract itself.

Why I am not cloning Bogus

Ernesto Herrera Salinas — Fri, 03 Jul 2026 01:06:00 +0000

v1.0 of Munchausen shipped. Before I designed a single v1.1 feature, I made myself answer the most uncomfortable question a .NET mock-data library can face: how far am I from Bogus?

Bogus is the giant in this space. It has been around for a decade; it has fifty locales; it has datasets for finance, company, hacker-speak, and music; and it has the community to match. If "be like Bogus" is the goal, the honest answer to my question is "years away, and I am one person." That is a depressing place to start a roadmap, so I sat with it until I realized I was asking the wrong
question.

Two axes, not one

Bogus and Munchausen are not the same kind of tool measured at different sizes.

They sit on different axes.

One axis is breadth. Bogus is rules-first: you write a rule per property, and in exchange you get enormous content, locales, and ecosystem. That is years of curation and a community, not a clever architecture.

The other axis is rigor. Munchausen is inference-first. You hand it a type, and it generates an owned deterministic generator, leaning on semantic inference, eager validation, an Explain() report, and a compiled performance model. The same task looks like this:

// Bogus: rules-first
var faker = new Faker<Customer>("en")
    .RuleFor(c => c.FirstName, f => f.Name.FirstName())
    .RuleFor(c => c.LastName,  f => f.Name.LastName())
    .RuleFor(c => c.Email,     (f, c) => f.Internet.Email(c.FirstName, c.LastName))
    .UseSeed(42);
var customers = faker.Generate(100);

// Munchausen: inference-first
var customers = Lie<Customer>.Generate(100);

Once I drew the two axes, the real peers came into focus. The library that generates an object graph from nothing is not Bogus, it is AutoFixture and AutoBogus. That is the company I am actually keeping.

The fork

So the strategic decision was a fork, and it is genuinely two different products.

Option A: chase Bogus on its home turf. Invest in locales, dataset breadth, coherent personas, ecosystem. This is the road to adoption against Bogus, and it is the right road if breadth is the bet. It is also content and community work, mostly, and it would take a solo developer a very long time to even look credible.

Option B: own the rigor lane. Go all-in on being the best deterministic, explainable, zero-config generator, a category Bogus does not really compete in.
The competitors to beat here are AutoFixture and AutoBogus, and on this axis Munchausen already leads or ties.

The choice, and the why

I chose B. Own the rigor lane, and deliberately cede breadth to Bogus.

The reasoning is about where a solo project with no deadline can actually win.
Breadth is content and ecosystem: you beat it with a team and ten years.
Rigor is architecture: determinism, explainability, a compiled plan, zero-config inference. That is the kind of advantage one person can build and defend, because it is design, not catalog size.

The part that took me the longest to accept is that "adoption" and "cede breadth" are not in conflict, as long as "adoption" means winning a category rather than winning Bogus's category. I am not trying to be the library you reach for when you need realistic German addresses. I am trying to be the default for people who care about their generated data being reproducible, inspectable, and configured by inference rather than by a hundred hand-written rules. Bogus does not compete there.

Two more things fell out of this, and I wrote them down so every later decision could be checked against them.

First, Munchausen has to serve two roles as co-equals: test fixtures (where any plausible value is fine) and seed or demo data (where the data must look real and internally consistent). That second job is the one that usually needs breadth, so ceding breadth puts pressure on a different lever entirely, making the in-locale
data genuinely coherent. I flagged that tension on purpose. It becomes the whole back half of this series.

Second, "scalable" had to be pinned down, because it pulls in several directions at once. For this project it means four things: extensibility, performance, contributor approachability, and feature growth without breaking the locked public surface.

That is the brief, and it is the rubric I now hold every decision against:
deterministic, explainable, zero-config; displace AutoFixture and AutoBogus, not Bogus; both jobs co-equal; scale four ways; built solo, no deadline, quality over speed.

Takeaway

Do not benchmark yourself against the market leader's strengths. Find the axis where your advantage is structural rather than accumulated, and commit to being the best there. Half the hard decisions get easier once you can say out loud what you are choosing not to be.

What's next

With the lane chosen, the obvious next move was to read my own frozen v1.0 design against it with fresh, slightly hostile eyes. That audit found a contradiction hiding inside my own binding documents, which is where the next post starts.

Designing Munchausen v1.1 in public

Ernesto Herrera Salinas — Wed, 01 Jul 2026 00:27:02 +0000

I am picking Munchausen back up.

This is an old project of mine: a .NET mock-data library with the modest ambition of generating useful fake data from your types without asking you to write a rule for every property. I already took it through a v1.0 foundation series, starting with the traditional first step of any responsible software project: deleting code
and breaking the build on purpose.

Now I want to revive it, not because anyone is waiting for a quarterly roadmap, but because I still like the problem. I like the tiny design arguments. I like deciding where a public API should bend and where it should refuse. I like the part of software where nothing exists yet except a few constraints and the dangerous belief
that this time the abstractions will behave.

So this series is for me, and it is for fun.

Also, if the Baron is going to do anything in public, I may as well design in public too.

What v1.1 is about

Munchausen v1.0 is the foundation: deterministic generation, semantic inference, Build() validation, Generate(), datasets, and an Explain() surface that makes the automatic parts inspectable.

v1.1 asks what the library has to become if it wants to be more than a convenient object generator. The work has two main threads.

The first is composition: how one definition controls the nested objects inside a generated graph. If an Order has a Customer, and that customer has addresses, collections, abstractions, and reusable child definitions, the library needs a surface that can express those relationships without becoming a maze of rules.

The second is coherence: making generated data agree with itself. A customer named Maria should not have a full name of John Smith and an unrelated email address.
An address should not combine a city, state, and postal code from three different places. For tests, any plausible value is often enough. For seed data, demos, and screenshots, incoherent data looks broken.

Those two threads sound separate. They are not entirely separate. Composition changes object boundaries, and coherence depends on object boundaries. That is where the interesting design work lives.

Why design gets its own series

These posts are design records, not release notes. Most of the work here is deciding what should exist before writing the code that ships it.

Each article will follow the same shape: the problem, the options, the choice, and the reason. I want the rejected options in the record too, because a design without the alternatives is hard to trust later. If I decide not to build something yet, I will say why. If I reserve a public surface for later, I will write that down too.

The goal is not to make the decisions look inevitable. The goal is to make them reviewable.

That matters more on a hobby project than it sounds. When nobody is forcing a deadline, the easy failure mode is not shipping the wrong thing under pressure. It is drifting, forgetting why a decision was made, then rediscovering the same fork three weekends later with worse notes. Writing the design down is how I keep the project honest with future me, who is technically the maintainer and also the most
likely person to ask, "why did I do this?"

The rules of the game

Munchausen is not going to beat Bogus by becoming a smaller Bogus. Bogus owns breadth: locales, datasets, categories, and years of ecosystem. Munchausen's lane is rigor: deterministic output, explainable inference, zero-config generation, a compiled plan, and failures that are local and diagnosable.

That gives me the rubric for the whole series:

deterministic, explainable, zero-config
better than AutoFixture and AutoBogus on inference-first generation
not a Bogus clone
test fixtures and seed/demo data as co-equal jobs
solo project, no deadline, quality over speed

Every v1.1 decision has to survive that rubric. If a feature makes the library broader but less explainable, that is suspect. If a shortcut helps one release but forces the next one to reopen the same surface, that is suspect too.

I want a design that can be reviewed cold. Someone should be able to read this series later and understand not just what v1.1 intends to do, but why the shape is the way it is. That someone might be another engineer. More likely, it will be me, after forgetting the entire argument and acting surprised by my own repository.

That means writing down the uncomfortable parts: where the public API drifted, why some convenient features are deferred, where automatic behavior becomes too magic, and what remains unresolved. The reasoning is part of the product. If the design is
going to be rigorous, the record of the design has to be rigorous too.

The first decision is strategic: before I design any v1.1 feature, I need to decide what kind of library Munchausen is trying to be, and what kind it is deliberately not trying to be.

The Iron Man Protocol: Turning Agent Mistakes Into Durable Engineering Feedback

Ernesto Herrera Salinas — Fri, 26 Jun 2026 22:00:00 +0000

How a Team Can Make a Coding Agent Improve Without Retraining It

Tony Stark does not get stronger by force of will. He gets stronger because he is an engineer who treats every failed suit as evidence: he records what broke, changes the armor, adds a safeguard, and tests the next version under harder conditions. The suit does not learn. Stark learns, and the suit carries the lesson forward.

That distinction is the whole point of this article, so it is worth being precise about who plays which role. A coding agent like Codex is the suit. Your team is Stark. Codex does not update its model weights when it misunderstands a repository convention, skips a test, or patches the wrong file. That interaction does not retrain the underlying model. But the system around the agent can be re-engineered after failures so the next run carries the lesson:

That loop is the Iron Man Protocol. The rest of this article is how to run it without it collapsing under its own weight.

Who This Is For

This article is for teams using a coding agent for repository work, code review, CI fixes, migrations, generated-code workflows, and other repeated engineering tasks, especially anyone who has watched the same agent mistake happen twice and wants something stronger than "please remember next time."

The named surfaces below (AGENTS.md, skills, hooks, memories, evals) are spelled the way Codex spells them. Many coding-agent platforms expose comparable primitives, though names, locations, and enforcement behavior differ. Treat "Codex" throughout as the concrete example, not as the only possible implementation.

The One Idea That Matters

Make the next failure harder to repeat by changing the environment, not by exhorting the model.

This rests on a single research finding, stated once: language models are unreliable at correcting their own reasoning when the only signal is their own reflection. They improve far more dependably when given an external, executable signal: a failing test, a trace, a CI log, a reviewer comment, a concrete rule. [4][5] The nuance matters: self-correction is not worthless; it works when the model can check its work against something real. [1] So the engineering goal is not "tell the agent to reflect harder." It is "put a real signal in front of the agent."

That is the difference between a weak loop and a strong one:

Weak:   Think harder and avoid that mistake again.

Strong: Reproduce the failing case, find the root cause, patch the smallest
        responsible code path, add a regression test, update the repository
        rule if this convention recurs, and rerun the relevant suite.

The strong loop hands the agent evidence. It turns a mistake into a constraint. Everything below is machinery for producing those constraints, and for removing them once they stop paying rent.

The Research, Briefly

A few results carry this argument:

Self-correction has limits. Models often fail to fix their own reasoning without a reliable external signal. [4][5] This is the reason the protocol refuses to depend on the model "noticing" its mistake.
Verbal feedback can persist across attempts. Reflexion showed agents improving without weight updates by storing linguistic feedback from prior tries. [2] That is the mechanism the protocol industrializes.
A growing, reusable skill library compounds. Voyager paired execution feedback and self-verification with an ever-expanding skill set. [8] Tests, skills, hooks, evals, and AGENTS.md are that skill library for a coding agent.

Software-engineering benchmarks reinforce the practical shape: coding agents need repository context, usable tools, good localization, and validation loops. [9][10] Agentless adds a useful caution: simpler structured pipelines for localization, repair, and patch validation can outperform more autonomous loops. [11] The lesson is not "maximize autonomy." It is "build reliable feedback channels."

Adjacent research on iterative refinement, tool-using agents, and search-style reasoning is relevant to the broader agent landscape, but it is not the core proof for this protocol. [3][6][7] Model-level approaches such as process supervision, reasoning bootstrapping, and reinforcement learning for self-correction may improve future models, but most teams cannot touch training-time systems. [18][19][20] For day-to-day engineering teams, the leverage stays in the environment.

What "Learning" Actually Means Here

Three kinds of learning, only one of which you control:

Kind	Where it lives	Lifespan	Your leverage
Training-time	The model weights	Permanent	Almost none
In-context	The current thread	Until the thread ends	Real but temporary
System-level	The repo, CI, instructions, skills	Until you retire it	High: this is the protocol

If the agent fixes a bug after seeing a failing test, it improved in the session. If the repository now carries a regression test that blocks that bug from returning, the system improved. The Iron Man Protocol is entirely about that third row.

The Protocol

Seven steps. The seventh is the one most "lessons learned" systems forget, and the reason most of them rot.

1. Capture the failure

Do not let the failure live only in a chat transcript. Record it while the evidence is fresh:

## Mistake Record
- Date / Task:
- What the agent did / What was wrong:
- How it was detected:
- Root cause:
- Correct behavior:
- Durable artifact needed (test / rule / skill / hook / memory / eval):

This is compression, not bureaucracy: a confusing interaction becomes a reusable lesson.

2. Diagnose along two axes

Earlier versions of this protocol used a flat list of seven failure "types." That list leaks: real failures rarely sit in one bucket. The worked example below is a context failure and a verification failure at once. Diagnose along two orthogonal axes instead:

Context axis: did the agent lack something it needed to know? A convention, dependency, product rule, hidden workflow step?
Verification axis: did the agent fail to check what it changed? Wrong test, skipped lint/typecheck, no reproduction, missed regression?

Most mistakes score on both. The axes matter because they point to different artifacts: context gaps become knowledge (an AGENTS.md rule, a skill); verification gaps become enforcement (a test, a hook, a CI check). The familiar labels, such as bad localization, over-broad change, tool misuse, review miss, and memory gap, are common positions on these two axes, not separate categories.

3. Choose the durable artifact

The rule: never store an important lesson only as prose in a conversation. Match the artifact to the failure:

Failure	Durable upgrade	Use when	Do not use when
Recurring repo convention	`AGENTS.md` rule	It should govern future work in the repo/subtree.	It is temporary, personal, or better enforced by a test.
Skipped validation	Hook / CI check / test command	The same check keeps getting missed.	It is noisy, slow, or not yet well understood.
Behavioral bug	Regression test	The issue is user-visible and reproducible.	It is not yet deterministic enough to encode.
Repeated multi-step task	Skill	The task has repeatable steps, references, or scripts.	One sentence in `AGENTS.md` is enough.
Stable local preference	Memory	It helps future local sessions only.	It must bind a team or repo; use checked-in rules.
Agent-behavior failure	Eval case	You need to measure whether behavior improved.	You have no harness, expected behavior, or scoring.
Sequence/handoff failure	Trace + postmortem	The lesson depends on ordering or context.	A small test or rule already captures it.

Codex-specific surfaces have sharp edges. AGENTS.md is durable repository guidance. [12] Memories are optional local recall, not a system of record for mandatory team policy. [13] Skills are reusable workflows. [14] Hooks are lifecycle automation and require configuration and trust review. [15] Evals require a defined task, expected behavior, harness, and scoring method. [16][17]

Every artifact you add is a liability as well as an asset: it must be maintained, it consumes attention or context, and it can go stale. Add it deliberately, and tag it so Step 7 can find it later.

4. Make the lesson verifiable

A lesson is only as good as the evidence the agent can check it against. This is the highest-leverage move in the protocol, so be ruthless about specificity:

Weak:    - Be careful with authentication.
Strong:  - When editing OAuth callback handling, run `npm test -- auth callback`
           and verify expired-state rejection.

Weak:    - Do not break migrations.
Strong:  - When changing DB migrations, include rollback coverage and run
           `make test-migrations`.

A strong instruction names a trigger ("when editing X"), an action ("run Y"), and an observable result ("verify Z"). The agent can follow specific rules and check specific evidence; it cannot reliably operationalize a vague warning. If you can write only one artifact per failure, write this one.

5. Patch the workflow, not just the code

After every fix, ask: what should exist now so this class of mistake is less likely next time? Step 3 chooses the artifact; Step 5 wires it in: checked into the repo, referenced from AGENTS.md, enforced in CI, packaged as a skill, or run by a configured hook. A lesson that is not wired into the workflow is just a nicer transcript.

6. End with evidence, when the change warrants it

Consequential or correctness-affecting changes should close with evidence. Trivial edits should not; forcing a ceremony on every keystroke just trains people to rubber-stamp it. When it matters, the final report answers:

What changed, and what failure was reproduced?
What check now passes that previously failed?
What durable artifact was added, updated, or retired?
What risk remains?

7. Retire what no longer earns its place

This is the step that keeps the protocol from defeating itself. A pure "capture every lesson" ratchet ends exactly where the agent performs worst: a bloated AGENTS.md that dilutes its own context window, a slow and flaky regression suite, and rules that contradict each other. Durability without pruning is just accumulation.

Give the system a maintenance budget and a retirement rule:

Budget the surfaces. Cap AGENTS.md, for example by requiring the active rules to fit on one screen of scoped, action-oriented lines. When you add a rule that pushes past the cap, retire, merge, or demote an older one.
Date and review. Tag each artifact with the failure it came from and the last time it fired. Review instruction files monthly, or whenever a rule causes confusion. Sweep for artifacts that have not triggered, no longer apply, or duplicate a stronger check.
Resolve conflicts at write time. Before adding a rule, search the existing ones. If the new rule contradicts or supersedes an existing rule, edit in place rather than appending a second voice.
Prefer enforcement over prose. A CI check is more self-checking than prose because failure creates a visible signal. But checks also get flaky, slow, and obsolete, so they need the same retirement discipline as instructions.

Retirement is not optional cleanup; it is what makes "durable" mean load-bearing rather than permanent.

A Minimal Template

# Iron Man Protocol Entry
## Failure        - what happened / how detected
## Diagnosis      - root cause / context-axis + verification-axis scoring
## Upgrade        - code or test change / instruction update / skill|hook|memory|eval
## Verification   - command run / result / remaining risk
## Bookkeeping    - artifact id + date added; rule it replaces, if any

The point is not a long incident report for every small issue. It is to avoid losing the lesson, and to leave a trail Step 7 can prune.

How You Know It Is Working

The protocol's own thesis is that assertion is weak and evidence is strong, so it must hold itself to the same standard. Do not claim the protocol works. Measure it.

The minimum viable version is a mistake log with tags:

- 2026-06-10: schema-client-drift / context + verification / fixed by check-generated
- 2026-06-18: missed-auth-test / verification / fixed by AGENTS rule + test command
- 2026-06-25: bad-localization / context / fixed by routing note in AGENTS.md

Once the log exists, track three simple signals:

Repeat-failure rate: how often a previously captured failure class recurs.
Artifact hit-rate: how often each durable artifact catches a real problem.
Surface budget: size of AGENTS.md, suite runtime, and number of active rules.

If repeat failures stay flat, the artifacts are not changing behavior. If artifacts never fire, they are retirement candidates, not trophies. If surface budget climbs monotonically, your retirement step is broken.

A worked example follows. It is illustrative, not evidence. Substitute your own numbers from a real incident, because a real before/after is the only thing that proves the loop pays off.

Worked Example (illustrative)

A task adds a preferred_locale field to an account API response. The code looks fine and backend tests pass, but CI fails later because the generated TypeScript client still expects the old shape. A reviewer notes this has happened before.

Diagnosis. Context axis: the schema-to-client generation step was invisible in the prompt and the repo rules. Verification axis: no drift check ran before the change was considered done. Both axes, one incident.

AGENTS.md, before -> after.

- Run backend tests before opening a PR.

- When changing API schema files, run `make clients` and commit the regenerated
  client in the same change.
- After clients regenerate, run `make check-generated` before stopping.

Enforcement. Add or confirm a make check-generated drift check in CI that fails when schema changes are not reflected in generated client files. The check is the primary guard; the prose rule tells the agent when to run it before CI does.

Verification.

- Reproduced: `make check-generated` failed before regeneration.
- Repair: ran `make clients`.
- Result: `make check-generated` passed; schema, mapper, and generated client all changed.
- Remaining risk: a future schema package with a different generator would need the rule extended to name it.
- Bookkeeping: schema-client-drift-2026-06; supersedes the bare "run backend tests" line.

Step 7 follow-through. Because the drift check now enforces the behavior in CI, the prose rule is the weaker, higher-maintenance copy. At the next sweep, consider trimming the AGENTS.md lines to a single pointer: "schema changes are gated by make check-generated in CI." That keeps the green check, not the paragraph, closest to the source of truth.

A 30-Minute Start

Pick one recurring failure from the last week. Write one mistake entry. Add or update one regression test, validation command, or CI check that would have caught it. Add one concrete, triggered AGENTS.md rule that says exactly when to run that check. Rerun the validation and record the result.

Then, if instructions already exist, delete, merge, demote, or mark one stale rule for later review. You have now run the full loop, including retirement, on a single failure.

What to Avoid

"Be more careful" prompts. Too vague to enforce; this is the entire reason for Step 4.
Memory as the system of record. Mandatory project rules belong in checked-in instructions, tests, CI, hooks, or skills, not in per-session memory.
Append-only instruction files. Capturing every lesson and keeping files small are only compatible if you actually run Step 7. Adding a rule without budgeting for its retirement is how AGENTS.md becomes noise.
Evals from imaginary failures. The best eval cases come from real bugs, user corrections, production incidents, CI failures, and reviewer comments.
Automating too early. Understand the failure first. A hook that runs the wrong check is negative value.

The Final Idea

A coding agent does not permanently retrain itself after a bad interaction. But the environment around it can be re-engineered after failures, and pruned when the lessons go stale. The repo gains tests; CI gains checks; the agent gains skills and sharper rules; the eval suite gains real failure cases; and the instruction files stay small because what no longer fires gets removed.

Stark survives because the next suit carries the lesson from the last fight, and because he strips the parts that stopped working. Build the agent's environment the same way.

References

[1] OpenAI Developers, "Prompting - Codex." https://developers.openai.com/codex/prompting

[2] Noah Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning." https://arxiv.org/abs/2303.11366

[3] Aman Madaan et al., "Self-Refine: Iterative Refinement with Self-Feedback." https://arxiv.org/abs/2303.17651

[4] Jie Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet." https://arxiv.org/abs/2310.01798

[5] Ryo Kamoi et al., "When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs." https://arxiv.org/abs/2406.01297

[6] Shunyu Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models." https://arxiv.org/abs/2210.03629

[7] Shunyu Yao et al., "Tree of Thoughts: Deliberate Problem Solving with Large Language Models." https://arxiv.org/abs/2305.10601

[8] Guanzhi Wang et al., "Voyager: An Open-Ended Embodied Agent with Large Language Models." https://arxiv.org/abs/2305.16291

[9] Carlos E. Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" https://arxiv.org/abs/2310.06770

[10] John Yang et al., "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." https://arxiv.org/abs/2405.15793

[11] Chunqiu Steven Xia et al., "Agentless: Demystifying LLM-based Software Engineering Agents." https://arxiv.org/abs/2407.01489

[12] OpenAI Developers, "Custom instructions with AGENTS.md - Codex." https://developers.openai.com/codex/guides/agents-md

[13] OpenAI Developers, "Memories - Codex." https://developers.openai.com/codex/memories

[14] OpenAI Developers, "Agent Skills - Codex." https://developers.openai.com/codex/skills

[15] OpenAI Developers, "Hooks - Codex." https://developers.openai.com/codex/hooks

[16] OpenAI Developers, "Working with evals." https://developers.openai.com/api/docs/guides/evals

[17] OpenAI Cookbook, "Build an Agent Improvement Loop with Traces, Evals, and Codex." https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop

[18] Hunter Lightman et al., "Let's Verify Step by Step." https://arxiv.org/abs/2305.20050

[19] Eric Zelikman et al., "STaR: Bootstrapping Reasoning With Reasoning." https://arxiv.org/abs/2203.14465

[20] Aviral Kumar et al., "Training Language Models to Self-Correct via Reinforcement Learning (SCoRE)." https://arxiv.org/abs/2409.12917

AGI, Are We There Yet? A Follow-Up

Ernesto Herrera Salinas — Thu, 25 Jun 2026 03:38:00 +0000

In my previous article, AGI, Are We There Yet?, I asked a simple question with a complicated answer:

Are we at AGI yet?

At the time, OpenAI’s o3 had just made headlines for its performance on ARC-AGI-1. Depending on the compute budget, o3 reached 75.7% on the semi-private evaluation set and 87.5% in a much higher-compute configuration, according to ARC Prize’s analysis of the o3 result.

That was a serious milestone. It was not just another benchmark bump. ARC was designed to test something closer to fluid intelligence: the ability to solve novel problems, not just repeat patterns from training data.

But my conclusion then was cautious:

This was a pit stop on the road to AGI, not the destination.

A year and a half later, that still looks like the right call.

The story since then has not been “AGI arrived.” The story is more interesting than that. Frontier models have become much better at reasoning, coding, tool use, multimodal understanding, long-context processing, and agent-like workflows. They are more useful, more capable, and more economically relevant than the systems we were discussing even a short time ago.

But they are still not robustly general in the human sense.

So where are we today?

We are not at AGI.

But we are no longer talking about simple chatbots either.

We are now in the era of frontier reasoning-and-agent systems.

The short version

Question	Current answer
Did o3’s ARC-AGI-1 result matter?	Yes. It was a real milestone.
Did it prove AGI had arrived?	No. ARC Prize itself warned against that interpretation.
Have frontier models improved since then?	Yes, significantly. Especially in reasoning, coding, multimodality, long context, and tool use.
Are old benchmarks still enough?	No. Many are saturated or noisy.
Are newer benchmarks still difficult?	Yes. ARC-AGI-2, ARC-AGI-3, HLE, and long-horizon task evaluations still expose major gaps.
Are we at AGI?	No. We are at increasingly general-purpose, increasingly agentic systems, but not robust human-like general intelligence.

What the previous article claimed and how it reads today

The previous article defined AGI as the broad adaptability of a human mind, contrasted it with narrow AI, and argued that o3’s ARC result was important but not conclusive.

That framing still holds.

Claim from the original article	Current evidence	Updated reading
AGI should imply broad adaptability, not just task-specific skill.	François Chollet’s On the Measure of Intelligence frames intelligence around skill-acquisition efficiency and fluid generalization, not static benchmark skill.	Still correct. Benchmarks are useful, but not equivalent to AGI.
o3’s ARC-AGI-1 result was a milestone.	ARC Prize reported o3 at 75.7% under the public compute budget and 87.5% in a high-compute configuration.	Still correct. It was a major jump.
Passing ARC-AGI-1 should not be treated as AGI.	ARC Prize explicitly said the result did not mean AGI had arrived and later introduced harder ARC-AGI benchmarks.	Still correct, and even more strongly supported now.
Chain-of-thought is not necessarily a faithful window into reasoning.	Research like LLMs Do Not Think Step-by-step In Implicit Reasoning and later chain-of-thought monitorability work suggests caution.	Still correct. Reasoning traces are useful, but not proof of internal cognition.
Scaling laws matter, but do not guarantee AGI.	OpenAI’s Learning to reason with LLMs and Google’s Gemini 2.5 report show gains from both training-time and inference-time compute.	Correct, with an update: the frontier now also scales through test-time reasoning and tool use.

The o3 ARC result aged well, but not as proof of AGI

The o3 result on ARC-AGI-1 was real and important. It showed that test-time compute, better reasoning methods, and stronger model scaffolding could dramatically improve performance on tasks that had resisted previous systems.

That matters.

But the caveats also mattered.

ARC Prize itself warned that passing ARC-AGI-1 should not be treated as “AGI achieved.” They noted that o3 still failed some tasks humans would find easy, that the result depended heavily on compute, and that the tested system had been trained using part of the public ARC training set.

That distinction is critical.

A model can perform extremely well on a benchmark without possessing the full flexible intelligence that the benchmark was trying to approximate.

And that is exactly what became clearer afterward.

ARC-AGI-1 started to saturate, so ARC-AGI-2 was introduced. This newer benchmark kept the original spirit of ARC but increased the difficulty around compositional reasoning, contextual rule application, and novelty.

The result?

As of the ARC-AGI-2 paper, frontier models were still below meaningful performance. OpenAI o3 scored around 3%, and other leading systems were similarly weak. The benchmark authors considered scores under 5% not meaningful.

Then ARC-AGI-3 went further. Instead of static puzzles, it introduced interactive environments where agents must infer goals, explore, learn rules, and adapt through interaction.

Again, frontier systems struggled. As of the ARC-AGI-3 paper, leading AI systems were below 1%, while humans could solve the environments reliably.

That is the cleanest update to my previous article:

o3 broke through ARC-AGI-1, but the broader ARC program did not confirm AGI. It confirmed that evaluation had to move.

That is progress, but it is not arrival.

Timeline: AGI-relevant milestones since the previous article

Date	Milestone	Why it matters
Dec 2024	ARC Prize reports o3 at 75.7% high-efficiency and 87.5% high-compute on ARC-AGI-1.	A major reasoning benchmark breakthrough, but not proof of AGI.
Jan 2025	Humanity’s Last Exam is introduced.	A harder, broad academic benchmark designed after older benchmarks became easier for frontier models.
Mar 2025	METR publishes work on measuring AI ability to complete long tasks.	Moves evaluation from static questions toward task duration and real-world work horizons.
Apr 2025	OpenAI releases o3 and o4-mini.	Reasoning models become more explicitly tool-using and agentic.
May 2025	Anthropic releases Claude 4.	Stronger agentic coding and long-running software workflows become a major frontier battleground.
May 2025	ARC-AGI-2 paper shows frontier systems still below meaningful performance.	Strong evidence that novelty and compositional generalization remain hard.
Jun 2025	Google publishes the Gemini 2.5 technical report.	Shows broad gains in coding, reasoning, multimodality, long context, and thinking-budget performance.
Mar 2026	ARC-AGI-3 is introduced for agentic intelligence.	Moves evaluation from static reasoning puzzles to interactive environments.
May 2026	METR publishes updated task-completion time horizon measurements.	Tracks how long frontier models can work reliably on realistic tasks.

The important shift is not just that model scores went up. The field also changed what it measures.

By early 2025, older tests were already too easy or too noisy. By mid-2025, the most capable models were being compared on coding agents, difficult science questions, long-context retrieval, multimodal reasoning, and expert-level exams. By 2026, benchmark designers were moving from static test items to interactive environments.

That evolution itself is strong evidence that “AGI” cannot be reduced to one benchmark win.

Benchmarks improved, but benchmarks also got weirder

Since the previous article, model scores have improved across a lot of hard benchmarks.

Frontier systems now perform strongly on:

GPQA
MMMU
AIME
SWE-bench
LiveCodeBench
Humanity’s Last Exam
long-context retrieval tasks
tool-use evaluations
multimodal reasoning evaluations

This is not trivial. These are not toy tasks. Some of these benchmarks test graduate-level science, difficult math, software engineering, visual reasoning, and multi-step problem solving.

But there is a problem.

The better models become, the less stable old benchmarks become as evidence.

MMLU used to be a big deal. Now it is mostly saturated. HumanEval used to be a strong coding signal. Now it is too easy for frontier models. GSM8K used to be a standard math benchmark. Now it does not tell us much about the frontier.

Even worse, some benchmarks contain errors. The paper Are We Done with MMLU? found that a non-trivial percentage of MMLU questions contain mistakes, with some subsets much worse than others.

Stanford HAI’s 2025 AI Index technical performance chapter also notes that traditional benchmarks like MMLU, GSM8K, and HumanEval have been saturating, while newer benchmarks like MMMU, GPQA, and SWE-bench became more important.

So the benchmark story today is mixed.

Yes, models are getting much better.

But also, the measurement problem is getting harder.

Mid-2024 generation snapshot

The following table shows the mid-2024 generation of strong models. These values are useful for historical context because they show how quickly the conversation moved from “chatbot” to “general-purpose assistant.”

Important caveat: these numbers are not perfect apples-to-apples comparisons. Different labs use different prompts, scaffolds, evaluation setups, and reporting conventions.

Model	MMLU	MMLU-Pro	GPQA	HumanEval	BFCL tool use	Long-context example
GPT-4	86.4	-	-	67.0	-	Multimodal text and image input supported
GPT-4o	89.1	74.0	53.6	90.2	80.5	NIAH 100.0
Claude 3.5 Sonnet	89.9	77.0	59.4	92.0	90.2	NIAH 90.8
Llama 3.1 405B	87.3	73.3	51.1	89.0	88.5	ZeroSCROLLS/QuALITY 95.2
GPT-3.5 Turbo	70.7	49.2	30.8	68.0	85.9	-

Sources: GPT-4 Technical Report, The Llama 3 Herd of Models

This table explains why the conversation changed so quickly in 2024. The strongest systems were no longer merely fluent chatbots. By mid-2024, leading models were posting near-saturated MMLU scores, strong code generation, usable tool calling, and credible long-context retrieval.

But even here, GPQA remained much lower than MMLU, which foreshadowed the field’s shift toward harder reasoning benchmarks.

Mid-2025 frontier reasoning snapshot

By mid-2025, the frontier had moved from “models that answer” toward “models that reason with tools.”

Model	LiveCodeBench	Aider Polyglot	SWE-bench Verified	GPQA	Humanity’s Last Exam	AIME 2025	LOFT <=128K	MMMU
Gemini 2.5 Pro	74.2	82.2	59.6 single / 67.2 multi	86.4	21.6	88.0	87.0	82.0
OpenAI o3 high	72.0	79.6	69.1 single	83.3	20.3	88.9	77.0	82.9
OpenAI o4-mini high	75.8	72.0	68.1 single	81.4	18.1	92.7	60.5	81.6
Claude 4 Sonnet	48.9	61.3	72.7 single / 80.2 high-compute multi	75.4 with extended thinking / 70.0 without	7.8	70.5	81.6	74.4
Claude 4 Opus	51.1	72.0	72.5 single / 79.4 high-compute multi	79.6 with extended thinking / 74.9 without	10.7	75.5	-	76.5
DeepSeek R1	70.5	71.6	57.6 multi	81.0	14.0	87.5	-	No native multimodal

Sources: Gemini 2.5 technical report, Claude 4 announcement and benchmark appendix

This table captures the current situation better than any single “AGI score.”

The strongest systems now span a much larger capability bundle than the previous article discussed:

frontier coding
difficult science QA
difficult math
long-context retrieval
multimodal understanding
tool use
agentic workflows

Gemini 2.5 Pro looks especially broad in public vendor reporting. OpenAI’s o3 and o4-mini look especially strong on reasoning-heavy tasks and multimodal perception. Anthropic’s Claude 4 family looks especially strong in coding and tool-using software workflows.

But the caution is important.

Benchmark wins are not stable enough to map directly to AGI claims. Different labs use different scaffolds and compute settings. Anthropic explicitly distinguishes no-extended-thinking benchmarks from benchmarks run with extended thinking and tool use. Google also notes that many non-Gemini results are provider self-reports.

These comparisons are useful, but only as capability indicators. They are not a definitive scoreboard for general intelligence.

The biggest shift: models now think with tools

In the previous article, I talked about chain-of-thought and the question of whether models are really reasoning step by step or just producing text that looks like reasoning.

That question is still open.

But the field has shifted.

The most important change is that frontier models are no longer only “answer generators.” They are increasingly systems that can use tools while reasoning.

Modern frontier models can call Python, browse the web, inspect files, work with images, write and execute code, use external tools, and continue across longer workflows. Some models are explicitly trained to spend more time reasoning before answering. Others allow configurable “thinking budgets,” where more inference-time computation can improve performance.

This is a major shift.

The frontier is no longer just about bigger pretraining runs.

It is now scaling along at least three axes:

Pretraining scale
Post-training and reinforcement learning
Inference-time reasoning and tool use

That third axis is especially important.

A model that can pause, inspect evidence, run code, test a hypothesis, revise its plan, and continue working is meaningfully different from a model that only predicts the next token in a single pass.

But this still does not mean AGI.

It means we are building more capable systems around language models. Some of the intelligence is in the model. Some of it is in the scaffold. Some of it is in the tools. Some of it is in the feedback loop.

That matters because when people say “the model solved it,” we now need to ask:

Did the model solve it from internal reasoning?
Did it solve it through tool use?
Did it solve it through repeated attempts?
Did it solve it because the benchmark was already familiar?
Did it solve it because the environment was carefully scaffolded?

Those are not nitpicks. They are the difference between measuring a model and measuring a whole engineered system.

Long-horizon work is improving, but still limited

One of the most useful ways to think about current AI progress comes from METR’s task-completion time horizon work.

Instead of asking, “What score did the model get on this benchmark?”, METR asks something more intuitive:

How long are the tasks that AI systems can complete reliably?

Their March 2025 paper, Measuring AI Ability to Complete Long Tasks, estimated that frontier models had reached a 50% task-completion horizon of around 50 minutes on their task suite. METR also reported that this horizon had been doubling roughly every seven months since 2019.

That is impressive.

It means the systems are not just answering trivia. They are becoming capable of completing longer, messier, more realistic tasks.

But it also shows the limitation.

A 50-minute task horizon is not the same as a full workday. It is not a week-long project. It is not open-ended research. It is not reliable autonomous operation in the real world.

METR’s later time-horizon measurements continue tracking this trend, but the core conclusion remains: the trend is important, and the current level is still not AGI.

This matches what many software engineers experience in practice. The models are extremely useful. They can write code, review code, explain systems, generate tests, help debug, and accelerate development. But they still need supervision. They still make confident mistakes. They still lose track of constraints. They still require someone with judgment to decide whether the answer is actually correct.

That is not a small limitation.

That is the boundary between assistance and autonomy.

Chain-of-thought is still not a solved window into reasoning

The previous article mentioned research suggesting that implicit chain-of-thought may not always reflect genuine step-by-step reasoning.

That concern has not gone away.

If anything, it has become more important.

As reasoning models become more capable, their intermediate reasoning becomes more valuable for debugging, safety, and evaluation. If a model explains its reasoning, we might be able to catch mistakes, deception, hidden assumptions, or unsafe intent.

But that only works if the reasoning we see is faithful.

The paper LLMs Do Not Think Step-by-step In Implicit Reasoning argues that when models are prompted to use implicit reasoning, they often do not appear to internally compute intermediate steps in a robust way.

Later safety work, such as A New and Fragile Opportunity for AI Safety, frames chain-of-thought as a useful monitoring surface, but also a fragile one.

That creates a strange situation.

The more powerful reasoning models become, the more we want to inspect their reasoning.

But the more strategically capable they become, the less we can naively assume their visible reasoning is complete, faithful, or safe.

This does not mean chain-of-thought is useless.

It means it should be treated as a tool, not as proof.

A model writing a beautiful explanation is not the same as a model having robust, grounded, human-like understanding.

So what is missing?

The missing piece is not raw capability.

Raw capability is clearly improving.

The missing piece is robust generality.

Humans are not impressive because we can memorize internet-scale text. Humans are impressive because we can enter new environments, infer goals, test ideas, learn from sparse examples, adapt under uncertainty, and transfer knowledge across domains.

That is the kind of ability AGI is supposed to capture.

Today’s frontier models are broad, but brittle.

They can be superhuman on some tasks and surprisingly weak on others. They can solve difficult coding problems but fail simple novel puzzles. They can reason through expert-level questions and still hallucinate. They can use tools but also misuse them. They can follow instructions but still lose track of the real goal.

This unevenness is important.

A system that is excellent across many benchmarks is not automatically generally intelligent. It may simply be very powerful across many benchmark-like distributions.

AGI requires more than breadth.

It requires:

transfer
reliability
adaptation
grounded planning
calibration
robustness under unfamiliar conditions

That is where the strongest evidence still says: not yet.

Why this still is not AGI

The cleanest reason is that novelty remains hard.

ARC-AGI-2 was built specifically to preserve human accessibility while increasing compositional difficulty, contextual rule application, and in-context symbol definition. The benchmark authors report that scores below 5% are generally not considered meaningful. Leading systems remained below that threshold.

ARC-AGI-3 raises the bar again in interactive environments, and its paper reports frontier systems below 1% as of March 2026.

If one’s AGI criterion includes fluid intelligence and open-ended adaptation, those numbers matter more than a high score on a saturated benchmark.

The second reason is that benchmark saturation and benchmark quality problems make many impressive scores less informative than they first appear. Stanford’s 2025 AI Index explicitly says traditional benchmarks are saturating. Separate work on MMLU found errors that can mislead evaluation and model comparison.

The third reason is that long-horizon reliability is still limited. METR’s time-horizon framing is helpful because it shows real progress, but also shows that current systems are not yet reliable for long, messy, open-ended work.

The fourth reason is that reasoning transparency remains unsettled. Reasoning models are impressive, but their internal cognition is not understood well enough to use as evidence of full AGI.

The best current description: general-purpose, not generally intelligent

Here is where I would place today’s frontier AI systems:

They are not narrow AI in the old sense.

They are also not AGI.

They sit in a middle category that is becoming more important:

general-purpose AI systems with emerging agentic capabilities.

They can operate across domains. They can use tools. They can reason more deeply than previous models. They can handle text, images, audio, code, long documents, and structured tasks. They can assist professionals in meaningful ways.

But they still do not have the kind of robust, open-ended, human-like adaptability that most serious definitions of AGI imply.

This is why saying “AGI is here” feels premature.

But saying “it is just autocomplete” is also outdated.

The honest position is uncomfortable because it does not fit the usual internet debate.

We are not at AGI.

But we are much closer to something economically disruptive than many people expected.

Bottom line

In the previous article, I treated o3’s ARC result as a milestone, not a destination.

That conclusion has aged well.

The milestone was real. The hype was excessive. The research since then has clarified the picture.

ARC-AGI-1 showed that frontier systems could make surprising jumps.
ARC-AGI-2 and ARC-AGI-3 showed that novelty, interaction, and compositional generalization are still hard.
METR’s work showed that long-horizon task completion is improving quickly, but also that current systems are still far from reliable long-term autonomy.
New reasoning models showed that inference-time computation and tool use matter a lot.
Safety research showed that more capable systems also require more serious evaluation.
Benchmark research showed that our measurement tools are struggling to keep up.

So, are we there yet?

No.

But the road changed.

The question is no longer whether AI can be useful across many domains. That has already been answered.

The question is whether these systems can become robustly adaptive, reliable, and autonomous outside the distributions where we train, scaffold, and evaluate them.

That is the real AGI question now.

And today, the factual answer is:

We are not at AGI yet. We are at increasingly general-purpose reasoning systems that can look shockingly intelligent in many contexts, while still failing in ways that reveal they are not generally intelligent in the human sense.

That may be less dramatic than “AGI is here.”

But it is much more useful.

And honestly, much more interesting.

Sources

Primary sources and technical reports used for this follow-up:

How to Reduce Codex Token Spend Without Reducing Code Quality

Ernesto Herrera Salinas — Tue, 23 Jun 2026 22:46:00 +0000

Scope: Verified against Codex documentation on June 22, 2026.

The safe way to reduce Codex token spend is to optimize only while fixed quality gates remain unchanged. “Without reducing quality” should not mean trusting that a shorter transcript looks equally good. It means defining requirements, tests, static checks, and review criteria before the run, then rejecting any cheaper setup that performs worse against those gates.

This principle applies across the Codex CLI, IDE extension, app, and cloud, but the budget appears differently on each surface. ChatGPT users generally consume included usage, rate limits, or credits rather than receiving a direct token invoice. API-key users pay standard, model-specific API rates. Context usage affects how much information fits in a thread, while latency is simply elapsed time. These are related operational constraints, not interchangeable measurements. See the official Codex pricing, authentication, and API pricing documentation.

1. Define and measure the budget

A Codex thread contains more than the prompt and final answer. Codex gathers file contents, tool output, model responses, and the ongoing work record; all of it must fit in the model’s context window. Long logs and broad file reads can therefore consume context even when the final response is short. Codex may compact long threads automatically, but compaction summarizes and discards detail rather than making context free (Prompting Codex).

Measure four separate outcomes:

Context: input tokens, cached input tokens, and remaining context capacity.
Generated tokens: output tokens and reasoning-output tokens.
Account cost: API charges, ChatGPT credits, or rate-limit consumption.
Operational efficiency: elapsed time, turns, failed verification, and corrective attempts.

For a reproducible comparison, use this proposed method rather than relying on anecdotes:

Choose five representative tasks. Give each task a saved prompt, a fixed starting commit, an exact verification command, and a written review checklist.
Create a baseline profile and one candidate profile. Change one variable at a time. If testing context discipline, keep the model and reasoning effort fixed; test model changes separately.
Run each task three times per profile from a clean copy of the same commit. Use a new noninteractive thread for every initial run.
Capture the event stream:

   codex exec --profile baseline --json - \
     < prompts/task-01.txt > results/task-01-baseline-1.jsonl

Immediately filter the completion event instead of printing the full machine-readable stream into another model context:

   jq -c 'select(.type == "turn.completed") | .usage' \
     results/task-01-baseline-1.jsonl

Run the predefined verification command outside Codex. Record pass or fail. If it fails, take the exact session ID from the thread.started event, resume that thread with the verification log and the fixed instruction, “Make the smallest correction required to pass the stated gate,” and save each corrective turn to another JSONL file. Stop after three corrective attempts and record failure if the gate still does not pass.
Store one row per task and repetition with: profile, input tokens, cached input tokens, output tokens, reasoning-output tokens, completed turns across all JSONL files, verification result, and corrective-attempt count. Compare medians only after confirming that requirement, test, and review outcomes are equivalent.

codex exec --json emits JSONL events, and its turn.completed usage object reports input_tokens, cached_input_tokens, output_tokens, and reasoning_output_tokens (Non-interactive mode). This article does not claim benchmark results; the procedure is a way to produce evidence for your own workload.

2. Establish quality gates first

Token reduction often fails because the task was underspecified. A five-word prompt can trigger exploration, assumptions, rework, and repeated checks. OpenAI recommends giving Codex a goal, relevant context, constraints, and a definition of done, and recommends including reproducible validation steps (Codex best practices).

Use a prompt shape like this:

Goal: Fix duplicate invoice creation in the retry path.
Context: Start with src/billing/retry.ts and tests/billing/retry.test.ts.
Constraints: Preserve the public API and database schema. Do not refactor
unrelated billing code.
Done when: The regression test and billing unit suite pass, and the final
diff contains no unrelated changes.

For higher-risk work, add type checks, linting, security checks, performance thresholds, or a human review rubric. Keep these gates identical across the baseline and candidate runs. A configuration that uses fewer tokens but needs more corrective turns, misses requirements, or produces a riskier diff is not an improvement.

3. Reduce irrelevant context and tool output

Point Codex toward likely files and explicitly exclude generated code, dependencies, build artifacts, coverage output, and unrelated services. Prefer targeted search and line-range reads over printing whole trees or large files. Run the smallest relevant test first, then expand only when the risk or failure requires it.

Filter command output before it enters the thread:

rg "InvoiceRetry" src tests
git status --short
git diff --stat
npm test -- retry.test.ts --runInBand

Machine-readable output is useful only when it is immediately filtered to the fields needed for the decision. Raw JSONL, verbose test reporters, and full CI logs can be larger than concise human-readable output. Preserve the failing assertion, stack trace, and summary; omit repeated successes and unrelated diagnostics. Codex also has a tool_output_token_limit setting, but a hard cap is a backstop because truncation can hide the actual error (Configuration reference).

Use AGENTS.md for durable routing and verification commands, but understand discovery correctly. Codex loads one global instruction file, then walks from the project root down to the directory where the run started, loading at most one instruction file per directory. It does not universally load a nested AGENTS.md merely because it later edits a file beneath that directory. Start Codex in the relevant subtree when those nested instructions must apply, and keep instruction files concise because they enter context at startup (AGENTS.md guide).

The self-published Caveman project is a Claude API output-style benchmark, not evidence about Codex, reasoning tokens, or total task cost. Its useful lesson is limited: removing filler can reduce visible output, but terse prose does not compensate for irrelevant reads, noisy tools, or retries.

4. Match model and reasoning effort to the task

Use the least expensive setup that still passes the fixed gates. OpenAI currently recommends gpt-5.5 for demanding Codex work and gpt-5.4-mini as a faster, lower-cost option for lighter coding tasks. Reasoning effort can be low for well-scoped mechanical changes and medium or high for complex debugging; higher effort uses more tokens and can consume limits faster (Codex models and IDE features).

Do not reduce model capability, reasoning effort, and context simultaneously in the first experiment. If the candidate fails, you will not know which change caused it. Escalate deliberately when ambiguity, concurrency, security, or repeated failed verification demonstrates the need.

Fast mode is a latency control, not a savings control. With ChatGPT sign-in it makes supported models faster while consuming credits at a higher rate; API-key users instead remain on standard API pricing. Use /fast off when the objective is credit efficiency rather than response speed (Codex speed).

Cloud tasks currently do not expose local model selection, so optimize them through task scope, repository instructions, environment setup, and verification rather than a local profile (Codex models).

5. Manage threads, compaction, and subagents

Keep one thread aligned to one objective. In the CLI, /status reports session configuration, token usage, and remaining context capacity. /compact summarizes the conversation to free context, while /new or /clear starts a fresh conversation when prior investigation is no longer relevant. /usage is account-level ChatGPT activity or rate-limit monitoring; it is not a per-run benchmark. These controls and their current meanings are documented in the CLI slash-command reference.

Compact at a phase boundary, such as after diagnosis or implementation, rather than repeatedly. Start a new thread when a task changes materially. Resume only when previous decisions and repository state are still useful.

Subagents can keep noisy exploration out of the main thread and reduce elapsed time for independent work, but they do not reduce total token consumption. Each subagent performs its own model and tool work, so comparable multi-agent runs consume more tokens. Reserve them for parallel research, test analysis, or independent review dimensions where isolation or speed justifies the extra cost (Subagent concepts).

6. Apply a concise reusable configuration

The CLI supports named profiles stored as separate files under CODEX_HOME, which defaults to ~/.codex. For example:

# ~/.codex/economy.config.toml
model = "gpt-5.4-mini"
model_reasoning_effort = "low"
model_verbosity = "low"

Run it with codex --profile economy or codex exec --profile economy "task". Profile files use top-level keys; current Codex no longer reads legacy [profiles.name] tables from config.toml (Advanced configuration). model_verbosity applies to Responses API providers (Advanced configuration).

The CLI, IDE extension, and app share configuration layers (Codex best practices). For this workflow, the CLI provides named profiles, its slash-command interface, and codex exec --json usage events (profiles, slash commands, and noninteractive mode). Keep project-specific paths and checks in the repository’s AGENTS.md, and use the profile only where its model choice passes your gates.

Every concise final report should still preserve four review outputs:

changed behavior;
checks run;
results;
unresolved risks.

That is the quality boundary: spend fewer tokens only when those outputs and the underlying verification outcomes remain intact.

Dev Log: Shipping the Foundation

Ernesto Herrera Salinas — Tue, 23 Jun 2026 03:07:00 +0000

M8 is the last milestone, and it has the texture of every "last 10%" you've ever done: not one big feature, but a dozen smaller things that together turn "it works" into "it's done." Explain the zero-config path, an allocation budget, benchmarks, public documentation, and final goldens. Closing-time work. I like closing-time work.

Making the magic inspectable

The feature I cared most about here is Explain(). Munchausen infers a lot; it quietly decides your Make is a car make and your Email is an email, and any inferred behavior you can't inspect is just spooky action. So Explain() walks the compiled plan and tells you, per member, exactly what it decided and why:
the source, the generator, the confidence, and what rules got overridden. It generates nothing; it just reads the plan. Car.Make -> Vehicle.Make [property name + model] High. No more guessing why a value looks the way it does.

A nice cleanup came with it: two enums I'd kept internal since M4 to avoid committing them publicly before Explain() needed them finally became public.
The promotion was almost free thanks to a C# namespace-resolution quirk: moving the enums to the root namespace meant that every internal reference compiled, since nested namespaces see their parent namespace. Small, tidy, satisfying.

The allocation test that made me earn it

One of my performance goals was "no reflection per object," but implementation inspection did not feel like enough proof. So I wrote an allocation test: generate 50,000 objects, measure the bytes allocated, and divide. The accessors were already
compiled (M2), but construction still used reflection's ConstructorInfo.Invoke.
I compiled the constructor into a delegate too, an Expression that creates the object from a boxed args array.

First run of the test: 408 bytes per object. My threshold was 400. Off by a whisker. I stared at it, half-expecting a hidden reflective allocation, then realized: that's just boxing. The model has five value-type members, and every value source returns a boxed object. 408 bytes is five boxes plus the object plus
a list slot, entirely accounted for, zero reflection. I shaved the empty-args array allocation, nudged the threshold to a value that comfortably separates "boxing" from "reflection would be here," and documented the reasoning right in the test.
A good reminder that an allocation number you can fully explain is worth more than a green checkmark you can't.

A dumb little collision

One of those five-minute facepalms: LieDefinition<T>.Explain() is a method, and Munchausen.Explain is a namespace. So when I wrote Explain.InferenceReportBuilder inside the method, the compiler thought I meant the method and gave me a baffling CS0119. Fully-qualify it (Munchausen.Explain.InferenceReportBuilder), and it's
fine. The kind of error that's obvious the instant you see it and inscrutable for the thirty seconds before.

The last golden

For the final golden, I generated the canonical Car through the zero-config Lie<Car> path, but pinned both the seed and the reference time, because Vehicle.Year depends on "current year" and I didn't want a golden that breaks every January 1st. Out came a deterministic Mazda Sorento, model year 2020, $788.99, owned
by Linda King. That one fixture now locks the entire pipeline end to end: inference, datasets, traversal, the lot.

Done

And that's v1.0. Nine milestones, from an empty scaffold to a library that takes your type and hands you believable data, with a locked public surface, a deterministic core validated against published vectors, an inference engine you can interrogate, and
goldens that pin it all in place. Two hundred-odd tests, green, zero warnings.

The thing I'll remember about this build is how often implementation improved the design. Working one milestone at a time gave each idea room to fail clearly: the canary tested the public boundary, reference vectors tested determinism, and the inference contradiction forced me to articulate what good fake data should feel like. Treating seeded output and public API changes as deliberate decisions made the final milestone calm instead of a panic. Nothing important remained, only an assumption.

What's next

Not M9; there isn't one. v1.1 needs a new design pass: inference providers, attributes, hooks, validation, uniqueness, locales, child definitions. The seams are already sitting in the code, empty and waiting. But that's a new plan for another day.

For now:

Engineering Post: Making inference inspectable, generating cheaply, and the surface final

Ernesto Herrera Salinas — Tue, 23 Jun 2026 03:00:00 +0000

M8 is the finale: the Explain() report family, the cached Lie<T> automatic path, reflection-free construction, the benchmark suite, and the final goldens.
It tests whether the ideas developed independently across eight milestones hold together as one library.

Explain: a pure walk of the plan

I did not want inferred behavior to remain opaque, so Explain() materializes an InferenceReport by walking the frozen plan, generating nothing. Each member maps to a MemberInferenceReport (source, generator, confidence, derivation order, overridden rules); nested members appear as a single entry, so a recursive
Employee.Manager is one line, not infinite expansion. ToText() renders a human form whose exact prose is explicitly not contractual, the tests assert structure, not wording.

This is also where InferenceConfidence and InferenceSource graduate from the internal mirrors of M4 to public enums in the root namespace. Because a Munchausen.Inference file resolves an unqualified InferenceConfidence against the enclosing Munchausen namespace, the promotion is almost free, move the enums up, delete the internal copies, and every internal reference still compiles.

A subtle fix rode along: nested/collection members now report ChildDefinition as their source (they reported Type before), so the report reads correctly, Car.Owner -> (nested object).

The automatic path

public static class Lie<T>
{
    private static readonly Lazy<GenerationPlan> CachedPlan =
        new(() => DefinitionCompiler.Default.CompileAutomatic(typeof(T)),
            LazyThreadSafetyMode.ExecutionAndPublication);
}

A generic static gives a free, thread-safe, per-closed-type cache;
CompileAutomatic sits over a process-wide plan cache; and Lazy caches a
compilation failure too, so a bad type fails consistently instead of racily. The
acceptance test proves the automatic path equals an inferred-only definition for
the same seed, Lie<Car>.Generate and Lie.Define<Car>().Build().Generate agree
byte for byte.

Zero per-object reflection

One of my performance goals was no reflection at generation time. Rather than trusting inspection alone, I verified it with an allocation test. The last reflective holdout was construction: M5/M6 used ConstructorInfo.Invoke. M8 compiles the constructor to a delegate at build time:

// arguments => (object)new Ctor((T0)arguments[0], (T1)arguments[1], ...)
Expression.Lambda<Func<object?[], object>>(body, arguments).Compile();

Combined with the M2 compiled accessors and Array.Empty for parameterless ctors, steady-state generation touches no reflection. The allocation test generates 50,000 objects and asserts per-object allocation stays under a small bound, boxing of value-type members is accepted in v1.0, but per-object Invoke/GetValue would
blow past it.

Hardening

Benchmarks: a BenchmarkDotNet suite covering warm Build, per-object and batch generation, a collection model, and PRNG throughput, tracked for trend, not gated.
Docs: every public member is documented (CS1591 is an error, so a clean build is the proof). Ignore/Preserve docs open by contrasting each other; builder methods name the LIE codes they can cause.
Surface: PublicAPI.Shipped.txt records the final v1.0 API and the analyzer rejects accidental changes.
Goldens: a final Lie<Car> golden with a fixed seed and reference time (so Vehicle.Year is deterministic) pins the whole inference + dataset + traversal pipeline.

What's next: v1.0 is done🎉; v1.1 is a new conversation

That's the v1.0 Foundation, complete: M0 scaffold through M8 hardening. The seams for v1.1 already exist and sit empty by design: provider-stage slots in the pipeline, hook/validator/uniqueness arrays on each TypePlan, the reachable-plan dictionary that child definitions (Use) can substitute into, and a SequenceTable field reserved for the operation. Inference providers, DataAnnotations attributes, hooks, validation, uniqueness, locale packs, and child definitions deserve a new design pass rather than being squeezed into the finished plan.