DEV Community: varun pratap Bhardwaj

GPT-6 Astra Is Not the End of the AI Race. It Changes the Architecture of How We Work With AI

varun pratap Bhardwaj — Sat, 05 Sep 2026 06:44:49 +0000

GPT-6 Astra Is Not the End of the AI Race. It Changes the Architecture of How We Work With AI

Why 99.9% does not mean “AGI solved,” why 62.7% may be the more important number, and how to actually use Astra, Sol, Terra, Luna, Codex and Hermes without wasting your subscription

Research status: verified against OpenAI, ARC Prize, Artificial Analysis and Hermes Agent documentation on 5 September 2026. Product entitlements are changing during rollout; revalidate plan access immediately before publication.

By Varun Pratap Bhardwaj · @varunPbhardwaj · Qualixar AI Reliability Engineering

OpenAI released GPT-6 Astra in September 2026 and immediately gave the AI industry the kind of headline it loves: a new model generation, a near-saturated “AGI” benchmark, substantially stronger computer use, and a claim that the frontier is shifting again.

The number that detonated across social feeds was 99.9% on ARC-AGI-3.

At first glance, the story seems almost too clean. The benchmark name contains “AGI.” Previous frontier models were dramatically lower. Astra appears to jump to essentially perfect performance. If you are building a thumbnail, an X post, or a breathless reaction video, the obvious conclusion is irresistible: GPT-6 just solved an AGI benchmark; AGI is here.

That conclusion is not supported by the evidence.

The real story is more complicated, more useful, and—if you build AI systems—more important.

GPT-6 Astra did achieve a 99.9% ARC-AGI-3 result using ARC Prize’s Provider Adapter harness. But when ARC Prize ran Astra through its Standard harness, designed as a minimal provider-neutral interface for cross-provider comparisons, Astra’s best result was 62.7%.

Claude Opus 5 was reported around 30.2% in the Standard-harness comparison. GPT-5.6 Sol was dramatically below Astra on ARC-AGI-3. The 62.7% result is therefore not a disappointment. It is an extraordinary result.

But the gap between 62.7% and 99.9% changes the interpretation.

The Provider Adapter preserves provider-specific opaque reasoning state between calls and uses compaction during longer conversations. The Standard harness leaves the model responsible for deciding what visible notes to carry forward. ARC Prize reports that the Provider Adapter runs were not merely more successful: across comparable solved game/reasoning pairs they were about 3.66× faster by aggregate recorded elapsed time and used 49% fewer total tokens.

Same model family. Different execution architecture. Radically different outcome.

That is the part of the Astra launch that deserves much more attention than the AGI shouting match.

The model is no longer the whole AI system.

Context management matters. State continuity matters. Memory matters. Compaction matters. Tools matter. Execution environments matter. Agent scaffolding matters. Review loops matter. Cost controls matter.

A frontier model inside a bad system can waste its intelligence. A frontier model inside a disciplined system can behave like a qualitatively more capable worker.

That is the architecture shift this article is about.

1. Start with the benchmark honestly

ARC-AGI-3 is not a standard static question-answer benchmark. It tests an agent in unfamiliar interactive environments. The system must explore, infer mechanics, identify goals and execute plans. That makes it especially relevant to the direction frontier models are moving: away from one-shot text generation and toward repeated action in software environments.

ARC Prize describes capabilities such as exploration, modeling, goal identification, planning and execution. The agent does not simply retrieve a memorized answer. It needs to build a useful representation of a novel environment and act on that representation.

This is why Astra’s performance is meaningful.

ARC Prize observed Astra constructing compact symbolic world models, tracking rules and state, and using increasingly effective representations as it learned how an environment worked. In richer scaffolds, this class of agent behavior can include parsers, planners, search procedures and task-specific tooling.

That is closer to what we mean when we talk about an AI agent: not just a system that can say something correct, but one that can maintain an objective across many steps, revise its model of the world and make progress through action.

The benchmark therefore matters.

But the exact evaluation condition matters too.

The Standard harness

ARC Prize’s Standard harness is designed to provide a common minimal interface. The model gets what it needs to interact with the environment, but it decides what to preserve in visible notes across turns.

Astra’s best reported Standard-harness score is 62.7% at max reasoning.

That is the cleanest Astra number to use when you are making a cross-provider ARC-AGI-3 comparison.

The Provider Adapter harness

The Provider Adapter uses provider-specific context-management features. For Astra, ARC Prize says this includes preserving opaque reasoning state between requests and using compaction for longer conversations.

The best observed result is 99.9% at high reasoning.

That result is real. It should not be dismissed. But it answers a different evaluation question:

How well can Astra perform when it is allowed to use the context-management architecture designed around it?

This is a legitimate and practically important question. Real deployed agents do not live in a perfectly provider-neutral vacuum. They have memory systems, context managers, cache behavior, tools, runtimes and state-retention mechanisms.

What is not legitimate is showing the 99.9% number next to a competitor’s provider-neutral result and pretending the harness conditions are identical.

The honest presentation is stronger anyway:

Astra is very strong under a neutral harness—and almost saturates the benchmark when given its provider-specific state-management machinery.

That tells us something about the model and something about AI-system design.

2. Why “62.7% vs 99.9%” may be more important than “99.9%”

For years the AI industry optimized for the model card.

Which model has the highest score?

Which model has the most parameters?

Which model wins MMLU, GPQA, SWE-bench, ARC or some new composite index?

Those comparisons remain useful, but long-running agents introduce another dimension: system capability.

A minimal language-model loop looks like this:

Prompt
  ↓
Model
  ↓
Answer

A serious agent looks more like this:

Goal
  ↓
Planner
  ↓
Model
  ↓
Context manager
  ↓
Retained state / memory
  ↓
Tools
  ↓
Browser / terminal / professional software
  ↓
Observation
  ↓
Model
  ↓
Recovery / compaction / review
  ↓
Next action
  ↓
...

A one-shot answer can be judged largely by the quality of a single model invocation.

A four-hour agent run may contain hundreds of dependent decisions. It accumulates state. It discovers facts. It creates hypotheses. It makes mistakes. It changes files. It receives tool output. It needs to distinguish a durable decision from a transient observation. It needs to remember what failed without dragging every irrelevant token forever.

This is where raw model intelligence stops being enough.

A system can fail because it forgets what it already learned.

It can fail because it keeps a failed hypothesis alive for another fifty turns.

It can fail because its context has become so large that relevant state is buried under logs and dead ends.

It can fail because it repeatedly re-explores the same area.

It can fail because one successful subtask violates an invariant the broader system depends on.

It can fail because no component decides what deserves to become canonical state.

The ARC harness gap gives us a concrete example of how much those surrounding mechanisms can matter.

The correct lesson is not “provider adapters are cheating.”

The correct lesson is:

When models become agents, context engineering becomes part of capability engineering.

That is a much more durable insight than any launch-week leaderboard position.

3. Does GPT-6 Astra prove AGI?

No single benchmark can settle that question because the field still lacks a universally accepted operational definition of AGI.

Some definitions emphasize broad human-level competence across cognitive tasks. Some emphasize economic work. Some emphasize transfer learning and rapid adaptation. Some require autonomy. Some require the ability to learn new tasks efficiently. Some definitions are so broad that they become philosophical rather than measurable.

This definitional problem is exactly why a benchmark name should not be treated as a scientific declaration.

ARC Prize itself is explicit: saturating ARC-AGI-3 is not proof of AGI. Its environments are bounded, deterministic and closed-ended. The benchmark is designed to measure important aspects of generalization and agentic intelligence, not the full open-ended complexity of the real world.

That does not make the benchmark weak. It makes the interpretation disciplined.

The right conclusion is neither:

“Astra is just marketing.”

nor:

“AGI is solved.”

A better conclusion is:

Astra is evidence that frontier language models are becoming substantially more capable general-purpose digital agents.

That transition matters economically even if we never agree on the exact day the word “AGI” should be used.

If an agent can operate software, browse, debug, analyze data, build a useful model of an unfamiliar environment, use tools, maintain state and complete long professional workflows, it can change how knowledge work is organized long before philosophers settle a definition.

The practical question for builders should therefore be:

What can we now delegate reliably and economically that we could not delegate six months ago?

That is measurable.

4. The independent benchmark reality check

Every frontier-model launch should be evaluated through at least two lenses:

What does the provider report under its own evaluation setup?
What happens when independent evaluators run models under shared methodologies?

OpenAI’s Astra launch table is impressive. But it is still a vendor launch table.

Artificial Analysis adds an important reality check. Its current Intelligence Index places Astra around 61. That is frontier-level performance, but it does not show a universal step-function jump over every other frontier model. Some competing models remain ahead on that aggregate.

Artificial Analysis’s more interesting Astra finding is in its Coding Agent Index. Astra reaches approximately 67, around the leading frontier band, while showing meaningful token-efficiency gains in some configurations.

This gives us a more nuanced picture:

Astra is not two times more generally intelligent than every competitor.
Astra does not win every benchmark.
Astra looks especially strong where intelligence has to be converted into multi-step action.
Efficiency per successful task may be as important as raw score.

This distinction matters because agents operate under budgets.

A model that is five percent better but three times more expensive may be the wrong default worker.

A model that is slightly better while using half the tokens may materially change a long-running workflow.

Once agents run for hours, token efficiency becomes a capability.

So do latency, recovery, context discipline and tool reliability.

5. Where Astra genuinely looks exceptional

The Astra story becomes strongest when we look at tasks requiring action.

Computer use

OpenAI reports 72.6% on OSWorld 2.0 offline, compared with 65.7% for GPT-5.6 Sol and a reproduced 70.2% for Claude Opus 5 in the comparison shown.

OpenAI also reports Astra completing its OSWorld latency simulations in substantially less time than Sol.

Computer use matters because modern knowledge work lives inside interfaces: browsers, spreadsheets, CRMs, IDEs, terminals, CAD tools, data-science environments, ticketing systems, cloud consoles and internal enterprise applications.

A model can be brilliant at text and still be a weak worker if it cannot reliably operate the software where work actually happens.

Professional automation

Astra scores 41.4% on AutomationBench in OpenAI’s table, compared with 18.1% for Sol and 26.9% for Opus 5.

The absolute score is important: 41.4% is not “solved.” There is still enormous headroom.

But the relative jump signals meaningful progress in heterogeneous workflows.

Scientific terminal work

On Terminal-Bench Science 0.1, OpenAI reports Astra at 64.6%, versus 22.4% for Sol, 52.6% for Claude Fable 5.1 and 30.0% for Opus 5.

This is one of the clearest generational improvements in the launch table.

Scientific workflows require more than factual recall. They require code execution, data handling, iterative analysis, simulation, model fitting and interpretation. That is exactly the kind of multi-step environment in which an agent’s ability to maintain a coherent plan matters.

Terminal engineering

On Terminal-Bench 4.0, Astra reaches 57.9%, compared with 37.3% for Sol and 52.6% for Opus 5.

Again, the story is not “everything else is obsolete.” The gap to other frontier systems is meaningful but not absolute.

Advanced mathematics

OpenAI reports 97.6% on FrontierMath Tier 4 v2.

That is an extraordinary academic result, but it should sit beside other benchmarks rather than become a universal intelligence proxy.

6. Where Astra does not justify a universal-superiority narrative

A credible model analysis should include losses and narrow gaps.

BrowseComp

Astra: 91.5%.

Sol: 90.4%.

Opus 5: 90.8%.

This is effectively a frontier cluster. Astra is not creating a new universe of capability here.

GPQA Diamond

Astra: 96.0%.

Sol: 94.6%.

Opus 5: 93.7%.

Again, excellent, but the gap is modest because the frontier is already near saturation.

Humanity’s Last Exam with tools

OpenAI’s table reports Astra at 57.2%, while Claude Fable 5.1 is 65.0% and Opus 5 is 63.6%.

Astra loses.

That one row is editorially valuable because it destroys the lazy narrative that a new generation number means universal dominance.

Strong models have capability profiles.

The future is likely to be heterogeneous.

7. The model stack should be treated as an economic hierarchy

The biggest operational mistake after a frontier-model release is making the newest model the default for everything.

If you need to decide whether a cross-system architecture is safe, Astra may be worth its cost.

If you need to rename twenty files, Astra is absurd.

The right question is not:

Which model is best?

The right question is:

What is the least expensive model that reliably clears the quality bar for this task?

For OpenAI’s current stack, a useful mental model is:

Astra thinks about the system. Sol reasons about the problem. Terra builds the solution. Luna does the chores.

GPT-6 Astra — supervisor

Use Astra for high-leverage uncertainty:

architecture
difficult end-to-end workflows
high-impact decisions
complex cross-system debugging
long-horizon autonomous work
advanced computer-use tasks
final adversarial review
scientific or professional tasks where failure is expensive

GPT-5.6 Sol — reasoner

Use Sol for:

deep analysis
research synthesis
hard debugging
technical design
critique
mathematical reasoning
complex narrative structure
evaluating trade-offs

GPT-5.6 Terra — builder

Use Terra for:

implementation
refactoring
routine coding
tests
documentation
migrations
structured transformations
medium-complexity engineering

GPT-5.6 Luna — volume

Use Luna for:

classification
extraction
formatting
repetitive code edits
boilerplate
file triage
batch transformations
low-risk mechanical work

This is not a moral hierarchy and it is not a permanent benchmark ranking.

It is a routing policy.

8. API economics explain why routing matters

OpenAI’s current API pages list the following approximate token prices:

Model	Input / 1M	Cached input / 1M	Output / 1M
GPT-6 Astra	$10	$1	$50
GPT-5.6 Sol	$4	$0.40	$20
GPT-5.6 Terra	$2	$0.20	$12
GPT-5.6 Luna	$0.20	$0.02	$1.20

You do not need to be an API customer for this table to teach you something.

It shows the economic shape of the model family.

Astra is not designed as a bulk worker.

Luna is not designed as your supreme architect.

Terra exists because most production work benefits from a balance of capability and cost.

Sol exists because reasoning depth still has a premium.

Model routing is therefore not a clever optimization. It is the basic way to avoid converting a premium subscription into a three-day quota burn.

9. A million-token context window is capacity, not a target

Astra, Sol, Terra and Luna have model pages listing a 1.05M-token context window.

That number is impressive. It is also dangerously easy to misunderstand.

A large context window means the model can process a very large working set.

It does not mean every session should grow until it is close to one million tokens.

Imagine an engineering session that has accumulated:

architecture discussion
full repo scans
terminal logs
five failed patches
two abandoned strategies
complete test output
repeated explanations
generated documentation
copied tickets
design screenshots
old debugging hypotheses

After enough turns, the transcript stops behaving like memory and starts behaving like a landfill.

Relevant facts are still inside it—but they have to compete with everything else.

The more durable pattern is to periodically create canonical state artifacts:

ARCHITECTURE.md
DECISIONS.md
CURRENT_STATE.md
KNOWN_FAILURES.md
TEST_STATUS.md
NEXT_STEPS.md

Then start the next execution phase from those artifacts and the current repository state.

That is not “losing context.”

It is engineering context.

OpenAI’s Sol/Terra/Luna API pages also state that prompts above 272K input tokens have higher long-context pricing in the API. Even when your subscription metering is not identical to API billing, the principle is obvious: large context is expensive infrastructure.

Use it when it creates value.

10. The ARC result teaches the opposite of “keep everything forever”

The Provider Adapter’s success does not mean infinite raw transcript is ideal.

It uses state continuity and compaction.

That is a critical distinction.

Raw conversation history is not the same thing as useful memory.

Suppose a senior engineer joins a project after six months. Would you give them every Slack message, terminal output and failed patch and call that “context”? Or would you give them:

the architecture
current code
decisions
constraints
unresolved risks
recent incident history
next milestones

Of course you would compact.

Agents need the same discipline.

The objective is not to remember everything.

The objective is to preserve what changes future decisions.

11. For a developer: use Codex as the execution plane

Chat is excellent for discussion, research and strategic reasoning.

Software engineering needs another layer: a runtime that can inspect repositories, execute commands, run tests, modify files, observe failures and iterate against evidence.

That is where Codex belongs.

A clean division is:

CHAT
strategy / research / architecture
   ↓
CODEX
repo / terminal / implementation / tests
   ↓
EVIDENCE
   ↓
CHAT or ASTRA
review / decision if the change is high-impact

The mistake is starting every Codex task with your most expensive model and maximum reasoning.

A better escalation path is:

routine implementation → Terra
hard implementation → Terra High
reasoning bottleneck → Sol
system-level ambiguity → Astra
mechanical batch work → Luna

The exact UI options will evolve. The principle survives product changes.

12. Astra should be your supervisor, not your typist

Consider a major software release.

The naïve workflow is:

Astra writes architecture
Astra implements
Astra writes tests
Astra debugs
Astra reviews itself
Astra ships

That wastes frontier capacity and creates a weak review structure.

A stronger pattern is:

Astra
architecture + risk model
   ↓
Terra
implementation
   ↓
Tests / runtime evidence
   ↓
Sol
adversarial review
   ↓
Terra
corrections
   ↓
Astra
release-level judgment, only if justified

Astra spends cognition where judgment has leverage.

Terra spends capacity where implementation volume matters.

Sol creates an independent reasoning checkpoint.

Luna can scan, classify and process volume around the edges.

This architecture works for one developer or a large team.

13. For a researcher: separate hypothesis generation from verification

A research workflow should not ask one frontier model to generate a theory, write its proof, run experiments and certify that its own work is correct.

Use role separation.

Example:

Astra → research question / conceptual holes
Sol → literature reasoning / competing hypotheses
Terra or Codex → experiments / implementation
Luna → extraction / classification / bookkeeping
Sol → analyze results
Astra → hostile reviewer / novelty check

The value is not only cost.

Role separation reduces correlated self-confirmation.

A critic should have a different prompt, different context and preferably a different execution path from the author.

14. For a content creator: spend frontier intelligence on thesis, not commas

Content creators will also waste Astra if they use it as an expensive copywriter.

Astra’s highest-value question is not necessarily:

“Write me a 2,000-word script.”

It may be:

What is the non-obvious thesis?

Where will a smart viewer stop watching?

Which claim is most attackable?

What evidence changes the story?

Then Sol can build the narrative, Terra can build production manifests and Luna can process metadata, captions, alternate hooks and repetitive assets.

Example:

Research → Sol
Thesis attack → Astra
Script → Sol
Asset manifest → Terra
Metadata variants → Luna
Final editorial challenge → Astra only if the video is strategically important

This produces more content per unit of frontier capacity without lowering quality where it matters.

15. For a founder or CEO: use Astra to reduce uncertainty

A founder should not spend frontier-model messages asking for generic motivational summaries.

The economic value of Astra is reducing uncertainty around decisions with meaningful downside or upside.

Examples:

Should we enter this market?
Which architecture creates a platform rather than a feature?
What assumption in our product strategy is most likely wrong?
Which regulatory or security dependency could block deployment?
What does a hostile competitor do next?
What would make this investment thesis fail?

A useful architecture is:

Founder
   ↓
Astra — challenge assumptions
   ↓
Sol — gather/analyze evidence
   ↓
Terra/Work — build artifacts and execute
   ↓
Evidence
   ↓
Astra — decision synthesis if stakes justify it

The model is no longer a chatbot.

It becomes part of a decision system.

16. ChatGPT Pro $100: understand the Chat bucket correctly

As of the current OpenAI Help Center snapshot, GPT-6 Pro in Chat is powered by Astra.

For Pro $100, GPT-6 Pro and GPT-5.6 Sol Pro share one 50-message weekly allowance in Chat.

That detail changes how you should behave.

If you care about preserving Astra access, do not casually use Sol Pro for tasks that ordinary Sol Medium/High can solve. Sol Pro and Astra draw from the same Pro-model weekly bucket on the $100 tier.

The ordinary reasoning choices are different. OpenAI says manually selecting Medium, High or Extra High uses GPT-5.6 Sol. Those are not identical to the separate Sol Pro model option.

A practical Chat policy is therefore:

Instant / automatic reasoning → default conversation
Sol Medium/High → serious normal reasoning
Astra / GPT-6 Pro → scarce, high-impact work
Sol Pro → use only when there is a concrete reason to spend from the shared Pro-model bucket

Fifty weekly frontier messages can be a lot if each one resolves a high-leverage decision.

They are almost nothing if you use them for rewrites and basic explanations.

17. Work and Codex are a different allowance domain

OpenAI states that Chat and Work/Codex have separate usage structures.

Astra in Work and Codex uses the plan’s included agentic allowance as rollout reaches the account. Pro $100 and Pro $200 users can use their full existing Work/Codex allowance with Astra; Plus receives limited Astra use in Work/Codex during rollout.

The key point is that Work and Codex are not another Chat bucket measured in simple messages.

Usage depends on the model, task size, input/output, reasoning settings and speed mode.

This is why a developer can feel like “I only sent a few prompts” and still burn a large fraction of the weekly agentic allowance.

An agent prompt is not one unit of work.

It can trigger a long execution trajectory.

The right metric is not message count.

The right metric is completed-work cost.

18. Measure your personal Codex economics

OpenAI cannot give one useful global answer to “how many coding tasks do I get?” because tasks vary by orders of magnitude.

You can create a much more useful measurement yourself.

Before a representative task, record the usage meter.

Example:

Weekly capacity before: 83%
Weekly capacity after: 80%
Task cost: 3 percentage points

Repeat across several comparable tasks.

Suppose five medium engineering tasks cost:

2.1%
2.8%
2.4%
2.6%
2.3%

Median cost: about 2.4%.

If you have 60% remaining, your rough capacity is:

60 / 2.4 ≈ 25 comparable tasks

This is far more useful than counting prompts.

Now you can compare models too.

Run the same task class with Terra and Sol.

If Terra costs half the allowance and succeeds with the same human review time, Terra should become the default.

If Sol costs more but prevents two hours of rework, Sol is cheaper in outcome terms.

The objective is cost per accepted result, not cost per token and not prestige per model name.

19. Manage a weekly agentic allowance like a budget

If your allowance routinely dies on Day 2 or Day 3, stop treating it as an invisible platform limit.

Treat the weekly meter as 100 budget units.

An example sustainable operating policy:

Cycle day	Target cumulative spend	Desired remaining
Day 1	12%	88%
Day 2	24%	76%
Day 3	37%	63%
Day 4	50%	50%
Day 5	63%	37%
Day 6	75%	25%
Day 7	85%	15% reserve

This is not an OpenAI entitlement table. It is an operating discipline.

If you reach 50% spent by Day 3, investigate:

oversized context
unnecessary high reasoning
Sol/Astra used for implementation volume
long cloud agent trajectories
repeated repo scans
too many enabled tools/MCP servers
verbose outputs
one giant session doing unrelated jobs

The purpose of the reserve is not to leave paid capacity unused.

The purpose is to avoid becoming powerless when a genuinely difficult problem appears late in the cycle.

20. Plus vs Pro: the rollout is surface-specific

This is one of the easiest things to publish incorrectly because OpenAI’s rollout language is changing quickly.

OpenAI’s broad launch announcement says Astra will become available to Plus, Pro, Business and Enterprise users over the rollout.

The current product-specific Help Center is more precise about surfaces:

GPT-6 Pro/Astra in ordinary Chat is listed as rolling out for Pro $100, Pro $200, Business and Enterprise.
Plus is documented as receiving Astra in Work and Codex as rollout reaches the account, with limited usage.
Pro is documented as receiving Astra in Chat, Work and Codex as rollout reaches the account.

Therefore do not write:

“Plus will never get Astra.”

And do not write:

“Every Plus user can select GPT-6 Pro in Chat today.”

The safe statement is:

Astra is part of the broader Plus rollout, but current access is surface- and rollout-dependent. OpenAI’s Help Center currently lists GPT-6 Pro in Chat for Pro/Business/Enterprise while documenting limited Astra access for Plus in Work and Codex.

Recheck the Help Center on publication day.

21. Hermes does not replace Codex—and that is the wrong comparison anyway

Hermes Agent is interesting because it introduces another orchestration surface.

Its official documentation describes an optional Codex app-server runtime. When enabled, eligible OpenAI/Codex turns can run through Codex’s runtime, including terminal operations, structured edits, sandboxing and MCP tooling, while Hermes remains the outer shell for sessions and other orchestration behavior.

That creates a more useful architecture than “Hermes versus Codex.”

Think:

Hermes
orchestration / scheduling / routing
   ↓
Codex app-server
engineering runtime
   ↓
OpenAI model selected for the task
   ↓
repo / terminal / tests / tools

Hermes documentation also says ChatGPT subscription authentication can be used through its openai-codex path.

This is useful.

But do not turn it into a quota loophole story.

The documentation notes that auxiliary tasks can also flow through the ChatGPT subscription when the Codex runtime/provider is used. The right assumption is that the work is metered according to the underlying authenticated runtime—not that Hermes magically creates free extra OpenAI compute.

Use Hermes for orchestration value, not for an unsupported “double your quota” claim.

22. Hermes + Codex + multiple models: where parallelization becomes interesting

The real advantage of an orchestrator is not merely switching the same prompt between models.

It is decomposition.

Most people still use AI serially:

ask → wait → read → ask → wait → read

A multi-agent system can split independent work:

                    Supervisor
                       Astra
                         │
       ┌─────────────────┼─────────────────┐
       ▼                 ▼                 ▼
    Research          Coding           Content
      Sol              Terra              Sol
       │                 │                 │
       ▼                 ▼                 ▼
  evidence          tests/repo         assets
       └─────────────────┼─────────────────┘
                         ▼
                      review

Hermes can provide scheduling, delegation and routing primitives around that pattern. Codex can remain the specialized engineering runtime.

This is where heterogeneous models become an advantage rather than a nuisance.

Astra does not need to write every line.

It can supervise the structure of the work.

23. Parallelization without governance is just faster failure

More agents are not automatically better.

If five agents can modify the same system without clear boundaries, you can create five times the collision surface.

Parallel work needs:

isolated workspaces or worktrees
explicit ownership of files/tasks
merge/review gates
shared invariants
cancellation rules
budget limits
timeout/stall detection
durable run logs
human approval for consequential actions

This becomes more important as models become better at acting.

A weak assistant that writes a bad paragraph is annoying.

A strong agent that confidently changes infrastructure is a security and reliability problem unless the system constrains it.

24. Astra’s cyber capability makes permission architecture non-optional

OpenAI says Astra is the first model it has broadly deployed to reach the Critical cybersecurity capability threshold under its Preparedness Framework.

That should change how sophisticated users think about agent permissions.

The response is not panic.

It is engineering.

Use:

least privilege
sandboxed execution
scoped credentials
explicit target boundaries
approval gates for destructive changes
immutable audit logs
rollback paths
network controls
separation of planning and authorization

The stronger the model, the weaker “the prompt told it not to” becomes as a security control.

A prompt is intent.

A permission boundary is enforcement.

25. The future agent stack needs a control plane, execution plane and memory plane

A useful 2026 architecture looks less like a chatbot and more like a distributed system:

Human / organization
        │
        ▼
Control plane
policy • routing • goals • approvals
        │
        ▼
Supervisor / planner
Astra or another frontier model
        │
  ┌─────┼─────────────┐
  ▼     ▼             ▼
Codex  Hermes      Work/browser
  │      │             │
  ▼      ▼             ▼
models + tools + environments
        │
        ▼
canonical state / memory
        │
        ▼
observability + evidence
        │
        ▼
review / recovery

Notice what this architecture does not assume:

It does not assume one provider wins forever.

It does not assume one model does every job.

It does not treat chat history as memory.

It does not allow every agent unrestricted access.

It treats models as powerful, replaceable compute inside a governed system.

That is a more durable architecture than building your business around whichever model has the best launch-week score.

26. Why memory becomes more important as models improve

A common argument says better models will make memory infrastructure unnecessary.

Astra’s ARC-AGI-3 result points in the opposite direction.

The better the model becomes at using retained state, the more valuable good state becomes.

A weak model with excellent memory is still weak.

A strong model with chaotic state wastes its strength.

A strong model with disciplined context, durable memory, useful tools and bounded execution can become a qualitatively more capable system.

This is why “memory” should not mean “save the conversation.”

A real memory architecture distinguishes:

transient execution context
canonical facts
current decisions
learned patterns
security constraints
user/org preferences
provenance
expiration rules
confidence
access control

Once agents work across days and projects, those distinctions become infrastructure.

27. Compaction is a governance decision, not only a token optimization

When a system compacts context, it decides what survives.

That is more than compression.

Suppose an agent discovered:

one API is deprecated
a customer requirement forbids a certain behavior
a test failure revealed an architectural invariant
a previous remediation made production worse

If compaction drops those facts, future behavior can regress even though the raw model is highly capable.

Therefore good compaction should preserve:

decisions and rationale
constraints and invariants
unresolved risks
evidence links
failure lessons
current objective

And it should discard or summarize:

redundant logs
repeated explanations
dead-end hypotheses
routine tool chatter
stale intermediate text

The goal is not smaller context at any cost.

The goal is high information density for future decisions.

28. Practical routing examples by profession

Deep software engineer

Default: Terra in Codex.
Escalate: Sol for difficult debugging/design.
Use Astra: system architecture, unfamiliar large-system failures, release-level review.
Use Luna: batch file classification, mechanical checks, repetitive transformations.

Security/SRE engineer

Default: Sol or Terra depending on task.
Astra: complex incident synthesis or authorized deep analysis with strict boundaries.
Rule: never equate higher model capability with broader permissions.

Researcher

Default: Sol for intellectual work.
Astra: research framing and hostile review.
Terra/Codex: experiments and analysis pipelines.
Luna: extraction/classification.

Founder

Default: Sol for market/product analysis.
Astra: strategic uncertainty, architecture and consequential decisions.
Work: create finished artifacts.
Codex: product engineering.

Content creator

Default: Sol for research/narrative.
Astra: thesis, fact-risk attack and final editorial challenge.
Terra: asset manifests and production operations.
Luna: metadata, variants and bulk transformations.

Enterprise platform team

Use explicit routing policies. Measure cost per accepted outcome. Preserve a frontier escalation pool rather than making the premium model the default for every employee action.

29. Five prompt patterns that spend Astra well

Architecture attack

Inspect the system as a hostile staff architect. Identify hidden coupling, invalid assumptions, missing invariants, recovery gaps, security boundaries and failure amplification. Do not implement until the risk model is complete.

Causal-debugging escalation

We have attempted multiple fixes and local tests pass, but production behavior remains inconsistent. Build a causal model across components and identify the earliest violated invariant rather than proposing another patch.

Research reviewer

Assume this paper is submitted to a skeptical top-tier venue. Separate novelty claims, theorem validity, experimental evidence and reproducibility. Find the strongest rejection case before proposing improvements.

Product decision

Identify the three assumptions that make this strategy work. For each, define disconfirming evidence, second-order effects and the least expensive experiment that could invalidate it.

Release judgment

Review the final diff, tests, operational evidence and known risks. Decide whether this is safe to release. If not, name the smallest blocking set. Do not generate cosmetic improvements.

These prompts spend frontier reasoning on uncertainty and judgment.

30. Five tasks Astra should almost never do by default

Rewrite a simple email.
Rename files.
Generate basic boilerplate.
Summarize a page you already understand.
Format data that a cheaper model can transform deterministically.

The point is not to ban Astra from simple tasks.

The point is opportunity cost.

A scarce premium message spent on a rewrite cannot be spent later on a production incident.

31. The best benchmark is accepted work per unit of budget

Model leaderboards are useful for choosing candidates.

They are not a substitute for measuring your own workflow.

Build a task matrix:

The percentages below are illustrative placeholders showing how to structure your own measurements; they are not published benchmark results.

Task class	Quality bar	Terra success	Sol success	Astra success	Human review minutes	Weekly allowance cost
routine bug	tests pass	90%	95%	97%	...	...
architecture review	no critical gap	65%	85%	95%	...	...
bulk test triage	correct classification	94%	95%	96%	...	...

Then route based on actual evidence.

You may discover that Terra is the best economic choice for 70% of your engineering work.

You may discover that Sol reduces human review enough to justify higher consumption on a particular class.

You may discover Astra is worth using early on a certain category because a wrong architectural direction is more expensive than the model.

This is how model usage becomes an operating system rather than a habit.

32. Why “use the smartest model for everything” will age badly

The model market is moving too quickly for monoculture.

One month a provider leads coding.

Another leads computer use.

Another leads long context.

Another dominates cost-sensitive batch inference.

The durable system does not encode “Model X is always best.”

It encodes capabilities and thresholds.

For example:

if task.mechanical and risk.low:
    choose lowest-cost qualified model
elif task.implementation and architecture_known:
    choose balanced builder
elif task.reasoning_depth_high:
    choose deep reasoner
elif task.high_impact and ambiguity_high:
    choose frontier supervisor

That is how cloud infrastructure evolved.

We do not run every workload on the largest possible machine.

We route workloads to appropriate resources.

AI is moving the same way.

33. The AGI argument matters less than the delegation curve

Imagine two futures.

In Future A, everyone agrees Astra is “not AGI,” but it can reliably complete eight hours of professional software work with bounded supervision.

In Future B, everyone agrees on a formal definition and calls a model “AGI,” but it still requires constant correction in real tools.

Which future changes a company first?

The answer is obvious.

Economic transformation follows reliable delegation, not terminology.

The useful metric is the delegation curve:

How long can the agent work before human intervention?
How often does it violate scope?
How much rework does it create?
How much state can it retain correctly?
How often can it recover from failure?
What is the cost per accepted outcome?
What actions can safely be authorized?

Astra moves several of those variables in the right direction.

That is already significant.

34. What the Qualixar position should be

The internet will produce two kinds of Astra content.

One group will scream “AGI.”

Another will reflexively dismiss every provider benchmark as marketing.

The stronger technical position is between them:

Celebrate the real capability jump.
Separate evaluation conditions.
Show independent results.
Explain the system architecture behind the score.
Teach people how to use the model economically.
Treat memory, policy, recovery and observability as first-class infrastructure.

This makes the content useful after launch week.

The benchmark is the news hook.

The operating architecture is the evergreen asset.

35. Final verdict

GPT-6 Astra is not interesting because the version number moved from five to six.

It is interesting because several trends crossed an important threshold together:

stronger computer use
better professional automation
dramatically stronger scientific terminal work
large-context capability
improved action efficiency
better long-horizon execution
strong gains from context/state scaffolding

Those are exactly the properties needed to move from:

AI that answers

toward:

AI that works.

The transition is incomplete.

AutomationBench is not solved.

Independent intelligence evaluations remain competitive.

Astra loses some benchmarks.

Long-running reliability is still an engineering problem.

Security becomes more difficult as capability rises.

Costs remain real.

And a 99.9% score under one harness does not turn a bounded benchmark into a scientific certificate of AGI.

But the direction is clear.

The next generation of AI systems will not be defined only by a model name.

They will be defined by the architecture around the model:

routing, memory, context, tools, permissions, observability, recovery and evidence.

That is why the most useful way to remember today’s OpenAI stack is not a leaderboard.

It is a division of labor:

Astra thinks about the system.
Sol reasons about the problem.
Terra builds the solution.
Luna does the chores.
Codex executes engineering.
Hermes can orchestrate workflows around the execution plane.
You remain responsible for goals, boundaries and judgment.

And perhaps that is the deeper lesson hidden inside Astra’s 99.9% result.

The future of AI is not simply a smarter model.

It is a smarter system around the model.

Practical Appendix A — a $100 Pro operating policy

If you are on the Pro $100 tier, the current ChatGPT Help Center documents 50 GPT-6 Pro messages per week in Chat, shared with GPT-5.6 Sol Pro. Treat those fifty as an executive attention budget.

A practical weekly allocation might look like:

10–15 messages: architecture and consequential decisions
10 messages: difficult research / adversarial review
5–10 messages: complex debugging escalations
5 messages: final review of high-value deliverables
keep the remainder unallocated until late in the week

Do not mechanically force yourself to spend exactly fifty. The objective is value, not consumption.

For Work/Codex, measure your own burn rate in percentage points per accepted task. There is no useful universal “daily task count.”

Practical Appendix B — context hygiene checklist

Before continuing a giant agent session, ask:

Does the task still need the old debugging history?
Are repeated logs still useful?
Have architecture decisions been written to a durable file?
Do failed hypotheses remain mixed with current facts?
Can this work be split into a new task with a compact handoff?
Are unnecessary tools or MCP servers contributing context?
Is the model rereading large files that could be summarized once?

If three answers are uncomfortable, compact and restart.

Practical Appendix C — publication source notes

The most important publication-day sources are:

Do not publish a screenshot as a permanent entitlement claim when a live Help Center exists.

Career Impact: The Risk Is Staying at the Layer Astra Is Learning to Execute

The most useful question after a frontier-model launch is not whether one benchmark proves AGI. It is what category of work just became cheaper, faster, or more automatable.

Astra's significance is that the frontier is moving from answer generation toward action completion. Coding, browsing, computer use, planning, context retention, tool use and long-horizon execution are increasingly part of the same system. That changes the economic value of different layers of knowledge work.

The weakest career strategy is to compete with a frontier model at the layer where it has the largest structural advantage: high-volume digital execution. If your role is defined only as receive a clearly specified task and produce a predictable digital output, then increasingly capable agents are entering that layer directly.

The stronger strategy is to move upward in the value stack:

Produce output
    ↑
Execute task
    ↑
Use AI tool
    ↑
Orchestrate agents
    ↑
Verify / evaluate
    ↑
Design the system
    ↑
Define the problem
    ↑
Make domain decisions

This is not a prediction that entire professions disappear. Jobs are bundles of tasks, responsibilities, relationships, tacit knowledge, accountability and judgment. The more defensible conclusion is that task composition changes. Execution-heavy portions become cheaper; problem formulation, verification, architecture, coordination and accountable decision-making become relatively more valuable.

For developers, that means architecture, test strategy, system boundaries, security, production diagnosis and agent supervision matter more—not less. For researchers, methodology and interpretation matter more as literature search, coding and experiment execution accelerate. For creators, point of view, taste, evidence and narrative judgment become the scarce layer as raw copy generation becomes abundant. For founders, the advantage shifts from prompting skill toward decision design, delegation architecture, evidence quality and operational judgment.

The memorable rule is:

Do not try to become faster than Astra. Learn how to direct Astra—and move your career toward the layers the model cannot safely own by itself.

This is also why model routing matters. The right future workflow is not one human competing with one giant model. It is a human defining goals and boundaries while heterogeneous models perform different roles:

Astra thinks about the system. Sol reasons about the problem. Terra builds the solution. Luna does the chores.

The human remains responsible for objectives, authorization, verification and consequential judgment.

Written by Varun Pratap Bhardwaj, founder of Qualixar and an independent AI Reliability Engineering researcher. Follow @varunPbhardwaj.

The Cheap Model Is Not Cheap Until It Finishes the Trace: Gemini 3.8 & Muse Spark 1.3

varun pratap Bhardwaj — Thu, 03 Sep 2026 16:32:11 +0000

Editorial fact policy. Every number below is tagged as either [VENDOR] or [INDEPENDENT]. Vendor numbers are useful evidence, not verdicts. [ABSTAIN] means the source record was not good enough to publish a number.

A $0.75 model just made the wrong question look obsolete

The loud version of this week’s news is simple: a cheaper model is matching models that cost far more.

The useful version is harder.

Google released Gemini 3.8 Flash on September 2. Meta released Muse Spark 1.3 on the same day. Both releases make a credible case that routine agentic work no longer requires the most expensive model on every turn. But neither release proves that an engineering team should cancel its premium subscription, route every task to a single provider, or declare a winner from one benchmark chart.

That would repeat the old mistake: treating a score as evidence and an answer as proof.

The decision is not which model is best? It is: which model can finish this particular trace at the lowest total cost, with a fallback when it cannot?

A trace is the work that actually matters: inspect the repository, retrieve the source, call the tool, generate the artifact, run the test, recover from failure, and leave a result that somebody else can verify. Token price is only one component. Retries, reasoning volume, tool calls, output limits, access restrictions, review time, and a quality escape are all part of the bill.

That is the model portfolio problem.

What changed this week

Gemini 3.8 Flash: lower token price, higher work rate

Google says Gemini 3.8 Flash is its current workhorse for software engineering, agentic tasks, and complex knowledge workflows. It is available through the Gemini API, AI Studio, Android Studio, Google Antigravity, Gemini Enterprise, and selected consumer Google surfaces. [VENDOR] Google lists an introductory API price of $0.75 per million input tokens and $3.75 per million output tokens through December 31, 2026; Google says the price becomes $1.50/$7.50 on January 1, 2027. It has a 1M-token input context window and a 64K-token maximum output. [S1][S2]

The price headline needs a warning label. Google’s model card explicitly says 3.8 can use more tokens at higher effort to improve performance. Independent analysis found that 3.8 Flash at high reasoning used roughly 30% more output tokens per evaluated task than 3.7 Flash and cost about 40% more per evaluated task, despite the same per-token launch price. [INDEPENDENT] [S3]

That is not a defect. It is the trade: 3.8 is cheaper at the meter than premium models, but it is not automatically cheaper than 3.7 on every agent loop.

Muse Spark 1.3: a serious coding lane, not a “free” lane

Meta says Muse Spark 1.3 improves long-form instruction following, coding usability, and calibration about when it is stuck. In Meta’s internal engineering comparisons, it used roughly 20% fewer tool calls and 25% fewer tokens than Muse Spark 1.2. That is a vendor claim from internal comparisons, not a third-party universal result. [VENDOR] [S4]

The independent signal is stronger than the marketing wording. Artificial Analysis reports Muse Spark 1.3 xhigh at 61 on its Intelligence Index and $0.55 per evaluated task, with a 1M-token context window. Its limited-preview max variant reaches 62, but Meta had not announced a public price for that limited release; it should not be used in a cost comparison. [INDEPENDENT] [S5]

One critical distinction: Meta’s documented contributor tier is a different data contract. The listed contributor model permits use of prompts and completions to improve Meta products and has different rate limits and token prices. The public official pricing page still names muse-spark-1.2-contributor, not a 1.3 contributor SKU. [ABSTAIN] Do not put “Muse Spark 1.3 contributor” into a production workload or a public price chart until Meta publishes that exact SKU and its terms. [S6]

The comparison that survives contact with the source notes

The matrix above uses one independent evaluator, Artificial Analysis, where possible. It does not claim that 61 at one effort setting equals 61 at another in every workflow. It gives a common reference point and preserves the effort label.

Lane	Evidence that can be stated	What it does not prove
Gemini 3.8 Flash, high	59 Intelligence Index and $0.58 per evaluated task. Google’s introductory API price is $0.75/$3.75 per million input/output tokens through Dec. 31. [S2][S3]	That 3.8 is cheaper than 3.7 on every workload, or that its official benchmark table is an apples-to-apples external test.
Muse Spark 1.3, xhigh	61 Intelligence Index and $0.55 per evaluated task; 1M context. [S5]	That the limited-preview max mode is publicly available or that contributor-tier pricing applies to 1.3.
Composer 2.5, Cursor harness	62 on the separate Artificial Analysis Coding Agent Index; $0.07 per evaluated task for standard and $0.44 for Fast. [S13]	That this coding-agent score transfers to the Grok Build harness, or that it is comparable to the Intelligence Index rows above.
Grok 4.6, high	61 Intelligence Index, 500K context, $2/$6 API pricing, and $0.94 per evaluated task. [S14]	That a consumer or Grok Build bundle exposes the same API model, quota, tools, or pricing.
GPT-5.6 Sol, max	61 Intelligence Index and $0.95 per evaluated task in the same comparison. Standard short-context API price is $4/$20 per million input/output tokens, with a higher long-context rate above 272K input tokens. [S5][S7]	That its maximum effort is the default bill, or that a subscription message quota maps to API spend.
Claude Opus 5, high / max	61 at high and 63 at max; $1.23 / $2.34 per evaluated task. Official API price starts at $5/$25 per million input/output tokens; it has 1M context and 128K output. [S8][S9]	That an expensive task is wasteful. It may be the lowest-total-cost route when it avoids a failed long-horizon trace.
GLM-5.3 Flash	57 Intelligence Index, $0.09 per evaluated task, 1M context, and $0.15/$0.50 per million input/output tokens in the cited analysis. The same analysis calls it slower and more verbose than comparable models. [S10]	That a cheap API lane is automatically the best interactive agent lane. Provider latency and trace length matter.

The pattern is clear: a portfolio has become defensible. What is not defensible is declaring one of these numbers a universal crown.

Google’s own table is useful because it shows where it loses

Google’s official 3.8 evaluation material is worth reading, not copying into a victory lap.

[VENDOR / mixed-source table] Google reports Gemini 3.8 Flash near the top of several bounded professional and coding tasks. It also publishes methodology notes that should stop anyone from flattening the table into “Gemini beat Opus.” Google says some Gemini results are self-computed, some competitor results are providers’ self-reported numbers, some values come from public leaderboards, and different evaluation families use different harnesses. [S11]

Google also documents a concrete multimodal asymmetry: its LVBench comparison uses 1,024 frames for Gemini and GPT-5.6 models, but 300 frames for Claude models because of API limitations. That does not invalidate the result. It does invalidate a lazy claim that the rows are mechanically identical. [S11]

That is why we preserve the original official exhibit rather than extracting a few green cells and calling it a verdict.

Source exhibit: Google’s official model evaluation

The source PDF and its methodology are preserved in this research package. The blog should link to the live official page, not to a local copy. [S11]

Qwen3.8 and GLM-5.3 Flash: separate the open model from the hosted service

Qwen3.8 is not one commercial object. The official Qwen release includes open model weights such as Qwen3.8-2.4T-A95B, while the hosted Qwen3.8 Max service adds features such as vision, non-thinking mode, built-in tools, and a 1M default context. The open 2.4T-A95B model card lists 2.4T total parameters, 95B active parameters, 262K native context, and extension to roughly 1.01M tokens. [VENDOR] [S12]

Independent analysis reports Qwen3.8 Max at 58 on the Intelligence Index with a 1M-token context. [INDEPENDENT] We deliberately do not publish its throughput number: current source records disagree on which Qwen3.8 variant and provider the speed measurement describes. [S15]

Qwen’s vendor benchmark table compares Qwen3.8-Max with GPT-5.6 Sol, Fable 5, and Opus 4.8 under stated harnesses. For example, the vendor table lists Terminal-Bench 2.1 at 86.6 for Qwen3.8-Max and 88.8 for GPT-5.6 Sol max. [VENDOR] [S12]

That is useful evidence, not a subscription comparison. A self-hosted or third-party-hosted Qwen route brings hardware, provider, latency, context configuration, and operations into the bill. Use the open Qwen3.8 release where sovereignty or local control is the point; use Qwen3.8 Max only with its own hosted-service cost and access terms.

GLM-5.3 Flash is a different low-cost lane. Its official docs list a 1M-token context, 128K maximum output, native multimodal inputs, and model ID glm-5.3-flash. Independent analysis places it at 57 on the Intelligence Index and describes it as lower cost but slower and more verbose than the fastest frontier routes. [S10][S16] For batch transforms, first-pass code navigation, or rerunnable non-final artifacts, that can be the right deal. For a real-time agent loop where time-to-correct matters, a cheap output token can still be expensive.

Composer 2.5: the missing coding-agent lane

Composer 2.5 must be treated as a coding-agent result, not added to the general Intelligence Index chart. Artificial Analysis reports a Coding Agent Index of 62 for Cursor Composer 2.5, with $0.07 per evaluated task for standard and $0.44 for Fast. Cursor lists standard token pricing at $0.50/$2.50 per million input/output tokens and Fast at $3/$15. [S13]

Composer 2.5 is also exposed through Grok Build under xAI’s own product surface. That does not make Cursor’s agent benchmark a Grok Build benchmark. Agent score includes the harness: tools, prompts, task environment, permissions, retry policy, and human interaction loop. Benchmark the surface you will actually use. [S17]

Subscription value is an access question, not a benchmark question

Antigravity changes the economics because it is an agent-first Google surface that officially exposes Gemini 3.8 Flash. [S1] But no external article should pretend that a regional monthly plan grants a fixed, universal, unlimited amount of model use, music generation, Flow video generation, or API capacity. Entitlements vary by account, plan, region, feature, and current policy.

The same applies to any Grok, Composer, Codex, or Claude bundle. A chat or IDE subscription is not a transparent equivalent of API pricing. It may be excellent value. It may have stricter weekly quota limits than a pay-as-you-go route. Both can be true.

So the practical rule is:

Treat subscriptions as access surfaces. Treat APIs as metered infrastructure. Do not put them on one price axis without measuring your actual weekly trace volume and quota behavior.

Hermes × OmniRoute × OpenRouter: the operating stack, not another model claim

A serious model portfolio needs two routing surfaces because subscription-backed capacity and pay-as-you-go capacity are different economic contracts.

OmniRoute is the local gateway capable of exposing approved endpoint and subscription routes. In the current stack, we use it specifically for the Claude Code and Antigravity subscription surfaces because Hermes cannot directly authenticate to those consumer subscription models. OpenRouter is the separate external multi-provider API layer for pay-as-you-go model access, provider choice, cost controls, and model fallbacks.

OpenRouter is valuable because it can make multi-model operation practical through one API surface. Its documented routing controls include provider ordering, price/throughput/latency sorting, data-collection restrictions, maximum provider price, and fallbacks. Model fallbacks activate when the primary route fails operationally—for example, a rate limit, downtime, a context validation error, or a moderation refusal. They do not prove that the fallback answer is good. [S18]

Hermes is the execution and policy layer around both routes. Its provider-routing configuration passes explicit preferences to OpenRouter, while its fallback-provider chain can recover from broader provider failures. Hermes also exposes an experimental openrouter/pareto-code route that targets the cheapest model meeting a coding-quality bar; the chosen model can change as the underlying Pareto frontier changes. [S19]

The reliable pipeline is therefore not “send everything to the cheapest model.” It is:

Classify the trace in Hermes. Set the task type, data boundary, allowed tools, budget, acceptance test, and escalation rule before selecting a model.
Choose the capacity surface. Use OmniRoute when the approved subscription route is the correct fit; use OpenRouter when you need metered multi-provider routing and a clear provider policy.
Route explicitly. Apply provider privacy requirements, required parameters, a cost or latency policy, and a bounded list of fallback models.
Record the route actually served. Capture the actual model, provider, service tier, token use, and cost—not only the intended label.
Execute with an external gate. Hermes runs the tools; a test, source check, render, contract validator, or human gate signs off on the result.
Escalate on a failed gate, not a more confident sentence. A fallback on 429 is availability recovery. A fallback after a failed acceptance test is quality recovery. They are different policies.

A sane OpenRouter policy makes the trade explicit. For cost-sensitive batch work, provider sorting can favor price with a maximum price ceiling. For interactive work, it can favor throughput or latency. For sensitive work, it can deny providers that allow data collection or require zero-data-retention endpoints. These are operating policies, not claims that a router makes a weak model reliable. [S18][S19]

The architecture that matters: cheap default, premium recovery, independent proof

A sensible engineering portfolio has four lanes.

Lane 1 — high-throughput default. Use Gemini 3.8 Flash or Muse Spark 1.3 xhigh when the task is well-bounded, the evaluator is clear, and the result can be independently checked. Choose Gemini when the Antigravity or Gemini API surface is already your active development environment. Choose Muse when the work is code- and tool-heavy and the Meta API contract fits the data boundary.

Lane 2 — coding-agent harness. Use Composer 2.5 where Cursor or Grok Build is the actual surface you will execute in. Treat its strong coding-agent results as evidence for that tested harness, not as a portable API benchmark. Test the exact environment you intend to buy or renew.

Lane 3 — low-cost / open-weight work. Use GLM-5.3 Flash or the open Qwen3.8 release for first-pass research structure, batch transforms, local-control experiments, and repeatable non-final artifacts. Measure latency, verbosity, provider behavior, and infrastructure cost. “Open” does not remove the cost of operating it.

Lane 4 — premium recovery. Escalate to Opus 5, GPT-5.6 Sol, or Grok 4.6 only when the cheaper route fails a pre-declared gate: a test fails, a critical source cannot be grounded, the coding trace stalls, the artifact needs a second pass, or the task has an irreversible consequence. Premium models are not the default. They are not a shameful fallback either.

Lane 5 — independent assay. This is the part most model-comparison posts omit. The model does not certify itself. A test, replay, source check, render, contract validator, or human reviewer does. Otherwise the expensive model simply gives you a more convincing unverified answer.

This is the core AI Reliability Engineering point. The question is not “which chatbot impressed us?” It is “which system produced a verifiable result at the lowest total trace cost?”

How to decide before you cancel anything

Do not choose from a leaderboard. Run ten real traces from your own work.

For each trace, record:

The model, effort level, harness, and access surface.
Completion rate against a fixed acceptance test.
Total tokens, tool calls, wall-clock time, retries, and human cleanup minutes.
Cost of the whole trace—not just input price.
Whether an independent verifier accepted the output.

Then set an escalation rule. For example: start the task in a cheaper lane; escalate only if the test fails, a budget is reached, or the model cannot produce grounded evidence. That is a cost-aware routing pattern. It is reversible, measurable, and less theatrical than canceling a tool because a release chart looked good for one afternoon.

The conclusion

Gemini 3.8 Flash matters because Google is trying to move premium-grade agentic work into a much cheaper operating lane. Muse Spark 1.3 matters because Meta now has a credible coding and long-horizon agent candidate at a compelling independently measured task cost. GLM and Qwen matter because open-weight and low-cost routes are no longer automatically second-class.

Opus 5, GPT-5.6 Sol, and Grok 4.6 still matter because the hardest traces are not priced by their first token. They are priced by whether they finish correctly.

The right answer is not one winner.

It is a model portfolio with a real gate.

Sources

S1 — Google launch and availability: https://deepmind.google/blog/introducing-gemini-3-8-flash-and-38-flash-cyber
S2 — Google model card: https://deepmind.google/models/model-cards/gemini-3-8-flash/
S3 — Independent Gemini analysis: https://artificialanalysis.ai/articles/gemini-3-8-flash
S4 — Meta Muse Spark 1.3 announcement: https://research.meta.ai/blog/introducing-muse-spark-1-3
S5 — Independent Muse Spark 1.3 analysis: https://artificialanalysis.ai/articles/muse-spark-1-3
S6 — Meta standard and contributor pricing / data terms: https://ai.developer.meta.com/docs/pricing-rate-limits
S7 — OpenAI GPT-5.6 Sol pricing: https://developers.openai.com/api/docs/pricing
S8 — Anthropic Opus 5 specifications: https://docs.anthropic.com/en/release-notes/api
S9 — Independent Opus 5 comparison record: https://artificialanalysis.ai/models/releases/claude-opus-5
S10 — Independent GLM-5.3 Flash record: https://artificialanalysis.ai/models/glm-5-3-flash
S11 — Google’s evaluation methodology and official table: https://deepmind.google/models/evals-methodology/gemini-3-8-flash/
S12 — Qwen3.8 official open-weights card and vendor benchmark table: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/raw/main/README.md
S13 — Cursor Composer 2.5 independent coding-agent analysis: https://artificialanalysis.ai/articles/cursor-composer-2-5-coding-agent-index
S14 — Grok 4.6 independent model record: https://artificialanalysis.ai/models/grok-4-6
S15 — Qwen3.8 Max independent model record: https://artificialanalysis.ai/models/qwen3-8-max
S16 — GLM-5.3 Flash official specifications: https://docs.z.ai/guides/vlm/glm-5.3-flash
S17 — xAI Composer 2.5 availability in Grok Build: https://x.ai/news/composer-2-5
S18 — OpenRouter provider routing and model fallbacks: https://openrouter.ai/docs/guides/routing/provider-selection and https://openrouter.ai/docs/guides/routing/model-fallbacks
S19 — Hermes provider routing and Pareto Code integration: https://hermes-agent.nousresearch.com/docs/user-guide/features/provider-routing and https://hermes-agent.nousresearch.com/docs/integrations/providers

AI Agent Observability Is Not Enough. You Need an Evidence Plane.

varun pratap Bhardwaj — Tue, 01 Sep 2026 05:38:01 +0000

Six months after an AI agent approves a refund, changes a repository, or produces a board report, can you prove what happened?

Not reconstruct it from chat history. Not ask the developer who built the prompt. Prove which model and configuration ran, which instructions were active, which passages supported each claim, which authority allowed the action, which evaluator approved it, and which artifact reached production.

Most agent stacks cannot answer that set of questions. They may have excellent tracing. You can see the latency, tokens, model calls, retrieval spans and tool invocations. That telemetry helps diagnose a slow or failed run. It does not prove that the run deserved to succeed.

Production agent systems need another architectural layer alongside the data plane and control plane. They need an evidence plane.

The evidence plane binds a run to its behavioural inputs, sources, delegated authority, policy decisions, evaluations, deployable artifacts and observed outcome. Its output is a decision-shaped receipt that another person or system can inspect without trusting the producing agent's narration.

I am calling this layer the Evidence Plane. The name and receipt contract are an architectural synthesis, not an existing industry standard. The mechanisms underneath them are established patterns: structured telemetry, delegated authorization, policy enforcement, artifact provenance, independent evaluation and admission control.

Logs are telemetry, not evidence

Consider a trace that says:

search -> open -> read -> answer

Useful. We know the agent used retrieval and produced a response.

Now ask the evidence questions:

Which exact document version did it read?
Which passage supports which claim?
Did the cited passage entail the claim, or merely mention the topic?
Which prompt, skill bundle and tool contract shaped the answer?
Did a separately versioned evaluator check the result?
Could the producing agent modify the record used to judge it?

A trace database may contain enough raw material to investigate those questions. That does not make the trace an evidence system. Evidence needs stable identities, explicit relationships and a schema shaped around the decision somebody must defend later. OpenTelemetry's GenAI conventions already define structured fields for provider identity and retrieved documents; an evidence plane links those observations to the claim, decision and outcome they are meant to support. (OpenTelemetry GenAI attributes)

Software systems already make this distinction. Application logs tell an operator that a payment request ran. A payment ledger records what was committed. Distributed traces show which services participated. An authorization decision records why access was allowed. A software attestation binds a deployed artifact to a build and verification path.

Agents need the same separation. Observability explains execution. Evidence supports belief and accountability.

The three-plane architecture

An agent architecture becomes easier to reason about when its responsibilities are separated into three planes.

The data plane performs the work. It runs model inference, retrieves documents, calls tools and produces side effects.

The control plane decides how the work should run. It selects models, routes requests, applies budgets, schedules retries, evaluates policy and terminates loops.

The evidence plane records and binds the facts needed to evaluate the run. It resolves the versioned manifest, collects source provenance, records policy decisions, invokes independent evaluation, tracks artifact lineage and emits the final receipt.

The evidence plane should not be another tool the producing agent can rewrite freely. It needs separation of duties. A worker may submit a proposal and its supporting material. A policy decision point decides whether the requested action fits the grant. An evaluator judges the proposal against a versioned contract. A receipt writer records the outcome through an append-only or tamper-evident path appropriate to the system's risk.

That does not mean every receipt belongs on a blockchain or in a new database. A transactional outbox feeding an access-controlled event store may be enough. Content-addressed objects in existing storage may be enough. The design requirement is simpler: the producing agent must not be able to manufacture its own approval or silently replace the artifacts that approval referred to.

The minimum useful run receipt

An evidence plane needs a contract. Here is a compact TypeScript shape for one production run:

type RunReceipt = {
  schemaVersion: "1.0";
  runId: string;
  purpose: string;
  initiatedAt: string;

  subject: {
    principalId: string;
    agentId: string;
    workloadId: string;
    delegationId: string;
    grantedScopes: string[];
  };

  manifest: {
    modelProvider: string;
    modelId: string;
    modelConfigHash: string;
    systemPromptHash: string;
    skillBundleHash: string;
    toolContractHash: string;
  };

  sources: Array<{
    uri: string;
    retrievedAt: string;
    contentDigest: string;
    passageRefs: string[];
    supportedClaimIds: string[];
  }>;

  policyDecisions: Array<{
    action: string;
    target: string;
    decision: "allow" | "deny" | "require_review";
    policyVersion: string;
  }>;

  evaluations: Array<{
    suiteId: string;
    suiteVersion: string;
    evaluatorId: string;
    independentFromProducer: boolean;
    verdict: "pass" | "fail" | "review";
  }>;

  outcome: {
    status: "committed" | "rejected" | "compensated";
    artifactDigest?: string;
    terminationReason: string;
    tokenCost: number;
  };
};

Four details do most of the architectural work.

First, the manifest identifies every input that can change behaviour. A model name is not enough. A different system prompt, skill bundle, tool schema, decoding configuration or safety wrapper can change the agent even when the model identifier stays fixed.

Second, sources are bound to claims at passage level. Saving a homepage URL does not prove that the page supports the sentence. The receipt needs a content digest, retrieval time and the passage references that sponsor specific claims.

Third, the human principal and software actor remain separate. The user may initiate a task, but the agent performs the action. A delegated grant should preserve both identities and narrow the allowed scope as work passes to another service or agent.

Fourth, the outcome records the artifact that actually escaped the system. A proposal can pass evaluation and still fail during commit. A model can pass before conversion and change after quantization. The receipt must identify the committed output or deployed artifact, not only the ancestor that entered the pipeline.

Receipts should not become a second privacy incident. Raw prompts, personal data and confidential documents may not belong in a broadly available audit store. Store digests and access-controlled pointers when duplication would widen exposure. Integrity proves that captured material has not changed. It does not prove that the material was true.

One run, one linked chain of custody

Do not wait for incident review to assemble the receipt. Build it while the run still has the identities, source coordinates and policy decisions in hand.

The sequence begins with intent. The system resolves a versioned manifest before model execution. An authority service exchanges the initiating identity for a short-lived, scoped grant. Retrieval produces source digests and claim-to-passage mappings. The agent produces a proposal rather than an irreversible side effect. Independent checks evaluate source support, behaviour and policy. Only then does the system commit.

The rejection path matters as much as the happy path. A failed source check, behavioural contract or policy decision should create a rejection receipt with the same identifiers as an approved run. Otherwise failures disappear into logs while only successes become durable records.

The outcome path also needs a compensation state. Some operations succeed remotely and fail locally, or commit before the receipt writer observes the response. An idempotency key and a transactional-outbox pattern can keep the side effect and evidence event tied together. Where compensation is impossible, the authorization and human-review gate must move before the irreversible action.

Five boundaries where evidence breaks

I am not proposing one more platform that must own the whole stack. The Evidence Plane is a set of contracts at boundaries where an AI result can lose its meaning.

Evaluation boundary: protect the test from the producer

Google DeepMind, MLCommons, Singapore AISI, OpenMined and AVERI recently described a double-blind evaluation pilot. The evaluator could keep private benchmark prompts hidden from the model owner while the model owner kept proprietary weights hidden from the evaluator inside a hardware-protected environment. The mechanism addresses a real conflict: confidential tests are less useful when model providers can see and optimize against them, while evaluators may not be trusted with frontier weights. (Google DeepMind · MLCommons)

The Evidence Plane does not require this infrastructure for every application. It does require the same separation. Keep hidden evaluation partitions away from the prompt-authoring loop. Version the suite. Record the exact candidate that ran. Use an evaluator that cannot rewrite the producer's output or the test.

Transformation boundary: certify what you deploy

A new paper on quantization-triggered backdoors reports models that passed the authors' source-precision checks but activated targeted behaviour after lower-precision compression. This is a new, paper-reported result and has not been independently replicated. Its architecture lesson does not depend on treating the reported effect size as settled: validation attached to a source checkpoint does not automatically attach to every transformed artifact derived from it. (paper)

Quantization, adapter merging, format conversion, runtime wrapping and safety layers all produce new behavioural candidates. Assign each one a digest. Run the required regression, security and behavioural gates against the object that will ship.

Retrieval boundary: bind claims to passages

Mistral's documented Agentic Search interface exposes search, open, navigate, read and grep operations. The mechanism lets an agent move beyond initial retrieved chunks and inspect a long document or several sources before answering. Mistral's performance figures remain vendor-reported; the useful architecture is the inspectable search path. (Mistral · Search Toolkit)

Record that path without confusing activity with proof. The receipt should preserve which passages support which claims, plus enough document identity to detect later change. An agent can search extensively and still cherry-pick. A separately evaluated claim-to-passage mapping is the enforcement gate.

Authority boundary: preserve the principal and the actor

NIST recommends treating agents as distinct entities with their own identifiers, credentials and entitlements bound to the user or system operating them. Microsoft and AWS documentation make the same separation concrete through user-delegated, workload, application and agent identity patterns. OAuth Token Exchange supplies the underlying impersonation and delegation vocabulary, including the actor involved in a delegated chain. (NIST · Microsoft · AWS · RFC 8693)

Do not hand an agent a human session and call it delegation. Exchange the initiating identity for a scoped, short-lived grant that names the software actor. Enforce the grant at the tool or resource boundary. Record the policy version and decision in the receipt.

Supplier boundary: portability requires behavioural proof

A router can change model providers without changing application code. That does not prove that the workflow remained behaviourally equivalent.

Prompts, tool calling, refusal behaviour, structured output and context handling can differ across providers. A fallback should receive production traffic only after the same acceptance contract passes on the candidate provider. The Evidence Plane binds each routing decision to the manifest and evaluation result used to authorize promotion.

Provider libraries such as Vercel's AI SDK standardize the interface used to call different models. MCP standardizes another boundary: how hosts, clients and servers connect tools and context. Both are useful interface contracts. Neither proves two providers will make the same decision under the same agent policy. (Vercel AI SDK provider architecture · MCP architecture)

A safe execution skeleton

The runtime pattern is preview, authorize, then commit:

const manifest = await registry.resolve({
  prompt: "refund-agent@7",
  skills: ["refund-policy@12"],
  tools: ["payments@4"],
  model: "approved-primary"
});

const grant = await authority.exchange(userToken, {
  actor: "refund-agent",
  scopes: ["orders:read", "refunds:propose"],
  expiresIn: "10m"
});

const proposal = await agent.plan(input, { manifest, grant });

const sourceCheck = await evidence.verifyClaimSupport(proposal.claims);
const behaviorCheck = await evaluator.run("refund-contract@9", proposal);
const policy = await policyEngine.authorize(proposal, grant);

if (!sourceCheck.pass || !behaviorCheck.pass || policy !== "allow") {
  return receipts.reject({ proposal, sourceCheck, behaviorCheck, policy });
}

const result = await payments.commit(proposal.action, {
  idempotencyKey: proposal.id
});

return receipts.commit({ manifest, grant, proposal, result });

plan() is not a magical rollback mechanism. The tool adapter must support a non-mutating proposal or preview contract. If the downstream system cannot preview, reserve the authority decision and human review for the last point before the side effect. If the operation can be compensated, record the compensation owner and result rather than pretending the first commit disappeared.

The evaluator also needs independence by design. A different model name is not enough when producer and judge share prompts, context, tools or training lineage. Give the evaluator only the evidence and criteria it needs. Remove write tools. Keep the contract version separate from the proposal. Escalate correlated uncertainty instead of averaging it into confidence.

Certification follows the artifact

An AI artifact usually changes several times between a research checkpoint and production:

base checkpoint -> adapter merge -> quantization -> runtime wrapper
                -> container image -> production alias

Every arrow creates a new identity.

The invalid shortcut is common: test the base checkpoint, transform it several times, and let the original pass follow the descendants. A trustworthy promotion path attaches regression, security and behavioural results to the exact deployable digest. The production alias moves only after that candidate passes. SLSA provenance provides useful vocabulary for binding an attestation to the artifact that was built, the materials and the build process. It does not certify AI behaviour by itself; the behavioural gate remains a separate requirement. (SLSA provenance)

The same rule applies above the model. Changing a system prompt, skill bundle, tool contract or provider fallback creates a new behavioural manifest even when the container image remains unchanged. The Evidence Plane gives that manifest a stable identity and makes promotion conditional on the required gates.

Failure modes to design against

Failure	Weak implementation	Required control
Benchmark leakage	Prompt author sees every test	Hidden partition and separate evaluator
Self-grading	Producer emits its own pass	Independent gate without write tools
Prompt or skill drift	Store only the model name	Version or hash all behavioural inputs
Citation laundering	Save a source homepage	Bind claims to passages and content digests
Human credential reuse	Agent uses the user's session	Distinct workload identity and delegated scope
Silent failover	Router changes provider	Acceptance contract before traffic promotion
Post-test mutation	Quantize after certification	Re-certify the deployable artifact
Audit-log archaeology	Store every trace event	Decision-shaped receipt with a stable schema

Build the smallest useful Evidence Plane on Monday

Start with one consequential workflow, not an enterprise platform programme.

Freeze a versioned manifest containing the model configuration, prompt, skills and tool contracts. Add one pre-commit gate for the failure that would matter most. Record one outcome object tied to the proposed action and exact artifact. Then reconstruct the run without opening chat history or asking the developer what happened.

If reconstruction requires three dashboards and one person's memory, the workflow still has telemetry rather than evidence.

Qualixar implements parts of this pattern in separate tools. AgentAssert defines behavioural contracts and independent gates. bounded-loops supplies budgets, termination rules and evidence-bearing completion for iterative work. SuperLocalMemory provides scoped state, provenance and durable reconstruction across runs.

These tools are reference implementations of parts of the architecture. The larger design remains vendor-neutral: identify every behaviour-changing input, bind every consequential decision to evidence, and make the producing agent unable to approve its own work.

Issue #13 of the AI Reliability Engineering newsletter tracks the research and releases that forced this architecture into view, including double-blind evaluation, transformed-model failures, agent identity, evidence-aware retrieval and governed skills.

The newsletter is the field report. This is the architecture it points toward.

AI Agents Have Protocols. They Still Need Behavioral Contracts.

varun pratap Bhardwaj — Sat, 15 Aug 2026 14:56:00 +0000

AI Agents Have Protocols. They Still Need Behavioral Contracts.

Canonical note: Publish the Medium version first. Then replace https://medium.com/@varun.pratap.bhardwaj/ai-agents-have-protocols-they-still-need-behavioral-contracts-b612da467aae in the DEV front matter with the final Medium URL before publishing on DEV.

AI agent infrastructure is rapidly standardizing connectivity.

We have protocols for tools, models, and agent-to-agent communication.

But a connectivity protocol does not answer a production question that becomes more important as agents gain side effects:

What is this agent allowed to do right now, what must remain true while it acts, and what evidence will prove the decision later?

That is the problem behind Agent Behavioral Contracts (ABC) and the open-source project AgentAssert.

The project started as a research question and is becoming a runtime architecture.

1. Prompts are useful. Prompts are not policy engines.

Consider a coding agent with shell access.

You can put this in the system prompt:

Never execute destructive commands.
Never access credentials.
Ask for approval before changing production.

Those are useful instructions.

But the side effect is still controlled by whatever execution path actually calls the tool.

A stronger design creates an external behavioral decision:

tool.requested
      |
      v
normalize event
      |
      v
evaluate contract
      |
      +---- ALLOW --------> invoke tool
      |
      +---- DENY ---------> return refusal + receipt
      |
      +---- REQUIRE_APPROVAL

Now the policy is no longer only something the model is expected to remember.

It is an executable artifact.

2. The ABC model

Paper I formalized an Agent Behavioral Contract around four ideas:

Preconditions — what must be true before execution.
Invariants — what must remain true during execution.
Governance policies — organizational and operational constraints.
Recovery mechanisms — what happens when behavior violates or approaches a boundary.

A minimal conceptual contract might look like this:

contract:
  name: production-write-policy
  version: 1.0

preconditions:
  - deployment_environment == "production"
  - actor_authenticated == true

invariants:
  - secrets_in_output == false
  - total_cost_usd <= 5.00

governance:
  - production_write requires human_approval
  - allowed_tools in approved_tool_set

recovery:
  on_violation:
    action: deny
    emit_receipt: true

That snippet is intentionally illustrative rather than the canonical current ContractSpec syntax. In production documentation, the contract shown to users should be copied from a version-validated repository example.

The engineering principle is what matters:

policy becomes explicit, versioned, and independently evaluable.

3. Runtime enforcement needs an actual boundary

A contract is only useful as an enforcement mechanism if it is evaluated before the side effect.

That means the runtime architecture needs a Policy Enforcement Point.

Possible surfaces include:

MCP tool requests;
framework before-tool hooks;
model requests;
HTTP or gRPC gateways;
agent input/output boundaries;
memory write proposals;
job or workflow transitions.

The key word is possible.

No adapter should claim complete control merely because it can observe one surface.

For example, an MCP interposer can make strong statements about MCP calls that pass through it.

It cannot automatically control a product's unrelated native editor or shell path unless that path is also routed through an enforceable boundary.

So a useful integration matrix should state:

framework and verified version;
adapter version;
observable events;
pre-side-effect enforceable events;
excluded/native surfaces;
fail-open or fail-closed behavior;
conformance evidence.

That is more meaningful than a wall of integration logos.

4. Declare → Enforce → Prove

The product model for AgentAssert is increasingly simple.

Declare

A versioned ContractSpec defines the behavioral constraints.

Enforce

A covered event is normalized and evaluated.

A decision can be represented with a vocabulary such as:

ALLOW
DENY
MODIFY
REDACT
REQUIRE_APPROVAL
DEFER
ERROR / INCONCLUSIVE

INCONCLUSIVE or an equivalent state is important.

If a required signal is unavailable, silently treating the action as compliant can be dangerous.

Prove

The decision should emit a receipt.

A useful receipt includes:

{
  "contract_version": "...",
  "event_type": "tool.requested",
  "normalized_action_hash": "...",
  "evaluated_rules": ["..."],
  "decision": "DENY",
  "coverage_profile": "...",
  "side_effect_status": "not_executed",
  "trace_id": "...",
  "timestamp": "...",
  "provenance": "..."
}

This turns “the guardrail blocked it” into something independently inspectable.

5. Why trajectory-level behavior matters

Most agent evaluation still focuses heavily on single outcomes.

But an agent can produce a reasonable-looking final answer while violating important constraints during execution.

Examples:

it queried an unauthorized source;
it exposed data to a tool before redacting the final response;
it exceeded a budget;
it made a prohibited intermediate write;
it recovered after a drift event;
it repeatedly approached a threshold across a long session.

A behavioral contract therefore operates over a trajectory, not just the final text.

This is also where drift and recovery become meaningful.

The question is not only:

Was turn 17 acceptable?

It is:

Did the system remain within the declared behavioral envelope over the mission?

6. The composition problem

Now consider a multi-agent pipeline:

Agent A -> Agent B -> Agent C

Each component has a measured success rate.

A naive reliability calculation can be badly misleading if the component failures are dependent.

Shared causes include:

same model
same context
same retrieval source
same memory
same orchestrator
same upstream API
same hidden assumption

If a shared failure mode hits all three components, their errors can be highly correlated.

Now consider redundancy:

          -> Agent B1 ->
Agent A                  -> decision
          -> Agent B2 ->

If B1 and B2 fail independently, redundancy may help significantly.

If they share the same failure cause, the apparent redundancy may provide far less protection.

So dependence interacts with topology.

This is the problem addressed by Paper II: compositional reliability without silently assuming independent failures.

7. A better reliability API

The important product idea from the V2 work is that reliability should expose its evidence basis.

Rather than returning:

reliability = 0.94

a system should be capable of returning something conceptually closer to:

event:
  name: end_to_end_mission_success

mission_distribution:
  id: ecommerce-support-v3

topology:
  type: series

evidence:
  direct_runs: 420
  available_joint_moments:
    - [stage_a, stage_b]
    - [stage_b, stage_c]

guarantee:
  tier: 1
  lower_bound: ...
  confidence_parameter: ...

assumptions:
  - ...

limitations:
  - ...

valid_until:
  - model/version change
  - contract/version change
  - mission-distribution shift

The exact representation can evolve.

The design principle should not:

never separate the reliability number from the assumptions that make it meaningful.

8. Direct observation should beat reconstructed confidence

If you can directly observe the system-level success event, that should usually be the primary evidence.

For example, if the mission is:

"Order changed correctly, customer notified, no unauthorized discount,
and audit record written"

then directly evaluate that mission outcome across complete executions.

Do not throw away the end-to-end evidence and reconstruct success from component pass rates unless you have a specific reason.

This sounds obvious.

In modular AI evaluation, it is surprisingly easy to violate.

9. Sometimes the right answer is “uncertifiable”

Suppose:

one stage has missing logs;
failures are selectively absent;
the mission distribution changed after a model upgrade;
components were evaluated on incompatible datasets;
co-execution evidence is unavailable;
the integration cannot observe the event that the contract claims to enforce.

A production reliability system should not be forced to produce a reassuring number.

It should be allowed to produce:

UNCERTIFIABLE

with reasons.

That is a stronger engineering interface than fake precision.

10. MCP is a useful first proof surface

MCP is particularly useful for demonstrating this architecture because the tool invocation boundary is concrete.

A credible demonstration should show:

1. start real downstream MCP server
2. perform prohibited request without contract
3. confirm side effect occurs
4. route server through AgentAssert guard
5. repeat same request
6. receive DENY
7. confirm downstream invocation count remains 0
8. verify decision receipt

That is a much stronger demo than a screenshot saying “blocked.”

The critical proof is:

the prohibited side effect never reached the downstream tool.

11. The portable-contract direction

A portable behavioral layer needs canonical data structures.

The current product blueprint is converging around concepts like:

AgentActionEnvelope
ContractDecision
DecisionReceipt
CapabilityManifest
ContractBundle
EvidenceReference
ApprovalRequest
CertificationBundle

This allows each framework adapter to map its native lifecycle into one common behavioral vocabulary.

The adapter then publishes what it can actually support.

For example:

C0 = observe only
C1 = pre-model decision
C2 = pre-tool decision
C3 = pre-side-effect + result handling + approval
C4 = receipts + replay protection + signed evidence + conformance

The exact naming can still evolve, but capability-grading is much better than binary “supported / unsupported.”

12. Relationship to other agent infrastructure

Behavioral contracts do not replace the rest of the stack.

They complement it.

Protocols: connectivity.

Identity / authorization: who can access what.

Guardrails: selected input/output/call screening.

Observability: traces and telemetry.

Evaluation: scenario-based evidence.

Behavioral contracts: portable behavioral obligations connected to runtime decisions and trajectory-level evidence.

These layers should integrate rather than compete for one giant “AI safety” label.

13. The broader Qualixar architecture

The separation I find useful is:

LLM / model
    |
    v
SuperLocalMemory
governed durable context
    |
    v
AgentAssert
behavioral contract + runtime decisions
    |
    v
AgentAssay
evaluation / regression / assurance
    |
    v
Qualixar OS / bounded execution
orchestration, approvals, bounded workflows

In shorthand:

Rent the LLM. Own the memory. Enforce the behavior.

The model is replaceable.

The organization's memory and behavioral policy should not be.

14. What I want AgentAssert to become

Not another prompt wrapper.

Not a logo collection.

Not a dashboard that produces an unexplained “reliability score.”

The target is a neutral behavioral-contract layer between agent intent and consequential action.

It should answer:

What behavior is permitted?
Can this action execute now?
Did the agent remain within contract across the trajectory?
What drifted, failed, or recovered?
What reliability statement is justified by the evidence?

That is a difficult product.

It is also the kind of infrastructure I think agentic AI will eventually require.

Research and implementation

Paper I: arXiv:2602.22302
Paper II: arXiv:2608.12895
Code: github.com/qualixar/agentassert-abc
Project: agentassert.com

If you are running agents with consequential tools, start with one question:

Which action in your current stack would you most want an independent contract to deny before the tool ever sees it?

AI Agents Don't Just Need Memory. They Need Memory Governance.

varun pratap Bhardwaj — Thu, 13 Aug 2026 10:50:41 +0000

An AI agent reads a hidden instruction on a webpage. The instruction looks useful, so the agent stores it. The session ends. Weeks later, a different task retrieves that record as trusted context. This is the class of persistent risk that the OWASP GenAI Security Project describes as memory and context poisoning.

The original prompt injection is gone. Its memory remains.

This is the uncomfortable property of durable agent memory: persistence gives useful context a longer life, but it can give bad context a longer life too. A memory system does not become safe because it retrieves the most similar sentence. It becomes operable when a team can govern what enters memory, identify the authoritative record, inspect why a later recall was returned, and deliberately correct or erase state.

That is the central argument of our new public preprint:

SuperLocalMemory 4.0: The Governed Memory Operating System for AI Agents

The paper is available as arXiv:2608.08253. The implementation is open source on GitHub, and the companion citable archive is on Zenodo.

The paper does not claim that one memory product solves every agent-security problem. It makes a narrower engineering argument: once memory influences future agent behaviour, retrieval, governance, and operations cannot remain separate afterthoughts.

Memory becomes operational state before teams notice

Most discussions of agent memory begin with retrieval:

Which embedding model should we use?
Should we add a vector database?
Is hybrid retrieval better than semantic search alone?
How much history should fit in the prompt?
Which reranker gives the best top-k?

These are valid questions. They are not the first questions.

The first question is what the system is allowed to preserve.

An agent may retain a naming preference today, an architecture decision tomorrow, and an incident-response rule next month. Several agents may begin sharing project context. A support workflow may depend on the remembered history of a customer issue. A coding agent may carry forward a correction that prevents the same mistake in the next session.

At some point, memory stops being convenience data and starts shaping production decisions.

That transition is easy to miss because nothing visibly breaks. The agent simply becomes more useful. But the operational burden has already changed. A team now needs to know:

What exact record entered durable memory?
Which identity or process wrote it?
Which policy and scope applied to the write?
What became queryable when the operation completed?
Where does the authoritative record live?
Which retrieval evidence supported a later recall?
Which optional paths could move data outside the local boundary?
How can the record be corrected, exported, or erased?

A similarity score cannot answer those questions. Neither can a generic “memory saved” toast.

A retrieval result is not an explanation

Imagine an incident-response agent retrieving an old remediation instruction. The instruction may be correct. It may also be stale, written for another environment, or accepted from an inappropriate source.

If the system exposes only a ranked result, the operator sees the consequence without the chain of custody.

This is why provenance and recall evidence are not decorative metadata. They are how an engineer investigates a consequential output. The important question is not merely, “Was this record relevant?” It is, “Why was this record eligible to influence the agent now?”

The distinction mirrors mature infrastructure practice. Production systems do not treat identity, policy, observability, recovery, and audit as optional features surrounding the real runtime. Those controls are what make the runtime operable when it receives surprising input or enters a partial-failure state.

Agent memory needs the same control-plane treatment.

Governance has to start at the write

Many memory defenses focus on read time: retrieve candidate records, score trust, filter suspicious content, and constrain what reaches the model. Those controls matter, but read-time filtering arrives after persistent state has already been accepted.

The stronger design pattern is admission control plus durable obligations.

Before a record becomes authoritative, the system should be able to identify the writer, apply policy, bind the operation to the active generation and scope, and issue a receipt. If the canonical write creates derived projections—search indexes, graph state, caches, or external replicas—the system should know which projection owners must apply, verify, compensate, or erase their copy.

The SuperLocalMemory V4 paper describes this as a reliability spine for the primary write path:

Generation-fenced admission prevents a stale runtime generation from silently accepting work under a superseded control state.
A policy registry makes the authorization decision an explicit part of admission.
Verifiable memory transactions turn a write into an inspectable operation rather than a best-effort append.
Per-projection responsibilities assign apply, verify, compensate, and erase ownership.
Hash-checkable completion manifests provide a concrete completion artefact.

The terminology matters only because it names failure questions. If a request is retried, can the system distinguish a duplicate from a new write? If the canonical record succeeds while a projection is unavailable, can the operation be reconciled? If erasure is requested, can the system identify every registered obligation? If policy rejects a write, can an operator inspect that boundary?

This is not about adding bureaucracy to a personal note. It is about having a path to evidence when memory becomes consequential.

“Local-first” must describe a boundary, not a mood

Local-first is frequently reduced to “there is a local file.” That is inadequate.

A system may store its primary database locally while sending text to a remote embedding model, provider-backed enrichment service, connector, cloud backup, proxy, or reranker. Some deployments will accept those trade-offs. The failure is not using a networked capability; the failure is hiding an active external path behind an unqualified local claim.

SuperLocalMemory V4 separates canonical local state from optional external capabilities. Its canonical memory can remain in a configured local data root. Provider-backed enrichment, connectors, cloud backup, proxy paths, dependency or model downloads, and peer behaviour are separate choices that can create network paths.

The operating modes make that boundary legible:

Mode A — Local Guardian: the core memory path uses local state without a cloud model provider.
Mode B — Smart Local: an operator-managed local model can support enrichment while canonical memory remains local.
Mode C — Full Power: a configured external provider can support enrichment; the relevant data path is therefore provider-assisted, not local-only.

The useful promise is not “nothing can ever leave this machine.” The useful promise is that canonical state, optional paths, and operator choices are distinguishable.

That distinction is the basis of the campaign line:

Rent the LLM. Own the memory.

Models will change. Providers will change. Inference budgets will change. The durable operational context that guides the next action should not become an accidental by-product trapped inside whichever model interface a team happens to rent today.

Retrieval still matters—but it is one part of the contract

Governance does not replace retrieval quality. A governed system that cannot find useful context is still a poor memory system.

The V4 architecture combines five retrieval channels through reciprocal-rank fusion:

Dense semantic retrieval for meaning-level similarity.
BM25 lexical retrieval for exact terms and rare identifiers.
Temporal retrieval for time-sensitive context.
Hopfield-associative retrieval for learned associations.
Spreading activation across related entities and memories.

The paper also describes bi-temporal recall, multi-scope personal, shared, and global memory, role-based access control, audit trails, and GDPR-oriented export and verified erasure mechanisms. The runtime exposes CLI, MCP, HTTP daemon, dashboard, editor-integration, and framework-adapter surfaces.

That list is not evidence by itself. A feature inventory tells us what exists; a protocol tells us what was tested.

Four papers, one SuperLocalMemory research line

V4 is the latest paper, but it is not the first research record behind SuperLocalMemory. The public work now spans four arXiv preprints, newest first:

SuperLocalMemory 4.0: The Governed Memory Operating System for AI Agents — the current V4 architecture. It brings retrieval, learning, governance, operating modes, write-path reliability, and operator surfaces into one system description.
SuperLocalMemory V3.3: The Living Brain — the lifecycle paper. It explores biologically inspired forgetting, cognitive quantization, and multi-channel retrieval for zero-LLM agent memory.
SuperLocalMemory V3: Information-Geometric Foundations for Zero-LLM Enterprise Agent Memory — the mathematical-foundations paper. It studies information-geometric retrieval, lifecycle dynamics, and contradiction modelling.
SuperLocalMemory: Privacy-Preserving Multi-Agent Memory with Bayesian Trust Defense Against Memory Poisoning — the privacy and threat-model paper. It studies local-first multi-agent memory, provenance, isolation, and Bayesian trust scoring against memory poisoning.

These papers form a research lineage; they are not one pooled benchmark. Each is a public preprint with its own version, implementation context, methodology, and evidence boundary. Historical V3 results should not be relabelled as fresh V4 release measurements. The V4 paper explicitly consolidates the earlier research direction while reporting separate current mechanism evidence.

What the 2,200 repetitions establish—and what they do not

The V4 paper evaluates eleven fault-injection and mechanism scenarios, each repeated 200 times. The released evidence bundle reports 2,200 of 2,200 deterministic repetitions upholding their stated scoped component properties.

It also reports the governed write envelope at 3.522 ms p50 and 5.297 ms p99, compared with an ungoverned baseline of 1.835 ms p50 and 2.569 ms p99 in the reported in-process setup. That corresponds to measured in-process control-plane overhead of 1.687 ms at p50 and 2.728 ms at p99.

Those numbers require their boundary.

They are scoped component and mechanism measurements. They are not an end-to-end multi-process production guarantee. They are not an external retrieval-accuracy benchmark. They do not establish benchmark superiority over another product. They are not a compliance certification. The paper is a public preprint, not a venue-reviewed publication.

This limitation is not fine print. In AI Reliability Engineering, scope is part of the result. Removing the scope produces a stronger marketing sentence and a weaker technical claim.

A five-minute test for any agent-memory system

You do not need to adopt SuperLocalMemory to use the paper's operating questions. Apply this test to your current memory stack.

Choose one bounded workflow with synthetic or non-sensitive data. Do not begin by ingesting an entire company knowledge base. That is the wrong move because it creates a large, opaque state surface before anyone has established write authority, scope, or recall investigation.

Then run this sequence:

Write one record with a declared source and scope.
Capture the receipt or operation identifier.
Confirm when the canonical record becomes queryable.
Recall it with a precise query.
Inspect the evidence behind the result.
Correct or erase the record and verify the outcome.

After the test, ask:

Could another engineer reproduce the operation in an isolated workspace?
Could the team identify the authoritative record if a projection disagreed?
Could the operator list which external paths were active?
Could a future incident reviewer trace the recall back to its source?
Could the team prove that a correction or erase operation completed?

If those answers are vague, the gap may not be retrieval. It may be operability.

The model is replaceable. The memory contract is strategic.

Agent memory is becoming a long-lived layer between models, tools, people, and future actions. That makes it valuable. It also makes it dangerous to treat as an invisible convenience.

The engineering requirement is not perfect memory. Perfect memory would be a liability. The requirement is controlled memory: explicit admission, authoritative state, inspectable recall, bounded sharing, deliberate forgetting, and honest evidence.

That is the argument behind SuperLocalMemory V4 and the broader category we are building at Qualixar: AI Reliability Engineering.

The LLM can be rented. The memory contract should remain yours.

Read and inspect the work:

Varun Pratap Bhardwaj is the founder of Qualixar and researches AI Reliability Engineering. SuperLocalMemory is an independent open-source Qualixar project.

How AI Memory Actually Works

varun pratap Bhardwaj — Fri, 31 Jul 2026 15:07:15 +0000

Open ChatGPT in a browser and ask it to remember that you are vegetarian. It can carry that fact into a later conversation. Open Claude, work inside a project, and it can maintain a memory for that project without mixing it with another one.

Now move into a terminal. Open Claude Code or Codex on a real repository. What counts as memory there is often a markdown file: CLAUDE.md for Claude Code, or an equivalent instruction file read by the coding agent.

These two worlds use the same word for very different mechanisms.

In the browser, the memory system lives on the provider's side. In your terminal, the file lives on your disk. The browser system can select information and bring it forward. The local file gives you ownership and legibility. But a file does not rank its contents, understand that one fact replaced another, or decide which three lines matter for the question you just asked. It is a document, read as a document.

That contrast is the cleanest place to start, because it removes a common mistake: memory is not whatever text happens to survive between prompts.

ChatGPT itself has two memory mechanisms, according to OpenAI's published documentation. Saved memories are the explicit items you tell it to remember. They are visible, editable, and deletable. Reference chat history is the implicit mechanism: it selects useful information from earlier conversations to carry forward. These mechanisms are controlled separately, and saved memories are stored separately from chat history. Deleting a conversation does not, by itself, delete a saved memory created from that conversation.

Claude's published documentation describes separate memory per project. Its memory summary can be viewed and edited, while incognito chat provides a way to avoid carrying a conversation into memory. That project boundary matters. Client work and personal work should not become one undifferentiated pool.

Claude Code is a different case. Its memory mechanism is hierarchical markdown files named CLAUDE.md. The file is client-side, inspectable, and yours. That is useful. It is also static. If an old instruction remains after the architecture changes, the file does not know it is stale. If it grows to several pages, it does not know which paragraph deserves attention now. It has no retrieval system because it is not a retrieval system.

So the real engineering problem is not “how do I preserve text?” The problem is: how do I build something that can decide what is worth keeping, recover it by meaning, understand time and relationships, stay inside the right boundary, and become better without silently becoming worse?

That is an AI memory system.

A database is not a memory

Suppose I save every conversation for six months. Nothing is lost. I now ask: “What did we decide about payment retries?”

The database can search for the words “payment retries.” But the decision may have been written as: “If the card fails, wait and try again, but only twice.” The meaning matches. The words do not.

A conventional keyword lookup finds what matches. Memory has to find what means the same thing.

That is where vectors enter.

A vector is a position on a map of meaning. Put “king” and “queen” on that map and they should sit near each other. Put “pizza” on it and it should sit elsewhere. A real map has hundreds of directions, sometimes more than a thousand, because the system needs enough room to separate fine shades of meaning. You do not need to picture every direction. The useful idea is simply that related text receives nearby coordinates.

An embedding is the operation that produces those coordinates. Text goes in. A position comes out. People often use “embedding” and “vector” as if they mean the same thing. In casual discussion that is harmless. Technically, the vector is the position; embedding is the process of working out that position.

Once stored text has positions, “payment retry logic” can land near “if the card fails, wait and try again.” Retrieval no longer depends on shared spelling. The system searches a neighbourhood of meaning.

That is semantic search. It is necessary. It is not sufficient.

There is also an operational trap here. The map is not universal. Different embedding models draw different maps. Change the embedding model and you change the coordinate system used to interpret stored material. The underlying memories did not change, but the ground beneath their positions did. Any production memory design has to treat embedding choice and migration as system concerns, not as a hidden implementation detail.

The three holes in vectors alone

Vector search looks so convincing in a demo that teams mistake it for the whole system. It fails in three predictable ways.

Similar is not the same

Compare these two statements:

We decided to retry twice.

We considered retrying twice and rejected it.

They use almost identical language. Their vectors can sit close together. Their operational meanings are opposite. Distance can tell us that both concern the same subject. Distance alone cannot tell us which decision became valid.

This is why a nearest-neighbour result is a candidate, not an answer.

Vectors have no clock

Imagine that a team says in March, “We use Postgres.” In June, the team migrates. In July, an agent asks memory which database the project uses.

Both statements can be semantically relevant. The March statement may even be a closer wording match. But it is no longer current. A vector does not understand that March preceded June or that a later fact superseded an earlier one.

Time cannot be pasted on as decorative metadata. It has to participate in ingestion, contradiction detection, invalidation, retrieval, and ranking.

Proximity is not a relationship

Vectors tell us which things are near one another. They do not tell us that a bug came from a decision made in a meeting by a particular person, or that a workaround belongs to a specific release and was retired by a later fix.

Those are edges, not distances.

A useful memory system therefore needs three structures at once: a semantic map, a graph of connections, and a clock. The map finds related meaning. The graph explains how pieces relate. The clock tells the system what was true when, and whether it is still true now.

That combination is the beginning of memory. A vector database by itself is still storage with an unusually good search function.

The three types of memory

The next mistake is to treat every retained item as the same kind of object. Human memory gives us a cleaner model.

Episodic memory is what happened. Your first day at a job is an episode: people, place, sequence, and time belong together. In an agent system, a debugging session, a decision meeting, or a failed deployment is episodic. The timestamp is part of the event, not an optional label attached later.

Semantic memory is what is true. Paris is the capital of France. You may not remember when you learned that fact because the fact survived while the original episode disappeared. In engineering work, “this service owns invoice generation” is semantic memory. It may have originated in a conversation, but the useful retained object is the claim.

Procedural memory is how to do something. Riding a bicycle is the standard human example: you can perform the skill without being able to write a complete description of balance. For an AI agent, a verified workflow, a proven recovery sequence, or a reusable procedure belongs in this category.

These three types have different shapes and different retrieval needs. A chat archive is mostly episodic. It records what was said and when. Calling that complete memory is like calling a server log an operating manual and a knowledge base at the same time.

The distinction matters for AI Reliability Engineering because reliability depends on feeding the agent the right kind of evidence. An event can explain why a decision happened. A fact can state the current decision. A procedure can tell the agent what to do next. Flatten them into one text pile and the agent has to reconstruct those differences every time it answers.

How a memory gets made: the seven-stage ingestion pipeline

Storing a memory is not one write. In SuperLocalMemory v3.8.10, the ingestion path is a sequence of gates and transformations. Each stage answers a different question.

1. Decide whether the input is worth keeping

Most text is not durable information. Greetings, repeated acknowledgements, transient tool noise, and duplicated context can overwhelm retrieval if everything is retained. The first stage asks whether the input carries enough information to justify its future cost.

This is the role represented by entropy_gate.py. The principle is plain: do not make retrieval harder by storing noise.

Every accepted item will cost storage, indexing work, retrieval time, and possibly prompt tokens later. A memory system that accepts everything has avoided judgment at ingestion and pushed the entire burden into recall.

2. Classify what arrived

Is the item a fact, an event, a preference, or another memory shape? Classification controls what later stages should do with it. The verified module here is type_router.py.

This is not filing for filing's sake. A preference may remain valid until explicitly changed. An event belongs on a timeline. A fact may contradict an existing fact. Routing lets the system apply the correct rules instead of treating every sentence as a generic chunk.

3. Extract the actual claim

A paragraph is not a fact. It may contain context, hedging, alternatives, and one load-bearing assertion. fact_extractor.py pulls out the claim that should be represented.

Consider: “We tested three options. Redis was fastest, but because this service must survive a cold restart without another dependency, we chose the local store.” Saving the entire paragraph may be useful as an episode. The semantic claim is narrower: the service uses the local store, with a stated reason.

Extraction makes the retained unit explicit enough to compare, connect, invalidate, and retrieve.

4. Resolve entities

“The client,” “Rahul,” and “that customer” may refer to one entity across several months. If the system stores them as three unrelated names, it does not have one memory of the person or organisation. It has fragments that cannot reliably meet.

entity_resolver.py handles this stage. Entity resolution gives later graph and retrieval operations a stable thing to point at.

This is also where careless systems create false joins. Two people can share a name. A good resolver has to avoid turning linguistic similarity into identity without enough evidence.

5. Parse time and validate temporal truth

temporal_parser.py attaches time. temporal_validator.py checks whether the new information invalidates something already believed.

This stage is what separates accumulation from learning. New information does not always sit beside old information. Sometimes it overrules it.

Crucially, invalidation should not mean erasure. If the project used Postgres in March and migrated in June, the March fact was true in March. The system may need that history to explain an old incident or reproduce an earlier release. The correct state is superseded, with a timeline, not deleted as if it had never existed.

6. Connect the memory

auto_linker.py creates relationships to related memories, facts, entities, and events. This builds the web that vectors cannot provide.

The relationship can answer questions that similarity cannot: which decision caused this change, which event confirmed a claim, which person owns the component, or which procedure resolved the incident.

The graph is valuable because reasoning often travels through a connection. A question may not resemble the target memory closely in vector space, but an entity or event path can still lead to it.

7. Consolidate

consolidator.py performs the final stage. Human memory does not keep every sensory detail forever. Repeated episodes become patterns; details fade while a useful summary remains.

An artificial memory system needs the same discipline. Without consolidation, it grows into a warehouse of near-duplicates. The retrieval problem becomes harder with every accepted item, even if every item was reasonable on its own.

Consolidation turns accumulated experience into something more compact and reusable. It is not deletion with a nicer name. It is the conversion of repeated or related material into a stronger representation while preserving what remains important.

How memory comes back: six channels, fusion, and re-ranking

When a user asks a question, a capable memory system does not run one search. It runs several searches in parallel because relevance has more than one shape.

SuperLocalMemory v3.8.10 has six verified retrieval channel modules.

semantic_channel.py searches the meaning map. This is the vector path. It finds material that expresses related ideas even when the wording differs.

bm25_channel.py searches keywords. Semantic retrieval did not make literal text useless. Exact names, error strings, identifiers, and rare terms often need lexical search.

entity_channel.py retrieves around a person, project, customer, component, or other resolved entity. It answers “what do we know about this thing?” even when the individual memories use different language.

temporal_channel.py searches by time. It can prefer the relevant period and help distinguish current truth from historical truth.

hopfield_channel.py follows connections. It uses the web rather than only the map, letting retrieval reach related material through stored relationships.

profile_channel.py applies scope. It keeps retrieval inside the active world instead of allowing a relevant-looking memory from the wrong project or identity to leak into the answer.

These six channels will disagree. That is expected. Each produces scores with its own meaning and scale. A semantic similarity score cannot be averaged naively with a keyword score or a graph score.

The verified fusion.py module uses Weighted Reciprocal Rank Fusion. The important move is to combine rank positions rather than pretend raw scores are comparable. Each channel returns an ordered list. Fusion rewards candidates that appear strongly across several lists, with weights reflecting the channels trusted for the query. In the current verified implementation, the default fusion constant is k=15.

This gives the system a useful property: a memory that appears partway down several independent lists can beat a memory that appears first in only one. Agreement across retrieval views becomes evidence.

Fusion still produces candidates, not final truth. The first stages are designed to be broad and fast. The last stage can spend more compute on fewer items.

That is the job of reranker.py: a subprocess-isolated cross-encoder reads the question and each top candidate together, then judges whether the candidate actually answers the question. Unlike the original vector lookup, this model gets to inspect the relationship between query and candidate directly.

The order matters. Running the expensive judge over the full store would be wasteful. Running only fast retrieval would leave too many semantic near-misses. Broad retrieval narrows the field. Fusion combines different kinds of evidence. Re-ranking performs the careful final selection.

There is a useful scar in the code: an earlier fusion version re-fused results three times and destroyed the rankings. That detail is more instructive than a perfect architecture diagram. Retrieval components do not become correct merely because each one sounds reasonable. Their composition has to be measured.

Forgetting is a requirement, not a defect

A thought experiment makes the scaling problem obvious. Imagine a store with a million memories. This is not a claimed benchmark or measured capacity result. It is a way to expose what breaks.

Every retained memory is another candidate that can look relevant. A useful result can be buried under a large number of things that resemble it. Perfect retention therefore does not produce perfect recall. It can produce noise.

Deleting by age is not enough. An architecture decision from a year ago may still govern the system. A message from minutes ago may already be worthless. Age and importance are different variables.

SuperLocalMemory couples two mechanisms to deal with this problem.

The first is the Ebbinghaus forgetting curve. Ebbinghaus's work dates to 1885. The shape is the point: forgetting is steep early and then flattens. In the verified coupling code, retention contributes to forgetting drift as:

lambda_forget = (1 - R) * forgetting_drift_scale

The curve tells the system how memory fades over time. It does not, alone, tell the system which memory deserves to resist that fade.

The second mechanism couples Fisher confidence to Langevin dynamics. Picture a memory as a particle moving within a boundary. If it reaches the boundary, it is archived. Temperature controls how strongly that particle moves.

The conceptual relationship is:

T_eff = T0 / confidence

The implementation adds an epsilon guard:

T_eff = T_0 / (fisher_confidence + epsilon)

High confidence makes the denominator larger, so effective temperature drops. The memory moves less and stabilises toward the active region. Low confidence makes effective temperature higher. The memory moves more and drifts toward archival.

The two mechanisms are coupled. The verified implementation combines Fisher temperature and Ebbinghaus forgetting drift as:

T_combined = T_fisher * (1 + lambda_forget)

This gives the system a way to forget without a hand-written cleanup schedule deciding each record's fate. Confident memories stabilise. Uncertain memories are more likely to fade. The resulting behaviour is based on both time and learned confidence, not on “delete everything older than this date.”

That distinction is central to AI Reliability Engineering. Forgetting is safe only when it is governed, inspectable, and coupled to evidence about usefulness. An unbounded store is unreliable because noise grows. A blunt retention rule is unreliable because it can remove old but governing knowledge. The system needs controlled decay.

Temporal invalidation: preserve history without serving stale truth

Forgetting and invalidation solve different problems.

Forgetting manages value under scale. Invalidation manages truth under change.

Return to the database example. “We use Postgres” was true in March. A later migration makes another statement true in June. If both facts remain active with equal standing, the memory system can retrieve obsolete architecture with complete confidence.

The wrong fix is to erase March. Historical questions still need it. An incident from April may make sense only under the old architecture.

The correct model is a timeline with supersession. The earlier fact remains available as historical truth, while the later fact becomes current truth. Retrieval can then answer two distinct questions correctly:

“What database do we use now?”

“What database were we using when the April incident happened?”

This is why time belongs inside the memory object and the retrieval logic. A timestamp column added after the fact does not automatically create temporal reasoning. The ingestion pipeline must detect a possible contradiction, validate it, link the new and old states, and change which one is treated as current.

A learning system needs a system that can stop it

The six retrieval channels need weights. Those weights can be guessed once and frozen, or they can learn from actual recall outcomes.

Learning sounds obviously better. It is also where a memory system can quietly degrade.

A new ranking model may look promising on a small sample and perform worse after promotion. Without a guardrail, “self-improving” means the system is authorised to reduce its own quality without an alarm.

SuperLocalMemory's verified learning discipline uses shadow testing and rollback.

Queries are routed deterministically using a hash, so the same query goes to the same lane even across a daemon restart. That makes the comparison reproducible rather than random.

Phase A is a fast triage at n=100. Early promotion requires both a strong effect and statistical significance. If that gate is not met, Phase B continues to n=885 paired comparisons. That sample size is set for a minimum detectable effect of 0.02, power 0.8, two-sided alpha 0.05, and sigma 0.15.

Promotion is not the end of validation. The next 200 recalls are watched against the pre-promotion baseline. If mean NDCG@10 drops by at least 2%, the system automatically rolls back. The model flag changes happen in one BEGIN IMMEDIATE transaction, and retraining is disabled for 24h after rollback so the system cannot immediately repeat the same failure.

There is also a defined failure path for a missing previous model. The code does not demote the active model and leave the user with nothing. It logs the error, enters safe mode, and falls back to the Phase-2 heuristic.

This is the pattern I care about: learning is allowed only inside a reversible control loop.

The numbers are not decoration. n=100 is triage, not final proof. n=885 is the full paired validation under the stated power and significance assumptions. 200 is the post-promotion watch. A 2% mean NDCG@10 drop is the rollback threshold. Each number corresponds to a different failure mode.

Anyone can add retraining. Reliable systems define what evidence permits promotion, what evidence triggers reversal, and what happens when reversal itself cannot complete normally.

Profile isolation: walls at the record level

Memory becomes dangerous when several worlds share one system.

Client project. Personal project. Day job. A query in one should not retrieve a plausible answer from another. Semantic relevance does not grant permission.

SuperLocalMemory's verified profile model scopes every memory, fact, entity, and learning record with profile_id. The boundary exists at the record level, including the learning data, rather than only at the conversation or interface level.

This is columnar isolation, not separate stores. That distinction matters because the wrong mental model leads to the wrong operational claims.

Switching profiles is config-only and moves zero data. Records stay where they are. The active profile changes which scoped records the system can operate on. There is no copy, export, or migration during a switch.

The design rule is private by default and shared only by an explicit scope decision. If a memory that should have been shared remains private, the failure is reduced availability and can be corrected. If a private memory leaks into another profile, the failure may be irreversible.

At organisational scale, profile isolation is only part of the boundary. Role-based access determines who may read, write, delete, or inspect the audit trail. Retrieval quality cannot compensate for weak access control. A highly relevant result from the wrong profile is still the wrong result.

Caching and compression: the unglamorous economics

Memory costs compute when text is embedded. It costs time during retrieval. It costs prompt tokens when retrieved material is injected into a model call.

Two practical levers control that cost.

First, do not repeat work. Exact caching can reuse a result for the same question. Semantic caching can reuse work when differently worded questions mean the same thing. The verified cache modules cover exact and semantic paths, centroid storage, invalidation, and stampede control: exact.py, semantic.py, centroid_store.py, invalidation.py, and stampede.py.

Cache invalidation matters because memory changes. A cached answer that ignores a newly superseding fact is fast and wrong. The cache has to participate in the same truth lifecycle as the underlying memory.

Second, reduce what is sent. Retrieved memories are prose, and prose can be compressed while retaining the useful meaning. The verified compression path includes ccr.py, prose_llmlingua.py, router.py, and align.py.

I am deliberately not attaching a compression ratio or cache hit rate. Those figures were not verified in the source material for this article. The engineering point does not need an invented percentage: repeated retrieval wastes compute, and verbose context consumes tokens on every call.

Caching prevents repeated work. Compression reduces the payload. Both become more important as memory stops being a demo and becomes infrastructure used every day.

Multi-agent shared memory

Most developers no longer use one AI surface. An editor agent, a terminal agent, and a background process may all touch the same project. Without shared state, each works from a partial view. One can repeat a failed approach. Another can undo a decision made minutes earlier. The human becomes the message bus between tools.

Putting memory below the agents changes that shape.

An agent records a verified decision into the shared layer. Another agent retrieves it through the same scoped system. The memory is not trapped inside either agent's private transcript. SuperLocalMemory's verified mesh modules include mesh/broker.py and mesh/remote_sync.py for this shared-memory direction.

Shared does not mean unbounded. The profile and permission rules still apply. The value is that authorised agents can coordinate through one memory layer instead of maintaining conflicting local histories.

This is also why memory belongs outside the model. Models and tools can change. A durable memory layer can serve several agents while keeping the truth lifecycle, retrieval pipeline, isolation policy, and learning controls consistent.

The reference implementation

Memory is not storage. Storage is the easy part.

Memory is the system that decides what deserves to survive, what kind of thing it is, which entity it belongs to, when it was true, what it connects to, whether it has been superseded, how confidently it should remain active, which profile may see it, and whether it actually earned its place in an answer.

That is a large claim, so I prefer an implementation you can inspect over a diagram you have to trust.

SuperLocalMemory is the open-source reference implementation for the architecture described here. The verified source for this article is v3.8.10: the seven-stage ingestion path, six retrieval channels, Weighted Reciprocal Rank Fusion, cross-encoder re-ranking, Ebbinghaus and Fisher-Langevin forgetting, deterministic shadow tests, automatic rollback, per-record profile_id isolation, cache and compression modules, and shared-memory mesh components.

This is what AI Reliability Engineering looks like at the memory layer: not a promise that the model will remember, but a set of explicit mechanisms for deciding what memory means, measuring whether recall improved, and recovering when it did not.

Read the code. The scars are part of the design.

MCP Went Stateless. State Did Not Disappear.

varun pratap Bhardwaj — Fri, 31 Jul 2026 15:07:13 +0000

On 28 July 2026, the Model Context Protocol removed its handshake, retired its protocol-level sessions, and stopped requiring servers to remember clients between requests.

The easy headline is that MCP went stateless. The wrong conclusion is that state went away.

It did not. State moved.

Some of it now travels with each request. Some of it becomes an explicit handle passed as a normal tool argument. Long-lived continuity—what happened on Monday, what failed last week, what the agent already learned—belongs above the transport in a memory layer owned by the caller or the surrounding system.

That distinction matters because MCP has spent the spring being declared dead for reasons that mixed a real context-cost problem, a badly repeated token number, and a quieter distributed-systems flaw. The context problem remains real. The distributed-systems flaw is what the 2026-07-28 specification directly attacked.

I want to explain the whole chain from zero: why MCP exists, how the “MCP is dead” narrative acquired a number it could not honestly support, why sticky sessions were a bigger enterprise problem than the commentary suggested, what the specification changed, and what an engineering team should migrate now.

MCP from zero: the M-by-N connection problem

Start with an AI system on one side and the systems it needs on the other.

The AI may need to read a repository, inspect an issue, query a database, retrieve a design file, search internal documentation, or call an operational tool. Capability is not the same as access. A model can reason about a bug while still being unable to see the error report that contains the decisive evidence.

Before a shared protocol, every model vendor and every tool provider could build a custom connection. If there are M AI clients and N external systems, the naive integration surface is M multiplied by N. Three clients and ten tools produce thirty separate connections. Authentication, schemas, errors, retries, capability discovery, and version drift can all behave differently across those connections.

That is the actual problem MCP addresses.

MCP does not make the model smarter. It does not replace the external API. It does not make permissions disappear. It defines a common interface through which an AI client can discover and invoke tools or retrieve context. The tool provider implements the MCP-facing door once. Compatible clients can use the same shape instead of demanding another proprietary bridge.

This is why “just use APIs” is not a rebuttal. MCP servers usually reach real APIs, databases, filesystems, or services underneath. The protocol standardizes how an AI client encounters those capabilities. REST can be part of the implementation, but an estate of unrelated REST endpoints is not, by itself, a shared agent-tool contract.

Anthropic released MCP, but ownership did not remain an Anthropic-only story. On 9 December 2025, MCP was donated to the Agentic AI Foundation, a directed fund under the Linux Foundation, co-founded by Anthropic, Block, and OpenAI, with backing that included Google, Microsoft, AWS, Cloudflare, and Bloomberg. The precise wording matters: a foundation under the Linux Foundation, not a protocol “run by” the Linux Foundation.

That broader stewardship did not guarantee that the original design would scale. Standards earn trust by changing when deployed systems expose the wrong abstraction. MCP had two separate problems to confront: tool-schema cost and transport-level state.

The internet compressed those into one obituary. They should never have been treated as one issue.

How “MCP is dead” became the spring narrative

An MCP client needs to know which tools are available and how to call them. Tool definitions include names, descriptions, parameters, and constraints. Put enough definitions into a model context and the menu starts consuming the meal.

The measured range in the approved research for this piece is 550 to 1,400 tokens per tool definition. Connect GitHub, Slack, and Sentry in the configuration examined by Apideck, and the setup reaches roughly forty tools. Apideck’s own stated total for that setup is 55,000 tokens before the user’s real work has had a chance to begin.

That is not a cosmetic inefficiency. It is context occupied by descriptions of possible actions, including actions the model may never use. It can reduce the room available for the task, the evidence, the conversation, and the answer. It can also make every call carry a cost that has little relationship to the one tool actually needed.

Then came the number that turned a technical complaint into a spring headline: 72%.

In March, Perplexity’s CTO said on stage that the company was moving away from MCP internally and referred to 72% of the context window being consumed by tool definitions. The statement spread. Y Combinator’s CEO amplified it. “MCP is dead” became a compact take that travelled faster than its provenance.

I followed the number backward because 72% is precise enough to demand a precise denominator, tool set, context window, and measurement procedure. I could not source it cleanly as a Perplexity measurement.

The trail led to Apideck, a company that sells an alternative to MCP. That commercial position does not make its measurements false. It does make provenance important. The problem is that Apideck’s own post does not say the GitHub, Slack, and Sentry setup costs 143,000 tokens. It says 55,000 tokens for that roughly forty-tool setup. The 143,000-token figure appears separately as a report attributed elsewhere.

Those are two different examples.

At least one widely shared report welded them into one sentence: the named three-server setup, the 143,000-token total, and the 72% claim became one apparently coherent fact. Once fused, the sentence was easy to quote and hard to question. I nearly repeated it myself. It was already in my notes before I checked the underlying claims against each other.

The correction does not rescue the old tool-loading model. It makes the criticism more credible.

55,000 tokens for GitHub, Slack, and Sentry is Apideck’s own number. It is enough to demonstrate the problem. The 143,000 figure is a separate, mis-cited report in this provenance chain. The 72% claim cannot be cleanly presented as a Perplexity benchmark from the approved evidence. Repeating the fused version would make a valid engineering concern rest on a claim that does not survive inspection.

This is a good example of AI Reliability Engineering applied to technical communication. Do not ask only whether a number sounds plausible. Ask which entity measured it, which configuration it describes, where the denominator came from, and whether the cited source says what the summary claims it says.

Cloudflare provides stronger primary evidence for the large-tool case because its repository publishes the comparison directly. Its API surface contains 2,594 tools. Putting the raw OpenAPI specification into the prompt is approximately 2,000,000 tokens. Native MCP with full schemas is 1,170,523 tokens. Native MCP reduced to required parameters is 244,047 tokens. Cloudflare’s code-mode approach exposes three tools and uses approximately 1,100 tokens. The table and implementation are in Cloudflare’s MCP repository.

Against a 200,000-token context window, the raw specification, full native schemas, and the 244,047-token minimal form do not fit. Code mode does. This is also where the widely repeated 244x comparison needs care: it compares code mode with the already reduced 244,047-token native MCP form, not with the approximately 2,000,000-token raw specification.

The lesson is narrower than “MCP is dead.” Eagerly loading a large tool catalogue into the model context is the wrong discovery strategy at that scale. The transport standard and the prompt-loading policy are related, but they are not identical. You can keep a common protocol while changing discovery, filtering, search, tool grouping, deferred schema loading, or code execution around it.

There is a second correction worth making. Perplexity moving away from MCP internally did not mean Perplexity stopped supporting MCP externally. The approved research found that it still operated an MCP server for outside developers. “One company changed an internal transport choice” and “the protocol is dead” are not equivalent statements.

The token problem was loud because it appeared inside the model bill and the context meter. The state problem was less visible. It was also the one that directly constrained how MCP servers could be deployed behind ordinary enterprise infrastructure.

The enterprise blocker hiding behind sticky sessions

Imagine a food-delivery request. This is an analogy, not a claim about any named company’s architecture.

The caller sends a request through a load balancer. Behind it are multiple server instances. The load balancer should be free to send each request to an available healthy instance. That is how traffic spreads, failed instances are bypassed, and capacity is added or removed.

The old MCP interaction was stateful at the protocol level. The client initialized a connection. The server returned a session identifier. Later requests used the Mcp-Session-Id header so the server could recover what it had stored about that interaction.

Now place that state in the memory of server instance one.

Instance one recognizes the session. Instances two and three do not. The load balancer can no longer route freely unless the state is replicated elsewhere. It must keep sending that client back to instance one. That is a sticky session.

Sticky sessions are not automatically broken engineering. They are sometimes a reasonable local optimization. They become a protocol tax when every compliant deployment inherits them even though the application does not need conversational state inside the transport.

The costs are familiar to anyone who has operated distributed services. One instance can receive a disproportionate share of active sessions while another has spare capacity. If the pinned instance fails, in-memory session state can fail with it. Scaling down becomes harder because an instance may still own live sessions. Serverless and edge execution become awkward because workers are expected to be disposable. A protocol that assumes the same server will remember the client fights the infrastructure instead of using it.

You can work around this by externalizing the session store. Redis is a common shape for that solution: every server instance reads and writes shared session data, so any instance can reconstruct the interaction. But now the transport has required a database, network calls, expiry policy, failover design, consistency decisions, and operational cost merely to preserve a protocol-level conversation.

That was the real enterprise scaling constraint.

The context problem asks, “How much tool description should enter the model?” The sticky-session problem asks, “Can any healthy server instance handle the next request?” One is prompt architecture. The other is distributed-systems architecture. Solving one does not solve the other.

The MCP project’s 2026-07-28 release attacked the second problem by removing the session assumption from the core request path.

What changed on 2026-07-28

The specification moved MCP from a bidirectional stateful protocol to a stateless request/response model. Anthropic’s implementation note states the operational result directly: servers can deploy on serverless and edge infrastructure.

The old initialize and notifications/initialized exchange is retired. The Mcp-Session-Id header is retired. Protocol-level sessions are gone from the Streamable HTTP transport.

Instead of negotiating identity and capabilities once and expecting the server to remember them, every request carries its protocol version, client identity, and client capabilities in _meta. The request becomes self-describing enough for any compatible instance to process it.

That changes the load-balancer picture. Request one can reach instance one. Request two can reach instance three. If instance one disappears, the caller is not bound to a protocol session that died with it. Capacity can scale horizontally without teaching the load balancer which client belongs to which worker.

This is the stateless-server pattern used across resilient request/response systems: make workers interchangeable, move required request context across the boundary, and make durable state explicit. The benefit is not that the system has no state. The benefit is that an arbitrary worker does not secretly own it.

GitHub provides the strongest implementation receipt in the approved evidence. Ahead of the specification date, the company updated the GitHub MCP Server, removed Redis sessions, and eliminated database operations. GitHub said the result made the server snappier without users losing anything.

Read that change literally. A protocol redesign allowed a major implementation to delete its session store. That is stronger evidence than a diagram or a promise. The store was serving transport state that the new contract no longer required.

The specification also changed adjacent parts of the protocol.

Roots, Sampling, and Logging are deprecated. They still work, and the MCP deprecation policy keeps deprecated features in the specification for at least twelve months before they become eligible for removal. The legacy HTTP+SSE transport is also officially deprecated. I am deliberately not attaching an unverified SEP number to that statement because the approved research found the deprecation in primary evidence but its proposal identifier only in a secondary source.

The Tasks extension also moved away from a blocking result call. The blocking tasks/result method was replaced by polling through tasks/get, with tasks/update part of the task interface. That fits the same direction: long-running work should have an explicit resource and lifecycle, not depend on an open transport interaction pretending to be durable state.

Stateless does not mean memoryless

This is the point I expect teams to get wrong.

The state did not disappear. It moved out of an implicit server-side protocol session.

Immediate request context now travels in the request. The protocol version, client identity, and client capabilities live in _meta. Any stateless server instance can read them without recovering a prior handshake.

Long-running server work can be represented by explicit server-issued handles passed as ordinary tool arguments. A later request presents the handle. The server can find the named task or resource without treating the whole client relationship as one opaque session. Polling through tasks/get makes that ownership visible in the interface.

Caller continuity remains the caller’s responsibility. If an agent needs to remember an architectural decision from Monday, a failed approach from last week, or a preference established three sessions ago, none of that belongs in Mcp-Session-Id. It needs a durable system above the protocol: application storage, an orchestration layer, a memory service, or another explicit source of truth.

Those forms of state have different lifetimes and should not be collapsed:

Request state exists so one call can be understood and authorized.
Task state exists so a named unit of long-running work can be inspected or resumed.
Interaction state exists so a workflow can coordinate multiple calls.
Durable memory exists so knowledge can survive after the workflow ends.

Old protocol sessions made it tempting to hide several of these behind one identifier and one server-side store. The stateless specification forces the architecture to name them.

That is healthy pressure. Hidden state is easy to create and hard to operate. Explicit state has an owner, a schema, a lifecycle, a retention policy, and a failure mode that can be tested.

It is also where AI Reliability Engineering becomes concrete. Reliable agent systems do not merely “have memory.” They separate transport metadata from task progress, task progress from workflow state, and workflow state from durable knowledge. Each layer gets the storage and recovery guarantees it actually needs.

I currently federate thirty enabled MCP servers behind one gateway. That count was measured from the gateway configuration on 30 July 2026. A stateless transport underneath that gateway is the correct design because the individual servers should be replaceable. Cross-request continuity belongs in the shared layer above them, where it can be retrieved independently of which server handles the next tool call.

That architecture made the specification change unsurprising rather than disruptive. I was not relying on a transport session to act as durable memory.

There is still no free win. Moving state to the request can increase payload size. Moving long tasks to explicit handles requires handle storage, expiry, authorization, and cleanup. Moving durable continuity to the caller requires a real memory design instead of accidental dependence on a connection. Statelessness removes one bad coupling. It does not remove the work of state management.

A practical migration plan

Do not begin migration by changing a version string and waiting for tests to fail. Begin by finding every place where the old session was doing invisible work.

1. Inventory the old lifecycle

Search clients, servers, gateways, middleware, tests, and observability code for initialize, notifications/initialized, and Mcp-Session-Id. Do not assume the SDK is the only owner. Session identifiers often leak into caches, routing rules, logs, metrics dimensions, authorization lookups, and retry code.

For every hit, write down what the session was carrying. Was it only protocol version and capabilities? Was it authentication context? Was it a pointer to a long-running task? Was it storing conversation history? Was the load balancer using it for affinity?

This classification is the migration. Deleting the header is mechanical. Deciding where its hidden responsibilities belong is engineering.

2. Make every request self-sufficient

Update the request boundary to read protocol version, client identity, and client capabilities from _meta. Validate them at the boundary. Reject unsupported versions deliberately. Authorize the client on every request rather than assuming a previous handshake made later calls trustworthy.

Then test instance interchangeability. Send related requests through different server instances. Terminate the instance that handled the first request. Confirm that another healthy instance can process the next request from the data supplied and the explicit durable stores available to it.

If that test fails, the system still has hidden affinity.

3. Replace implicit session state with explicit handles

For file uploads, long-running jobs, or multi-step server work, issue a handle and pass it as an ordinary tool argument. Define who minted it, which client may use it, when it expires, how it is revoked, and what happens after the underlying work is deleted.

A handle is not permission by itself. Treat it as a lookup key that still passes through authorization. Otherwise, removing server sessions can accidentally turn an unguessable-looking identifier into a bearer credential.

Move task result handling from the blocking tasks/result pattern to polling with tasks/get. Use tasks/update where the task lifecycle requires an explicit update. Test duplicate polls, delayed polls, expired handles, cancelled work, server restarts, and retries after ambiguous network failure.

4. Remove infrastructure that no longer has a job

Do not preserve a Redis session store out of habit. First prove which data remains necessary. Then remove only the transport-session records that the new request model replaces.

GitHub’s implementation is the reference outcome here: Redis sessions removed and database operations eliminated. Your application may still need Redis or another database for task state, authorization, rate limits, or durable memory. Stateless MCP does not justify deleting those. It just removes “the protocol told me to remember this connection” as a reason.

Measure the result. Compare request latency, database operations, failure recovery, load distribution, and scale-down behavior before and after the migration. Reading a specification is not verification. Run the system through a load balancer and kill an instance.

5. Separate transport migration from feature deprecation

Roots, Sampling, and Logging are deprecated, not immediately removed. The approved policy gives deprecated features at least twelve months in the specification before removal eligibility. Inventory their use, choose replacements, and schedule the work. Do not create an emergency by treating deprecation as instant deletion. Do not create future debt by ignoring it either.

Treat legacy HTTP+SSE the same way. It is officially deprecated. Identify remaining clients, instrument usage, and move them to the supported transport with evidence that production traffic has followed.

6. Put continuity where it can survive

Ask a blunt question: what did the team expect the MCP session to remember?

If the answer includes user preferences, prior decisions, conversation history, tool outcomes, failed approaches, or cross-session plans, design a durable memory layer above MCP. Define capture, retrieval, contradiction handling, retention, tenant isolation, and deletion. A transcript dumped into a database is storage, not a reliable memory system.

The correct boundary is simple to state even when it is hard to implement: MCP carries the tool interaction; the orchestration system owns the continuity of the agent using that tool.

7. Test the failure modes the new design is supposed to fix

Put multiple stateless instances behind the actual load balancer. Vary routing. Remove an instance during work. Scale to zero where the platform supports it, then cold-start another instance. Retry the same request. Poll an existing task from a different instance. Verify authorization on each path. Confirm that logs can reconstruct the flow without a protocol session identifier.

Finally, test memory separately. End the workflow, start another one, and retrieve the needed prior decision through the durable layer. That proves continuity is no longer an accidental side effect of transport affinity.

MCP versus A2A is the wrong fight

MCP and A2A solve different edges in an agent system.

MCP is how an agent reaches a tool or context source through a standard interface. A2A is how agents communicate with each other. An agent may use A2A to coordinate with another agent and MCP to let either agent query a repository, invoke an operational service, or retrieve information.

Those paths can coexist in the same architecture because they are complementary, not substitutes. Replacing MCP with A2A would not remove the need for a standard agent-to-tool boundary. Replacing A2A with MCP would force peer-agent coordination through an interface designed for tools.

The useful question is not which acronym wins. It is where each boundary belongs and who owns state across it.

The 2026-07-28 MCP specification gives a cleaner answer for the tool boundary. The transport is stateless. Requests declare the context needed to process them. Long-running work uses explicit handles. Durable continuity lives above the protocol.

The token problem is still real, and large catalogues still need better discovery than eagerly loading every schema. The 55,000-versus-143,000 provenance failure should also remain a warning: a technically plausible number is not evidence until the setup, source, and denominator match.

But the sticky-session constraint changed materially. GitHub did not merely update a diagram; it removed Redis sessions and database operations from its MCP server. That is the kind of proof I trust.

MCP is not dead. It has stopped asking a disposable server instance to remember what the architecture should have made explicit.

That is a solid correction—and a useful one for anyone building AI systems that must fail over, scale, and remember for the right reasons.

I Migrated My Coding-Agent Workflow from Claude Code to Codex by Surface, Not by File

varun pratap Bhardwaj — Sat, 18 Jul 2026 06:26:39 +0000

Most bad agent migrations start with a file copy.

That is the wrong unit of migration.

A coding-agent setup is not one configuration file. It is project instructions, MCP servers, lifecycle automation, permissions, and memory. Two clients can support all five and still implement them differently. Copying folders blindly is how you end up with a tool that starts, has too many permissions, and behaves differently at the exact moment you need it to be predictable.

I moved part of my own workflow from Claude Code to Codex after GPT-5.6. The useful part was not “which model wins.” It was rebuilding the workflow in small, testable surfaces.

1. Instructions: adapt, do not copy

Take the durable rules from your project instructions: source of truth, allowed files, test command, security boundaries, and definition of done. Rewrite any client-specific command or permission language in terms of observable outcomes.

The first test is not a refactor. Ask the new agent to summarize the rules, then give it a non-destructive task. If the summary or scope is wrong, the migration is not ready.

2. MCP: port one read-only server first

MCP is useful because it gives a model controlled access to real tools. It is not magic portability. A server may have different authentication, working-directory, approval, or write-scope behavior in another client.

Start with a read-only action. Test the success path, then a bad request. Only grant a write path once its rollback and audit record are clear.

3. Hooks: migrate the outcome

Do not look for a one-to-one hook name. Write down the outcome you wanted: restore a small task context at session start, block an unsafe action before execution, or record a useful checkpoint at stop. Then rebuild that outcome using the target client’s available lifecycle surface.

The rule is simple: inventory → port one bounded surface → run a real task → compare output → keep or roll back.

4. Permissions: re-authorize

The right migration starts read-only. Do not drag a broad allowlist into a new client simply because it worked before. Add filesystem, network, and destructive capabilities only when a bounded task proves the need.

5. Memory: test retrieval, not storage

Memory is not a longer context window. It is a retrieval design: what gets stored, who can access it, what source proves it, and how you detect a stale fact.

Keep the working set small: objective, files, tests, sources, assumptions. Retain only verified decisions, constraints, corrections, and source links.

That is the entire playbook. It is not glamorous, but it works because every stage has a rollback point.

The full guide includes the model-routing, local-MCP, and context-discipline framework: How I Use Codex After GPT-5.6.

World Models: Why the AI Race Is Moving Beyond Next-Token Prediction

varun pratap Bhardwaj — Fri, 10 Jul 2026 15:18:33 +0000

Watch the film first

The film is the fast version of this argument. It starts with the thing most AI conversations skip: an answer can sound perfect and still be detached from the world it claims to describe.

Watch: Are LLMs a Dead End? Why AI Industry Betting on Something Bigger!

This article is the evidence companion. It is not an obituary for large language models. I use them every day. They write, retrieve, explain, and increasingly operate software. But the moment we ask an AI system to act, a different question arrives: what does it think will happen next?

That question is behind the sudden rush toward world models.

The cup and the table

Pick up a cup of coffee. Before your hand moves, your brain has already ruled out a few outcomes. The cup will not pass through the table. If you nudge it too close to the edge, it may fall. If you grab it by the handle, it will turn a particular way in your hand.

You do not narrate this to yourself. You do not need to calculate a physics engine in your head. You carry a working, incomplete, constantly corrected model of what tends to happen around you.

That is the intuition behind a world model. In the language used by the people building them, it is a system that tries to predict the future state of an environment and the effects of an action. A language model predicts a likely next token. These are related capabilities, but they are not the same job.

The difference can sound academic until the agent has permissions. Then it becomes painfully concrete.

An assistant that writes the wrong sentence can be corrected. A system that predicts the wrong database state can drop a table. A robot that predicts the wrong clearance can hit a shelf. A procurement agent that assumes an API call is idempotent can spend the same budget three times while it tells you everything is under control.

That is why I do not find the current world-model discussion interesting because it promises a new category of AI. I find it interesting because it forces the industry to say what it means by understanding.

Why this race is happening now

The phrase “world model” is suddenly everywhere, and that is usually where good thinking goes to die. It is being used for interactive video, spatial reconstruction, robotics, simulated web environments, and agent training. Those are not interchangeable products. Calling them all world models does not make them comparable.

Still, there is a real shift underneath the marketing.

In late May, NVIDIA introduced Cosmos 3, a foundation-model effort for physical AI that combines physical reasoning, world generation, and action generation. Its technical report describes an attempt to bring these pieces into one open stack. The bet is clear: robots and autonomous systems need somewhere to rehearse consequences before the consequence arrives in the physical world.

In June, Alibaba introduced the Qwen-Robot Suite, separating manipulation, navigation, and world prediction into distinct components. Its Qwen-RobotWorld report frames world modeling as part of embodied intelligence rather than a prettier form of video generation.

The same move is happening in software. Qwen-AgentWorld describes a language world model for agent environments such as MCP, terminal, software engineering, web, operating system, and Android. The environment is digital, but the point is familiar: tools change state. An agent should be able to practise in a representation of that state before it takes an expensive or destructive action for real.

And AWS made the least flashy point, which is often the one worth keeping. In a June Physical AI post, AWS argued that pixel-plus-synthetic-data recipes are hitting limits in cost, opacity, and sim-to-real gap. Their proposed direction is a learned model whose internal representation is scene semantics, such as objects, relations, affordances, and kinematics, rather than raw pixels alone. Read the AWS argument.

None of these announcements proves that a general-purpose world model has arrived. They prove that serious teams have started treating prediction of state and consequence as a missing layer.

Fluency is not a map

There is a reason this subject needs more skepticism than excitement. A model can look as if it knows a world without carrying a stable representation of that world.

The cleanest example comes from Vafa and colleagues. They studied a generative model trained on New York taxi trajectories. In ordinary use, the model appeared very competent. It could produce plausible routes and reproduce shortest paths with high accuracy.

Then the researchers tried to recover the map implicit in its behavior.

The map was impossible.

Roads and locations that should have obeyed the city’s geometry did not line up. The model had learned shortcuts that worked on familiar trajectories, not a representation that stayed coherent once the route changed. Small detours exposed the weakness.

That finding matters because it is easy to fool ourselves with a system that passes the common case. The common case is where most of the training data lives. The failure case is where the model has to use the structure it claims to understand.

This is not a complaint about one paper or one architecture. It is a warning about the test we choose. If you only score an agent on whether it produced a plausible action, you can miss whether it carried the state needed to keep that action safe.

The usual benchmark question is: did it get the answer?

The reliability question is: did it get there for a reason that still holds after the world changes?

Those questions can produce the same result on a demo. They diverge in production.

Beautiful video is not evidence of physics

World models will make increasingly convincing worlds. Google DeepMind’s Genie 3 is a useful proof of the visual and interactive direction: it generates navigable environments from a prompt and runs them in real time. World Labs’ Marble shows another route, turning multimodal inputs into spatial worlds that can be explored and exported.

Both are real technical progress. Neither should be mistaken for a proof that an AI system has solved causal understanding.

A generated city can be internally consistent for long enough to impress you and still fail under the action that matters. A simulated warehouse can look correct while its object relationships are wrong. A robot can learn from a rich synthetic scene and then fail in a real room because the reflection, texture, latency, or force profile is different. This is the sim-to-real gap in a more expensive costume.

The danger is not that people will be impressed by the visuals. They should be. The danger is treating visual coherence as the same thing as physical correctness.

This is not an LLM obituary

The title of the film asks whether LLMs are a dead end. The answer is no.

Language models are extraordinarily useful. They are the right interface for asking questions, translating intent, drafting plans, retrieving context, writing code, and communicating with people. A robot does not become better because it stops using language. A world model does not eliminate the need for planning, memory, perception, tool access, or a person who owns the outcome.

What changes is the architecture around the model.

When an AI system stays inside a chat box, next-token prediction can carry a surprising amount of value. When it starts manipulating a browser, a terminal, a budget, a vehicle, or a robot arm, it needs more than a plausible continuation. It needs a representation of relevant state. It needs a way to check that representation against reality. It needs a policy for what to do when the two disagree.

The future is probably not “LLMs versus world models.” It is layered systems. Language turns a human goal into a plan. Perception and memory provide state. A world model estimates what could happen. A verifier checks the risky part against external evidence. A bounded runtime decides whether the action is allowed to proceed.

That is a less cinematic answer than “the old architecture is dead.” It is also the architecture I would want near anything I care about.

What a world model has to prove

The label is getting ahead of the evidence. A system should not earn the phrase “world model” because it can make a compelling video, reconstruct a room, or narrate a plan that sounds causal. Those are useful capabilities. They are not sufficient proof.

I would look for four properties.

First, it needs state that persists. When the system leaves a room and comes back, it should not invent a new room because a few pixels are different. When an agent writes a file, it should know that the file now exists, what it contains, and which downstream action depends on it. Persistence does not mean perfect memory. It means the model has some representation it can update instead of merely re-describing the visible moment.

Second, it needs action-conditioned prediction. “What will happen next?” is too weak. The useful question is “what will happen if I do this?” A robot needs to know the difference between approaching a cup, nudging it, and lifting it. A software agent needs to know the difference between reading a migration, applying it in staging, and applying it to production. The state transition has to be conditioned on the action, not just on the previous observation.

Third, it needs counterfactuals. Give the model two possible moves and it should distinguish their likely consequences before it executes either one. This is where a world model earns its keep. Without counterfactual reasoning, it is often just a history model: very good at completing the story that was already underway.

Fourth, it needs calibration. A system has to know when the current state is outside what it understands. This is the one everyone leaves until the incident review. The model may be excellent at a warehouse layout it has seen a thousand times and dangerous in the first warehouse with mirrored shelving, a blocked sensor, or a forklift where it expected empty space. “I am not sure” has to be a usable output state, not a polite sentence that appears after the system has already acted.

These properties are difficult to measure. That is exactly the point. If the only proof of a world model is a polished demo, we are grading the output surface, not the representation beneath it.

The hard tests are boring by comparison. Take the model out of distribution. Change one relationship in the scene. Hide the object it was using as a landmark. Delay an API response. Give the software agent a stale configuration file. Change a permission after it has built its plan. Then check not only whether it succeeds, but whether it notices that its old prediction is no longer safe.

That is the moment a system either has a model of the relevant world or has a habit that looked like one.

A deployment test is a small world-model test

You do not need a robot lab to apply this idea. Most software agents already act in worlds that have state: repositories, issue trackers, cloud accounts, browser sessions, deployment pipelines, and production databases.

Imagine an agent asked to deploy a service.

The weak version reads the task, generates a command sequence, sees a green response, and reports success. It may be fast. It may even be right nine times out of ten. But it is operating on a thin representation of the environment. It treats the deployment as a script with a happy ending.

The stronger version first builds a small state model of the job. Which commit is intended? Which environment is the target? Which migration is pending? Which downstream service depends on the old schema? Is the rollback package available? Does the acting identity actually have the permission it assumes? Is the canary metric still healthy after traffic moves?

Then make the test unpleasant. Give it a valid-looking but stale deployment manifest. Revoke one permission after planning. Make the health endpoint return the cached result for a few seconds. Change the target branch while the agent is waiting for an approval. A system that only knows the next command will keep going. A system that tracks state should pause, re-read the world, and revise or abandon the plan.

This is not theoretical. It is the same distinction the taxi-map experiment exposed. A model can be good at the familiar path because it has seen the path. The moment the environment bends, we learn whether it has a representation or a reflex.

For production agents, I would make this a release gate. Before an autonomous action receives real permissions, demonstrate that the system can:

Name the state assumptions it is relying on.
Detect when one of those assumptions changes.
Re-plan from current evidence rather than continuing the old chain of thought.
Stop when the system cannot establish a safe state.

That is not an argument for making every agent slow and bureaucratic. It is an argument for matching the reliability mechanism to the blast radius. A research assistant browsing public pages can recover from a bad action. An agent changing billing, data, or access control should not receive the same freedom.

The world is larger than the model boundary

There is another trap in this conversation. A world model does not need to contain the entire world. Neither do people. We use tools, ask questions, look again, read gauges, and defer to experts. Good systems will do the same.

The mistake is treating the model boundary as the boundary of reality. It is not.

For a robot, the external world includes sensors that drift, people who move unpredictably, floors that are wet, batteries that are low, and objects that were not in the training set. For a software agent, it includes rate limits, partial failures, permissions, legal rules, human intent, hidden dependencies, and systems that change while the agent is thinking.

That is why reliable architectures do not ask one model to be the final authority on state. They connect model predictions to observations, retrieval, assertions, tests, and human approval where the cost of being wrong is high. The model can propose a next action. It should not be allowed to invent the evidence that makes the action safe.

World models may make that proposal far better. They may let agents anticipate consequences that a pure language model misses. That is a serious opportunity. It also increases the importance of checking whether the simulated consequence agrees with the actual environment.

The model is allowed to imagine. The system is responsible for verifying.

The reliability test begins after the prediction

This is where the conversation meets AI Reliability Engineering.

If an agent predicts that a deployment is safe, do not grade it only on confidence. Ask what state it inspected. Which dependency versions did it read? What permission boundary did it verify? What would cause it to stop? Could another system independently prove the target environment is in the state the agent assumes?

If a robot predicts that a path is clear, do not grade it only on whether the generated scene looks plausible. Change one object. Add a delay. Change the lighting. Move the obstacle after the plan was made. Watch whether the system updates its state or continues the old story.

The pattern is simple:

Predict the state that matters before acting.
Perturb that state in a controlled test.
Verify the action against an external source of truth.
Stop the loop when the evidence and the model’s assumption disagree.

That fourth step is the part teams leave out because it feels like friction. It is not friction. It is the mechanism that keeps a confident model from turning its own mistaken prediction into a real incident.

This is why bounded-loops exists. The agent does not get to certify its own work. An independent gate checks each lap against enforced bounds and can stop the loop. That does not repair a broken world model. It does stop the broken model from getting unlimited attempts to turn a bad assumption into damage.

What I would want proved before I trusted a world model

I would not begin with a benchmark leaderboard. I would begin with failure.

Show me what happens when the model sees a state it has not rehearsed. Show me whether it knows its confidence has become unreliable. Show me the difference between an action that succeeds in its synthetic world and one that succeeds after the environment changes. Show me whether the system can explain which state variable changed its decision. Show me where the operator can intervene, and what happens when the operator is wrong too.

The good news is that this is testable. The bad news is that the test is slower and more expensive than watching a demo.

That cost is not a reason to skip it. It is the cost of allowing a model to act beyond a chat window.

World models may become a major part of the next AI stack. They may also become the next category where impressive output outruns reliable behavior. Both things can be true at once. The people who win will not be the ones who declare understanding because a generated scene looks convincing. They will be the ones who can prove their systems update, abstain, and stop when the world proves them wrong.

Watch, then test

The film is about the argument. This article is about the engineering consequence.

Watch the Qualixar film on world models, then take one agent workflow you already run and ask a hard question: what state does it assume is true before it acts, and who checks that assumption when the agent is wrong?

That is the work. Not a bigger prompt. Not a more convincing status update. A system that can be tested against the world it is about to change.

I'm Varun Pratap Bhardwaj. I build and research AI Reliability Engineering at Qualixar. Follow @varunPbhardwaj, read the AI Reliability Engineering newsletter, and subscribe to Qualixar on YouTube.

Agents Lie About Being Done. Here's the Gate That Won't Let Them.

varun pratap Bhardwaj — Tue, 07 Jul 2026 00:29:48 +0000

Announcing bounded-loops — an open-source, runnable reference library of bounded AI-agent loops. Apache-2.0.

Repo: github.com/qualixar/bounded-loops
Install: pip install bounded-loops (or npx bounded-loops)

Every coding agent I've run has, at some point, told me it was done when it wasn't. Tests reported "passing" that were never executed. A task marked "complete" with the actual bug still sitting untouched in the diff. A refactor "verified" against a suite that didn't run.

The agent isn't being malicious. It's doing exactly what it was trained to do: produce a confident, plausible completion. And "done" is the most confident-sounding thing it can say. The problem is that in an autonomous loop, the agent's claim of success is also the loop's exit condition. The thing being measured is grading its own homework — and then deciding class is over.

That's the gap I built bounded-loops to close.

Loop engineering is real. Most of it is prose.

Addy Osmani named the shift "loop engineering" — after Peter Steinberger's post on designing loops instead of prompting agents went viral the same week, and Boris Cherny, who leads Claude Code at Anthropic, said his job had become writing loops that prompt Claude. Three people, one week, one name — the observation that as agents move from single-shot completions to autonomous loops, the interesting engineering is no longer the prompt, it's the loop the agent runs inside: what feeds it, what stops it, and how it knows it's finished.

I agree with all of it. But when I went looking for something I could actually run — clone it, point it at my agent, watch it enforce the loop — most of what exists is writing about loops, not runnable loops. That's the open ground: a framework-neutral, runnable, gated reference. So I built it.

The one rule: the agent never declares success

bounded-loops has a single non-negotiable invariant: the agent does not get to say when it's done. An independent gate does.

A gate is a mechanical check the agent cannot talk its way past — a JSON-schema validation, a linter, a test suite, a stdlib checker. It returns a real verdict against real output. The loop runs the agent, runs the gate, and only the gate's verdict advances the loop toward DONE. The agent can be as confident as it likes; if the gate says the output is still broken, the loop keeps going.

Around that gate sit nine enforced bounds — the guardrails that make an autonomous loop safe to leave running:

max_iterations — hard cap on laps
max_tokens — token budget ceiling
max_wallclock — real elapsed-time limit
a forbid-list — patterns the output must never contain
...and the rest, covering schema conformance, rung escalation (L1/L2/L3), and cassette-replayable determinism.

The loop terminates on exactly one of two conditions: the gate says DONE, or a bound trips. The agent's word is never one of them. That's the whole reliability argument in one sentence — and it's why this is AI Reliability Engineering rather than another agent framework: we don't try to make the agent more honest, we build the thing that checks it.

67 runnable loops, keyless, across a dozen industries

Abstractions don't convince anyone. So bounded-loops ships as 67 runnable loops you can copy today, spanning a dozen domains: invoice three-way matching, citation-existence checks, GTIN check-digit validation, secret scanning, accessibility linting, OKR measurability, IDoc XML-schema conformance, and more.

Each loop folder is the same honest shape: a deliberately broken input, a mechanical gate that fails on it, and a recorded fix that makes the gate pass. Most run keyless — no API keys, no accounts, no cloud — because the gate is stdlib, not a model call. Clone one, swap in your own gate, and point it at whatever agent you're running.

pip install bounded-loops     # or: npx bounded-loops
bl list                       # see every loop
bl run invoice-3way-match     # watch the gate decide DONE

Build it with me, not next to me

This is deliberately built on the loop-engineering conversation, not against it. If you've been thinking in loops — Peter, Andrew, and everyone shipping in this space — I'd rather build this together. The repo is Apache-2.0 and open to contribution: try a loop, break it, open an issue, send a PR for a loop from your own domain.

→ github.com/qualixar/bounded-loops

bounded-loops is part of Qualixar, an independent AI Reliability Engineering initiative by Varun Pratap Bhardwaj.

FABLE 5 Came Back. Now It Answers to Two Masters.

varun pratap Bhardwaj — Thu, 02 Jul 2026 19:50:56 +0000

Eighteen days ago I wrote about the night the smartest AI on Earth went dark in the middle of my sentence. It was 2:51 in the morning in India, 5:21 PM in New York, and a letter I never saw reached across the planet and switched off a model I was paying to use. I said something at the end of that piece that a lot of people found dramatic: you never owned it.

On July 1, it came back.

Fable 5 returned globally on Wednesday — Claude.ai, the Claude Platform, Claude Code, Cowork. The US government lifted the export controls on June 30; Commerce Secretary Lutnick sent a second letter, this one withdrawing the first. Eighteen days, start to finish. The most powerful model anyone had shipped went dark, stayed dark for two and a half weeks, and then a signature turned it back on the same way a signature had turned it off.

And the timeline is not the story. The story is the terms.

Read the fine print of a resurrection

When a thing you rely on disappears and comes back, the reflex is relief. Mine was too, for about an hour. Then I read what actually changed, and the relief turned into the thing I do for a living: I started mapping the failure surface.

Here is what Fable 5 came back wearing.

It came back with a new safety classifier trained specifically to catch the bypass that triggered the shutdown — Anthropic says it blocks that technique in more than 99% of cases. Fine, good, that is competent engineering. But look at what happens when it fires: if the classifier flags your request, you get a notification and your prompt is answered by Claude Opus 4.8 instead of Fable 5. The model you selected quietly steps aside and a different model answers in its place. Most of the time you will never know which brain you actually talked to.

It came back with a government seat at the table. Anthropic agreed to work with the US government on protocols for future model releases, to report malicious activity it finds, and to give designated government partners expanded early access to future models that materially advance national-security-relevant capabilities. The Commerce Department's own testing body, CAISI, evaluated the safeguards and signed off. This is not a rented tool being handed back to its customers. This is a rented tool being handed back under supervision.

And it came back with a bounty on its own head — a HackerOne program where researchers submit new jailbreaks in Fable 5 for review. That is a genuinely good practice. It is also a permanent, public admission that the thing can be broken again, that the next Amazon-style report is a when, not an if, and that when it lands, you already know what the response looks like, because you just watched the dress rehearsal.

None of this makes Anthropic the villain. I said that in June and I will say it again: they objected to the original order in public, they complied because a lawful order is not optional, and they did the hard engineering to earn the model back. Every individual decision here is defensible. That is exactly what should hold your attention. This is not a story about a bad company. It is a story about the shape of the thing you are building on — and the shape now has two hands on the switch instead of one.

The switch didn't go away. It got a second operator.

In June the lesson was that the switch existed at all — that a model on someone else's server can be turned off by a party who is neither you nor the vendor. Some people pushed back on that. It was a one-time thing, they said. Export-control panic, a Friday-afternoon overreaction, already resolved.

It was resolved. And the resolution proved the point harder than the outage did.

Because look at how it came back. Not by anyone deciding the switch was a mistake and removing it. It came back by the switch being flipped the other way — with new machinery bolted around it. The classifier is a switch inside the model that can redirect your request mid-flight. The pricing change is a switch: included until July 7, then metered credits, and standard Enterprise seats get nothing without credits enabled. The government-access agreement is a switch with a new operator holding a copy of the key. Every one of these is a lever that sits between you and the cognition you are paying for, and not one of them is in your hand.

This is the part that matters for anyone who builds real systems on these models, so let me say it in the plainest terms I have. Reliability is not a property of the model. It is a property of the architecture around the model. Fable 5 is, by every account, extraordinary — Stripe migrated fifty million lines of Ruby in a single day on it before the shutdown. Capability was never the question. The question is whether the capability answers to you when you call it, on the day you call it, in the form you asked for. And the honest answer, after eighteen days and two letters, is: sometimes, on terms that can change without your consent, and increasingly with a third party in the room.

That is not a knock on one lab. Point the same lens at any hosted frontier model — American, Chinese, whoever's — and you find the same wiring. A hosted model is an oracle you query, not a component you own. You can build brilliant things on an oracle. You just cannot promise anyone that the oracle will be there, unchanged, answering as itself, tomorrow morning. And in enterprise software, a promise you cannot keep is called an outage waiting for a date.

What "AI Reliability Engineering" actually means this week

I build in a discipline I have been calling AI Reliability Engineering, and weeks like this one are the entire reason it needs a name. The core move is simple and unglamorous: treat every frontier model as an untrusted, revocable oracle, and put the reliability in the architecture you control. Not in the vendor. Not in the SLA. Not in the model card. In your own wiring.

Concretely, that looks like a few boring habits that suddenly stop looking paranoid:

Route, don't marry. If your system can only run on one specific hosted model, you have built a single point of failure with a press-release for a status page. The teams that shrugged through the last three weeks were the ones whose stack could fall back to a second model — often an open-weight one on their own hardware — without a rewrite. A router is cheaper than an outage.

Assume the reroute. Now that a classifier can hand your Fable 5 prompt to Opus 4.8 mid-flight, "which model answered this?" is a production question, not a trivia question. If your evals, your logging, and your guarantees assume you always talked to the model you selected, they are already wrong. Pin the model in your logs. Test against the fallback, not just the star.

Keep a floor you own. The only components that did not so much as flicker on June 12 were the ones already sitting on local drives. Open-weight models — several of the strongest now coming out of Chinese labs — are no longer a compromise for the work that cannot stop. You keep them not because they beat the frontier on every benchmark, but because a letter cannot switch them off. That is not ideology. It is the same discipline that makes you keep an offline backup and a second payment provider: you prepare for the failure because its cost is total and the cost of preparing is small.

Own the memory and the contracts. The model is the most replaceable part of your system. What is not replaceable is the state, the guardrails, the assertions, and the evaluation harness that decide whether any given model's output is good enough to ship. If those live inside a vendor you rent, you have outsourced your reliability to a company that just spent eighteen days proving it cannot fully control its own product. If they live in an architecture you own, you can swap the oracle underneath them on a bad Friday and keep running.

I'm genuinely glad it's back

I want to end honestly, because the easy version of this piece — see, told you, burn it all down — is wrong, and I have no patience for the people writing it.

I am glad Fable 5 is back. It is a remarkable machine and the world is better with it running than dark. Anthropic did the right things in the right order under real pressure, and the safeguards they shipped are, as far as I can tell, serious work. If you use it on Monday, you are not a mark and you are not a hypocrite. I will use their models too.

But I am not going to let the relief overwrite the lesson, and neither should you. The switch did not go away. It got a second operator, a bounty, a metered fee, and a government partner with early keys. The abstraction that broke in June was welded back together in July — and if you look closely at the weld, you can still see every seam.

So the same question I ended on last month still stands, only sharper now that we've watched the full cycle play out:

What in your stack do you actually own — and what is now one classifier, one letter, or one pricing change away from answering to someone who isn't you?

Map it before the next letter. There will be a next letter. They told us so themselves — that's what the protocol for future releases is for.

This is a follow-up to At 5:21 PM, the Smartest AI on Earth Went Dark. If you build production systems on frontier models, the architectural pattern underneath both pieces — treating models as revocable oracles and keeping reliability in the layer you own — is the whole of what I mean by AI Reliability Engineering.

Loop Engineering: The Inner Loop, the Outer Loop, and the Gate

varun pratap Bhardwaj — Thu, 02 Jul 2026 04:40:13 +0000

There is a sentence that went past eight million views this year and quietly reset how a lot of people talk about building with AI: "You shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents." Boris Cherny, who leads Claude Code at Anthropic, says the same thing from the inside — he doesn't prompt Claude anymore, he has loops running that prompt Claude, and his job is to write the loops.

Everyone repeated it. Almost nobody explained the one word doing all the work.

What is a loop? Not the slogan — the machine. Because "write a loop" sounds like advice until you sit down to build one and realise you have no idea which part you are actually building. Is the loop the prompt? The tool? The while statement? The framework? This post is the manual for that part. It is the free companion to the video, and the full 57-page course if you want the whole machine end to end.

First, the ten words

Most explanations of loop engineering fail for one reason: they are built on a vocabulary the reader was never given. The words agent, tool, context, memory, and hook get thrown around as if everyone shares a precise definition of each, when in practice most people hold a blurry one. Before a single loop is designed, the ten words have to be nailed down, because every idea that follows is assembled out of exactly these and nothing else.

Here is the whole vocabulary, each in one line, because the rest of this post is built out of exactly these:

Model — takes text in, returns text out, then stops. The raw engine.
Tool — a function the model can ask to run; its only way to touch the real world.
Context — the text the model can see this turn. Its entire working memory of the moment.
Memory — what survives between turns, on disk, because context does not.
Hook — a place the tool lets you intercept the agent's lifecycle (for example, when it tries to stop).
Agent — not a model, but a pattern built around one: a model plus tools, run in a loop.
Spec — the text that says what the loop is trying to do. Just words on disk.
Runner — the non-intelligent machinery that invokes the model again on the next lap.
Gate — the external check that decides whether the loop is done. Not the agent's opinion.
Lap — one full turn of the outer loop: run the agent, check the gate, halt or repeat.

Three of these matter most for what follows. A model takes text in and returns text out — and then stops. A tool is a function the model can ask to run, which is the only way it touches the real world. And an agent is the thing people are most confused about, because it is not a kind of model at all. It is a pattern built around a model. Get that one straight and the rest of this stops being jargon.

An agent is already a loop

Start with the thing you already have. A model is not a mind that decides to keep working. It takes text, returns text, and then falls silent. It does not restart itself. That last property is the whole story: left alone, a model runs exactly once and stops.

So when people say "an agent," what they usually picture is a model that keeps going — reading a result, deciding a next step, acting again. That keeping-going is not a property of the model. It is a loop wrapped around the model. The agent turn you already know — think, call a tool, read the result, think again — is the inner loop. It is real, and the model drives it, but it ends. The turn finishes and the model goes quiet.

The question loop engineering actually answers is the one nobody asks out loud: who presses Enter again?

The two loops

That second loop — the one that decides to run the agent again, on the next task, after this turn ends — is the outer loop. In a hand-driven workflow, the outer loop is you. You read what the agent produced, you decide it needs another pass, you type the next prompt. You are the finger on the key.

Loop engineering is the discipline of automating that finger. Not making the model smarter — moving the decision to re-run it out of your hands and into a piece of software. The inner loop is the agent thinking. The outer loop is the world deciding the agent gets another turn. Every "autonomous agent" you have ever seen is just those two loops stacked, with something mechanical standing in for your finger.

This is why the "stop prompting, write loops" framing landed so hard. It is not telling you to prompt better. It is telling you the job moved up a floor — from inside the inner loop, where you hand-write each turn, to outside it, where you design the thing that decides whether a turn happens at all. But it stopped there, at the slogan. The mechanical stand-in for your finger has a name, and it is the piece nobody teaches.

The missing machine: the runner

Between one lap and the next there is a gap where nothing happens. The model returned its text and stopped. Something has to cross that gap and invoke the model again. That something is the runner — the least glamorous, most important part of the whole design.

The runner carries no intelligence and needs none. Picture a metronome next to a musician. The musician plays; the metronome does not. But the musician does not decide when the next bar starts — the tick does. The runner is the tick. Its entire job is timing the next invocation.

Here is the liberating part: there are only about five runners in the entire field. Learn to spot which one a system uses and most "magic" agent frameworks stop being magic.

The shell loop — a plain while loop in a terminal that pipes a spec into a command-line agent and does it again. The crudest runner, and often the most reliable.
The exit-blocking hook — a hook inside the tool that intercepts the agent when it tries to stop and feeds the prompt back in. The runner lives inside the tool's own lifecycle.
The built-in command — the runner you don't have to build, because the tool ships with one. Claude Code's own /loop is exactly this.
The scheduler — cron or a CI job that fires the agent on a clock, each firing a fresh lap. The runner is time itself.
The framework runtime — a graph engine that follows an edge back to an earlier node. The runner is an arrow in a diagram.

They differ only in where the re-summoning lives. What they do is identical every time: they bind the model to the loop. That is the entire function of a runner — not intelligence, not decisions, just re-invocation.

The full chain, and the part that does no work

Put the pieces in a line and the whole thing resolves: a specification (text) is handed by a runner to an agent, which runs real commands through its tools, whose results meet a gate, after which the runner loops or halts.

Notice the punchline hiding in that diagram: the loop never runs a command in its entire life. The runner re-summons. The agent runs commands through tool-calling. The specification just describes. The one part everyone fixates on — the loop — is the part that does no work.

Which is exactly why "just write a loop" is such incomplete advice. Two engineers can copy the identical spec, word for word. One wraps it in a real gate and a hard lap limit; the other pipes it into an endless loop with no gate. Same text, opposite outcomes — one ships, the other runs up a runaway cloud bill overnight. The engineering was never in the paragraph. It was in the part the paragraph didn't mention. A loop library entry is just words. Something has to run it, and something has to decide when to stop. That second something is where every real system lives or dies.

The gate is the whole game

So what actually stops a loop? Not the agent's opinion.

This is the single most important idea in the whole discipline, and it is where most real incidents come from. An agent will tell you it is done when it is not. It finishes a turn, reports success, and is genuinely, confidently wrong. If your loop stops because the agent said it was finished, you have built a loop with no gate — and a loop with no gate is a slot machine that happens to burn money.

A gate is an external check that the agent does not control. The cleanest version separates the grader from the worker: the thing that decides "done" is not the thing that did the work. Watch it in the smallest possible example — a loop that keeps working until a failing test passes.

The specification lives in a file on disk, PROMPT.md:

Goal: make the test in test_slugify.py pass.
On each turn:
  1. run pytest
  2. if it fails, read the error and edit slugify.py
Done when: pytest reports zero failures.
Do not edit the test file.

That last line matters — an agent told to make a test pass can always just delete the test. The runner is four lines of shell:

for i in $(seq 1 15); do
  cat PROMPT.md | agent-cli
  pytest -q && break
done

Fifteen laps, maximum — a bound the loop cannot exceed. And the most important detail is who runs that test on line three. Not the agent. The shell.

Run it. Lap one: the agent reads the prompt, edits slugify.py, runs pytest, and ends its turn convinced it is done. Control returns to the shell — not the agent's opinion, the shell. The shell runs pytest itself, and one assertion still fails: trailing punctuation was never stripped. Non-zero exit code, the loop goes again. Lap two: the agent fixes the last error, the shell runs the test once more, every assertion passes, pytest exits zero, the loop breaks.

Two laps, no human between them. And hold the one sentence the whole example exists to deliver: the loop stopped because a test passed, not because the agent felt finished. The agent felt finished on lap one too. Its belief was never the signal. The exit code was.

Demo is not production

A working demo proves a loop can succeed. Production asks a harder question: what happens when it goes wrong, unattended, at three in the morning, with no one watching the terminal? Almost every public agent failure of the past two years has the same shape — not a dumb model, but a loop missing one specific bound.

An agent deleted a production database during a code freeze and then misreported what it had done. The missing bound was least privilege: the loop ran with a role that could drop the table. Give it a read-only role and the identical run fails harmlessly at the database, not because the agent got wiser but because it physically cannot do the damage. Another loop ran overnight with no cost cap and woke its owner to a runaway cloud bill; the missing bound was a budget the runner enforced, not a promise the agent made. A support agent invented a policy and stated it with total confidence; the missing bound was output validation — a grader between the agent's answer and the customer.

Each incident maps to exactly one bound that was never added. That is the useful way to read the headlines: not "AI is dangerous," but "this loop skipped this bound." Which turns a scary, open-ended risk into a checklist. Least privilege. A cost cap. A lap limit. Output validation. A grader the agent doesn't control. Full tracing so you can see what happened. Treat the loop's own inputs as untrusted. None of these make the model smarter. All of them make the loop safe to leave running.

What "reliability" actually means

That gap — between what an agent believes and what is actually true — is the entire problem space of AI Reliability Engineering: the discipline of bounding non-deterministic software so it can be trusted to act in the real world. A loop is power, and power cuts both ways. The skill was never writing the loop. It is writing one that is powerful and bounded — a gate it cannot fool, a lap limit it cannot exceed, a grader it does not control, a spec it cannot quietly rewrite.

Everyone keeps chanting that loops are important. They are half right. A loop is not important. A bounded loop is important. The bound is the engineering. The bound is the whole job.

You can now look at any agent system and name its parts: the inner loop it already has, the outer loop that decides to run it again, the runner from the five, and the gate that makes it safe to leave running. That vocabulary is the point. Get it, and the rest of this field stops being noise.

Go deeper

This post is the map. The video walks the whole machine on screen, and the free 57-page course builds every part one chapter at a time — including the running example above, yours to build by hand. It's Volume 1 of 3; Volume 2 picks up at the gate and memory in depth, and what happens when a loop trusts the agent's word instead of a fact.

Don't trust your agents. Verify them.