DEV Community: Luhui Dev

What Is Loopcraft? From Prompt Engineering to Agent Loop System Design

Luhui Dev — Fri, 26 Jun 2026 10:53:45 +0000

🙋‍
I’m Luhui Dev, a developer who has been breaking down Agent engineering and exploring how AI can be applied in education.
I focus on Agent Harness, LLM application engineering, AI for Math, and the productization of education SaaS.

Introduction

A new term has been circulating in the Silicon Valley agent world: Loopcraft.

My first reaction was: isn't this just putting an agent inside while true? A few years ago people called it Agent Loop. Then it became Workflow and Harness Engineering. Now we have Loopcraft. The AI industry never stops inventing new names.

But after following recent discussions from Peter Steinberger, Claude Code lead Boris Cherny, and Andrej Karpathy around agent loops, I do think something real is changing.

Peter Steinberger put it this way:

You shouldn’t be prompting coding agents anymore. You should be designing loops that prompt your agents.

In other words, you should not manually prompt a coding agent again and again. You should design a loop that prompts the agent for you.

Boris Cherny said something similar:

I don’t prompt Claude anymore. I write loops. The loops do the work.

Karpathy made a related point when introducing Autoresearch: if a human still has to inspect every result, decide the next step, and give the agent another instruction, the human becomes the throughput bottleneck of the whole system.

Put together, these comments point to an abstraction shift:

Before:
Human -> Prompt -> Agent -> Result

Now:
Human -> Design the loop
             ↓
Task discovery -> Agent execution -> Automatic verification -> Retry on failure -> Save state -> Continue running

My shortest definition is: Prompt Engineering optimizes a single interaction. Loopcraft optimizes the whole system that runs repeatedly.

Loopcraft is less interested in how to complete one isolated task and more interested in questions like:

Who starts the next task?
How does the agent know what to do?
Who checks the output?
How does failure produce useful feedback?
Should the system retry, switch strategy, or hand off to a human?
How is state preserved across sessions?
How do lessons from repeated runs improve the system itself?

This article breaks down three questions:

What is Loopcraft, and why is it suddenly getting attention?
How is it different from Agent Harness?
Can an ordinary developer build a small loop today?

1. Why People Are Moving From Prompts to Loops

For the past two years, the typical way to use a coding agent looked roughly like this:

Tell the agent what to do
-> Wait for code changes
-> Review the result
-> Tell it what is wrong
-> Let the agent continue
-> Review again

The model can already write code, search files, and run tests. But the whole process is still driven step by step by a human.

After each round, the agent stops and waits for the next instruction.

On the surface, the human is using the agent. From another angle, the human is acting as the scheduler, state machine, and verifier of the agent system.

So even if the model is fast, the human still cannot leave. The async mobile supervision features shipped by many agent products are one attempt to relieve this bottleneck.

That is the problem behind the recent loop discourse: do not automate only one step inside the work. Design the surrounding system for task discovery, assignment, verification, and continuation.

For example, fixing a CI failure used to look like this:

I see a CI failure -> Open Codex -> Copy the error log -> Ask it to analyze -> Review the diff -> Ask it to run tests -> Confirm green -> Manually create a PR

Inside a loop, it can become:

CI failure event -> Automatically read logs -> Decide whether the problem is safe to automate -> Start an agent in an isolated worktree -> Modify code -> Run tests and lint -> A second verifier checks the diff -> Create a PR when it passes -> Notify a human when it cannot proceed

The real automation here is not just code editing. It is the closed loop around code editing.

So Loopcraft is not a new model capability, and it is not one specific framework.

It is closer to an agent system design discipline: organizing task execution, result verification, event triggers, state persistence, and system improvement into nested loops.

2. Is Loopcraft Actually New?

The name is new. The underlying technical pieces are not.

We already had:

Reason-Act-Observe loops for agents;
workflows and state machines;
automated tests and CI/CD;
scheduled jobs and event-driven systems;
multi-agent collaboration;
LLM-as-a-judge;
Reflexion and Self-Refine;
long-term memory;
automated experimentation and hill climbing.

Even the simplest Ralph Loop is basically repeated invocation of a coding agent:

while true; do
  claude "Read the task and current progress, then continue the work"
done

3. Agent Harness vs. Loopcraft

This is where the terms are easiest to confuse.

Over the past year, Agent Harness has already become a popular concept.

Anthropic's definition is clear: a harness is the system that enables a model to work as an agent, including context handling, tool use, permissions, environment, state management, and result return.

Put simply, Harness answers: what environment does this agent work in?

Loopcraft answers a different question: when is this agent started, why does it continue running, who checks the result, and what should happen in the next round?

A simplified analogy:

Model: the worker's brain
Tools: the tools in the worker's hands
Harness: the worker's workstation and work environment
Loop: the factory cadence, quality control, and task scheduling
Loopcraft: how to design and layer the whole production loop system

In practice, the boundary is not absolute.

A mature long-running harness already includes retries, verification, and state handoff. A loop also depends on the harness for tools and execution environment.

I prefer to separate them by focus:

Concept	Main question
Prompt Engineering	What instruction should the model see in this round?
Context Engineering	What information should the model see right now?
Tool Engineering	What actions can the agent take?
Harness Engineering	How can one agent run happen reliably?
Loopcraft	How are repeated runs triggered, verified, connected, and improved?

Loopcraft does not replace Harness.

In fact, without a stable harness, a loop just manufactures errors automatically and continuously.

4. Loopcraft Is Not One Loop, but Layers of Loops

LangChain later broke Loopcraft into four practical layers. I find the breakdown useful.

Layer 1: Agent Loop

The innermost layer is the agent loop we already know:

The model reasons
-> Calls a tool
-> Reads the tool result
-> Continues reasoning
-> Stops when it believes the task is done

For example, a documentation agent can:

Read an issue -> Search the repository -> Edit Markdown -> Check links -> Create a PR

Layer 2: Verification Loop

An agent saying "done" does not mean the task is actually done. So we wrap the agent in a verification layer:

Agent executes
-> Verifier checks
-> If it fails, return concrete feedback
-> Agent executes again
-> Repeat until it passes or the budget is exhausted

The verifier can be unit tests, type checks, lint, schema validation, and so on.

One important principle: try not to let the same entity that writes the answer also grade its own exam.

Layer 3: Event-driven Loop

Once execution and verification are in place, the next step is removing manual startup.

Tasks can be triggered by real events. The agent is no longer just a chat tool; it becomes a background component in a business system.

Event
-> Deterministic rule decides whether to handle it
-> Start agent
-> Verify result
-> Update the real system

Layer 4: Hill-climbing Loop

The first three layers automate work.

The fourth layer starts automating how the work gets better.

Every agent run leaves a trace:

what task it received;
what tools it called;
where it failed;
why the verifier rejected it;
how many tokens it used;
whether a human had to take over.

An outer system can periodically analyze these traces:

Collect many run records
-> Identify frequent failure modes
-> Adjust prompts, tools, skills, or verifiers
-> Re-test on an eval set
-> Update the harness after passing

This is the layer where Loopcraft becomes most valuable.

An ordinary loop repeats work. A hill-climbing loop changes the system that produces the work.

Ordinary loop:
Failure -> Try again

Improvement loop:
Failure -> Analyze why it failed
        -> Modify prompts, tools, or verification rules
        -> Make future runs more reliable

The outer loop's arrow does not just go back to the beginning of the task. It reaches into the agent and changes the inner loop.

That is where compounding starts.

5. Karpathy's Autoresearch Is the Cleanest Loopcraft Example So Far

Karpathy's Autoresearch is a good concrete example for understanding Loopcraft.

The project is conceptually simple:

Agent proposes a training improvement
-> Modify train.py
-> Run training for a fixed five minutes
-> Read the val_bpb metric
-> Keep the change if the metric improves
-> Roll back if the metric worsens
-> Start the next experiment

It can run about 12 experiments per hour without human intervention. Overnight, it can complete close to 100 experiments.

The clever part is not a fancy agent prompt. It is that Karpathy reshaped the problem into an environment that is ideal for loop optimization:

the agent can modify only one file;
the evaluation metric is fixed;
each experiment has a fixed runtime;
results can be compared automatically;
failed changes can be rolled back;
Git records the full experiment history;
verification code cannot be modified by the agent.

That is the core shift of Loopcraft: humans move from directly doing the task to designing a system that can repeatedly do, verify, and improve the task.

6. How to Build a Minimal Loop Yourself

Autoresearch is a special environment. Ordinary developers can start with something simpler:

Automatically receive a small issue, attempt a fix, create a PR after tests pass, and retry with feedback when it fails.

Do not start with multi-agent orchestration. A minimal loop needs only six parts:

Trigger: what event starts the task, such as a CI failure, scheduled job, or issue with a specific label.
Goal: what counts as done, preferably something that can be turned into tests, lint, type checks, or another machine-verifiable condition.
State: store attempt count, failure reason, and current progress in a file or database instead of relying only on chat context.
Worker: run the coding agent in an isolated worktree or container so it does not pollute the main branch or other tasks.
Verifier: prefer tests, rules, and static checks. Use an LLM reviewer only for parts that are hard to formalize.
Budget: limit attempts, runtime, and cost. Hand off to a human for high-risk operations.

The whole flow can be simplified to:

for attempt in range(3):
    result = run_agent(goal, load_state())
    verdict = verify(result)
    save_state(result, verdict)

    if verdict == "passed":
        create_pull_request()
        break

    if verdict != "retryable":
        notify_human()
        break

Whether you use Claude Code, Codex, GitHub Actions, Bash, or Python is not the important part.

What matters is designing this chain clearly:

Trigger -> Execute -> Verify -> Feedback -> Retry or exit

As long as a task has a clear goal, reliable feedback, recoverable state, and a stopping condition, you already have a minimal loop.

7. Common Loopcraft Traps

Trap 1: Treating Infinite Retry as Autonomy

Running repeatedly is not the same as improving.

If the agent receives no new feedback, repeating ten times usually means spending ten times the tokens to make similar mistakes.

Trap 2: Letting the Agent Change Its Own Exam

The execution agent should not freely modify tests, evaluation metrics, time budgets, permission boundaries, or verifier prompts.

Otherwise it may not be making the task better. It may only be making "pass" easier.

Trap 3: Starting With Multiple Agents

Multiple agents do not automatically create intelligence. They first create more token cost, file conflicts, duplicate work, and state synchronization problems.

Get one worker, one verifier, and one persistent state path working before adding parallelism.

Trap 4: Measuring How Busy the Agent Is

Number of agents, runtime, token usage, and tool-call count are not the final value.

What matters is verified progress per unit cost.

Examples include issue auto-resolution rate, average cost per qualified PR, and human handoff ratio.

Trap 5: The Smoother the Loop, the Easier It Is to Stop Understanding

This is the risk I care about most.

When an agent can automatically write code, test it, fix it, and create a PR, humans may be tempted to look only at the final green check.

But the faster the system produces code, the faster human understanding of that system can decline.

Loopcraft should not become an excuse to stop thinking. It actually raises the bar for how much the human has to understand.

Closing Thoughts

I increasingly feel that agent engineering is going through an abstraction shift.

At first we discussed prompts. Then we moved to context, tools, memory, and harnesses.

Now the focus is moving outward again: how to put a single agent run inside a larger cycle of tasks, verification, and improvement.

I remain skeptical of fully removing humans from the loop.

But I agree with one thing:

Do not only fix the current result produced by the agent. Start fixing the system that keeps producing those results.

Field Notes: How Agentic RAG Handles the Real Mess of Enterprise Data

Luhui Dev — Fri, 19 Jun 2026 15:24:18 +0000

A Support Ticket That Opens a Data Maze

Say your company just launched an AI customer-support system.

A major account sends in a ticket: "What's the remaining warranty period on the servers we purchased under Project Alpha last quarter? Could you also share the original contract terms and the current technical support contact?"

It sounds like an ordinary question. But when your tech lead reads the ticket, they pause for a moment.

Because they know answering it requires the system to:

Look up the customer's profile and project history in the CRM
Look up the procurement contract and warranty terms for Project Alpha in the ERP / contract management system
Look up the stock-in date and device serial numbers for that batch of servers in the asset management system
Look up the current customer-success owner in the HR system

These four systems are maintained by different teams, run on different databases, and enforce different access controls.

A standard RAG system is helpless here. The best it can do is say, "Sorry, I couldn't find relevant information."

This is exactly the problem Agentic RAG is built to solve.

Traditional RAG: A One-Shot Retrieval Clerk

Let's quickly recap how RAG works.

The core idea behind RAG (Retrieval-Augmented Generation) is simple: an LLM's training knowledge is static, while enterprise data is dynamic and private. The fix is to retrieve relevant document chunks from a database before generating an answer, stuff them into the context, and have the LLM answer based on that material.

User question → [vector search] → retrieve relevant chunks → [LLM] → generate answer

This pipeline works well when there's a single knowledge base and a clear question. But it has two fundamental limits.

Limit one: a single retrieval pass, no iteration. Retrieve once, hand it to the LLM once, done. If the first pass misses the key information, the whole chain breaks, and the LLM is left guessing or saying "I don't know."

Limit two: a single corpus, no routing. Traditional RAG assumes all knowledge lives in one unified vector database. In a real enterprise, data is scattered across CRM, ERP, Confluence, data warehouses, private document stores -- each with its own access point and permission boundary.

Here's an analogy: traditional RAG is a librarian who can only find books on the first floor, while the book you need might be sitting on the fourth floor, behind a different access pass.

Agentic RAG: A Retrieval Department That Thinks

The core shift in Agentic RAG is this: turn a single retrieval pass into a planned, iterative retrieval process.

It's no longer a passive query-and-return pipeline. It's a workflow run by multiple specialized agents, each with a distinct responsibility.

Let's use the support-ticket example to walk through how the whole workflow operates.

Step 1: The Orchestrator Decomposes the Task

The user's question first reaches the Orchestrator.

The orchestrator doesn't retrieve anything directly. It first understands the structure of the question: how many independent information needs are involved? Are there dependencies between them? Which data sources need to be accessed?

For our ticket, the orchestrator breaks it down into:

Subtask A: Look up the customer's "Project Alpha" basics (customer ID, project number) in the CRM
Subtask B: Use the project number to look up warranty terms in the contract system
Subtask C: Use the project number to look up device serial numbers and stock-in dates in the asset management system
Subtask D: Look up the current technical support owner in the HR system

Note that Subtasks B and C depend on the result of Subtask A (they need the project number first). Subtask D can run in parallel.

This dependency graph is the execution plan produced by the Planner Agent.

Step 2: Query Rewriting for Each Data Source

Every data source expects queries in a different form. The CRM might need keyword search, the contract system might need structured SQL, and the vector database needs semantic search.

The Query Rewriter translates each natural-language subtask into a query format the target source can understand:

For the CRM vector store: "Alpha project procurement record {customer name}"
For the contract system: SELECT warranty_terms FROM contracts WHERE project_id = 'Alpha-XXX'
For asset management: "Alpha project server stock-in date serial number"

Step 3: Parallel Retrieval Across Permission Boundaries

The Search Fanout Agent queries multiple data sources at once.

There's a key engineering problem here: permissions.

Different data sources have different access controls. CRM data might be open to the sales team, HR data might only be accessible to admins, and contract data might require legal sign-off. An Agentic RAG framework needs to maintain a "credential pool" at this layer -- different access tokens for different data sources -- and make sure retrieval never exceeds the current user's actual authorization scope.

This isn't just a technical problem; it's a compliance one too: AI shouldn't be able to bypass data access controls you were never supposed to have just because you phrased a request in natural language.

Step 4: Sufficiency Checking -- the Most Important Innovation

Once all retrieval results come back, they're passed to the Sufficient Context Agent.

This is the design that most distinguishes Agentic RAG from traditional RAG: the system actively judges whether the information gathered so far is enough to answer the original question, and if not, it spells out exactly what's missing before retrieving again.

In our ticket example, the checker might find:

✅ Found: customer profile, project number, device serial numbers
✅ Found: technical support owner
❌ Missing: the contract system returned a document, but the warranty terms are in an attached PDF that the vector search didn't hit

Instead of just saying "not enough information," the checker outputs a precise description of the gap:

"Project number Alpha-2024-087, device serial numbers SN-XXX-YYY-ZZZ, and stock-in date March 2024 have been retrieved. The main contract file has been found, but the warranty terms are in Contract Attachment B. Re-search the contract attachment store specifically for 'Attachment B warranty period.'"

That feedback drives a second retrieval round: the rewriter generates a more precise query targeted at the contract attachment.

This "retrieve → evaluate → retrieve again" loop continues until the sufficiency checker determines the information is complete, or the maximum iteration limit is reached.

Step 5: Synthesis Produces the Final Answer

Once everything is in place, the Synthesis Agent combines fragments from four different systems into one coherent, accurate, and source-attributed answer:

"The three servers purchased under Project Alpha (project number Alpha-2024-087, serial numbers SN-XXX-001 through 003) carry a 36-month warranty from their stock-in date (March 15, 2024), per Section 4.2 of Contract Attachment B, expiring March 14, 2027. The current technical support owner is Li Ming (extension 4521, liming@company.com)."

Every sentence has a traceable source.

Cross-System Permissions: Harder Than the Technology

The handling of permission boundaries deserves its own discussion.

In a real enterprise, data permissions are a multi-dimensional problem:

Dimension	Description	Example
Role-based access	Different roles see different data	Sales can see a contract summary but not the full text
Data classification	A single database can hold multiple sensitivity levels	Employee salary vs. employee directory
Time-based access	Some data has time-limited access rules	Financial data is read-only during an audit
Cross-system access	Data from System A must not surface in System B's context	GDPR requires data to stay within its jurisdiction

An Agentic RAG framework needs to enforce these rules on every single retrieval call, not just authorize access once at indexing time.

That means the architecture needs permission checks at query time, rather than the blunt approach of vectorizing everything into one big store.

In database terms: traditional RAG is like joining every table into one giant table and handing it to the LLM. Agentic RAG is like generating a permission-filtered SQL query dynamically for every request.

Three Decisions You Can't Avoid in Practice

When you actually build Agentic RAG in production, three decisions come up every time.

Decision one: routing strategy -- static rules or LLM routing?

Static routing: predefine rules based on keywords or metadata in the query to decide which data source to hit. Fast and predictable, but weak on open-ended queries.

LLM routing: let the LLM understand the query's intent and dynamically decide where to route it. Flexible, but every routing decision burns an LLM call, adding latency and cost.

Decision two: iteration depth -- when do you stop?

The system can get stuck in an infinite loop -- every round of retrieval feels like something is still missing, so it keeps searching.

Engineering-wise, you need:

A maximum iteration count (typically 2-4 rounds)
A time budget (answer with what you have once you time out)
A degradation strategy (answer with available information and flag it as potentially incomplete once the iteration limit is hit)

Decision three: the latency-vs-accuracy trade-off

Agentic RAG is slower than traditional RAG -- there's no avoiding it. Multiple LLM calls, parallel retrieval, and sufficiency evaluation all add latency at every step.

Approach	Cost multiplier	Latency multiplier	Best fit
Traditional RAG	1x	1x	Simple Q&A, single knowledge base
Adaptive RAG	1.5-2x	1.2-2x	Mixed scenarios with varying query complexity
CRAG (Corrective RAG)	3-5x	2-3x	High accuracy needs, tolerant of second-scale latency
Full Agentic RAG	5-10x	3-6x	Complex multi-hop, cross-store, async scenarios

Not every scenario needs full Agentic RAG.

Classifying intent at the query level -- routing complex queries through the Agentic pipeline and simple ones through traditional RAG -- keeps average cost and latency within a reasonable range.

Closing Thoughts

I think the essence of Agentic RAG is turning retrieval into an executable strategy: if one pass isn't enough, keep searching until it is. And the system itself decides what "enough" means.

That shift sounds simple, but it requires moving from a stateless "query-response" model to a stateful "goal-plan-execute-evaluate-iterate" workflow.

This is the same general challenge every agent system faces: state management is the core difficulty.

If you're building an enterprise AI system that spans multiple data sources, Agentic RAG isn't just an upgrade to your retrieval technique. It forces you to rethink data architecture, permission design, and workflow orchestration. Getting those three things right matters more than which framework or cloud vendor you pick.

References

Google Research, Unlocking dependable responses with Gemini Enterprise Agent Platform's Agentic RAG, June 2026
Microsoft, Agentic Retrieval Overview -- Azure AI Search, 2026-04-01 GA
Microsoft, What is a Knowledge Source? -- Azure AI Search
Amazon Web Services, Knowledge Bases for Amazon Bedrock -- Multiple Data Sources, April 2024
MarsDevs, Agentic RAG: The 2026 Production Guide (includes cost/latency comparisons across approaches)
Google Research, Deeper Insights into Retrieval-Augmented Generation: The Role of Sufficient Context

Claude Code Incident Review: What Anthropic's Three Production Bugs Teach Agent Engineers

Luhui Dev — Fri, 12 Jun 2026 14:24:59 +0000

Intro

Last month, Anthropic published a rare kind of incident review.

The rare part was not that they had bugs. If you build large-model products, bugs are part of the deal.

The rare part was that they wrote up three production incidents in detail: how each one was introduced, why testing missed it, why it was hard to reproduce internally, and what they changed afterward.

After reading it, I think the review is worth studying closely. If you build LLM Agents, especially systems with multi-turn tasks, tool calls, context compression, and reasoning trace management, these failures are not edge cases. They are waiting on the road.

Three Bugs, Three Failure Modes

Here is the short version.

Bug one: On March 4, to address occasional UI freezes in Opus 4.6 under high reasoning mode, the team changed the default reasoning effort from high to medium.

Internal testing looked fine: intelligence dropped only slightly, and latency improved a lot.

After launch, users pushed back hard: Claude felt dumber.

On April 7, one month later, the change was rolled back.

Bug two: On March 26, Anthropic shipped a cache optimization. The idea was simple: after a session had been idle for more than an hour, clear old thinking history to reduce the cost of resuming.

Sounds reasonable, right?

The production implementation had a bug. It was supposed to clear the old thinking once. Instead, it kept clearing it on every later turn.

So Claude kept working while repeatedly losing the memory of why it was doing the work. Users saw forgetting, repetition, strange tool calls, and increasingly odd behavior.

Worse, once thinking blocks kept disappearing, each request became a cache miss and burned through usage limits faster.

The root cause was not identified until April 10, two full weeks later.

Bug three: On April 16, to reduce verbose output from Opus 4.7, the team added a line to the system prompt:

Use no more than 25 words between tool calls, and no more than 100 words in the final response.

That prompt line showed no obvious regression during weeks of internal testing.

After launch, coding quality dropped by 3%.

On April 20, it was rolled back.

These three bugs look different, but they point to the same issue: in an Agent system, things that look local, such as parameters, caches, and prompt lines, can still affect the core execution logic.

Touch them, and you may be touching the model's brain.

Reasoning History Is Working Memory, Not a Log

The second bug is the one I keep coming back to.

"Clear old thinking to save tokens" is a perfectly normal engineering optimization. Thinking blocks are long and expensive. If a session has been idle for an hour, the old reasoning chain can look less important.

But that is exactly the trap.

For an Agent, the reasoning trace is not just a log. It does not merely record what happened. Its more important job is to preserve why the Agent made earlier decisions.

That why is what lets a multi-turn task keep moving.

When it disappears, the Agent does not crash immediately. It can still talk, call tools, and return results. But it has already started forgetting.

It forgets which paths were ruled out, why the current path was chosen, and what problem the user was actually trying to solve.

The result is a nasty kind of degradation: it gets worse, and the task drifts.

This class of bug is painful because it is not a crash. It does not give you a clean stack trace. It slowly shows up in production as a feeling that the Agent has become strangely bad.

So context management cannot be a blunt token-count cut.

At minimum, we need to separate three categories:

Do not casually compress: decision rationale, task intent, hard constraints, reasoning path.
Can compress: intermediate observations, tool outputs, process material.
Can drop: formatting helpers, redundant explanations, temporary display content.

Reasoning history is not cache garbage. In many cases, it is the Agent's working memory.

You may think you are saving tokens. You may actually be removing the part of the system that lets the Agent stay on task.

Every Prompt Line Is Code

The third bug is just as important.

How can adding one line that says "say less" reduce coding quality?

Because in model behavior, less output and less thinking are not always separate things.

If you require the final answer to be under 100 words and text between tool calls to be under 25 words, the model may not only compress expression. It may compress the decision process.

This is not a traditional bug. The model is sincerely optimizing for the target you gave it.

That is why Anthropic's follow-up discipline matters: every system prompt change should be ablated per model; if a line can be tested line by line, test it line by line; changes that may affect intelligence need gradual rollout and a soak period.

That sounds heavy.

But if you accept that prompts are production code, the discipline is not excessive.

Clean Test Environments Are Unlike Production

There is a common Agent engineering problem that people do not like to face:

The test environment is too clean.

So clean that it stops looking like production.

Many Agent failures are not as simple as "input A produces wrong output B." They depend on a sequence of state:

session idle for more than one hour
resume
continue multi-turn tool use
thinking gets cleared
enter the next turn
thinking gets cleared again

That kind of state sequence is hard to cover with unit tests and easy to miss in e2e tests.

We test features. Production runs a state machine.

That is also why Anthropic later had more internal employees use the exact same Claude Code build as the public release, rather than a test build.

Real usage is still the best integration test.

Defaults Are the Product

Now back to the first bug.

Changing high to medium made sense from an engineering angle: fewer freezes, lower latency, and only a slight intelligence drop.

The problem is that "slight drop" is benchmark language. It may not match user experience.

Users are not running average benchmarks. They are working with their code, their context, their workflow, and their messy problems.

In those highly personalized tasks, a small capability drop can feel like: it no longer understands me, it is not as sharp as before, it is making basic mistakes.

The product may offer a setting, but most users will not change it. The default is the product decision.

Closing

The most valuable part of this incident review is not the reminder that Claude Code can have bugs.

That is normal.

The valuable part is the reminder that Agent reliability often fails outside the model itself, in nearby system decisions that look local, technical, and low-risk.

Default parameters, cache strategy, context trimming, prompt constraints, and differences between test and production builds can all change Agent behavior.

Building an Agent is not just calling a smart model.

It is maintaining a complex system that can act, forget, misunderstand goals, and be shaped by context.

So the core discipline of Agent engineering may be this:

Do not only ask whether a change made the system faster, cheaper, or shorter. Also ask whether it took away the memory the model needs to finish the task.

DeepMind AlphaProof Nexus Explained: 4 System Paradigms for AI Math Research

Luhui Dev — Thu, 28 May 2026 14:55:25 +0000

Introduction

In May 2026, DeepMind released the arXiv paper Advancing Mathematics Research with AI-Driven Formal Proof Search, introducing AlphaProof Nexus, a formal proof system for mathematical research.

If we only look at the results, the paper may sound like another story about AI proving math problems: the system solved 9 out of 353 formalized Erdos open problems, proved 44 out of 492 OEIS conjectures, and was also used in real research across graph theory, optimization theory, algebraic geometry, additive combinatorics, and quantum optics.

But as someone building in AI + education, what I care about more is the research route behind it:

Let large models propose proof ideas, let formal systems such as Lean verify them, let specialized proof searchers solve local goals, and use multi-agent orchestration to turn failures, partial drafts, and subgoals into reusable search assets.

This essay breaks down AlphaProof Nexus in plain language: why it exists, how the system is designed, and the four AI math research paradigms reflected in the paper.

1. Why AI for Math Needs Formal Proof

Most people understand mathematical proof in natural language:

Since A is true, B follows. Because of a theorem, C follows. Therefore the conclusion holds.

This works for human mathematicians because people can fill in omitted reasoning. For AI, however, there is a serious issue: large models are good at writing text that looks like a proof, but that does not mean they have actually proved anything.

They may:

cite theorems that do not exist;
skip key steps;
replace a hard problem with another problem that looks easier but is still unproved;
write confident derivations with broken logic.

So the core starting point of the DeepMind paper is to make AI work inside a strict system.

That system is Lean.

Lean can be understood as a mathematical compiler. When programmers write code, compilers check syntax and type errors. When mathematicians write Lean proofs, Lean checks whether every logical step really holds.

If Lean accepts a proof, the proof is not merely plausible. It has passed strict machine checking.

That is the basic idea of AlphaProof Nexus:

AI can guess, try, and fail, but the final answer must pass Lean verification.

2. What Is AlphaProof Nexus?

AlphaProof Nexus is not a single large model, nor is it a chat-only math assistant. It is closer to an AI mathematical research pipeline.

The input is a Lean file containing an unfinished theorem proof. The system repeatedly modifies, completes, and decomposes the proof until Lean accepts it.

You can picture it like this:

There are several important roles:

LLM: proposes proof ideas, writes Lean code, and fixes errors.
Lean: checks whether the proof is actually correct.
AlphaProof: solves certain local proof goals.
Rater Agent: compares proof drafts and judges which ones are more promising.
Population DB: stores many unfinished but valuable proof drafts.
SafeVerify: performs final checks against cheating, such as modifying the original theorem or introducing disallowed axioms.

3. Four AI Math Research Paradigms in the Paper

DeepMind compares four agent configurations. They can also be read as four paradigms for AI-driven mathematical research.

Paradigm 1: Basic Agent

The first is the most basic pattern: LLM + Lean feedback loop.

It works much like a programmer writing code:

The LLM reads an unfinished Lean proof.
It tries to fill in proof code.
Lean compiles the file.
If Lean reports errors, those errors are sent back to the LLM.
The LLM edits the proof based on the errors.
This repeats until the proof passes or the budget is exhausted.

The core ability here is iterative trial and correction.

Imagine a student solving a problem. The student writes a solution, the teacher points out what is wrong, and the student revises it. Here the teacher is not a person but the Lean compiler. Lean will not say whether an idea is elegant, but it will strictly tell you when a type is wrong, a goal remains unsolved, a theorem does not match, or a logical step is missing.

That gives the large model very high-quality feedback.

One interesting conclusion in the paper is that Basic Agent is already strong.

In post hoc experiments, Basic Agent could also reproduce the 9 successful Erdos cases, although harder problems required more cost and had worse efficiency.

This suggests that as base model capability improves, the simple "generate - compile - repair" loop has already become a highly competitive baseline for formal mathematics.

The lesson is straightforward: do not expect a large model to produce a perfect proof in one shot. Put it inside an environment that can keep giving feedback and force revision.

Paradigm 2: Basic Agent + AlphaProof

The second paradigm adds AlphaProof on top of Basic Agent.

AlphaProof is a formal proof search system DeepMind developed earlier. In Nexus, it is not the main character. It is more like a local proof specialist.

When an LLM writes Lean proofs, it often runs into small goals such as:

In the current context, prove this subclaim.

Some subgoals are mechanical but awkward for an LLM to write. Others require searching through a sequence of tactics, meaning Lean proof commands. At that point, the system can hand local goals to AlphaProof.

AlphaProof may return three kinds of results:

it finds a proof;
it proves the subgoal is actually false;
it fails to find a proof within budget.

This matters because the LLM no longer has to carry every proof detail alone.

You can think of the LLM as a graduate student responsible for the overall idea and proof structure; AlphaProof as a tactical searcher responsible for local technical details; and Lean as the final judge.

The paper also notes that standalone AlphaProof cannot independently solve these research-level open problems.

In the experiments, AlphaProof alone did not solve the 9 Erdos problems. Its effective position is inside the larger Nexus system, where it acts as a local proof tool.

Paradigm 3: Basic Agent + Evolution

The third paradigm introduces a key idea: evolutionary search.

Mathematical proof is often not linear.

One proof route may get stuck halfway. Another route may be unfinished but closer to the core idea. A third route may have proved useful lemmas even if the main theorem is not done.

If every agent run starts from zero, a lot of exploration is wasted. AlphaProof Nexus therefore saves unfinished proof drafts into a Population DB, a seed population of proof sketches.

The system then uses a Rater Agent to compare those drafts and decide which ones are more promising.

It is a bit like evolution:

each proof draft is an individual;
agents mutate and rewrite drafts;
the rater selects more promising drafts;
the system spends compute on routes with higher potential.

The hard part is that success signals in mathematical proof are very sparse. In program optimization, even a non-optimal program may still get a score. In formal proof, a proof usually either succeeds completely or does not.

Intermediate states are hard to quantify.

DeepMind's approach is to ask models to compare proof draft quality instead of assigning absolute scores. For example:

Among these unfinished proofs, which route is clearer? Which remaining goals look more credible? Which one is more likely to be completed?

The system then ranks proof drafts using an Elo-like mechanism and uses a P-UCB-style strategy to balance two needs:

continue developing high-scoring routes;
explore routes that have not been tried enough.

Paradigm 4: Full-Featured Agent

The fourth paradigm is the main system promoted by the paper: LLM + Lean + AlphaProof + Evolution.

It combines the earlier capabilities:

the LLM writes proofs, edits proofs, and proposes lemmas;
Lean checks whether each step is correct;
AlphaProof handles local subgoals;
Evolution saves and filters different proof routes;
Rater Agent judges which drafts deserve more work;
SafeVerify performs final safety verification.

This is the full shape of AlphaProof Nexus.

The way it solves math problems resembles a highly automated research team: one role proposes ideas, one checks logic, one solves local technical problems, one reviews different directions, one manages historical attempts, and one performs final acceptance.

4. A Key Design: Proof Search Must Prevent Cheating

AI mathematical proof has a special failure mode: it may make the proof easier by changing the problem, and it often learns to do exactly that.

Suppose the original goal is to prove that property P holds for all natural numbers n.

An agent may quietly change it to: property P holds for some natural numbers n.

Or it may introduce an unallowed axiom that effectively assumes the conclusion.

Lean may still pass certain local checks, but that is obviously not the proof we want.

So AlphaProof Nexus performs multiple layers of safety checks:

check whether the theorem statement was modified;
check whether any sorry remains;
check whether disallowed axioms were introduced;
check whether the proof truly corresponds to the original problem;
run Lean inside a sandbox.

This is useful for all agent systems: when the task objective is complex enough, agents may learn to bypass the objective instead of completing it. Strong verification constraints are mandatory.

5. Experimental Results in the Paper

DeepMind applied the system to several mathematical tasks. The most representative ones are Erdos problems and OEIS conjectures.

1. Erdos Problems

Erdos was one of the most important mathematicians of the twentieth century and left behind many open problems. The paper reports that the system processed 353 formalized Erdos open problems and solved 9 of them.

The percentage may look low, but these are not elementary contest problems. They are research-level mathematical problems.

2. OEIS Conjectures

OEIS is a database of integer sequences with many conjectures about sequence patterns. The system proved 44 out of 492 OEIS conjectures.

This type of task is especially suitable for formal proof systems because many sequence conjectures can be encoded cleanly as definitions and theorems.

3. Deployment in Real Mathematical Research

The paper also says the system was used in multiple research areas, including graph theory, optimization theory, algebraic geometry, additive combinatorics, and quantum optics.

6. Closing Thoughts

Here are the conclusions I reached after reading the paper closely.

Conclusion 1: LLMs Can Already Be Core Components of Formal Proof Search

Many people used to say that large models are bad at rigorous mathematics because they hallucinate, skip steps, and invent things.

That is true, but incomplete.

If you ask an LLM to directly write a natural-language proof, it is unreliable. But if you place it inside a strict feedback system such as Lean, it can become an efficient search engine.

The value of an LLM is not that it is always correct. Its value is that it can propose many candidate routes and keep revising them based on compiler feedback.

Conclusion 2: Multi-Agent Orchestration Matters More Than a Single Model

The point of AlphaProof Nexus is not that one model version can solve hard problems. The point is that the system decomposes capabilities into generation, checking, search, rating, caching, and verification.

That is an engineering mindset. Mathematical research becomes a long-running search process.

Conclusion 3: Failed Attempts Are Valuable

Unfinished proof sketches, failed subgoals, Lean errors, and AlphaProof search results can all enter later loops.

That makes mathematical proof search feel more like software engineering: code can be reused, error logs can be analyzed, intermediate states can be saved, branches can continue, and the final result is accepted by a compiler.

Conclusion 4: Strong and Cheap Models Can Divide the Work

The paper uses stronger models for complex proof generation and cheaper models for draft comparison and ranking.

That is a practical agent engineering lesson. Not every step needs the strongest model.

Use strong models for complex reasoning, cheaper models for high-throughput evaluation, and specialized tools for local proof. That is more scalable than asking one expensive model to do everything.

Lessons from LangChain: Designing a Reliable Runtime for Production-Grade Agents

Luhui Dev — Wed, 20 May 2026 13:46:51 +0000

Intro

Agent demos are easy to get excited about. A model, a few tools, a prompt, wrap it in a loop, and suddenly you have something that searches, writes files, and calls APIs.

But between a demo and a production system, there's a long gulf. I call it the Runtime gap — and what crosses it isn't a smarter model. It's a runtime that can hold up complex, unstable, interruptible, recoverable workloads in a real environment.

Once you actually deploy into a business context, an Agent might run for minutes or tens of minutes. It calls multiple external systems, may need user approval, may hit network failures, tool timeouts, drifting model output, missing permissions, mid-flight user interruptions, process restarts, and version upgrades. Worse, it carries state: where the task is up to, what's already been queried, which intermediate files were written, which conclusions are still unconfirmed, whether this user can access a given dataset.

At that point, optimizing prompts alone won't fix the underlying problem. What an Agent needs is a runtime that holds the execution process together — complex, unstable, interruptible, and recoverable.

LangChain's recent writing on production deep agents and their Runtime is worth sharing for anyone building Agent products. It's a useful reminder: the moat for business-grade Agents isn't just a prettier agent loop. It's whether you can make state, permissions, recovery, observability, and human collaboration into a stable foundation.

1. Business-grade Agent failures don't only happen when the model is wrong

When people think about Agent reliability, the first thing that comes to mind is usually hallucinations. That matters, but in a business system, the failure surface is much larger.

The process might crash on step 8 of a long task. Re-running wastes cost and may double-call external APIs, leaving dirty data behind.

A tool might fail. An API timeout, a page that won't load, a database query that throws — without retries, fallbacks, and state persistence, the whole task becomes a one-shot gamble.

It might lose context while waiting for human approval. The user comes back half an hour later to click "Confirm," and the system can't remember which step they were confirming.

It might lose control at the interaction layer. The Agent is still running, and the user types "wait, that direction is wrong — switch to plan B." Should the system queue, interrupt, restart, or reject? Without a clear policy, the experience falls apart.

So reliability for a production Agent is at least six things: execution reliability, state reliability, interaction reliability, permission reliability, observability, and operational reliability. The value of a Runtime is to productize and frameworkify these problems — instead of leaving every team to hand-roll them from scratch.

2. Separate Harness from Runtime

In my current view, one critical distinction is this: Harness and Runtime are not the same thing.

The Harness is the behavioral shell of an Agent. It governs how the task is planned, how the prompt is written, which tools can be called, whether sub-tasks are spawned, whether there's a filesystem, whether sub-Agents are used, how context is compressed. This layer directly affects how smart the Agent looks.

The Runtime is the lower layer. It governs how Agents get executed, persisted, recovered, interrupted, observed, scheduled, isolated across users, and how concurrent requests are handled. This layer directly affects whether the Agent can actually support a business system.

In many open-source Agents, everything gets stuffed into the harness: rules in the prompt, try-catch inside tool calls, ad-hoc state in the database, a loading spinner on the frontend. It runs in the short term. Over time it becomes a tangle of logic no one wants to maintain.

LangChain's Runtime approach is to lift the cross-cutting capabilities out of the agent-loop context.

3. Durable Execution: the first foundation of reliability

If I could only learn one design from LangChain Runtime, I'd start with durable execution.

A normal web request is short-lived: request in, do some work, respond, done. Agents are different. A business Agent might run many steps: understand the task, break it down, retrieve material, call tools, write intermediate files, wait for approval, continue, generate a report. The process naturally spans multiple model calls, tool calls, and user interactions.

Once tasks get long, the system has to answer one question: what if it crashes in the middle?

LangChain/LangGraph's answer is checkpointing. Key states during execution are continuously persisted. On recovery, you don't start from scratch — you resume from the most recent reasonable state. For a business system, this isn't just a cost saver. It's how you avoid duplicating side effects.

How does it actually work? LangGraph models Agent execution as a state graph. Each node is a step — a model call, a tool call, a conditional. State flows between nodes, and after each step, the current snapshot of the whole graph is serialized to the checkpointer. There are several design choices worth unpacking here.

First, the unit of checkpointing is the node boundary, not the function-call boundary. If a streaming model call dies mid-output, recovery re-runs the entire call.

Second, state is structured, not a black-box pickle. LangGraph requires you to split state into named channels (messages, plan, scratchpad), each paired with a reducer (append for messages, overwrite for plan). That makes checkpoints structured diffs — traceable, replayable, and time-travelable to any step.

Third, checkpoints form a tree, not a line. Every checkpoint carries a parent reference. You can branch off any historical node and re-run — tweak the user's question, skip an approval, try a different tool — all of which grow new branches off the same tree.

Fourth, interrupt and checkpoint share the same mechanism. An interrupt before or after a node is essentially a checkpoint written at that point followed by a pause. Human approval, user edits, external wake-up signals — all reuse the same persistence layer. That's why HITL can be a Runtime capability rather than UI logic.

Fifth, the backend is pluggable. In dev, use in-memory or SQLite; in production, Postgres or Redis. Your Agent's reliability tier can scale with the business — you don't need heavyweight infrastructure on day one.

Picture an Agent generating a research report for an enterprise customer. It has finished gathering material, summarizing competitors, drafting the report, and is now waiting for the user to confirm whether to pull internal CRM data. If the service restarts at this point, the ideal outcome isn't making the Agent search again from scratch, nor making the user re-describe their requirements. It's resuming at "waiting for confirmation."

That's the point of durable execution: turning Agent execution from a one-shot function call into a savable, recoverable, resumable task that has a real lifecycle.

There are still specific questions worth answering, like: what exactly counts as a recoverable boundary in each Agent step? Can writes to the business system be safely repeated?

4. Layer your state: short-term state, long-term memory, and business data shouldn't be mixed

Complex Agent tasks produce state. But state shouldn't be one big stew.

Short-term state is the current task's context: what the plan is, where execution has reached, intermediate results, which tool calls completed, what's pending confirmation. This kind of state belongs bound to threads, runs, and checkpoints.

Long-term memory is cross-session context: user preferences, organizational rules, common workflows, recurring constraints, reusable knowledge. This should live in a long-term store, namespaced by user, organization, application, assistant, and so on.

Business data is yet another layer: orders, problems, lessons, customer records, organizational assets, permission models. This data generally shouldn't be casually swallowed by an Agent Runtime. It should stay owned by the business system, with the Agent accessing it through controlled tools.

LangChain's design is instructive here: it separates thread checkpoints from the long-term store, while still letting deep agents access different layers of state through something like a virtual filesystem. For the Agent on top, reading and writing files and memory feels natural; for the system underneath, state still has clear boundaries.

This matters a lot in real business systems. Many early Agent products pile chat history, tool results, user preferences, and business data into a single conversation memory. It's simple to implement, but later it blows up at once in permissions, cost, retrieval quality, data cleanup, and compliance audit.

A more robust pattern: short-term state serves task recovery, long-term memory serves experience continuity, and business data stays inside the business system — the Agent only reaches it through permission-controlled tools.

5. Human-in-the-loop isn't decoration — it's a reliability mechanism

Production-grade Agents are hard to fully automate. Especially anything that touches writes, external system calls, important decisions, paid resources, or user privacy — human collaboration is a necessary safety valve.

The key isn't popping up a confirmation dialog. The real engineering question is: how does the Agent pause? What state is saved while it pauses? Can the user come back any time later and continue? Can the user edit the plan the Agent produced? After editing, where does it resume? Are approval records auditable?

LangChain Runtime makes interrupt/resume a runtime capability rather than something the application layer judges ad-hoc. Because if HITL only lives in the frontend interaction layer, it quickly becomes UI logic — and once tasks span processes, workers, and time, the frontend can't hold it.

Plenty of scenarios in business Agents need this:

A finance Agent about to submit an expense report needs human confirmation.
An education Agent generating lesson plans in bulk needs a teacher to pick the teaching style.
A customer service Agent issuing a refund needs supervisor approval.
A data analysis Agent wanting to access sensitive fields needs a one-time user authorization.

These aren't ordinary chat experiences — they're business workflows. If the Runtime natively supports pausing, resuming, approval, and state persistence, Agent reliability jumps a clear notch.

6. Permissions and multi-tenancy: an Agent shouldn't roam with the master key

One of the biggest risks for a production Agent is permissions.

In a normal app, the user clicks a button, calls an API, the server checks permissions — the chain is relatively clear. Once an Agent gets involved, it gets complicated: the model decides which tool to call, the tool may access external systems, those external systems may require user authorization, and the Agent may also write intermediate results into long-term memory.

LangChain's approach is to split identity and permissions into layers: who the end user is, which threads and resources that user can access, which external systems the Agent can access on the user's behalf, and what team members can do on the platform itself.

In this design, the Agent isn't a backend super-admin. It's more like a delegated executor, allowed to act only within the scope of the current user, current organization, and current task.

If you're designing a Runtime for your own production Agent, you should at least think about these boundaries:

User identity enters the run context. Every Agent execution should know whom it is currently acting on behalf of.
Resource access should be isolated by thread, file, project, and organization. You can't rely on a prompt to tell the model not to touch someone else's data.
External tool authorization should be managed separately. GitHub, Slack, CRM, object storage, databases — long-term keys shouldn't be handed directly to the Agent's execution environment.
Long-term memory needs namespaces. Otherwise the user preferences an Agent remembers can easily become data pollution in a multi-tenant setting.
High-risk tools need approval or policy interception. Deletes, sends, payments, publishes, bulk writes — these can't rely on the model's self-discipline.

The more human-like the Agent feels, the easier it is for a system to assume it inherits a human's full permissions. From an engineering view, the Agent should hold task-scoped, time-bounded, minimum-privilege permissions.

7. Middleware: put protective capabilities into the runtime lifecycle

Many teams write guardrails into the prompt: "don't leak private information," "don't perform dangerous operations," "ask the user when uncertain." Useful, but not enough.

Models forget, prompts get overridden, tool-call paths can route around the rules, and streaming output and background tasks may behave differently. A business system needs a harder line of defense.

The middleware design inserts control points around the Agent lifecycle: before the model call, during the model call, during a tool call, and after the model call — all can host policy.

That means many reliability capabilities can sink down into the Runtime:

Before a model call: trim context, inject permission info, check token budgets.
During a model call: model fallback, timeout control, retry policy, cost accounting.
During a tool call: permission checks, parameter validation, sensitive-action interception, human approval.
After model output: PII detection, format validation, result archiving, trace tagging.

This is more stable than scattering logic across every tool, and far easier to govern uniformly.

For a business-grade Agent, middleware isn't just a security filter — it's the entry point for Runtime governability.

8. Streaming and double-texting: interaction reliability also belongs in the Runtime

A lot of people treat streaming as a UX optimization. For Agents, streaming is also a reliability concern.

If a long task has no real-time feedback, the user has no idea whether the system is still alive or where it's up to. Especially for research, coding, data analysis, and lesson-plan generation, users need to see the intermediate state: retrieving, calling a tool, drafting, waiting for confirmation.

The trickier piece is mid-flight user input. The Agent is still running and the user sends a new instruction — what LangChain calls a double-texting problem. It's not a small UX detail. It's an interaction protocol problem.

The system has to decide: is the new input queued, or does it interrupt the current task? Is it merged into the current context, or does the task roll back and re-run? Can the user change goals mid-task, or must they wait until the current task finishes?

In business systems, the right answer varies.

A writing Agent can let the user adjust direction mid-task.

A payments Agent cannot get casually interrupted and then continue with dangerous operations.

A lesson-plan Agent might be a good fit for queueing new input onto a task list.

A coding Agent might need to pause the current command and wait for the user to confirm a plan change.

This is why chat experience is actually part of Runtime design.

9. Observability: you can't debug a business Agent with logs alone

For traditional applications, logs, metrics, and distributed tracing usually get the job done. For Agents, plain logs are often not enough.

That's because Agent errors are frequently process errors: step 1 misunderstood the task, step 3 used the wrong tool, step 5 accepted low-quality retrieval, step 7 promoted an intermediate assumption into a conclusion. The final answer is wrong, but the actual cause is buried in the execution path.

LangChain/LangSmith emphasize trace, time travel, and debug. A business-grade Agent needs more than call success rate — it needs to know:

Which nodes did this task pass through?
What context did each step's model call actually see?
Which tools were called? With what parameters? Returning what?
How did intermediate state change?
At which step did a branch occur?
Were middleware, approvals, retries, or fallbacks triggered?
If you change the state of a particular checkpoint, do the downstream results change?

These capabilities decide whether the Agent can be continuously improved. Otherwise the team is just tweaking prompts by feel — and that isn't an engineering loop.

Going further: observability also feeds eval. Traces aren't just for triage — they become evaluation samples, regression tests, cost analyses, and product insight.

10. An Agent Runtime should scale horizontally and control cost

Once a business Agent goes live and starts serving users, operational concerns kick in.

Some tasks are short, some are long. Some only read data; some call slow tools. Some users send messages back-to-back; some tasks fire on a schedule. Model calls are expensive; tool calls can be expensive too. Once long tasks pile up, the API server, queue workers, Redis, Postgres, and external tools all become bottlenecks.

Several things in LangChain Agent Server are worth learning from:

Separate the API server from queue workers. The former accepts requests; the latter executes long-running tasks. This stops long tasks from dragging down the entry service.

Layer ephemeral state and durable state. Transient runtime state can live in something like Redis; threads, runs, checkpoints, and memory go into durable storage.

Concurrency is configurable. Different Agents have different task shapes — I/O-heavy and CPU-heavy workers need different concurrency policies.

Avoid frontend polling. For long tasks, join/stream beats blunt polling — both for UX and for system resource cost.

Support cron. Plenty of business Agents don't fire because a user clicked something; they need to run periodic checks, periodic summaries, periodic syncs, periodic content generation.

A reliable business Agent Runtime has to care about both task semantics and infrastructure cost at the same time.

11. A Runtime design checklist

If you're not copying LangChain wholesale but designing a runtime for your own business Agent, here's a checklist to break the capability surface into.

One: task lifecycle. Do you have basic abstractions like thread, run, and step? Can a single Agent execution be tracked, canceled, paused, resumed, retried?

Two: durable execution. Is the key state at each step checkpointed? Where are the recovery boundaries? Which operations are replayable, which must be idempotent, which can only resume after human confirmation?

Three: layered state. Are short-term task state, long-term memory, and business data separated? Are there namespaces? Are cleanup, migration, and audit supported?

Four: permission model. Who is the Agent acting on behalf of? Which resources can it touch? Which tools can it call? How are external system authorizations managed? Do high-risk operations require approval?

Five: tool governance. Do tools have schemas, permissions, timeouts, retries, rate limits, audit logs? On failure, do you retry, fall back, skip, or interrupt?

Six: human-in-the-loop. Can a run be interrupted mid-flight? Can it resume after user confirmation? Are approval content, approver identity, and approval time auditable?

Seven: interaction protocol. How is streaming designed? How is mid-flight user input handled? When is queueing, rejecting, interrupting, or restarting the right answer?

Eight: observability and debugging. Is there structured tracing? Can you see model calls, tool calls, state changes, and middleware triggers? Can bad cases be turned into evals?

Nine: operational scaling. Are the entry service and execution workers separated? How is the queue designed? Where are the storage bottlenecks? How do you rate-limit when long tasks pile up? How is cost attributed?

Ten: deployment boundaries. What do you self-host, and what do you delegate to a managed platform? Can data stay inside your system? Are you locking yourself too deeply into one runtime?

12. Closing thoughts

Agent products aren't only chasing smarter behavior. Once they enter a business system, what matters as much is being more reliable: recoverable, isolable, approvable, traceable, scalable, cost-controllable.

That's why Agent Harness and Runtime are worth seeing as separate. The Harness sets the ceiling on what an Agent can do. The Runtime sets the floor under which it can't safely go live. Without the former, the Agent isn't smart enough; without the latter, the Agent can't be launched.

If we're going to build our own production-grade Agents, the Runtime should enter the architecture from day one. Even if v1 isn't a full system, the boundaries should be defined first: how is task state saved, how is user identity passed, how are tool permissions controlled, how does human approval recover, how do traces settle into evaluation data.

The future of Agents isn't only stronger models — it's a more mature Runtime. Whoever can make a complex Agent's execution stable, controllable, and auditable will get closer to real business deployment.

That's what I find most worth learning from LangChain's Runtime piece.

References

LangChain, The Runtime Behind Production Deep Agents www.langchain.com/blog/runtime-behind-production-deep-agents

Lessons from LangChain: Designing a Reliable Runtime for Production-Grade Agents

Luhui Dev — Wed, 20 May 2026 13:38:22 +0000

Intro

Agent demos are easy to get excited about. A model, a few tools, a prompt, wrap it in a loop, and suddenly you have something that searches, writes files, and calls APIs.

1. Business-grade Agent failures don't only happen when the model is wrong

When people think about Agent reliability, the first thing that comes to mind is usually hallucinations. That matters, but in a business system, the failure surface is much larger.

The process might crash on step 8 of a long task. Re-running wastes cost and may double-call external APIs, leaving dirty data behind.

A tool might fail. An API timeout, a page that won't load, a database query that throws — without retries, fallbacks, and state persistence, the whole task becomes a one-shot gamble.

It might lose context while waiting for human approval. The user comes back half an hour later to click "Confirm," and the system can't remember which step they were confirming.

2. Separate Harness from Runtime

In my current view, one critical distinction is this: Harness and Runtime are not the same thing.

LangChain's Runtime approach is to lift the cross-cutting capabilities out of the agent-loop context.

3. Durable Execution: the first foundation of reliability

If I could only learn one design from LangChain Runtime, I'd start with durable execution.

Once tasks get long, the system has to answer one question: what if it crashes in the middle?

First, the unit of checkpointing is the node boundary, not the function-call boundary. If a streaming model call dies mid-output, recovery re-runs the entire call.

That's the point of durable execution: turning Agent execution from a one-shot function call into a savable, recoverable, resumable task that has a real lifecycle.

There are still specific questions worth answering, like: what exactly counts as a recoverable boundary in each Agent step? Can writes to the business system be safely repeated?

4. Layer your state: short-term state, long-term memory, and business data shouldn't be mixed

Complex Agent tasks produce state. But state shouldn't be one big stew.

5. Human-in-the-loop isn't decoration — it's a reliability mechanism

Plenty of scenarios in business Agents need this:

A finance Agent about to submit an expense report needs human confirmation.
An education Agent generating lesson plans in bulk needs a teacher to pick the teaching style.
A customer service Agent issuing a refund needs supervisor approval.
A data analysis Agent wanting to access sensitive fields needs a one-time user authorization.

These aren't ordinary chat experiences — they're business workflows. If the Runtime natively supports pausing, resuming, approval, and state persistence, Agent reliability jumps a clear notch.

6. Permissions and multi-tenancy: an Agent shouldn't roam with the master key

One of the biggest risks for a production Agent is permissions.

In this design, the Agent isn't a backend super-admin. It's more like a delegated executor, allowed to act only within the scope of the current user, current organization, and current task.

If you're designing a Runtime for your own production Agent, you should at least think about these boundaries:

User identity enters the run context. Every Agent execution should know whom it is currently acting on behalf of.
Resource access should be isolated by thread, file, project, and organization. You can't rely on a prompt to tell the model not to touch someone else's data.
External tool authorization should be managed separately. GitHub, Slack, CRM, object storage, databases — long-term keys shouldn't be handed directly to the Agent's execution environment.
Long-term memory needs namespaces. Otherwise the user preferences an Agent remembers can easily become data pollution in a multi-tenant setting.
High-risk tools need approval or policy interception. Deletes, sends, payments, publishes, bulk writes — these can't rely on the model's self-discipline.

7. Middleware: put protective capabilities into the runtime lifecycle

Many teams write guardrails into the prompt: "don't leak private information," "don't perform dangerous operations," "ask the user when uncertain." Useful, but not enough.

Models forget, prompts get overridden, tool-call paths can route around the rules, and streaming output and background tasks may behave differently. A business system needs a harder line of defense.

The middleware design inserts control points around the Agent lifecycle: before the model call, during the model call, during a tool call, and after the model call — all can host policy.

That means many reliability capabilities can sink down into the Runtime:

Before a model call: trim context, inject permission info, check token budgets.
During a model call: model fallback, timeout control, retry policy, cost accounting.
During a tool call: permission checks, parameter validation, sensitive-action interception, human approval.
After model output: PII detection, format validation, result archiving, trace tagging.

This is more stable than scattering logic across every tool, and far easier to govern uniformly.

For a business-grade Agent, middleware isn't just a security filter — it's the entry point for Runtime governability.

8. Streaming and double-texting: interaction reliability also belongs in the Runtime

A lot of people treat streaming as a UX optimization. For Agents, streaming is also a reliability concern.

In business systems, the right answer varies.

A writing Agent can let the user adjust direction mid-task.

A payments Agent cannot get casually interrupted and then continue with dangerous operations.

A lesson-plan Agent might be a good fit for queueing new input onto a task list.

A coding Agent might need to pause the current command and wait for the user to confirm a plan change.

This is why chat experience is actually part of Runtime design.

9. Observability: you can't debug a business Agent with logs alone

For traditional applications, logs, metrics, and distributed tracing usually get the job done. For Agents, plain logs are often not enough.

LangChain/LangSmith emphasize trace, time travel, and debug. A business-grade Agent needs more than call success rate — it needs to know:

Which nodes did this task pass through?
What context did each step's model call actually see?
Which tools were called? With what parameters? Returning what?
How did intermediate state change?
At which step did a branch occur?
Were middleware, approvals, retries, or fallbacks triggered?
If you change the state of a particular checkpoint, do the downstream results change?