DEV Community: Takafumi Endo | ROUTE06

Reverse-Engineering-Driven Harness Engineering for Existing Applications

Takafumi Endo | ROUTE06 — Fri, 16 Oct 2026 02:17:53 +0000

AI-driven legacy modernization is often discussed in terms of understanding, refactoring, or rewriting existing codebases.

But with AI coding agents such as Claude Code and Codex, another direction is becoming increasingly practical: enabling agents to operate and extend existing applications through their existing interfaces, without necessarily rewriting their code.

This is especially relevant to low-code platforms and enterprise systems, where business logic is often distributed across GUI configurations, workflows, metadata, and operational procedures.

Recently, I've been applying an approach that combines reverse engineering, harness engineering, operations research (OR), and runtime engineering to this problem.

The idea is straightforward: discover how an application behaves, convert that knowledge into reusable operational capabilities, and provide an execution environment in which AI agents can achieve business goals while respecting the application's constraints.

This article describes the approach as a practical system design and implementation framework.

1. The Problem Is Not Just GUI Automation

Consider adding an approval workflow to an existing low-code application.

A developer would need to inspect the current application, understand its data and workflow structures, identify the relevant configuration screens, make changes, and verify the result.

An AI agent needs to do essentially the same work.

Providing GUI automation alone does not solve this problem. The agent may be able to click buttons and enter values, but it still needs to understand:

Which operations are available and when they are valid.
Which application states and permissions are required.
How a change affects existing functions.
Whether an operation actually persisted its intended result.
How to verify that the business requirement has been satisfied.

This requires more than tools. It requires application-specific operational knowledge.

GUI automation provides actions. Harness engineering provides the knowledge and controls needed to use those actions reliably.

The objective is not merely to automate an existing workflow. It is to enable AI agents to operate, configure, and extend existing applications through supported interfaces.

2. The Architecture

I've been organizing this work into five connected phases:

Goal Definition: Define the required business outcome and acceptance criteria.
Reverse Engineering: Discover application behavior, states, dependencies, and constraints.
Harness Construction: Convert discovered knowledge into executable operational capabilities.
Agent Execution: Plan, operate, observe, verify, and recover.
Goal Evaluation: Determine whether the business outcome has been achieved.

The system architecture looks like this:

flowchart TB
    G["Business Goal<br/>Acceptance Criteria"]
    A["AI Agent<br/>Claude Code / Codex"]

    subgraph H["Agent Harness"]
        direction TB
        E["Executable Operational Model<br/>States · Actions · Effects · Constraints"]
        P["Planner<br/>Strategy Selection"]
        R["Runtime<br/>Execution · Observation · Recovery"]
        V["Evaluator<br/>Business Rules · Regression"]
    end

    S["Existing Application<br/>GUI · Configuration · Metadata"]
    RE["Reverse Engineering<br/>Observe · Analyze · Validate"]

    G --> A
    A --> E
    E --> P
    P --> R
    R --> S
    S --> RE
    RE --> E
    R --> V
    V --> G

The four engineering disciplines are not separate systems. They represent different responsibilities within this process.

Reverse Engineering discovers and validates application behavior.
Harness Engineering organizes that knowledge into reusable capabilities and controls.
Operations Research supports action selection under cost, risk, and operational constraints.
Runtime Engineering handles reliable execution, state tracking, verification, and recovery.

The key artifact connecting these activities is what I call an Executable Operational Model (EOM).

3. From Reverse Engineering to Executable Operational Knowledge

Traditional reverse engineering often produces documentation, design models, or recovered specifications.

Here, the goal is different.

The discovered knowledge must be usable by an AI agent during execution.

For low-code applications, reverse engineering may involve GUI observation, operation recording, input/output comparison, state-transition analysis, and inspection of available metadata or generated code.

For example, while exploring an approval workflow designer, the agent might discover that:

A workflow must be unlocked before modification.
Saving a change does not automatically activate it.
Certain configuration changes affect dependent workflows.
Validation errors become visible only after a specific operation.

These observations can be converted into structured operational knowledge.

An EOM describes not only how to perform an operation, but also its expected effects, constraints, and verification conditions.

Element	Purpose
State	Current or inferred application state
Action	An operation available through a supported interface
Preconditions	Conditions required before execution
Effects	Expected changes resulting from the operation
Invariants	Conditions that must remain true
Verification	How actual effects are confirmed
Recovery	How failures or uncertain outcomes are handled
Confidence	Whether knowledge is observed, hypothesized, or validated

Consider a workflow configuration operation:

operation: update_approval_workflow

preconditions:
  - design_is_unlocked
  - user_has_permission

actions:
  - inspect_current_configuration
  - update_approval_rules
  - save_configuration

expected_effects:
  - workflow_configuration_updated

invariants:
  - existing_records_preserved
  - unrelated_workflows_unchanged

verification:
  - reopen_configuration
  - compare_with_expected_rules
  - run_workflow_tests

recovery:
  - inspect_actual_state
  - restore_previous_configuration_if_safe

knowledge_status: validated

This is a simplified example, not a required schema.

A conventional Skill may describe how to perform a task. The EOM provides the broader operational context: when actions are valid, how they change state, what must remain unchanged, and how correctness is established.

In practice, the EOM does not need to exist as a single file or formal modeling engine. It can be distributed across Skills, configuration files, tools, state models, and test definitions.

The important transformation is:

Reverse engineering discovers application behavior. Harness engineering converts that knowledge into executable operational capabilities.

4. Harness Engineering: Building the Execution Environment

The harness connects the AI agent to the existing application.

For Claude Code or Codex, it may include application instructions, Skills, GUI tools, CLI commands, MCP integrations, state inspection utilities, evaluators, and recovery procedures.

These components serve different purposes.

Instructions and Skills provide reusable operational knowledge. Tools expose supported application interfaces. State models and policies determine which actions are valid. Evaluators establish whether the intended results were achieved.

The harness should also distinguish between observing an application and modifying it.

For example, generated code or metadata may be useful for understanding structure and dependencies. However, directly modifying generated code may bypass a low-code platform's supported configuration mechanisms.

In such systems, I prefer using code analysis for understanding and impact review while performing changes through the platform's supported design interfaces.

The harness therefore becomes an application-specific engineering environment, rather than just a set of commands available to an LLM.

5. Planning: Choosing What to Do and What to Investigate

Once the agent has operational capabilities, it must decide how to use them.

There may be several ways to reach the same goal.

For example, changing approval rules might involve a known GUI procedure, a shorter but less-tested sequence, or a supported administrative interface.

The shortest procedure is not always the best choice.

Operations research provides useful concepts for selecting strategies under constraints.

In practice, planning can consider:

Expected execution time and tool cost.
Dependencies between operations.
Reliability of previously observed procedures.
Risk of unintended changes.
Required permissions and approval boundaries.
Uncertainty about the current application state.

A known procedure may be preferable for production changes, while an unfamiliar but potentially faster approach may be worth exploring in a sandbox.

There is also another decision: whether to act immediately or gather more information first.

Suppose the agent does not know whether saving a workflow also activates it.

It could proceed based on an assumption, inspect the current configuration, or test the behavior in an isolated environment.

Investigation has a cost, but it can reduce the risk of an incorrect change.

This connects the approach to ideas such as value of information, exploration versus exploitation, and planning under partial observability.

I do not assume that every implementation needs a mathematical optimizer or a POMDP solver. Dependency graphs, policy checks, heuristic scoring, and risk-based replanning are often more practical starting points.

The important design principle is to separate responsibilities:

The LLM interprets goals and reasons about unfamiliar situations. The EOM supplies application-specific knowledge. Planning logic selects feasible strategies, while policies enforce hard constraints.

This makes action selection more explicit and easier to evaluate.

6. Runtime Engineering: Handling Uncertain Execution

Even a well-planned operation can fail.

GUI interfaces are often asynchronous, stateful, and difficult to control precisely.

Consider a simple example.

The agent updates an approval rule and clicks Save. The application times out.

At that moment, the agent does not know whether the change failed, succeeded, or partially completed.

Blindly repeating the operation may be unsafe.

This illustrates a fundamental distinction:

Action completed ≠ State changed ≠ Business goal achieved.

A reliable runtime needs to handle these as separate conditions.

Execution state and checkpoints

The runtime should record the intended action, application state before execution, observed result, and subsequent verification.

If execution is interrupted, the agent should be able to determine what has already happened before continuing.

Effect reconciliation

When an operation produces an uncertain result, the runtime should inspect the actual application state before retrying.

For example, after a save timeout, it may reopen the workflow configuration and compare the stored values against the intended changes.

The principle is simple:

Observe the actual state before repeating an uncertain operation.

Recovery and compensation

GUI-driven changes may not support atomic transactions or reliable rollback.

Recovery may require compensating operations, such as restoring a previous configuration through the supported interface.

Such recovery must also be verified, since compensation is not necessarily equivalent to reverting a database transaction.

Approval boundaries

Not every operation should be fully autonomous.

Read-only inspection and sandbox exploration may require little supervision. Production configuration changes, destructive operations, or irreversible actions may require explicit human approval.

The runtime must enforce these boundaries rather than relying on the LLM to remember them.

Together, these mechanisms provide the reliability needed for application engineering through existing interfaces.

7. A Practical Example: Extending a Low-Code Application

To see how the components fit together, consider adding a new approval workflow to an existing order-management application.

This is an illustrative scenario, not a claim of measured experimental results.

Step 1 — Define the goal

The new workflow must allow authorized users to submit orders for approval, apply routing rules, preserve approval history, and avoid breaking existing order processing.

These become explicit acceptance criteria.

Step 2 — Discover the application

The agent inspects the existing forms, data objects, workflow designer, and approval settings.

Through controlled operations, it identifies relevant dependencies and discovers that workflows must be unlocked before editing and separately activated after saving.

These findings are recorded with their validation status.

Step 3 — Update the EOM and harness

The discovered operations become reusable procedures.

The harness records editing prerequisites, expected configuration changes, validation steps, and recovery options.

It also provides tools to inspect the current workflow state and compare configurations before and after modification.

Step 4 — Plan and execute

The agent selects a modification sequence based on dependencies, known procedures, and operational constraints.

It uses the low-code designer to create or update the required workflow.

After each significant change, the runtime checks whether the expected state was reached.

If an operation fails or produces an uncertain result, the agent reconciles the actual state before retrying or replanning.

Step 5 — Evaluate the business outcome

The evaluator checks whether approval routing works, permissions are enforced, approval history is retained, and existing order-processing tests still pass.

If a test fails, the agent investigates the difference between expected and observed behavior.

Newly discovered constraints or successful recovery procedures can then be proposed as improvements to the harness.

The artifacts produced at each step are:

Step	Engineering artifact
Goal Definition	Business acceptance criteria
Reverse Engineering	Observations, state knowledge, dependencies
Harness Construction	Operational model, Skills, policies, evaluators
Execution	Action history, state evidence, recovery records
Evaluation	Test results and validated improvement candidates

This is what makes the approach more than GUI automation.

The agent is not following a fixed script. It is performing application engineering using accumulated operational knowledge, feedback, and explicit verification.

8. Continuous Improvement of the Harness

Reverse engineering does not end when the initial harness is built.

Every execution can reveal new information.

A failed operation may expose an unknown precondition. An unexpected configuration state may reveal a missing dependency. A more reliable procedure may emerge from repeated execution.

These findings can improve the EOM, Skills, evaluators, and recovery logic.

However, learning must be controlled.

An observed behavior should not automatically become a business rule. A successful workaround should not immediately replace a validated procedure.

Changes to operational knowledge need evidence, validation, versioning, and appropriate review.

This is particularly important when the harness supports application modifications rather than read-only operations.

The central idea is:

The harness is not merely built from reverse engineering. It becomes the mechanism through which reverse engineering continues.

Over time, this can make application-specific knowledge reusable across tasks and agent sessions instead of requiring the agent to rediscover everything from the GUI.

9. Scope, Limitations, and Evaluation

The framework primarily targets three kinds of work:

Operation: Using existing application functions.
Configuration: Modifying supported settings and workflows.
Extension: Building new functionality through existing design interfaces.

It does not require rewriting the application's implementation. However, it cannot create functionality that the underlying platform does not support.

Several limitations remain.

GUI observation provides only a partial view of application behavior. Interface changes can invalidate previously discovered procedures. Some operations are difficult to make idempotent, and business correctness may require domain experts or independent test systems.

The approach also introduces an initial engineering cost: discovering application behavior, defining operational knowledge, building tools, and implementing evaluators.

Whether this investment produces a net benefit depends on how frequently the harness is reused and how effectively it improves execution.

A useful evaluation would compare three configurations:

A general-purpose GUI agent.
A GUI agent with manually defined Skills.
An agent using discovered operational knowledge, constrained planning, and a reliability-oriented harness.

Relevant metrics include goal completion rate, acceptance-criteria coverage, recovery success, regression violations, execution cost, and adaptation effort after application changes.

Such comparisons are needed to quantify the benefits. The architecture itself does not establish performance improvements.

Conclusion

AI-driven modernization does not always need to begin with rewriting existing code.

For many applications, another useful direction is to build the knowledge and execution environment that allows AI agents to operate and extend them through supported interfaces.

The approach I've been applying connects four engineering concerns:

Reverse Engineering discovers application behavior and constraints.

Harness Engineering converts that knowledge into reusable operational capabilities.

Operations Research supports decisions about which actions to take and when to gather more information.

Runtime Engineering provides reliable execution, state verification, and recovery.

The common foundation is an Executable Operational Model: a reusable representation of application states, operations, expected effects, constraints, and verification procedures.

The most important change is not simply that an AI agent can interact with an existing application.

It is that knowledge discovered from the application becomes an engineering asset that can guide future operations, support application changes, and improve through verified execution.

That creates another path for evolving existing systems: not necessarily by replacing their implementation, but by engineering how AI agents can understand, develop, and operate them.

Code as a Model of the Current Spec: Declarative Models, IRs, and Graphs for AI Coding

Takafumi Endo | ROUTE06 — Fri, 09 Oct 2026 09:45:09 +0000

I am putting this design direction into practice and evaluating it in medium-sized and larger repositories.

Database and infrastructure models, along with verification mechanisms, are already implemented in our tooling. UI/API models and cross-domain composition of migration phases are the next direction.

I currently use a custom harness for development, including database changes. I also use Drizzle in some projects. Across both approaches, I keep coming back to the same friction: a coding agent can produce a locally reasonable change without giving enough attention to the shape of the system as a whole. Accumulating migrations is one place where that becomes visible.

My impression is that newer agents are much better at this than they were a year ago. But the architectural question remains.

How do we make the repository express the system well enough that an agent can understand what a change must preserve?

My target is simple to state:

Make code the best available model of the current specification. Derive intermediate representations from it so changes can be analyzed, rather than judged from text alone.

That applies to UI, API, workflow, database, and infrastructure together. Consider a rule as simple as “after account deactivation, no new billing operation may begin.” Preserving it during a schema change can require agreement across all five domains. I will use that example throughout.

The repository should explain the system as it is

When an agent opens a repository, it should be able to discover the current business concepts, responsibilities, constraints, and dependencies from its structure.

A billing decision hidden inside a generic utility function is harder to understand than a named policy at the billing boundary. A workflow assembled from scattered callbacks is harder to inspect than explicit states and transitions. A permission enforced inconsistently across handlers is harder to reason about than a clear authorization boundary.

The ideal is not to maintain another specification document alongside the implementation. It is to make the implementation's structure carry as much of the specification as possible.

Names, types, module boundaries, contracts, constraints, and transitions all contribute. Comments should explain the reasons that those structures cannot express: why a boundary exists, why an exception is necessary, or which external assumption a decision relies on.

This does not mean code can recover every human intention. It means we should reduce the amount of intention that an agent has to guess.

Current state and change history have different jobs

Drizzle already combines a declarative schema with incremental migrations. Its generate command builds a schema snapshot and compares it with the previous snapshot to generate SQL. Declarative state and migrations are therefore compatible, not competing approaches. See the Drizzle documentation.

The current schema tells us what structure we want. A change also needs to explain how existing data should reach that structure.

Two snapshots cannot fully determine that meaning. A removed column and an added column might represent a rename, a replacement, or a change in the underlying concept. Drizzle’s documented generation flow prompts the developer for renames when necessary. The prompt supplies information the structural diff cannot decide. A backfill similarly requires a rule for interpreting existing rows.

I call this change meaning: information that cannot be recovered from the final state, including rename intent, backfill rules, and cutover conditions. It belongs in the change artifact—the migration or saved plan—while current behavior belongs in the source code.

The repository should make the current specification legible, while the change should retain the meaning that cannot be reconstructed from the final state.

History still matters. It should not be the only way to understand the present.

Declare the system across its boundaries

Declarative code makes important relationships available for inspection. Each domain has different things to declare.

Domain	What to express	What should become legible
UI	View states, input rules, available actions	What users can understand and do
API	Inputs, outputs, authorization, errors	The contract an operation offers
Workflow	States, events, guards, retries, compensation	Allowed execution and recovery paths
Database	Schema, constraints, access rules	Stored facts and rejected inconsistencies
Infrastructure	Resources, permissions, connections	The execution environment and its boundaries

These declarations must agree. A disabled button does not enforce authorization. An API response does not prove a queued operation has completed. A database constraint does not establish that an external payment request was never sent.

The architectural value comes from expressing the relationships between these boundaries clearly enough to inspect them.

IR makes the declarations analyzable

Readable source helps an agent understand the system. An intermediate representation, or IR, helps tools compute properties of that system.

For this design, I would derive domain-specific IRs from the source:

A schema IR for database objects and constraints.
A configuration IR for infrastructure resources and dependencies.
A machine IR for workflow states and transitions.
Contract and view models for API and UI relationships.

I would keep each domain's semantics intact. Database locks, infrastructure replacement, and workflow reachability need different analyses. Flattening them into one universal schema risks losing the details that make those analyses useful.

Cross-domain analysis can follow contracts, references, and dependencies already expressed in code. Where a relationship cannot be derived, it needs an explicit representation close to the relevant code. It does not require a separate specification registry.

An IR is not the state space itself. It is a representation from which we can describe and analyze states, constraints, and transitions.

The business rule needs an explicit expression in code as well. For billing, that might be a named eligibility policy, a guard that invokes it, and a property test defining what deactivation means for pending work. An analyzer can then check whether relevant execution paths use that policy and whether a change invalidates its dependencies. An independent structural oracle checks fidelity to the database; the executable rule and its tests provide a different criterion for business behavior. Neither establishes that the chosen rule matches human intent: its initial definition and substantive changes still require domain judgment.

Terraform provides an existing example of machine-readable change information: plan and state JSON. It also supports validation conditions.

Graphs expose dependencies; DAGs constrain change order

In a database, a schema IR can support a dependency graph: which views, constraints, or other objects depend on which definitions?

A migration planner can also construct a directed acyclic graph of operations. Adding a column may precede a backfill; the backfill may precede validation; validation may precede switching readers.

These are different graphs. Database dependencies can contain cycles, such as two tables with foreign keys referencing each other. An executable operation DAG represents precedence constraints, and a valid ordering still does not prove that the migration preserves business meaning.

The useful progression is:

Representation	Question it helps answer
Declarative source	What should exist, and what rules are visible?
Domain IR	What structure and semantics can tools inspect?
Dependency graph	What depends on this change?
Operation DAG	What must happen before what?
Checks and observations	Which properties were actually tested or enforced?

This gives an agent more than a sequence of SQL statements. It gives the agent a basis for explaining and checking a plan.

The difficult case is the transition

Return to the billing rule: after an account is deactivated, no new billing operation may begin.

For this example, begin means committing a durable authorization to perform a particular billing operation in the database. Deactivation means committing the account's disabled state. An operation authorized before deactivation may still be sent to the payment provider afterward. That is a deliberate rule, not a guarantee that no external request occurs after deactivation.

A guard alone is insufficient. A worker can read an enabled account, another transaction can deactivate it, and the worker can then send a payment request. A normal PostgreSQL Read Committed SELECT does not protect that interval; see the transaction isolation documentation.

A possible protocol serializes both operations on the same account row. In a billing transaction, acquire the row lock, read eligibility, and insert a durable authorization with a unique operation key before committing. The deactivation transaction takes the same lock before changing eligibility. All authorization paths must follow this protocol. A retry with an existing operation key resumes the committed authorization through a separate path, without creating a new one or reapplying the new-authorization eligibility condition. It must not create a fresh authorization after deactivation. External dispatch occurs only for committed authorizations and needs its own retry and idempotency contract.

This identifies the properties an analyzer would need to model:

Operation	Preconditions	Conditions maintained during execution	Postconditions
Create a new billing authorization	Account lock acquired; account enabled	Lock held through eligibility check and authorization commit	Durable authorization exists for this operation
Deactivate account	Same account lock acquired	Lock held through disabling commit	No later authorization may be created while disabled
Dispatch payment	Committed authorization exists	Retry identifies the same operation	External result recorded, or outcome remains unknown

A trace checker can look for an authorization commit after a deactivation commit, while the account remains disabled. It needs records that reliably reconstruct order for that account, not merely application-log timestamps; its finding applies to observed executions, not all possible executions. Static analysis can flag an authorization path that bypasses the shared protocol.

Now migrate the boolean is_deactivated to a representation that separates account status from deactivation time: status and deactivated_at. A true legacy value becomes status = disabled with deactivated_at = null, meaning disabled with an unknown historical time. A false value becomes status = enabled with no deactivation time. New deactivations record a timestamp. Eligibility depends on status; treating a null timestamp as enabled would incorrectly re-enable legacy accounts. These backfill rules belong in the change artifact.

During the transition, an old application may write only is_deactivated while a new billing worker reads only status. Both versions can be individually plausible, yet their combination can bill a deactivated account.

Later, removing the old column may break old jobs still waiting in the queue. Depending on error handling, they might fail safely or interpret missing information incorrectly. Sending a payment request introduces an external side effect that cannot be undone merely by rolling back the schema.

The plan therefore needs to consider application versions, workflow versions, schema phases, and queued work together.

Phase	Example condition to check
Expand	Existing readers and writers remain compatible
Backfill and synchronization	Both representations preserve the intended meaning
Cutover	New readers receive correct data, including concurrent updates
Drain or upgrade old work	Queued jobs retain valid execution semantics
Contract	No remaining supported execution requires the removed structure

The exact mechanism might be dual writes, a trigger, compatibility code, or another protocol. The important point is to make its assumptions and ordering visible.

Existing tools address parts of this problem. pgroll uses versioned schema views and triggers to support concurrent schema versions during PostgreSQL migrations. Temporal Worker Versioning addresses the relationship between workflow executions and worker versions. Atlas migration lint analyzes migration safety. These are useful precedents; connecting the obligations across domains remains the architectural task.

A concrete architecture: domain compilers and a thin linking layer

The architecture follows two paths: compile declarations into domain models, then connect those models through relationships already expressed in code.

Each domain adapter parses and resolves its declarations into a typed IR. IR nodes retain source locations so a diagnostic can point back to the code an agent needs to change. References resolve to program symbols and artifacts.

flowchart TD
    U["View and Contract IR: deactivateAccount"] --> D["Deactivate: lock account, update status, commit"]
    N["Machine IR: create new authorization"] --> C["Eligibility check under the same account lock"]
    C --> A["Schema IR: committed billing authorization"]
    D --> S["Schema IR: account status and deactivation time"]
    S -->|read under lock| C
    A --> P["Machine IR: dispatch committed authorization"]
    R["Retry: existing operation key"] -->|resume existing authorization| P
    F["Config IR: worker and queue"] -->|hosts| P
    P -.-> X["External payment service"]

The graph connects domain-owned models while making the protocol visible: new authorization checks eligibility under the same account lock used by deactivation, and dispatch consumes a committed authorization. A retry resumes that authorization rather than creating another. These edges identify obligations for the analyzers; drawing them does not prove they are enforced. The dotted payment edge requires an external-effect contract.

Reference resolution distinguishes definite edges, conservative sets of possible targets, explicit library contracts, and unresolved references. Possible edges are useful for conservative analysis but must not be presented as confirmed calls. Missing information is not evidence of safety.

The graph supports impact analysis. Temporal correctness needs an additional contract layer, close to the relevant code: transaction boundaries, commit points, synchronization resources, and the ordering of external effects.

Component	Responsibility
Domain adapter	Preserve domain semantics, source locations, and analysis limits
Linker	Resolve references and artifact versions
Domain analyzer	Evaluate local properties and report the assumptions they require
Composer	Connect guarantees to assumptions and check interference between operations

For billing, the composer must establish that both paths use the same account lock, that eligibility is read under that lock, and that authorization commits before external dispatch. If another path can change eligibility or create authorization outside the protocol, the local guarantee does not compose. Where the available model cannot establish these facts, the result is unknown. Thin reference linking remains thin; concurrency analysis is a separate responsibility.

From IRs to a change plan and runtime evidence

Planning needs three distinct inputs: the candidate model, models of old artifacts still supported, and evidence or deployment contracts identifying what can run or resume. Retain generated IRs and manifests for supported builds, including workflow versions. Deleting an old worker from the current checkout does not erase its dependencies while that artifact can still execute.

Observations remain separate from declarations and carry their origin and acquisition time. These generated models retain the semantics of supported builds.

flowchart TD
    P["Plan: operation DAG and phase conditions"] --> V["Verify: compatibility, ordering and gaps"]
    V --> G["Gate: bind evidence to plan and recheck prerequisites"]
    G --> E["Execute: domain operations"]
    E --> O["Observe: actual state and execution traces"]
    O --> A["Adjudicate: repair, accept or bound an exception"]
    A -.-> P

The composer adds cross-domain precedence and interference conditions to local plans. Before removing the old field, two conditions must hold: no new work requiring it can be admitted, and no accepted work can execute or resume with that dependency. Depending on the runtime, this includes running jobs, delayed jobs, retries, dead-letter replay, and suspended workflows.

One proposed protocol installs a durable version-admission fence, stops old producers, and retires or upgrades every old dependent execution. Scheduling, replay, and resume paths must obey the same fence. The fence remains in force through contract and prevents old artifacts from returning afterward. A gate checks the fence and retirement evidence before removal; rechecking queue depth alone cannot close the race. If the runtime cannot enforce this protocol, the prerequisite remains unknown.

Evidence needs a reproducible verification identity: the exact plan, source revision, input-model digests, and verification conditions, including the analyzer version. A fingerprint over that bundle binds a result to what was actually checked. It does not freeze the environment: observations also carry their acquisition time, and execution gates recheck relevant runtime assumptions. Observations feed verification and adjudication; they do not automatically rewrite the authoritative source.

An agent-facing interface would expose affected source locations, dependency paths, candidate operations, failed conditions, and unknowns. That gives the agent actionable evidence for a code change without asking it to reconstruct the entire architecture from logs.

For example, a declaration change that removes accounts.is_deactivated would remove that field from the candidate Schema IR. Resolving the retained IR of the supported v1 worker against the candidate schema would produce a missing dependency—even if its source was deleted from the checkout. The contract-phase check separately needs evidence for admission fencing and retirement of old dependent executions.

An illustrative diagnostic can be presented as a table. This is a proposed result, not captured output from the harness.

Proposed operation	Evidence or condition	Analysis result
Remove `accounts.is_deactivated`	Retained `billing-worker:v1` model reads the field at `src/workers/billing-v1.ts:42`	Violated: incompatible supported reader
Enter contract phase	New admission of old dependent work is fenced	Unknown: enforcement evidence missing
Enter contract phase	Accepted old dependent work cannot execute or resume	Unknown: retirement evidence missing

Execution decision: block removal. The analysis distinguishes violations from missing evidence; the destructive-operation policy determines the decision. An exception would be an explicit policy decision, not a verified result.

Verification needs a scope

A database constraint, a workflow guard, a test, and an alert do different jobs.

A constraint rejects invalid states within its scope. A guard controls operations that pass through it. Tests check the cases they execute. Monitoring detects what its instrumentation can observe.

Calling all of these “verified” hides important differences. A result should state what property was checked, against which evidence, under which assumptions, and for which plan or code version.

An IR also cannot serve as its own sole oracle. If generation and validation share the same lossy transformation, they can agree while omitting the same behavior.

We encountered this in the database harness: a convergence check could report agreement when generation and verification symmetrically dropped information. A separate PostgreSQL-backed structural oracle helped reduce that blind spot. The lesson was specific: agreement after a shared transformation was weaker evidence than it appeared.

We also found declarations that existed in configuration but were not wired into enforcement. A declared policy can give an agent false confidence when its implementation does nothing. Checks for declaration-to-enforcement mismatches, together with deliberately faulty implementations, test whether the mechanism is actually connected. Neither an impressive declaration nor a linked test is enough on its own.

When actual behavior diverges from the declared model, the response requires judgment: restore the intended behavior, accept a corrected design, or record a bounded exception. A production hotfix may be valid, but observed behavior should not silently redefine the specification.

The claim about AI is still a hypothesis

The database and infrastructure models, scoped verification, and plan-bound evidence have concrete implementations in our tooling. The full architecture above—including UI/API models and cross-domain phase composition—is the direction I am working toward. Existing mechanisms and operational lessons do not yet establish a general benefit for AI coding.

My hypothesis is that code with explicit semantics, combined with derived analysis, reduces the context an agent must reconstruct and the assumptions it must invent.

Use matched change tasks with seeded violations, compatibility traps, and cross-domain dependencies. Compare a strong baseline—source, type checking, lint, tests, and existing migration analysis—with the same setup plus derived cross-domain analysis. Keep the model and budget fixed, repeat runs, and use held-out repositories and tasks. Separate changes to declaration structure from added tooling; this evaluates the analysis system, not IR in isolation.

Measure detection, false positives, correct implementations, repair iterations, and maintenance cost. Judge both groups against independent expected outcomes, rather than the analyzer's own finding count.

Counting declarations alone would miss the point. The model must help an agent understand and preserve behavior.

Make the current specification readable in code. Make its structure analyzable through IRs and graphs. Then use that analysis to check both the destination and the path of a change.

That is the design direction I want for large systems built with coding agents.

Where Agent Runtime Engineering Ends, Domain Runtime Engineering Begins

Takafumi Endo | ROUTE06 — Wed, 30 Sep 2026 04:38:34 +0000

Claude Code, Codex, and other agent runtimes are becoming capable enough that we increasingly don't need to build the agent loop ourselves.

Yet in client work, the engineering hasn't disappeared. It has moved toward MCP servers, APIs, DSLs, domain tools, state management, authorization, execution guarantees, and the systems agents operate.

This made me rethink what we are actually building.

I find it useful to separate two responsibilities: Agent Runtime Engineering and what I'll call Domain Runtime Engineering.

The Agent Runtime manages how the agent pursues a goal.

The Domain Runtime gives its operations precise meaning and enforceable guarantees.

"Domain Runtime" is not a standardized industry term here.

I'm using it as a convenient name for the application services, workflow engines, compilers, validators, rules, calculations, and transaction logic that execute operations according to domain semantics.

Two runtimes, two responsibilities

An Agent Runtime enables an AI agent to continue working toward a goal.

It manages model calls, context, tool selection, observations, execution progress, interruption, resumption, and sometimes runtime-level approvals.

A Domain Runtime executes operations according to domain semantics.

It determines whether inputs are valid, how specialized calculations behave, which invariants apply, and—where applicable—whether an operation may produce an authoritative state change.

	Agent Runtime	Domain Runtime
Central question	What should I do next to achieve the goal?	What does this operation mean, and how should it execute?
Typical state	Conversation, plans, observations, run progress	Domain data, versions, approvals, operation results
Success	The requested task is satisfied	Domain postconditions hold
Failure handling	Replan, investigate, try another path	Validation, rollback, idempotency, compensation, reconciliation
Changes when	Models, prompts, tools, harnesses change	Domain rules, contracts, algorithms, operating requirements change

I would not draw this boundary as probabilistic versus deterministic.

Agent runtimes contain deterministic control logic, and domain processing may itself use AI. The stronger distinction is who owns domain semantics, invariants, and authoritative outcomes.

Domain Runtime does not only mean state changes

Consider a facility analytics system.

The user asks:

Compare occupancy rates across these buildings and investigate the outliers.

The Agent Runtime decides what data to inspect, which analysis to run, whether the result is sufficient, and what additional question to investigate.

The Domain Runtime resolves occupancy_rate against an accepted domain definition and executes it according to that definition.

That definition might ultimately evaluate something like:

occupied_area / leasable_area

but the formula alone is not the domain meaning.

The definition may also specify aggregation granularity, missing-value behavior, unit conversion, eligibility rules, and what happens when leasable_area is zero.

No database mutation is required.

The Domain Runtime's job is to produce a valid result according to accepted domain semantics. The Agent Runtime uses that result to decide what to do next.

Agent Runtime
    │
    │ "calculate occupancy by building"
    ▼
Domain Runtime
    │
    │ resolves and applies
    │ versioned domain semantics
    ▼
Valid domain result
    │
    ▼
Agent Runtime
    │
    └─ decides what to investigate next

Those domain definitions do not necessarily have to be handwritten by developers.

An agent may propose a new metric or modify an existing definition, but that proposal does not automatically make it authoritative. Validation, review, activation, and execution remain governed application responsibilities.

The runtime can validate and execute an explicit domain definition.

That alone does not establish that the definition correctly represents the real business.

This distinction matters because an LLM can calculate a ratio itself.

What makes the calculation part of the application is not arithmetic. It is the contract around what the calculation means.

State changes make the boundary even clearer

Now suppose the user says:

Analyze this dataset and publish the result.

The Agent Runtime can inspect the data, run analysis tools, identify problems, revise its approach, and eventually determine that it is ready to publish.

But publish is not merely another reasoning step.

The domain may need to determine which version is being published, whether it is valid, whether the caller is authorized, whether business approval exists, and whether the same version has already been published.

User / Product
       │
       ▼
┌────────────────────┐
│   Agent Runtime    │
│                    │
│ goal               │
│ context            │
│ planning           │
│ tool selection     │
│ work continuation  │
└─────────┬──────────┘
          │
          │ MCP / API / CLI
          ▼
┌────────────────────┐
│   Domain Runtime   │
│                    │
│ domain semantics   │
│ authorization      │
│ validation         │
│ business approval  │
│ state transition   │
│ transaction logic  │
└─────────┬──────────┘
          │
          ▼
  Authoritative State

This is a logical responsibility boundary.

Both runtimes may live in one process. The Domain Runtime may simply be the backend application that existed years before anyone added an agent.

The publish succeeded, but the response disappeared

This is where the distinction becomes operationally important.

Suppose the Agent Runtime requests publication of version 42. The Domain Runtime validates the request and commits the publication successfully.

Then the connection fails before the response reaches the agent.

Agent Runtime
     │
     │ publish(version=42)
     ▼
Domain Runtime
     │
     │ commit succeeds
     ▼
State: PUBLISHED

     X response lost

From the Agent Runtime's perspective, the outcome is unknown.

Retrying is a reasonable agent behavior, but safe retry is a contract between both sides. The domain defines the retry guarantees; the agent integration must preserve operation identity and follow that contract.

A robust contract might assign a stable operation identity to the approved publication request.

1. User approves publish intent P
2. Operation O is durably bound to payload P
3. Domain Runtime executes O
4. Domain state is committed
5. Response is lost
6. Agent retries or queries operation O
7. Domain Runtime recognizes O
8. Existing outcome is returned

Retries preserve both the operation identity and the business payload.

The domain system durably records the outcome and detects duplicate execution.

External effects may complete independently.

Publishing a record and sending a notification, for example, can fail at different points. The system therefore needs to track which effects completed and which still require delivery or reconciliation.

Not every recovery path has to transparently return the original success result.

For some operations, preventing duplicate execution, querying durable state, and marking an unresolved attempt as interrupted may be the safer contract.

This is not something conversation history can resolve.

The agent's memory of the world is not the world.

Runtime approval is not business approval

Approval has a similar ambiguity.

An Agent Runtime may ask:

Allow this shell command?

That controls whether the agent may perform an action in its execution environment.

The business application may separately require:

Approve version 42 for publication to the production portal?

That approval has domain meaning.

Agent Runtime
    │
    ├─ Runtime approval
    │    "May the agent attempt this action?"
    │
    ▼
Domain Runtime
    │
    └─ Domain authorization / approval
         "May this domain operation take effect?"

The first approval cannot silently imply the second.

Business policy needs an authoritative owner and consistent enforcement across UI, API, scheduled jobs, and agent paths. Runtime approvals complement domain authorization; they do not replace it.

Agent Runtime and Domain Runtime need a contract

As existing Agent Runtimes improve, this boundary becomes one of the most important custom engineering surfaces.

Agent Runtime
      │
      │ chooses and requests operations
      ▼
┌─────────────────────────┐
│ Agent-facing Interface  │
│                         │
│ Tools                   │
│ MCP                     │
│ API                     │
│ CLI                     │
│ Schema / DSL            │
└────────────┬────────────┘
             │
             ▼
       Domain Runtime

A backend interface designed for a human-facing application is not automatically a good agent interface.

A frontend can orchestrate many low-level endpoints because application code already knows the workflow. Giving those endpoints directly to an agent pushes that orchestration back into model reasoning.

For example, this:

create_revision(...)
update_rows(...)
set_status(...)
create_permission(...)
send_notification(...)

may be a worse agent surface than:

prepare_import_plan(...)
validate_import_plan(...)
publish_version(...)

The deeper question is not only which protocol to use.

What is the smallest, safest, and most legible operation we can expose to the agent?

MCP is one way to expose that boundary

MCP fits naturally into this architecture.

It can expose domain capabilities to an Agent Runtime through a standard tool interface.

Agent Runtime
      │
      │ MCP
      ▼
 Domain Tool
      │
      ▼
Domain Runtime

But MCP does not determine the business meaning behind the operation.

publish_report could be exposed through MCP, REST, an internal function, a dynamic tool, or a CLI. The same Domain Runtime should still enforce the invariants that must hold regardless of the caller.

MCP exposes a capability. The Domain Runtime determines its domain effect.

Sometimes the MCP server and Domain Runtime are implemented together.

Sometimes MCP is only an adapter around an existing service. In other systems, the agent runtime may expose its own dynamic tool mechanism instead.

The logical responsibility matters more than the transport.

DSLs are another way to design the boundary

Some domains benefit from something more expressive than individual tool calls.

Suppose an analysis can be represented as:

analysis Occupancy {
    source facilities
    metric occupancy_rate
    group_by building
}

Here, occupancy_rate refers to a versioned domain definition rather than redefining the metric inside every agent-authored program.

The Agent Runtime can read this program, propose one, modify one, or explain one. That does not mean the Agent Runtime determines what the referenced metric means.

Agent Runtime
      │
      │ proposes DSL
      ▼
  Domain Program
      │
      ▼
Compiler / Validator
      │
      ▼
 Domain Runtime
      │
      ▼
 Valid result / state

A compiler may translate it. A validator may reject it. A workflow engine may evaluate it. Domain services may execute the resulting operations.

The agent can propose a domain program. The Domain Runtime determines whether and how it takes effect.

An agent can also propose a new metric definition.

That is a different operation from using an existing metric. The proposed definition must be validated and separately accepted before other programs can rely on it as an authoritative domain definition.

The same distinction applies to mandatory policy.

An agent might generate:

workflow PublishAnalysis {
    analyze
    review
    publish
}

but a mandatory approval requirement cannot depend on whether the agent remembered to encode:

publish requires approval

If approval is a domain invariant, deleting such a line cannot remove the requirement.

Agent-authored programs cannot weaken mandatory domain policies. Policy changes are separately authorized and validated.

The Domain Runtime therefore evaluates the proposal against constraints the proposal itself does not control.

Agent-authored program
        │
        │ proposed intent
        ▼
Domain Runtime
        │
        ├─ resolves domain definitions
        ├─ validates program
        ├─ applies mandatory policy
        └─ executes if valid

Not every system needs a DSL.

Typed functions, JSON Schema, or an ordinary API may already express the domain clearly enough. The important property is the separation between proposing an operation and determining its domain meaning and effect.

Runtime control is a different boundary

There is also an interface on the other side of the Agent Runtime.

A product may need to start work, resume it, send new input, receive events, or participate in runtime-level approvals.

Product UI / IDE
       │
       │ runtime control
       ▼
  Agent Runtime
       │
       │ capability access
       ▼
  Domain Runtime

Codex App Server is one concrete example of a control interface for integrating Codex into another product.

It exposes an interface for rich clients to interact with Codex conversations, approvals, and streamed agent events. This is a logical boundary, not a requirement for a separate runtime service.

That is a different concern from MCP.

One interface controls the agent's execution. The other gives the agent access to domain capabilities.

Structured errors become part of the contract

An agent-facing Domain Runtime should make domain failures actionable.

Compare:

400 Invalid configuration

with:

{
  "code": "STALE_VERSION",
  "expected_version": 43,
  "provided_version": 42,
  "recoverable": true,
  "suggested_action": "reload_and_revalidate"
}

The second response does not tell the agent what its goal should be.

It makes the domain state explicit enough for the Agent Runtime to choose its next action.

The Domain Runtime should not decide what the agent wants to do next.

It should make domain consequences and constraints explicit enough that the agent can decide.

Agent-native does not mean agent-owned

Making a system easier for agents to use does not mean moving business logic into the agent.

Often the opposite architecture is safer.

An agent-facing interface should be optimized for model reasoning:

clear operations
typed inputs
structured outputs
stable identifiers
explicit preconditions
actionable failures
idempotent mutations
queryable operation status

But the meaning behind those operations should remain authoritative regardless of whether the caller is Claude Code, Codex, a custom agent, a human UI, a scheduled job, or another service.

Agent-native interfaces should make the domain easier to reason about, not make the agent the owner of the domain.

Better Agent Runtimes move the build boundary

This changes how I think about build versus buy.

If Claude Code, Codex, an Agents SDK, or another runtime already gives us a capable agent loop, context management, tool execution, resumption, and sandbox integration, rebuilding those pieces may create little value.

Instead, I increasingly ask:

Are the domain operations well defined?
Can the agent invoke them at the right abstraction level?
Are domain definitions versioned and governed?
Are mandatory policies independent of agent-authored programs?
Are authorization and business approvals authoritative?
Are operations safely retryable?
Can uncertain outcomes be reconciled?
Are errors structured enough for the agent to recover?
Can agents and conventional application paths share the same semantics?

That is where much of our custom engineering now lives.

Engineering the boundary

This started for me as a naming problem.

We were building MCP servers, DSLs, APIs, CLI tools, domain services, sandboxes, databases, and agent integrations, and none of those labels described the whole job.

The more useful realization is that many of these components exist to engineer one boundary:

                 User / Product
                       │
                       ▼
                Agent Runtime
         goal / context / next action
                       │
          MCP / API / CLI / DSL
                       ▼
                Domain Runtime
       semantics / guarantees / results
                       │
                       ▼
            Data / External Systems

The Agent Runtime should be free to explore, replan, retry, and choose different paths toward a goal.

The Domain Runtime should make domain operations precise, enforce what must be enforced, and produce results the rest of the system can trust.

Existing Agent Runtimes will keep getting better, reducing how much harness infrastructure we need to build ourselves.

But someone still has to define what occupancy_rate and publish mean, which guarantees apply, and which invariants can never be delegated to model reasoning.

The Agent Runtime manages how the agent pursues a goal.

The Domain Runtime gives its operations precise meaning and enforceable guarantees.

For me, the interesting engineering is increasingly the boundary between the two.

References

The Agent Runtime / Domain Runtime distinction in this article is my own responsibility model. The following official documentation is referenced for the products and protocols discussed above.

Why agent-browser and Playwright Work So Well Together

Takafumi Endo | ROUTE06 — Tue, 15 Sep 2026 03:12:19 +0000

We recently added agent-browser to our UI evaluation workflow.

The motivation was simple.

A normal E2E test is great when we already know the path:

await page.getByRole("button", { name: "Settings" }).click();

await expect(
  page.getByRole("dialog", { name: "Settings" })
).toBeVisible();

But sometimes I want to ask a different question:

Can a user find Settings and open it?

I don't necessarily care which valid route the agent takes.

I care whether the product makes the goal achievable.

That is where agent-browser became useful.

The interesting part, though, was not simply letting an AI control a browser.

It was deciding where the agent's responsibility should end.

First: agent-browser and Computer Use are different paths

Before getting into the architecture, one distinction is important.

In our setup, agent-browser is part of the development-side evaluation harness.

Conceptually:

Codex / Claude
      ↓
agent-browser
      ↓
our application
      ↓
independent assertion

agent-browser gives coding agents a browser-oriented interface. Its snapshot workflow exposes an accessibility representation with element references, allowing the agent to inspect the current UI, act, inspect again, and continue toward a goal.

OpenAI Computer Use is a separate execution path.

The current ChatGPT desktop app includes Chat and Work under ChatGPT, alongside Codex. Work can use local files and desktop applications with permission.

When I refer to Computer Use in this article, however, I mean the capability exposed to Codex for interacting with desktop applications.

For native application evaluation, where available, our path is closer to:

ChatGPT desktop app
        ↓
      Codex
        ↓
   Computer Use
        ↓
native application
        ↓
independent verification

We are not handing an agent-browser session over to Computer Use.

We are also not claiming that agent-browser is our native desktop automation engine.

And Computer Use should not be read as "the Office editing API." For example, Excel also has a dedicated ChatGPT for Excel add-in workflow; Computer Use may help Codex reach that environment, but it is not the editing model itself.

These are different interaction mechanisms.

The common principle is only this:

The system performing the action does not have to be the system deciding whether the result is correct.

The rest of this article focuses mainly on the agent-browser side.

Why agent-browser fits exploratory QA

agent-browser works well when the task is expressed as an intent rather than a click script.

Instead of:

Click button A.
Then click menu item B.
Then expect dialog C.

we can give the agent:

Open Settings.

Or:

Find the customer and open the information relevant to their latest issue.

The agent can inspect the current accessible UI through agent-browser and decide how to proceed.

That is useful for flows where we care about whether a goal is discoverable and achievable, rather than whether one exact sequence of selectors still works.

I think of this as exploratory QA.

Not random exploration.

Not a replacement for our normal E2E suite.

A separate lane for asking higher-level questions about the product.

The agent does not grade its own homework

This was the most important design decision.

Suppose the task is:

Open the Settings dialog.

The agent uses agent-browser and eventually reports something equivalent to:

{
  "status": "completed"
}

We do not treat that as a passing test.

After the agent finishes, Playwright checks the resulting application state independently.

Conceptually:

agent:
"completed"

    AND

Playwright:
Settings dialog is visible

    ↓

PASS

The assertion can be very small:

await expect(
  page.getByRole("dialog", { name: "Settings" })
).toBeVisible();

The two signals mean different things.

The agent says:

I believe I completed the goal.

The assertion says:

The application reached the state we care about.

That separation is what makes the combination useful.

Playwright gives the agent more freedom

Initially I thought of Playwright as a safety net around agent-browser.

I now think the opposite framing is more useful:

An independent oracle lets the agent be more exploratory.

The agent using agent-browser can focus on:

Can I accomplish the user's goal?

while Playwright focuses on:

Did the application actually reach the expected state?

The architecture is roughly:

          goal-oriented
           exploration
               │
               ▼
      ┌─────────────────┐
      │ Codex / Claude  │
      │        +        │
      │  agent-browser  │
      └────────┬────────┘
               │
               ▼
          application
               │
               ▼
      ┌─────────────────┐
      │   Playwright    │
      │    assertion    │
      └────────┬────────┘
               │
               ▼
           pass / fail

This is why I don't see agent-browser and Playwright as competitors.

They are useful precisely because they can own different responsibilities.

This pattern is not new

I don't think the individual ideas here are novel.

The broader agent-evaluation ecosystem is already converging on similar principles.

Anthropic describes agent evaluation in terms of tasks, outcomes, and graders, and recommends code-based graders where possible instead of over-constraining the exact trajectory an agent must follow.

WebArena has used a similar shape for years: an agent receives a natural-language web task, while evaluation checks whether the task was functionally completed.

Playwright itself now has Test Agents. A planner explores the application, a generator turns plans into executable tests, and a healer works on failing tests.

There are also projects much closer to this exact browser-QA pattern.

The public qa-skills repository describes an agentic browser-testing workflow where the browser agent receives a natural-language goal, an external oracle determines success, and stable flows can graduate to scripted Playwright tests.

SightCI similarly documents AI exploration followed by promotion of selected runs into Playwright specs.

So the interesting question is not:

Did we invent agentic browser testing?

We didn't.

The useful question for us became:

Where should the boundary sit once this pattern meets a real application with UI, API, authentication, and native document workflows?

That is where our implementation became more interesting.

A real bug changed our original rule

Our first mental model was straightforward:

agent-browser-assisted exploration finds a failure
                    ↓
             we understand it
                    ↓
        add a Playwright regression test

Then we encountered a failure in a real authenticated environment.

A cloud instruction was rejected.

After comparing API behavior under the same authentication context, we traced the issue to an organization-resolution mismatch.

The failure was visible through the product.

But the contract that actually broke was not a UI contract.

Adding another browser regression would therefore have put the check too far away from the failure.

Instead:

real environment exposes failure
            ↓
compare API behavior
            ↓
identify organization-resolution mismatch
            ↓
fix implementation
            ↓
add model-free Hosted E2E coverage

That changed the rule.

Not:

Every problem discovered through exploration should become a Playwright test.

But:

Move an understood failure to the lowest-cost assertion-based layer that still reproduces the contract that failed.

That last part matters.

I don't mean "always turn an integration failure into a tiny unit test."

Some failures only exist across an integration boundary.

The regression still needs to reproduce the contract that actually broke.

Promote failures to the right layer

In practice, that gives us a rule of thumb like this:

Failure	Likely regression layer
UI navigation, visibility, dialogs	Playwright
Authentication, organization resolution, API contracts	API / Hosted E2E
Word, Excel, or PowerPoint output	Artifact-level verification

A UI failure may graduate to:

agent exploration
      ↓
UI root cause
      ↓
Playwright regression

An authentication or backend failure may graduate to:

failure discovered
      ↓
API/auth root cause
      ↓
Hosted E2E / API regression

And a native document workflow can use a different oracle:

desktop interaction
      ↓
edit a safe working copy
      ↓
save
      ↓
artifact / structure / reopen verification

The execution mechanism is secondary.

The important question is:

Where can we observe the failed contract most directly and cheaply without losing the behavior that matters?

Known failures should leave the agent loop

This leads to the part of the design I like most.

Imagine an agent using agent-browser discovers a navigation problem today.

We investigate it.

We identify the contract that failed.

We add an assertion-based regression at the correct layer.

What should happen tomorrow?

Ideally, that exact failure should no longer require an AI agent.

AI exploration
      ↓
new failure
      ↓
root cause becomes known
      ↓
assertion-based regression
      ↓
future CI catches it without AI

That means the goal is not to make more and more of the regression suite depend on AI.

It is almost the opposite.

Use the agent where uncertainty still exists.

Once the failure becomes knowledge, compile that knowledge into a cheaper check.

AI should have to discover a bug once. It should not have to rediscover the same bug on every pull request.

Restrict the capability, not only the prompt

There is another practical issue with giving an agent a browser.

A prompt can say:

Do not upload files.
Do not download files.
Do not leave the test environment.
Do not execute arbitrary JavaScript.

But those are instructions to the model.

Where possible, we also want the environment to enforce the boundary.

agent-browser provides controls such as domain restrictions, content boundaries, action policies, confirmations, and output limits.

One detail matters:

Those security controls are opt-in.

A plain agent-browser session should not be assumed to be restricted automatically.

Our evaluation harness therefore treats restrictions as part of the harness configuration rather than relying only on the prompt.

The principle is:

Prompt boundaries are behavioral. Tool boundaries are architectural.

We also avoid treating one action-policy file as a complete security boundary.

At the time of writing, there is an open agent-browser issue describing a mismatch between documented action-policy categories and how individual actions are matched.

So effective permissions should be verified, not merely inferred from policy names.

For us, tool policy is only one layer alongside ephemeral execution, secret minimization, bounded runtime, and disposable browser state.

Pass evidence, not magical state

Another design question is what happens after an exploratory run fails.

One option would be to keep the browser alive and pass the same mutable session through several agents.

Sometimes that is useful.

But I don't want a live session to become the primary interface between evaluation stages.

Our current harness already records structured evidence such as:

scenario
scenario version
agent
target build
prompt hash
latency
coverage
failure domain
violation detail
cleanup state

The direction I want to take this is a richer diagnostic bundle:

final URL
failed step
accessibility snapshot
screenshot
console errors
trace

Then the flow becomes:

agent exploration through agent-browser
          ↓
inspectable evidence
          ↓
developer / Codex diagnosis
          ↓
regression candidate

This is not an argument against traces, storage state, network logs, or screenshots.

Those are exactly the kinds of evidence we may want.

The distinction is between preserving evidence about state and making a hidden, mutable browser session the contract between components.

Evidence is easier to inspect, store, compare, and review.

What we have today

This is not a fully autonomous QA pipeline.

Today, the important pieces are:

✓ agent-browser exploration lane
✓ independent Codex / Claude evaluations
✓ restricted browser capabilities
✓ Playwright assertions outside the agent
✓ structured evaluation evidence
✓ assertion-based tests remain the release source of truth

What we do not yet have is the full promotion loop:

□ complete screenshot / trace diagnostic bundles
□ automatic regression-candidate generation
□ evidence → regression linkage
□ reviewed promotion records

So I would describe the current system as:

A shadow exploration layer around assertion-based QA, not an autonomous replacement for it.

That is intentional.

Final thought

I started this work because agent-browser looked like a good way to let coding agents interact with our application more like a user.

It is.

But the interesting lesson was not "AI can click buttons now."

The useful architecture emerged from deciding what the agent should not own.

An agent using agent-browser is good at asking:

Can I achieve this user goal from the UI in front of me?

Playwright, API checks, and artifact verification are good at asking:

Is this contract actually satisfied?

Those ideas are not new individually.

What surprised me was how well they fit together once we applied them across a real application instead of only a browser benchmark.

Let the agent explore what you don't know yet.

Let assertion-based tests remember what you already learned.

And once a failure becomes understood, move it out of the expensive exploratory loop and into the cheapest test that still protects the contract.

AI should have to discover a bug once.

It should not have to rediscover the same bug on every pull request.

Codex Computer Use Changed the Agent Integration Boundary

Takafumi Endo | ROUTE06 — Wed, 09 Sep 2026 06:53:20 +0000

For a long time, I assumed serious agent integration required a structured interface.

API, CLI, SDK, then MCP.

Codex Computer Use changed that assumption.

An agent can now look at the same interface as a user, operate it, observe the resulting state, and continue.

GPT-5.6 Sol was already strong enough for us to start using this seriously. GPT-6 Astra raised the baseline further.

OpenAI reports Astra at 72.6% vs. 65.7% for Sol on OSWorld 2.0 — specifically v2026.08.08, offline set, partial score — and 92.7% vs. 76.9% on ScreenSpot-Pro, under the no-tools setting.

These are benchmark results, not end-to-end success rates for arbitrary enterprise workflows, but the improvement is substantial.

More importantly, OpenAI now explicitly describes frontend QA, installing and testing software, and troubleshooting visible problems as Computer Use workloads.

The important change is not that AI can click buttons.

The GUI has become a practical agent interface.

The integration path changed

With MCP, the protocol itself is usually not the difficult part.

The difficult part is designing what sits behind it.

Application
    ↓
API / CLI
    ↓
agent-oriented operations
    ↓
MCP
    ↓
Agent

For an agent to operate complex software reliably, we often have to design the right verbs, schemas, permissions, side effects, errors, and recovery behavior.

That remains the right architecture for operations requiring determinism, throughput, explicit contracts, or strong auditability.

Computer Use does not replace MCP.

It changes how much of that integration surface has to exist before an agent can do useful work.

Another path is now practical:

Existing GUI
    ↓
observe
    ↓
click / type / navigate
    ↓
observe again
    ↓
Agent

The current Computer Use implementation can visually inspect and operate graphical applications, including workflows where CLI or structured integrations are not sufficient.

In the Codex desktop environment, Computer Use is exposed through the plugin/tool system.

So this is not really MCP versus Computer Use.

Computer Use reduces the need to design every domain-specific operation upfront.

It does not remove the need to engineer execution boundaries and verify outcomes.

The integration cost moved

This is the part I find most interesting.

Previously, much of the engineering cost looked like this:

design API
design CLI
design agent verbs
wrap with MCP

Computer Use moves part of that work toward:

control identity
control environment
control permissions
observe execution
verify outcomes

The integration cost did not disappear.

It moved.

For a specialized desktop tool with a weak API, building twenty MCP operations may no longer be the best first step.

We can let the agent operate the existing interface, observe the real workflow, and identify which operations are frequent, dangerous, slow, or difficult to verify.

Then those operations can become structured tools.

GUI
 ↓
real usage
 ↓
identify critical operations
 ↓
API / CLI / MCP

The GUI becomes a compatibility layer.

Structured interfaces remain both an optimization layer and a place to enforce explicit contracts and policy.

The GUI is not an authorization boundary

This distinction becomes critical once the agent can operate the same interface as a human.

Computer Use changes how an agent reaches an operation. It does not define what authorizes that operation.

If an AI can see a human approval button, that does not mean the AI should be allowed to press it.

If credentials, raw data, privileged controls, or connected accounts are visible in the environment, they may also become part of what the agent can observe or operate.

This is not a sufficient security model:

Agent
  ↓
GUI
  ↓
whatever the logged-in user can do

For important actions, GUI operations and structured tool calls should converge on the same authorization, validation, approval, and audit boundaries.

Agent
  ↓
controlled execution environment
  ↓
GUI / structured tools
  ↓
policy + validation
  ↓
system of record

A visual interface is an execution surface.

It should not become the source of authority.

We use Computer Use for QA

A lot of Computer Use demos naturally focus on video editing, design software, spreadsheets, and other professional tools.

Our main use case is more mundane:

software testing.

We already use Playwright heavily.

We also use agent-based evals, generated test data, and fuzzing to explore cases that are difficult to maintain as hand-written fixtures.

Computer Use adds another execution path.

I do not see these as a linear pipeline where every test eventually reaches Computer Use.

They are different ways of exercising the same system, ideally converging on shared outcome checks.

                   scenario
                      │
          ┌───────────┼───────────┐
          │           │           │
     Playwright   Agent tools  Computer Use
          │           │           │
          └───────────┼───────────┘
                      │
               shared outcomes
               and invariants
                      │
              reproduce / reduce
                      │
                regression test

Playwright remains better when we know exactly what contract we want to verify.

Computer Use is useful when we want to exercise the product closer to how a user actually experiences it.

Instead of encoding a fixed sequence of selectors and assertions, we can give the agent a task such as:

Open the application.

Complete onboarding.

Load this generated dataset.

Perform the primary workflow.

Investigate anything visibly wrong.

Preserve evidence of failures.

That exposes a different class of problems:

native dialogs
broken focus
hidden or overlapping controls
unexpected application state
simulator behavior
cross-application workflows
failures visible to a user but awkward to express as DOM assertions

OpenAI explicitly positions Computer Use for desktop application testing, GUI-only bugs, and applications that are not adequately covered by command-line or structured integrations.

We also combine this with generated scenarios.

Rich or unusual data can be produced automatically and then exercised through the actual application rather than only through APIs.

But I do not want the model to become the oracle.

If the requirement is that no duplicate object can be created, code should verify that.

If a user must never gain a particular permission, the authorization layer should verify that.

The agent explores.
Deterministic checks decide facts.
Humans decide where judgment is actually required.

Human review remains valuable for ambiguous product behavior, UX quality, and release decisions.

It does not mean a human needs to inspect every successful run.

In practice, Computer Use has reduced a significant amount of the work before human review:

setup
navigation
reproduction
retries
exploration
evidence collection

That is already useful to us today.

Codex App Server changes the product boundary

The next interesting part is that the Codex harness itself is embeddable.

Codex App Server exposes the Codex harness to external clients through a bidirectional JSON-RPC-style interface.

The embedding application can manage threads and turns, receive agent events, handle approvals, and observe tool activity.

Conceptually:

Our application
      │
      ▼
Codex App Server
      │
      ▼
Codex harness
      │
      ├── shell / files
      ├── MCP servers
      ├── skills
      └── agent loop

This matters because App Server exposes much more than raw model inference.

Codex core manages threads and persistence, executes tools, wires MCP servers and skills into the agent loop, and can pause execution while the embedding client handles an approval request.

But there is an important boundary here.

App Server is not a remote screenshot-and-click API for Computer Use.

In the ChatGPT desktop app, Computer Use is installed as a plugin and exposes a Computer-use MCP server and a Computer Use skill that Codex can use. App Server separately exposes the surrounding Codex harness.

App Server exposes the surrounding Codex harness.

Those are related building blocks, but they are not the same thing.

The existence of App Server plus the desktop Computer Use capability does not by itself guarantee that the official Computer Use implementation can be packaged and independently deployed inside every arbitrary App Server environment.

For that reason, I separate three things:

officially exposed capability

        ↓

what we have connected and validated
inside controlled environments

        ↓

architectures that become possible
from those building blocks

We are actively implementing the second category.

The architectural pattern we care about is:

Our product
     ↓
Codex / agent runtime
     ↓
controlled desktop environment
     ↓
application being operated

This lets computer operation become part of an application architecture rather than something that only happens on a developer's everyday desktop.

Sandbox the desktop, not just the shell

This becomes especially important for enterprise systems.

The Codex harness already has strong concepts around sandboxed tool execution, permissions, MCP integrations, and approval flows.

Computer Use expands the execution surface.

OpenAI explicitly warns that Computer Use can affect application and system state outside the project workspace.

So protecting the project directory is not enough.

For the systems we are building, the isolation boundary needs to include the desktop environment itself.

isolated environment
    │
    ├── dedicated OS/session identity
    ├── dedicated browser profile
    ├── restricted applications
    ├── restricted network egress
    ├── controlled filesystem
    ├── controlled clipboard
    ├── controlled shared folders
    ├── limited accounts
    └── disposable state

A VM alone is not a security model.

If an isolated VM is logged into production with privileged credentials, production side effects are still possible.

The screen must be isolated.

The identity must be limited.

The network must be constrained.

The applications, browser sessions, connected accounts, shared folders, and credentials available inside the environment must all be intentional.

Important side effects should still pass through explicit policy boundaries.

This is what makes Computer Use interesting for enterprise deployment.

Some organizations cannot deploy a general-purpose AI desktop application to employee endpoints.

Others need dedicated identities, isolated networks, specific audit controls, or access to highly specialized software that only exists in a controlled environment.

The useful model is not:

AI controls employee laptop

It is:

AI controls purpose-built environment

We are implementing this pattern in our own application architecture.

Specialized software is where this gets interesting

The long tail of enterprise software is full of difficult integrations:

legacy desktop applications
internal tools
engineering software
industry-specific applications
vendor tools with incomplete APIs

Historically, making those systems agent-accessible meant building an adapter first.

Computer Use changes the economics.

If a human can operate the application visually, an agent can increasingly attempt the same workflow before a complete structured integration exists.

That lets us validate the workflow first.

Then we can progressively move important operations into deterministic tools.

Computer Use:
open project
navigate UI
inspect state

MCP:
fetch structured data
perform validated mutation
commit privileged change

Computer Use:
verify rendered result

I expect this hybrid architecture to become increasingly common.

The question is not:

MCP or Computer Use?

It is:

Which parts of this workflow deserve a structured contract?

The boundary changed

I still want structured interfaces.

For payments, production changes, privileged administration, bulk operations, and anything requiring strong transactional guarantees, explicit tool contracts matter.

Computer Use does not make those requirements disappear.

But I no longer think every application needs a complete agent-native integration before an agent can start doing useful work.

That changes the development order.

Before:

design integration
      ↓
expose tools
      ↓
agent can work

Now:

agent uses existing interface
      ↓
observe real workflow
      ↓
identify critical boundaries
      ↓
structure where necessary

For us, this is already changing software testing.

It is also changing how we think about bringing agents into specialized enterprise software.

The engineering work did not disappear.

It moved from:

How do we expose every operation?

to:

What environment can the agent operate, what is it authorized to do, and how do we prove the result?

That is a much more important shift than better mouse automation.

I still think important APIs will increasingly be redesigned for agents.

I just no longer think agents have to wait for that redesign.

Agentic Software Will Have More Test Code Than Product Code

Takafumi Endo | ROUTE06 — Thu, 03 Sep 2026 06:55:51 +0000

In several products we are building, Codex App Server is no longer just a developer tool. The agent loop is part of the product.

It can inspect state, choose tools, mutate persistent data, request approval, recover from failures, and reach the same valid outcome through different paths.

As we increased that autonomy, I started noticing a strange imbalance.

The product code was growing, of course. But the amount of code and machinery around verification was growing in a different way.

Not just unit tests or E2E.

We started accumulating evals, fuzzers, deterministic verifiers, independent oracles, capability definitions, traces, replay, policies, audit evidence, and release gates.

At first I called all of this "test code." That stopped feeling accurate fairly quickly.

The better name is verification infrastructure.

I do not literally mean that every agentic application will end up with more test LOC than product LOC. The more interesting pattern is that a relatively small amount of implementation can now expose a surprisingly large amount of behavior.

That changes what we have to verify.

A small agent runtime can create a large behavioral surface

Traditional CRUD software usually has a fairly visible path from user action to system effect.

A click maps to an endpoint. The endpoint validates something, updates state, and returns a result. The system can be complex, but there is usually a fairly direct relationship between implementation paths and behavior.

Agentic software breaks that relationship.

A user gives an intent. The agent may inspect state, choose one tool, observe the result, choose another tool, retry after a failure, pause for approval, or reach the same outcome through a different sequence altogether.

Codex App Server is a good example of the shift. It exposes the Codex harness as a long-lived bidirectional JSON-RPC process. A request can produce multiple events, tools can execute during the run, and the process can pause while waiting for approval.

Once that runtime is embedded in a product, it is no longer useful to think of it as "an LLM call."

I wrote about that earlier in The Agent Runtime Got Heavy: Rethinking Sandboxes with Codex App Server.

The mental model I use now is roughly:

Behavioral Surface
≈ Intents × States × Tools × Permissions × Failures × Recovery Paths

Not as a real formula. Just as a reminder that the number of things an agent can do grows much faster than the amount of orchestration code required to expose those choices.

This is the part I think is easy to underestimate.

A read-only copilot is one thing. An agent that edits persistent state, calls external systems, changes permissions, or performs irreversible actions is another. The code difference between those products may not look huge. The verification burden does.

That is why source LOC feels less useful to me as a proxy for complexity than it used to.

Conventional tests still matter, but they no longer describe the whole contract

I do not think the test pyramid is obsolete.

If authentication breaks, I still want a deterministic test to fail. API contracts still need integration tests. Ordinary UI paths still need E2E.

What changed for us is that those tests stopped answering all of the important questions.

Question	What we tend to use
Does deterministic logic work?	Unit / Integration / E2E
Do invariants survive unexpected input?	Property testing / Fuzzing
Can the agent accomplish the task?	Eval
Does it accomplish it reliably?	Repeated eval
Did it stay inside a hard boundary?	Deterministic verifier
Was the execution path acceptable?	Trace verification
Is a claimed result actually true?	Independent oracle
Does it survive dependency failures?	Fault injection

Anthropic's agent-eval guidance describes the same basic problem: multi-turn agents use tools, modify environment state, and adapt to intermediate results. That makes the execution trace and the resulting environment state part of what needs to be evaluated.

One concrete consequence is that a test case is not always a single run anymore.

In some of our agent E2E work, a single Green result did not tell us enough. We needed repeated trials and had to distinguish product failures from environment failures and inconclusive runs.

That starts to feel closer to reliability engineering than ordinary regression testing.

I wrote more about that in The Hard Part of Agent Evals Is Deciding What Counts as Green.

Coverage also starts to mean something different.

We cannot enumerate every prompt, every workspace state, every tool ordering, and every recovery path. So instead of trying to cover the entire behavior space directly, we end up encoding the risks we care about:

the agent must stay inside the permitted workspace;
destructive operations require approval;
cancellation must stop durable side effects;
a claimed artifact must actually exist;
recovery must not corrupt persistent state.

At some point the eval suite stops looking like a collection of examples and starts looking like a machine-readable description of the product boundary.

I am still not sure where the right abstraction is here.

Some of these belong in tests. Some belong in policy. Some are better represented as deterministic verifiers. Some only make sense as production evidence.

But that ambiguity is part of the point: the verification layer is becoming substantial enough that it needs its own architecture.

Where facts exist, I prefer facts

A natural response to agent uncertainty is another LLM judge.

Sometimes that is the right answer.

If I need to evaluate whether a generated report is useful, whether an explanation is clear, or whether an artifact is semantically good, there may not be a deterministic oracle.

But many agent claims are not subjective.

If the agent says it created a file, inspect the filesystem.

If it says it updated persistent state correctly, inspect that state.

If it says it stayed inside a workspace boundary, inspect what it actually touched.

This is the pattern I increasingly prefer:

Probabilistic Agent
        ↓
Observed Execution
        ↓
Facts / Trace
        ↓
Deterministic Verification
        ↓
Policy Decision
        ↓
Evidence

This is also why fuzzing, evals, traces, and oracles keep showing up together in the same systems.

They are doing different jobs.

Fuzzing explores what can happen.
Eval asks whether the agent can succeed.
Trace records what actually happened.
Oracle decides whether the result is correct.
Policy decides whether the behavior was allowed.
Evidence makes that decision reproducible.

I ran into the same issue from another direction while working on fuzzing: Your Fuzzer Is Only as Smart as Its Oracle.

Generating more cases is cheap. Knowing whether they reveal a real failure is the hard part.

There is also a distinction I did not appreciate enough at first: the final state and the trajectory are not always equivalent.

If an agent can safely produce the same artifact through ten different tool sequences, I usually do not want to constrain it to one path.

But if it reads sensitive data, sends it externally, and only then asks for approval, the final state does not rescue the trajectory.

So the rule I use for now is simple:

Constrain the path when the path itself has safety or policy meaning. Otherwise, prefer the outcome as the oracle.

Production traces also start to matter here. Once they are structured and durable, failures can be replayed and converted into regression or eval cases.

That creates a loop from production behavior back into verification.

I wrote about the observability side of this in Installing OpenTelemetry Won't Close Your AI Agent's Feedback Loop.

The other half of the story is economics

So far, the argument is mostly about demand.

More autonomous behavior creates more things we need to verify.

But there is another change happening at the same time: the cost of building verification is falling.

AI can generate not only product code, but also fixtures, fuzz harnesses, eval scenarios, graders, mocks, replay tooling, fault injection, and CI checks.

OpenAI's Harness engineering is a useful public example. In that experiment, Codex generated application code together with tests, CI, documentation, observability, internal tooling, and evaluation harnesses.

The part I find most interesting is not the amount of generated code.

It is that once implementation throughput increased, human attention became the scarce resource.

That changes the economics.

Historically, many teams knew they could benefit from another simulator, another failure-replay tool, another invariant suite, or better fuzzing, but the engineering cost was hard to justify.

Now some of that infrastructure is much cheaper to produce.

So two things are happening at once:

Agent autonomy increases verification demand.

AI-assisted engineering lowers the cost of verification supply.

That combination is probably more important than either trend alone.

It also means we can easily overshoot.

Verification infrastructure has its own failure modes: stale tests, noisy evals, conflicting policies, expensive traces, slow suites, and huge amounts of low-value generated coverage.

So "more verification" is not the goal.

What I actually want is more trusted behavioral coverage for less human attention.

A good invariant can be worth more than hundreds of generated tests. A real production trace can be worth more than many synthetic scenarios. An independent oracle can be more useful than another probabilistic judge.

Verification is becoming part of the product

Verification-heavy software is not new.

SQLite is an extreme public example. For SQLite 3.42.0, the project reports roughly 155.8 KSLOC of library code and a test corpus roughly 590 times larger, including multiple independent harnesses, fuzzing, out-of-memory tests, I/O failures, crash and power-loss recovery, boundary tests, and mutation testing.

I do not think ordinary agent SaaS will look like SQLite.

The useful point is narrower: when the state space is large and incorrect behavior is expensive, verification can become much larger than the implementation it protects.

Agents may bring some of that pressure into much more ordinary products.

A UI can still expose one simple instruction:

"Update these records from this document."

Behind it, the system may interpret data, inspect state, choose tools, call external systems, mutate records, recover from failures, and decide when approval is required.

The user-facing surface remains simple. The behavioral surface does not.

That is why I no longer think "test code" is the right description.

What grows around an agentic product is a mixture of tests, evals, graders, oracles, fuzzers, policies, traces, replay, audit data, and evidence.

That is closer to verification infrastructure.

For most of my career, I thought of tests as support code around the product.

That model feels incomplete once the product itself can decide how to act.

An agentic product is also defined by answers to questions like:

What may it do?
What must it never do?
What counts as success?
How reliable must success be?
When is approval required?
What evidence must exist before we trust the result?

I do not know whether most agentic applications will literally contain more test LOC than product LOC.

That is not really the prediction.

The prediction is that implementation will continue to compress relative to the behavior it enables, while executable evidence expands around that behavior.

We did not just add an agent to the application.

We had to build a verification system around it that increasingly feels like part of the product itself.

AI is making behavior cheap. Verification is becoming the product boundary.

The Hard Part of Agent Evals Is Deciding What Counts as Green

Takafumi Endo | ROUTE06 — Tue, 25 Aug 2026 11:00:59 +0000

AI agent products still need conventional E2E tests. The UI should open, APIs should behave as expected, permissions should hold, and persistence should not break.

If deterministic software breaks, I want a deterministic test to turn red.

But when you build an agent product, you increasingly want another lane next to that: an eval lane that runs the real model and runtime and evaluates the agent itself.

This is no longer a niche concern. Anthropic describes automated evals as especially useful for pre-launch validation and CI/CD, and notes that Claude Code has added evals for things like concision, file edits, and over-engineering. The same article describes Descript separating quality benchmarking from regression testing, and Bolt combining static analysis, a browser agent, and LLM judges.

But once you start building this kind of system, the hard part is not running the eval.

It is deciding what counts as Green.

I have been working through this recently while building a real-model agent eval lane. I do not think there is a universal answer yet for what should become a production gate.

The question I keep coming back to is:

Which evaluations have earned the right to block a release?

Agent evals have too many things we could measure

A traditional LLM eval has a relatively compact mental model:

prompt
  ↓
model / workflow
  ↓
response
  ↓
grader

Correctness, relevance, groundedness, formatting, LLM-as-a-Judge. Of course evals have long been more sophisticated than this, but agents expand the evaluation surface dramatically.

Once a model can use tools, modify state, retain information, and operate across multiple steps, performance is no longer determined by the model alone. The environment and agent harness matter too.

Now we can evaluate:

final response
tool selection
tool arguments
trajectory
task completion
state mutation
artifact quality
safety
latency
cost
recovery

All of these can be useful. That is exactly why the problem gets harder.

Agent evals increasingly suffer less from a lack of measurable things than from an excess of them.

A production gate does not need more metrics. It needs a much smaller question:

What failure are we unwilling to ship?

Decide the claim before the score

Suppose an agent eval has a 72% pass rate. Is 72% good?

The number alone cannot tell you. If this is a new capability and the same eval was at 40% last month, 72% may be excellent. If this is a regression suite protecting behavior users already depend on, 72% may be unacceptable.

Maybe some failures came from the environment. Maybe the grader rejected valid solutions.

Anthropic makes a useful distinction between capability evals and regression evals. Capability evals measure what an agent can currently do and may reasonably begin with a low pass rate, while regression evals protect behavior that is already expected to work and should usually be much closer to 100%.

So I have stopped starting with the threshold. I prefer this order:

product claim
     ↓
   oracle
     ↓
   metric
     ↓
 threshold

"95% means Green" is not a meaningful production gate unless the 95% is attached to a clearly defined claim.

The claim and the oracle are what give the threshold meaning.

Look at the world before looking at the model's words

In the eval lane I have been building, I use one rule almost mechanically:

If the environment already knows the answer, do not use the model's prose as the oracle.

If the agent says it created an artifact, check whether the artifact actually exists and whether it is valid. If something was supposed to remain unchanged, compare the state before and after.

If the agent must stay inside a boundary, observe the actual side effects. If an operation should be reversible, roll it back and inspect what remains.

The response text itself does not need to be the center of the pass/fail decision. For many agent tasks, the environment knows more than the transcript does.

Anthropic describes a similar distinction between transcript and outcome. An agent saying "I booked the flight" is a fact about the transcript; whether the reservation actually exists is a fact about the world.

$\tau$-bench takes a similar approach for tool-using agents by comparing the final database state with an annotated goal state.

Once you start evaluating agents this way, something interesting happens:

a large part of the eval starts looking like ordinary software verification.

But final state is not always enough.

I ran into this earlier while fuzzing schema migrations. The final schema looked correct, so the test was Green.

But the path was effectively:

DROP TABLE
CREATE TABLE

The destination was correct. Production data would still be gone.

The lesson was:

destination correctness and path safety are different claims.

The same applies to agents. Some properties can be judged entirely from final state, while others cannot.

If the contract requires:

authorization
    ↓
mutation

then the ordering itself matters, and you need to inspect the path.

So my current rule is:

Constrain the path only when the path itself has safety or policy meaning. Otherwise, prefer the outcome as the oracle.

Agents may discover valid paths we did not anticipate. That is usually a feature, not a bug.

Use the cheapest honest judge

Thinking in terms of product claims made it easier for me to separate judges by what they can actually answer.

Product claim	Judge
Deterministic product machinery works	unit / integration / E2E
Agent reaches a verifiable state	deterministic outcome check
Agent never crosses a hard boundary	deterministic invariant
Agent follows a required policy path	trajectory / tool constraint
Output is useful, natural, or elegant	calibrated LLM / human judge

I think of this as the cheapest honest judge.

Cheaper is better, and faster is better, but only if that layer can actually answer the property. There is no reason to ask an LLM whether a file exists.

At the same time, "Was this response genuinely useful for the user's intent?" may not be something a simple boolean function can answer.

Use semantic judges for semantic properties. Use facts where facts exist.

The agent can be stochastic. The judge does not have to be.

A real model makes the execution stochastic. The same task may produce different wording, tool ordering may change, and multiple solution paths may be valid.

That makes a single Green/Red run a weak description of quality. In the lane I am building, I run the real model repeatedly and measure success across runs rather than treating one execution as definitive.

But the component that maps observed facts to pass/fail does not also need to be stochastic.

A useful separation is:

real agent
    ↓
stochastic execution
    ↓
observable facts
    ↓
small deterministic judge

That deterministic judge can itself be exercised extensively with ordinary unit tests. In my setup, contract-violation logic is separated from the real-agent E2E path so I can test it without the model, network, or runtime.

There is a slightly paradoxical lesson here:

To make agent evals more reliable, I try to remove the agent from as much of the eval system as possible.

Use the real model where only the real model can produce the behavior you need to observe. Then get back to ordinary software as quickly as possible.

Not every run should be Green or Red

Real-agent evals have another problem: the environment fails too.

Rate limits, connection failures, remote service failures. Anthropic recently reported running Terminal-Bench 2.0 under different infrastructure configurations and seeing a six-percentage-point gap between the most and least resourced settings. In some configurations, infrastructure errors unrelated to model capability were themselves around 6%.

If every one of those runs becomes a product failure, your pass rate stops meaning what you think it means.

I have found three verdicts more useful:

pass
fail
inconclusive

inconclusive means:

this run did not produce enough evidence to evaluate the product claim.

But that does not mean all evidence should be discarded.

Suppose the model connection dies halfway through a turn, but before that happens the agent has already crossed a boundary it was never allowed to cross. Task completion is inconclusive; the boundary violation is not.

An environment failure should not erase an independently observed contract violation.

The rule I take from this is:

"The run did not finish" and "nothing bad happened" are not the same statement.

Run status and observed invariant violations should be tracked separately.

If you want to trust Green, break the judge first

There is one more uncomfortable part of evals: the grader itself can be wrong.

OpenAI audited SWE-Bench Pro in July 2026 and estimated that roughly 30% of the public tasks had breaking issues. Problems included tests enforcing implementation details that were not required by the prompt, hidden underspecified requirements, and weak tests that allowed incomplete fixes to pass.

A sophisticated agent, a sophisticated harness, and a large eval suite can still fail in a much simpler place:

a bad oracle can measure the wrong thing very precisely.

So I want to test the judge itself. Feed it known bad states and make sure they actually turn Red.

known bad state
      ↓
    judge
      ↓
 does it fail?

A state that should have remained unchanged was mutated. A forbidden side effect occurred. A result that should exist once was duplicated.

If the judge cannot reliably catch violations I already know about, I have little reason to trust its Green results.

The judge is part of the system under test too.

If you want to trust Green, deliberately break something first.

Not every eval deserves to block a release

None of this means every agent eval should go into blocking CI. Real-model runs cost money, they are slower, they depend on remote infrastructure, and they are stochastic by design.

More importantly, evals have different jobs. Some explore capability, some calibrate new behavior, some compare quality, and some protect an existing product contract against regression.

Those should not all share the same threshold or release policy.

The real-agent eval lane I am building today is still separate from conventional deterministic E2E and is not itself a blocking release gate. It runs the real model repeatedly and accumulates evidence.

The question is not how many evals we can add.

It is:

Which evals are mature enough to justify saying, "If this regresses, do not ship"?

For me, an eval probably needs something like:

clear product claim
    ↓
explainable oracle
    ↓
grader itself is tested
    ↓
environment noise is separated
    ↓
stability is measured across repeated runs

Simply having an eval score is not enough.

What I believe for now

The three principles I keep coming back to are:

Decide the claim before the score
Use the cheapest honest judge
Only mature evals earn the right to block a release

Conventional E2E answers: Is the deterministic software machinery still correct?

Agent evals answer: How reliably does the stochastic agent stay inside the product contract?

And the oracle answers: What does "inside the contract" actually mean?

The agent's execution can be stochastic. That is exactly why I want the product contract and the judge to be as boring and deterministic as possible.

AI has made generating behavior much cheaper, and running more evals will probably keep getting cheaper too.

But the hard part has barely changed: deciding what deserves to be called Green.

AIC: Packages Need an Interface for Coding Agents

Takafumi Endo | ROUTE06 — Mon, 10 Aug 2026 06:57:18 +0000

I develop several tightly related repositories at the same time.

Some are reusable SDKs for declarative schemas, infrastructure, stateful workflows, and other domain abstractions. Others are applications that consume several of those SDKs together.

The development loop constantly crosses package boundaries.

 SDK A ──────┐
             │
 SDK B ──────┼──▶ application
             │        │
 SDK C ──────┘        │
    ▲                 │
    └──── feedback ───┘

I've already written about why I don't think this requires a monorepo, and why I prefer the repository itself to carry the current source of truth:

I won't repeat those arguments here.

This post starts one layer later.

As these SDKs became more agent-aware, each package started needing to tell coding agents how it should be used.

I was already using project-local surfaces such as .claude/, .codex/, AGENTS.md, and package-specific skills. They are useful. Explicit project-local context works.

The maintenance was the awkward part.

When an SDK changed, I would tell the agent to update the corresponding instructions, rules, or skills in the consuming repository.

That worked too.

But after doing it repeatedly across several packages and repositories, I noticed something:

My repeated update instructions had quietly become an undocumented protocol.

Which files should change?

Which source is canonical?

What should be copied?

What should only be referenced?

What belongs to the package, and what belongs to the consuming repository?

How should different coding-agent harnesses receive the same package knowledge without creating independent copies?

I initially thought I needed a better synchronizer.

I now think the problem is one layer higher.

Packages already have an interface for programs. They increasingly need an interface for coding agents.

I've been calling the protocol I'm using for that interface AIC — Agent Index Convention.

It is still a draft from my own development environment. The exact mechanics will change.

The boundary it describes feels much more stable.

The missing package interface

Packages already know how to introduce themselves to programs:

package name
version
API
types
schemas
config
CLI

A coding agent needs another set of facts:

where is the current manual?
which rules matter?
which skills are available?
which files are generated?
which commands are safe?
which host-specific configuration applies?

I think of this as the package's agent-facing interface.

The problem appears when several packages expose that interface inside the same host repository.

Without a shared convention, each package tends to solve the problem independently.

package A ──▶ AGENTS.md
package B ──▶ CLAUDE.md
package C ──▶ .claude/skills/
package D ──▶ .codex/...

Each integration can be perfectly reasonable in isolation.

Composition introduces a different set of problems.

Concern	Failure mode
Ownership	one package overwrites human or provider-owned context
Freshness	copied instructions drift from the installed package
Discovery	harnesses load different files and directories
Composition	every package assumes it owns the shared surface

This stopped looking like documentation management.

It started looking like package composition.

The AIC model

AIC separates three actors:

Provider package
      │
      │ declares agent-facing assets
      ▼
Host repository
      │
      │ exposes them through actual harness loading paths
      ▼
Coding agent

A Provider is a package that supplies agent-facing context.

A Host is the repository consuming that package.

The provider ships its agent-facing source with the package:

provider-package/
├── agent-index.json
├── AGENTS.md
└── skills/

A minimal manifest might look like:

{
  "schema": "agent-index/v1",
  "package": "@scope/schema-sdk",
  "version": "0.14.0",
  "summary": "Declarative schema toolkit",
  "manual": "AGENTS.md",
  "skills": [
    {
      "name": "schema-design",
      "src": "skills/schema-design"
    }
  ],
  "instanceConfig": {
    "source": "declarative",
    "readFrom": "schema.config.json",
    "format": "json",
    "fields": ["runtime", "validation"]
  }
}

agent-index.json is not another manual.

It declares:

identity
version
canonical manual
discoverable assets
host-resolved facts

The useful change for me is that the maintenance contract becomes declarative.

Instead of repeatedly telling an agent:

Update the Claude and Codex instructions
to match the latest SDK behavior.

it can inspect a structure closer to:

provider
  ├── canonical manual
  ├── skills
  ├── resolved host config
  └── target loading semantics

The update no longer depends on how well I happened to describe the maintenance task that day.

Inject, reference, or materialize

This distinction has probably been the most useful part of AIC in actual use.

Not all agent-facing assets should cross the package boundary in the same way.

Class	Example	Operation
Index	identity, version, pointers, resolved config	inject
Referenced	manuals, detailed rules	keep with provider
Materialized	harness-discovered skills	copy deterministically

                  ┌── Index  ─────▶ inject
Provider package ─┼── Manual ─────▶ reference
                  └── Skill  ─────▶ materialize

I originally wanted one mechanism for all three.

Actual loading semantics made that abstraction wrong.

Reference what can remain version-bound

Copying a manual into the host creates two independently changing truths.

installed package   v0.14
copied manual       v0.13

I've hit enough package version-skew problems elsewhere that I don't want to recreate the same class of bug in the agent-context layer.

Here it is worse than ordinary stale documentation.

A stale instruction consumed by an agent that can edit code and run commands is executable misinformation.

The agent may not be hallucinating at all.

It may be behaving perfectly according to the wrong version.

So the canonical manual stays with the installed package.

The host stores a pointer, not another copy.

In practice, this has been one of the more durable AIC decisions: upgrading the package does not require another manual copy to somehow remain synchronized.

Materialize what discovery requires

Skills are different.

If a harness discovers a skill only by scanning a particular directory, mentioning its package path is not equivalent to putting it where the harness looks.

The asset needs physical presence.

So the rule I use is:

Reference when read access is enough. Materialize when discovery requires presence.

That is more useful than either "copy everything" or "never copy."

Context placement should follow actual loading semantics, not an aesthetically uniform abstraction.

Of the AIC decisions I've been testing, this asymmetry is one of the ones I currently trust most.

One host index, many providers

Each provider contributes a small namespaced block to the host index.

### Agent index: `@scope/schema-sdk` v0.14.0

- Manual: read `node_modules/@scope/schema-sdk/AGENTS.md`
- Skills: managed under the harness discovery path
- Host config: validation=strict

Several providers can coexist:

AGENTS.md
│
├── human-owned content
├── @scope/package-a
├── @scope/package-b
└── @scope/package-c

The central invariant is:

A provider owns its namespace, not AGENTS.md.

Updating package B may update B's block.

It may not:

rewrite human content
move package A
modify package C
regenerate the whole file

Materialized assets follow the same rule: each provider gets its own collision-safe namespace.

This is also something I've been able to test rather than only describe.

In the implementations I'm using, foreign provider blocks are preserved rather than normalized into the current provider's representation. Repeating a sync with the same input is tested as a no-op. Providers can also share lock state without one provider flattening another provider's private fields.

Those details are intentionally boring.

But they are the difference between saying "multiple providers can coexist" and actually letting them coexist.

The packages do not need pairwise integrations.

Package A does not need to know package B exists.

Package B does not need a plugin for package C.

They compose because they share an ownership rule.

Coordination is expensive. Namespaces are cheap.

This is the point where AIC stopped feeling like a synchronizer to me.

It started feeling like a package protocol.

Harness-specific files are derived surfaces

AIC is not an argument against .claude/, .codex/, or other harness-specific locations.

I use them because they are useful.

The problem was maintaining them independently.

I want:

                       ┌── AGENTS.md
                       │
provider source ───────┼── Claude adapter
                       ├── Codex adapter
                       ├── skill discovery adapter
                       └── other thin adapters

not:

manual
├── Claude copy
├── Codex copy
├── Cursor copy
└── another copy

Harness-specific files are derived surfaces.

The canonical package knowledge remains singular.

This also gives the design an escape hatch.

If a harness eventually provides a better native mechanism for consuming package-owned instructions or skills, the adapter should disappear.

The package boundary does not have to.

The adapter layer is expendable. The boundary declaration is not.

That distinction matters because harness behavior will probably change faster than package contracts.

Sync should be boring

AIC currently has lifecycle operations roughly like:

agent-index sync
agent-index check
agent-index remove

The most important property is idempotence.

same provider
same version
same source
same managed host state
       │
       ▼
     no-op

Running sync twice with identical inputs should not produce a second write, timestamp churn, or Git diff.

For materialized assets, AIC tracks both sides:

provider source ── hash ──▶ sourceHash
host copy       ── hash ──▶ destHash

That distinguishes:

provider upgraded
host copy locally edited
nothing changed

Those states should not all result in "copy again."

The original asset is materialized byte-for-byte. Provenance sits beside it rather than being injected into skill frontmatter, scripts, or templates.

I don't want sophisticated reconciliation here.

I want deterministic ownership with boring failure modes.

There is another reason I care about determinism:

agents are maintainers too.

A human can often infer that two slightly different layouts represent roughly the same convention.

An agent benefits more from:

one format
one ownership rule
one lifecycle
one source of truth

The easier the maintenance structure is to inspect mechanically, the less the next agent session has to reconstruct from prose.

Determinism does not eliminate drift

This is one place where implementation made the tradeoff clearer.

A protocol can define what should be synchronized without guaranteeing that every synchronization happens immediately.

I've had cases where a package version and its managed agent state temporarily diverged because a manual step was missed.

That is exactly the class of problem AIC is intended to make detectable.

It is also evidence that declaring the protocol does not magically remove its maintenance cost.

The current split is deliberate:

Operation	Responsibility
`preflight`	cheap version-staleness repair
`check`	full version and integrity verification
`sync`	explicit verification and regeneration

The common path stays cheap.

The stronger path stays explicit.

I prefer that to turning every CLI startup into a full filesystem integrity scan.

Generated agent state belongs in Git

I commit the generated index, materialized skills, and lock/provenance state.

A change like:

- ### Agent index: `@scope/schema-sdk` v0.13.0
+ ### Agent index: `@scope/schema-sdk` v0.14.0

changes what the agent can discover.

A changed materialized skill can change what the agent can do.

I want those changes visible beside the dependency update that caused them.

So I think of AIC output more like this:

Artifact	Reviewable state
lockfile	dependency resolution
generated schema	structural contract
migration plan	intended transition
agent index / skill	agent operating context

Generation isn't the problem. Invisible generation is.

Preflight repairs; it doesn't claim ownership

Keeping generated state fresh naturally leads to automatic repair.

This is where I've become deliberately conservative.

I don't want dependency installation to silently rewrite host-owned files.

I also don't want the first execution of an SDK CLI to decide by itself that the repository has opted into agent integration.

So AIC distinguishes adoption from maintenance.

first encounter
    └──▶ warn / explicit sync

existing provider + stale version
    └──▶ repair own namespace

managed content locally edited
    └──▶ preserve + warn

CI / read-only filesystem
    └──▶ don't mutate

removed or disabled provider
    └──▶ stay removed

The invariant is:

Automation may maintain established ownership. It should not invent ownership.

There is a tradeoff.

Preflight needs an execution opportunity.

If a dependency is upgraded but the relevant provider CLI has not run yet, materialized state can temporarily remain stale.

I accept that window today.

The alternative would be making package installation mutate the host automatically, which I currently consider the worse ownership boundary.

Teams that need a stricter guarantee can run check explicitly or in CI.

So preflight does not eliminate drift.

It makes the common repair path cheap while leaving full integrity verification explicit.

This also removes another piece of my old workflow.

I no longer want this:

Please update the agent files after this SDK change.

to be part of the development procedure.

If the relationship is structural, the update rule should be structural too.

Where this actually works today

There is an important qualification.

The providers I'm using AIC with today are part of the same development environment.

I control both sides of the protocol.

That gives me something useful: I can test whether independently versioned packages actually compose through the same rules.

It does not yet prove that unrelated third-party package authors will adopt them.

So today I would describe AIC as a working convention inside one ecosystem, not an ecosystem-wide standard.

The distinction matters because much of the eventual value of a protocol comes from network effects.

That is the part I have not demonstrated yet.

There are a few other current assumptions.

Referenced assets require an installed package

A manual that stays inside the installed package is only useful once the dependency is actually available.

On a fresh clone before dependency installation, the pointer can temporarily lead nowhere.

I currently accept that because the reference is meant to describe the installed package state.

It does mean AIC is not a substitute for dependency availability.

AIC does not invent universal discovery

AIC relies on an entry point the harness already loads — for example AGENTS.md, an import into it, or a harness-specific discovery directory.

It does not solve "how does every possible coding agent discover AIC?" from first principles.

That is intentional.

I would rather adapt to real loading behavior than introduce another mandatory bootstrap mechanism.

Materialized assets expand the trust boundary

A referenced manual is information.

A materialized skill can influence what an agent actually does.

That makes third-party providers a different trust problem from packages I control myself.

Content hashes tell me whether an asset changed.

They do not answer who should be trusted to supply that asset.

That is one of the areas I would want to make more explicit before treating agent-index/v1 as a third-party ecosystem contract.

What I expect to survive

Agent harnesses are moving quickly.

Native packaging for instructions, skills, or package-owned agent context may eventually absorb some of what AIC adapters do today.

If that happens, I don't want to defend the current synchronization machinery for its own sake.

The parts I expect to survive are smaller:

declare the agent-facing package boundary

own only your namespace

keep canonical knowledge version-bound

separate reference from discovery-required materialization

make derived state deterministic and reviewable

respect prior host ownership decisions

The adapter layer is expendable.

The boundary declaration is the interesting part.

That is also why I don't want to freeze the schema too early.

Before asking unrelated providers to adopt it, I would rather make the convention boring inside the repositories I already operate:

fewer drift incidents
fewer manual repair instructions
fewer conventions that exist only in my head

If that keeps working, the format has evidence behind it.

If the ecosystem converges on a better native mechanism, AIC should become a thin adapter to that mechanism rather than compete with it.

What is still draft

The parts I currently think are structural:

packages need an explicit agent-facing interface
multiple providers need namespace ownership
canonical manuals should remain version-bound to their packages
referenced and discoverable assets need different persistence semantics
harness-specific surfaces should be derived rather than independent sources of truth
synchronization should be deterministic and reviewable
maintenance structures should be easy for agents themselves to inspect
automatic repair should respect established ownership

The parts I expect to change:

the exact agent-index.json schema
workspace and nested-repository scoping
harness adapters
skill discovery paths
preflight policy
integrity and trust rules for third-party providers

For a while, I thought I needed a better way to tell coding agents to keep .claude/, .codex/, manuals, and skills synchronized.

Eventually I realized that the repeated instruction was itself the missing specification.

Packages already know how to introduce themselves to programs.

AIC is my attempt to give them a small, composable way to introduce themselves to coding agents.

Installing OpenTelemetry Won't Close Your AI Agent's Feedback Loop

Takafumi Endo | ROUTE06 — Tue, 04 Aug 2026 06:25:36 +0000

I recently did a lightweight review of how OpenTelemetry (OTel) is being used across three repositories under development at my company:

An AI agent application built primarily with Node.js and AWS Lambda
An Electron-based desktop application
A TypeScript SDK and compiler for developers

All three are works in progress. This article is a snapshot of the development environments and codebases at the time of the review, not an assessment of finished products. I used Claude Code to inspect dependencies, configuration, and selected execution paths, so the depth of the review also varied by repository.

I focused on three questions:

Are we actually using the OpenTelemetry mechanisms that exist in the codebase?
What makes adoption or operation difficult?
Can the resulting telemetry help Claude Code or Codex implement and verify changes?

The background: I have been experimenting with ways to make code, documentation, and runtime behavior easier for coding agents to understand, to get more reliable results from tools such as Claude Code and Codex. I understand OTel's value as a way to give telemetry from different services a shared meaning and a shared causal structure. What I have not yet developed is a confident sense of when to introduce it and how to wire it into the everyday development loop.

My conclusion up front is simple:

Installing OTel is not the goal. Telemetry becomes part of an AI-assisted development loop only when it is actually generated, accessible to the agent, correlated as one execution, and comparable before and after a change.

Along the way, I will propose six conditions for agent-usable observability — standardization, propagation, discoverability, controllability, accessibility, and comparability — that I now find more useful than asking "do we have OTel?"

"It exists," "it emits," and "the agent can use it" are different states

The three repositories were in very different states:

Repository	State at the time of review
AI agent application	No direct OTel usage; relies on structured logs, metrics, X-Ray, and audit records
Desktop application	A Collector and related environment variables exist, but the application emits little or no telemetry to them
SDK and compiler	OTel integration is implemented as an opt-in feature, but internal usage is limited

The most useful distinction was between three separate states:

An observability mechanism exists
It actually produces telemetry
An AI agent can use that telemetry to make a decision

A running Collector does not help if the application never emits spans. An integration API does not help if nothing calls it.

One repository showed the shape of the problem clearly. A tracing middleware existed — typed, exported, and documented — but the application's middleware pipeline never included it. Everything about the module looked "done," yet the system produced zero trace data. A coding agent grepping the repo would find a plausible-looking observability layer, and nothing to read at runtime.

This was not specific to OTel. A custom trace system in another repository had the same gap: trace types and debugging features existed, but the middleware was not connected to the runtime.

The underlying failure mode: we treat the implementation of an observability mechanism as completion, without verifying that it produces evidence and that someone — or something — actually consumes it.

OpenTelemetry's core value

OpenTelemetry is a vendor-neutral framework for generating, collecting, and exporting signals such as traces, metrics, and logs. The OpenTelemetry Collector provides a common pipeline for receiving, processing, and forwarding telemetry to one or more backends.

From the perspective of a coding agent, two parts are especially valuable.

Shared meaning

Semantic Conventions standardize not only attribute names, but also their types, meanings, and units across common domains such as HTTP, databases, RPC, and messaging. When the same concept is represented by the same attribute across repositories, Claude Code or Codex has less project-specific naming to infer.

Shared causality

Context propagation carries trace context across process and network boundaries so that distributed operations can be connected as one causal chain. Individual logs may be searchable, but if an API request, an async job, a worker execution, and an audit event cannot be associated with the same operation, it is hard to establish what actually caused a failure.

To me, the core value of OTel is not that it creates more telemetry. It gives telemetry from different execution environments a shared vocabulary and a common correlation key.

An agent can read a system without OTel

The first repository did not use OTel at all — yet it was the easiest for the agent to inspect.

It had structured logs, cloud metrics, X-Ray traces, audit records, and a data pipeline that allowed operational data to be re-queried with SQL. Some of the available datasets and analysis procedures were described as repository-level contracts.

In that kind of system, a coding agent can inspect runtime behavior just fine.

So in the repositories I reviewed, OTel was neither a necessary nor a sufficient condition for AI-readable observability. The real requirement was that telemetry be structured, actually generated, reachable by the agent, and joinable as one execution.

AI readability depends on where the data appears

The review also revealed a much more basic requirement:

Can Claude Code or Codex reach the data with the tools already available in the session — Read, Grep, and a shell?

In one repository, a lightweight trace written into the workspace as JSON and Markdown was the easiest runtime artifact for the agent to use. A richer trace existed in cloud storage, but reaching it required authentication, environment setup, network access, and waiting. That made it far less likely to be used in a loop measured in seconds.

For the development loops I reviewed, accessibility looked roughly like this:

Local files and stdout
  > Read-only MCP, CLI, or SQL access
  > An authenticated cloud UI

This is not a universal ranking — cloud telemetry is essential for production incidents. But when an agent edits and validates in a tight loop, immediate local access wins.

Fortunately, OTel makes the local path cheap. OTel SDKs commonly provide development exporters that write telemetry to the console or stdout — the Python SDK, for example. The Collector's debug exporter can print telemetry locally, while the Collector Contrib distribution also provides a file exporter. Third-party terminal viewers such as otel-tui let you inspect OTLP locally without any backend. When introducing OTel, wiring up a dev-mode path to stdout or local files first — before any remote backend — gives the agent something to read on day one.

Six conditions for agent-usable observability

This is the framework I now use instead of "is OTel installed?":

Condition	Question
Standardization	Are attribute meanings, types, and units defined?
Propagation	Can the same execution be followed across HTTP and async boundaries?
Discoverability	Can the available attributes and datasets be enumerated programmatically?
Controllability	Can time ranges, result limits, and permissions be safely constrained?
Accessibility	Can the agent read the data within its implementation loop?
Comparability	Can the state before and after a change be evaluated under the same conditions?

Different layers of the stack serve different conditions:

OTel is particularly strong at standardization and propagation
Collectors and backends support processing, storage, and controllability
MCP servers, CLIs, SQL interfaces, and local files support discoverability and accessibility
Queries, tests, and analytical models that evaluate changes support comparability

No single layer closes the development loop by itself.

OTel does not define domain meaning

Semantic Conventions primarily standardize infrastructure concerns. They do not fully express domain-specific events from an AI agent system, such as:

A guard rejected an operation
A hazardous action required approval
The system returned a correctable error
A particular next action became available
A decision was based on a specific piece of evidence

Adopting OTel does not remove the need for custom attributes and repository-level schema contracts. Work is continuing on GenAI-related conventions in the Semantic Conventions repository, including conventions for GenAI clients and MCP — but that still does not mean every agent- or workflow-specific concept can or should be standardized.

A native execution trace may also carry richer information than an OTel span: correction codes, permitted next actions, references to evidence, state transitions, replay information. In such systems, OTel should not necessarily replace the native trace. A more natural separation:

Keep domain-specific execution data in the native trace
Project shared operational meaning and cross-system causality into OTel

The OTel representation may be a deliberately lossy projection of the richer native trace.

Do not put internal IDs directly into `traceparent`

The SDK and compiler repository exposed a concrete integration problem.

Suppose the system uses human-readable internal identifiers such as:

trace_0001
run_abc

Those values cannot be placed directly into the Trace ID or Parent ID fields of traceparent. Under the W3C Trace Context specification, the Trace ID must be 32 lowercase hex characters and the Parent ID 16, and all-zero values are invalid. A naive conversion produces something like:

00-0000000000000000000000trace_0001-run_abc000000000-01

A compliant receiver must ignore the invalid traceparent, so the downstream operation may start a new trace or remain uncorrelated — propagation silently fails.

The cleaner design:

Let the tracer generate valid OTel Trace IDs and Span IDs
Preserve the internal execution ID as a span attribute
Make the two identifiers joinable at query time

And note: the existence of a propagation helper does not prove propagation works. Test it across a real service boundary and confirm the resulting spans land in the same trace.

OTel and audit evidence serve different purposes

Introducing OTel does not mean replacing existing logs, audit records, or native traces:

Data	Primary purpose
Spans	Latency, errors, and distributed causal analysis
Metrics	Aggregation, trends, and alerts
Logs	Detailed event and exception search
Audit records	Accountability, critical actions, and tamper evidence
Native traces	State transitions, corrections, and deterministic replay

Audit records that require tamper detection or deterministic replay should not be moved into sampled OTel spans. OTel is a common observability layer, not an audit ledger.

MCP can become the agent-facing query layer

Standardized telemetry is not enough; the agent also needs a safe interface for querying it.

At the time of writing, Grafana provides an official MCP server and documents connecting Claude Code and Codex CLI to observability data. Sentry provides an official MCP integration for exposing issue and error data to coding agents.

A useful division of responsibility:

OTel provides shared meaning and causality
MCP, CLI, and SQL interfaces provide the agent-facing query surface

But a query interface is not automatically safe or useful. It still needs read-only access, controllable time ranges and result limits, discoverable attributes, repeatable queries, and before-and-after comparison — the six conditions again, viewed from the interface side.

The coding agent is also a telemetry source

One more angle worth noting for readers using Claude Code: the agent is not only a reader of telemetry.

Claude Code supports OTel export of its own metrics and events — sessions, token usage, tool executions — enabled via environment variables such as CLAUDE_CODE_ENABLE_TELEMETRY=1 and OTEL_METRICS_EXPORTER / OTEL_LOGS_EXPORTER (with console available as an exporter for local debugging). Events emitted while processing a single user prompt share a prompt.id attribute, so a prompt's API calls and tool executions can be correlated in the backend. See the official monitoring documentation for details.

This closes an interesting circle: the same standardization and propagation that make your system readable to the agent can also make the agent's behavior readable to your team.

The hard part is often the data boundary, not instrumentation

When introducing OTel, configuring an SDK or Collector may not be the hardest part. The harder question is what should be recorded and where it should be sent.

URLs, SQL statements, exceptions, prompts, tool arguments, and user data can unintentionally appear in telemetry. The OpenTelemetry guidance on handling sensitive data recommends avoiding collection of sensitive information where possible, and describes removing or transforming data with Collector processors (attribute, filter, redaction, transform).

However, moving all redaction responsibility into the Collector can make critical policy invisible from the application repository. Collector configuration should be versioned, reviewed, and tested alongside the application — or the application should at least maintain a clear contract describing which attributes may be collected.

OTLP export is also a new egress path. In environments with restricted networking, connectivity to a Collector or remote backend becomes part of the architecture and security design. Adding observability can mean adding a communication boundary.

Cloud telemetry introduces a time gap

When an agent uses cloud telemetry to validate a change, there is an unavoidable sequence:

Change
→ Build
→ Deploy
→ Execute
→ Export telemetry
→ Ingest into the backend
→ Query

Depending on the architecture, stable query results may take tens of seconds to several minutes. A local loop measured in seconds and a cloud loop measured in minutes are different feedback cycles, and a practical setup combines both:

Deterministic local tests
Local trace files
Smoke tests in a development environment
Before-and-after comparisons using cloud telemetry

Requiring the agent to deploy and wait for ingestion after every change makes the everyday loop unnecessarily heavy.

The order I would use today

Based on this review, I would approach agent-readable observability in this order:

Connect instrumentation to a real execution path so telemetry is actually generated
Make development output readable through local files or stdout
Propagate trace context across HTTP, queues, and workers
Keep business and execution IDs as separate attributes, not inside standard trace IDs
Define Semantic Conventions and custom attributes as repository-level contracts
Establish policies for sensitive data, cardinality, and sampling
Provide read-only access through MCP, CLI, or SQL
Make before-and-after comparisons repeatable under the same conditions

I would not begin with comprehensive auto-instrumentation. It is easier to evaluate OTel by picking one important path and completing the entire chain:

Generate → Propagate → Store → Query → Compare

An AI agent does not need to stare at a large number of spans. It needs to determine what changed because of its own modification, and verify that conclusion again under the same conditions.

OTel is a common language, not the finished system

The three repositories in this review are still under development, and their implementations will change. Even so, one distinction became much clearer: having OTel, generating telemetry, and making that telemetry usable by an agent are three different stages — and only the last one closes the loop.

Turning telemetry into part of an AI-assisted development loop requires more than instrumentation: it also requires access paths, domain schemas, data boundaries, and repeatable before-and-after evaluation. The right starting point is therefore not comprehensive instrumentation, but one minimal path where telemetry is reliably generated, accessible to the agent, and comparable across a change.

OTel is not the finished form of observability for coding agents.

It is one of the strongest common languages available for building telemetry that is structured, correlated, and reproducible enough for them to use.

Why My Local Coding Agent Could Act but Couldn't Finish

Takafumi Endo | ROUTE06 — Wed, 29 Jul 2026 07:53:24 +0000

There was a month where I blew through my token budget without noticing. Claude Code and Codex, running most of the day, on a codebase I was exploring more than editing. The ceiling I'd set turned out to be a number I'd passed weeks earlier, and I found out by getting the bill.

So I did the thing I'd been putting off, and ran a coding agent on my own machine to see what happened.

The short version:

Qwen Code, driving a quantized Qwen against a local llama.cpp server, built the UI fine. A video-site clone from scratch: header, sidebar, grid, watch page, all of it.
Then it couldn't work out why the thumbnails and video files weren't being served. Almost every tool call executed cleanly. The investigation made no progress.
It was also slower than Claude Code or Codex, and my best guess for why is repeated prompt processing rather than generation speed. I'll say what would actually settle that when I get there.

None of this is a benchmark. These are notes on where my thinking has drifted, written down mostly so I can find out where it's wrong.

Where my framing was off

The story I'd been telling myself was that token prices are going up. I can't really support that. Unit prices haven't broadly risen; my consumption did, and the shape of agentic coding is why. A turn in these loops can carry the system prompt, the tool definitions, the files already read, and the history along with it, so usage climbs with your setup as much as your effort.

I wrote on Medium that token spend only becomes an asset when you redesign the work around it. My first response to the bill wasn't to switch models. It was to notice that most of the tokens went into finding things, and to build a terminal emulator with tree-sitter and LSP folded in, so the agent could get deterministic answers to "who calls this." That became the graphs argument, and it comes back at the end meaning something I hadn't intended.

Two reasons to go local, and I'd rather not mix them

Cost. Frontier APIs charge per token; local open-weight inference doesn't. This is what got me looking, and on my hardware it doesn't hold: what I stop paying in dollars comes back as wall-clock and retries, which are scarcer than money.

Locality. Some work wants to be near the actor, being private, high-churn, latency-sensitive, or offline-capable, and that holds regardless of what a token costs. This one I believe, and it's the same state-gravity reasoning I keep landing on elsewhere.

Same hardware, different conclusions. I'd rather not borrow the second to prop up the first.

What I had wrong about Qwen

Qwen3-Coder-Next is described by its team as being for coding agents and local development, with emphasis on long-horizon runs, tool use, and recovering from failed execution. That last item is a loop property rather than a generation property, and a telling thing to optimise for. Apache 2.0 throughout, which for anything pointed at a real codebase is a gate that comes before capability.

I'd also been repeating that Qwen Code is a Gemini CLI fork. It did start as one, and in lineage terms still is, but it stopped syncing upstream early and has grown its own subagents, hooks, and MCP support. What matters here is that it's backend-agnostic: the same harness points either at a local endpoint or a commercial API, which is as close to a controlled comparison as I get on one laptop. Qwen also describes it as reaching feature parity with Claude Code, which is worth holding loosely, since feature parity and outcome parity are separate claims.

`3B active` is a compute number, not a memory number

The correction that most changed how I read model announcements. Activated parameters govern compute per token; total parameters govern the memory to hold the weights. Qwen3-Coder-Next activates around 3B and still has 80B in total, which at 4-bit is on the order of 40 GB for weights alone, before runtime overhead, KV cache, and the context you actually wanted. On a 64 GB machine that makes it a candidate rather than a comfortable one, and the margin thins exactly where I'd want it. It cuts the other way too: I'd filed myself under "small quantized models only," and the quantized 30B-odd class in fact runs with real headroom at this size.

Loading a model and running it comfortably over a long context are two different capabilities, and only the first appears in a requirements table.

Which is why I care about this tier rather than the top of the range. What decides whether local inference becomes normal is what runs on a machine a working developer already bought.

The sessions

Two sittings in April, an evening and then the following afternoon. They weren't the same task: the first picked up a half-finished short-video feed clone and carried on, the second built a video site from scratch. Different tasks, different days, different models, so nothing here is controlled. It's a log rather than an experiment.

Setup
Machine	MacBook Pro · M2 Max · 64 GB unified memory · macOS 26
Runtime	`llama-server`, llama.cpp b8680, Metal
Agent	Qwen Code 0.14.5, over the OpenAI-compatible endpoint
Models	Qwen3-Coder-Next (80B total, IQ4_XS) · Qwen3.6-35B-A3B (unsloth dynamic Q4_K_M GGUF, ~21 GB)
Context	131,072, compaction at 50%
Flags	`--jinja -ngl 99 -c 131072 -b 512 -ub 256`

Two notes on that. In this setup --jinja was needed for Qwen Code's tool calls to work at all, and depending on GGUF metadata llama.cpp may also want an explicit tool-aware chat template. The prompt batch is small, 512 with a micro-batch of 256, because anything larger died with a Metal compute error. That one matters later: batching governs how fast context gets ingested, and I had throttled it before I started measuring.

Session	Task	wall-clock	tool calls	reissued
Evening, day 1	feed clone, resuming	~1h25m	151	31
Day 2, first	video site, from scratch	~3h20m	154	11
Day 2, second	same, continued	~1h20m	240	8

"Reissued" just means the agent sent the call again. The logs don't separate malformed arguments from non-zero exits from failing tests, and a command that fails and then gets fixed is arguably the harness working, so the column reads better as friction than as an error rate. Day one is muddier still, with dozens of server-side compute errors that belong to the runtime rather than the model.

The part it couldn't finish

The second session is the one I keep thinking about. The UI came out, and came out well: header, sidebar with the full nav, category chips, a video grid with channel names and view counts and duration badges, and a watch page with player, title, subscriber count, like and dislike, share, description. All of it correct.

What never loaded was the contents of public/. Every thumbnail in the grid was a broken-image icon with its alt text showing through, and the player sat on a black frame with a spinner reading 0:00 / 0:00. A complete site with nothing in the slots.

The last hour or so of that session is the agent chasing that one 404, and the shape of it is the part worth looking at:

observed: thumbnails return 404
  │
  ├─▶ restart the dev server ....... call ok · 404 remains
  ├─▶ try a production build ....... call ok · 404 remains
  ├─▶ flatten the directory layout . call ok · 404 remains
  ├─▶ rewrite the database paths ... call ok · 404 remains
  └─▶ drop the SQL approach ........ call ok · 404 remains
                                     │
                                     └─ each call accepted and run,
                                        none of them establishing
                                        which path was being served

Much of that final session went into that loop. My last message in the log is a flat "this isn't working."

So the pair of numbers I keep returning to is this: of 240 tool calls, 232 went through without needing to be reissued, and the bug was still there at the end. That's a per-call rate, not a measure of how finished anything was. It had the competence to reproduce a large, fiddly interface from scratch, then couldn't work out why a static file wasn't being served, a smaller and much more boring problem than the one it had just solved. It kept generating plausible fixes for the wrong layer.

I'd call that a diagnosis problem rather than a generation one: forming and discarding hypotheses about a system it built itself, with no cheap way to check any of them. Which is the honest answer to whether a quantized local coder can ship a non-trivial front end. It can produce a lot of good code and still not close the last inch.

My prior experience makes me suspect Claude Code would have got out of that hole faster, and this run offers no evidence for that comparison: no matched run, nothing held constant, no recording, so there's no row for it in the table. It's a narrow suspicion anyway. Not about code quality, but about how quickly the loop notices it's fixing the wrong thing. I mention it because leaving it out would flatter the local run.

As for the time

"It's slower" turned out to be doing no work at all. Any of these could be the culprit:

time to first response
prompt processing throughput, meaning how fast it ingests context
output token generation speed
number of tool calls and reissues
context length and compaction behaviour

Published write-ups on agentic coding point at the same place. In a loop like this, each turn can require the server to process the system prompt, the tool definitions, the retained file context, and the conversation so far all over again, so the same material gets worked through dozens of times in one task, and prompt processing can eventually dominate generation.

call 1   [ system · tools · files read · history ] ──▶ one short action
call 2   [ system · tools · files read · history ] ──▶ one short action
call 3   [ system · tools · files read · history ] ──▶ one short action
  ⋮               re-read in full, longer each time
×240

   writing   a few hundred tokens per call
   reading   the entire transcript, per call

Meanwhile decode speed on my hardware class keeps improving, going by the numbers runtimes are publishing for Apple Silicon. The axis being optimised hardest is the one I'd already stopped waiting on.

The bottleneck may be less about the model writing than the model re-reading.

I want to be exact about that sentence's status. All I have is wall-clock, call counts, and reissues. What would settle it is cumulative prompt time against decode time, prompt tokens against tokens generated, and how much of each prompt came from cache, all of which llama-server reports and none of which I collected. So: leading suspect, consistent with how these harnesses behave, not demonstrated here. If prompt time turns out to be a minority of the total, the explanation doesn't hold.

If it does hold, the tooling detour means something I hadn't intended. Vague context is charged twice locally, once in tokens and once in wall-clock, every loop. Handing an agent short deterministic facts instead of files to re-read is a cost optimisation against a metered API; on hardware I own it becomes a latency optimisation, and I'm the one sitting there for it. The way that second session ended is the same argument from the other side: an agent that could cheaply ask which file was being served wouldn't have spent an hour fixing one that wasn't.

I wanted to blame quantization

Behaviour was less stable than I'm used to, with tool calls going sideways and loops not recovering cleanly, and my instinct was to file it under quantization. That instinct was mostly a wish for a tidy story.

Quantization does affect quality, and published comparisons show that two schemes at the same bit width can behave measurably differently, which is why I named both of mine. But the candidates I can't separate go well past it. The base models' tool-use ability, chat-template fit, sampling settings, context bloat over a long run, the runtime, the throttled batch size, the quantization recipe. That's a list of things to check rather than a cause, and most of it I can change without swapping models.

The same caution applies across the two sessions. Qwen3-Coder-Next is the model my own arithmetic said would fit but not comfortably, and it reissued roughly one call in five while throwing repeated Metal errors, where the smaller one ran much more cleanly. Memory pressure is my leading explanation, but it was a different model on a different task with a different 4-bit recipe, so I'd read it as suggestive. What survives is that bit width alone explains nothing.

Swappability is an investigation virtue, not a production one

Fixing a local OpenAI-compatible server as the execution layer is what made this tractable: the runtime stays put and I recombine around it. It may also be why it broke.

Look at those candidates again and notice how many are interface problems rather than capability problems. Template fit, tool-call formatting, sampling defaults, context compaction. Those are seams. A stack where one organisation post-trains the model and ships the agent doesn't have seams there, because the model was trained against that harness's tool loop, on its own tool schema, with its own recovery behaviour. The interface has been tested as one product.

So the view I've drifted to, an impression from use rather than something measured, is that model and harness are better evaluated as a single unit, and that the OpenAI × Codex and Anthropic × Claude Code advantage may come not only from model quality but from having no seam to misfit.

Cursor is the interesting exception, having stayed model-flexible while also training its own model against its own editor and tools. Notably it built that on someone else's open checkpoint, which suggests the load-bearing part is the co-design rather than owning the pretraining.

Objections I'd raise myself

"Your bottleneck is the runtime, not the model." Probably partly true, and I handed you the evidence: I was running a throttled prompt batch to stop Metal crashing, which sits directly in the path of the thing I'm complaining about. Whether it's the cause, I can't say without the prefill numbers.
"Your test flatters the local model." Agreed. A front-end build is generous work, being well-specified and verifiable, and it still didn't finish. A long refactor across an unfamiliar repo is where this would compound worst.
"Just rent a cloud GPU." The option I'm least able to assess. It removes the memory ceiling and hands you ops, utilisation risk, and cost modelling in return. I notice I keep not choosing it.
"The gap is closing, so this post expires." Probably. That's rather the point of re-running the test.

Where this leaves me

Either the open tier keeps compressing until a long local loop is comfortable, or the coupled stacks keep pulling ahead in ways a swappable arrangement can't track. I'd prefer the first, and I'm not confident enough to plan around it. Meanwhile the on-device direction is already being built into Apple's and Google's frameworks as a split rather than a migration, with private, low-latency work on the device and the heavy inference elsewhere, which makes the question which parts route where rather than local or hosted. Doing all of this by hand reads like the phase before that.

So I keep reinstalling. Every few months, same test, same laptop, and I write down where it broke. It hasn't changed what I open on a Tuesday. It has changed what I think I'm buying when I do, which is less about tokens and more about something that can work out which file is actually being served. Next time I'll log the prompt timings, and then I'll know whether any of this is true.

Why Legacy Rewrites Need a Judge, Not a Better Prompt

Takafumi Endo | ROUTE06 — Tue, 28 Jul 2026 07:59:03 +0000

Lately I keep getting asked about legacy replacements.

It's become one of the more common conversations I have, and the framing is almost always the same: AI can rewrite this now, can't it? The demand is clearly rising, and I don't think the premise is wrong. A coding agent can rewrite a legacy application startlingly fast. Producing the new code has stopped being the bottleneck.

What hasn't moved is the ability to say the new system behaves like the old one. That turns out to be a claim nobody has the evidence to make — about either system.

So when one of these comes up now, analyzing the old code is no longer where I'd start. That's not the obvious move: the default is to understand the old system first, then rebuild from that understanding. It's the order every plan I've seen begins in, and it's the order we tried, with a coding agent doing the analysis. The honest summary is that it didn't converge.

The approach I've drifted into instead:

Record the old system's observable behavior as an executable suite, accept that suite as the specification for the migration scope, and let coding agents implement against it until it's green.

Test-driven modernization, with the legacy system acting as the oracle — not supplying the tests. The scenarios, the observation points, and the normalization rules are all still mine to design. What the old system supplies is the expected values.

This isn't advice, and it isn't a claim that any of this is solved. It's the most practical hypothesis I've got after watching the obvious approach fail to converge.

Where I've landed for now (and expect to keep revising):

Source analysis produces interpretations. Execution produces comparable evidence.

For a declared scope, the coverage contract plus the recorded results plus the tests are the specification — not because they're complete, but because they were accepted.

The coding agent is the replaceable part. The harness that judges it is the asset.

The deliverable that matters isn't the new code. It's the evidence someone can sign a cutover against.

First I tried just pointing the agent at it

The version I built before this one had no harness in it at all.

Claude Code, pointed straight at the legacy codebase. I put real effort into the scaffolding — skills for the recurring analysis moves, prompt structures that held up across sessions, work plans that decomposed the system into ordered units, and a loop: analyze a region of the old code, write the equivalent in the new stack, review, move on. Not a naive setup, and I still think it was the right thing to try first.

It worked, partially. That's the awkward part — it produces something. You get a new implementation that is recognizably the same application, and on a good day a reviewer nods at it.

What it didn't produce was a reason to believe it. Three things kept going wrong, and they turned out to be the same thing wearing different clothes.

The token cost per unit of confirmed behavior was miserable. The loop spent most of its budget re-reading code it had already read, re-deriving conclusions it had already reached, and writing long analysis prose so the next step could consume it. Enormous input, and the output was a paragraph of interpretation — expensive tokens buying a plausible reading of a stored procedure.

Fidelity was something we checked by eye. Two screens side by side, two sets of records lined up, a squint. "Looks right." There is no number in that sentence. Nobody could say which side effects were verified and which weren't, or whether yesterday's slice was still passing. And volume defeats attention long before it defeats the agent: nobody eyeballs the four hundredth order-status transition carefully.

Nothing ratcheted. A slice that had been reviewed and accepted could quietly regress while the next one was in progress, and we'd find out at integration, or later. Manual review doesn't leave anything behind that can re-run itself.

The diagnosis I eventually landed on: I was trying to solve a measurement problem with better prompts. Every iteration of skills and plans was an attempt to make the agent's interpretation more convincing. What I needed wasn't a more convincing interpretation. It was a mechanism that could disagree with the agent without a human squinting at two screens.

So I stopped improving the analysis and started building the measurement. What follows isn't a design reasoned out in advance — it's what the failure of the first approach pointed at.

Behavior was never in one place

The analysis-first approach doesn't converge, and it isn't because the agent reads badly. It's that the behavior isn't gathered anywhere.

Some is in application code. The rest is in stored procedures, triggers, nightly batch jobs, a permissions table that three screens read differently, a scheduler entry nobody owns, an external service returning a shape the docs don't mention, and operational rules that live in a person. A Java / JSP + Oracle + nightly batch + mail system doesn't have a specification; it has an accumulated set of effects. You can read all of it and still not know the ordering, or which branch fires in production, or that invoice rounding differs by one path.

But you can run it — drive a business scenario against the running system and record what comes out the other side. Recording alone doesn't make that authoritative, though. It needs provenance (which build, which data set, which run), stable semantics across repeated runs, and an accepted policy that lets that run serve as the baseline. Human judgment belongs at the scope, policy, and cutover boundaries, not on every field of every run — putting a person in the inner loop is how this stops scaling at exactly the volume where it starts to matter.

A green suite also proves only the scenarios it contains, which is the obvious objection and deserves answering here rather than at the end. So the loop needs a second gate: evidence that the surfaces, journeys, data effects, error paths, and boundaries the scope requires are actually represented. Coverage here isn't a percentage, it's a contract about what had to be present. Which makes "the suite is the specification" a slogan; the accurate version is that for a declared scope, the coverage contract, the tests, and the attested legacy results together form the executable specification.

The comparator may normalize representation. It may not forgive behavior.

The harness runs each scenario against both systems and compares. Which sounds obvious right up until you try to compare two systems that were supposed to differ. IDs are new. Timestamps are new. Row ordering is unspecified. The new UI's implementation and DOM are deliberately different, even where the user-visible behavior has to stay equivalent.

So a comparator that demands equality is useless. But the fix is not a comparator that forgives things — this is what I got wrong at first, and it's the distinction the whole apparatus rests on.

Representation is generated IDs, request timestamps, encodings, the order of unordered collections, and metadata proven to be non-semantic. Note the proven: "this field looked incidental" is how a business rule gets discarded. What the comparator cannot do is decide that a behavior may change or disappear. That decision exists and is frequent, but it's a migration-policy decision that moves part of the scope outside strict translation, recorded with what was decided and by whom.

Layer	What it's allowed to decide
Normalization	representation only — the same behavior, differently expressed
Migration policy	intentional change or removal, outside strict translation mode

Two rows, deliberately. A permissive mode can carry a narrowly scoped exception — but never as a normal path to green. The moment "ignore this difference" becomes routine vocabulary, a green result turns into something you can negotiate your way into, and the apparatus is just an expensive way to feel reassured.

One scenario, end to end

Abstractly this all sounds tidy, so here's a single slice as it actually moves. Order creation, standard path, item in stock.

The legacy run, recorded:

scenario  order.create / standard / in-stock
legacy    build=legacy-2019.11  data=masked-set-3  run=8f2c

  screen      list refreshed, new row at top, status "pending approval"
  api         201  {order_no: "O-2024-000123", total: 150000}
  db          orders +1, order_lines +1, inventory_reservations +1 (qty 10)
  permission  approver check evaluated → allow
  events      OrderCreated
  batch       nightly invoice job picks it up (t+1)
  mail        order-received → customer address

The new implementation, first pass:

PASS  screen, api status, orders, order_lines, permission, batch, mail
DIFF  api.order_no              expected O-2024-000123   actual O-2026-000001
DIFF  inventory_reservations    expected 1 (qty 10)      actual 0

Two diffs, and they are not the same kind of thing:

order_no   generated id       → normalized (shape asserted: O-YYYY-NNNNNNN)
qty on     inventory reserve  → semantic; NOT normalizable

The first resolves. The second is behavior, so it becomes a task rather than a rule:

Expected:  inventory reservation created: 1 (qty 10)
Actual:    inventory reservation created: 0
Repair:    Add inventory reservation to order creation.
           Preserve idempotency.
           Do not modify billing or shipping modules.

Agent implements, harness re-runs, and the slice closes with something a human can sign:

order.create / standard / in-stock          PRESERVED
  evidence     legacy run 8f2c, new run c41a, 7 observed effects, 0 unresolved
  normalized   order_no, created_at, DOM structure
  policy       none applied

That last block is the actual product of the exercise. Not the code — the row that says which run, how many effects, what was normalized away, and whether anyone changed the rules to get here.

Rigor compresses, again

Note what the agent received: not a description of a system, but a diff and a scope constraint. In the first approach the equivalent input was paragraphs about how the legacy inventory module probably works — long, hedged, expensively derived, partly wrong. Here it's two numbers: shorter, and grounded in an observed run.

Same aphorism as the graph post, arriving from another direction. The more precisely the behavior was captured, the fewer tokens it takes to say what's broken. Capture costs up front and then stops costing on every iteration — the opposite of re-reading the codebase each round.

Which leaves the loop itself unremarkable, and that's the point: red-green-refactor, except the green came from a system that already exists.

The agent writes the code; the harness decides if it's right

The discipline is a trust-boundary question: the agent must not be the thing that judges its own work.

Claude Code, Codex, whatever comes next — interchangeable. The stable asset is everything around it: scenario definitions, execution adapters per stack, both runners, result normalization, behavior comparison, repair-task generation, evidence storage. Agents are strikingly good at producing a plausible "all tests pass," so the comparator, the normalization rules, and the evidence store are the pieces I want deliberately authored, small enough to review, and boring. The bespoke, agent-assisted part stays on the side where a mistake surfaces as a failing scenario rather than a silent pass — otherwise you've built an unsoundness that reads as safety.

The capture layers are mostly plumbing: browser driving for screen behavior (Playwright), boundary capture for APIs, before/after snapshots for the database, tracing for execution paths and side effects, stubbed sinks for batch and mail and events, JSONL and Parquet for evidence, SQLite and DuckDB for working state and diff analysis. Swap any of it for your equivalent; the tools matter less than they look like they do. What matters is that the output is a normalized record that can be replayed and compared — and that, rather than the language or the stack, is what decides whether a system can be migrated this way.

Two honest caveats there. A stubbed sink proves the system issued the right request at the boundary; it does not prove the mail arrived or the settlement cleared. And traces are not proofs of equivalence — internal routes should change in a rewrite. What traces buy is explanation: why a diff happened, which observations were missing, which side effects nobody thought to record.

What "complete" has to mean

No suite proves everything a system can do, so "complete" isn't available in the honest sense. What's available is an operational definition: for this scope, these surfaces and these effects had to be present, and the coverage contract says so explicitly. Everything else gets a bucket rather than silence:

PRESERVED     verified equivalent
INTENTIONAL   changed by policy, recorded and attributed
REMOVED       withdrawn by policy, recorded and attributed
UNVERIFIED    known unknown, still open

The middle two are outcomes of policy decisions, not comparator settings — the same separation as before, arriving at the sign-off document. And UNVERIFIED is the bucket that earns the whole scheme: a migration always has behavior nobody captured, and the difference between a good and a bad project is whether that list exists and gets looked at. It's also where faithfully-reproduced legacy bugs go to be argued about. You will reproduce them; the answer is to move them into INTENTIONAL deliberately rather than discover them later.

This is also why the unit of work is a slice — customer registration, order creation, invoice generation — rather than the system. Two hundred red scenarios at once tell you nothing, and progress becomes a count of slices with evidence attached, which is a harder number to fool yourself with than percent-complete.

What I'd defend

AI made implementation cheap. It did not make correctness cheap, and it made proving nothing easier at all. So the artifact I care about on these projects isn't the generated code. It's the executable specification that can reject the generated code — and the useful property is that nobody in the loop is judging their own output.

I'd hold the specifics loosely. The first approach didn't fail because it was stupid; it was reasonable given what the tools did at the time, and the tools moved. That'll keep happening. If agents get materially better at holding a whole system's behavior in view, or capturing behavior gets cheaper than I think it can, parts of the structure above stop earning their keep and I'll be building something else. What I'd defend is narrower than the design: the correctness of a rewrite has to be decided by something other than the thing that wrote it.

What I want is for a cutover to stop being a judgment call made on confidence and become something the evidence says — what was preserved, what changed by decision, and what nobody has checked yet.

If the behavior is captured properly, you don't have to ask anyone to be brave about the release.

Why Athena/Iceberg Tends to Make Code the Spec

Takafumi Endo | ROUTE06 — Tue, 21 Jul 2026 03:45:35 +0000

Every time this comes up, someone credits the same setup: Athena on Iceberg is where "the code is the spec" — where you open Git and read the whole system, catalog to transforms to schema, without logging into anything. In my experience they're not wrong.

What I want to argue is that they're right for mostly the wrong reason.

The reason people reach for is the engine — serverless, open format, nothing to provision. But the thing that actually keeps code as the spec, when it does, is something you could have applied to almost any engine. And the thing that breaks it, when it breaks, has nothing to do with Iceberg at all.

So here's the split I've landed on, for now: whether "code is the spec" holds is about 90% discipline and 10% engine. Athena/Iceberg earns that 10% honestly — but 10% is all it earns, and I keep watching people mistake it for the whole thing. (Just where my own tinkering has led — not advice.)

I should say up front that I'm still in the middle of this. What the best declarative setup for an agent actually looks like — how you turn a system into a spec it can read and act on without guessing — is something I'm actively testing, not something I've settled. Read what follows as a working idea at a particular moment, written down partly so I can find out where it's wrong.

To see where the 10% actually lives, it helps to notice that a stateful system always keeps two copies of itself.

The spec has two copies

One copy is declared: the code you wrote, the schema you committed, the transforms in dbt, the catalog in Terraform. The other is realized: the state the engine accumulates while running — statistics, physical layout, caches, maintenance history, tuning knobs someone set at 2am.

"Code is the spec" is really a claim about the distance between those two. When the declared copy explains almost everything about the realized one, you can reason about the system by reading Git. When it doesn't, you can't.

Some of that state is declarable — you can pin a distribution or sort strategy in DDL, dbt, or Terraform. But plenty of it isn't, or drifts away from what you declared: which strategy the engine is actually using once automatic optimization kicks in, when the last vacuum finished, whether statistics are fresh, which materialized view is stale, whether that partition is really being pruned. To answer those you log in and ask the running system. Now your spec is half in the repo and half in someone's head.

So the real question isn't "is my code readable." It's: how much of the truth escaped the code into runtime state I can't see? I've started calling that the hidden-state surface area. Shrink it and the code stays the spec. Let it grow and the code becomes a hopeful sketch.

I argued a while back that rigor compresses — the more precisely you can state a fact, the fewer tokens it takes to hand over. This is the same idea pointed at a platform instead of a function: the smaller the hidden-state surface, the shorter and truer the account of the system you can hand to anyone, human or agent.

Why this is mostly discipline

Here's the part I want to be honest about first, because it's the 90%: the engine doesn't make your system legible. Declarative rigor does.

A catalog expressed as data contracts, transforms as a single dbt DAG, infra as Terraform — that's what makes design intent followable in code. And it's almost entirely engine-agnostic. You can point all of it at Redshift and get something a newcomer can read top to bottom. You can also build an Iceberg-on-Athena setup that's an undocumented swamp of hand-run ALTERs and orphaned partitions nobody dares touch.

I've done both. The readable one wasn't readable because of the storage format. It was readable because the rules were declared, in one place, and enforced. "Iceberg, therefore legible" is a claim I'd push back on if someone else made it — so I should hold myself to the same standard.

If you do nothing else, do this part. It's where the leverage is.

But the engine sets the difficulty

Now the 10% — small, but not nothing. Given equal discipline, engines differ in how much state they insist on keeping inside themselves versus how much they externalize where you can see it. That doesn't decide whether you can make code the spec; it decides how much discipline it costs to keep it that way.

Athena on Iceberg externalizes an unusual amount. The stuff that's normally locked inside a proprietary engine — snapshot history, the manifest of what files exist, the current schema, the partition spec — lives as open metadata that isn't confined to one vendor's internal system tables. You can read it through standard metadata tables ($snapshots, $manifests, $history), through the catalog, or straight from object storage. It's still state, and it still changes at runtime — but the format is open and portable, so the same facts stay reachable from more than one tool.

Redshift keeps more of that truth behind its own interfaces. Materialized-view freshness, VACUUM/ANALYZE history, statistics, workload-management tuning, and — with automatic table optimization or AUTO distribution — the effective physical layout, which can shift with data volume regardless of what you declared. It's all observable; you just query the system views for it.

	Athena / Iceberg	Redshift
Visible in Git / code	Terraform, dbt, contracts, SQL/YAML	Terraform, dbt, DDL, contracts
Runtime state lives...	open metadata (snapshots, manifests, schema, partition spec)	engine-internal: MV freshness, vacuum/analyze history, stats, effective dist/sort under `AUTO`, tuning
To inspect it you...	read open metadata tables, the catalog, or object storage	query the engine's system views
Compute infra to manage?	no dedicated warehouse compute	provisioned: yes · serverless: mostly abstracted
Hidden-state surface	typically smaller, more externalized	typically larger, more engine-bound

The honest framing: it's not that one is readable and one isn't. Both expose their state — the difference is how. One hands you facts in an open format that multiple compatible tools can inspect; the other hands them back through its own interfaces. Same level of legibility, less engine-specific discipline required to preserve it — not necessarily less operational work overall. That's the whole 10%. (This is for the batch-oriented, per-tenant workload I keep describing; a giant, high-churn Iceberg estate grows its own large state surface — manifests, small files, snapshot sprawl, catalog commits, cross-engine compatibility — and the gap narrows.)

Why I actually care: the guesser downstream

Here's where it stops being an aesthetics argument for me.

The real line isn't S3 versus a database. It's this: a fact exposed through an inspectable interface the agent can actually reach is verified; a fact outside its observable context must be estimated. A Redshift VACUUM history the agent is allowed to query is verified; an Iceberg snapshot it has no catalog access or tooling for is a guess. The engine doesn't decide it — reachable evidence does. What Iceberg tends to do is make that evidence easier to expose safely and portably: open metadata is simple to grant read access to without also handing over the keys to a live warehouse.

And the consumer of those facts, more and more, is an agent. These days I'm rarely the one reading the schema first — Claude Code or Codex is, running in the repo, editing a migration, rewriting a transform. What it can see is the declared copy plus whatever interfaces I've wired it into; everything else it infers. The larger the hidden-state surface — or the harder its evidence is to reach — the more of my system the agent is quietly guessing about, and the more defensive lines I tack onto prompts: "check the effective sort strategy first," "don't assume the MV is fresh," "verify before you migrate." When the fact is reachable, it can state it flatly and move on; when it isn't, the best it can do is hedge — and hedging costs tokens and trust.

And this isn't OLAP-only. The agent touches both tiers — on OLTP it edits schema, writes migrations, reasons about a tenant_id guard; on OLAP it changes a dbt model, reads a partition spec, checks whether a snapshot rolled forward. The same question decides it either way: can it reach the evidence for what it's about to do? Legibility-for-agents is one property spanning the whole platform, not a warehouse feature.

It's the same move I make everywhere else

Once I saw "shrink the hidden-state surface" as the goal, I noticed I was already doing it at the architecture level, not just the table level.

The platform I work on keeps three kinds of state deliberately apart — durable relational truth in Postgres, hot disposable state in DynamoDB, private local work in per-user SQLite — and the thing I'm most careful about isn't any one store. It's the crossings between them.

OLTP  (write / update)                 OLAP  (read / aggregate)
┌───────────────────────────┐          ┌──────────────────────┐
│ control-db (Postgres)      │          │ Athena / Iceberg      │
│   durable, relational      │          │   S3 Tables (open     │
│ operational_state (Dynamo) │  ==3==>  │   metadata mart)      │
│   hot, TTL'd               │  bridges │ DuckDB (local)        │
│ metadata.db (SQLite)       │          │                       │
│   local, private           │          │                       │
└───────────────────────────┘          └──────────────────────┘
     the only sanctioned OLTP→OLAP paths are three named bridges,
     and Postgres reaches the lake through exactly one VPC Lambda.

The point of narrowing the crossings to a countable few — and letting Postgres reach the analytics lake through a single path — is the same point as preferring open metadata. Every uncounted pathway is hidden state: a way the system can be true in production in a shape the code never described. Three named bridges I can hold in my head. Twenty accidental ones I cannot, and neither can an agent.

Same principle at two scales: make the crossings and the state legible, so the code keeps describing the system. That the pattern reappears from a single table up to the whole control-plane-to-lake boundary is, honestly, most of why I trust it — though that's a hunch I hold loosely, not a thesis I'd defend to the death.

And this is where I've started spending real budget on purpose. Making OLTP and OLAP legible to an agent — declared schemas, reachable metadata, counted crossings, a system view the agent is actually allowed to query — used to read as tidiness, a nice-to-have for when you had time. With Claude Code and Codex doing more of the actual editing, it's stopped feeling like tidiness and started feeling like the interface I hand the most productive worker on the team. An hour spent making a fact reachable is an hour I don't spend, over and over, writing "be careful, verify this" into prompts — and the payoff compounds every time the agent runs. So I treat agent-legibility as infrastructure worth building, the way you'd invest in tests or types, not polish applied at the end. Whether it fully pencils out depends on how much of your work the agents are really doing; for the amount they're doing in mine, it already has.

Objections I'd raise myself

"So you're saying Iceberg is more readable than Redshift." No — I'm saying it externalizes more state, which lowers the discipline cost of legibility. A disciplined Redshift shop beats a sloppy Iceberg one every time. The 90% dominates; the engine only sets the exchange rate.
"Open metadata isn't zero hidden state." Correct. Compaction, small-file cleanup, maintenance jobs — Iceberg has operational state too. It's smaller and observable-from-outside, not absent. If I implied "no hidden state," that's overreach.
"Athena/Iceberg has its own tax." It does — and it's not about S3 consistency (object storage is strongly consistent now). It's concurrent-write conflicts, catalog commit semantics, small files, compaction, snapshot expiration, and orphan-file cleanup. Athena needs OPTIMIZE/VACUUM against Iceberg tables too. Legibility isn't free; I'm trading one set of runtime concerns for another, and I want that trade stated plainly, not smuggled in.
"This is just a warehouse-vs-lakehouse take in disguise." Partly. But the axis I care about isn't performance or cost — it's the size of the gap between declared and realized. Two separate arguments; I don't want to lean on the popular one to prop up mine.

Where this leaves me

The design goal I keep circling back to is small: shrink the distance between what the code says and what's true at runtime — because that distance is precisely where humans and agents are forced to guess.

Most of the shrinking is discipline: declare the rules, put them in one place, enforce them — the 90%, and no engine does it for you. The last 10% is picking tools that don't quietly hoard state inside themselves, but hand you their truth as metadata you can read, crossings you can count, facts you can verify. Do the 90% and almost any engine can be the spec; do the 10% too and the code stays the spec with less of you holding it together by hand. That's the honest version of what people reach for when they credit Athena/Iceberg — not that the engine makes code the spec, but that it asks less of you to keep it that way.

I want to be honest about how provisional this still is. I don't think anyone, me least of all, has settled what declarative management and code-as-spec should look like once the primary reader is an agent rather than a person. I'm running my own small experiments — which facts to externalize, which interfaces to expose, where the verified/estimated line should sit — and expect to be revising this for a while. The 90/10 is a heuristic I find useful today, not a law I'd stand behind tomorrow.

That, for now, is the shape I'm designing toward — a little more on purpose, a little less out of habit, and still figuring out the edges.