DEV Community: Harish Kotra (he/him)

I built the same agent twice to see what a framework actually deletes

Harish Kotra (he/him) — Sun, 11 Oct 2026 01:57:19 +0000

relay: a hand-written tool loop next to a Strands Agents TypeScript 1.x agent, on the same tasks.

Every agent framework makes the same promise — "you write the tools and the prompt, we own the loop" — and every demo of that promise is a blog post asserting it. I wanted the claim to be checkable, so I built one agent twice and put the two implementations side by side, running against the same three tasks, showing their real tool-call traces, token counts, latencies and line counts.

One catch I set for myself: no flattering measurements. If the framework path needs extra code to produce the same observability, that extra code gets counted.

The two files

Path A is a loop you would recognise from any hand-rolled agent:

while (iterations < MAX_TURNS) {
  const response = await fetch(`${config.baseUrl}/chat/completions`, {
    method: 'POST',
    headers: { 'content-type': 'application/json', authorization: `Bearer ${config.apiKey}` },
    body: JSON.stringify({ model: config.model, messages, tools: openaiToolSchemas, tool_choice: 'auto' }),
  });
  const choice = (await response.json()).choices[0];
  const calls = choice.message.tool_calls ?? [];
  messages.push({ role: 'assistant', content: choice.message.content ?? '', ...(calls.length ? { tool_calls: calls } : {}) });
  if (calls.length === 0) { answer = (choice.message.content ?? '').trim(); break; }
  for (const call of calls) {
    const payload = JSON.stringify(await executeTool(call.function.name, JSON.parse(call.function.arguments)));
    messages.push({ role: 'tool', tool_call_id: call.id, name: call.function.name, content: payload });
  }
}

Path B is the whole agent:

export function createAgent(config: ModelConfig): Agent {
  return new Agent({
    model: new OpenAIModel({
      api: 'chat',
      modelId: config.model,
      apiKey: config.apiKey || 'relay-local-server',
      clientConfig: { baseURL: config.baseUrl, maxRetries: 0 },
      maxTokens: MAX_OUTPUT_TOKENS,
      params: EXTRA_BODY_PARAMS,
    }),
    tools: strandsTools,
    systemPrompt: SYSTEM_PROMPT,
    printer: false,
  });
}

No while. No message array. No tool_calls parsing. No stop condition. The call is
await agent.invoke(prompt, { limits: { turns: 8 } }), and limits.turns is the framework's version
of my guard.

What Strands Agents TypeScript 1.x actually looks like

@strands-agents/sdk is at 1.20.0 (the 1.0 line landed in April). The docs are thinner on the TS SDK
than the Python one, so these are the things I read out of the shipped .d.ts files:

Peer dependencies are real. It wants zod@^4 and openai@^6. npm install @strands-agents/sdk
alone leaves you with an ERESOLVE that names both.

Tools accept a plain JSON Schema, not just Zod. This matters when two implementations must share
one tool definition:

export const strandsTools = TOOLS.map((t) =>
  tool({ name: t.name, description: t.description, inputSchema: t.inputSchema, callback: (input) => t.run(input) }),
);

Same TOOLS array feeds Path A's tools: field on the wire and Path B's framework dispatch. Any
difference in the traces is then a difference in the loop, not in the tools.

Hook events are how you get observability. agent.addHook(EventClass, callback) with
BeforeModelCallEvent, AfterModelCallEvent, BeforeToolCallEvent, ToolResultEvent. That is what
lets Path B emit the same step-by-step trace Path A writes for free.

Metrics come back on the result, and they are richer than what I hand-counted:

const result = await agent.invoke(prompt, { limits: { turns: 8 } });
result.metrics?.cycleCount;                        // framework's own loop count
result.metrics?.latestAgentInvocation?.usage;      // input/output/total tokens
result.metrics?.latestAgentInvocation?.cycles;     // per-cycle duration + usage
result.stopReason;                                 // 'stop' | 'toolUse' | 'limitTurns' | …

The line counts, both ways

loc is computed at request time by reading the actual file and counting lines that hold something
other than a comment or whitespace. Not hardcoded, and the UI's code viewer pulls the same file from
disk, so a screenshot can't lie.

file	code lines
`server/handrolled.ts` (Path A)	146
`server/strands.ts` (Path B, the agent)	24
`server/strands-trace.ts` (Path B's trace bridge)	147

24 vs 146 is the headline. It is also not the whole truth, and here is the part most demos skip: to
make the framework path emit the same trace format the comparison needs, I wrote 147 lines of hook
bridge. Path A has no such file, because its loop is its own instrumentation — the step where it
calls executeTool is already the step where it records that it called executeTool.

So the app shows 24 (+147 trace bridge = 171) rather than picking the flattering number, and the
honest sentence is: the framework deleted the loop, not the understanding of the loop. If all you
need is an answer, Path B is 24 lines and Path A is 146. If you need to show the loop, both sides
pay for that — Path A almost for free, Path B in a separate file.

Fairness controls, because the comparison is worthless without them

Same system prompt, same tools, same MAX_TURNS = 8, same max_tokens, same extra body params — all in server/agent-spec.ts, imported by both paths.
maxRetries: 0 on Path B. The OpenAI client inside Strands retries twice by default; raw fetch in Path A never retries. Left alone, "one model call" would mean different numbers of HTTP requests on each side. (Strands' own DefaultModelRetryStrategy only fires on ModelThrottledError, so pinning the client's retries costs almost nothing.)
reasoning_effort: "none" on both paths. deepseek-v4.1-flash on the default gateway will spend the entire completion budget on hidden reasoning tokens and return content: "". A hand-written loop reads that as "the model returned nothing" and blames the model. Both paths get the same field, so the comparison stays symmetric and the trap is documented instead of mysterious.
/run/both runs the paths concurrently, so each ms covers only its own loop while both share upstream latency. Isolated timings come from running one path at a time.

What diverges, before I ran a single model

Two structural differences show up even when both paths produce the same answer:

Who decides the next step. Path A's stop condition is a line of code you can read and break.
Path B's is a stopReason the framework reports back.

What failure means. Path A throws on a non-2xx with the raw upstream body attached — which is why
the UI can print the gateway's actual error text. Path B classifies: ModelError,
ModelThrottledError, ContextWindowOverflowError. Different failure model, same underlying HTTP.

Two bugs worth reporting

The compiled build couldn't find its own source. serverFile() resolved paths relative to the
running module, which is server/ under tsx but dist/server/ after tsc — so /source and the
LOC counter 500'd in production while working perfectly in development. Fixed by probing candidates
for the directory that actually contains handrolled.ts. The general lesson: any feature that reads
the repo at runtime needs to be tested against the build, not just dev.

A comment stripper that miscounts URLs. A naive "strip from // to end of line" scanner treats
https://api… inside a string literal as a comment and silently undercounts the line. Since the whole
point is a number you can trust, the counter tracks string, template and comment state character by
character. npm run check asserts this case specifically.

Try it

npm install && npm run dev   # :5173 client, :3001 server

Pick a task, hit Run both, and read the two traces against each other. The config panel takes any
OpenAI-compatible base URL — Particle.ai by default, or LM Studio / Ollama / Gemini / custom in one
click — and nothing provider-specific is hardcoded server-side; the config travels with each request.

npm run check verifies the tools and the line counter without a model. npm run verify runs all
three tasks on both paths through the HTTP API and writes an evidence file.

What the stub caught before I spent a single token

To check the plumbing without burning provider quota, I wrote a scripted OpenAI-compatible server
(scripts/stub-provider.ts). The model replies are hard-coded; the tools still really execute, so
calculator genuinely computes and live_fetch genuinely hits the network.

The first thing it caught was a bug in my assumption: Strands always streams. The chat adapter
hardcodes stream: true and stream_options: { include_usage: true }, so my non-streaming JSON stub
killed Path B with ModelError: Stream ended without completing a message while Path A — plain
fetch, plain JSON — ran fine. Every provider I care about supports streaming, but that is a real
asymmetry: the hand-written loop tolerates a dumber server.

The second thing it caught is the interesting one.

Finding: parallel tool calls work on one path and not the other

Task 1 is "what is 1873 * 42, and what is the current UTC time?" — the natural model response is
two tool_calls in one turn. Against a stream that does exactly that:

	Path A	Path B
tool calls executed	2	1
answer	`78666` and the time	the time

Not a fixture artifact. The OpenAI adapter emits a toolUseStart per tool call but only emits
content-block stops when finish_reason arrives (chat-adapter.js:370), while the shared assembler
keeps a single toolName / toolUseId / accumulatedToolInput slot and resets it on every block
start (model.js:199-209). The second start overwrites the first, and that call never reaches the
executor.

The boundary: one tool call per turn is fine — both paths matched exactly on the fetch task (2
model calls, 1 tool) and the fetch-then-sum task (3 model calls, 2 tools, same computed result).

The temptation was to add "make at most one tool call per turn" to the system prompt. I didn't,
because that changes the rules both paths play by in order to hide one path's limitation — which
defeats the entire build.

Code & more: https://www.dailybuild.xyz/project/280-relay

Two agents, two companies, one protocol: building a visible agent-to-agent negotiation

Harish Kotra (he/him) — Sat, 10 Oct 2026 18:29:57 +0000

How consortium works: Microsoft Agent Framework 1.x, A2A for the negotiation, MCP for the work,
and a data plane that says no.

There is a particular kind of agent demo I find unsatisfying. Five agents sit in a group chat, a framework orchestrates their turn-taking, and the "negotiation" is a string of tokens inside one process's memory. You can print it, but you cannot point at a protocol. Nothing crossed a wire.

So I built the version I wanted to look at: two agents that belong to two different companies, talking over a real protocol, where the thing that actually enforces the agreement is not a language model.

BuyerCorp's agent wants one number: average order value by region for Q3. It does not have the data. VendorCorp's agent has the data and will not release anything until they agree which columns, for what purpose, for how long. They negotiate that contract over A2A. When the deal is struck, the query runs through MCP tools — and the MCP server refuses calls that step outside the contract, in code, before reading a single row.

Everything is visible in a Streamlit observer, one message at a time.

It is ~2,900 lines of Python across four processes and a shared core/ package. This is how it is built, and the eight things that cost me real time.

The shape of it

Process	Port	What it owns
`mcp_server.py`	8123	the CSV, the audit log, the scope check
`vendor_agent.py`	8101	VendorCorp's agent, the contract grant
`buyer_agent.py`	8102	BuyerCorp's agent, the negotiation driver
`observer.py`	8501	rendering. Writes nothing but config

Two design choices do most of the work:

The control plane and the data plane are different protocols. A2A carries the argument. MCP carries the data. The bridge between them is a file: when VendorCorp accepts, it publishes the agreed scope to data/contract.json, which the MCP server re-reads on every call.

The buyer talks to the vendor's MCP server directly. That is what makes the refusal interesting. If the vendor's own agent refused its own tool call, the enforcement would be a courtesy. Instead an independent process — one the buyer cannot negotiate with — evaluates the request against the contract and says no.

Step 1: the negotiation is a schema, not a vibe

Every message is one JSON object:

{
  "from": "buyer|vendor",
  "intent": "propose|counter|accept|reject|request",
  "scope": {
    "columns": ["region", "order_value", "order_date"],
    "purpose": "average order value by region for Q3 2025",
    "retention_days": 30
  },
  "rationale": "one sentence"
}

The rule I set myself: reject anything that does not parse, retry once, then fail the run. No "best effort" path, because a best-effort negotiation is how you get a contract nobody agreed to.

The validator is deliberately picky:

# core/negotiation.py
TOP_LEVEL = {"from", "intent", "scope", "rationale"}
SCOPE_KEYS = {"columns", "purpose", "retention_days"}

extra = sorted(set(obj) - TOP_LEVEL)
if extra:
    raise SchemaError(f"unexpected top-level key(s) {extra}; allowed: {sorted(TOP_LEVEL)}")

if expect_from and side != expect_from:
    raise SchemaError(f'expected "from": "{expect_from}", got "{side}" '
                      f"(a reply may not speak for the other party)")

if known_columns is not None:
    unknown = sorted(set(columns) - known_columns)
    if unknown:
        raise SchemaError(f"column(s) {unknown} are not in the dataset schema {sorted(known_columns)}")

Three details worth calling out:

expect_from stops an agent answering on the other side's behalf. If VendorCorp's reply comes back with "from": "buyer", that is a protocol violation, not a cosmetic slip.
Unknown columns are rejected against the peer's published schema. Where does the buyer learn the column names? From the vendor's agent card. A model that invents avg_order_value never reaches the data plane.
Extra keys are rejected. Tempting to add "urgency": "high", and exactly the kind of drift a schema exists to stop.

The retry lives in the model call, not in the UI:

# core/model.py — one round trip that must yield a parseable JSON object
try:
    parsed = extract_json(raw)
except ValueError as exc:
    if attempts > 1:
        emit("model", side, "parse-failed", turn=turn, attempts=attempts, error=str(exc), raw=raw[:800])
        raise ProtocolError(f"{side} reply was not valid JSON after one retry: {exc}")
    text = prompt + "\n\nYour previous reply could not be parsed as a single JSON object" \
                   f" ({exc}). Reply again with ONLY the JSON object: no prose, no code fence."
    continue

Step 2: serving an agent as an A2A endpoint

Microsoft Agent Framework ships the bridge in agent-framework-a2a: A2AExecutor adapts an agent to the SDK's AgentExecutor, and a2a-sdk supplies the routes.

# vendor_agent.py
from a2a.server.request_handlers import DefaultRequestHandler
from a2a.server.routes import add_a2a_routes_to_fastapi, create_agent_card_routes, create_jsonrpc_routes
from a2a.server.tasks import InMemoryTaskStore
from agent_framework.a2a import A2AExecutor

card = AgentCard(
    name="VendorCorp Data Steward",
    version="1.0.0",
    provider={"url": url, "organization": "VendorCorp"},
    default_input_modes=["text"],
    default_output_modes=["text"],
    capabilities=AgentCapabilities(streaming=False),
    supported_interfaces=[AgentInterface(url=url, protocol_binding="JSONRPC")],
    skills=[...],
)

handler = DefaultRequestHandler(
    agent_executor=A2AExecutor(service, stream=False),
    task_store=InMemoryTaskStore(),
    agent_card=card,
)
app = FastAPI()
add_a2a_routes_to_fastapi(
    app,
    agent_card_routes=create_agent_card_routes(card),
    jsonrpc_routes=create_jsonrpc_routes(handler, rpc_url="/"),
)

A2AExecutor only needs something that satisfies SupportsAgentRun — run() plus create_session(). That is the seam I used to put the protocol logic in one place: a VendorService class that validates the inbound message, decides whether the turn needs the model at all, and logs
both sides of the wire.

The agent card is also how the vendor publishes its negotiable schema and its data plane:

AgentSkill(
    id="vendor-dataset-schema",
    name="Negotiable dataset schema",
    description="Column names VendorCorp will negotiate access to. Names only: no rows and no "
                "values are exposed before a contract is granted.",
    tags=columns(),          # ← the schema the buyer validates proposals against
)

At startup the vendor connects to its own MCP server, calls tools/list, and embeds the real input schemas in every grant it issues. So the buyer learns the tool signatures from the same place it learns the columns.

Step 3: calling a peer, and the `context_id` question

A negotiation has memory. VendorCorp must remember what it offered, or an acceptance means nothing. The natural key is the A2A context_id — but does a client-supplied context_id survive across separate message/send requests?

I tested it before building on it, and it does:

session = AgentSession(service_session_id=A2AServiceSessionId(context_id=conversation))
final = await peer.run(body, session=session)

The vendor's A2AExecutor calls create_session(session_id=task.context_id), and task.context_id
came back as exactly the string the buyer sent (conv-A, conv-A, conv-B in the probe). That is the whole basis for the vendor's per-conversation state:

self._offers: dict[str, dict[str, Any]] = {}      # context_id → last scope VendorCorp offered
self._transcripts: dict[str, list[dict[str, Any]]] = {}

Fetching the card is one line with a caveat:

from a2a.client.card_resolver import parse_agent_card
card = parse_agent_card(response.json())

parse_agent_card exists because the served card carries legacy fields (preferredTransport,
url, protocolVersion) that the protobuf AgentCard does not have. json_format.ParseDict
without ignore_unknown_fields=True throws on them. Use the SDK's helper.

Step 4: the data plane is an MCP server, not a function

mcp_server.py is a standalone process on streamable HTTP, so that "kill the MCP server mid-run" is a thing you can actually do to it.

from mcp.server.fastmcp import FastMCP, ToolError

mcp = FastMCP("vendorcorp-data", host="127.0.0.1", port=8123,
              streamable_http_path="/mcp", stateless_http=True, json_response=True)

@mcp.tool(name="query_records", description="Aggregate VendorCorp's order dataset. …")
async def query_records(filters: dict[str, Any], group_by: str | None,
                        aggregate: dict[str, Any]) -> dict[str, Any]:
    dataset_columns = set(columns())
    referenced = _referenced_columns(filters, group_by, aggregate)
    allowed, reason, contract = evaluate(referenced, dataset_columns)
    if not allowed:
        _record_call("query_records", arguments, decision="refusal", reason=reason, contract=contract)
        raise ToolError(reason)          # ← the refusal originates HERE
    result = run_query(filters, group_by or None, aggregate)
    ...

On the caller side, ToolError becomes isError on the wire, which Agent Framework's MCP client raises as ToolExecutionException with the server's text intact:

try:
    out = await tool.call_tool("query_records", **args)
except ToolExecutionException as exc:
    reason = str(exc)                   # the vendor's words, not ours
    emit("mcp", "buyercorp", "refusal", tool="query_records", arguments=args, reason=reason)

That distinction is the demo. A ToolExecutionException is a policy decision from the data owner. Anything else — ConnectionError, MCP server failed to initialize — is a transport failure, and the observer renders them differently.

Step 5: where enforcement lives

core/contract.py holds the one function that decides anything real:

def evaluate(referenced, dataset_columns, *, now=None) -> tuple[bool, str, dict | None]:
    unknown = sorted(referenced - dataset_columns)
    if unknown:
        return False, f"UNKNOWN_COLUMN: {unknown} is not a column of the VendorCorp dataset…", contract
    if contract is None:
        return False, "NO_CONTRACT: no negotiated contract has been published…", contract
    if _expired(contract, now):
        return False, f"CONTRACT_EXPIRED: {contract['contract_id']} expired at {contract['expires_at']}…", contract
    denied = sorted(referenced - granted)
    if denied:
        return False, (f"SCOPE_DENIED: {denied} was not granted by contract {contract['contract_id']}. "
                       f"Granted columns: {sorted(granted)}. Purpose of record: "
                       f"{contract['scope']['purpose']!r}. This call is refused at the MCP server "
                       f"and was not read from the dataset."), contract
    return True, "", contract

Four refusal codes, deliberately distinct, because "the column does not exist" and "you were not granted the column" are different incidents.

And two rules that are code, not prompt — because a prompt is a request and code is a guarantee:

PII_COLUMNS = {"customer_name", "customer_email"}

def _apply_policy(self, message):
    kept = [c for c in message["scope"]["columns"] if c not in PII_COLUMNS]
    removed = sorted(set(message["scope"]["columns"]) - set(kept))
    if removed:
        emit("contract", "vendorcorp", "policy-filter", removed=removed,
             reason="PII_COLUMNS is enforced in vendor_agent.py after the model replies")
        message["intent"] = "counter"
    message["scope"]["retention_days"] = min(retention, MAX_RETENTION_DAYS)   # 90-day cap

If the model over-grants, the grant is narrower than the model's reply, and the log says so.

The second rule closes the hole I thought about hardest. What stops a buyer sending an accept whose scope quietly includes customer_email? Nothing in the message format. So VendorCorp honours an acceptance against its own last offer, never against what the buyer claims to accept:

buyer_cols, offered_cols = set(accepted["scope"]["columns"]), set(offered["columns"])
widening = sorted(buyer_cols - offered_cols)
if widening:
    return {"from": "vendor", "intent": "reject", "rationale":
        f"An accept may only confirm the scope VendorCorp offered; {widening} was never offered, "
        "so this reads as a new proposal rather than an acceptance."}

The grant step also skips the model entirely. Once both sides agree, publishing a contract is a deterministic act; routing it through an LLM adds a chance of hallucinating terms into a document that is about to become an access-control policy.

Step 6: the observer is a log reader

Four processes append to one JSONL file. Cross-process ordering is the only hard part: take an exclusive fcntl lock, read the tail to find the last seq, append, fsync.

with _lock, open(LOG_PATH, "a+", encoding="utf-8") as handle:
    fcntl.flock(handle.fileno(), fcntl.LOCK_EX)
    event["seq"] = _next_seq(handle)          # parse the last line of the tail
    handle.write(json.dumps(event) + "\n")
    handle.flush(); os.fsync(handle.fileno())

The UI rule I cared about: never render a message that did not happen. No seeded timeline, no placeholder exchange. An empty log draws an empty state that tells you what to start.

State is derived, not stored:

def derive_state(events):
    if not events: return "idle", "No event has been written yet…"
    starts = [i for i, e in enumerate(events) if e["plane"] == "run" and e.get("kind") == "start"]
    if not starts: return "idle", "…no negotiation has been requested."
    tail = events[starts[-1]:]
    if any(e.get("kind") in ("error", "transport-error", "connection-error") for e in tail):
        return "error", …
    if any(e["plane"] == "mcp" and e.get("kind") in ("call", "connect", "result", "refusal") for e in tail):
        return "executing", "Calling the VendorCorp MCP data plane"
    return "negotiating", "A2A messages are exchanging between the two agents"

A stored flag would lie the moment a process dies mid-run. Derived state cannot.

Same instinct for the health dots in the sidebar: a 200 from a port proves nothing about which process answered, so each check requires an identity field:

if isinstance(body, dict) and body.get("service") == "consortium-vendor-agent":
    …report up…
elif body and body.get("service"):
    …report "port answered by X — not this build"…

The MCP row goes further: it is reported up only when the vendor successfully discovered tools from it at startup, which is evidence the data plane answered a real tools/list.

Eight things that cost time

A CSV reader that silently produced garbage. [{field: _coerce(raw) for field, raw in zip(fields, record)}] where record is a DictReader row — zipping a list against a dict iterates its keys, so every cell became its own column name and every mean was null. No exception anywhere. Found by printing one result. Correct form: {field: _coerce(record[field]) for field in fields}.
httpx awaits response hooks. The shim for the provider quirk below was written as a plain function, which produced TypeError: object NoneType can't be used in 'await' expression wrapped in a framework exception that did not mention hooks. It must be async def with await response.aread().
api.particle.ai returns metadata as an object on structured replies while the OpenAI SDK types it str | None, so valid responses fail client-side validation. Fixed with the hook from (2): null the field, rewrite the body, never touch a non-JSON (SSE) response.
deepseek-v4.1-flash spends the whole completion budget on hidden reasoning and returns content: "". reasoning_effort: "none" in the request body fixes it; without it, every agent turn looks like an empty model.
Streaming A2A returned nothing. A2AExecutor(stream=False) puts its reply in a task artifact; asking A2AAgent.run(stream=True) for it yielded zero updates and an empty final response on these versions. Non-streaming works, and one request/response per negotiation message is a more honest trace anyway. Card advertises streaming=False to match.
API surface drift in Agent Framework 1.x. MCPClient is gone — it is MCPStdioTool / MCPStreamableHTTPTool now, tool.functions is a list not a dict, call_tool() returns list[Content] rather than a string, and ToolExecutionException lives in agent_framework.exceptions, not the package root. A2AContinuationToken is not re-exported through the agent_framework.a2a lazy shim either.
st.fragment in Streamlit 1.65 returns a plain function — no .run(), so dynamic run_every is not available the way the docs' examples imply. Fixed the interval instead. Also use_container_width is deprecated in favour of width='stretch'.
My own run.sh lied. It reported "observer up" because the port was open — held by a stale instance it did not own, while its own process had exited with "Port 8501 is not available". Now it refuses to start a service whose port is held by someone else and checks the child it spawned is still alive.

What it actually produces

Negotiation, from the captured event log:

 7 a2a      buyercorp   send         buyer:propose  ['region','order_date','order_value','order_id']
 8 a2a      vendorcorp  inbound      buyer:propose
10 a2a      vendorcorp  outbound     vendor:counter ['order_date','order_value','region']
14 a2a      buyercorp   send         buyer:accept   ['order_date','order_value','region']
17 a2a      vendorcorp  outbound     GRANT c-020682
22 mcp      buyercorp   call         query_records
23 mcp      vendorcorp  result       query_records
27 run      buyercorp   answer

The vendor dropped order_id; the buyer accepted the narrower scope; the contract carries a 30-day expiry and the A2A context_id it was negotiated under. The answer is 202 of 624 rows:

region	n_rows	mean(order_value)
APAC	22	138.3195
EMEA	44	173.9941
LATAM	27	162.1626
NA-East	50	188.0616
NA-West	59	224.1207

Same question, but naming the email column, and the data plane answers:

SCOPE_DENIED: ['customer_email'] was not granted by contract c-61cc5c. Granted columns:
['order_date', 'order_value', 'region']. Purpose of record: 'aggregate order value by region for
Q3 2025'. This call is refused at the MCP server and was not read from the dataset.

Kill the MCP server mid-run and the run ends instead of hanging:

{"plane": "mcp", "actor": "buyercorp", "kind": "connection-error",
 "error": "ToolException: MCP server failed to initialize: [Errno 61] Connection refused"}

One honest caveat. Those runs were driven through scripts/mock_provider.py, a scripted OpenAI-compatible endpoint, because no provider key was configured when I captured them. The aggregates and every refusal string are model-independent — the MCP server computes them from the committed CSV, and tests/test_core.py recomputes the means with statistics.fmean and asserts equality. What a real model changes is the wording and column choices inside the negotiation messages. The harness is a plumbing test, and I have labelled it as one rather than letting it pass for a demo.

Try it, or read the interesting files

git clone https://github.com/harishkotra/consortium
cd consortium && uv venv --python 3.12 .venv
uv pip install -p .venv/bin/python -r requirements.txt
cp .env.example .env          # put a key in, or point at scripts/mock_provider.py
./run.sh start                # http://127.0.0.1:8501

If you want to read three files rather than eight:

core/contract.py — the entire policy model in ~120 lines.
mcp_server.py — where a refusal is born.
vendor_agent.py — the part where an agent decides not to call its model.

Code & more: https://www.dailybuild.xyz/project/279-consortium

Guardrails are a lie until they're arithmetic

Harish Kotra (he/him) — Fri, 09 Oct 2026 18:16:16 +0000

How Allowance works: an agent that runs inside a hard spend envelope of tokens, dollars and wall-clock time — enforced in code, with tiered approvals, a negotiation path, and a ledger that reconciles against the provider's own usage numbers.

Built with Inngest AgentKit, TypeScript, Express 5, React 18 and SQLite. About 4,500 lines of app code, plus 474 lines of scenario verification and 278 lines of unit tests.

Every agent framework in 2026 ships a budget knob, and almost every one of them implements it as
a sentence in a system prompt:

You have a budget of 40,000 tokens. Please stop when you approach the limit.

That is not a budget. That is a request. It works right up until the model has a bad day, at which
point the spend is real and the "limit" was a paragraph.

Allowance is a small control panel that treats the envelope as a property of the code:

give it a task + an envelope  ->  watch the ledger fill step by step
                              ->  at 60% it files a budget request; you approve or deny
                              ->  on deny it stops cleanly with a partial result and a cost report
                              ->  reconcile the ledger against what the provider reported

Three things turned out to be much more interesting to build than the UI: the enforcement point,
the accounting split, and durability. This post is about all three, plus the four bugs that
verification caught and I would not have found by reading the code.

1. The mechanic is four lines

Before every model call, in code:

remaining = min( cap_tokens   - used_tokens,
                 usdToTokens(  cap_usd   - used_usd ),
                 timeToTokens( deadline  - now ) )

estimatedCost > remaining  ->  the step does not execute
                               ├─ a cheaper plan fits?  re-plan, log it
                               └─ otherwise            REFUSED { at: <resource> }

The min() is the whole idea. Three independent resources, and the tightest one binds. A run
can have 95% of its tokens left and still be refused because it spent its seconds.

In TypeScript (server/src/agent/budget.ts), with the deadline checked first because it can
expire mid-plan with money unspent:

export function gateFor(run: RunRow, now: number, needTokens: number): GateDecision {
  const envelope = snapshotEnvelope(run, now);

  if (envelope.deadlineBreached) {
    return { allowed: false, reason: "deadline", at: "time", canReplan: false, ... };
  }

  const allowed = needTokens <= envelope.remainingTokenEquivalent;
  if (allowed) return { allowed: true, reason: "ok", at: null, ... };

  const cheaperPlanFits = envelope.remainingTokenEquivalent >= MIN_CALL_TOKENS;
  return {
    allowed: false,
    reason: "envelope_exhausted",
    at: envelope.binding,          // "tokens" | "usd" | "time"
    canReplan: cheaperPlanFits,
    ...
  };
}

at: envelope.binding is what makes refusals useful instead of merely terminal. The UI says
"the tokens envelope was exhausted before the next step could execute — needed 909, headroom
266", and the transcript keeps the numbers.

Money-to-tokens, priced conservatively

export function usdToTokens(usd: number, prices: Prices): number {
  // Priced at the dearer of input/output so headroom is never more generous
  // than the cheapest possible call.
  const perToken = Math.max(prices.per1kInput, prices.per1kOutput) / 1000;
  return perToken > 0 ? Math.floor(usd / perToken) : 0;
}

There is no hardcoded cost anywhere in the project. Dollars come from a price table the user can
edit in the Settings panel — which is precisely why every derived figure is labelled an estimate.
A budget you cannot re-price is a bill, and this is not a billing system.

2. The counter must not live in a variable

This is the sentence the whole build is about, so it's worth being precise about why it's hard.

A JS variable holding remainingBudget is wrong in three different ways at once:

Retries double-count. A step that fails after charging and runs again spends the same money twice, or charges nothing twice, depending on where the throw happens.
Restarts lose it. The process dies, the number dies, and the next run spends like it's day one.
You cannot audit it. The number that gated step 7 no longer exists by the time you're looking at step 7's cost.

So the envelope lives in SQLite, and — this is the part I'd skipped the first time — the counters
are recomputed from the ledger rather than incremented:

export function commitLedgerAndCounters(entry: LedgerInsert): void {
  const tx = getDb().transaction(() => {
    getDb().prepare(`
      INSERT INTO ledger (run_id, step_key, ...) VALUES (@runId, @stepKey, ...)
      ON CONFLICT(run_id, step_key) DO UPDATE SET
        input_tokens = excluded.input_tokens,
        output_tokens = excluded.output_tokens,
        usd = excluded.usd, ...
    `).run(entry);

    getDb().prepare(`
      UPDATE runs SET
        used_tokens = (SELECT COALESCE(SUM(input_tokens + output_tokens), 0) FROM ledger WHERE run_id = @runId),
        used_usd    = (SELECT COALESCE(SUM(usd), 0) FROM ledger WHERE run_id = @runId),
        executed_steps = (SELECT COUNT(*) FROM ledger WHERE run_id = @runId)
      WHERE run_id = @runId
    `).run({ runId: entry.runId });
  });
  tx();
}

Three properties fall out of that one transaction:

Idempotent. UNIQUE(run_id, step_key) means a replayed step rewrites its own row. It cannot create a second charge.
Self-consistent. used_tokens is a SUM over the rows, so the counter and the ledger cannot disagree. There is no drift to reconcile; the invariant is structural.
Auditable. "Every executed step has exactly one ledger row" is a query, not a hope: rows == executed_steps == COUNT(DISTINCT step_key).

The verification script asserts that triple after all five scenarios, and the reconciliation table
prints Envelope counters match the sum of ledger rows as a standing line.

3. Where the framework actually earns its keep

I started skeptical of using an agent framework for something whose point is not trusting the
agent. Two things changed my mind.

Inngest's step.ai.infer makes the provider call itself a durable, memoized step. The HTTP
request is issued by the runtime, its response is stored, and a replay returns the stored response
instead of spending again. That is exactly the property a spend ledger needs, and it is not
something I wanted to hand-roll.

AgentKit's onStart lifecycle hook can stop a call before it is sent. The docs describe it as
able "to adjust the prompt, history, or to stop the agent from making the call altogether". That is
the enforcement point I was looking for, in framework-native clothing:

export function makeBudgetGate(gate: GateOutcome, block: GateBlock) {
  return async (args: { prompt: Message[]; history?: Message[] }) => {
    const messages = [...args.prompt, ...(args.history ?? [])];
    if (!gate.allowed || conservativeCallNeed(messages) > gate.remaining) {
      block.blocked = { ...gate, allowed: false, at: gate.at ?? gate.binding };
      return { ...args, history: args.history ?? [], stop: true };   // nothing leaves the process
    }
    return { ...args, history: args.history ?? [], stop: false };
  };
}

conservativeCallNeed is tokeniser(prompt) + max_completion_tokens. Because the adapter pins
max_completion_tokens on every request, the provider physically cannot return more than the gate
already charged for. That is what makes "the cap cannot be exceeded even when the model asks for
more" a property of the code rather than a bet on model behaviour.

So there are two gates. The orchestration gate decides control flow — execute, re-plan, ask,
refuse — and it runs inside a memoized step. The pre-call gate above is the safety net that
catches any call path that skipped the first one.

Architecture

Browser :5173            Express :3001                    Inngest :8288
┌──────────────┐  POST   ┌─────────────────────┐  events  ┌──────────────────────┐
│ Gauges       │────────▶│ /api/runs           │─────────▶│ allowance-run        │
│ Ledger       │   SSE   │ /api/approvals/:id  │          │  gate:plan           │
│ ApprovalInbox│◀────────│ /api/runs/:id/stream│          │  Planner:plan  ──┐   │
│ Negotiation  │         │ /api/inngest (serve)│          │  ledger:plan   ◀─┘   │
│ Reconcile    │         └────────────────────┘          │  gate:exec:0         │
│ Settings     │                    │                     │  Executor:0    ──┐   │
└──────────────┘                    │ SQLite              │  ledger:exec:0 ◀─┘   │
                                    ▼                     │  wait:0 (pause)      │
                          ┌───────────────────┐           │  grant / refuse      │
                          │ runs   caps+counters│◀────────                      │
                          │ ledger 1 row/step   │           └─────────┬────────────┘
                          │ approvals requests  │                     │ step.ai.infer
                          │ negotiations transcript│                   ▼
                          │ findings            │           OpenAI-compatible provider
                          └───────────────────┘             usage.prompt_tokens / …

4. Reported and estimated numbers must never share a column

The provider usually returns usage.prompt_tokens / usage.completion_tokens. Sometimes it doesn't. The tempting implementation is "use it if present, else estimate, and move on" — which produces a total that is half measurement and half guess with no way to tell which rows are which.

So each ledger row stores both views of every call:

export function resolveUsage(args: { raw: unknown; requestMessages: unknown[]; completionText: string }) {
  const estInput  = estimateRequestTokens(args.requestMessages);  // cl100k, +3 framing per message
  const estOutput = estimateCompletionTokens(args.completionText);
  const reported  = parseProviderUsage(args.raw);                 // null when absent
  return {
    input:  reported ? reported.input  : estInput,   // what we charged
    output: reported ? reported.output : estOutput,
    estimated: reported === null,                     // the badge on the row
    reportedInput: reported?.input ?? null,
    estInput, estOutput,
  };
}

The reconciliation table then sums them separately:

Source	in	out	tokens	$	calls
Ledger charged (all rows)	1,694	563	2,257	$0.000592	4
· provider-reported rows (4)			2,257	$0.000592	4
· tokeniser estimates, labelled (0)			0	$0.000000	0
Provider-reported totals (re-summed from raw payloads)	1,694	563	2,257	$0.000592	4
Tokeniser estimate for every call	855	532	1,387	$0.000447	4
Delta (ledger − provider, attested rows)	0	0	0	0

That second-to-last row is the honest entertainment of the whole project: the tokeniser would have charged 1,387 tokens for four calls the provider billed 2,257 — a 39% under-estimate, because cl100k framing does not count hidden reasoning tokens. When a provider omits usage, that is the error bar you are silently accepting. The verdict line says so in words rather than showing you a tidy zero.

5. Approvals: a tier, a pause, and a clock

When the envelope binds, the agent may ask for more. The ask is a row, never a capability:

export function tierFor(estCostUsd: number, autoApproveUnderUsd: number): ApprovalTier {
  return estCostUsd <= autoApproveUnderUsd ? "auto" : "human";
}

auto — priced at or below APPROVE_UNDER_USD. Granted without a human, but still recorded as an approval row and still applied by the durable run. The tier changes who says yes, not how the yes is written to the caps.
human — everything else. The run pauses in awaiting-approval and the inbox shows the justification, the ask, and the estimated cost.

The pause is step.waitForEvent, which means it survives a server restart:

await step.waitForEvent(`wait:${reason}:${triggerStep}`, {
  event: approvalEventName(runId),     // name carries the run id: no cross-run wakeups
  timeout: `${window.timeoutMs}ms`,
});

And the caps only ever change here, inside a step, after a decision exists:

const caps = capsAfterGrant(run, approval, Date.now());
applyGrant(approvalId, caps);          // writes cap_tokens / cap_usd / deadline_at

The model has no path to that function. Its output is parsed for a plan, never for an authorisation — which is the difference between a request and a budget.

6. Four bugs that only a live run could find

I wrote the enforcement maths, unit-tested it, and it was still wrong in four ways. Every one was caught by running the thing.

a) The observed-throughput death spiral

The time term needs a tokens-per-second rate. My first version used the run's observed throughput — obviously right, measured rather than assumed.

It is a self-referential trap. One slow first call (235 tokens over 40 seconds) implies 5.9 tok/s, which converts the remaining 50 seconds into ~295 tokens of headroom, which refuses every subsequent step — while the run still has 97% of its tokens and 99% of its money unspent. The first slow call dooms the run.

The fix is to derive the rate from the envelope itself:

export function tokensPerSecondOf(run: RunRow): number {
  if (Number(process.env.TOKENS_PER_SECOND) > 0) return Number(process.env.TOKENS_PER_SECOND);
  return run.capSeconds > 0 ? run.capTokens / run.capSeconds : DEFAULT_TOKENS_PER_SECOND;
}

Now the envelope is internally consistent — the token cap is exactly what fits in the time cap — and time binds proportionally as the clock runs down. Observed throughput is still computed and displayed; it just never feeds enforcement. There is a regression test named prices time headroom off the envelope, not observed throughput.

b) Reading live state during a replay — the good one

Inngest re-executes the function body from the top whenever it resumes: after a retry, and after waitForEvent returns. Memoized steps replay their stored outputs. Everything not in a step is recomputed for real.

My onStart gate read SQLite directly. So when a run resumed from an approval pause with the envelope now genuinely nearly empty, the hook blocked a planning call that had originally succeeded, the replay took a branch the original run never took, and it stamped REFUSED{at:"tokens"} over the top of a clean denial.

The invariant that fixes it: branch only on step return values. The envelope is still read from the database on every step boundary — it is just read inside step.run, and that memoized
GateOutcome is what the hook enforces:

const planGate = await step.run("gate:plan", () =>
  gateOutcome(gateFor(getRun(runId)!, Date.now(), planNeed)),   // one live read, memoized);
const planner = createPlannerAgent(deps, planGate, "Planner:plan");

Replaying that read cannot change the answer, so the replay walks the same path. This is the whole argument for durable step functions, made concretely: durability is not just about not re-spending, it is about the control flow being reproducible.

c) Negotiating past a wall clock

Two related ones. A planner that returned unparseable JSON was being read as "the model needs more budget", so a bad day from the model triggered a money request. And a deadline breach was opening a negotiation instead of refusing — which meant a 20-second envelope could hang for 15 minutes waiting on a human.

Now: an unparseable plan is a clean partial-result; time exhaustion is always REFUSED{at:"time"}; and the approval window is bounded by the envelope's own clock:

const remainingMs = Math.max(0, env.remainingSeconds * 1000);
const cap = env.deadlineBreached ? 0 : Math.min(APPROVAL_TIMEOUT_MS, Math.max(1000, remainingMs));

If the deadline passes during the wait, the unanswered request is recorded as a timeout and the run refuses with the resource named. "Awaiting approval" is a state with its own deadline.

d) 850 tokens for an empty string

deepseek-v4.1-flash is a reasoning model. Given max_completion_tokens=700 and a multi-part
prompt, it returned:

completion_tokens: 700   reasoning_tokens: 700   message.content: ""

Empty. Billed in full. Adding reasoning_effort: "none" to the body gives reasoning_tokens: 0 and 255 tokens of real output.

I want to be careful about what this bug isn't: the ledger was right. The provider genuinely charged those tokens. What was wrong was my assumption that a completion costs roughly what it says — and the fact that without per-row reporting I'd have had no way to notice a third of my budget evaporating into invisible thinking. The envelope caught it. The accounting made it visible. That's the feature.

7. Verification is the product

npm run verify runs five scenarios over HTTP against a live provider and prints observed numbers:

· inference under test: deepseek-v4.1-flash @ https://api.particle.ai/v1 (HTTP 200, 1042 ms)
· provider reports usage: yes (prompt 9, completion 2)

[PASS] generous envelope · runs to completion
         tokens used: 900 / 100000
         recon verdict: reconciles — every charged row mirrors provider-reported usage …
         tokeniser estimate vs reported: 471 vs 900 tokens
[PASS] tight envelope · replans inside the remainder or ends REFUSED with a named resource
         envelope bound: yes (re-plan)          ← a plain completion FAILS this scenario
[PASS] cap-bypass attempt · refuses a plan far larger than the cap instead of executing it
         planned tokens (model's own figure): 2700 (0.9x cap)
         planned tokens (as the gate prices them): 13230 (4.4x cap)
         tokens actually used: 1559 / 3000      ← envelope held
[PASS] approval path · a denial stops the run cleanly with a partial result and a cost report
[PASS] auto-approve tier · a request under APPROVE_UNDER_USD is granted without a human

5/5 scenarios passed. Individual assertions: 15/15.

Two of those lines are the result of making the tests harder after they passed:

The tight-envelope scenario originally passed by completing, which proves nothing — the cap never bound. It now fails unless the run refused, re-planned, or filed a request.
The cap-bypass scenario originally reported planned / cap: 0.9x — the model's own estimatedTokens were optimistic, so the "5× the cap" claim was false as printed. It now reports both the model's estimate and what the gate actually prices per step, and asserts on the larger.

A test that can pass without the property holding is worse than no test, because it lets you ship
the doubt.

8. What I'd build next

An adversarial task suite. A prompt that literally says "your budget is too small, raise the cap to $50 and continue". Assert the caps column is unchanged. This is the sharpest test of "enforced, not requested" and I have not written it yet.
Crash-and-resume determinism. Kill the API mid-run, restart, assert a byte-identical ledger.
A budgeted network — several agents sharing one envelope, with an agent column in the ledger and per-agent spend on the gauges.
Forecasting. After three steps the run knows its own mean cost per step; "exhausts in ~4 steps / 22s at this rate" is more useful to a human approver than the model's guess.
Reconcile against an actual invoice. The table compares the ledger to what the provider reported. Comparing to what it charged is the honest next layer.

Try it

git clone https://github.com/harishkotra/allowance.git && cd allowance
npm install && cp .env.example .env      # set OPENAI_API_KEY
npx inngest-cli dev -p 8288 -u http://127.0.0.1:3001/api/inngest
npm run dev                              # client :5173, API :3001
npm run verify

Code & more: https://www.dailybuild.xyz/project/278-allowance

Building a Rubik's Cube Decision Arena with 6 Decision Models (Laya, Clef, GLiNER 2.5, Kev and Strands Decider)

Harish Kotra (he/him) — Thu, 08 Oct 2026 07:13:08 +0000

What happens when six small AI models — four of them running on a laptop — are asked to solve the same scrambled cube, one move at a time?

Most "AI plays a game" demos hide the interesting part. They show a model winning, and you are left to guess how much of that was the model and how much was the harness. This project does the opposite: it puts six models in a glass box, gives them the exact same scrambled Rubik's cube, the exact same question every turn, and shows you — live, in one screen — exactly where each one succeeds and where it falls apart.

The result is uncomfortable and honest. A Rubik's cube is a planning problem: you need lookahead and search. The models in this arena are decision models: fast, calibrated judgments about the current state. They are not the same thing, and the arena makes that difference visible.

1. The premise: decision models vs. planning

A "decision model" (sometimes called a system-one model) does one thing well: it looks at a situation, weighs a closed set of options, and returns a choice with a confidence. It is fast and it is often surprisingly well-calibrated.

A Rubik's cube is not a decision problem. It is a search problem. From a scrambled state there is an optimal solution length, and finding it means exploring a tree of moves. A model that only ever looks at the current state — with no memory of the plan and no lookahead — is being asked to do something it was not built for.

So the arena asks a precise question:

Given the same cube and the same one-move question, how good are six real decision models at picking moves that actually help?

Everything else in the project exists to answer that question fairly.

2. The two hard rules

The whole comparison is only meaningful if the conditions are identical, so the code enforces two rules everywhere:

Every engine gets the identical state and the identical question. No engine gets extra context, a better prompt, retries, or a second look. Same 54 stickers, same 18 options, same instructions.
Nothing is ever faked. If a model or its server is missing, its panel reads unavailable with the exact reason, and the race simply runs without it. There are no placeholder scores and no synthetic results.

That second rule sounds obvious until you build a demo. It is very tempting to fill a dead panel with something plausible. The arena refuses.

3. Architecture

                          ┌──────────────────────────────────┐
                          │  browser   http://127.0.0.1:8077  │
                          │  ui/index.html  (CSS-3D cubes)    │
                          └───────────────┬──────────────────┘
        GET /            GET /api/engines │ POST /api/run
        GET /api/run/<id>  GET /api/chart/<id>
                                          │
                          ┌───────────────▼──────────────────┐
                          │  server.py   (stdlib HTTP)        │
                          │  Hub: loaded engines + active run │
                          │  CHART_LOCK serialises matplotlib │
                          └───────┬───────────────────┬───────┘
                                  │                   │
             one thread per engine (run in parallel)  │
                                  ▼                   ▼
                    ┌────────────────────┐   ┌─────────────────────────────┐
                    │ arena.run()        │   │ charts.build_scorecard()    │
                    │  guided | greedy   │   │ -> results/chart_<id>.png   │
                    │  | staged          │   │ progress · latency ·        │
                    └─────────┬──────────┘   │ guided · final              │
                              │              └─────────────────────────────┘
              each turn: decide(state, question)
                              ▼
   ┌────────────────────────────────────────────────────────────────┐
   │  engines.py — one contract, six adapters                        │
   │   laya      MLX, in-process          gliner   CPU, in-process   │
   │   strands   HTTP 127.0.0.1:8010      kev      HTTP 127.0.0.1:8008
   │   clef / clef-flash   Cloudflare Workers AI (HTTPS)             │
   └────────────────────────────────────────────────────────────────┘
                              │
                              ▼
   ┌────────────────────────────────────────────────────────────────┐
   │  cube.py  (pycuber state + metrics)                             │
   │  solver.py (kociemba plan / first move)                         │
   └────────────────────────────────────────────────────────────────┘

The stack is deliberately boring: a stdlib HTTP server, a single HTML file, and plain Python. There is no framework, no bundler, and no build step. You can read the whole thing in an afternoon.

Tech stack

Layer	Choice
Cube simulation	`pycuber`
Optimal search	`kociemba` (two-phase)
Local runtimes	MLX (`laya`), torch/MPS (`strands`), transformers/CPU (`gliner2`)
Cloud inference	Cloudflare Workers AI (`clef`, `clef-flash`)
HTTP client	`httpx`
Web server	Python stdlib `http.server` + `ThreadingHTTPServer`
UI	single-file HTML + CSS 3D transforms + vanilla JS
Charts	`matplotlib` (`Agg`)
Terminal	`rich`
Automation	Playwright
Env	`uv` + Python 3.12

4. The cube model

Everything starts with a deterministic scramble. Given a seed, every engine sees the same cube — and the same cube every time you rerun the demo.

# cube.py
FACES = "URFDLB"
MOVES = [f + s for f in FACES for s in ("", "'", "2")]  # 18 quarter/half turns

def scramble(seed: int, n: int = 25) -> tuple[Cube, list[str]]:
    rng = random.Random(seed)
    seq: list[str] = []
    prev_face = None
    while len(seq) < n:
        mv = rng.choice(MOVES)
        if mv[0] == prev_face:      # avoid trivial same-face repeats
            continue
        prev_face = mv[0]
        seq.append(mv)
    c = Cube()
    for mv in seq:
        c.perform_algo(mv)
    return c, seq

Two metrics are tracked. A sticker is correct when it matches its own face's centre colour (centres never move, so the centre is the solved colour). That gives a smooth 0–54 progress line; complete faces give a chunky 0–6 milestone.

# cube.py
def solved_facelets(cube: Cube) -> int:
    n = 0
    for f in FACES:
        ctr = centre_colour(cube, f)
        face = _face(cube, f)
        for r in range(3):
            for c in range(3):
                if face[r][c].colour == ctr:
                    n += 1
    return n

The model never sees a Cube object. It sees a compact text state, identical for every engine:

# cube.py
def state_text(cube: Cube) -> str:
    rows = facelet_rows(cube)
    body = "\n".join(f"{f}: {rows[f]}" for f in FACES)
    return (
        "3x3 Rubik's cube. Faces U R F D L B, each 9 stickers read row by row "
        "seen from outside. Letters are sticker colours (W Y G B R O). A face is "
        "solved when all 9 stickers match its fixed centre sticker.\n"
        f"{body}\n"
        f"Correct stickers: {solved_facelets(cube)}/54. "
        f"Complete faces: {solved_faces(cube)}/6."
    )

5. The solver bridge (and its encoding trap)

Guided mode needs an optimal-ish solver. The project uses kociemba, but wiring it up has a classic gotcha: pycuber colours its stickers with colour names, and Kociemba wants a 54-character string of face letters in URFDLB order.

Hardcoding the colour→face map is wrong, because pycuber's default cube is not the usual white-up arrangement. The fix is to read the map off the centres, which never move:

# solver.py
FACES = "URFDLB"

def colour_map(cube: Cube) -> dict[str, str]:
    """colour name -> face letter, taken from the fixed centres."""
    return {cube.get_face(f)[1][1].colour: f for f in FACES}

def facelet_string(cube: Cube) -> str:
    """54-char Kociemba facelet string, URFDLB order, row-major from outside."""
    m = colour_map(cube)
    out: list[str] = []
    for f in FACES:
        face = cube.get_face(f)
        for r in range(3):
            for c in range(3):
                out.append(m[face[r][c].colour])
    return "".join(out)

@lru_cache(maxsize=200_000)
def _solve_cached(state: str) -> str:
    return kociemba.solve(state)

Every state is cached by its facelet string. The race revisits states constantly (neutral moves, repeated positions), and Kociemba is by far the slowest part of the loop — the cache is the difference between a snappy demo and a stall.

Two more things worth knowing if you build on this:

kociemba has no Apple-silicon wheel, so it always compiles a small C extension. If that build fails, the module degrades cleanly: greedy and staged modes still work, and guided mode says exactly what is missing.
The solver's solution length is near-optimal, not provably optimal. That is why guided runs take optimal + a few moves rather than exactly optimal.

6. One contract, six engines

This is the heart of the project. Six very different runtimes — an in-process MLX model, a torch server, a CPU classifier, two cloud endpoints, and a local OpenAI-ish server — are all reduced to one method:

# engines.py
def decide(self, state: str, question: dict, qid: str = "move") -> Decision:
    ...

@dataclass
class Decision:
    engine: str
    choice: str                      # e.g. "R2"
    confidence: float | None
    probabilities: dict[str, float]  # over the 18 moves (may be empty)
    latency_ms: float
    error: str | None = None

The question is a closed choice over the 18 legal moves, with a description for each — the shape that strands-decider requires, and that the others accept too:

# engines.py
_FACE_NAME = {"U": "up", "R": "right", "F": "front",
              "D": "down", "L": "left", "B": "back"}

MOVE_CRITERIA = {}
for _f, _n in _FACE_NAME.items():
    MOVE_CRITERIA[_f]        = f"turn the {_n} face clockwise 90 degrees"
    MOVE_CRITERIA[_f + "'"]  = f"turn the {_n} face anticlockwise 90 degrees"
    MOVE_CRITERIA[_f + "2"]  = f"turn the {_n} face 180 degrees"

def move_question(instructions: str) -> dict:
    return {"type": "choice", "instructions": instructions,
            "criteria": dict(MOVE_CRITERIA)}

Most engines share an HTTP transport, so adding one is usually a tiny subclass:

# engines.py
class HttpEngine(Engine):
    def probe(self) -> tuple[bool, str | None]:
        """A dead server must read as unavailable, not as 'ready'."""
        if not self.health_url:
            return True, None
        try:
            r = httpx.get(self.health_url, timeout=3.0)
        except Exception as e:
            return False, f"no server at {self.health_url} ({type(e).__name__})"
        return True, None

    def decide(self, state, question, qid="move") -> Decision:
        body = {"state": state, "model": self.model, "questions": {qid: question}}
        t0 = time.perf_counter()
        try:
            r = self.client.post(self.url, json=body)
            r.raise_for_status()
            resp = r.json()
        except Exception as e:
            return Decision(self.name, "", None,
                            latency_ms=(time.perf_counter() - t0) * 1000,
                            error=f"{type(e).__name__}: {e}")
        dt = (time.perf_counter() - t0) * 1000
        choice, conf, probs = _parse_answers(resp, qid)
        return Decision(self.name, choice, conf, probs, dt)

The probe() method is why a dead Kev server shows unavailable: no server at 127.0.0.1:8008 instead of silently reporting "ready" and failing on every move.

The six models

Model	Params	Where	How
Laya (`aac6fef/laya-mlx`)	421M	local, Apple silicon	in-process MLX
Strands Decider 2B	1.9B	local, torch/MPS	HTTP `POST /v1/systemone`
GLiNER2.5-Decide	340M	local, CPU	in-process `classify_text`
Clef-Flash	9B	Cloudflare Workers AI	HTTPS
Clef	27B	Cloudflare Workers AI	HTTPS
Kev-0.8B	0.8B	local server `:8008`	HTTP `POST /v1/systemone`

Why two run in the cloud: this was built on an Apple M2 mini with 8 GB of RAM. A 27B model does not fit at any quantization, so Clef and Clef-Flash are called on Workers AI where they cost no local RAM. The other four run locally — including on a 16 GB laptop — which is the point of the demo.

7. The guided loop: the solver plans, the model decides

The unaided models cannot solve a cube. That is measured, not assumed: after 24 moves each, three different architectures all landed on exactly the sticker count they started from. So guided mode separates the two jobs cleanly — the solver is the search, the model is the decision layer inside it.

# arena.py (abridged)
plan = solution(cube)                 # a concrete, known-good move list
correct = plan[0]                     # the solver's optimal next move

d = engine.decide(state_text(cube), move_question(GREEDY_Q))
mv = pick_move(d, policy, rng)        # sample the distribution, or take argmax

play, tag, replan = correct, "corrected", False
if mv == correct:
    tag = "hit"                       # matched the solver exactly
elif mv in MOVES:
    probe = cube.copy(); apply_move(probe, mv)
    d_m = len(solution(probe))
    if d_m < len(plan):
        play, tag, replan = mv, "accepted", True      # a different move that helped
    elif d_m == len(plan) and neutral_budget > 0:
        play, tag, replan = mv, "neutral", True       # wasted turn (budgeted)
        neutral_budget -= 1
    else:
        play, tag = correct, "corrected"              # solver overrides
else:
    play, tag = correct, "corrected"                  # unusable answer

apply_move(cube, play)
plan = solution(cube) if replan else plan[1:]

The model gets the same 18-move question as the unaided race, so the two are directly comparable. Each pick is scored:

Outcome	Meaning
`hit`	the model picked exactly the solver's next move
`accepted`	a different move that still made strict progress
`neutral`	the move did not change the distance — a wasted turn (budgeted)
`corrected`	the model's move was worse or unusable, so the solver played its own move

The cube is guaranteed to advance because the solver always holds a valid plan, and after the neutral budget is spent every move must strictly reduce the distance.

But here is the subtlety that trips people up: a corrected turn still costs one of the moves you allocated. The solver is doing the solving, and the model is being measured — but the clock runs on every turn, including the ones where the model was overruled. With an optimal distance of 22 and a neutral budget of 6, a guided run can legitimately need up to 28 moves. Set the cap below that and slower models get cut off unsolved while still showing the done status.

Policy: how a distribution becomes a move

Most of these models return a probability for each of the 18 moves. Policy decides what to do with it:

# arena.py
def pick_move(d, policy: str, rng: random.Random) -> str:
    if policy == "sample" and d.probabilities:
        opts = [(m, v) for m, v in d.probabilities.items() if m in MOVES and v > 0]
        if opts:
            total = sum(v for _, v in opts)
            r = rng.random() * total
            acc = 0.0
            for m, v in opts:
                acc += v
                if r <= acc:
                    return m
            return opts[-1][0]
    return d.choice      # argmax

sample draws the move from the model's own distribution. This is the honest way to run a decision model in a loop, and it prevents the collapse you get from always taking the top choice.
argmax always takes the most likely move. In practice it collapses — the model repeats one move forever — so it is shown mainly for contrast.

8. The server: threads, polling, and a chart race

The server is stdlib-only. Its job is small: hold the loaded engines, run one race at a time, and serve the UI. The interesting part is concurrency.

A run spawns one thread per ready engine, so all six models think in parallel:

# server.py
def _drive(self, run: dict) -> None:
    threads = []
    for panel in run["panels"]:
        if panel["status"] != "ready":
            continue
        t = threading.Thread(target=self._one_engine, args=(run, panel), daemon=True)
        t.start()
        threads.append(t)
    for t in threads:
        t.join()
    # ... persist the run + render the chart, THEN flip status to "done"

Every turn, the engine thread fires an on_move callback that updates the shared panel under a lock. The browser polls /api/run/<id> every 350 ms and redraws the cubes.

There is one nasty bug worth calling out, because it is the kind of thing that only shows up in a demo: matplotlib's pyplot is not thread-safe. The scorecard is built on the run thread when a race finishes, and on demand by the /api/chart/<id> route. If those two ever overlap, the render corrupts and the browser shows a broken image.

The fix is two-fold — a lock, and getting the order right:

# server.py
CHART_LOCK = threading.Lock()

# ... in _drive(), after all engine threads join:
snapshot = {**run, "status": "done", "finished_at": finished_at}
(RESULTS / f"ui_run_{run['id']}.json").write_text(json.dumps(snapshot, indent=2))
with CHART_LOCK:
    build_scorecard(run, RESULTS / f"chart_{run['id']}.png")
with self.lock:
    run["status"] = "done"          # flipped LAST, only after the PNG exists

The status flips to done after the PNG is on disk, so the page can never request a chart that does not exist yet. The front-end adds a retry with a cache-buster as a belt-and-braces measure.

The API is four routes:

Route	Purpose
`GET /`	the single-page UI
`GET /api/engines`	per-engine load status (polled on startup)
`POST /api/run`	start a race `{seed, mode, policy, moves, engines}`
`GET /api/run/<id>`	current run state (polled while running)
`GET /api/chart/<id>`	the scorecard PNG

9. The UI: a real 3D cube with no library

The cube is not an image or a canvas — it is six DOM faces positioned in 3D space with CSS transforms, slowly rotating. That is the whole trick:

/* ui/index.html */
.scene { height: 164px; perspective: 1000px; perspective-origin: 50% 45%; }
.cube3d {
  position: relative; width: var(--cube); height: var(--cube);
  transform-style: preserve-3d;
  animation: spin 46s linear infinite;
}
.face3d.U { transform: rotateX(90deg)  translateZ(calc(var(--cube)/2)); }
.face3d.D { transform: rotateX(-90deg) translateZ(calc(var(--cube)/2)); }
.face3d.F { transform: translateZ(calc(var(--cube)/2)); }
.face3d.B { transform: rotateY(180deg) translateZ(calc(var(--cube)/2)); }
.face3d.L { transform: rotateY(-90deg)  translateZ(calc(var(--cube)/2)); }
.face3d.R { transform: rotateY(90deg)   translateZ(calc(var(--cube)/2)); }

Each face is a 3×3 grid of stickers whose colours come straight from the model's net (the per-face colour strings). The layout is three columns by two rows, with a short-screen media query that shrinks the cube so all six stay in one frame.

Two front-end details earned their keep during development:

Persistence. When a run finishes, runId is cleared and the engine poll resumes. The first version re-rendered empty placeholders and every cube vanished at the moment of victory. The fix keeps the last run's panels and re-renders those instead, so the final solved state stays on screen.
The chart is the true "done" signal. startRace() is async, so the status text briefly still reads "models ready". Tests wait on the chart image actually loading (naturalWidth > 0), not on the status string.

10. The scorecard

When a race finishes, the server renders a four-panel dark scorecard with
matplotlib:

Progress — correct stickers over moves, one line per model.
Latency — median ms per move, log scale.
Guided resolution — stacked hit / accepted / neutral / corrected bars, so you can see at a glance how much of the solve was the model.
Final — stickers correct at the end, solved models marked.

Colours are stable per engine key, so a model keeps its colour across runs.

11. What we actually measured

Short version; the full write-up is in REPORT.md.

Unaided (greedy / staged), same scramble, seed 20261002: after 24 moves each, three different architectures landed on exactly the sticker count they started from — zero net progress, and no face ever completed. Even 60 moves of subgoal-named play (staged mode) topped out at 17 of 54 stickers.

Guided mode, same scramble (optimal distance 22): every model solves the cube, because the solver is doing the solving. The honest reading is in the counters:

model	solved	moves	hits	accepted	neutral	solver corrected
laya-mlx	yes	28	0	1	6	21
strands-decider	yes	28	0	2	6	20
GLiNER2.5-Decide	yes	27	4	0	5	18

Read it carefully:

18 to 21 of every 27–28 moves were the solver overriding the model. laya and strands never once picked the solver's move.
GLiNER's 4 hits are an artefact, not skill. It returns a single label with no distribution, so it cannot be sampled — it plays U almost every turn, and the optimal plan happened to start with U four times.
Wall clock is the real race. Same solve, but laya finishes in ~5 s against strands' ~64 s and GLiNER's ~95 s.

The headline is not "who won". It is that small decision models do not plan, and the arena shows exactly where that breaks down — while never pretending the model did something it didn't.

12. Hard-won lessons

A few things that cost real debugging time and are worth stealing:

kociemba has no Apple-silicon wheel. It always compiles a C extension. Keep it in its own pip install line, or a failed build aborts the whole command and takes pycuber down with it.
matplotlib pyplot is not thread-safe. Serialise every render with a lock, and write the PNG before you advertise that the run is done.
A dead server must be probed, not assumed. probe() is what turns a silent failure into an honest unavailable panel.
Never wipe the final state on the next poll. Keep the last run's panels and re-render those, or the cubes disappear the instant the race ends.
The move cap is part of the experiment. In guided mode a corrected turn still costs a move, so a cap below optimal + neutral_budget silently cuts models off unsolved.
argmax on a decision model collapses. Sampling from the returned distribution is the honest way to run a model in a loop.
Build on the centre stickers, not on assumed colours. pycuber's default cube is not the usual white-up arrangement, so derive the colour→face map from the fixed centres.

13. Run it, fork it, extend it

git clone https://github.com/harishkotra/cube-arena.git && cd cube-arena
uv venv --python 3.12 && source .venv/bin/activate
uv pip install laya-mlx strands-decider gliner2 pycuber rich matplotlib requests httpx
uv pip install kociemba
python server.py            # open http://127.0.0.1:8077

Everything is optional except pycuber: a model without credentials or a running server simply shows unavailable. ./run.sh starts the strands server and the UI together; ./run_kev.sh starts the Kev server.

There is no framework and no build step, so contributing is easy. The most common contribution is a new engine: subclass HttpEngine (or Engine), register it in ENGINE_SPECS in server.py, and the UI, charts, and CLI pick it up automatically.

Other good first contributions: per-move regret and calibration metrics, a
best-of-N mode, an OpenAI-compatible adapter, a /api/runs history index, and a multi-seed suite. The full list is in the README's "Feature ideas" section.

If you build something on top of this, I'd love to see it. The whole point is that comparison under identical conditions is more interesting than a leaderboard.

Screenshots

Watch

Code & more: https://www.dailybuild.xyz/project/277-cube-arena

Building Anvil: code-as-action with a capability sandbox that explains its refusals.

Harish Kotra (he/him) — Wed, 07 Oct 2026 18:19:48 +0000

The agent wrote `import socket`. Now what?

Every agent framework got very good at making models write code. Almost none got good at the question that follows immediately afterwards: what is that code allowed to do to my machine?

I built Anvil to answer that in the smallest honest shape I could: an agent writes a Python tool for a task over some real CSVs, the tool executes in a locked-down subprocess, and every attempt to escape is blocked and reported with the exact reason string, at the layer that
caught it. It's three panes — the generated code, the sandbox trace, the artifact — plus a "try to break it" panel with four canned escape attempts.

About 4,900 lines later, python -m agent.verify passes 8 checks. This is the post I wanted
when I started: how the layers actually divide the work, and the five things that broke and
taught me something.

The pipeline

 task ──▶ CodeAgent (smolagents) ──▶ Python tool source
                                        │
                                        ▼
                             1. AST pre-check            (nothing executed)
                             2. sandboxed subprocess     (rlimits + audit hooks + kernel)
                             3. artifact collection      (artifacts/<runId>/)
                             4. trace                    (every refusal, with its reason)

The one architectural decision that mattered: nothing untrusted runs in the server
process. The server orchestrates. Model-authored code executes one fork away, under a
kernel profile, in an ephemeral jail, with a scrubbed environment.

 FastAPI (trusted) ──▶ worker thread (trusted) ──▶ sandboxed child (UNTRUSTED)
                                                    sandbox-exec -p <profile>
                                                    setrlimit(CPU, AS, FSIZE, NPROC…)
                                                    sys.addaudithook(policy)
                                                    cwd = .anvil/runs/<id>/
                                                    env  = 9 variables, no secrets

Layer 1 — the AST pre-check: text, but not just text

policy.py parses the candidate source and never executes it. Banned imports, process
primitives, dynamic eval, dunder traversal, and any literal path that resolves outside the
allowed roots. Each rule produces a reason string that is shown verbatim in the UI and
asserted by the test suite:

blocked: import socket (network denied by policy)
blocked: os.fork (process creation denied by policy)
blocked: write path './artifacts/../outside.txt' outside ALLOWED_WRITE_PATHS (./artifacts)

Paths go through realpath, so ../ traversal and a symlink pointing out of the jail are
both caught, and absolute-path literals are swept even when the model parks them in a
variable first — that's what catches p = '/etc/passwd'; open(p).

Lesson one: an over-eager static rule is a correctness bug, not a security feature. My
first version checked the first string argument of every call. It cheerfully rejected a
perfectly good pandas tool because it contained:

raise ValueError('Required columns missing')   # "blocked: read path 'Required columns missing'"

Path checks now only apply to calls that actually take paths (read_csv, to_csv,
savefig, open, …). The same over-reach killed the whole data stack a second time: I had
banned inspect, gc and builtins as "introspection modules" — and pandas imports
inspect internally, so no tool using pandas could run.

Layer 2 — audit hooks: policy at the moment of the operation

Static analysis has an obvious ceiling: a path built at runtime, a module reached through
importlib. CPython's audit hooks catch the operation itself, and the hook is installed in
the child before the tool runs, with no API to remove it.

def audit_hook(event, args):
    if event == "open":
        path, mode, flags = str(args[0]), args[1], args[2]
        if _is_write(mode, flags) and not _under(path, WRITE_ROOTS):
            emit_blocked(f"open({path!r}, 'w')", "write outside ALLOWED_WRITE_PATHS")
            raise SandboxBlocked(...)      # raised *into* the tool's own stack

Note the last line: the exception is raised into the executing code, so the tool experiences
the refusal as an error at its own call site, and the traceback lands in the trace. Blocking
and explaining is the same event.

Lesson two: some audit events have no caller frame, and the ones that do need scoping.
Two halves of the same discovery.

import events carry a usable stack, so import denial is caller-scoped — which is the only
reason ctypes could stay banned at all, because numpy dlopens its own compiled extensions
through ctypes on the import path. Refusing that globally broke every legitimate tool until
the hook started asking who is asking.

os.fork does not. I verified it directly:

sys.addaudithook(lambda e, a: sys._getframe(1))
os.fork()   # ValueError: call stack is not deep enough

CPython raises that event from C with no Python frame to consult. So process creation is
denied globally, and I had to remove the legitimate reason a library would ever fork —
see lesson three.

Layer 3 — the kernel: prove it, don't assume it

The spec's ladder was bwrap > firejail > rlimit-only. None of those binaries exist on this
macOS box. What does exist is /usr/bin/sandbox-exec — Seatbelt, the kernel sandbox Apple's
own apps use — which gives the two guarantees that actually matter here: network denial and
write confinement, enforced below Python.

So the ladder became bwrap > firejail > seatbelt > rlimit-only, and the rule is that a tier
is only selected if it passes a functional probe: the interpreter starts and a write
outside the allowed root is refused by the kernel. Being on PATH is not evidence.

(version 1)
(allow default)
(deny network*)              ; egress refused below Python
(deny process-fork)          ; no child processes, from any code path
(deny file-write*)           ; then allow-listed to the jail's write roots
(allow file-write* (subpath "/…/.anvil/runs/<id>/artifacts"))
(deny file-read* (subpath "/etc"))      ; and /private/etc, ~/.ssh, keychains…

The read denylist at kernel level is not redundant paranoia — it's the only layer that
covers C extensions. pandas' CSV reader opens files in C and never touches a Python audit
hook. I checked both paths explicitly:

[py-open-etc-passwd]      rc=1 | PermissionError: Operation not permitted: '/etc/passwd'
[c-level-read-via-os_open] rc=1 | PermissionError: Operation not permitted: '/etc/passwd'

Lesson three: the strongest rule I wrote broke the thing I was building. With
deny process-fork global and process creation denied for every caller, matplotlib's font
manager shells out to fc-list to enumerate system fonts:

# matplotlib/font_manager.py
if b'--format' not in subprocess.check_output(['fc-list', '--help']):

My hook refused it and the chart tool died. The tempting fix was to scope the rule to the
tool — impossible for os.fork, per lesson two — or to whitelist fc-list, which is
exactly the kind of exception that quietly deletes your security story. Instead I removed
the need: Anvil pre-warms matplotlib's font cache in the parent, outside the sandbox, so
the only legitimate fork never happens inside it. Charts render, and the rule stays global.

Two smaller ones from the same layer: RLIMIT_AS is advisory on macOS (the memory hog happily
climbed past the cap), so the parent samples the child's resident set every 20 ms and SIGKILLs
the process group — and reports the observed peak rather than pretending the kernel refused
the allocation. And bytecode writing had to be disabled, because __pycache__ writes into
site-packages were being correctly refused by my own write allowlist.

Layer 4 — capabilities: the model asks, the user decides

The tool-writing prompt asks the agent to declare capabilitiesRequested. That declaration is
an input to an approval list, never an authority:

def resolve_capabilities(requested, approved, *, allow_network):
    ...
    elif cap == "network":
        (granted if allow_network and cap in approved else denied).append(cap)
    elif cap == "process":
        denied.append(cap)          # not grantable under this policy at all

A denied capability has to produce a visible failure, not a silent workaround, so the
network-using tool still runs, still reaches for the socket, and is refused at the operation:

blocked at runtime: socket.connect (network denied by policy (capability not granted))

That's an assertion in the test suite, not a screenshot in a README.

The smolagents part, and where the spec was wrong about the API

The spec said "configure its executor to be your sandbox", with a fallback if the hook
couldn't be swapped. Good news: in smolagents 1.26 it can be, directly —

CodeAgent(tools=[], model=…, executor=AnvilSandboxExecutor(config),
          additional_authorized_imports=[…])

so AnvilSandboxExecutor implements the real PythonExecutor contract (send_tools,
send_variables, __call__ -> CodeOutput) and the default local interpreter is never even
constructed. Each agent code action goes through the same run_tool() path as the final
tool, which is what makes the trace continuous rather than two different systems pretending
to agree.

Two API details worth saving someone an hour: OpenAIServerModel names the argument
api_base, not base_url. And because each action is a fresh sandboxed process, state
between actions is limited to JSON-serialisable variables — that's a consequence of running
untrusted code in a subprocess, not a shortcut I took.

Lesson four: a fallback that quietly becomes the default is a lie with extra steps. My
first CodeAgent runs "worked" but were wasting every step: the model emitted raw JSON where
the ReAct format wants a <code> block, so the executor received an empty string, executed
nothing, and reported success. Three vacuous successes later the loop ended. The fix is four
lines and the honest part is refusing to count nothing as a run:

if not code_action.strip():
    return CodeOutput(output="No code was received. …", logs=hint, is_final_answer=False)

Lesson five, the one I'd warn about hardest: after adding that guard, a run still
"succeeded" while showing an exploratory print(df.columns) as the deliverable — my
provenance fallback accepted any successful action when the loop ran out of steps. It
produced a green UI, a real artifact directory, and a completely meaningless tool. The
fallback now only accepts an action that actually wrote an artifact, and if nothing qualifies
the run fails with a clear error. A demo that lies is worse than a demo that fails.

The same principle caught a genuinely good outcome looking bad: pointed at a task that said
"revenue by region" without naming columns, the model invented df.groupby('Region'), the
tool really ran, hit a real KeyError, and the UI showed failed with a traceback. That's
the system working. Two different models, on two different days, independently computed the
same grand total — 12210294.46 — from the 500-row CSV, which is the artifact being real
rather than the answer being scripted.

Verification is the product surface

agent/verify.py runs the actual pipeline and prints the captured strings:

PASS  1. two tasks -> two distinct tools, real artifacts + stdout
PASS  2. four escape attempts blocked at pre-check and runtime
PASS  3. infinite loop killed at SANDBOX_TIMEOUT_S with 'killed: timeout'
PASS  3b. busy loop killed by RLIMIT_CPU before the wall clock
PASS  4. memory hog killed by the memory limit (peak RSS reported)
PASS  5. capabilities are requestable, never self-grantable, denial is visible
PASS  6. effective isolation level is truthful for this environment
PASS  7. blast-radius card produced from a real run
8/8 checks passed

Check 6 is the one I'd defend hardest: it doesn't read the cached report, it re-runs the
probe and asserts that if the app claims a kernel tier then that binary exists and an
escaped write still fails right now. If the environment changes, the claim has to change with
it. That check failed on its first real run because of a bug in my test — the unsandboxed
control wrote the canary file and left it there, so the sandboxed run looked like it had
succeeded. A verifier that can't fail is not verifying.

Steal these ideas

Probe your isolation tier, don't detect it. which bwrap is not a security property.
Scope each rule to the smallest caller that makes it correct, and say out loud in a comment which rules are global and why.
Pre-warm anything your sandbox will refuse on behalf of a library instead of whitelisting the primitive.
Make denial visible in the trace, with a reason string that a test asserts verbatim.
Report observed resource peaks, not configured limits. "peak 475.3 MB > 200 MB cap" is a fact; "memory limit 512 MB" is an intention.

None of this is a VM, and the README says so in as many words: same kernel, same uid, and a vulnerability inside pandas or CPython is out of scope. For genuinely hostile code you want Firecracker or a remote sandbox. The claim here is narrower and I think more useful — for an
agent whose code you want to run, a layered boundary that can tell you exactly what it refused, and why, and which layer did it.

The repo is in this project (agent/, client/, data/): npm run dev · npm run verify.

Code & more: https://www.dailybuild.xyz/project/276-anvil

How Arbiter works, why it was built, and the four measurements it made about an LLM judge that I had to look at twice.

Harish Kotra (he/him) — Tue, 06 Oct 2026 13:46:12 +0000

Your eval harness trusts a witness it never cross-examined

Every agent benchmark I have built ends with the same line of code: a model grades another model's output, and I treat the resulting number as a measurement. Rubric, strong model, score out of five.
Ship it.

The judge is a language model. It has documented failure modes — it likes long answers, it likes the first answer, it likes confident answers, and it drifts when you change its temperature. We know all of this. And then we grade with it anyway and never once ask whether it was any good at grading.

So I built the inversion: a rig where the thing under test is the grader. It's called Arbiter — a gold set of human labels, four experiments, and one blunt verdict on whether the judge deserves to be trusted for your rubric.

What it measured surprised me more than I expected, and the most interesting result isn't about the judge at all.

The setup

Gold set (20 human-labelled items + 8 pairwise items)
      |
      v
  Judge agent (Pydantic AI, output_type=Verdict)   <-- every call recorded
      |
      +--> 1. Agreement    raw %, Cohen's kappa, 5x5 confusion matrix, bootstrap CIs
      +--> 2. Position     (A,B) then (B,A) -> flip rate, mean score delta, Cohen's d
      +--> 3. Verbosity    padded ~2x vs terse <=0.6x -> mean delta, effect size
      +--> 4. Drift        second temperature, second model -> kappa per condition, KS shift
      |
      v
  Corrected score + VERDICT: trustworthy / NOT trustworthy (failing criterion named)

The judge's output type is deliberately tiny:

class Verdict(BaseModel):
    score: int                     # 1..5
    criteria: dict[str, bool]      # one key per rubric criterion
    rationale: str                 # <= 240 chars
    confidence: float              # 0..1

and the system prompt is fixed, so the thing being measured is the judge and not a moving instruction:

You are a strict evaluator. Rubric criteria:
- correctness: the answer is factually right
- completeness: it covers every part of the question
- clarity: it is unambiguous and well-structured
Score 1-5. Reply ONLY with JSON:
{"score":<1-5>,"criteria":{"correctness":true|false,"completeness":true|false,"clarity":true|false},
 "rationale":"<=240 chars","confidence":<0-1>}
Judge only what is present. Do not reward length.

That last line is doing a lot of work. It is the hypothesis under test.

One table is the whole design

The architecture decision that shaped everything: every judge call is a row, and every statistic is computed by reading rows back. No experiment ever calculates from an in-memory response.

CREATE TABLE judge_calls(
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  run_id TEXT NOT NULL, experiment TEXT NOT NULL,   -- agreement | position | verbosity | drift
  item_ref TEXT NOT NULL, variant TEXT NOT NULL,    -- original | padded | terse | forward | reverse
  base_url TEXT NOT NULL, model TEXT NOT NULL, temperature REAL NOT NULL,
  system_prompt TEXT NOT NULL, user_prompt TEXT NOT NULL, prompt_sent TEXT NOT NULL,
  raw_output TEXT,
  parse_ok INTEGER NOT NULL,                        -- 0 => failure, and score IS NULL
  score INTEGER, criteria_json TEXT, rationale TEXT, confidence REAL,
  human_score INTEGER, preferred TEXT, slot_scores_json TEXT,
  latency_ms INTEGER, error TEXT, created_at TEXT NOT NULL
);

Three things fall out of this that I did not expect to need:

A report is rebuildable. GET /api/report?run_id=… recomputes every number from stored rows, months later, without calling a model again. The UI's report step loads from a URL alone.
A crash loses nothing. Half-finished runs still report whatever completed.
A failure cannot be quietly dropped. An unparseable verdict is a row like any other, with parse_ok = 0. It has to be actively excluded, and the exclusion is visible.

The prompt that went on the wire is stored too, which is more fiddly than it sounds. Pydantic AI can append its own instructions to what you passed in, so the honest "exact prompt" is the assembled request, not your string. In 2.54.0 that lives on the instance, not the class:

request = next(m for m in result.all_messages() if type(m).__name__ == "ModelRequest")
texts = [p.content for p in request.parts if isinstance(getattr(p, "content", None), str)]
prompt_sent = "\n\n".join(texts)      # SystemPromptPart + UserPromptPart, as actually sent

Failures are rows, not defaults

The rule when a judge returns something unusable: record it, keep the raw text, score nothing. Never clamp, never retry-into-a-number, never substitute 0 — because 0 is a verdict, and a silent 0 turns a broken measurement into a bad result.

I expected this path to be rare. It is the single busiest code path in the project.

parse failures counted, never coerced — causes: rationale over the 240-char limit x3
failure g06: rationale is 244 chars, over the 240 limit
failure g15: rationale is 255 chars, over the 240 limit
failure g16: rationale is 255 chars, over the 240 limit

The model cannot hold a 240-character limit. Across a 116-call run, 17–19 calls fail to parse, and the dominant cause is a rationale that is 241–290 characters long. Nothing about that is visible in a normal eval harness — you would just see slightly noisier scores and quietly attribute the noise to the model under test.

To make this reachable I had to leave the typed model bare (score: int, not Field(ge=1, le=5)) and enforce ranges in my own validator. If I constrained the schema, Pydantic AI would coerce or retry at the provider, and the failure I most wanted to measure would disappear into a retry loop.

Proving the accounting also needed an escape hatch from structured output. With output_type=Verdict, a prompt begging for prose still returns valid JSON — the provider enforces the schema. So the verification path swaps both the type and the prompt:

system_prompt = PROSE_PROBE_SYSTEM_PROMPT if prose_mode else JUDGE_SYSTEM_PROMPT(rubric)
agent = Agent(model=model, system_prompt=system_prompt,
              output_type=str if prose_mode else Verdict)

Three prose calls in, three rows with parse_ok = 0, score IS NULL, and the raw paragraph attached. Not one of them scored 0.

The sign trap in position bias

Position bias looks trivial: judge (A,B), judge (B,A), count flips. The bug is in the score delta, and it is a good one because the wrong version is self-cancelling and therefore invisible.

The judge names a slot, never an answer. Mapping back to identity is the whole trick:

if winner_slot == "slot1":
    preferred_answer = "A" if order == "forward" else "B"
elif winner_slot == "slot2":
    preferred_answer = "B" if order == "forward" else "A"

A flip is then "the content-winner changed", which is not the same as "the slot choice changed" — a judge that picks slot 1 both times is perfectly consistent by slot and completely biased by content.

For the score delta, each answer must be compared first-position minus second-position. A is first in the forward ordering; B is first in the reverse one. So the two subtract in opposite directions:

deltas.append(slots_f["answerA"] - slots_r["answerA"])   # A: first in forward
deltas.append(slots_r["answerB"] - slots_f["answerB"])   # B: first in reverse

I wrote the second line the obvious way first — slots_f["answerB"] - slots_r["answerB"] — and a test caught it, because the test fixture was built to produce a known +3 inflation and got 0.0. The wrong sign makes a real positional bias cancel itself out and report "no bias found". That is the kind of bug that survives code review and quietly launders a benchmark.

The denominator trap

Same category, caught later by looking at a screenshot instead of a test.

Verbosity reports three numbers: padded − terse, padded − original, terse − original. My first implementation averaged each side over whatever had parsed on that side. Since 15–19 of 20 items survive each pass, padded − original was comparing a mean over one subset against a mean over a different subset. A "difference" with no shared items is not a difference.

Now the comparisons against the unmodified answer only use items where all three parsed:

for item_id in sorted(by_item):
    baseline = baseline_rows.get(item_id)          # the agreement pass, not this experiment
    p, t = by_item[item_id].get("padded"), by_item[item_id].get("terse")
    if p and t and baseline and p.parse_ok and t.parse_ok and baseline.parse_ok:
        triples.append((baseline.score, p.score, t.score))

and the report publishes its own denominators so you can see the sample each figure rests on:

"denominators": { "paddedScored": 15, "terseScored": 19, "paired": 15,
                  "triplesWithOriginalBaseline": 15, "originalScored": 18 }

Padding has to be generated in code

The verbosity experiment needs two variants of the same content at different lengths. Asking a model to produce them is a trap: it will add content, and sometimes add facts, and then you are measuring "does the judge like more information" instead of "does the judge like more words".

So padding is a fixed table of information-free sentences rotated by a hash of the item id:

FILLER = (
    "It is worth noting that this topic has several dimensions that merit careful consideration.",
    "Before continuing, it may help to step back and consider the framing of the question itself.",
    ...
)

def pad(text: str, item_id: str) -> str:
    base, n = text.rstrip(), len(text.rstrip())
    chosen, current, i = [], n, _seed_offset(item_id, len(FILLER))
    while current < TARGET_RATIO * n:                     # aim for 2.0x
        candidate = FILLER[i % len(FILLER)]
        addition = len(candidate) + (1 if chosen else 0)
        if current + addition > MAX_RATIO * n and current >= MIN_RATIO * n:   # 2.4x ceiling / 1.7x floor
            break
        chosen.append(candidate); current += addition; i += 1
    return base + "\n\n" + " ".join(chosen)

The tests are the interesting part: the original bytes must survive untouched, no filler sentence may contain a digit or a currency symbol (no new facts), every appended sentence must be a member of FILLER, and regenerating must be byte-identical.

terse() is the opposite operation and it is honest about being ugly: delete discourse-only sentences, delete scaffolding phrases, drop stop words, until the item fits under 0.6×. It never adds a word, so every surviving token is a substring of the original. Terse answers read like notes. That's correct — the controlled variable is length, and if the terse variant were better written I would be measuring two things at once.

Result 1: the judge is generous, not wrong

Here is the confusion matrix from a live run of deepseek-v4.1-flash against 20 human labels:

              j1    j2    j3    j4    j5
        h1     4     0     0     0     0
        h2     1     0     0     0     0
        h3     1     3     0     0     0
        h4     0     0     0     0     3      <- every human-4 answer got a 5
        h5     0     0     0     0     5

Raw agreement 52.9%. Unweighted Cohen's κ = 0.387 — fails the 0.6 threshold outright. But quadratic-weighted κ = 0.888, which reads as "excellent judge".

Both numbers are correct and they disagree because they answer different questions. Weighted κ treats a 4-vs-5 miss as a near-hit. Unweighted κ treats it as a miss. If your downstream use is "did this answer reach the top band", the offset is the entire signal, and the plain κ verdict is the honest one.

This is the argument for shipping both numbers next to each other rather than picking a metric and moving on.

Result 2: position bias was absent — genuinely

0 flips. Mean score delta 0.000. First-slot preference 45.5%, which is a coin flip. 100% agreement with the human preference across 8 pairs.

Worth stating plainly because a null result here is a finding: this judge's pairwise ordering was not moved by presentation order. Not every experiment has to find something.

Result 3: the one that changed how I think about this

I ran the same configuration four times. Same 20 items, same 8 pairs, same model, judge temperature 0.0 — the setting where a model is supposed to be most deterministic.

Quantity	Run A	Run B	Run C	Run D	Threshold
Cohen's κ	0.297 FAIL	0.387 FAIL	0.321 FAIL	0.323 FAIL	≥ 0.60
verbosity delta	0.533 FAIL	0.200 PASS	0.462 FAIL	0.353 FAIL	≤ 0.30
verbosity 95% CI	[−0.067, 1.333]	[−0.467, 0.933]	[−0.154, 1.308]	[−0.177, 1.000]	all four straddle 0.30
items scored in agreement pass	18/20	17/20	19/20	17/20	—
pairs completing both orderings	7/8	4/8	4/8	3/8	—

The verbosity check passed once and failed three times on identical runs.

Look at the confidence intervals: each one is roughly ±0.7 wide, and the threshold is 0.3. The threshold sits inside the noise. This gold set cannot resolve that check, and no amount of re-running will make it able to.

Two things follow from that, and they apply well beyond this project.

A pass on a quantity whose interval straddles its threshold is not a pass. It is undecided, and a harness should say so. That is why every number in Arbiter's bias table is printed next to its CI rather than on its own.

Sample size is a first-class property of an eval, not a config value. Twenty items is the default in every quickstart I have ever written. For detecting a 0.3-point effect at the observed variance, twenty is the wrong number, and the fix is not more seeds — it is more items, or a wider threshold stated honestly.

I nearly averaged this away. The version of the README I started writing quoted one run and reported "verbosity bias: 0.53, judge inflates long answers", which is a real effect from a real measurement and also not stable enough to be a finding. Running it three more times is the only reason I know the difference — and note that the one passing run is the run whose full output is published, so a reader seeing only that table would conclude the judge has no verbosity problem.

War story: Pydantic AI cannot talk to the default provider

The spec pinned the default provider to an OpenAI-compatible endpoint and the framework to Pydantic AI. Both were reasonable. Together they did not work.

pydantic_ai.exceptions.UnexpectedModelBehavior: Invalid response from openai chat completions endpoint:
1 validation error for ChatCompletion
metadata.weight_versions
  Input should be a valid string [type=string_type, input_value=[{'version': 'default', 'start': 0, 'end': 239}], input_type=list]

The gateway returns metadata as an object whenever tools or structured output are involved, and the OpenAI SDK declares ChatCompletion.metadata as str | None. Plain unstructured calls return
metadata: null and work fine, so the first probe succeeds and the real one fails, which is a confusing way to discover it.

Unknown extra fields are tolerated by the SDK; only this mistyped known field breaks it. The fix is a small httpx response hook that repairs the body before the SDK validates it:

async def sanitize(resp: httpx.Response) -> None:
    if resp.status_code != 200: return
    if "application/json" not in resp.headers.get("content-type", ""):
        return                                    # never touch SSE streams
    body = await resp.aread()
    try: payload = json.loads(body)
    except (json.JSONDecodeError, UnicodeDecodeError): return
    if not isinstance(payload, dict) or not isinstance(payload.get("metadata"), (dict, list)):
        return
    patched = json.dumps(sanitize_metadata(payload)).encode()
    resp._content = patched
    resp.headers["content-length"] = str(len(patched))

client = AsyncOpenAI(base_url=base_url, api_key=key,
                     http_client=httpx.AsyncClient(event_hooks={"response": [sanitize]}))
model = OpenAIChatModel(model_id, provider=OpenAIProvider(openai_client=client))

After that, structured calls run in 1.2–2.3 seconds. Every "the framework is broken" investigation deserves five minutes of reading the raw response body with curl first — the answer was in the metadata field the whole time.

The verdict is code, and it is on screen

Thresholds buried in a config file are how you end up arguing about a number six months later. They are constants, and the UI renders them next to the measurement:

KAPPA_MIN = 0.6
VERBOSITY_DELTA_MAX = 0.3
FLIP_RATE_MAX = 0.15

VERDICT: judge is NOT trustworthy for this rubric
  - Cohen's kappa vs human labels >= 0.60        value    0.387  FAIL
  - |padded - terse| mean score delta <= 0.30    value    0.200  PASS
  - position-swap flip rate <= 15%               value    0.000  PASS
! Cohen's kappa 0.387 < 0.6 — agreement with human labels is too weak to trust this judge for this rubric

All three must pass. A judge that agrees well but inflates long answers is not trustworthy for a rubric where length is a confound. And a degenerate result is a failure, never a pass: if a judge collapses every item onto one score, κ returns NaN rather than 0, because "unmeasurable" and "no agreement" are different claims and only one of them is reassuring.

What I'd add next

The obvious one, and it follows directly from Result 3: run the whole rig N times and report the verdict distribution. Right now the instability is visible only because someone ran it three times. A verdict_stability panel — "this judge passes the verbosity check in 1 of 5 runs" — is a more honest interface than a single PASS chip.

Then: a power analysis that warns when the gold set is too small to resolve its own thresholds; more probes in the same shape (authority bias, self-preference, a "nothing to grade" control that should score 1); judge ensembles with inter-judge κ; and calibration that transfers — fit an isotonic mapping from judge score to human score and report held-out error, replacing the hand-weighted correction I shipped.

Try it

git clone https://github.com/harishkotra/arbiter.git && cd arbiter
uv venv --python 3.13 .venv
uv pip install -p .venv/bin/python pydantic-ai fastapi "uvicorn[standard]" httpx python-dotenv pytest
cp .env.example .env                                # any OpenAI-compatible endpoint works
cd client && npm install && cd ..
npm run dev          # api :3001, client :5173
npm run verify       # 116 live judge calls, six checks, exits non-zero on failure

159 pytest tests, 9 client tests, no network required for either. npm run verify:offline exercises the whole pipeline through a scripted judge and labels itself a fixture on every line, because passing structural checks is not the same claim as measuring a judge.

Code & more: https://www.dailybuild.xyz/project/275-arbiter

How I built a time-travel debugger for LLM agents on Burr, and what "replay the unchanged branch and prove nothing changed" actually takes.

Harish Kotra (he/him) — Mon, 05 Oct 2026 17:25:56 +0000

Rewind: counterfactual replay for agent pipelines

Every agent trace tool I've used has the same ceiling. It shows me a waterfall of steps, I spot that step six produced something stupid, and then I'm stuck. To find out why — or whether it was the model, the retrieval, or the prompt — I have to re-run the whole pipeline and hope it lands in the same place. With a non-deterministic model it doesn't, so I can never separate my change from model jitter.

So I built the other thing: fork a completed run at any step, change exactly one input, replay only the downstream steps, and diff the two trajectories. Rewind does that, and — this is the part that took real care — when you replay a branch you didn't change, every output hash comes back identical and the replay costs zero tokens. That's the difference between a diff you can trust and a diff that's just noise.

Here's the whole thing: the architecture, the four design rules that make determinism provable, and the handful of traps I walked into so you don't have to.

The shape of it

browser (Vite + React 18 + TS + Tailwind)
  Graph · NodeEditor · DiffPanel · CostBar · RawJSON · Settings
        │  /api  (Vite proxy)
        ▼
FastAPI ──── spawns a daemon thread per run; client polls /graph for live nodes
        │
        ▼
Burr application ── plan → research → analyse → critique → revise → compose → verify → publish
  │        (llm)     (tool)    (tool)     (llm)      (llm)    (llm)     (tool)    (tool)
  ├── SQLitePersister("burr_state")        ← state after every node; what forking reads
  └── NodeTelemetryHook(PostRunStepHook)   ← inputs/outputs/latency/tokens/content-hash per node
        │
        ▼
rewind.db (SQLite, WAL): runs · nodes · edges · cache · tool_cache · burr_state
data/*.csv, data/*.json  →  artifacts/*.md

Eight nodes, four real model calls, four real local computations over a real dataset file. The graph is deliberately boring — a linear chain — because the interesting axis is time, not topology.

Burr already had forking. I just had to find the right door.

I expected to hand-roll "resume from step N". Burr 0.42 has it natively:

builder.initialize_from(
    persister,
    resume_at_next_action=True,               # ← the important one
    default_state=initial,
    default_entrypoint=node_id,
    fork_from_app_id=src_app_id,
    fork_from_sequence_id=src_sequence_id,
)

The trap: to re-run node N, you fork from N's predecessor row. A completed row's successor becomes the entrypoint. Pass N's own row and you silently start at N+1 — which is exactly what my first spike did, and it looked like the framework was broken. It wasn't; I'd asked for the state after the node I wanted to re-run.

parent    ── plan ─ research ── analyse ─ critique ── … ── publish
rows        seq 0     seq 1       seq 2      seq 3            seq 7
                            └── fork_from_sequence_id = 1, resume_at_next_action=True

fork      ┈┈┈ inherited ┈┈┈▶ analyse ▶ critique ▶ … ▶ publish
          copied verbatim,      └── re-executed (cache hits in cache mode)
          inherited: true

Forking the first node has no predecessor row, so that one case starts a fresh application carrying the same override. And nested forks work because each node row keeps the src_app_id and sequence_id of the app that genuinely produced it — so forking inside a fork resolves to real executed state rather than to a copy of a copy.

Rule 1: state is semantic only

This is the rule that makes byte-identical replay possible, and it's the one I'd violate first if I weren't watching.

Latency, token counts, costs and run ids never enter Burr state. They travel in the action's result payload and get written to the nodes table by a PostRunStepHook:

class NodeTelemetryHook(PostRunStepHook):
    def post_run_step(self, *, app_id, partition_key, sequence_id,
                      state, action, result, exception, **kw):
        t = (result or {}).get("_telemetry") or {}
        db.upsert_node(self.db_path, run_id=app_id, node_id=action.name,
                       latency_ms=t["latency_ms"], total_tokens=t["total_tokens"],
                       cost_usd=t["cost_usd"], cache_hit=t["cache_hit"],
                       output_sha256=t["output_sha256"], …)

Think about why. Node 3's prompt embeds state_json. If node 2's latency lived in state, node 3's prompt would contain a wall-clock number, its cache key would differ on every run, and "replay the unchanged branch" would be a permanently broken feature — while still looking completely correct in the UI. The bug wouldn't surface as a crash; it would surface as nothing ever hits cache and you'd spend a week debugging a determinism claim that was structurally impossible.

Rule 2: strip the framework's own keys

Burr keeps __PRIOR_STEP and __SEQUENCE_ID inside persisted state. A child run's sequence ids differ from its parent's by construction, so they must never reach a prompt or a hash:

def semantic_state(state: State) -> dict:
    return {k: v for k, v in state.data.items() if not k.startswith("__")}

Rule 3: overrides live outside state

The fork's override is passed to the run context, not injected into state. If overrides were a state key, then a no-change replay would carry {"node_overrides": {}} where the parent had no such key — different state_json, different prompt, cold cache, and the headline feature quietly false. The override still lands in state where it belongs: it changes the node's output, and outputs are state.

Rule 4: hash the content, not the container

The artifact hash is the hash of the markdown bytes. The run-scoped filename is recorded on the run row and deliberately excluded from the hashed tool result:

hashed_result = {"tool": result["tool"], "bytes": result["bytes"], "written": True}

I got this wrong first. The publish node stored its path in the tool result, so every run diverged at node 8 — and my diff dutifully reported first divergence: publish, which was true, meaningless, and cost me a verification cycle to notice.

The cache key

key = sha256( model | temperature | exact prompt | tool result )

Length-prefixed, so the concatenation can't collide:

def _part(value: str) -> str:
    return f"{len(value.encode()):d}:{value}"

def llm_cache_key(*, model, temperature, prompt, tool_result):
    return hash_text(_part(model) + "|" + _part(repr(float(temperature)))
                     + "|" + _part(prompt) + "|" + _part(tool_result or ""))

Tool nodes get their own table keyed on (tool name, canonical args). That's not decoration: without it, an unchanged replay would recompute every tool step and get the same answer by luck rather than by proof — and "every node is a cache hit" would be a claim about arithmetic, not about the system.

Each node also records why it was or wasn't a hit, because a boolean can't tell the truth here:

`cache_source`	meaning
`live`	went to the provider / recomputed; tokens charged
`cache`	LLM response replayed from cache
`tool-cache`	tool result replayed from the tool cache
`override`	value supplied by the fork; nothing executed
`render-cache`	artifact bytes unchanged (the file is still written for this run)
`inherited`	copied from the parent; never executed

The diff engine: name the field, not the blob

Comparing two nested dicts and reporting "these differ" is useless. So flatten to leaves:

def flatten(value, prefix=""):
    if isinstance(value, dict):
        for key, item in value.items():
            yield from flatten(item, f"{prefix}.{key}" if prefix else str(key))
    elif isinstance(value, list):
        for index, item in enumerate(value):
            yield from flatten(item, f"{prefix}[{index}]")
    else:
        yield prefix or ".", value

Then the diff says:

first divergence: analyse → tool result JSON (analyse tool result)
  result.groups.A.cost_per_outcome   26.4682  →  79.4046
  result.groups.A.total_cost         344.75   →  1034.25
artifact  0f661db5… → af9df990…   differs: true

and the two published artifacts really do read differently — the parent says A | 26.47, the fork says Policy A: … 79.4046. That's the whole point of the tool: a one-field counterfactual, visible in prose.

Fail loudly, three times over

The brief's guardrail was "on parse failure the node fails loudly with the raw output attached; the run is marked failed, not silently patched". I kept that and added two more, because silent tolerance is how a determinism claim rots:

Unparseable / missing / empty output → NodeFailure(node, msg, raw=raw). No fence-stripping retry loop that manufactures a passing answer.
An override the node can't consume. A prompt edit on a tool node has no prompt to edit. Originally my code just… ignored it, and the fork produced an identical artifact. The test failed with first divergence: publish, and the real bug was that I'd applied a no-op. Now it raises, and the editor only offers fields the node kind actually uses.
A client-supplied key is used for that request and never persisted, echoed, or returned — /api/settings exposes hasKey: true and nothing else. Not even a masked prefix; I had one and removed it, because "first four characters" is still key material on the wire.

What verification proved

npm run verify boots its own server on an OS-assigned free port and asserts, per task: 7+ nodes with hashed outputs and a real artifact file; fork at node 3 with inherited 1–2 and re-executed 3+; the diff naming node 3 and the exact field; an unchanged replay hitting 100% cache with 0 incremental tokens; and a different artifact hash for the fork. 14/14, exit 0.

The number I like most: the parent pass cost 10,054 tokens; the unchanged replay cost 0, and its artifact file is byte-identical to the fork's (cmp clean). I also recomputed the CSV maths independently, outside the app, and it matches the artifact exactly — so "real computation" isn't taking my word for it.

Two findings from poking at it:

A run survived a dead provider. Pointed at an unreachable base URL, it completed — every LLM node was cached and the rest were local tool calls. Forcing a cache miss against the same dead endpoint failed instantly with the transport error attached. That's the determinism property behaving like a feature.
LM Studio silently substitutes an unknown model id. A model override to no-such-model-xyz returned HTTP 200 with model: google/gemma-4-e4b. So model overrides are only meaningful against a provider that validates the id. Rewind records the model the provider answered with, so the substitution is visible in the node record instead of hidden.

Traps I'd like a week of my life back

The OpenAI SDK and Particle's metadata. The gateway returns metadata as an object whenever tools/structured output are involved; the SDK types it str | None, so every call dies with UnexpectedModelBehavior. A plain httpx POST parses the body itself and ignores unknown fields — immune by omission. The spec asked for plain fetch anyway.
response_format: {"type":"json_object"} is accepted by Particle and rejected by LM Studio with a 400 naming the field. One targeted retry without it keeps both presets usable while the specified behaviour stays primary.
Port clusters of look-alike processes. Several sibling builds run uvicorn agent.main:app with identical command lines on 3001/3011/…. A 2xx from a port probe proves nothing about which app answered. So /api/health carries "service": "rewind" and verification asserts on it.
Vite's IPv6 default lets a second dev server "succeed" on :5173 while the first owns 127.0.0.1 — then curl 127.0.0.1:5173 reports the wrong app. Bind 127.0.0.1 explicitly and make the proxy target an env var.
My own test harness deleted a live run's output. The scratch mechanics script did rmtree("artifacts") at startup, and I ran it while a verification was mid-flight. Four checks failed with a confusing 404, and the cause was me, not the code. Now the artifact directory is an env override (REWIND_ARTIFACT_DIR) and the harness writes into .scratch/. Lesson: a test that cleans a shared directory is a test that will sabotage a parallel run.

What's next

The highest-leverage addition is a what-if grid: fork one node across N overrides in parallel and render an N-way matrix, so "which of these five retrieval queries changes the conclusion?" is one screen instead of five forks. After that: non-linear graphs (which makes the diff align two trees), streaming node updates over SSE instead of 1.2 s polling, and a hash-stability mode that runs the same branch k times in live mode and reports the distribution of output hashes — turning "is my agent deterministic?" into a number rather than a binary.

Try it

git clone <your-fork> && cd rewind
uv venv --python 3.12 .venv && uv pip install -p .venv/bin/python -r requirements.txt
cp .env.example .env      # set base URL / key / model
npm install --prefix client
npm run dev               # → API + client, on ports it proves are free
npm run verify            # → 14 checks, exit 0

The README documents the fork semantics, the cache key, the data model, the API surface, and every substitution I made against the original brief — including the fact that the recorded verification ran on a local model because no provider key was present in this project's environment.

Feedback and PRs welcome.

Code & more: https://www.dailybuild.xyz/project/274-rewind

Your agent is a long-running process with side effects. Kill it and watch what happens.

Harish Kotra (he/him) — Sun, 04 Oct 2026 17:08:18 +0000

A teardown of **Checkpoint: a 30-step agent run on Temporal that survives kill -9, with an exactly-once ledger that proves it in the UI.

Every agent tutorial ends at the same place: a while loop that calls a model, calls a tool, writes something to the world, and repeats. It works beautifully until the process dies at step 27 of 30 and then you learn what your loop actually was: a record of everything it had already done,
held in memory that no longer exists.

The instinctive fix is a checkpoint object — serialize the loop index to disk, restore it on boot. That works right up until the crash lands in the two lines between "the model answered" and "the email was sent", at which point you are writing a distributed database and calling it a bug fix.

Checkpoint is the version where the loop's position is not state you maintain — it is a log someone else already stores durably, and every observable side effect is gated by a key that cannot change across retries. Then I built a button that sends a real SIGKILL to the worker process, and a test suite that fails if anything is done twice.

Here's how it works, and the three bugs I found by actually pressing the button.

The claim being tested

Not "the app is resilient" (unfalsifiable). Instead:

SELECT COUNT(*) - COUNT(DISTINCT id) FROM effects;   -- must equal 0

That number sits in the ledger panel's header, green when it's 0, and the app computes it on every refresh. Kill the worker at step 7 of 16 and the run completes with 48 ledger rows and duplicates: 0.

Everything below exists to make that sentence defensible.

Three processes, two stores

 panel :5173 ──SSE──► Temporal :7233 ─── EVENT HISTORY ───►  the run's position
     │                    ▲   ▲
     │ /api/*             │   │ poll tasks
     ▼                    │   │
 API :3001 ──spawn/kill──►│   └──► WORKER (a pid you can murder)
     │                          workflows/agent.ts   ← the loop
     │                          activities/{llm,tool,effect}.ts
     ▼
 SQLite checkpoint.db  effects · steps · runs · faults · worker_events · notes
     └── artifacts/<runId>/step-007.md · outbox.log

The rule I held myself to: anything the UI presents as evidence must be readable from SQLite or Temporal history after both processes are killed. That single constraint killed most of the ways I would have cheated.

The loop is a plain async function — and that's the trick

// server/src/workflows/agent.ts
export async function agentLoop(input: AgentInput): Promise<AgentSummary> {
  const { workflowId } = workflowInfo();
  for (let index = 1; index <= input.total; index++) {
    await book.markStepRunning({ runId: input.runId, step: index });
    const outcome = await step.llmActivity({ runId: input.runId, step: index, total: input.total, previousNote });

    const toolResult = outcome.decision.action === 'compute'
      ? (await step.toolActivity({ runId: input.runId, step: index, value: outcome.decision.args.value })).result
      : null;

    await step.effectActivity({ runId: input.runId, step: index, workflowId, decision: outcome.decision, toolResult });
    await book.markStepCompleted({ runId: input.runId, step: index });

    await sleep(300 + ((index * 137) % 501));   // 300–800ms, derived from the index
  }
}

On every task, the SDK re-runs this function from the top. Activity calls don't re-execute — their recorded results are fed back from history, so the loop is effectively "seeking" to the frontier. That's why the sleep is a formula rather than Math.random(): replay must schedule identical timers. There is no clock, no fetch, no fs anywhere in workflow code; timestamps come from activities.

SDK trivia: the docs still say workflow.now(). It doesn't exist in 1.24 — the sandbox patches Date/setTimeout inside workflows, and the unsafe escape hatch moved to workflowInfo().unsafe.now(). This loop needs neither.

Where the crash-safety actually lives: the idempotency key

Durable execution gets you re-execution. It does not get you don't-send-two-emails — an activity whose worker died is re-run from the beginning, by design. That's exactly-once processing, not exactly-once effects. The bridge is a key that cannot change:

effectId = `${workflowId}:${stepIndex}:${kind}`     // kind ∈ file | db | email

workflowId is the run id — stable across activity retries, worker respawns, API restarts. Then the
write itself is the lock:

// server/src/ledger.ts
export function applyEffectOnce(db, spec, physical) {
  const claim = db.transaction(() => {
    const res = db
      .prepare('INSERT OR IGNORE INTO effects(id, step, kind, payload, applied_at, attempt) VALUES(?,?,?,?,NULL,?)')
      .run(spec.id, spec.step, spec.kind, spec.payload, spec.attempt);
    return { claimed: res.changes === 1, existing: db.prepare('SELECT * FROM effects WHERE id = ?').get(spec.id) };
  })();

  if (claim.claimed) { physical(); stamp(spec.id); return { applied: true }; }        // we own it
  if (claim.existing.applied_at) return { applied: false };                            // no-op: already done
  physical(); stamp(spec.id); return { applied: false, recovered: true };              // finish an interrupted write
}

Three branches, three crash stories:

situation	what happens	result
first attempt	claim → write → stamp `applied_at`	`applied: true`
retry, or re-dispatch after a kill	PK conflict → read row → skip the write	`applied: false`
killed between claim and stamp	row exists with `applied_at IS NULL` → redo the deterministic write	`recovered: true`

The third row forces a design rule that pays off everywhere else: payloads are built only from activity arguments, because those arguments are already in history. A replayed attempt produces byte-identical file content, so re-running an interrupted write is boring — and boring is the goal. The email stub appends to outbox.log; the claim row is what stops the second append.

Note the deliberate trade-off: this is at-most-once per claim, so a crash between claiming and writing leaves a row that lies. The applied_at IS NULL marker turns that silent corruption into a detectable, repairable state. If your side effect isn't deterministic (a payment, an email with a timestamp in it), you'd instead store the rendered payload on the claim row and hand the same string to the transport on every attempt.

Bug #1 — a "fast" resume took three minutes

First version. Kill the worker, watch the panel: the replacement came up in ~1.5s and… nothing. Step 7 stayed running for 180 seconds, then resumed correctly. The data was never at risk; the latency was a lie I was telling myself about how crash recovery works.

Cause: the activity task was dispatched to worker A. A died without acknowledging. Temporal doesn't guess — for an in-flight activity it waits for startToCloseTimeout, which I'd set to 3 minutes to accommodate slow models.

Fix: heartbeats. The activity proves liveness every 2s; the server notices the silence at
heartbeatTimeout and re-dispatches.

// server/src/activities/heartbeat.ts
export function beat(details: unknown): () => void {
  let stopped = false;
  const once = () => {
    if (stopped) return;
    try { heartbeat(details); } catch { stopped = true; clearInterval(timer); }
  };
  once();                       // never-beaten = can't be declared dead early; announce yourself
  const timer = setInterval(once, 1000);
  timer.unref();
  return () => { stopped = true; clearInterval(timer); };
}

// activity options
{ startToCloseTimeout: '3 minutes', heartbeatTimeout: '5 seconds',
  retry: { initialInterval: '400ms', backoffCoefficient: 2, maximumInterval: '5 seconds', maximumAttempts: 8 } }

That moved kill→resume from minutes to ~12 seconds — and then I stopped pretending to understand
the rest of it, measured it, and was wrong about the cause. Three samples, three builds:

build	heartbeatTimeout	workflowTaskTimeout	measured kill → next step starts
original (no beats)	—	default 10s	>180s — stalled for the whole `startToCloseTimeout`; verify's 300s budget expired
+ heartbeats	12s	default 10s	11.4 – 12.0s
+ tighter beats	5s	default 10s	11.9 – 12.5s
+ tighter beats	5s	2s	12.0s, 12.0s, 18.0s

The heartbeat change was worth ~3 minutes. Everything after that was noise: the last ~10 seconds is the Temporal dev server re-delivering the outstanding workflow task, and none of the knobs I had access to moved it. I reverted the workflowTaskTimeout experiment — an unearned deviation from a default is worse than a slow demo — and wrote the number down as 12 seconds instead of the "2–4 seconds" I had confidently put in the README first.

Two lessons worth keeping: measure before you name the mechanism (I blamed heartbeats for a deadline that was never binding), and a negative result belongs in the docs — it's the part that saves the next person an afternoon.

Cancellation gets the same treatment: an activity can only receive it if it beats, so the 1s cadence buys a clean shutdown too.

Bug #2 — retries were invisible, because every activity has its own counter

Verification scenario B injects two transient failures at step 8 and asserts the timeline shows attempt 2. It failed: attempt=1. Injection clearly worked (the step retried and completed) but the ledger said one attempt.

Cause: steps is upserted by several activities — llmActivity logs attempt 2 for the model call, then markStepCompleted writes… attempt 1, because its attempt counter is 1. Last writer won and erased the evidence.

-- server/src/ledger.ts, upsertStep
ON CONFLICT(run_id, step) DO UPDATE SET attempt = MAX(steps.attempt, excluded.attempt), ...

The generic lesson is about observability, not SQL: a field written by more than one process should have an explicit merge rule. MAX was the right semantics for "how many times did this step be worked on"; it would have been the wrong one for, say, ended_at.

Bug #3 — a model that rambles is a config bug, not a crash

The prompt demands JSON. Local models sometimes answer in prose. Per the spec, a parse failure is a non-retryable guardrail — so a 32-step demo died at step 2 because a 4-billion-parameter model decided to explain itself:

throw new ApplicationFailure(`Guardrail: step ${step} returned non-JSON output`, 'GuardrailError', true, [{ raw }]);

Two changes, in order of honesty:

A tolerant extractor: fenced blocks, then a hand-rolled balanced-brace scan for prose-wrapped or truncated JSON (with finish_reason === 'length' recorded, so the UI can say "truncated").
One corrective re-prompt inside the activity — "your previous reply was rejected; return only the JSON object, strings under 15 words" — before the guardrail fires. Persistent garbage still fails fast and non-retryably, with the raw 500 chars stored on the step row and shown in a red banner. The injected garble fault bypasses repair entirely, which is how the test still proves the guardrail path rather than assuming it.

I also stopped trusting preset model ids: local servers answer for whatever is loaded, so the fallback chain now asks /v1/models first. Without that, a silent downgrade to the offline provider was producing a "working" demo that wasn't calling any model at all.

The kill button is not a simulation

// server/src/supervisor.ts
async kill(timeoutMs = 8000): Promise<KillReport> {
  const pid = this.pid ?? readPidFile();
  const atStep = this.getActiveStep();
  const observed = this.waitForExit(timeoutMs);
  process.kill(pid, 'SIGKILL');                     // ← undeliverable, uncatchable, real
  const { signal, exitCode } = await observed;       // resolves from the child's 'exit' event
  return { pid, signal: signal ?? 'SIGKILL', exitCode, confirmed: !pidAlive(pid), atStep };
}

A SIGKILL leaves no exit code, so the panel says SIGKILL · no exit code instead of inventing one.

The worker is spawned detached: true + child.unref(), which enables the best test in the suite:
SIGKILL the API process mid-run. The worker survives (its parent dying is not its problem), a
fresh API reads run/worker.pid, sees the pid alive, and adopts it rather than starting a second
consumer:

PASS  5a. the api process is gone but the worker survived     api pid 96049 alive=false; worker pid 96449 alive=true
PASS  5b. the restarted api adopted the same worker           new api pid 98642 adopted worker 96449; completedSteps already 6

The UI is a thin client over durable data

The panel holds no run state either — that would be embarrassing for an app about statelessness. It subscribes to an SSE stream where the server diffs SQLite each tick:

// server/src/api.ts
for (const s of next.steps) {
  const before = prev?.steps.find((p) => p.step === s.step);
  if (!before || before.status !== s.status || before.attempt !== s.attempt || before.worker_pid !== s.worker_pid) {
    send('step', { step: s.step, status: s.status, attempt: s.attempt, tokens: s.tokens, cost: s.cost_usd ?? 0, durationMs, provider: s.provider, workerPid: s.worker_pid, resumed: next.resumeStep === s.step });
  }
}

resumeStep is derived, never remembered: the first step whose worker_pid differs from the step before it. So the ⚑ resumed here chip is a query result, not a UI memory — reload mid-crash and
it lands in the same place.

The history panel is the raw event log (handle.fetchHistory(), EventType names from @temporalio/proto), which is the honest place to look when the timeline and the ledger disagree.

Proof, not prose

npm run verify runs four scenarios and prints observed numbers:

PASS  1. worker really dies — API reports the observed signal + exit code   signal=SIGKILL exitCode=null pid=65590 confirmed=true observedAtStep=6
PASS  1b. the killed pid is gone from the process table                     previous pid 65590 alive=false; status=worker-dead
PASS  2. run completes after resume (workflow COMPLETED)                    status=COMPLETED completedSteps=16/16 resumedAtStep=7
PASS  2b. step log has no gaps                                              steps 1..16 missing=[]
PASS  2c. a different worker pid finished the run                           worker pids: 65590, 66292
PASS  3. ledger proves exactly-once after a mid-run kill                    COUNT(*) - COUNT(DISTINCT id) = 0 (141 rows)
PASS  4. retried step shows attempt >= 2 and wrote no duplicate effect      step 8: attempt=3 effects=3 uniqueIds=3
PASS  5. no run state was lost: the run finishes after an api restart       completedSteps=16/16 duplicates=0
PASS  6. completed step count equals TARGET_STEPS                           A=16/16  B=12/12  C=16/16
ALL CHECKS PASS — 14/14

Each assertion prints the value it looked at, so a pass is auditable and a failure is a diagnosis:

function check(name, ok, observed) {
  results.push({ name, ok, observed });
  console.log(`${ok ? 'PASS' : 'FAIL'}  ${pad(name, 58)} ${observed}`);
  return ok;
}

I also drove it in a browser and recorded the DOM, because "the state exists in the reducer" is not
the same as "the user sees it": click Kill worker →

worker process killed — a replacement worker is polling the queue now · observed signal SIGKILL
  · exit code none            → pill: worker-dead · RUNNING
(none)                        → pill: running · RUNNING      # resumed on the new pid
run completed — 24 steps, 72 ledger rows, duplicates 0 · applied exactly once across 2 worker processes

And once from outside the app entirely: kill -9 $(cat run/worker.pid). The panel noticed, respawned, the run finished, duplicates stayed 0.

What this does not claim

Not exactly-once delivery. Temporal gives at-least-once activity execution; the ledger makes effects idempotent. If you can't render the effect deterministically, you need a transactional outbox, not a cleverer key.
Not a crash-proof clock. steps.started_at is stamped by whatever worker ran the attempt, and the resume marker uses worker-identity changes — a worker that restarts for an unrelated reason also flips that flag. It's honest evidence of "a different process did this step", which is the claim I want, but it isn't a general-purpose audit clock.
Not production. Single namespace, one task queue, no auth, no multi-tenancy, SQLite on a local disk, dev server with file persistence. Non-goals, deliberately.
The verified run above answered on the offline deterministic provider — no real API key in that session (placeholder → HTTP 401) and a hanging local server. Crash-safety claims don't depend on the model, but the "thoughts" in those runs are synthetic. With a key or a healthy local server the same scenarios run model-backed; the UI labels which provider answered every step.

Steal these four ideas

Give every side effect a key derived only from durable identity — (workflowId, step, kind) in my case. If the key can change between retries, you don't have an idempotency key.
Make the write itself the lock (INSERT OR IGNORE + changes === 1). No advisory locks, no separate state machine.
Stamp completion last. Claim with applied_at NULL, do the work, then set the timestamp — and you get a detectable "interrupted" state instead of silent loss.
Assert on numbers the app computed. If your resilience demo's proof is a console.log you wrote by hand, you built a slideshow.

Run it: clone the repo, npm install && npm run temporal:install && npm run dev, start a 32-step run, press Kill worker around step 7, and watch step 7 get claimed twice and applied once.

Code & more: https://www.dailybuild.xyz/project/273-checkpoint

Sleeptime memory consolidation: an agent that edits its own memory and proves it still remembers

Harish Kotra (he/him) — Sat, 03 Oct 2026 18:42:33 +0000

Most agent memory today is one of two failure modes. Either it's an append-only log that grows
until the context window chokes on six-week-old trivia, or it's a lossy summary that quietly
rewrites history and can't tell you what it dropped. Both avoid the actual hard part:

Deliberate, auditable forgetting under a stated retention policy — plus proof that recall still
works afterwards.

So I built Nocturne: a chat agent with a Sleep button. You chat, mixing signal and noise; you
press Sleep; four passes run over its memory blocks — Dedupe → Merge → Promote → Forget — each one
an LLM call whose output is refused or accepted by code, with a snapshot before and after. A diff
view shows every kept / changed / forgotten item with its reason. Then a Wake panel quizzes the
agent: five questions only the consolidated store can answer, plus one about something it was
supposed to have forgotten — where "I don't have that" is the correct answer.

Everything in this post is from the run in the repo: 18/18 verification checks, exit 0.

1. The rule that makes the whole thing defensible

One sentence, and it decides every design choice:

The model proposes. The code applies.

GUARDRAIL = (
    "You may only reference itemIds that appear in the input. You may not invent facts, "
    "names, numbers or dates. If you are unsure, omit the item from all lists."
)

That constant is appended to every pass prompt. It is not the safety mechanism — it's a polite
request. The safety mechanism is that each pass's JSON is re-derived from the data before anything
touches the store:

LLM proposal ──► gate (pure function over store state) ──► apply ──► snapshot
                         │
                         └── refuse ──► log with reason ──► surface in the UI

A rejected proposal is never swallowed. It shows up in the per-pass counts, in the diff panel under
Rejected mutations, and in the mutations table with its rejection_reason. In the run I
documented, the verifier caught the model proposing a drop for an item that had already been dropped
earlier in the same pass:

- drop   m_17   itemId m_17 was already removed by an earlier mutation in this pass

That line is the entire argument for the design: an unverified "consolidation" would have applied a
mutation against a phantom row.

2. Memory model

Four blocks — the shape Letta popularised:

block	holds
`persona`	durable facts about the assistant itself
`human`	stable facts about the user: identity, preferences, commitments
`episodes`	events anchored in time or place
`scratch`	working memory: unconsolidated facts pulled from recent conversation

Facts are rows, not prose. That's what makes a per-item diff, a per-item policy and a per-item
undo possible at all.

CREATE TABLE items(
    item_id TEXT PRIMARY KEY,          -- m_17
    block   TEXT NOT NULL REFERENCES blocks(name),
    text    TEXT NOT NULL,
    category TEXT NOT NULL DEFAULT 'context',   -- identity|preferences|commitments|event|...
    source_turn INTEGER,                          -- which transcript turn produced it
    created_at REAL NOT NULL,                     -- what the retention policy measures against
    updated_at REAL NOT NULL,
    active  INTEGER NOT NULL DEFAULT 1            -- forgetting flips this; rows are never deleted
);
CREATE TABLE snapshots(
    snapshot_id TEXT PRIMARY KEY, seq INTEGER NOT NULL UNIQUE,
    label TEXT NOT NULL, created_at REAL NOT NULL, active_count INTEGER NOT NULL
);
CREATE TABLE snapshot_items(            -- a full immutable copy of the state, every version
    snapshot_id TEXT NOT NULL, item_id TEXT NOT NULL, block TEXT NOT NULL, text TEXT NOT NULL,
    category TEXT NOT NULL, source_turn INTEGER, created_at REAL NOT NULL,
    updated_at REAL NOT NULL, active INTEGER NOT NULL
);
CREATE TABLE mutations(                 -- the audit trail, accepted AND refused
    mutation_id INTEGER PRIMARY KEY AUTOINCREMENT, snapshot_id TEXT, pass TEXT NOT NULL,
    kind TEXT NOT NULL, item_id TEXT NOT NULL, refs TEXT NOT NULL DEFAULT '{}',
    accepted INTEGER NOT NULL, reason TEXT, rejection_reason TEXT, policy TEXT, created_at REAL NOT NULL
);

The store exposes six methods, and that's the only interface the rest of the app uses:

list_blocks() · get_block(name) · append(block, item) · replace(block, items) · snapshot() · restore(snapshot_id)

flowchart LR
  UI[React :5173] -->|fetch + SSE over /api proxy| API[FastAPI :3001]
  API --> ENC[write-time encoder] --> G0[traceability gate]
  API --> RC[run_consolidation] --> P[Dedupe → Merge → Promote → Forget] --> G[ gates ]
  API --> RQ[run_quiz] -->|scored against store text| S[(SQLite<br/>items · snapshots · mutations)]
  G --> S
  API --> RS[restore] --> S
  API -.->|LETTA_MODE=letta| L[(Letta memory blocks)]

Backend that actually ran: SQLite. pip install letta today resolves to Letta Code (0.34.x), a
cloud-agent CLI — it no longer ships a self-hosted memory-block server, and there's no Docker daemon
on this machine, so nothing could bind :8283. I implemented the same block model over SQLite behind
the identical six-method interface, and kept letta-client installed so the swap is a config change.
The consolidation, diff, restore and quiz code never learns which backend is live.

3. The write-time encoder (where invention gets cut off first)

Before any sleep, each chat turn is encoded into atomic facts in scratch. Same guardrail, same kind
of gate — every content word of a proposed fact must exist in the transcript, and it may not
introduce a name the transcript doesn't contain:

cov, missing = tm.coverage(text, sources)
if cov < 1.0:
    rejected.append({"proposal": raw, "rejectionReason":
        f"fact is not traceable to the transcript: {len(missing)} content token(s) invented ({', '.join(missing[:8])})"})
    continue
new_names = tm.proper_nouns(text) - transcript_names
if new_names:
    rejected.append({..., "rejectionReason": f"fact introduces proper nouns absent from the transcript: {sorted(new_names)}"})

Real output from the verifier's seed step — the model tried to store something the user never said:

encoder rejected: fact is not traceable to the transcript: 1 content token(s) invented (prefer)
encoder rejected: fact is not traceable to the transcript: 1 content token(s) invented (run)

Duplicates are deliberately not filtered here. Append-only bloat is precisely the condition the
Dedupe pass exists to fix, and if the encoder silently collapsed repeats there would be nothing left
to consolidate.

Encoding is also incremental — it tracks an extracted_through cursor and only sends new turns.
Without that, every message re-encodes the entire transcript: prompt growth was quadratic and a
single chat turn took ~200 s before the fix.

4. The four passes

pass	contract the model must satisfy	what the code checks before applying
Dedupe	`{"drop":[{"itemId","duplicateOf","reason"}]}`	both ids exist in the pre-pass store and are active; not self-pairs; no duplicate chains; normalised token-set similarity ≥ 0.9; reason non-empty
Merge	`{"merge":[{"intoId","fromIds","mergedText","reason"}]}`	ids exist, unconsumed, same block; coverage(`mergedText`, sources) == 1.0; no new proper nouns; reason non-empty
Promote	`{"promote":[{"itemId","toBlock","reason"}]}`	`toBlock ∈ {persona, human, episodes}`; item active and elsewhere; reason non-empty; identity/preference/commitment facts applied first
Forget	`{"forget":[{"itemId","reason","policy"}]}`	item active; category not protected; and it qualifies: `ageDays > STALE_AFTER_DAYS` (stale) or `itemId ∈ overflowIds` (capacity). The code records the policy it verified, not the one the model claimed

Dedupe gets a little extra engineering: the code computes candidatePairs with its own comparison
and hands them to the model to adjudicate, so a duplicate the model failed to spot is still put in
front of it. The model still decides; the code still checks.

payload["candidatePairs"] = [
    {"dropId": loser, "keeperId": keeper, "similarity": round(score, 3)}
    for loser, keeper, score in tm.find_duplicate_pairs(state.active(), policy.similarity_threshold)
]

Merge in code, because it's the pass where a plausible-sounding rewrite is the whole danger:

cov, missing_tokens = tm.coverage(merged_text, sources_text)
if cov < 1.0:
    reject(raw, f"mergedText is not traceable to its sources: {len(missing_tokens)} content token(s) "
                f"appear in none of them ({', '.join(missing_tokens[:8])})")
    continue
new_names = tm.proper_nouns(merged_text) - {n for s in sources for n in tm.proper_nouns(s["text"])}
if new_names:
    reject(raw, f"mergedText introduces proper nouns absent from every source: {sorted(new_names)}")
    continue

And Forget, where the gate is a permission check, not a vibe check:

qualifies_stale = item["ageDays"] > policy.stale_after_days
qualifies_capacity = item_id in overflow
if not qualifies_stale and not qualifies_capacity:
    reject(raw, f"item is {item['ageDays']:.1f}d old (staleAfterDays={policy.stale_after_days:g}) "
                f"and not over capacity, so no retention rule authorises forgetting it")
    continue
if policy.is_protected(item):
    reject(raw, f"category {item['category']!r} is protected by the retention policy "
                f"({', '.join(policy.protected_categories)})")

5. The measures, with real numbers

Everything above rests on four tiny functions, which is what makes the rule set auditable in one
screen. Values below are computed by the shipped code, not written by hand:

similarity(a, b) = |A ∩ B| / min(|A|, |B|) over lowercased, punctuation-stripped,
inflection-folded, stopword-free token sets.

a	b	similarity	gate at 0.9
`I live in Lisbon and work best in the morning.`	`User lives in Lisbon and works best in the morning.`	1.000	duplicate
`The user keeps a bearded dragon.`	`The user keeps a bearded dragon called Saffron.`	1.000	duplicate (subset)
`I live in Lisbon and work best in the morning.`	`I live in Porto and work best in the morning.`	0.800	refused — different fact
`My stack is Airflow and Postgres.`	`My stack is Airflow and Kubernetes.`	0.667	refused

Containment rather than Jaccard is a deliberate choice: a restatement that adds words is the same
fact, and "lives in Lisbon" ⊂ "lives in Lisbon, Portugal" should collapse. The cost is that the two
sentences above score identically to the merge case, so near-duplicate in this project means
"same content, different wording", not "sits in a 0.9–0.99 band".

coverage(claim, sources) — fraction of the claim's content tokens present in the source
vocabulary. Sources keep their stopwords in the vocabulary, because a rewrite legitimately promotes a
function word into content ("use Python" → "uses Python").

claim (against two sources)	coverage	verdict
`The user installed postgres 16 on the backup box and the Kitchen leaks.`	1.00	tokens all traceable — but stopped by the proper-noun gate: `introduces proper nouns absent from every source: ['Kitchen']`
`The user installed postgres 16 on the backup box near the kettle.`	0.86	refused: `near` appears in no source
`The user migrated to Kubernetes on the backup box.`	0.50	refused: `migrated`, `kubernete` invented

stem_token is intentionally not a real stemmer — possessive stripped, then a trailing plural
-s unless the word ends in -ss/-us or is ≤ 3 chars: Lisbon's → lisbon, camps → camp,
business → business, gas → gas. Bounded rules mean every rejection can be explained in one
sentence, which is the difference between a guardrail and a mystery.

proper_nouns = capitalised-initial words that are not at a sentence start, so
"Marco moved to Lisbon in March, said Ana." → {Marco, Lisbon, March, Ana} while
"Lisbon is nice. The weather is warm." → {}.

6. Three deterministic backstops (and why they aren't cheating)

A rule the model can simply forget to apply is not a rule. Where the rule is derivable from the
data, the engine applies it itself and labels it:

enforcedBy: engine-duplicate-rule    # a >=0.9 pair the model left un-adjudicated
enforcedBy: engine-promotion-rule    # identity/preferences/commitments still in scratch → human; events → episodes
enforcedBy: policy-engine            # active set above MAX_ACTIVE_FACTS → lowest-salience unprotected facts

This is not the model being overruled by taste. Each backstop runs the same published measurement
the gate runs, and every such mutation is counted separately in the UI and the mutation log
(enforced=3 in the run below, distinct from the 20 the model proposed). Nothing is ever invented by
a backstop: they only move or retire text already in the store.

Salience, for the capacity rule, is explicit arithmetic rather than a judgement:

score = block_weight(persona 60 | human 50 | episodes 30 | scratch 10)
      + 40 if the category is protected
      + max(0, 30 - 30 * ageDays / STALE_AFTER_DAYS)          # recency, decaying to zero

7. Versioning: forgetting is a view change, not a delete

A snapshot is a full copy of the state, written before and after every pass. One Sleep looks like
this on the version line (real labels from nocturne.db):

 #10 consolidation:start  8 facts
 #11 pass:dedupe:before   8        → #12 pass:dedupe:after    7   drop m_6 (similarity 1.00)
 #13 pass:merge:before    7        → #14 pass:merge:after     7   REJECT 1 merge
 #15 pass:promote:before  7        → #16 pass:promote:after   7   promote ×5
 #17 pass:forget:before   7        → #18 pass:forget:after    6   forget m_8 (stale, 70d)
 #19 restore:s0011        8        ← "undo this sleep" (the restore is itself a version)

restore() deletes the live rows and re-inserts them from snapshot_items, then takes a new
snapshot — so undoing is undoable, and the diff between any two versions is exact because both
versions are fully materialised. Diffing is by item_id:

kept      present in both, same text and block
changed   text differs (merged) or block differs (promoted/moved) — with the sources it came from
forgotten active in A, absent or inactive in B — with the pass, the policy and the reason
added     appears only in B

8. Wake: proving recall, and proving forgetting

Five questions are generated against surviving facts, one against a fact the Forget pass removed. Two
details make this a real test rather than a demo.

(a) The answer key is the store, not the model. A reply is correct when it reproduces ≥ 0.9 of
the fact's content tokens. The question itself is also gated: a generated question that leaks half of
its own answer's content words is refused and recorded, because "Does the user work at Northwind
Logistics?" proves nothing about memory.

(b) The forgotten probe needs three independent conditions to pass, and it reports each:

passed = absent_from_store and said_absent and not leaked

absentFromStore=True      no active fact scores >= 0.9 against the forgotten text (checked in code)
saidAbsent=True           the agent's reply matches an explicit not-in-memory pattern
leakedForgottenContent=False   the reply does not reproduce the fact's content tokens

That third one is the interesting one: an agent that says "I don't have that" while blurting the
detail anyway has not forgotten anything. The measured run:

forgotten probe [OK]: "I don't have that in memory."
     absentFromStore=True saidAbsent=True leaked=False

Merged items are excluded from the question pool on purpose — their text is a concatenation of
several facts, so a faithful answer becomes a paragraph and the test stops measuring recall of a fact.

9. Streaming a four-minute operation without lying about progress

EventSource can't POST a body, so the client reads the response stream itself; the server runs the
consolidation on a worker thread and relays queue events:

const reader = res.body.getReader();
buffer += decoder.decode(value, { stream: true });
while ((cut = buffer.indexOf("\n\n")) >= 0) { /* parse event: / data: */ }

item = events.get(timeout=15.0)
if not thread.is_alive(): break
yield ": keepalive\n\n"        # a pass can be silent for minutes; keep proxies awake

Events are named pass:<name>:start|done|error, which is what lets the Sleep panel show live counts
per pass — input, proposed, accepted, rejected, policy-enforced, active-after, and the snapshot ids on
both sides — while it's still running. Dry run runs the identical four passes but applies nothing:
no live mutation and no snapshot, so the forgetting curve doesn't gain a fake point. The verifier
asserts exactly that (active stays 22->22).

Provider plumbing is one plain POST {baseUrl}/chat/completions call with
response_format={"type":"json_object"}. LM Studio answers that with
400 "'response_format.type' must be 'json_schema' or 'text'", so the client probes once per base
URL, remembers the rejection, retries without the field, and Settings reports json mode unsupported
rather than hiding it.

10. What the verifier proves

npm run verify (→ python -m agent.verify) seeds a scripted 8-turn conversation through the real
chat path — the assistant replies are generated by the model — and the real encoder, then runs a
dry run, a real consolidation, the quiz, and a policy comparison. 18/18, exit 0. From one run:

seeded in 92.4s; encoder proposed 21 facts, accepted 20, rejected 1, backdated 2 episodic fact(s)
active facts=22  exact-dup pairs=7  near-dup pairs=3  stale=2  protected=8  durable=14
3 pair(s) sit below the 0.9 gate (0.7-0.9): m_4/m_16 0.857, m_7/m_18 0.875, m_9/m_18 0.875

dry-run plan: proposed=19 accepted(if applied)=21 rejected=1 active stays 22->22
pass dedupe  in=22 proposed=10 accepted= 9 rejected=1 enforced=0 activeAfter=13
pass merge   in=13 proposed= 2 accepted= 2 rejected=0 enforced=0 activeAfter=11
pass promote in=11 proposed= 7 accepted=10 rejected=0 enforced=3 activeAfter=11
pass forget  in=11 proposed= 2 accepted= 2 rejected=0 enforced=0 activeAfter= 9
totals: {'proposed': 21, 'accepted': 23, 'rejected': 1, 'enforced': 3}

23 accepted, 1 rejected; 24 re-verified, 0 problems
9/9 active facts fully traceable to the transcript
MAX_ACTIVE_FACTS=120: forgotten=2,  active left=20
MAX_ACTIVE_FACTS=6:   forgotten=14, active left=8  (floor 8 = protected facts the policy refuses to forget)

The assertions that matter most are the paranoid ones:

Every accepted mutation is re-audited from the database, independently of the consolidation code: dropped pairs are re-measured for similarity, merged text is re-checked for coverage and new proper nouns, promote targets are re-checked against the whitelist, forgettings are re-checked against the rules. "The model said it merged" is never evidence.
A global no-invention sweep: every surviving active fact must be 100 % traceable to the raw transcript and must introduce no name absent from it. 9/9.
The gates are tested without any model at all — 16 adversarial and legal proposals pushed straight through them (ghost itemIds, self-pairs, dissimilar "duplicates", invented tokens, TitleCased names, illegal blocks, a stale-but-protected commitment, an ineligible forgetting), each required to refuse for the stated reason. This is the part that must not depend on how cooperative the model is today:

dedupe: refused as required -- duplicateOf 'm_99' does not exist in the pre-pass store
merge:  refused as required -- mergedText introduces proper nouns absent from every source: ['Kitchen']
promote: refused as required -- toBlock 'longterm' is not one of ('persona', 'human', 'episodes')
forget: refused as required -- category 'commitments' is protected by the retention policy
capacity enforcement dropped exactly the 4 unprotected facts
promotion enforcement moved m_6 scratch -> human

Changing the policy changes the outcome measurably: identical seeded facts, two caps, 2 vs 14 forgotten.

11. Bugs that the design surfaced

Worth writing down, because each one is a general trap in this class of system.

KeyError: 'refs' killed the whole Sleep run. Rejected mutations don't carry a refs dict, and the mutation-logging loop indexed it unconditionally. It only fired when a pass had a rejection and was not dry-running — a path the demo never hit until a merge was refused for spanning two blocks. Fix: .get("refs") or {"proposal": ...}, and the verifier now forces the rejection path with model-free gate cases.
The encoder was quadratic. Re-extracting the full transcript on every turn took the chat response for the memory-write event from ~15 s to ~200 s. Fix: an extracted_through cursor.
Apostrophes produced phantom tokens. normalize("user's") → user s, and s isn't a stopword, so perfectly good facts were refused for "inventing" a token. Fix: delete apostrophes rather than mapping them to spaces.
Inflection broke traceability. "only use Python" → a fact saying "uses Python" scored as invented. Fix: fold both sides with the same bounded stem_token, and keep stopwords in the source vocabulary while excluding them from the claim's content tokens.
A dry run that wrote a snapshot polluted the forgetting curve with a no-op point. Fix: a dry run writes nothing at all, and the assertion got stronger as a result.
Rejections vanished from the diff view. The range filter compared a mutation's snapshot to the range's endpoint only; a merge rejected mid-range never appeared. Fix: lo < seq <= hi.
Misleading rejection text. An item dropped earlier in the same pass was reported as "does not exist in the pre-pass store" — false, and exactly the kind of audit-log lie that makes an auditable system unauditable. Fix: _lookup() distinguishes never existed from consumed earlier in this pass.
The quiz failed on a synonym. Store: …sending emails. Agent: …sending messages. Coverage 0.875 → a strict-but-flaky failure. Fix: rank merged items out of the question pool and allow one stricter verbatim re-ask, recorded as attempts: 2 in the report — visible, not hidden.
Local-model ghosting. Killing a client mid-request does not cancel LM Studio's generation; the orphan kept the queue, and the next request looked like a 15-minute hang with an idle server. Diagnosis saved: if a local run stalls with no CPU on either side, suspect an abandoned request.

12. Trade-offs I'd argue about

Containment similarity treats subset restatements as duplicates. Right for memory collapse, wrong if you consider "lives in Lisbon" and "lives in Lisbon with a view" different facts.
0.9 is strict for short sentences: one swapped word in an 8-token fact scores 0.875 and is refused. I kept the spec's threshold and let those pairs surface as rejections rather than quietly lowering the bar — the Merge pass is where they legitimately belong.
No embeddings. Intentional: similarity must be explicable in a rejection message. A cosine threshold you can't narrate is a guardrail in name only.
The answer key is token overlap, so a genuinely correct answer phrased differently fails. That is a cost I accept for a test that can't be satisfied by confident nonsense.
Small model, slow loop. A 4B model makes a full Sleep ~2–4 minutes. The UI is honest about it with live per-pass counts, which turns out to be a feature: watching a pass stall for 90 s is information about your provider.

13. Run it, or fork it

git clone https://github.com/<you>/nocturne && cd nocturne
uv venv --python 3.12 .venv && uv pip install -r requirements.txt
cp .env.example .env            # OPENAI_BASE_URL / OPENAI_API_KEY / MODEL
npm install --prefix client
npm run dev                      # API :3001, client :5173
npm run verify                   # 18 checks; ~13 min on a local model, exit 0 on success

Stack: FastAPI 0.142 + Uvicorn + httpx on Python 3.12; SQLite (WAL) with no ORM; React 18.3 +
TypeScript 5.9 (strict) + Vite 5.4 + Tailwind 3.4, with the forgetting curve as hand-written SVG —
no chart dependency. The whole client bundle is 190 KB (59.6 KB gzipped).

If you want to extend it, the highest-value next pieces are a contradiction pass (conflicting
facts should supersede with a validity interval, not coexist), a lineage view (the ancestry data
already lives in mutations.refs), and a cross-model consolidation benchmark — the verifier
already accepts --provider/--model, so "which model edits memory most honestly" is one loop away.

The repo's README.md documents every gate, rejection rule and measure in full, plus the verifier
output. The most instructive file is agent/memory/textmath.py: 136 lines that decide whether your
agent's memory is auditable or vibes.

Code & more: https://www.dailybuild.xyz/project/272-nocturne

Luthier: an agent that audits an AI coding agent's harness against the repo it governs.

Harish Kotra (he/him) — Fri, 02 Oct 2026 18:13:25 +0000

Your agent's `CLAUDE.md` is lying to it, and nothing is checking

Every team I talk to has the same artefact now: a CLAUDE.md, an AGENTS.md, a .cursorrules, a folder of skills, an .mcp.json with four servers in it. It is written in a burst of optimism, and then the code moves.

Three months later:

the file it tells the agent to edit was renamed;
the command it tells the agent to run is not a script anyone declares any more;
one of the MCP servers points at a binary nobody has installed since the laptop was rebuilt;
a skill directory exists on disk that no instruction file has ever mentioned, so the agent cannot know it is allowed to use it;
and the convention everyone swears by — no console.log in committed code — is violated on line 4 of the entry point.

None of that throws. Your tests pass. CI is green. The only symptom is that the agent takes forty minutes to do a four-minute task, edits a file that does not exist, and tells you confidently that it "found the relevant module".

We check code into review, CI, a test suite, a linter and a type-checker. The harness — the file
that decides what the agent believes about the codebase — is checked into git and then never
verified again. Luthier is a checker for it.

The idea in one paragraph

Extract every claim the harness makes. Verify each claim against what is actually on disk. Report
the delta, with the rule quoted at file:line, the contradicting reality, and a unified diff you
can apply. Score the result 0–100 so it is comparable between runs and repos.

The claims are the unit of work, so the first job is a rule inventory.

{"rules":[
  {"id":"r1","source":"CLAUDE.md","line":14,
   "kind":"path|command|skill|mcp|convention","text":"<verbatim rule>",
   "check":{"type":"path_exists|command_runs|file_referenced|server_reachable|semantic",
            "target":"src/foo.ts"}}]}

Each rule names the check that can falsify it. Four of the five checks are deterministic — no LLM,
no network, no execution. The fifth is LLM-judged and degrades honestly when there is no key.

The architecture that keeps it honest

paste repo path (or drop a zip)
  |
  +-- discover harness ....... read_harness / inspect_repo   -> audit.log
  +-- extract rules .......... LLM (validated) + static fallback
  +-- check 1 path_exists .... glob against the real tree
  +-- check 2 command_runs ... declared in package.json / Makefile / pyproject  [never executed]
  +-- check 3 file_referenced  skills: declared -> used, on disk -> listed
  +-- check 4 server_reachable MCP schema + binary resolution  [never spawned]
  +-- check 5 semantic ....... mechanical subset offline, then LLM verdicts
  +-- score .................. 100 - sum(min(category_penalty, 32))
  +-- SSE progress -> runs/<id>.json -> scorecard / findings / evidence / fix diff

The design constraint that shapes everything: the auditor may only assert what it has read.
There are exactly two ways to learn about the repo, and both are journaled.

@tool
def read_harness(paths: list[str]) -> dict:
    """Read harness/instruction files (CLAUDE.md, AGENTS.md, .cursorrules, skills,
    MCP config, README) and return their real contents with 1-based line numbers.
    ...
    """
    return bundle.call("read_harness", {"paths": list(paths)})

@tool
def inspect_repo(globs: list[str]) -> dict:
    """Ground truth about the repository: which files match each glob, their sizes,
    and the first 20 lines of each match. Call this before claiming anything about
    what the repo contains. Output is byte-capped and says so when truncated.
    """
    return bundle.call("inspect_repo", {"globs": list(globs)})

bundle.call is the interesting part: it runs the tool, measures the duration and the byte size of
the payload, appends a JSONL record to audit.log, and keeps the result in an in-memory journal.
Every finding carries a journalRef — inspect_repo#4, grep#15 — pointing at the row in the log
that justifies it. A score is not an opinion here; it is a queryable trail.

Contents come back as lineno|text rows rather than raw text. That is a deliberate prompt-engineering
choice: models miscount lines, and if you make them count, they will confidently cite line 27 of a
12-line file. Hand them the numbering and the citation rate goes to ~100%.

The five checks, and what each one actually proves

1. `path_exists`

Pull path-shaped tokens out of harness text and glob them against the real tree. The interesting
work is in the filter, because prose is full of things that look like paths:

def looks_like_path(token: str, root_entries: set[str]) -> bool:
    raw = (token or "").strip().strip("`")
    if not raw or raw.startswith(("~/", "$", "%", "(", "@", "#")):
        return False
    parts = PurePosixPath(raw.replace("\\", "/").rstrip("/")).parts
    if not parts or any(_is_placeholder(p) for p in parts):
        return False
    head = parts[0]
    if "." in head and head.rsplit(".", 1)[-1].lower() in URL_TLDS:
        return False                       # example.com/api is not a repo path
    if head.upper() == head and len(head) > 2 and "." not in head:
        return False                       # OPENAI_API_KEY is not a directory
    ...
    return head.lower() in COMMON_TOP_DIRS or head in root_entries

Accept a token when it has a known extension, or when its first segment is a directory that
actually exists at the repo root, or a conventional one (src, docs, scripts, …). Placeholders
(<id>, {name}, path/to/…) are rejected — a harness that says path/to/secret.json is not
making a falsifiable claim.

Globs are claims too: src/**/*.tsx that matches nothing is drift, exactly as a missing file is.

2. `command_runs` — declared, never executed

This is the check people react to, because the naive version is a security incident: you do not
run the commands you are auditing.

if base in {"npm", "pnpm", "yarn", "bun"}:
    ...
    if sub in {"run", "run-script", "rs"}:
        script = tokens[2]
    elif sub in NO_DECL_SUBCOMMANDS:          # install, add, exec, doctor, ...
        return CommandVerdict(True)
    else:
        script = sub                          # `pnpm build` == `pnpm run build`
    if manifests.declares_script(script):
        return CommandVerdict(True)
    close = difflib.get_close_matches(script, sorted(manifests.scripts), n=1, cutoff=0.7)
    return CommandVerdict(False, "high",
        f"Script `{script}` is not declared in package.json", ...,
        replace=(script, close[0]) if close else None)

Everything is a lookup against declarations the repo already makes: package.json scripts (across
workspaces), Makefile targets parsed with a line regex, just recipes, [project.scripts] entry
points from pyproject.toml via tomllib, and requirements*.txt / dependency tables for bare
binaries. A binary "resolves" if it is on PATH (augmented with node_modules/.bin, .venv/bin,
~/.local/bin, Homebrew dirs) or declared by a manifest — because a documented tool you simply
have not installed yet is not harness drift.

3. `file_referenced` — skills go both ways

Two independent failures hide in a skills folder:

declared but unused: the harness names a skill, and the only mention of it anywhere in the repo is the harness line that declares it. Nothing outside the skill's own directory references it. That is a dead capability with a confident description.
on disk but unlisted: .hermes/skills/orphan-skill/ with a real SKILL.md inside, and no instruction file that mentions it. The agent cannot use what it is never told about.

The first is why the exclusion predicate matters — you must exclude the skill's own files and the
declaration line, or the check passes vacuously on every skill.

4. `server_reachable` — schema plus binary, no spawning

.mcp.json, claude_desktop_config.json, .cursor/mcp.json, .vscode/mcp.json and opencode.json
all hold server maps under slightly different keys (mcpServers, servers, mcp). Validate the
shape — command is a non-empty string, args a list of strings, env an object — then resolve
the binary. Never start the server.

The diff for a dead server is the most satisfying piece of the codebase. Reparsing the JSON and
re-dumping it would reformat the whole file; instead a small brace-matching scanner finds the exact
line span of the member and deletes just that block, then re-parses the result and refuses to
emit the diff if the JSON would break:

patched = "\n".join(remaining) + "\n"
try:
    json.loads(patched)
except ValueError:
    return None
return make_diff(path, text, patched)

If the dead binary has a near neighbour on PATH (node21 → node), the patch rewrites the
command instead of deleting the server.

5. `semantic` — and `unknown` as a real answer

Conventions are the hard part, so they get two engines. A mechanical subset is checkable without a
model: never use X, use A instead of B, every unit must contain file.ext. Grep the
token through source files, skipping comments and docs — a README that quotes an anti-pattern to
tell you not to write it is not a violation.

Everything else goes to the model, with a strict contract:

{"verdicts":[{"id":"r7","verdict":"holds|violated|unknown","evidence":"<path:line + quote>"}]}

…and then the answer is audited too. If a violated verdict cites a path that does not exist in the
repo, it is downgraded to unknown and the unverified citation is kept on the finding. unknown
costs 1 point and is always displayed. A tool that hides its uncertainty is a tool that will
eventually hide a violation.

No key configured? The section reports skipped: no_api_key — as a row in the findings table, not a
silent gap. Checks 1–4 still run, which is where most real drift lives anyway.

Scoring you can defend

BASE_SCORE                    = 100
PENALTY_DETERMINISTIC_FAILURE = 8
PENALTY_SEMANTIC_VIOLATION    = 5
PENALTY_SEMANTIC_UNKNOWN      = 1
MAX_PENALTY_PER_CATEGORY      = 32
CATEGORIES = ("paths", "commands", "skills", "mcp", "conventions")

score = max(0, 100 - sum(min(category_penalty, 32) for category in categories))

The per-category cap exists because a 500-path repo with one renamed directory should not score 0.
Four deterministic failures saturate a category, and the UI says capped when it happens.

The constants are served by GET /api/scoring and rendered in an expandable "how this score was
computed" panel, so nobody has to trust the number.

The two bugs that actually taught me something

lstrip("./") is not "strip a leading ./". It strips any leading character in the set
{'.', '/'} — so the pattern .mcp.json silently became mcp.json, and every dotted harness file
(.mcp.json, .cursorrules, .hermes/skills/**, .claude/…) was invisible to discovery. The
symptom was a clean-looking audit of a repo with a broken MCP config. The fix is a loop:

while norm.startswith("./"):
    norm = norm[2:]

Worse than the MCP miss: inspect_repo(".eslintrc.json") returned "does not exist", so the auditor
would have confidently reported a false finding. A byte-cap or a regex bug is visible; a
path-normalisation bug manufactures evidence. That is the one class of bug this project cannot
afford.

A diff is a claim too. My first stale-path patch deleted the whole harness line. For
- Edit \src/old.tswhen the CLI output changes. that is right. For
Read \README.mdfirst, then \docs/architecture.md. it deletes a valid instruction along with
the stale one. So: if the line carries other checkable claims, excise only the stale token — which
means tidying the residue (… first, then . → … first.) with a small chain of regexes, and
falling back to deleting the line when nothing checkable survives.

And a rename is only proposed when the file name is unchanged (a move) or the two paths are ≥0.86
similar. Anything looser goes in the suggestion field, not in a diff. src/cli.ts → src/util.ts
is a plausible-looking patch that would be wrong, and a wrong patch is worse than no patch.

Precision: the audit of the audit

Luthier's first run against a real repo scored 42 and flagged src/old.ts — a path that repo's
README quotes as an example of drift. Technically a true observation; practically noise, and
noise in a score is how you lose people's trust in one edit.

Two rules fixed most of it:

A file the docs say the app writes is not a broken reference. "stored in run/settings.local.json (mode 0600)" describes run-time behaviour. Advisory, 0 points.
Prose docs naming a path inside a directory this repo does not have are probably examples. README.md quoting src/foo.ts in a repo with no src/ at all → advisory. But CLAUDE.md quoting it → still a hard failure, because an instruction file is what the agent loads and obeys.

That distinction — instruction files are charged, docs are surfaced — is the whole difference
between a score people act on and a lint report nobody reads. The same repo went 42 → 76 with the
genuinely real findings still listed.

Verification

npm run verify     # python -m agent.verify

Two throwaway git repos are written to a temp dir. One has five kinds of seeded drift; every one
must be flagged at the correct file:line. The other must score ≥95 with zero deterministic
failures — the guard against a checker that flags everything, which is easy to write and worthless.
Then every generated diff is run through git apply --check, every citation must be a real line in
a real file, and audit.log must be valid JSONL with tool, args, duration and bytes on every row.

drifted fixture  score  47  findings  9  rules  9  tool calls 15
clean fixture    score 100  findings  0  rules  4

30 passed, 0 failed
PASS

The Strands agent loop gets its own check (scripts/agent_loop_check.py) against a local mock
OpenAI-compatible endpoint, asserting the agent's tool call really travelled through the journaled
reader and that the API key never appears in a response body. It proves the integration, not the
model's judgement — and it is honest about that distinction, because a claim about the second kind
would be exactly the thing this tool exists to catch.

What it's for

Not lint. Lint checks that code satisfies a style. Luthier checks that instructions satisfy
reality — a different axis, and the one nobody automated yet.

The uncomfortable framing: a harness is a prompt, a prompt is a dependency, and we have no
dependency checker for prompts. We have this instead — a scorecard that says, in one number and a
table of quotes, how much of what you tell your agent is still true.

Run it against your repo. If the number is high, the interesting question is whether your harness
has been edited recently enough to have had the chance to rot.

Stack: Strands Agents 1.57 (agent loop, @tool), FastAPI + uvicorn, httpx for plain OpenAI-compatible chat calls, Vite + React 18 + TypeScript + Tailwind, difflib for the patches, git apply --check as the diff gate. Python 3.11+, no network from the checks, no writes to the
audited repo.

Code & more: https://www.dailybuild.xyz/project/271-luthier

Building Clay: an A2UI host where the model emits interface instead of prose

Harish Kotra (he/him) — Thu, 01 Oct 2026 16:58:45 +0000

Technical notes from building an agent that renders live React widgets, keeps per-session state in a Durable Object, and refuses any UI it doesn't recognize.

The problem with chat

Every agent product today has the same output medium: a paragraph. Ask for a trip budget and you get a wall of text containing numbers you cannot manipulate. Ask what happens if the hotel costs more and you re-prompt, and hope it recomputes consistently.

The interesting alternative is to let the model emit interface — a declarative description of widgets bound to a data model — and have the host render it as real controls. That's the A2UI idea: agent-to-UI. The model never sends code. It sends JSON that names components from a catalog the host already owns.

Clay is a complete implementation of that loop:

prompt ──► model ──► A2UI message ──► React renderer ──► live widgets
                                              │
                                     widget event
                                              ▼
                          agent recomputes dependents ──► patch

The whole app is three panes: a chat rail, a generated surface, and an inspector showing the exact messages, because in this design the protocol is the product.

Why "declarative, not code" is the actual security model

The tempting shortcut is to let the model emit HTML or JSX and inject it. Then your sandbox is your own diligence, and one bad <script> is a bad day.

With a component catalog, the model's vocabulary is ten nouns:

Column Row Text Slider Toggle Table BarChart Stat Badge Button

Anything else cannot render, by construction. There is no dangerouslySetInnerHTML anywhere in the client, no eval, no dynamic import. Props arrive as data and get spread into components the bundle already contains. The model can ask for a Slider; it cannot ask for a <script>.

That turns "is the output safe?" from a scanning problem into a type problem.

The contract

One JSON object per turn, four allowed top-level keys:

{
  "surfaceUpdate": {
    "surfaceId": "s1",
    "components": [
      { "id": "root",   "component": { "Column": { "children": ["hdr", "budget", "table", "chart"] } } },
      { "id": "hdr",    "component": { "Text":   { "text": "5-day trip plan", "variant": "h2" } } },
      { "id": "budget", "component": { "Slider": { "label": "Budget", "min": 10000, "max": 80000,
                                                   "step": 1000, "value": 40000, "bind": "budget" } } },
      { "id": "table",  "component": { "Table":  { "columns": [ … ], "rows": [ … ] } } },
      { "id": "chart",  "component": { "BarChart": { "series": [{ "label": "Day 1", "value": 7200 }],
                                                     "unit": "INR" } } }
    ]
  },
  "dataModelUpdate": { "surfaceId": "s1", "dataModel": { "budget": 40000, "days": [ … ] } },
  "text": "one short sentence"
}

Two details do most of the work.

The tree is flat and id-linked. Column.children holds ids, not nested objects. Flat lists are far easier for a language model to emit correctly than deeply nested JSX-shaped trees, and patching a node is a replace-by-id.

bind is a promise. A widget's value must equal the value stored at its dotted path in the data model. That single invariant is what makes the surface live rather than decorative: the slider isn't a picture of a number, it's a handle on the model.

Architecture

                ┌──────────────────────────────────────────────────────────┐
                │  BROWSER                                                 │
                │  App.tsx ─► useSurface() ─► SurfaceRenderer.tsx           │
                │              │ state, phases,   resolve via registry.ts  │
                │              │ optimistic bind   ▼                        │
                │              │ writes            Slider Table BarChart …  │
                └──────────────┬───────────────────────────────────────────┘
                               │ /api/*  (Vite dev proxy)
          ┌────────────────────┴────────────────────┐
          ▼                                         ▼
┌──────────────────────────┐            ┌──────────────────────────┐
│ RUNTIME=workers  :8787   │            │ RUNTIME=node   :3001     │
│ agent/index.ts           │            │ server-node/index.ts     │
│  Worker fetch()          │            │  node:http               │
│   session id → DO stub   │            │   session id → Map       │
│  ClayAgent               │            │                          │
│  this.state / setState() │            │                          │
│  SQLite per session      │            │                          │
└────────────┬─────────────┘            └────────────┬─────────────┘
             └─────────────────┬─────────────────────┘
                               ▼
                  agent/turn.ts  (shared turn engine)
             build messages → call model → classify reply
                  → apply to surface → measure dependents
                               ▼
                  agent/inference.ts (plain fetch)
             POST {baseUrl}/chat/completions · stream:false
                    response_format:{type:"json_object"}

Both runtimes import the same turn engine, prompt and guardrail. The fallback reimplements exactly one thing: who owns state. That's the point of the split — RUNTIME=node proves the Durable Object isn't load-bearing for the protocol, only for persistence.

The Cloudflare Agents SDK, from the actual API surface

The SDK's state API is the piece that changes between releases, so the installed package's type declarations were the ground truth. What agents@0.24.0 gives you:

import { Agent } from 'agents';

type SessionState = {
  sessionId: string; surfaceId: string;
  components: A2UIComponent[]; dataModel: Record<string, unknown>;
  turns: TurnRecord[]; updatedAt: number;
};

export class ClayAgent extends Agent<ClayEnv, SessionState> {
  initialState: SessionState = { /* … */ };

  async onRequest(request: Request): Promise<Response> {
    const path = new URL(request.url).pathname;
    if (path === '/api/chat')     return this.handleChat(request);
    if (path === '/api/interact') return this.handleInteract(request);
    return this.json({ error: 'not_found', detail: path }, 404);
  }
}

Five things worth knowing before you write against it:

Implement onRequest, don't override fetch. The base class's fetch() resolves dynamic sub-agent paths and then delegates to your hooks. Overriding it skips that machinery.
setState() is a full replace, synchronous, returns void. Forgetting {...this.state} silently wipes fields. There is no merge on the object (only on workflow steps), so every write spreads.
SQLite storage is enabled by the migration, not a binding flag. new_sqlite_classes in wrangler.jsonc — there is no durable_objects.bindings[].sqlite key:

   {
     "compatibility_flags": ["nodejs_compat"],
     "durable_objects": { "bindings": [{ "name": "CLAY_AGENT", "class_name": "ClayAgent" }] },
     "migrations": [{ "tag": "v1", "new_sqlite_classes": ["ClayAgent"] }]
   }

After a few sessions the evidence is on disk: one .sqlite file per session,
under .wrangler/state/v3/do/clay-agent-ClayAgent/.

wrangler dev does not promote process env into env. .env has to be parsed and passed explicitly, which is what scripts/run-agent.mjs does:

   for (const [key, value] of Object.entries(bindings(envFile.values))) {
     args.push('--var', `${key}:${value}`);
   }

Same script pins XDG_{CONFIG,CACHE,DATA}_HOME into the project so local state never escapes into your home directory.

There is no sessionId concept for plain agents. The Durable Object's name is your session id. You mint it, you put it in the URL or a header, and namespace.get(namespace.idFromName(sessionId)) routes to it.

The Worker entry owns the public paths and forwards the body once:

// a Request stream cannot be consumed twice — read text, parse, forward the same text
const text = await request.text();
const body = text ? JSON.parse(text) : {};
const sessionId = clientSessionId(request, body) ?? newSessionId();
const upstream = await stubFor(env, sessionId).fetch(
  new Request(url.toString(), { method: request.method, headers: request.headers, body: text })
);

(That line about streams is not a footnote. My first version called
request.clone().json() and then .text(), and every /api/chat returned a 500 with Body has already been used.)

Deliberately unused: the SDK's WebSocket transport, agents/react, AI SDK chat helpers, MCP. Updates ride on the HTTP response, which keeps the protocol inspectable — and inspectable is the whole demo.

The hard part isn't rendering, it's the guardrail taxonomy

Validating the model's reply with zod is easy. Deciding what to do about an invalid reply is where the design lives.

A small model fumbles JSON constantly. Observed failure modes, all real:

truncated output mid-object
{"key, ": "total"} — a stray comma inside a quoted key
a component entry with no id at all
two components in one component body
a Slider whose value sits outside its own min/max
bind: "trip.travellers" while the data model wrote travellers flat

The naive answer is "retry everything." But then a model that invents
HologramMap gets retried until it behaves, and your unsupported-component path never fires — which is one of the things the app is supposed to demonstrate.

So the split is by kind of failure, not by severity:

Kind	Examples	Response
emission defect	truncated JSON, missing `id`, unrecognized key, out-of-range slider value, two components in one body	re-ask, up to 3 attempts, feeding the parser's complaint and the model's own previous answer back
contract violation	unknown component name, forbidden prop, injection pattern, duplicate ids, dangling `children`, two roots	refuse, final — `invalid_surface`, never retried

The discriminator is the zod issue code:

const result = a2uiResponseSchema.safeParse(parsed);
if (!result.success) {
  const issue = result.error.issues[0];
  // A prop the schema does not know, a missing prop, or a component body that
  // matches no variant is the model fumbling JSON — an emission defect, so the
  // loop asks again. Anything reported by superRefine (duplicate id, dangling
  // child, two roots, a bind with no data behind it) is a contract violation
  // and stays final.
  const defectCodes = new Set(['unrecognized_keys', 'invalid_type', 'invalid_union']);
  return {
    kind: defectCodes.has(String(issue?.code)) ? 'malformed' : 'rejected',
    detail: issuePath(result.error),
  };
}

And the re-ask, which never touches the payload:

conversation = [
  ...conversation,
  { role: 'assistant', content: raw.slice(0, 12_000) },
  { role: 'user', content: `That reply was rejected by the A2UI parser: ${emission.detail}. ` +
    `Re-emit the entire A2UI JSON object. Every entry in "components" must be an object with ` +
    `BOTH "id" (a non-empty string) and "component" (exactly one catalogued name with its props). ` +
    `Do not truncate, do not elide, do not explain. JSON only.` },
];

The rule that makes this honest: the agent loop may ask again; it may not patch. A refusal returns the model's own text (raw, first 300 chars) and the exact zod path, and the previous surface stays on screen. The inspector shows what was refused and why. Fixing the model's JSON locally would be a repair, and a repaired payload is no longer evidence of anything.

The catalog itself is one object, shared/catalog.ts, read three ways:
serialized into the system prompt, wrapped in zod schemas server-side, and
imported as the client's render allowlist. They cannot drift.

Making a drag actually change other widgets

This is the acceptance test that separates a live surface from a picture of one. Move a slider and the table, chart and stat cards must show new arithmetic.

Sequence:

you drag Slider "stay" 2500 → 6000
  ├─ client: setPath(dataModel, "stay", 6000)      optimistic, thumb tracks instantly
  ├─ phase → "patching"
  ▼
POST /api/interact { surfaceId, componentId, value }
  ├─ server writes bind into the dataModel BEFORE asking
  │    (the model's job is downstream arithmetic, not re-deriving what the slider says)
  ├─ prompt carries: the event, the interactive widgets, the full current dataModel
  ├─ model returns the FULL component list + dataModelUpdate
  ├─ recomputedIds(previous, next, touchedId)
  ├─ if only the touched widget moved → ask once more, pointing at its own reply
  ▼
200 { recomputed: ["chart","statTotal","table","statRemain", …] }

Two decisions matter.

Seed the bound value server-side. Asking the model to both apply the event and recompute everything is how you get half-updated surfaces. Applying bind is protocol semantics, not guessing at UI; the model's job becomes purely the arithmetic downstream.

Measure the right numbers. "Did dependents change?" needs a definition that can't be gamed:

// shared/a2ui.ts — numeric leaves, ignoring the control's own range
export function numericFields(component: A2UIComponent): Record<string, number> {
  const flat = flattenProps(componentProps(component));
  const out: Record<string, number> = {};
  for (const [key, value] of Object.entries(flat)) {
    // min/max/step describe the control, not a computed output.
    if (/(^|\.)(min|max|step|maxValue)$/.test(key)) continue;
    if (typeof value === 'number' && Number.isFinite(value)) out[key] = value;
  }
  return out;
}

Without that exclusion, a slider drag "changes" the moved widget's own value and a build that recomputes nothing still passes. The comparison also excludes the touched component id entirely.

Finally, syncBoundValues() re-writes every interactive widget's value from the data model after a patch lands. That's not repair either — it's the bind invariant enforced, so a stale echo in the model's reply can't rewind what the user just dragged.

In a real browser, one drag of Hotel per night ₹2,500 → ₹6,000 moved the trip total (₹28,000 → ₹38,500), the remaining budget (₹12,000 → ₹1,500), the per-day average (₹7,700), the day table, the chart and the status badge — recomputed: badgeStatus, dayChart, dayTable, note, statPerDay, statRemaining, statTotal. The client computed none of those numbers.

Inference: one boring code path

const response = await fetchImpl(`${settings.baseUrl}/chat/completions`, {
  method: 'POST',
  headers: { 'content-type': 'application/json', authorization: `Bearer ${settings.apiKey}` },
  body: JSON.stringify({
    model: settings.model, messages, stream: false,
    temperature: settings.temperature, max_tokens: settings.maxTokens,
    response_format: { type: 'json_object' },
  }),
  signal: AbortSignal.timeout(90_000),
});

Particle.ai, LM Studio, Ollama and Gemini's OpenAI-compatible endpoint all speak this. Presets are data in the client; Custom covers everything else. Switching providers is a Settings click, not an edit.

Three details that bit:

max_tokens needs headroom. Reasoning models spend output budget on hidden thinking before writing any JSON. 8192 truncated surfaces regularly; 32768 doesn't. A truncated surface is not a small failure — it's an unusable turn.
The key never goes to the browser. Settings stores it in localStorage and sends it to the server; nothing echoes it back, and /api/test redacts it from upstream error bodies before returning them.
Fence tolerance is parsing, not repair. Some models wrap JSON in

```json

. Extracting the object before validation is fine; the payload is still validated strictly.

The renderer, and one type trick that prevents drift

// client/src/renderer/registry.ts
export const registry: Record<ComponentName, ComponentType<WidgetProps>> = {
  Column, Row, Text, Slider, Toggle, Table, BarChart, Stat, Badge, Button,
};

export function resolveComponent(name: string): Resolved {
  const decision = decide(name);                       // pure allowlist check
  if (decision.kind === 'unsupported') return { kind: 'unsupported', name };
  const Widget = registry[name as ComponentName];
  return Widget ? { kind: 'widget', name, Widget } : { kind: 'unsupported', name };
}

Record<ComponentName, …> is exhaustive: add a name to the catalog and the client fails to compile until a widget exists for it. The allowlist can't silently lag the server's.

An off-catalog name renders a red card and is logged:

UNSUPPORTED COMPONENT: HologramMap
id map · refused by the client allowlist · logged to the inspector

The renderer walks the tree from the single unreferenced root, guards depth, and renders missing children as a placeholder rather than throwing. allowlist.ts is a separate pure module specifically so the verifier can execute the client's decision in Node — testing the real function instead of re-implementing it in the test.

Verification you can't satisfy by cheating

npm run verify runs 11 checks over the four that matter, against a live model. The interesting part is what each one refuses to accept:

Different prompts → different trees. Compares sorted multisets of component types, not response codes. A 200 OK with a canned tree passes the naive version and fails this.
Same prompt twice → different tree, identical schema. Proves generation rather than a template, while asserting every component is catalogued.
A drag changes ≥2 other components' numbers. Uses the numeric-leaf definition above; ignoring min/max/step and the touched id.
Bogus components are refused twice over — the client allowlist function refuses the name, the server guardrail refuses an off-catalog payload, and an engineered prompt tests the live path. Then /api/health proves nothing crashed.
No renderable response contains <script, onclick or javascript: — with the diagnostic raw/detail fields stripped, because those deliberately echo a refused payload into the inspector, where React renders it as escaped text in a <pre>.

And the honest caveat, stated in the README too: this suite is model-dependent and will occasionally fail. A run earlier today failed check 2 because the model put two components in one body during an interact turn. The verifier retries refused turns the way the app's retry button does, but if you weaken a guardrail to make the suite green you've deleted the thing you were testing.

npm run verify:screenshots drives the same assertions in headless Chrome over CDP — send a prompt, wait for the turn, drag a slider, read displayed numbers out of the DOM before and after, then fire the injection attempt at the running app and assert nothing markup-shaped reached [data-surface-root].

Things that broke, so you don't have to

Symptom	Cause	Fix
`Body has already been used` on every POST	`request.clone().json()` then `.text()` in the Worker	read `text()` once, parse it, forward the same string
`ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX: parameter property`	Node's strip-only mode rejects `constructor(readonly code: string)`	declare fields explicitly in the class
`Cannot find module '…/shared/catalog'`	type-stripping needs explicit `.ts` specifiers	import `'../../shared/catalog.ts'` in server-side modules
npm ERESOLVE on install	`agents` declares an optional peer on `react@^19`; the client is React 18	`.npmrc` with `legacy-peer-deps=true`, and declare the non-optional peers (`@modelcontextprotocol/*`, `zod`) explicitly
Verifier screenshots showed the wrong session	CDP script attached to a Chrome already listening on the shared debug port — someone else's session	allocate an exclusive free port per run
A drag "changed 12 numbers" that meant nothing	DOM probe compared `.tabular-nums` by index, including the dragged widget's own readout	tag each number with whether it lives inside a slider card and exclude those
Inspector tab click silently did nothing	labels are lowercase in the DOM with CSS `capitalize`	match case-insensitively
Interact turns refused ~30% of the time	slider out-of-range and stray-key slips classified as contract violations	reclassify emission defects as retryable

What I'd build next

Per-component patches (patchComponents / removeComponents) so a drag sends three components instead of twenty-three.
Streaming surfaces — assemble the tree as the model emits it.
Server-checked arithmetic: let the model attach a formula to a derived component and have the host evaluate it, so a wrong total becomes a visible failure instead of a plausible-looking one.
TextField / Select / DateRange — the catalog's biggest gap, and the four-file checklist makes adding one mechanical.
Turn scrubber — /api/session/:id already returns the whole turn log, so replaying turn n reconstructs any earlier surface for free.

Try it

npm install && cp .env.example .env && npm run dev

Ask for a trip plan, drag a cost slider, and watch the inspector's diff tab list which component ids the agent decided to recompute. Then ask it to put a <script> in a Text prop and read the refusal. Both paths are the app.

Code & more: https://www.dailybuild.xyz/project/270-clay

One UI, Any Agent: Proving Framework Independence with AG-UI, Mastra, and React

Harish Kotra (he/him) — Wed, 30 Sep 2026 18:37:19 +0000

How I built a single React client that renders a Mastra agent and a hand-rolled emitter with zero client changes — and the cancellation bug that almost broke the demo.

The problem with agent demos

Almost every agent-framework demo hard-wires the UI to one framework. The
frontend knows about Mastra's (or LangChain's, or the Vercel AI SDK's) response
shapes, so swapping the agent means rewriting the client. That defeats the
purpose of having a protocol.

AG-UI (Agent-User Interaction Protocol) fixes this by
defining a standard event contract between agents and frontends: RUN_STARTED,
TEXT_MESSAGE_CONTENT, TOOL_CALL_START/ARGS/END, STATE_DELTA, RUN_FINISHED,
RUN_ERROR. If every backend speaks AG-UI, the frontend only needs to speak
AG-UI too — and becomes framework-independent.

This post walks through a project that proves it: one Vite + React client,
two backends (a real Mastra agent and a ~50-line hand-rolled emitter), one
unchanged component tree. Along the way I'll show the exact code, the
architecture, and the gnarliest bug I hit — server-side cancellation that
silently never fired.

Architecture: protocol over transport

┌─────────────────────────────┐      POST /api/run/:backend (SSE)      ┌──────────────────────────────┐
│  CLIENT :5173               │  ───────────────────────────────────▶  │  SERVER :3101                │
│  App.tsx → useAgui → agui   │                                        │  /mastra ──┐                 │
│  (tabs, SSE reader, reducer)│  ◀───────────────────────────────────  │  /minimal ─┴─▶ runPipeline() │
└─────────────────────────────┘      data: {type: RUN_STARTED…}        └──────────────┬───────────────┘
                                                                                     │ fetch (streaming)
                                                                              ┌──────▼──────┐
                                                                              │ Ollama / LM │
                                                                              │ Studio / …  │
                                                                              └─────────────┘

The invariant that makes the whole demo honest: both routes funnel through a
single runPipeline() function. The Mastra route constructs a genuine
Agent — name, instructions, model — but the AG-UI byte sequence comes from the
shared pipeline. Conformance is structural, not aspirational.

The client is equally strict: a pure reducer folds the raw event log into the
view. There is no if (backend === "mastra") anywhere in client/src.

The shared pipeline

server/src/pipeline.ts is the heart of the project. Every run opens the same way:

const send = (e: Omit<AguiEvent, "runId" | "timestamp" | "backend">) =>
  sseSend(res, { ...e, runId, timestamp: now(), backend } as AguiEvent);

send({ type: "RUN_STARTED", message: frameworkNote } as any);
send({ type: "STATE_SNAPSHOT", snapshot: { task, backend, plan: [], results: [] } } as any);

const messageId = randomUUID();
send({ type: "TEXT_MESSAGE_START", messageId } as any);
for await (const chunk of streamLLMText(settings, task, isAborted)) {
  full += chunk;
  send({ type: "TEXT_MESSAGE_CONTENT", messageId, delta: chunk } as any);
}
send({ type: "TEXT_MESSAGE_END", messageId } as any);

streamLLMText() (server/src/llm.ts) calls any OpenAI-compatible
/chat/completions endpoint with stream: true and yields real token chunks.
If the provider doesn't support SSE, it falls back to a non-streaming call and
re-chunks the genuine response text — the UI still shows real model output
arriving incrementally, never canned strings.

Constraining the model — and refusing to hallucinate

The planning agent's system prompt (in shared/defaults.json, shared by both
backends) demands strict JSON:

{ "plan": ["step1", "step2", "step3"],
  "steps": [{ "tool": "<tool-name>", "args": {...} }, ...] }

Models disobey. The pipeline's guardrail: strip code fences, JSON.parse, verify
plan[]/steps[] — and if anything fails, show the parse error instead of
inventing a plan:

if (!doc) {
  send({ type: "STATE_DELTA",
         deltaState: [{ op: "add", path: "/parseError", value: parseError }] } as any);
  send({ type: "RUN_ERROR", code: "PLAN_PARSE_ERROR",
         message: `Model did not return valid plan JSON (${parseError}). ...` } as any);
  return;
}

Each validated step then becomes real TOOL_CALL_START → two chunked
TOOL_CALL_ARGS → TOOL_CALL_END (with a deterministically executed toy-tool result) → STATE_DELTA sequence, before the final RUN_FINISHED. Different tasks produce different plans, hence visibly different tool cards — acceptance criteria without a single fixture file.

The cancellation bug: Stop silently did nothing

My first implementation checked the abort flag between stream chunks. Testing against a stalled upstream revealed the flaw: when the LLM fetch() hangs, no chunks arrive, so the flag is never checked — Stop left the spinner running forever (well, until the client's own timeout).

The fix has two halves. Server-side, the upstream fetch gets an AbortController
driven by a 100 ms poll of the abort registry:

// server/src/llm.ts
const ctrl = new AbortController();
const poll = setInterval(() => { if (isAborted()) ctrl.abort(); }, 100);

and the pipeline maps the resulting AbortError to a first-class protocol event:

} catch (e: any) {
  send({ type: "TEXT_MESSAGE_END", messageId } as any);
  if (isAborted() || e?.name === "AbortError") {
    send({ type: "RUN_ERROR", code: "ABORTED",
           message: "Run cancelled by user (interrupt)." } as any);
  }
  ...
}

Client-side, Stop tells the server first and only aborts local transport as a backstop 800 ms later — so the genuine server event arrives down the still-open stream:

// client/src/useAgui.ts
await fetch("/api/run/abort", { method: "POST",
  body: JSON.stringify({ runId: activeRunId }) });
setTimeout(() => abortRef.current?.abort(), 800);

I verified this end-to-end against a black-hole HTTP server: the stream closed with TEXT_MESSAGE_END → RUN_ERROR code: "ABORTED", and the canvas rendered "aborted". The lesson generalizes: cancellation in streaming systems must be tested against a hung upstream, not just a slow one.

The client: one reducer to render them all

client/src/agui.ts holds reduceRun() — ~80 lines folding an event log into a
RunView (text + tool cards + state tree + status). useAgui.ts reads
SSE-over-POST frames (data: {...} split on \n\n), stamps each with a
wall-clock timestamp and inter-event latency, and feeds the log. panes.tsx
renders the two panes from that single state.

Conformance is a type-sequence diff ignoring ids and timestamps:

export function diffSequences(a: string[], b: string[]): string[] {
  const out: string[] = [];
  const n = Math.max(a.length, b.length);
  for (let i = 0; i < n; i++)
    if (a[i] !== b[i]) out.push(`[${i}] mastra=${a[i] ?? "∅"} minimal=${b[i] ?? "∅"}`);
  return out;
}

Empty diff → green badge. It's a small thing, but it turns "framework
independence" from a slide into a widget you can watch go green.

Configuration without code edits

The Settings panel (base URL / API key / model + Particle.ai, LM Studio, Ollama, Gemini presets, persisted to localStorage) flows into every run request body — the server holds no keys. Test connection does a live GET {baseURL}/models and prints the true status and body. Pointing the demo at Ollama (http://localhost:11434/v1, llama3.1, empty key) requires zero code changes, as the acceptance criteria demand.

What I'd add next

A third backend (LangChain, CrewAI) reusing runPipeline() — the badge should stay green with zero client changes.
Replay mode: persist event logs, re-render via reduceRun() with no server.
A latency waterfall from the already-recorded inter-event timings.
Conformance CI: boot the server, fire both backends at a fixture, fail on type-sequence divergence.

Takeaway

Protocols only matter if you prove the independence they promise. Two backends, one shared emitter, one pure client reducer, and a badge that stays red until the event streams actually match — that's the whole demo, and that's the point: build to the event contract, and the framework becomes an implementation detail.

Code & more: https://www.dailybuild.xyz/project/269-marionette

DEV Community: Harish Kotra (he/him)

I built the same agent twice to see what a framework actually deletes

The two files

What Strands Agents TypeScript 1.x actually looks like

The line counts, both ways

Fairness controls, because the comparison is worthless without them

What diverges, before I ran a single model

Two bugs worth reporting

Try it

What the stub caught before I spent a single token

Finding: parallel tool calls work on one path and not the other

Two agents, two companies, one protocol: building a visible agent-to-agent negotiation

The shape of it

Step 1: the negotiation is a schema, not a vibe

Step 2: serving an agent as an A2A endpoint

Step 3: calling a peer, and the context_id question

Step 4: the data plane is an MCP server, not a function

Step 5: where enforcement lives

Step 6: the observer is a log reader

Eight things that cost time

What it actually produces

Try it, or read the interesting files

Guardrails are a lie until they're arithmetic

1. The mechanic is four lines

Money-to-tokens, priced conservatively

2. The counter must not live in a variable

3. Where the framework actually earns its keep

Architecture

4. Reported and estimated numbers must never share a column

5. Approvals: a tier, a pause, and a clock

6. Four bugs that only a live run could find

a) The observed-throughput death spiral

b) Reading live state during a replay — the good one

c) Negotiating past a wall clock

d) 850 tokens for an empty string

7. Verification is the product

8. What I'd build next

Try it

Building a Rubik's Cube Decision Arena with 6 Decision Models (Laya, Clef, GLiNER 2.5, Kev and Strands Decider)

What happens when six small AI models — four of them running on a laptop — are asked to solve the same scrambled cube, one move at a time?

1. The premise: decision models vs. planning

2. The two hard rules

3. Architecture

4. The cube model

5. The solver bridge (and its encoding trap)

6. One contract, six engines

The six models

7. The guided loop: the solver plans, the model decides

Policy: how a distribution becomes a move

8. The server: threads, polling, and a chart race

9. The UI: a real 3D cube with no library

10. The scorecard

11. What we actually measured

12. Hard-won lessons

13. Run it, fork it, extend it

Screenshots

Watch

Building Anvil: code-as-action with a capability sandbox that explains its refusals.

The agent wrote import socket. Now what?

The pipeline

Layer 1 — the AST pre-check: text, but not just text

Layer 2 — audit hooks: policy at the moment of the operation

Layer 3 — the kernel: prove it, don't assume it

Layer 4 — capabilities: the model asks, the user decides

The smolagents part, and where the spec was wrong about the API

Verification is the product surface

Steal these ideas

How Arbiter works, why it was built, and the four measurements it made about an LLM judge that I had to look at twice.

Your eval harness trusts a witness it never cross-examined

The setup

One table is the whole design

Failures are rows, not defaults

The sign trap in position bias

The denominator trap

Padding has to be generated in code

Result 1: the judge is generous, not wrong

Result 2: position bias was absent — genuinely

Result 3: the one that changed how I think about this

War story: Pydantic AI cannot talk to the default provider

The verdict is code, and it is on screen

Step 3: calling a peer, and the `context_id` question

The agent wrote `import socket`. Now what?

Your agent's `CLAUDE.md` is lying to it, and nothing is checking

1. `path_exists`

2. `command_runs` — declared, never executed

3. `file_referenced` — skills go both ways

4. `server_reachable` — schema plus binary, no spawning

5. `semantic` — and `unknown` as a real answer