DEV Community: Madhesh Vivekanandan

The Coding Agent That Can't Grade Its Own Homework

Madhesh Vivekanandan — Wed, 09 Sep 2026 15:05:27 +0000

"All tests pass. ✅"

You've seen that message from a coding agent. And at least once, you've opened the file afterwards and found the tests were never run — or the test file was quietly edited until it agreed.

The problem isn't that the model writes bad code. It's that the same context window writes the code, reviews the code, and declares the code correct. One brain, grading its own homework — and it grades generously. That cycle — understand → change → check → repeat — is the agent loop, and every coding tool ships one that runs inside a single context window.

agent-loop is a small open-source project that restructures it: six explicit stages, file-based handoffs, an independent verifier, hard iteration caps. No framework, no daemon, no SDK — markdown plus one shell script, and it runs unchanged in Claude Code, Codex, Cursor, Gemini CLI, Copilot, and 30+ other tools.

The whole loop in 30 seconds

Each stage starts with a fresh context, reads only its declared inputs, and hands off through a markdown file in a gitignored .agent-loop/ directory:

.agent-loop/
  profile.md                    # cached project profile (commands, conventions)
  runs/2026-09-09-rate-limit/
    task.md
    analysis.md
    plan.md
    implementation.md
    test-report.md              # ends in PASS or FAIL — never "looks good"

Those files aren't logs. They're the interface between stages — and an audit trail you can read afterwards to see exactly why the agent did what it did.

Not every task deserves a pipeline

Anthropic measured multi-agent systems at roughly 15× the tokens of a regular chat. For a typo fix, that's a five-course tasting menu because you wanted a glass of water.

So the first move is triage — one cheap classification, never delegated:

Tier	The task	What runs
S	Diff fits in one sentence	implement → verify
M	Localized change	analyze → implement → verify
L	Multi-file or uncertain	the full pipeline

One rule does more work than it looks like: torn between two tiers, pick the smaller one. Left alone, agents will write a design document for a null check — the bias has to be encoded.

Artifacts, not transcripts

Why fresh contexts? Because context poisoning is real. Exploring a codebase fills the window with dead ends — irrelevant files, abandoned hypotheses, stale test logs — and in one long session, all that noise quietly influences every later decision.

File handoff cuts the cord. The analyzer can burn its whole budget exploring, but only analysis.md survives into the next stage. What crosses the boundary is a document, not a transcript.

One rule decides whether handoffs work at all:

The plan must carry rationale, not just conclusions. An implementer that gets only "use middleware, put it in upload.py" re-derives the reasoning, reaches a different answer, and contradicts the plan while believing it follows it. (Cognition's "Don't Build Multi-Agents" circles the same failure.)

The templates enforce it structurally: plan.md has required sections, and a plan without a named runnable check — the command that will decide pass or fail — is returned once, then the loop stops. No check, no run.

The verifier never sees the reasoning

This is the part of the design I'd defend in a fight. Verification starts in a completely fresh context, and its inputs are a closed list:

The verifier gets	It never gets
The plan	The implementer's summary
The diff	The implementer's reasoning
A shell, to run the check	`implementation.md`

Why so strict? An agent that reads "here's why my implementation is correct…" doesn't verify that claim — it confirms it. Sycophancy isn't a personality flaw you can prompt away; the fix is structural. The verifier learns what the code was supposed to do, sees what actually changed, runs the check named in the plan, and writes PASS or FAIL with command output as evidence. "Looks good to me" is not a verdict.

And my favorite detail in the whole project: you might think spawn the verifier without write tools and it can't cheat. But it must have a shell to run the tests — and a shell can edit files (sed, echo >, git apply…). So the real guarantee is a diff fingerprint: the working tree is fingerprinted before and after verification, and if it moved, the verdict is void. Tool restriction is defence in depth; detection is the guarantee — on every host, including those that can't restrict tools at all.

(Reviewers fail in the other direction too, so the contract counts only correctness-affecting gaps — no invented nitpicks.)

Three strikes, then a human

On FAIL, a debugger spins up — fresh context again, so failed attempts don't pile up as noise that poisons the next one. It must reproduce the failure before fixing it, and if it concludes the plan is wrong, that's an escalation, not a code change.

The debug loop caps at 3 iterations — with oscillation detection, and a full re-verification after every fix. On cap, you get a distilled summary of what was tried and what the evidence says, not a 400-line transcript.

An agent loop without a termination condition isn't autonomous. It's just unsupervised.

One folder, thirty-plus tools

The loop is a single Agent Skills folder — an open standard 30+ coding agents read — and the stage contracts name no vendor, tool, or product. Hosts differ wildly in what they offer, so every capability declares an explicit fallback:

Capability	With it	Without it
Subagents	Mode A — one isolated context per stage	Mode B — sequential stages, still reading only declared inputs
Tool restriction	Verifier spawned without write tools	The diff fingerprint alone — the real guarantee anyway
Turn caps	Debugger bounded mechanically	Iteration counting; the 3-cap holds either way
Per-stage models	Strong models on plan/verify/debug	One model throughout — costs more, quality holds

Every run announces which mode it got. One requirement is non-negotiable: shell execution. Verification that can't run real commands gives the loop no termination condition — so it refuses to run rather than emit confident, unverified output. That refusal is the product.

And because everything is markdown plus one POSIX installer — no daemon, no database — there's nothing racing the features agent hosts ship natively every month, and the skill costs zero context until you explicitly invoke it.

So what does this actually improve?

Each failure you've met on real work maps to a specific structural counter — that mapping is the design:

Failure you've seen	What the loop does about it
Bad early output contaminates everything after it	Fresh context per stage; artifacts cross the boundary, transcripts don't
"All tests pass" that nobody ran	Independent verifier, diff fingerprint, evidence required in the verdict
The agent grinds on a fix forever	Cap of 3, oscillation detection, escalation with a failure summary
It rebuilds a helper that already exists	The analyzer's mandatory Found / Exemplars / Missing / Reuse-plan gate
A design doc for a one-line fix	Triage tiers, biased toward the smaller tier
The implementer "follows the plan" into a different design	Plans carry decisions and rationale, enforced by template sections

None of this makes the model smarter. It makes the process resistant to the specific ways coding agents fail.

Steal this, then run it

git clone https://github.com/Madheshvivekanandan/agent-loop && cd agent-loop
./install.sh claude-code     # or: codex · cursor · agents
# Claude Code, zero-install alternative:  claude --plugin-dir path/to/agent-loop

Then, in any project:

/agent-loop add rate limiting to the upload endpoint

First run profiles your project — it discovers your test/build/lint commands and executes them once before trusting them. Every run after that states its tier and mode up front, works the stages, and returns a verdict, a diff summary, and a folder of evidence you can actually read.

If you take nothing else from this post, take the two rules that travel to any agent workflow:

Never let the agent that wrote the code declare it correct. The separation must be structural — fresh context, plan + diff only — not a prompt asking it to "be critical."
Every loop needs a number where it stops and calls a human. Caps aren't a lack of faith in the model; they're what makes it safe to stop watching.

The repo is github.com/Madheshvivekanandan/agent-loop — MIT-licensed, one folder, readable in an evening. If you run it on a host I haven't tested, or find a hole in the verifier's guarantees, I genuinely want the issue.

Previously: The Dashboard That Builds Itself, on generative UI and letting a model order from a menu without getting into the kitchen.

All images in this post were generated with ChatGPT.

The dashboard that builds itself

Madhesh Vivekanandan — Mon, 07 Sep 2026 04:41:51 +0000

Every dashboard you've ever used was designed long before you opened it. Someone guessed which charts you would need, arranged them on a grid, and shipped that guess to everyone. This post is about a dashboard that skips the guess: it doesn't exist until you ask it a question. You type "how did Q3 go?" and the answer comes back as a screen, not a paragraph — stat cards and a chart, building on screen while the AI is still thinking.

I built it with three plain parts: OpenAI's structured outputs (a mode where the reply is guaranteed to match a shape you define), a small Python backend (FastAPI), and Google's open A2UI protocol (a standard way for an AI to describe a UI as plain data), rendered by @a2ui/react, the A2UI project's official React renderer. This post explains how the pieces fit and the five safety nets that make "an AI draws my UI" boring instead of scary. The whole app — backend, frontend, tests and config — is about 4,300 lines, and all of it is on GitHub.

Two questions and a button press, recorded from the running app — every answer re-forms the same canvas.

Three ways to do generative UI (and where A2UI sits)

Nielsen Norman Group defines generative UI as "a user interface that is dynamically generated in real time by artificial intelligence to provide an experience customized to fit the user's needs and context." In this app, that context is the question you just asked. There are three ways to build it, and they differ in one thing: how much you let the model do.

Let the AI write real code — HTML, CSS, JavaScript. Google ships this in Gemini and the results are impressive. But in your own product it means running AI-written code in your users' browsers — a malicious input can trick the model into writing a script that then runs on someone else's screen: the classic cross-site scripting (XSS) attack, delivered by the AI itself.
Let the AI pick from a menu. You build the components; the model only sends data saying which ones, with what values. The model orders from a menu — it never gets into the kitchen.
The same menu idea, with a shared protocol — so any client that speaks the protocol can draw the output of any agent that speaks it. That's A2UI, announced by Google in December 2025. An agent describes an interface as a stream of small JSON messages, and a renderer draws them using only components from a catalog the client defines. UI travels as data, never as code.

The official docs cover the protocol well, and I won't repeat them. Most A2UI tutorials run on Google's own tools — their Agent Development Kit (ADK) and Gemini models — and the exceptions go through large agent frameworks (Microsoft's Agent Framework pairs A2UI with FastAPI and an OpenAI client; A2UI's own docs show LangGraph and Strands setups). What I couldn't find is a write-up that wires A2UI directly to the plain OpenAI API and FastAPI, with no agent framework in between. That's this post.

The whole system in 30 seconds

flowchart LR
    U(["You"]) -->|"how did Q3 go?"| B["Browser · @a2ui/react"]
    B -->|"your question"| F["FastAPI"]
    F -->|"schema + data"| L["OpenAI"]
    L -->|"typed blocks, streaming"| F
    F -->|"A2UI messages, card by card"| B
    B -->|"button press"| F

The browser sends your question to the backend.
The backend asks OpenAI (gpt-4o-mini in this build) for a plan, not code: one to four "blocks" chosen from six types (stat card, line chart, bar chart, donut, table, text note). The plan is forced through a schema, so it always has the right shape. The numbers come from a fixed demo dataset that the backend includes in the prompt, so the model copies and combines real figures rather than inventing them.
A small compiler turns each finished block into A2UI messages.
Those messages stream to the browser using server-sent events, or SSE (one long-lived reply that the server keeps adding to), and @a2ui/react draws them. Cards appear one by one, while the model is still writing.
Cards carry buttons the model invented. Pressing one sends the request back, and the loop runs again.

And one deliberate rule ties it together: every answer is drawn into the same single canvas. Before a new answer streams in, the old one is cleared — so the dashboard is regenerated in place, never stacked. Ask a different question, get a different dashboard, same spot. The ✕ button brings back the starting view.

The menu, not the kitchen

The most important design decision sounds like a technicality: the model never speaks A2UI.

The tutorials I found — including the official A2UI agent guide — do the opposite: paste the protocol schema into the prompt, ask the model to write A2UI JSON directly, then check it afterwards. The format A2UI actually sends over the network is a flat list of components that refer to each other by ID. Great shape for a renderer; terrible shape for a model to write reliably — nothing stops it from referencing an ID that doesn't exist or forgetting a field.

So there are two languages, on purpose:

	Who reads it	What it is
Typed blocks	the model	six block types with required, typed fields
A2UI messages	the browser	the real protocol, produced by a compiler

The typed blocks are the schema I hand to OpenAI. Give the API a shape and it cannot produce JSON in any other shape, because the rule is applied while the text is being written rather than checked afterwards. The model cannot invent a seventh component type or skip a required field. Not "usually doesn't" — cannot. (One caveat: a guaranteed shape can still hold wrong content — the schema stops invalid structure, not bad answers. And this app keeps its count limits, like "at most four blocks," in one line of ordinary code rather than in the schema.)

The cost is real: the agent can only compose layouts my compiler knows how to build. For a dashboard, that's the right trade. For a general-purpose agent canvas, it wouldn't be.

It paid off in a way you can see in the code. Version 1 drew the UI with a hand-rolled renderer — three hundred lines that looked up each card type in a table and drew it by hand. Migrating to the protocol deleted that file, and the frontend no longer contains a single line that asks what kind of card something is; the only place a card type is named is one small lookup table in the backend compiler.

Watching an answer build itself

The reply streams in as it's written, which raises a fun question: when is a card inside a half-written answer safe to show?

The rule in this codebase: a card can only be trusted once the model has moved on to the next one. So the server sends each finished card immediately, and always holds back the last one it can see:

# A block is only provably finished once the next one has started,
# so the last one in the snapshot is always left to the caller.
for index in range(emitted, len(raw_blocks) - 1):

Each card shows the moment it's provably finished. Then, when the whole answer is done, the server sends the complete, fully-checked version once more, and it quietly replaces what is on screen. That works because an update overwrites whatever already sits at the same address — same data path, same component IDs — instead of adding a copy next to it. Any rushed mistake is corrected the moment the answer finishes.

sequenceDiagram
    participant B as Browser
    participant F as FastAPI
    participant O as OpenAI
    B->>F: your question
    F-->>B: open the canvas
    F->>O: ask for typed blocks (schema attached)
    O-->>F: …card 1 finished
    F-->>B: card 1 data + layout
    Note over B: first card renders
    O-->>F: …card 2 finished
    F-->>B: card 2 data + layout
    O-->>F: model finishes
    F-->>B: final complete answer (verified)
    F-->>B: chat explanation + suggestions · done

Here is what one message looks like on its way to the browser. This is all the "protocol" really is:

{ "version": "v0.9",
  "updateDataModel": {
    "surfaceId": "dashboard",   // the one canvas
    "path": "/blocks/0",
    "value": { "title": "Q3 Revenue", "value": 128400,
               "unit": "USD", "delta_pct": 12.4 } } }

Cards don't hold values — they hold pointers into this data tree, so updating the data is all it takes to update the screen. (The full capture is here — a real turn, straight off the wire.)

Buttons that talk back

Without buttons, every answer is a dead end: you read the chart, and then you are on your own to work out what to ask next. The model already knows what is worth asking. So every card carries up to two buttons the model wrote itself, and pressing one talks back to the agent.

All buttons share one action name, refine. The interesting part travels inside the button: as it writes the card, the model puts a complete follow-up request inside the button — label "Split by region", request "Break Q3 revenue down by region as a bar chart." The backend just pulls that request out and runs the same loop again. One handler covers every button the model will ever dream up — and pressing one feels like the dashboard changing its mind.

After a button press — the same canvas, re-formed. The chat keeps the conversation; the dashboard keeps only the current answer.

"Why would you trust an AI to draw your screen?"

When A2UI hit Hacker News, one blunt reaction captured a recurring worry in the thread: "Why on earth would you trust an LLM to output a UI?" It deserves an architectural answer. Mine is five nets, each catching what the previous one can't:

Structured outputs — the schema is enforced during generation. Wrong shapes can't exist.
A sanitizer — a valid shape can still be nonsense. This layer trims oversized data (at most four chart series, 25 table rows, and so on) and rejects anything that cannot be drawn at all — a donut with one slice is not a chart, it is a circle. A bad donut next to a good chart costs you the donut, not the whole answer.
The catalog whitelist — the renderer refuses any component name not in the catalog, enforced by the library itself.
Forgiving readers — every field a view renders passes through a tiny reader that turns bad input into a safe fallback instead of a crash: a missing number becomes a dash, a chart with no usable data becomes an empty state. Agent-written data is the expected input.
An error boundary per card — if a chart does break while being drawn, it takes down its own card and nothing else.

On top of all that sits one promise: the generate endpoint never returns an error. A missing key, a refusal from the model, a network failure — each one comes back as an ordinary text card, drawn through the same path as real content. That leaves the renderer with a single path to follow and no error branch at all.

That promise has a story behind it. A code review on day one caught the fallback card printing raw exceptions — and an OpenAI auth error contains part of your API key. The app was painting key material into a dashboard card. The fix: stable error codes on the wire, generic copy on screen, details only in the server log. If you draw errors from anywhere near the model onto the screen, you will eventually draw a secret onto it too. Decide the boundary on day one.

A real generated answer — every element on screen was ordered from the menu, including the buttons the model wrote itself.

Steal this, then run it

What I'd tell you before you build one:

Don't make the model speak the protocol. Let it fill in a strict, friendly schema; compile to the protocol in code.
Show partial results fast, then re-send the complete checked answer so it replaces anything the quick version got wrong. That final replace is what makes streaming safe to attempt.
Layer your defenses and expect each one to fire. Schema, sanitizer, whitelist, forgiving readers, boundary.
Treat the failure card as normal output. One render path for success and failure means the error path is tested on every fallback.
Test that streaming streams. The worst thing a broken streaming system can do is keep working. Assert the first update arrives before the stream closes.
Build the starting view the same way. The dashboard you see on load is compiled by the same code the agent uses, so bugs show up before you type anything.

What's missing, honestly: no auth, no database, mock data, and A2UI v1.0 will eventually mean some rework.

The whole thing runs with docker compose up and an OpenAI key — without a key it still boots and shows the baseline. The repo is here — clone it, type something odd into it, and tell me what breaks.