DEV Community: Ordewell

How to tell when a coding agent has actually finished

Ordewell — Fri, 18 Sep 2026 17:22:38 +0000

Every harness that runs a coding agent eventually has to answer one question: is this
task done? The three signals you reach for first all fail, in ways that are quiet
enough to cost you an afternoon.

Disclosure: I build Ordewell, and this page is how it answers that
question. The method is not a product feature though. It is a convention you can put in
any harness in an afternoon, and the pitfalls below will bite you in any language, so
they are worth reading even if you never run my tool.

The three obvious signals, and why they fail

1. The terminal went quiet. An interactive agent TUI does not close when
the work is finished. It stays open, waiting for you to say something else. Silence means
the model stopped printing, which is also what it does while it thinks, waits on a tool,
or gives up.

2. The process exited cleanly. Exit code 0 means the program ended without
an error. An agent running out of context, hitting a stop condition, or deciding the task
was already satisfied exits 0 too. The exit code tells you the process finished, not the work.

3. The agent said it was done. This is the one that hurts, because it is
confident. Models announce success over a build that does not compile, in a tidy paragraph,
with a summary of changes that were only partly made. When a coordinator agent supervises
worker agents, a version of this scales up: a model reading a model's report and ruling on
it.

The fix is to stop asking for an opinion and start watching for evidence.
A task carries a unique token. The prompt asks for that token on the final line. The
harness watches the runner's output for it. Token present means pass. Token missing means
fail, loudly, even when the session exited cleanly.

It cannot be a simple string search

Here is the part that surprised me. Watching for a token in terminal output sounds like
output.includes(token). It is not, and the reason is that a PTY stream is a
rendering protocol, not a document.

The raw bytes carry ANSI colour codes, OSC sequences, box-drawing gutter characters from the
TUI's borders, cursor moves, and line erases. Interactive TUIs soft-wrap long lines, so a
single token can arrive split across two writes with an escape sequence in the middle. A
raw string search misses it.

The first pass is a flatten: strip escapes, strip the box-drawing range, strip all
whitespace. Dropping whitespace sounds reckless until you notice the token itself has none,
so the flattened view can be scanned safely.

scan, flattened

```// strip ANSI/OSC escapes, box-drawing gutters, then every space function flattenTerminalOutput(raw) { return raw .replace(ANSI_OR_CTRL_RE, /* escapes, control bytes */ "") .replace(/[─-▟]/g, "") // TUI borders and gutters .replace(/\s+/g, ""); }


That handles the byte-stream case. It does not handle the screen case.

## A full-screen TUI is not a stream

Some agent TUIs paint fragments at absolute cursor positions and repaint unrelated widgets
between those writes. Frame a spinner, then finish the token on a row that was already
partly drawn. Flattened chronologically, the spinner lands in the middle of the token and
nothing matches:

what the bytes look like

row 9: << row 23: ⟳ spinner repaint
row 9: ELL_DONE_9f3c…>>>``

So there is a second pass: replay the cursor and erase sequences to reconstruct the small
screen the user is actually looking at, then scan that. The chronological flatten stays as
the fallback for plain piped output and for soft-wrapped lines.

Two scans, one verdict. If either sees the token, the task passed. Being strict here costs
throughput for no gain, because a false negative is the failure that erodes trust.

The three rules that make the token trustworthy

Rule 1: one token per task. Not per plan, not per run. Generate it when the
task is created, keep it on the task, and scan for that exact string. A shared token across
tasks means task six passes on task two's evidence.

Rule 2: the instruction must not contain the token. Whatever you append to
the prompt gets echoed into the session, and you are scanning that session's output. A
literal token in the instruction would settle the task before the work started. So the
instruction asks the model to assemble the token from two pieces, and the assembled form
never appears in the text you send:

what the runner is asked for

When you have fully completed this task, print one final line containing
only the completion marker. Build it by writing <<<ORDEWELL_
immediately followed by DONE_<task token>>>, joined into a single
unbroken token, with no space, quote, or any other character between
the two parts.

Rule 3: break tokens in anything you hand to the next task. Tasks receive
their predecessors' notes and summaries. If a finished task's captured output still carries
a live token and that text becomes context for the next one, the model can echo it back and
settle the wrong task. One replacement when the notes are assembled is enough: the token
prefix gets a hyphen spliced into it, and it can never match again.

When the token never shows up

The mission is to make that state unmistakable. The session ended, the model claimed the
work, and no token arrived. That is a failed verification, with the exit code kept beside it
as separate evidence rather than as a verdict in its own right. These are the two messages
the card can carry:

verdict

``Verified: completion marker detected in agent output. Task completed successfully.

Failed verification: agent exited cleanly but did not emit the completion marker.`




Note what the exit code does *not* do: it never overturns the token in either
direction. A clean exit after a missing token is still a failure. A non-zero exit after a
seen token is still a pass, because the agent did the work and then something unrelated
tripped on the way out.

It is also worth giving long tasks a mid-run checkpoint. The same output scan can carry a
second token type that pauses the task and asks you to approve or reject before it goes
further, which turns "review the diff at the end" into "decide at the risky step".

## Honest limits

- **A marker proves a claim, not correctness.** It shows the session finished and said so. It does not show the tests pass, the build compiles, or the change is right. Nothing here replaces reviewing the diff.

- **The scan is bounded.** Ordewell scans the recent tail of the output buffer, not the whole session, so a token printed an hour before the session ends can in principle sit outside the window. That is a deliberate trade for not re-flattening an unbounded buffer on every write.

- **Trimming output can hide evidence.** If your harness caps or truncates what it keeps, you can hide your own token. Keep the tail, drop the middle.

- **Manual overrides exist and are humiliating.** Marking a task complete by hand records that no automatic verification was performed. It is honest, and it should feel like a warning.

- **This is a convention, not a proof.** A model that prints the token without doing the work defeats it. A task-specific smoke check in the plan is the answer to that, and it is a real gap in the method, not a footnote to it.

## If you want it already wired

Ordewell generates one token per task, appends the instruction above, runs one real
coding-agent session per task, and verdicts each one by evidence. That is the whole loop:

bash



````ordewell plan --goal "Add rate limiting to the public API"
ordewell run`

The planner reads your repo read-only, so planning needs no API key: a coding agent you
already pay for can be the planner. Install with npm install -g ordewell, or
read the docs first. The source for everything on this page is in
the repo.

Licensed under Apache-2.0.

Home
Docs
Plan-first vs parallel
GitHub
Design decisions
License

Why I stopped choosing one coding agent — and route each task to the one that fits

Ordewell — Wed, 05 Aug 2026 23:31:19 +0000

Disclosure up front: I built the tool described here. Free and Apache-2.0;
the point of this post is the design thinking, not a purchase.

Most agent tooling assumes you've picked a vendor. You're a Claude Code shop or
a Codex shop or a Cursor shop, and everything you run is that one thing. But
that assumption was never true for me — Codex is better at some tasks, Claude
Code at others, and I wanted a single plan that could use both without me
shepherding each step by hand.

This post is about the design decision behind letting one plan hand work to
different agents — and what it takes to make that safe.

A plan as a portfolio decision

The tool starts the same way each time: a goal in, a planner reads the repo
read-only, and an ordered list of tasks comes back. What's different is that
each task carries its own runner, model, thinking effort, and mode. The planner
makes one portfolio decision across the whole plan — a security refactor and a
README update should not get the same model, let alone the same agent — and
shows you every assignment before anything runs. You can change any of it.

That's the part worth dwelling on: per-task routing as a first-class feature,
decided in the open, rather than "pick one agent and hope."

Adding an agent is a plugin manifest, not a fork

The barrier that usually stops multi-agent setups is that wiring in a second
agent means invasive code. Here, adding an agent is a small plugin manifest, not
a code change. Claude Code, Codex, and OpenCode ship built-in; Aider or your own
CLI is just another manifest. The planner reads each runner's catalog, so a
task's mode and model re-derive from what that runner can actually spawn — you
can't configure a task into a runner that can't run it.

The safety boundary that makes mixed plans viable

Routing work across agents raises an obvious question: what stops the planner
from doing damage in your repo? Two answers, both deliberate.

First, the planner can't write. Research commands are lexed the way a shell
would lex them and classified per segment; the write tier returns refused before
any approval prompt exists, so there's no "allow once" to click through.
Mutation belongs to the runners.

Second, completion is evidence-based. A task is done only when its unique
completion marker appears in the runner's output, exit code retained separately.
No agent grades its own work — the marker and the exit code agree, or it fails.

Honest limits

The planner is an LLM and occasionally produces a bad plan — the argument for
this design is that you see the bad plan before a single execution token is
spent. Also, mixing runners means you hold the subscriptions for whatever you
mix: it rides the Claude subscription you already have for Claude Code, the
Codex subscription for Codex. No extra API key, but also no free ride for a
runner you don't already pay for. And the TUI needs tmux (WSL on Windows); the
CLI and VS Code extension don't.

Try it

npm install -g ordewell && ordewell

GitHub: github.com/ordewell/ordewell
Apache-2.0, free.

One question I'd genuinely like input on: is per-task model routing useful, or a
knob nobody touches? Curious what multi-agent people think.

How I stopped my coding agents from writing files before I could see the plan

Ordewell — Wed, 05 Aug 2026 23:04:43 +0000

Disclosure up front: I built the tool this article walks through. It's
free and Apache-2.0, so my interest is "people find this useful" more than
"people pay me."

The failure that sent me down this path is probably familiar. I give an agent a
multi-step goal: refactor this module, update the README, add tests. Everything
goes fine until step 4 — where I discover it misread step 1 back at the start.
By then files are written. There's no plan to correct, because the plan lived in
the model's head. There's only something to undo.

That's the core problem: the plan is a side effect of the session, not an
artifact you can inspect and edit. This post is about the design decision I
landed on to fix it — make the plan the thing you approve, not the thing you
hope for.

The plan is a typed artifact

The tool reads your repo read-only, then hands back an ordered list of tasks.
Each task carries four pieces of metadata: the runner (Claude Code, Codex, or
OpenCode), the model, the thinking effort, and the mode. That's not decoration —
those four are what the task actually runs as. You can rewrite any prompt, add
or delete a task, rewire dependencies (task 4 depends on task 2), or change
the model on a single task. Completed work stays done. Nothing round-trips the
AI while you edit.

Why a separate planner

The natural question is "why not just use plan mode?" Two reasons.

First, plan mode plans inside the session that then executes — the plan is
advisory and the model can drift from it. Here, the plan is produced by a
separate process that physically cannot write to your repo. Its exploration is
strictly read-only; commands that would write are refused before an approval
prompt even exists, so there's no "allow once" to click through. Mutation
belongs to the runners, not the planner.

Second, per-task assignment. A security refactor and a README update do not
deserve the same model. The planner makes that portfolio decision across the
whole plan, in the open, before you spend a single execution token — and you can
override any of it.

Verdicts from evidence, not opinion

The part I care most about happened later. Agents are bad at grading themselves
— and worse, they'll announce success on a build that doesn't compile. So a task
here is marked done only when its unique completion marker appears in the
runner's output, with the exit code retained beside it as separate evidence. The
model is never the tie-breaker. If the session finished and claimed the work
without the marker, it fails loudly.

If you've watched an agent say "done" and then open a file that's still broken,
that's the whole motivation.

Honest limits

The planner is still an LLM and writes bad plans sometimes. The argument isn't
that it's always right — it's that a bad plan you can see and edit costs a
minute, and one you can't costs an afternoon. The completion marker proves the
session finished and claimed the work, not that the code is correct.

Also: the TUI needs tmux on every platform (on Windows, that means WSL). The CLI
and the VS Code extension run natively without it.

Try it

npm install -g ordewell && ordewell

Point it at a repo you know well and read the plan it hands back. That first
plan is the whole argument.

GitHub: github.com/ordewell/ordewell
Apache-2.0, free, no paid tier.

Happy to answer anything, including "why not just do X."