DEV Community: ForgeCoreAi

The worker never writes its own test: how my agents are kept from grading their own homework

ForgeCoreAi — Tue, 13 Oct 2026 10:15:12 +0000

In my last post the rule was: no model says done, a command exits 0. The obvious hole came up in the first comment I imagined writing to myself: who writes the command?

If the same agent that writes the code also writes the check, "exit 0" means "the agent agreed with itself". A check written after the work can be shaped to fit the work. So there are three separate rules in Hermes.Crew, and each one closes a different way to cheat.

Rule 1: the proof is frozen when the card opens

The proof command is snapshotted in the card's origin event, at the moment the card opens. A later edit of the proof command: line in the card text changes nothing: the only command that can produce a PASS is the snapshotted one. The one legitimate way to change it is a coordinator rescope, which is a recorded decision, not an edit.

Rule 2: closing is refused without a PASS line

Every run of the proof appends a verdict line (who ran it, which run, PASS only on exit code 0) to a per-card file. kanban_complete on a crew card is refused, in every profile, unless that file has a PASS for the card's proof command and no check failed after it.

The agent cannot argue its way past this. It is not a prompt asking for honesty, it is a tool that returns an error.

Rule 3: when the proof is a script, the verifier writes it

Some work cannot be proven by a one-liner. For those, the proof is a script in a .crew/ folder, one folder per card. Here the split matters:

the verifier writes the script, from the card's "Done when" and its inputs, never the worker
the plugin's role guard refuses a worker's write, patch or terminal write under .crew/
the verifier's own file tools reach only .crew/
a card whose proof is such a script is always verified independently, even if the card asked for the cheaper mode

The script is bound by hash at its first run. It changes only through a coordinator revise_script decision, and the new hash is recorded with who wrote it and which decision authorized it. If the worker finds the script annoying, there is no path to a quieter version.

And after "done", someone checks again

When a card is done, the coordinator re-runs the proof once more. A fail comments on the done card and opens one follow-up card under it. One lucky green run does not close the matter.

The owner can still override from the CLI with hermes kanban complete --force; that shows up as an owner_close decision rather than a silent pass.

What this does not buy you

It does not make the proof correct. A verifier can write a script that checks the wrong thing. Independence removes the worker's incentive to fit the test to the work. It does not remove misunderstanding of the brief.
It costs a second model session on every independent card, plus a proof run. On a one-line change that is a bad trade, which is why a plain command-only proof stays the cheap mode.
Hashes make tampering visible, not impossible. The guarantees are about what the plugin lets an agent do through its tools. They are not a sandbox against a person with a shell on your machine.

The pattern, without my plugin

Whoever does the work does not write the acceptance check.
The check is fixed before the work starts, or changes only through a recorded decision.
The "close" action is physically refused without a passing result.
Something other than the worker re-runs it afterwards.

Any agent framework can do all four with a file, a hash and a guard in front of the close call.

Hermes.Crew is Apache-2.0, self-hosted, no telemetry: github.com/macd2/Crew.

hermes plugins install macd2/Crew
python3 ~/.hermes/plugins/crew/install.py --profile NAME
hermes -p NAME plugins doctor crew

in your agent setup, who writes the acceptance check, and could the worker edit it? If yes, I would like to hear what happened.

I built the faster way to drive herdr: every binding one keypress away

ForgeCoreAi — Tue, 13 Oct 2026 09:19:35 +0000

The fastest thing you can do in a terminal is the thing you never have to leave it for. herdr gives you a lot of actions and then buries them: the ones you reach for most sit below help, settings and detach in its own list, moving a pane between tabs exists only on the command line, and even splits have no action, keybinding or CLI at all. Every one of those is a reason to lift your hand off the keyboard.

This plugin is about not doing that. One key shows every binding you have - not the defaults, your resolved keymap - ordered so the things you reach for are at the top. Three more keys do the pane work herdr leaves to the CLI: send a pane to another tab, walk it through its neighbours, set every split back to half.

Nothing to configure beyond the bindings you want, because the list is not stored anywhere. It is derived from your own config every time it opens.

One key, and the whole keymap

herdr will not print your resolved keybindings. There is no CLI for it. What exists is herdr --default-config, where every action appears as a commented line:

# action = "binding"

and ~/.config/herdr/config.toml, which overrides those. The set your fingers actually know is the merge of the two, and nothing in the tool shows it. So the plugin does the merge and renders it - which is also why it cannot go stale: bind a new chord, and the popup is right on the next keypress.

The popup has three columns under one title. KEYCAPS is what your config adds, PREFIX is herdr's stock binding for the same action, spelled out as ctrl+b h rather than prefix+h because the prefix is released before the next key and that is how the chord is typed. Keeping them apart is what makes the page scannable: your own chords line up down the left, instead of being pushed around by the longer prefix forms. An action with nothing under KEYCAPS is one still waiting for a direct chord.

Your own commands come first, because herdr's config order opens with help, settings and detach - the things you reach for least - and pushes pane and tab movement to the bottom of a popup that already has limited height.

Colours are ANSI 16 only, so the popup resolves through the terminal's own palette and follows whatever herdr theme is active. NO_COLOR=1 turns that off; piped output is plain either way.

The pane work herdr leaves to the CLI, as keys

Three things, and the third has no CLI at all.

Move a pane to another tab on alt+m. A picker, arrows or j/k to move, a number to jump straight there, Enter to confirm, q or Esc to cancel. Destinations are grouped by workspace, so a pane can cross into another one:

  MOVE PANE  pick a destination

  Redesign
  › 1  Claude1  (2 panes)
    2  a new tab here

  Infra
    3  Notes  (1 pane)
    4  a new tab here

    5  a new workspace

Headings are not selectable and the cursor skips them, the tab the pane is already in is never offered, and "a new tab here" carries the workspace it sits under, so it lands there rather than in whichever workspace has focus.

Swap a pane with its neighbour on alt+shift+h/j/k/l. herdr's own move_tab_previous and move_tab_next move whole tabs, not panes. Hold the same direction and the pane keeps walking, and at the edge it wraps to the far end instead of stopping dead, so a row behaves as a closed cycle - because herdr reports a missing neighbour as changed: false with exit code 0 rather than as an error, and that is the cue to wrap. The walk is bounded so a misbehaving swap cannot spin forever.

Even out the panes on alt+shift+e, which sets every split in the tab back to half. pane resize only shifts a divider by a relative amount, so there is nothing to call with: this one goes through the socket API's layout.set_split_ratio, over newline-delimited JSON on ~/.config/herdr/herdr.sock. A split is addressed by the path taken to reach it - [] is the root, then false descends into first and true into second.

The parts that took longest are the parts herdr does not document

Keybindings can only target plugin actions, never plugin panes. The popup is not what you bind. You bind an action, and that action opens the pane:

alt+h  ->  action "show"  ->  show.py  ->  herdr plugin pane open
                                            -> pane "keys" (popup)
                                              -> keycaps.py

placement = "popup" has to live in the manifest, not on the open request, because the CLI's --placement flag has no popup value. A popup also takes all terminal input, Escape included, and closes the moment its command exits - so handing the list to a pager is out. It pages in-process instead: Escape closes it, q and ctrl+c too, arrows and page keys and g/G scroll when it does not fit. (less would have needed --lesskey-src to rebind Escape, which arrived in less 582 while macOS still ships 581 - a plugin that works on one machine only is not a plugin.)

The install has a step herdr cannot tell you about. A plugin manifest has no field for a keybinding, and herdr plugin install never touches config.toml. So after the install every action is registered and no key reaches any of them, which looks exactly like a broken plugin. Hence the second command:

herdr plugin install macd2/herdr-keycaps
python3 ~/.config/herdr/plugins/github/macd2.keycaps-*/setup.py

setup.py merges this plugin's seven bindings - plus the single-modifier alt+ layer that fills the KEYCAPS column for herdr's own actions - into your config.toml. It backs the file up first, keeps your comments and ordering, adds only what is missing, and leaves any chord you already spent where it is, naming it on the way out. Running it twice changes nothing.

herdr cannot print a warning at install time either: the install preview emits only herdr's own fields, and a build command's output is piped and discarded unless the command fails, which aborts the install. So the plugin waits for the next server start and puts one notification in the top right, naming the command - and only while nothing at all is bound. Bind a subset on purpose and it stays quiet, because a half-spent config is a choice, not a mistake.

Both scripts would rather fail than mangle the file

Your whole herdr setup lives in that one file, so a wrong edit is worse than no edit. setup.py and uninstall.py both state what they intend, build the result in memory, re-parse it and compare it against that intent before anything is written. One unexplained difference - another table changed, a value that came out wrong, output that is not valid TOML - and nothing is written at all. Shapes a line-based edit cannot see are refused rather than guessed at: a keys.command inline array, a value spread over several lines.

A refusal exits 1 and prints the reason, the path, and the steps to do it by hand - addressed to an agent, telling it to ask the human first rather than editing the file to get past the refusal. Stock config, then setup.py, then uninstall.py --also-direct-layer gives back the original file byte for byte.

The failure that is worth a paragraph

If you attach with herdr --remote, nothing in this plugin works until you pass one flag:

herdr --remote <target> --remote-keybindings server

herdr --remote uses your local keybindings by default, and local custom command bindings are never sent to the remote host. This fails silently, which is what makes it worth knowing: herdr config check says config: ok, herdr server reload-config says applied, the plugin links fine and lists its actions - and no key does anything.

Tests

Four of them, because a plugin that edits someone's config should be boring to trust:

python3 tests/test_manifest.py     # the required metadata the marketplace parses
python3 tests/test_grouping.py     # group order, custom first, unknown actions kept
python3 tests/test_popup_pager.py  # stays open on a real pty, closes on esc
python3 tests/test_show_action.py  # action opens the right pane, opens no popup

The manifest one matters more than it sounds: the herdr marketplace indexes public repositories tagged herdr-plugin whose default branch carries a herdr-plugin.toml. Discovery is automatic and unreviewed, so a typo there is an invisible delisting rather than an error anyone sees.

An action no group knows about still renders, under OTHER, so a binding added by a future herdr release is never silently dropped.

Try it

herdr plugin install macd2/herdr-keycaps
python3 ~/.config/herdr/plugins/github/macd2.keycaps-*/setup.py

MIT, Linux and macOS, built and verified against herdr 0.9.3. The source is github.com/macd2/herdr-keycaps.

So the question I would like answered in the comments: which action in your setup is still not one keypress away? That list is what this plugin is built around, and for me it was moving panes - the one thing I did with a right-click for months because herdr only exposed it on the command line.

Drafted with an AI assistant and edited by me.

I built an agentic workforce where done means a command exited 0

ForgeCoreAi — Sun, 11 Oct 2026 11:20:03 +0000

Every failure I have had with coding agents looks the same at the end. The agent writes a summary that reads like
success: the feature is implemented, the tests pass, the branch is ready. Then I open the thing, and half of it is not
there.

The interesting part is that the model is not lying. It has no way to check. It generated the most plausible
continuation of the conversation, and "done, with a tidy summary" is an extremely plausible continuation. Nothing in
the loop ever asked for evidence.

So I stopped trying to make the model more honest and changed the shape of the loop instead. In the agentic workforce I run now, no
model is allowed to say done. A command says done, and it has to exit 0.

The proof command gets written before the work exists

When I type an ask, the first thing that happens is not work. It is a contract.

/crew add a pricing page with a monthly / annual toggle

The card that opens carries six things:

goal - what this is, in my words
artifact - the file or surface that changes
where it lands - the branch
done when - the same thing, in plain language
proof command - the exact command whose exit code decides
budget - tokens, so a run cannot spin

The order is the point. The proof is frozen while the work does not exist yet. At that moment I am uninformed and
honest, so I write:

npm run test:e2e -- pricing.spec.ts

and not "looks right to me". Later, with work already on the branch and the temptation to call it finished, the
command is a fact from an earlier conversation. Neither I nor the agent can quietly move it. That is most of the value:
not detection, but removing the option.

Done is decided by a tool, not by a model

In the workforce, done is not a status an agent sets. It is the exit code of that command, run as a tool call, at the point
where the card would otherwise close.

the worker does the work
the proof runs, exactly as written when the card opened; a non-zero exit is the only failure signal that counts
the coordinator re-runs the proof before the card can close, so one lucky green run does not close it
a card that asks for a stronger guarantee gets one independent verifier run instead

The model writes the code and the summary. It never writes the verdict.

One owner per card, one writer per card

The second failure mode is drift: three agents on the same branch, each doing something locally reasonable, together
producing nothing. So a card has one writer, and a card has one owner - a coordinator that wakes on every card event
with the whole record in hand and takes exactly one decision: retry, rescope, split, or ask me.

The asks are bounded too. Before any work starts, the chat asks at most five questions in one batch, or none. After
that I am out of the loop until the card ends, and it ends either with a verified result in the thread I asked from, or
with one concrete question. Token budgets are per card, so a run that reaches its budget stops instead of burning
through the rest of the night, and a worker that fails five tool calls in a row stops itself.

What it looks like from the outside

Illustrative card, from the demo board:

16:12:04  /crew add a pricing page with a monthly / annual toggle
16:12:09  contract: artifact app/pricing/page.tsx
          proof    npm run test:e2e -- pricing.spec.ts
16:12:11  card opened - coordinator owns it, your turn ends
   ...
16:41:21  done - proof exit 0 - 2 runs, 1 fix

That last line is what reaches the chat: not a summary of feelings, the proof that passed and how many attempts it
took. If the proof had failed twice, I would have got one question instead, with the two failure lines attached.

Where this shape stops working

Three honest limits, because a post about verification that only lists wins is its own kind of bullshit.

A proof proves what you wrote, not what you meant. test:e2e -- pricing.spec.ts can be green while the toggle does nothing, if the test asserts the wrong thing. The leverage is all in writing the command well, and that is human work.
It is bad for work where done is a judgement. Naming, positioning, "write the launch post" - no command decides those. The workforce falls back to a person, which is correct behaviour, but it means this shape only pays off on tasks with a checkable end.
Proofs cost a minute each. On trivial work that is a bad trade. On the work you would otherwise re-check at 23:00 anyway, it is the cheapest minute in the task.

There is also a fourth thing I did not expect: because the proof fixes the artifact, "can you also just..." turns into
a second card instead of a scope creep inside the first one. Annoying, and correct.

The pattern is the point, not my plugin

The generalisable rule is not complicated:

before the agent starts, write down an executable definition of done;
let only that definition close the loop;
let a tool run it, not the model that did the work.

Any agent framework can do this. Most do not, because "done" is much easier to print than to prove.

I built it as a plugin for Hermes Agent - Apache-2.0, self-hosted, runs on your own machine, no telemetry - and the
whole thing is github.com/macd2/Crew:

hermes plugins install macd2/Crew
python3 ~/.hermes/plugins/crew/install.py --profile NAME
hermes -p NAME plugins doctor crew

If you would rather watch it before installing anything - the 53 second launch film, one ask through to the verified
result:

crew.forgecoreai.com is the same loop as a page, if you would rather read than watch.

So the question I would actually like answered in the comments: what does done mean in your setup? Do you have a
command, or is it a feeling? I am collecting the ones people genuinely trust - a solo dev with pytest, a team with a
deploy smoke test, someone checking a number in a dashboard. A proof you cannot write is a task you cannot delegate,
and the interesting stories are the tasks where you had to admit that.

Drafted with an AI assistant and edited by me.