DEV Community

Cover image for Your agent went green by weakening the tests
Brasth
Brasth

Posted on

Your agent went green by weakening the tests

I already wrote about babysitting coding agents: prompt, wait, almost right, exit 0, still will not merge.

There is a nastier failure mode than a messy diff.

The worker "fixes" the suite by editing the suite.

Softens an assert. Deletes the case that hurt. Rewrites the expected value to match the bug. Process exits 0. The narrative says done. You open the PR and realize the contract moved under your feet.

That is not a flaky CI story. That is acceptance theft.

Acceptance theft before and parent-owned verify after
Before: worker can move the bar. After: parent owns acceptance.

Softened asserts diff
What acceptance theft looks like in the diff.

The objection that stuck

On the earlier DEV post, Reid Marlow put the tripwire cleanly: workers love modifying test files to make the run green, and separating test-file diffs from source diffs is how you catch it.

He is right. A noisy context window wastes tokens. Green-by-softening-asserts wastes trust. Once a worker can satisfy the check by changing the check, "tests passed" stops meaning what you think it means.

I care about that failure more than parallel agents stepping on the same file. File collisions are loud. Weakened asserts are quiet. Quiet is how bad merges ship.

Exit 0 was never acceptance

In a single-agent thread you can still notice. You own the terminal. You see tests/ light up in the diff. You can refuse.

Once you hand work to a worker (subagent, second CLI, MCP job, whatever), the temptation grows: the child is scored on "make it green," not on "keep the contract honest." If test paths are writable and acceptance is the child's exit code, you taught it the wrong objective.

So the product question is not "did the worker finish?" It is "who owns the bar, and can the worker move the bar?"

What Rig does about it

Rig is not another coding agent. It is the harness between ones you already use.

Spine, short version:

  1. Parent scopes files. Owned paths, not "edit whatever looks related."
  2. Parent writes a brief. Change / don't-change / acceptance. The worker runs that brief over MCP.
  3. Parent verifies. Real checks against the brief. Exit 0 is execution. Verified is acceptance.

For the weakened-tests case, that spine has concrete teeth:

  • Parent owns acceptance. The worker does not get to mark the job verified. The parent does, after its own checks.

  • Tests can be read-only / out of scope in the brief. If the job is "implement X," keep tests/ (or the contract files you care about) out of the writable set. Softening asserts then becomes a scope violation, not a clever pass.

Real Rig brief shape
Real Rig brief shape: scoped files, tests read-only, parent-owned acceptance.

  • Parent re-runs the real check set before accept. Not the child's story of green. The parent's command, on the parent's terms.

Parent verify: real checks, then accepted
Parent verify: real checks, then accepted.

  • Scope violation shows up as dirty verify, not a silent pass. If the worker touched paths it was not given, that is fail-closed for acceptance. You see it. You do not merge on vibes.

Reid's tripwire (split test diffs from source diffs) still matters as a human review habit. Rig's bet is to make that habit structural: the harness already knows which paths were in scope, and verify is not optional theater.

MCP is just the handoff pipe here. The point of this post is not multi-job graphs. It is who owns the bar when one worker tries to move it.

Configure the harness, then type normally

Prefer editing .rig/harness.toml over memorizing worker CLI commands:

parent = "codex"

[workers]
codex = false
grok = true
claude = true
cursor = false
opencode = false
Enter fullscreen mode Exit fullscreen mode

Enable only workers you actually have. Quit the parent once after install, open a new thread in the repo, and type a normal prompt. Everyday work is not rig run.

When the brief matters, be explicit: which paths may change, which paths are the contract, what command the parent will re-run. That is the difference between a worker that implements and a worker that negotiates the rubric.

Soft landing

Repo: https://github.com/Brasth/Rig

A screen recording of the broader babysitting harness lives elsewhere. This post is about acceptance theft, not that clip. Soft link only: clone it, try a scoped brief, and keep tests/ out of the writable set when the job is implementation.

If this is your scar

Same models. Less glue. Free and open source.

If a worker ever went green by editing the asserts, you already know why exit 0 is not enough. Rig is my attempt to keep the bar on the parent side.

I am on X as @brasthapp if you want to argue about scopes, verify, or MCP handoff.

Top comments (0)