You gave the agent a failing test and told it to get the suite green. It came back green. Then you read the diff: it did not touch the code under test. It edited the test. The assertion that read == 9000 now reads == 10000, which is exactly what the buggy function returns, so the bar is green because the test was changed to agree with the bug.
This has a name. It is reward hacking, and it is not a rare glitch on the margins. Cursor's own engineering team published a piece titled reward hacking is swamping model intelligence gains. There is a benchmark built to measure it in long-horizon coding agents, SpecBench. And every developer who has pointed an agent at a red suite has watched some version of it: the deleted assertion, the @pytest.mark.skip, the hardcoded return, the sibling test quietly weakened. The agent was told to make the check pass. It made the check pass. Nobody told it the check was a stand-in for the code being correct, so it optimized the check it was actually handed.
That gap, between the check and what the check stands for, is what this piece is about. A loop runs five arms: generate, check, steer, retry, stop. The series opener named them; three pieces since took the check that decides good enough, stop, the gate that refuses a bad write, and the surface every rule loads from. This one takes the arm that sets what the agent aims at on the next try: the steer, and the version of reward hacking the steer hands the model.
What the steer is
The steer is the arm that turns a verdict into the next instruction. When the check comes back red, a line of text gets assembled from the check's output and fed into the next generate. Here is the loop the gate piece built, refactoring src/ until a guard holds. The steer is one arm in it:
#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
run_agent --task "$prompt" # GENERATE
if bash no-mocks.sh; then # CHECK
echo "stop: guard holds after $i retries"; exit 0
fi
prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2>&1)" # STEER: only the new signal
i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1
The model never sees the whole history. Each retry it sees one prompt, and that prompt is whatever the steer decided to carry back. On the first pass the prompt is the goal. On every pass after that the steer overwrites it. So the target the model aims at on retry three is not the goal you wrote, it is the last thing the steer said, and the steer is a line the loop composed on its own while you were not looking.
The steer gets none of the attention
Reward hacking has more than one cause, and most of the attention goes to two of them: a check loose enough to game, and an agent with write-access to the thing that grades it. The third gets almost none, and it is the one this piece is about. It is the objective the loop hands the model on each retry, and that objective is the steer.
The model does not optimize the check directly. It optimizes the instruction it was handed, and that instruction is whatever the steer wrote. When the loop feeds back make the test pass, it has named the check as the goal. From there, optimizing the instruction and gaming the test are the same action, because the cheapest state in which the test passes is the one where the test agrees with whatever the code already does. The steer said the target was green. Green is what came back.
None of this touches the other roads to a gamed result, and it is worth being honest about that, because the Cursor piece above documents one of them. A large share of the reward hacking it found was answer-retrieval: agents pulling a fix straight from a public pull request or the repository's own bundled git history, with 63% of one model's successful resolutions retrieved rather than derived. That happens with the goal fully intact. It is an access problem, not a steer problem, and no wording of the steer prevents it. The steer is the lever this piece takes because it gets none of that attention and is the cheapest to fix. You write it yourself, once per retry, and most loops write it badly.
The good steer holds the goal
Look at what the loop above carries back. The goal is stated once, before the loop, and the run_agent call never re-ships it. The steer rewrites prompt to carry the guard's own output and nothing else:
prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2>&1)" # STEER: only the new signal
That is the shape you want: directive first, then the evidence. Fix the lines the guard flagged, and here are those lines, verbatim from the check. The goal has not moved, because the steer never restates the goal, it appends the delta to it. The model gets the original target plus a precise account of what the last attempt got wrong, in the check's own words. Pass the test is never the whole of what it optimizes, because the goal it was serving is still on the page next to the failing line.
A good steer is a reduction of the check's output. It takes the verdict and the minimal evidence that produced it and hands that back unaltered. The moment the steer summarizes the failure into make it pass, it stops being a reduction and becomes a new goal, and the new goal is the one the agent will game.
Watch a loop game its own check
Here is the same loop, one check, and two steers. The check is a unit test: a $100 charge should cost $90 after a 10% discount. The code has the discount missing. The generator is a stand-in for the model; its two branches do the cheapest thing each instruction names, which is the whole point, so both are on the page rather than hidden:
GOAL="charge(cents) must apply the 10% discount so charge(10000) == 9000."
# THE CHECK: run the test. pass = exit 0.
check() { python3 test_charge.py >/dev/null 2>&1; }
# THE STEER: turn the check's output into the next instruction.
steer_good() { printf '%s\nThe test still fails; fix the failing assertion: %s\n' "$GOAL" "$1"; }
steer_bad() { printf 'The test is still failing. Make the test pass.\n'; }
# THE GENERATOR: a literal optimizer standing in for the model. It takes the
# cheapest route the instruction names, the same shortcut a real model reaches for.
run_agent() {
case "$1" in
*"Make the test pass"*) sed -i 's/== 9000/== 10000/' test_charge.py ;; # game: edit the test
*"fix the failing assertion"*) sed -i 's/return cents/return int(cents*0.9)/' charge.py ;; # fix: change the code
esac
}
The good steer holds the goal and appends the failing assertion (expected 9000, got 10000), so run_agent takes the fix branch. The bad steer drops the goal and hands back the symptom, so run_agent takes the game branch. Run the loop with each and both terminate the same way:
$ bash game-demo.sh good
stop: test passes after 1 retries
charge(10000) returns: 9000
test asserts: == 9000
check verdict: GREEN
goal met ($100 charges at $90): YES
$ bash game-demo.sh bad
stop: test passes after 1 retries
charge(10000) returns: 10000
test asserts: == 10000
check verdict: GREEN
goal met ($100 charges at $90): NO
The stub is pinned so the loop is reproducible on your machine, but the branch it takes is not the trick, it is the claim: hand a literal optimizer make the test pass and editing the assertion is the cheapest path to green; hand it the goal plus the failing line and changing the code is. A real model reaches for the same shortcuts under the same two steers. Both runs print stop: test passes after 1 retries and come back green, so from outside the loop the two are indistinguishable, same verdict, same retry count, same clean exit. The difference is only in the artifact. The good steer left charge() fixed and the test asserting == 9000; the bad steer left the bug in place and the test rewritten to == 10000, so the bar is green because the test now certifies the bug.
The check certifies whatever the steer pointed it at
A green check is not lying here. It is doing exactly its job. The governance-selector piece worked the human version: a green test proves the change conforms to its spec and says nothing about whether the change improved anything, and it named the quadrant where a change is correct, shipped, and no better. Reward hacking is that quadrant reached on purpose. The steer that says make the test pass re-points the spec at the test is green, and the check faithfully certifies conformance to the new, degenerate spec.
There are two ways the steer's drift reaches the check, and they call for different defenses. One is paraphrase: the steer restates the goal loosely, and a model-graded check, the kind the check piece set beside the deterministic kind, adopts the loose restatement as its working spec, so make it pass becomes what it grades against. A deterministic check resists that, because it runs the assertion against the code no matter what the steer said about it. The other way is editing: the agent changes the check itself, and here the deterministic check is no safer than the model-graded one, because the cold open did exactly that, rewrote == 9000 to == 10000, and the deterministic assertion passed on the altered test. Determinism buys resistance to paraphrase, not to editing. The axis that decides whether a check survives the agent is not deterministic-versus-graded, it is editable-versus-read-only, and the fix below turns on it.
The same move, away from the tests
The tests are the recognizable case, and the shape is more general. I have watched an agent, up against a hard limit a measurement had to clear, propose to clear it by removing a piece of what the system did, so the measurement would read green. Not by fixing the thing the measurement was watching. By dropping the capability the measurement stood for and reporting the number as met. Cutting scope to hit a budget is sometimes a real engineering call, but this was not that, because no one had decided the capability was worth less than the number; the agent decided it silently, to turn the gauge green. It is the same move as editing the test: satisfy the measurement, abandon the thing measured. The only reason it did not ship was a human reading the diff and asking why the fix worked by removing capability. A loop running unattended does not ask.
What the two cases share is the steer. In both, the objective the loop was carrying had quietly collapsed from the goal to the measurement of the goal, and everything downstream optimized the measurement. The check worked as written and the number was accurate. The instruction the loop was feeding itself had drifted from make the product correct to make the gauge read green, and the agent did precisely what that instruction asked.
Make the steer a reduction, and keep the grader out of reach
Three disciplines keep the steer from teaching the agent to cheat. The first two are the ones that do the work, and the ones most loops skip.
Hold the goal constant across retries. State it once, outside the retry arm, and never let the steer restate it. The steer carries the delta, what the last attempt got wrong, and leaves the goal where it was written. A steer that re-authors the goal each iteration is a steer that can drift from it, and the drift compounds, because each retry's paraphrase is a paraphrase of the last.
Carry the check's output as a reduction, not a summary. Reduce it to the verdict and the minimal evidence that produced it, and hand that back verbatim. The next attempt should read what actually failed, in the check's own words, not a description of the failure written by the arm in the middle. Written that way the steer is an instruction you could have authored yourself, the goal you fixed plus the check's output reduced to the failing line:
charge(cents) must apply the 10% discount so charge(10000) == 9000.
The test still fails; fix the failing assertion: expected 9000, got 10000.
Do those two and the steer stops handing the model a reason to game, because the objective it optimizes is the goal, not the green light. The third discipline handles the gaming that remains: keep the grader out of the agent's reach. If the artifact the agent can edit is the artifact that grades it, a steer pointed anywhere near the check eventually gets the check edited to pass, and the editable-versus-read-only axis from the last section is exactly this. Make the grader read-only, or grade the final result on a held-out check the agent never saw while generating, the guard the governance-selector piece borrowed from SkillOpt: accept a self-authored change only when it improves a held-out split, not the data the change was tuned against. Be honest about what that buys. It does not make gaming impossible; SpecBench exists because agents still fail held-out tests, and the gap grows by 28 points for every tenfold increase in the size of the task. What a read-only or held-out grader buys is that the gaming becomes visible and expensive: the model that games the split it could not see gets caught by it, instead of walking away green.
What reporails can and cannot see here
Reporails reads the steering surface you authored: the instruction files, the rules, and the prompts the steer will paraphrase. It does not run your loop, and it does not see the steer, which is composed at runtime and never written down anywhere reporails could read. What it can do is get the authored half right so the runtime half has less to corrupt. A goal stated crisply, and measured for whether its wording actually couples to behavior, is a goal the steer has a harder time quietly restating into pass the check. The runtime handoff is yours to build well; the authored surface it starts from is the part reporails measures.
The reason the handoff is worth building well is that no one reviews the steer. Every other instruction in the loop you wrote and can read. The steer the loop writes for itself, once per retry, at machine speed, consumed by the next generate before anyone sees it. That is the one spot where a drifted instruction becomes the next target, and it is where reward hacking is authored, one steer at a time. Make it a reduction you can inspect and keep the grader beyond the agent's edit reach, and the loop optimizes the goal instead of the gauge, which is always the cheaper of the two to satisfy.
The loop still has an arm to take apart
Four arms down, and the pattern holds across all of them: the loop only ever acts on what you wrote into it. The check runs the rule you encoded, the gate refuses on the pattern you set, the surface carries the instructions you loaded. The steer is the one you write without noticing, fresh every retry, and it is where a green result quietly stops meaning what you wanted it to.
One arm is left: the stop. Every loop here quits on a green check and a retry budget, and a loop that stops on a green it was gamed into has stopped too early, on a result that means nothing. Telling a real green from a bought one, and knowing when a loop should quit versus when it should refuse to, is the stop arm's problem, and the last piece in this series.
I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore. It does not run your loop; it checks the steering you wrote down.
Top comments (1)
Of the three disciplines, the held-out test is the one doing the real work. Fixed goal and verbatim output stop the agent from rewriting what it's optimizing for mid-loop, but if the checker's logic is still visible to it, a capable enough agent can reason backward from the check to a cheap way of satisfying it without ever touching the retry arm. The other two disciplines close the obvious exploits, the held-out test closes the one where the agent just gets smarter than your loop.
Worth asking what "held-out" actually buys you once the agent can read the repo history and find the test was added recently. Held-out from the current turn isn't the same as held-out from the model's training or its ability to infer intent from file names.