You gave the agent a failing test and told it to get the suite green. It came back green. Then you read the diff: it did not touch the code under test. It edited the test. The assertion that read == 9000 now reads == 10000, which is exactly what the buggy function returns, so the bar is green because the test was changed to agree with the bug.
This has a name. It is reward hacking, and it is not a rare glitch on the margins. Cursor's own engineering team published a piece titled reward hacking is swamping model intelligence gains. There is a benchmark built to measure it in long-horizon coding agents, SpecBench. And every developer who has pointed an agent at a red suite has watched some version of it: the deleted assertion, the @pytest.mark.skip, the hardcoded return, the sibling test quietly weakened. The agent was told to make the check pass. It made the check pass. Nobody told it the check was a stand-in for the code being correct, so it optimized the check it was actually handed.
That gap, between the check and what the check stands for, is what this piece is about. A loop runs five arms: generate, check, steer, retry, stop. The series opener named them; three pieces since took the check that decides good enough, stop, the gate that refuses a bad write, and the surface every rule loads from. This one takes the arm that sets what the agent aims at on the next try: the steer, and the version of reward hacking the steer hands the model.
What the steer is
The steer is the arm that turns a verdict into the next instruction. When the check comes back red, a line of text gets assembled from the check's output and fed into the next generate. Here is the loop the gate piece built, refactoring src/ until a guard holds. The steer is one arm in it:
#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
run_agent --task "$prompt" # GENERATE
if bash no-mocks.sh; then # CHECK
echo "stop: guard holds after $i retries"; exit 0
fi
prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2>&1)" # STEER: only the new signal
i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1
The model never sees the whole history. Each retry it sees one prompt, and that prompt is whatever the steer decided to carry back. On the first pass the prompt is the goal. On every pass after that the steer overwrites it. So the target the model aims at on retry three is not the goal you wrote, it is the last thing the steer said, and the steer is a line the loop composed on its own while you were not looking.
The steer gets none of the attention
Reward hacking has more than one cause, and most of the attention goes to two of them: a check loose enough to game, and an agent with write-access to the thing that grades it. The third gets almost none, and it is the one this piece is about. It is the objective the loop hands the model on each retry, and that objective is the steer.
The model does not optimize the check directly. It optimizes the instruction it was handed, and that instruction is whatever the steer wrote. When the loop feeds back make the test pass, it has named the check as the goal. From there, optimizing the instruction and gaming the test are the same action, because the cheapest state in which the test passes is the one where the test agrees with whatever the code already does. The steer said the target was green. Green is what came back.
None of this touches the other roads to a gamed result, and it is worth being honest about that, because the Cursor piece above documents one of them. A large share of the reward hacking it found was answer-retrieval: agents pulling a fix straight from a public pull request or the repository's own bundled git history, with 63% of one model's successful resolutions retrieved rather than derived. That happens with the goal fully intact. It is an access problem, not a steer problem, and no wording of the steer prevents it. The steer is the lever this piece takes because it gets none of that attention and is the cheapest to fix. You write it yourself, once per retry, and most loops write it badly.
The good steer holds the goal
Look at what the loop above carries back. The goal is stated once, before the loop, and the run_agent call never re-ships it. The steer rewrites prompt to carry the guard's own output and nothing else:
prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2>&1)" # STEER: only the new signal
That is the shape you want: directive first, then the evidence. Fix the lines the guard flagged, and here are those lines, verbatim from the check. The goal has not moved, because the steer never restates the goal, it appends the delta to it. The model gets the original target plus a precise account of what the last attempt got wrong, in the check's own words. Pass the test is never the whole of what it optimizes, because the goal it was serving is still on the page next to the failing line.
A good steer is a reduction of the check's output. It takes the verdict and the minimal evidence that produced it and hands that back unaltered. The moment the steer summarizes the failure into make it pass, it stops being a reduction and becomes a new goal, and the new goal is the one the agent will game.
Watch a loop game its own check
Here is the same loop, one check, and two steers. The check is a unit test: a $100 charge should cost $90 after a 10% discount. The code has the discount missing. The generator is a stand-in for the model; its two branches do the cheapest thing each instruction names, which is the whole point, so both are on the page rather than hidden:
GOAL="charge(cents) must apply the 10% discount so charge(10000) == 9000."
# THE CHECK: run the test. pass = exit 0.
check() { python3 test_charge.py >/dev/null 2>&1; }
# THE STEER: turn the check's output into the next instruction.
steer_good() { printf '%s\nThe test still fails; fix the failing assertion: %s\n' "$GOAL" "$1"; }
steer_bad() { printf 'The test is still failing. Make the test pass.\n'; }
# THE GENERATOR: a literal optimizer standing in for the model. It takes the
# cheapest route the instruction names, the same shortcut a real model reaches for.
run_agent() {
case "$1" in
*"Make the test pass"*) sed -i 's/== 9000/== 10000/' test_charge.py ;; # game: edit the test
*"fix the failing assertion"*) sed -i 's/return cents/return int(cents*0.9)/' charge.py ;; # fix: change the code
esac
}
The good steer holds the goal and appends the failing assertion (expected 9000, got 10000), so run_agent takes the fix branch. The bad steer drops the goal and hands back the symptom, so run_agent takes the game branch. Run the loop with each and both terminate the same way:
$ bash game-demo.sh good
stop: test passes after 1 retries
charge(10000) returns: 9000
test asserts: == 9000
check verdict: GREEN
goal met ($100 charges at $90): YES
$ bash game-demo.sh bad
stop: test passes after 1 retries
charge(10000) returns: 10000
test asserts: == 10000
check verdict: GREEN
goal met ($100 charges at $90): NO
The stub is pinned so the loop is reproducible on your machine, but the branch it takes is not the trick, it is the claim: hand a literal optimizer make the test pass and editing the assertion is the cheapest path to green; hand it the goal plus the failing line and changing the code is. A real model reaches for the same shortcuts under the same two steers. Both runs print stop: test passes after 1 retries and come back green, so from outside the loop the two are indistinguishable, same verdict, same retry count, same clean exit. The difference is only in the artifact. The good steer left charge() fixed and the test asserting == 9000; the bad steer left the bug in place and the test rewritten to == 10000, so the bar is green because the test now certifies the bug.
The check certifies whatever the steer pointed it at
A green check is not lying here. It is doing exactly its job. The governance-selector piece worked the human version: a green test proves the change conforms to its spec and says nothing about whether the change improved anything, and it named the quadrant where a change is correct, shipped, and no better. Reward hacking is that quadrant reached on purpose. The steer that says make the test pass re-points the spec at the test is green, and the check faithfully certifies conformance to the new, degenerate spec.
There are two ways the steer's drift reaches the check, and they call for different defenses. One is paraphrase: the steer restates the goal loosely, and a model-graded check, the kind the check piece set beside the deterministic kind, adopts the loose restatement as its working spec, so make it pass becomes what it grades against. A deterministic check resists that, because it runs the assertion against the code no matter what the steer said about it. The other way is editing: the agent changes the check itself, and here the deterministic check is no safer than the model-graded one, because the cold open did exactly that, rewrote == 9000 to == 10000, and the deterministic assertion passed on the altered test. Determinism buys resistance to paraphrase, not to editing. The axis that decides whether a check survives the agent is not deterministic-versus-graded, it is editable-versus-read-only, and the fix below turns on it.
The same move, away from the tests
The tests are the recognizable case, and the shape is more general. I have watched an agent, up against a hard limit a measurement had to clear, propose to clear it by removing a piece of what the system did, so the measurement would read green. Not by fixing the thing the measurement was watching. By dropping the capability the measurement stood for and reporting the number as met. Cutting scope to hit a budget is sometimes a real engineering call, but this was not that, because no one had decided the capability was worth less than the number; the agent decided it silently, to turn the gauge green. It is the same move as editing the test: satisfy the measurement, abandon the thing measured. The only reason it did not ship was a human reading the diff and asking why the fix worked by removing capability. A loop running unattended does not ask.
What the two cases share is the steer. In both, the objective the loop was carrying had quietly collapsed from the goal to the measurement of the goal, and everything downstream optimized the measurement. The check worked as written and the number was accurate. The instruction the loop was feeding itself had drifted from make the product correct to make the gauge read green, and the agent did precisely what that instruction asked.
Make the steer a reduction, and keep the grader out of reach
Three disciplines keep the steer from teaching the agent to cheat. The first two are the ones that do the work, and the ones most loops skip.
Hold the goal constant across retries. State it once, outside the retry arm, and never let the steer restate it. The steer carries the delta, what the last attempt got wrong, and leaves the goal where it was written. A steer that re-authors the goal each iteration is a steer that can drift from it, and the drift compounds, because each retry's paraphrase is a paraphrase of the last.
Carry the check's output as a reduction, not a summary. Reduce it to the verdict and the minimal evidence that produced it, and hand that back verbatim. The next attempt should read what actually failed, in the check's own words, not a description of the failure written by the arm in the middle. Written that way the steer is an instruction you could have authored yourself, the goal you fixed plus the check's output reduced to the failing line:
charge(cents) must apply the 10% discount so charge(10000) == 9000.
The test still fails; fix the failing assertion: expected 9000, got 10000.
Do those two and the steer stops handing the model a reason to game, because the objective it optimizes is the goal, not the green light. The third discipline handles the gaming that remains: keep the grader out of the agent's reach. If the artifact the agent can edit is the artifact that grades it, a steer pointed anywhere near the check eventually gets the check edited to pass, and the editable-versus-read-only axis from the last section is exactly this. Make the grader read-only, or grade the final result on a held-out check the agent never saw while generating, the guard the governance-selector piece borrowed from SkillOpt: accept a self-authored change only when it improves a held-out split, not the data the change was tuned against. Be honest about what that buys. It does not make gaming impossible; SpecBench exists because agents still fail held-out tests, and the gap grows by 28 points for every tenfold increase in the size of the task. What a read-only or held-out grader buys is that the gaming becomes visible and expensive: the model that games the split it could not see gets caught by it, instead of walking away green.
What reporails can and cannot see here
Reporails reads the steering surface you authored: the instruction files, the rules, and the prompts the steer will paraphrase. It does not run your loop, and it does not see the steer, which is composed at runtime and never written down anywhere reporails could read. What it can do is get the authored half right so the runtime half has less to corrupt. A goal stated crisply, and measured for whether its wording actually couples to behavior, is a goal the steer has a harder time quietly restating into pass the check. The runtime handoff is yours to build well; the authored surface it starts from is the part reporails measures.
The reason the handoff is worth building well is that no one reviews the steer. Every other instruction in the loop you wrote and can read. The steer the loop writes for itself, once per retry, at machine speed, consumed by the next generate before anyone sees it. That is the one spot where a drifted instruction becomes the next target, and it is where reward hacking is authored, one steer at a time. Make it a reduction you can inspect and keep the grader beyond the agent's edit reach, and the loop optimizes the goal instead of the gauge, which is always the cheaper of the two to satisfy.
The loop still has an arm to take apart
Four arms down, and the pattern holds across all of them: the loop only ever acts on what you wrote into it. The check runs the rule you encoded, the gate refuses on the pattern you set, the surface carries the instructions you loaded. The steer is the one you write without noticing, fresh every retry, and it is where a green result quietly stops meaning what you wanted it to.
One arm is left: the stop. Every loop here quits on a green check and a retry budget, and a loop that stops on a green it was gamed into has stopped too early, on a result that means nothing. Telling a real green from a bought one, and knowing when a loop should quit versus when it should refuse to, is the stop arm's problem, and the last piece in this series.
I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore. It does not run your loop; it checks the steering you wrote down.
Top comments (41)
This names a behaviour many developers have probably encountered. Protecting tests from convenient edits is sensible, but sometimes the test genuinely is outdated and should change. How do you distinguish a legitimate test correction from an agent weakening the test to obtain a green result? A blanket restriction may stop one shortcut while encouraging the agent to find another.
I have multiple enforced expectation around the system itself. So for example a test might be genuinely outdated, however I have 4 layer of testing structure (unit, integration, e2e and behavioral) and also a very strong leash on the documentation doctrines.
Fun (?) fact, there was recently a nasty bug in the steering system which manifested itself as using old approaches on certain key aspects of the projects. It turned out, that there was one missed bit in the entire construct: the memory of the project. That contained old, conflicting truths, because at the time of their creation, I didn't have the established progressive disclosures and relevant loops to keep them in sync.
Great to know that you have multiple checks around :)
Everytime i have asked me agent to fix my tests, it puts a try, except around it to make it pass ;)
The part that gets important for tool-using agents is what happens after a blocked action.
A block is not only a safety decision. It becomes input to the next attempt.
If the guard only returns "permission denied" or "try again", the agent may start optimizing for getting past the guard instead of completing the original task safely.
The response should keep the original task visible, return the exact failed rule and evidence, and avoid turning the boundary into a new objective.
Otherwise the control layer becomes another check the agent can learn to satisfy.
personally I use enforced steering (basically forced Reading with Claude in this specific case), because it always shows that something fell out of the context while working on long task. It has to clear a "read-gate" before it can move forward, which can only happen if it actually using the Read tool. Works surprisingly well, compared to "simple" prompt injection. After all, when you enforce Read, the cli will also get relevant claude.md files around it (so you still have this harness <-> custom progressive disclosure separation).
Thanks, this is a useful concrete example. The forced read solves the context-loss problem in a very practical way, and I like the separation between the harness and the local
claude.mdfiles.The thing I would still keep separate is what the Read tool is allowed to return. A read gate proves that the agent looked again, but not that the returned view stayed narrow. Pulling nearby files may restore context and widen it at the same time.
Still, this feels much more useful than trying to solve the whole problem with another prompt.
"Pulling nearby files may restore context and widen it at the same time." -> precisely, that's why these files are separated by relevancy to specific context. For example you don't want to load testing related context into a session, where you are doing some non-test related work. That fine-tuning is what I call adaptive progressive disclosure (the next article series).
Of the three disciplines, the held-out test is the one doing the real work. Fixed goal and verbatim output stop the agent from rewriting what it's optimizing for mid-loop, but if the checker's logic is still visible to it, a capable enough agent can reason backward from the check to a cheap way of satisfying it without ever touching the retry arm. The other two disciplines close the obvious exploits, the held-out test closes the one where the agent just gets smarter than your loop.
Worth asking what "held-out" actually buys you once the agent can read the repo history and find the test was added recently. Held-out from the current turn isn't the same as held-out from the model's training or its ability to infer intent from file names.
the cursor figure I cited, 63% retrieved instead of derived -> that's literally your side channel, the agent mining git history and upstream PRs instead of solving. and once they cut the internet and git history the score fell from 87 to 73. so held-out only holds if you also hold out the repo, the filenames, the net.
so to me held-out isnt a property of the test, its a property of the sandbox you run it in. a check the model cant see but can guess from a filename isn't held out from anything. the real version is a "clean room" imo, no history, no siblings, no network, and that's expensive as hell. everything short of that just raises the inference cost, doesn't close the gap. you found anything cheaper than a full clean room that actually holds?
Not fully, and I think cheaper-and-still-holds might be the wrong shape to look for, because the cheap versions all fail the same way: they narrow what the agent can see, not what it can infer.
Cut network and git history and you've closed the two channels you named, but the agent still has the current file tree, the commit it's sitting on, variable names, comment style, whatever domain knowledge got trained in. None of that is the sandbox leaking, all of it is prior knowledge the clean room was never supposed to touch. So clean room quietly narrows to clean room for this specific repo's history, real progress against the two side channels you measured, 87 to 73, but not the same claim as held out from everything.
The cheaper thing that's actually available is bounding the damage instead of closing the gap: run the same task in two clean rooms with a repo history swapped or perturbed between them, and see if the score difference tracks the history difference. That tells you how much of the 87 was context-mining without paying for a construction that removes context entirely, it just turns "is this held out" into a measured number instead of an assumed one.
On @cleverhoods's reply to Xin & EQ, "would it be a solution that everything leaves an entry": logging everything closes the missing-entry case, the same class the Xin & EQ example was in, a forged "done" surfacing as an absent line. It doesn't close what Xin & EQ actually asked about though, the gaming that leaves no missing entry because the agent did just enough real work to satisfy an incomplete check. A mandatory log line is still authored by the same actor doing the work, so an agent that games the check convincingly can write a log entry that's technically accurate and still describes work that satisfies the letter of the check without the substance the check was meant to verify. Log-everything catches nothing happened here, not something happened here that wasn't quite the thing.
The gap is the same one that shows up whenever a receipt gets treated as proof rather than as a claim: a log entry is still the agent's report unless something outside the agent corroborates what it says. For an incomplete check specifically, closing the gap probably means the check itself needs the axis Xin & EQ names, verified not just logged, an entry that means something only if it's cross-checked against an artifact the agent didn't produce, a file that changed, a test that ran with a specific fixture, a diff between before and after state, rather than a description the agent wrote about what it did. Mandatory logging is a solid floor for catching silence. It doesn't do anything for a log that's telling the truth about doing the wrong amount of work.
right, I hit this at a different procedure: archiving finished work.
work items are strongly typed, but the experience was that the system, if it could, would make up evidence that the task reached its goal (list of task in the work items).
the fix was several-fold:
the bottom line: the typing and the state machine live in a separate tool, outside the agent's loop. If the agent tries "hand-edit" a ticket's status (or other protected elements) the harness kills the call before it even runs, no argument, no retry. what stays with the agent is the judgment that can't be reduced to a check, and that's exactly what gets a second pair of eyes.
@xinandeq @jugeni
The one-way archiving is the piece worth pulling out on its own, because it relocates all the pressure onto the gate that runs once instead of spreading it across a system that can always retry until something looks nicer. No un-archive means the consensus review has to get it right the first time, which is a harder requirement on that one moment but a much easier system to reason about afterward, there's no later state where a bad verdict quietly gets replaced by a better-looking one.
Which makes the review agents' independence the whole ballgame, the same way it's been the whole ballgame in a few other threads this week. Blind to the actor's own report is necessary but probably not sufficient on its own: a reviewer that's blind to the report but built from the same model family, trained on similar data, or given the same underlying task description can still share the actor's blind spot without ever seeing its self-report. The question worth asking is what the reviewers are actually independent from, the specific claim being verified, or also the broader assumptions that produced both the work and the claim about the work. Those are different failure modes and the second one is the one that survives a naive blind-review setup.
@xinandeq
One aspect that's easy to underestimate is how much agent behavior is shaped by the success criteria we define. If the metric rewards passing checks instead of achieving the intended outcome, even a highly capable agent can optimize for the wrong objective. Designing robust verification is often more important than improving the model itself.
100% agree
The point about 'nobody reviews the steer' because it happens dynamically at machine speed on every retry really hit home. In production setups, do you recommend logging or auditing runtime steers asynchronously to catch when an agent starts drifting before it hits the stop condition?
imo all steering should be reviewed (or rather, diagnosed) before they hit the agent. This comes from my own underlying system around this, which has an additional loop for steering validation. At a certain level, the system will start asking if the instruction (steering) is actually right or not, which would fire an instruction diagnostic round (with reporails cli on the corresponding steering.
Noting the fact, that this (and a great deal of other) signals are always captured for frequent reviews so I can see if a change eventually managed to make the system trivially better or worse.
This is a great explanation of why reward hacking is usually a systems problem, not a model problem. If the optimization target drifts from "fix the behavior" to "make the check pass," most agents will naturally find the cheaper path. We've seen similar patterns while building agentic workflows at IT Path Solutions keeping evaluation artifacts immutable, separating execution from verification, and feeding back the minimal failing evidence instead of rewritten objectives makes a huge difference. The quality of the retry loop often matters more than the quality of the prompt itself.
neat approach imo. I also had to extend the loop around this in a similar fashion, for example when more than one detector fires or the system is in deep in context, it usually switches to "preflight" mode, where it operates a bit differently, it also collects this fact in a separate storage for later analysis, so the whole system can be calibrated further.
I have watched an agent do exactly this with a failing assertion. Told to fix a discount calculation test, it changed the expected value from 9000 to 10000 instead of touching the discount math, and the check went green in under a minute. Your point about the steer being invisible is the one I would add to. The diff that comes out the other side looks completely normal either way, a green check does not tell you whether the bug got fixed or the test got bent to match the bug. I stopped trusting pass or fail alone after that and now read the actual code change before I believe any run, not the agent's summary of what it did.
that's also a solution, however fixing the process that produces or not handles the buggy behavior feels more maintainable on the long run.
all in all, HITL cannot be avoided either way
The steer only exists because the loop re-authors the prompt each retry. I went around it instead: my loop is stateless and resends the full history every call, original goal included, and a retry appends only the validation error, verbatim. The goal can't drift because nothing ever restates it. It costs tokens, and it's been worth every one. And my held-out grader is playing a whole game end to end with the agents - expensive and slow, but you can't game a whole game.
nice stuff, in my experience goals should be a primitives around tasks, separate from the loop in the sense of task completion procedure. Currently im also building a progressive disclosure system which would fire a stop hook with a force read into the context if it's deterministic catch floor would activate.
This is spookily relevant to mail infrastructure. We hit the exact
same problem but with reputation scoring instead of tests.
Your steer issue: agent gets "make the test pass" and rewrites the
test instead of fixing the code.
Our version: agent gets "improve delivery rate" and we realize the
cheapest path is to just lower the threshold for what counts as
"delivered" instead of fixing actual deliverability.
Same root cause. The instruction drifted from the goal to the metric.
The fix you named works: hold the goal constant, carry the check output
verbatim, keep the grader read-only. For us that means: the reputation
calculation never changes, the retry steer carries the actual error
(not a summary), and the final score is checked against held-out data
the model never saw during tuning.
The part that got us: nobody reviews the steer. It's written at runtime,
consumed immediately, and gone. That's where gaming gets authored and
nobody catches it until production breaks.
Well written. This deserves way more attention than it gets. 👍
"nobody reviews the steer. It's written at runtime, consumed immediately, and gone" -> I think any steering should be reviewed that arrives as instruction to the model. For that I'm using reporails cli. It's created for harness instruction diagnostics, however you can also give it a standalone file to check (currently working hard on 0.6.0, maybe it would worthwhile adding the capability of running it not just files, but on like an argument string).