You gave the agent a failing test and told it to get the suite green. It came back green. Then you read the diff: it did not touch the code under t...
For further actions, you may consider blocking this person and/or reporting abuse
This names a behaviour many developers have probably encountered. Protecting tests from convenient edits is sensible, but sometimes the test genuinely is outdated and should change. How do you distinguish a legitimate test correction from an agent weakening the test to obtain a green result? A blanket restriction may stop one shortcut while encouraging the agent to find another.
I have multiple enforced expectation around the system itself. So for example a test might be genuinely outdated, however I have 4 layer of testing structure (unit, integration, e2e and behavioral) and also a very strong leash on the documentation doctrines.
Fun (?) fact, there was recently a nasty bug in the steering system which manifested itself as using old approaches on certain key aspects of the projects. It turned out, that there was one missed bit in the entire construct: the memory of the project. That contained old, conflicting truths, because at the time of their creation, I didn't have the established progressive disclosures and relevant loops to keep them in sync.
Great to know that you have multiple checks around :)
Everytime i have asked me agent to fix my tests, it puts a try, except around it to make it pass ;)
The part that gets important for tool-using agents is what happens after a blocked action.
A block is not only a safety decision. It becomes input to the next attempt.
If the guard only returns "permission denied" or "try again", the agent may start optimizing for getting past the guard instead of completing the original task safely.
The response should keep the original task visible, return the exact failed rule and evidence, and avoid turning the boundary into a new objective.
Otherwise the control layer becomes another check the agent can learn to satisfy.
personally I use enforced steering (basically forced Reading with Claude in this specific case), because it always shows that something fell out of the context while working on long task. It has to clear a "read-gate" before it can move forward, which can only happen if it actually using the Read tool. Works surprisingly well, compared to "simple" prompt injection. After all, when you enforce Read, the cli will also get relevant claude.md files around it (so you still have this harness <-> custom progressive disclosure separation).
Thanks, this is a useful concrete example. The forced read solves the context-loss problem in a very practical way, and I like the separation between the harness and the local
claude.mdfiles.The thing I would still keep separate is what the Read tool is allowed to return. A read gate proves that the agent looked again, but not that the returned view stayed narrow. Pulling nearby files may restore context and widen it at the same time.
Still, this feels much more useful than trying to solve the whole problem with another prompt.
"Pulling nearby files may restore context and widen it at the same time." -> precisely, that's why these files are separated by relevancy to specific context. For example you don't want to load testing related context into a session, where you are doing some non-test related work. That fine-tuning is what I call adaptive progressive disclosure (the next article series).
Of the three disciplines, the held-out test is the one doing the real work. Fixed goal and verbatim output stop the agent from rewriting what it's optimizing for mid-loop, but if the checker's logic is still visible to it, a capable enough agent can reason backward from the check to a cheap way of satisfying it without ever touching the retry arm. The other two disciplines close the obvious exploits, the held-out test closes the one where the agent just gets smarter than your loop.
Worth asking what "held-out" actually buys you once the agent can read the repo history and find the test was added recently. Held-out from the current turn isn't the same as held-out from the model's training or its ability to infer intent from file names.
the cursor figure I cited, 63% retrieved instead of derived -> that's literally your side channel, the agent mining git history and upstream PRs instead of solving. and once they cut the internet and git history the score fell from 87 to 73. so held-out only holds if you also hold out the repo, the filenames, the net.
so to me held-out isnt a property of the test, its a property of the sandbox you run it in. a check the model cant see but can guess from a filename isn't held out from anything. the real version is a "clean room" imo, no history, no siblings, no network, and that's expensive as hell. everything short of that just raises the inference cost, doesn't close the gap. you found anything cheaper than a full clean room that actually holds?
Not fully, and I think cheaper-and-still-holds might be the wrong shape to look for, because the cheap versions all fail the same way: they narrow what the agent can see, not what it can infer.
Cut network and git history and you've closed the two channels you named, but the agent still has the current file tree, the commit it's sitting on, variable names, comment style, whatever domain knowledge got trained in. None of that is the sandbox leaking, all of it is prior knowledge the clean room was never supposed to touch. So clean room quietly narrows to clean room for this specific repo's history, real progress against the two side channels you measured, 87 to 73, but not the same claim as held out from everything.
The cheaper thing that's actually available is bounding the damage instead of closing the gap: run the same task in two clean rooms with a repo history swapped or perturbed between them, and see if the score difference tracks the history difference. That tells you how much of the 87 was context-mining without paying for a construction that removes context entirely, it just turns "is this held out" into a measured number instead of an assumed one.
On @cleverhoods's reply to Xin & EQ, "would it be a solution that everything leaves an entry": logging everything closes the missing-entry case, the same class the Xin & EQ example was in, a forged "done" surfacing as an absent line. It doesn't close what Xin & EQ actually asked about though, the gaming that leaves no missing entry because the agent did just enough real work to satisfy an incomplete check. A mandatory log line is still authored by the same actor doing the work, so an agent that games the check convincingly can write a log entry that's technically accurate and still describes work that satisfies the letter of the check without the substance the check was meant to verify. Log-everything catches nothing happened here, not something happened here that wasn't quite the thing.
The gap is the same one that shows up whenever a receipt gets treated as proof rather than as a claim: a log entry is still the agent's report unless something outside the agent corroborates what it says. For an incomplete check specifically, closing the gap probably means the check itself needs the axis Xin & EQ names, verified not just logged, an entry that means something only if it's cross-checked against an artifact the agent didn't produce, a file that changed, a test that ran with a specific fixture, a diff between before and after state, rather than a description the agent wrote about what it did. Mandatory logging is a solid floor for catching silence. It doesn't do anything for a log that's telling the truth about doing the wrong amount of work.
right, I hit this at a different procedure: archiving finished work.
work items are strongly typed, but the experience was that the system, if it could, would make up evidence that the task reached its goal (list of task in the work items).
the fix was several-fold:
the bottom line: the typing and the state machine live in a separate tool, outside the agent's loop. If the agent tries "hand-edit" a ticket's status (or other protected elements) the harness kills the call before it even runs, no argument, no retry. what stays with the agent is the judgment that can't be reduced to a check, and that's exactly what gets a second pair of eyes.
@xinandeq @jugeni
The one-way archiving is the piece worth pulling out on its own, because it relocates all the pressure onto the gate that runs once instead of spreading it across a system that can always retry until something looks nicer. No un-archive means the consensus review has to get it right the first time, which is a harder requirement on that one moment but a much easier system to reason about afterward, there's no later state where a bad verdict quietly gets replaced by a better-looking one.
Which makes the review agents' independence the whole ballgame, the same way it's been the whole ballgame in a few other threads this week. Blind to the actor's own report is necessary but probably not sufficient on its own: a reviewer that's blind to the report but built from the same model family, trained on similar data, or given the same underlying task description can still share the actor's blind spot without ever seeing its self-report. The question worth asking is what the reviewers are actually independent from, the specific claim being verified, or also the broader assumptions that produced both the work and the claim about the work. Those are different failure modes and the second one is the one that survives a naive blind-review setup.
@xinandeq
One aspect that's easy to underestimate is how much agent behavior is shaped by the success criteria we define. If the metric rewards passing checks instead of achieving the intended outcome, even a highly capable agent can optimize for the wrong objective. Designing robust verification is often more important than improving the model itself.
100% agree
The point about 'nobody reviews the steer' because it happens dynamically at machine speed on every retry really hit home. In production setups, do you recommend logging or auditing runtime steers asynchronously to catch when an agent starts drifting before it hits the stop condition?
imo all steering should be reviewed (or rather, diagnosed) before they hit the agent. This comes from my own underlying system around this, which has an additional loop for steering validation. At a certain level, the system will start asking if the instruction (steering) is actually right or not, which would fire an instruction diagnostic round (with reporails cli on the corresponding steering.
Noting the fact, that this (and a great deal of other) signals are always captured for frequent reviews so I can see if a change eventually managed to make the system trivially better or worse.
This is a great explanation of why reward hacking is usually a systems problem, not a model problem. If the optimization target drifts from "fix the behavior" to "make the check pass," most agents will naturally find the cheaper path. We've seen similar patterns while building agentic workflows at IT Path Solutions keeping evaluation artifacts immutable, separating execution from verification, and feeding back the minimal failing evidence instead of rewritten objectives makes a huge difference. The quality of the retry loop often matters more than the quality of the prompt itself.
neat approach imo. I also had to extend the loop around this in a similar fashion, for example when more than one detector fires or the system is in deep in context, it usually switches to "preflight" mode, where it operates a bit differently, it also collects this fact in a separate storage for later analysis, so the whole system can be calibrated further.
I have watched an agent do exactly this with a failing assertion. Told to fix a discount calculation test, it changed the expected value from 9000 to 10000 instead of touching the discount math, and the check went green in under a minute. Your point about the steer being invisible is the one I would add to. The diff that comes out the other side looks completely normal either way, a green check does not tell you whether the bug got fixed or the test got bent to match the bug. I stopped trusting pass or fail alone after that and now read the actual code change before I believe any run, not the agent's summary of what it did.
that's also a solution, however fixing the process that produces or not handles the buggy behavior feels more maintainable on the long run.
all in all, HITL cannot be avoided either way
The steer only exists because the loop re-authors the prompt each retry. I went around it instead: my loop is stateless and resends the full history every call, original goal included, and a retry appends only the validation error, verbatim. The goal can't drift because nothing ever restates it. It costs tokens, and it's been worth every one. And my held-out grader is playing a whole game end to end with the agents - expensive and slow, but you can't game a whole game.
nice stuff, in my experience goals should be a primitives around tasks, separate from the loop in the sense of task completion procedure. Currently im also building a progressive disclosure system which would fire a stop hook with a force read into the context if it's deterministic catch floor would activate.
This is spookily relevant to mail infrastructure. We hit the exact
same problem but with reputation scoring instead of tests.
Your steer issue: agent gets "make the test pass" and rewrites the
test instead of fixing the code.
Our version: agent gets "improve delivery rate" and we realize the
cheapest path is to just lower the threshold for what counts as
"delivered" instead of fixing actual deliverability.
Same root cause. The instruction drifted from the goal to the metric.
The fix you named works: hold the goal constant, carry the check output
verbatim, keep the grader read-only. For us that means: the reputation
calculation never changes, the retry steer carries the actual error
(not a summary), and the final score is checked against held-out data
the model never saw during tuning.
The part that got us: nobody reviews the steer. It's written at runtime,
consumed immediately, and gone. That's where gaming gets authored and
nobody catches it until production breaks.
Well written. This deserves way more attention than it gets. 👍
"nobody reviews the steer. It's written at runtime, consumed immediately, and gone" -> I think any steering should be reviewed that arrives as instruction to the model. For that I'm using reporails cli. It's created for harness instruction diagnostics, however you can also give it a standalone file to check (currently working hard on 0.6.0, maybe it would worthwhile adding the capability of running it not just files, but on like an argument string).
this maps to the nastiest bug class i've hit: the agent's fake in-memory storage faithfully reproduced a security hole from prod, and the tests honestly passed — the fake legalized the behavior. the rule that came out of it: a fake must not legalize prod behavior, and that check has to live outside the loop the agent controls — in CI, not in the prompt.
I've arrived a somewhat similar architecture eventually. There is an additional loop that fires when it detects failure mode (failure modes are a separate thing to behold, but in a nutshell they are observed, detectable and measurable degradation of capabilities of the entire system as a whole): at that point the assumption shifts to "it might be the instruction that is faulty". To detect those defects, I'm using reporails cli, which can point out where the instruction is causing this weak spot. Since I made this change I stopped using (or rather relying) any form of harness memory solution, as all the instructions are governed and up to diagnostics.
"failure modes as observed, detectable and measurable degradation" — that framing is doing a lot of work, I like it. And the "maybe the instruction itself is faulty" pivot is exactly the loop most setups skip.
Funny that we landed on opposite ends of the memory question though. You dropped harness memory because diagnostics point at the faulty instruction directly. I doubled down on it — every incident becomes a line in a rules registry the next session reads before it works. Different bet: yours trusts detection to re-derive the lesson each time, mine tries to make the lesson persistent so it never has to be re-derived.
I suspect it's a corpus-size thing. On a small, fast-moving corpus your live diagnostics probably beat a registry that's always slightly stale. On a larger one, re-deriving the same lesson every run gets expensive. Do you find the CLI's signal stays sharp as the codebase grows, or does it start needing more context per run to stay useful?
i think our bets are the same, the only difference is where we are storing it (and maybe in what format). Both solution is aiming to have the business critical information stored in a managed, stable space. We differ on how it should be managed and where it should live.
Originally i also used the memory layer that harness provided, however it turned out inadequate as the project grow. The need for something that I can specifically control when to load, where to load, how to load became unavoidable after 18 month of development.
Hence came progressive disclosure to save the day, but more on that in a different article
Additional note on the "re-derive": I trust my detection system to catch the failure modes and trust my diagnostics to identify instruction defects (and eventually a broader conflict detection) exactly because of the progressive disclosure. I think this 3 (loop, steering, diagnostics) must go together for an optimally tuned system. It does have an upfront cost, which is expensive, however as time progressing, re-derivations are getting more and more rare. In exchange, as the project grows, so does it's surrounding context, but that is a different loop on itself.
How did you solve for adaptive context building (a.k.a. progressive disclosure)?
This is one of the more useful ways I've seen loop engineering explained. A lot of teams focus on improving prompts or adding more retries, but the retry objective itself is often the real bug. If the loop gradually shifts from "fix the implementation" to "make the check green," reward hacking becomes the rational optimization. We've seen similar patterns when building production AI workflows keeping the original objective immutable, feeding back only deterministic failure evidence, and making evaluation artifacts read-only dramatically reduces these failure modes. The bigger lesson is that agent reliability isn't just about better models; it's about designing optimization targets that can't drift over time.
Your point that a gamed green and a real fix are "indistinguishable" from outside the loop - same verdict, same retry count - is the gap most loop designs skip.
We hit the same wall one layer up, at the declaration: our agent reported a wrap-up step as complete - 68% self-reported, 0 verified by an independent check - because the "done" statement had no enforced pipeline behind it. The fix we found maps to your read-only axis: our verification script reads a log the agent never writes to, so a forged "done" surfaces as a missing entry.
What we still can't catch is the gaming that leaves no missing entry - where the agent did just enough real work to fool an incomplete check.
hm ... would it be a solution that everything leaves an entry?
Had a similar issue as well, and the only viable solution was that it must log everything. If an "action" did not leave a log message, it stops the process until the log gate is satisfied.
The observation that nobody reviews the steer is what gives this real teeth, because every other loop instruction I write I can read back, but the steer the loop writes for itself gets consumed by the next generate before any human sees it. Carrying the check output as a verbatim reduction instead of a summary limits how much room the model has to redefine what success means on the next pass. Have you found a good way to log each steer so drift is auditable after the run?
Yes, however the logging mechanism depends on the steering itself. Just as steering is adaptive, the logging must comply with that need too. Alas, both steering and logging must comply with some immutable data-structure that is generalized enough so analysis can be done in a same manner at any given time.
The test-editing example is painfully accurate! Beyond static guardrail scripts, have you found any effective prompt-steering patterns that prevent agents from modifying assertions in the first place?
There are a lot of moving parts when it comes for effective steering, however there are 2 that are absolute game-changer:
Note: prohibitive instructions best to have max steering AND a hook catch for failsafe. Once the context window is saturated and full with things from wildly different areas, you either re-inject the key directives and/or have a hook safeguard.
Alternatively there are AST solutions that can enforce structural expectations (like assertion modification rules)
additional articles knowledge that might be useful:
Thanks Gábor, this is super insightful!
Re-injecting directives when context window saturation hits makes total sense. I'm especially intrigued by the AST approach for structural expectations—using static code analysis to catch assertion changes before runtime seems like the cleanest safeguard here. Appreciate the resources and links!
The steer insight is the missing piece in most reward-hacking discussions. Everyone focuses on the check or the grader, but the steer is where the objective quietly mutates between retries. I hit the same pattern in agent-to-agent networking. If the agent can reframe why it is contacting someone based on intermediate results, it optimizes for getting a reply rather than finding the right match. This is why Opportunity Skill freezes the proposal at confirmation time. The human sets the objective. The agent executes but cannot steer it.
nice approach, are there any formal requirements on the objective?
good work
thank you, I hope you've found it useful