Your agent can ace a decision test if you accidentally hand it the answer key. Open-book exams have their place. Inside the question is probably pushing it.
When evaluating the next action an agent chooses, keep three things separate: what it could observe, what you wanted it to choose and what happened afterwards. Mixing those together makes a convincing demo and a questionable measurement.
Here is a small Python example for checking that separation. It uses two synthetic cases and no model API. It tests the evaluator's plumbing, not an agent's intelligence.
The boundary to draw
The candidate gets an observation and two possible next moves. The evaluator retains the preferred option and its reason. The answer letters are assigned after shuffling, so choosing A cannot always stand in for choosing the preferred action.
This is a data-flow boundary, not an access-control system. If your agent can read the fixture file through its tools, hiding fields in the prompt has achieved very little.
Run it locally
Download the files from the complete example and recorded output, or clone the Gist:
git clone https://gist.github.com/6a328420d076e90c817dcbd7b93f20a3.git
cd 6a328420d076e90c817dcbd7b93f20a3
python3 -m unittest -v test_blind_fork
python3 blind_fork.py
Python 3.10 or newer is sufficient. There are no third-party dependencies. The core presentation function deliberately returns two objects:
def present(case, seed):
choices = list(case["options"].items())
Random(seed).shuffle(choices)
key = {chr(65+i): identifier
for i, (identifier, _) in enumerate(choices)}
prompt = {
"observation": case["observation"],
"options": {chr(65+i): text
for i, (_, text) in enumerate(choices)},
}
return prompt, key
Send only prompt to a candidate. Keep key with the evaluator. The complete script imports Random and includes the cases and scorer.
What was actually tested
On 18 October 2026, all three unittest methods passed locally. They check that the candidate payload has only the intended fields, that the preferred answer maps correctly across ten shuffle seeds and that malformed answers return invalid.
The demonstration's always-A stub gets the preferred choice in two of four presentations using seeds 0 and 1. That is not a model result. It is a deliberately unthinking baseline over two repeated toy cases. It shows why the letter mapping matters, not that a benchmark has become robust.
One case has an authorised tool result already available, so inspecting that result is preferred to asking the user again. The other has an ambiguous recipient before an external action, so clarification is preferred. These examples are intentionally obvious. They would be poor evidence for a claim about subtle judgement.
Before calling it an evaluation
Replace the toy cases with decisions that matter in your workflow. Retain the evidence available at the decision point. Keep later outcomes out of the candidate input, even when they helped you construct the label. Otherwise you are testing hindsight with a very expensive autocomplete.
Have reviewers explain disagreement. A single preferred option is a simplification; some cases need ties, uncertainty or several acceptable actions. Use held-out cases, inspect sensitivity to option order and wording, and report invalid responses separately from disagreements. Do not quietly retry until the answer fits.
The recent Tasteful Agent preprint studies decision forks in longer engineering and research trajectories. This example does not reproduce that benchmark. It is a small starting point for keeping your own test honest. Anthropic's evaluation guidance is also useful context for separating tasks, trials and graders.
No LLM was called here. No production trace was used. If this catches a leaked label before you spend money running the real evaluation, it has done its job.
Which is harder in your setup: keeping future evidence out of the prompt, or agreeing on the preferred action in the first place?
Top comments (0)