I built a blind AI pipeline to catch silent decisions in AI-written code. A single prompt did just as well.

TL;DR: I spent a few weeks building a pipeline that finds the decisions an AI coding agent makes without telling you. Its core idea was that the reader must never see the plan. Then I ran the comparison I'd been putting off: a single prompt that just asks a model to review the code against the plan. On my two test cases it found as many silent decisions, with better precision, in about a minute and at roughly a seventh of the cost. Here's the idea, how the pipeline works, and why it lost.

The problem: code that quietly decides things

You write a plan. An AI agent implements it. The code is clean and every test passes. Somewhere inside it, the agent decided that withdrawals are capped at 5,000 a day, that amounts round half-up, and that opening an account that already exists does nothing. None of that was in the plan.

I call these silent decisions. Each one is reasonable on its own. The agent had to pick something, and it picked without asking.

The tests don't catch them, because the tests came from the same reading of the same plan. If the agent decided overdrafts are rejected, its tests check that overdrafts are rejected. The suite is consistent with itself, and you find out it was the wrong call after deploy.

Code review doesn't reliably catch them either. A reviewer reads the code with the plan in mind, and a rule that looks sensible reads as intended. The question "did anyone actually decide this?" rarely gets asked.

The idea: start from the code, not the plan

Plan-compliance checks, like Traycer's verification pass or Qodo's ticket compliance, start from the plan and ask does the code do what was asked? I wanted to ask the opposite question: what does the code do that nobody asked for?

That means the thing you check has to be derived from the code, not from the plan. The plan tells you what was intended. The code is the only record of what was actually built.

It also changes the shape of the problem. "What's missing from this plan?" is an open-ended search over everything that could have been specified. "Here are 40 things this code does. Which of them can you point to in the plan?" is a matching problem over two concrete lists. Matching is something you can check.

The catch is that whoever describes the code must not have read the plan. If they have, they'll describe the code in the plan's words and every decision will look intended. So the describer has to be blind.

That was the hypothesis the whole pipeline is built on. As you'll see, the data didn't support it.

The pipeline

The tool is called silent-decisions. It runs after the agent has finished implementing, in six steps. The plan only enters at step 4, and even there no model sees more than one sentence of it at a time.

1. Prepare. Copy the source into a "clean room" with comments, tests, docs and git history removed. Agent-written code often carries the plan in its comments (// per spec: reject overdrafts), and that would break blindness. A plain AST walk also lists every decision point: literals, comparisons, rejection paths, rounding calls.
2. Extract, blind. A fresh agent that has never seen the plan reads the clean room and writes what the code does as concrete scenarios with real values. Each scenario comes with a contrary: the outcome a different, equally reasonable implementation would give.
3. Run everything. Every scenario becomes a test and is executed. Each also runs with a perturbed twin that asserts the wrong outcome, and the twin must fail. What survives is a verified description of behaviour, not a model's opinion.
4. Trace back to the plan. For every pair (plan sentence, scenario), a small decision-only classifier (Jev, from TypeSafe AI) answers one question: does this sentence require what the code does, require the contrary, or neither? It's asked twice with the two options swapped, and answers that move with the position are thrown away. A scenario with no sentence behind it is a silent decision.
5. Probe forwards. Every behavioural plan sentence gets a probe test. If the probe fails, the code dropped a requirement.
6. Control the run. A few plan sentences are hidden and the trace is repeated: behaviours that depended on them must come back unsourced. A canary string checks that the extractor really was blind. If a control fails, the run doesn't count.

What you review at the end is only the delta: the silent decisions and the dropped requirements. Each one is a card that asks you to choose:

Withdrawals are capped at 5,000 per day Given an account with balance 10,000 that has already withdrawn 4,000 When a further 1,500 is withdrawn Then the withdrawal is rejected with reason daily-limit

A. What the code does: rejected, balance stays 6,000 B. The alternative: accepted; there is no limit

You pick A or B, and your answer becomes a signed regression test. You read a handful of decisions instead of the whole diff.

The experiment

A pipeline this elaborate only makes sense if it finds things that cheaper methods miss. So I designed six "arms" to run on the same plan and code, and scored them against the same answer key.

Arm	What it does	Does the code reader see the plan?
a	Self-report: tell a model it implemented the code and ask "list the decisions you made that the plan didn't specify"	Yes
b	Sighted review: one pass that reads plan and code together and lists decisions	Yes
c	Full pipeline, but the extractor has read the plan	Yes
d	Full pipeline with an LLM tracer and adversary instead of the classifier	No
e	Full pipeline: blind extractor, execution, pairwise classifier, probes	No
f	Cheap static baseline: match each decision point from the AST walk against each plan sentence	No model reads the code

Arms a and b are the ones that matter. If a one-line prompt finds the same decisions, the rest of the machinery isn't earning its cost. Arm c tests the core idea directly: same pipeline, but the extractor is allowed to read the plan. If blindness matters, c should do worse than e.

There were two test cases, both small TypeScript ledgers with planted decisions:

• ledger: 10 silent decisions (5 planted up front, 5 added to the key after the first run, from reading the pipeline's output), 1 dropped requirement, 2 decisions the plan does make (to catch tools that flag everything).
• ledger-rounding: the same core plus rounding rules for fees, currency, interest and splits. 22 silent decisions, 3 dropped requirements, 8 planned rules.

Both cases were written on my side, with an AI agent doing most of the writing and running. Keep that in mind for everything below.

I developed the pipeline over five runs (arms e and f), then ran arms a to d in two more runs on the same code. Finally, every output from every arm was graded blind: a fresh model session got the plan, the code, the answer key and the shuffled lists, labelled L01 to L13, without being told which method produced which. Recall is the share of planted silent decisions found. Precision is the share of flagged items that are real unplanned decisions.

What happened

On the harder case, the one-minute prompts matched the full pipeline. These are blind decision-level grades on ledger-rounding (22 planted decisions):

Arm	Runs	Recall	Precision	Time per run	Cost per run*
a. Self-report	3	0.83	0.89	~1 min	~$0.30
b. Sighted review	3	0.82	0.93	~1 min	~$0.30
c. Pipeline, sighted extractor	1	0.86	0.82	~7 min	~$2.20
d. Pipeline, LLM tracer	1	0.91	0.77	~10 min	~$3.10
e. Full pipeline (blind)	3	0.83	0.85	~7 min	~$2 + classifier calls (not fully logged)
f. Static baseline	1	0	n/a	5 s	~$0

* API-equivalent cost reported by Claude Code for claude-opus-5-5. The real spend was subscription usage.

On the small ledger case every model-based arm scored between 0.90 and 1.0, so it can't separate them. Every arm caught every dropped requirement on both cases.

Arm f scores zero because of its format, not because it found nothing. It reports code locations ("a decision at DAILY_WITHDRAWAL_LIMIT = 5000") rather than rules, and the grader was told an item must state the decision. An earlier, more lenient grading gave it about 0.5.

A few details matter more than the averages:

• The pipeline does catch something the prompts miss. "Re-opening an existing account does nothing" was missed by the prompts in 11 of 12 runs and found by the pipeline arms in 9 of 10. One caveat: this is exactly the miss I fixed after run 004, and every pipeline run graded on the harder case came after that fix.
• The prompts found more real decisions that weren't in my answer key: about 9 to 10 per run against 0 to 4 for the pipeline. Some of that may be free text being easier to accept than an executed scenario, but the gap is large.
• The hardest decision was missed by everyone. "Transfers don't count towards the daily withdrawal limit" was missed by every arm in every run. That's not a weakness of blind extraction. It's hard for any model reading this code.
• The grader agreed with my earlier grades of the pipeline to within one decision per run, so the earlier results weren't inflated by my own grading.

Why it lost

Blindness didn't help. The whole design rests on one hypothesis: a model that has read the plan will describe the code as matching it, so the extractor must never see the plan. Arm c tested exactly that, and the pipeline with a sighted extractor did as well as the blind one (0.86 vs 0.83). The sighted review didn't rubber-stamp the code either. It had the best precision of any arm.

What the machinery buys is verification, not discovery. Every scenario the pipeline reports has been executed, and every dropped requirement it reports is confirmed by a failing test. The prompts' claims aren't checked against the code at all. That's worth something, and it's where the one consistent extra catch came from. On these cases it isn't worth seven times the time and seven to ten times the cost.

The limits are real, and they cut both ways:

• Small, self-made cases. Both are a single file of about 140 lines, written on my side with an AI agent, with every rule an explicit if or constant. That's exactly where one read-through should do best. On real code, with rules spread across many files, the result could change.
• One model grader. It was blind to the method, but it's a model, not a person, and the arms write differently (free text vs Given/When/Then), so it could have guessed which family a list came from.
• Few runs and lots of variance. c and d ran once. Three runs of the same arm differed by up to about 0.2 recall.
• Arm a isn't a real implementer. It's a fresh session told it wrote the code, with no memory of doing so.
• The pipeline was tuned on these cases over five runs. The prompts got no tuning at all, and they still matched it.
• On the small case, the answer key leans towards the pipeline. Half of ledger's 10 decisions were added after reading the pipeline's own output. On the 5 original ones, every model-based arm scored 1.0. The harder case was graded on the original key only, so the headline numbers aren't affected.

What I learned

Run the dumbest baseline first. I did build a cheap baseline, but the wrong one: a static scan that was easy to beat. The baseline that mattered was a one-line prompt. It took about a minute to run and it settled the question that five runs of pipeline tuning never could.

Write down what would prove you wrong, then actually run it. My own method write-up said that if the pipeline couldn't beat a one-line prompt, the prompt does the job and that's the finding. That turned out to be the finding. I could have known it weeks earlier.

Execution is the part I'd keep. Turning every claim into a test that actually runs, with a twin that must fail, worked in every run. Whatever replaces the rest of the pipeline, "show me a scenario that runs" is a better unit of review than "the model says".

Don't let the AI grade its own homework. An AI agent wrote a test case, ran the experiment, graded the results and proposed fixes, all competently. None of that could tell me whether the method worked. What settled it was running the comparison I'd been avoiding, and having every arm's output graded blind, shuffled and unlabelled, by a session that didn't know which method it was judging.

The closest published work is AssumptionMiner, which extracts implicit assumptions from LLM-generated code with the prompt in view and without running anything. Arm b is roughly that kind of approach, a single sighted pass, and on these cases it matched the pipeline.

What's next

The next step is probably a skill, not this pipeline: one well-prompted review pass (arm b's prompt is the starting point), with execution as an optional check on its claims. The cheapest experiment left is to feed arm b's list through the pipeline's execution and probe steps and see whether it matches the full pipeline's verified output.

The question that would actually settle this is independent cases: real repositories, with answer keys written by someone who hasn't seen any tool's output. That's where rules are spread out and harder to see in one pass, and where blind extraction plus execution might still pull ahead. Or might not.

The code, both test cases, the answer keys and a diary of every run (including what went wrong) are on GitHub: rodrinobrega/silent-decisions. Every run is tagged in git, and the blind grading packets are included so anyone can re-grade them. It's an experiment, not a product.