From shadow to gate: how we fixed agent citation fraud in one production batch

The previous post ended with a question: which of your zeros are just absences? This one is about what happened after we found ours.
Our research pipeline had a citation problem. A blind review of 150 generated reports found what three independent analyses converged on: references that did not match the text, method claims with no data behind them, and quality scores computed on a truncated prefix of each report.
We fixed it in one day, but not by making the model smarter. The fix was a pipeline: shadow audit, body-driven rebuild, then a gate. This is the recipe.
Step 0: Blind review before you fix anything
Two agents analyzed the same corpus independently — one implementation agent, one audit agent from a different harness, without seeing each other's conclusions. Both found the same three priority failures:

1. Citation mismatch: the reference table was built from the source material list, not from what the text actually cited.
2. Method dishonesty: reports claimed "survey research" when no survey had been conducted.
3. Scoring on a prefix: the quality grader read only the first 6000 characters; references and conclusions were appended after that. Independent convergence is the cheapest signal you will ever get that the problem is real. Step 1: Shadow audit — record, do not block The first change was a shadow audit. It ran before every report was saved and computed two numbers:
4. inbound rate: what fraction of in-text citations appear in the reference table?
5. dirty-entry count: how many entries contain internal markers, section fragments, or pipe residue? It wrote to a JSONL file and changed nothing. On the existing 148-report batch, the inbound rate median was 0.32 — roughly one in three citations was broken. The point of shadow mode is calibration without risk: you learn the distribution before you set any threshold. Step 2: Fix the source, not the symptom The root cause of the mismatch was structural: the reference table was reconstructed from the input material list, not from the body text. The fix inverted the direction — at finalize time, the system now rebuilds the reference table from the citations that actually appear in the body, resolving each through the knowledge base for a canonical title. Replaying the same 148 reports: inbound rate median 0.32 → 1.00 reports rebuilt 142/148 (6 correctly fell back) dirty entries 120/326 → 8/326 The remaining 8 were legitimate long titles, not defects. Step 3: Gate — block ingestion, not delivery Only after shadow numbers were calibrated did we turn on a gate. The design choice matters: A failed citation check blocks the report from entering the knowledge base. It does not block delivery.

Rationale: a report with broken citations may still be useful to the requester; poisoned knowledge is not. The cost asymmetry runs in one direction. We set the threshold at 0.5 — below the 10th percentile of the repaired batch (0.857), high enough to catch structural failure without blocking honest edge cases.
Step 4: A dashboard that runs itself
The final step was a batch quality contract: five mechanical metrics computed after every batch, stored in a governance database, with thresholds calibrated from the pre-fix distribution.
Metric Before After
Citation reconciliation (median) 0.45 1.00
Dirty entries (P50) — 0
Fixed closing-sentence rate 97.3% 0%
Method honesty — 100%
Deep-source share 31.6% 58%
Blank-source share 74% 20%

No LLM in the metrics. No human in the loop for the counting. If the pipeline regresses, the dashboard catches it on the next batch.
What made it work
Three things, in order:

1. Shadow before gate. We could not have chosen 0.5 without knowing the distribution. Blocking on a guess would have either frozen production or let everything through.
2. Fix the mechanism, not the prompt. The reference table was wrong because of how it was built, not because the model was careless. Rebuilding from body citations is deterministic.
3. Block ingestion, not delivery. This is the absent-zero principle applied to failure handling: a bad report is visible and correctable; a poisoned knowledge base is silent and compounding. The part that is reusable None of this is specific to our stack. Any agent pipeline that produces claims with citations can run the same sequence: blind review → shadow audit → source fix → calibrated gate → batch dashboard The three-state validity rule from the previous post does the same job at the measurement layer: absent evidence is unknown, not zero. This recipe does it at the production layer: a broken citation blocks knowledge, not the user. The trust layer that made the audit reproducible is at janzong/agent-lab-trust, and the validity-state specification is docs/validity-states.md. If your agents write anything that gets cited later, run the shadow audit first. The first number is usually worse than you expect.