Code review can't keep up with agents. Gates can.

Originally published on the Zoroval blog.

The question

Coding agents like Claude Code and Codex now write most of our code. Most of the resulting pull requests are small, with a median of 133 changed lines, but about one in eight changes more than 1,000 lines.

And it is not only volume. With agentic coding, a new engineer can open a PR in their first week that changes a prompt or a context builder, and while writing that change is now easy, knowing what it does to the product is as hard as it ever was.

So here is the question this post is about:

How do you keep quality when code is written faster than anyone can read it?

There are two obvious answers, and both fail.

Answer 1: read every line. This was the rule of code review for decades, and it was a good one, but nobody can carefully review 1,000 lines of agent code several times a day, so people skim. A skimmed review looks like a review while catching far less than you think, and it turns the reviewer into the slowest part of the system.

Answer 2: block the PR until someone has time. This is safe, but it kills you slowly: if your process holds every change for days, you are not really using agentic coding, and you give up the speed it offers.

We chose a third answer. This post is about one part of it, the changes that alter how an LLM behaves, because those are exactly the changes that reading code cannot check.

How a context builder silently broke our LLM-as-a-judge

Here is the failure that shaped all of this. Our product uses LLM-as-a-judge: a model reads traces, which are logs of conversations between a company's AI agent and its users, and decides whether a named failure mode is present. Our customers depend on that judge.

On July 13, our judge missed every failure we had already confirmed. Our monitoring job ran 1,138 judge calls, and although our golden dataset had four cases where a person had confirmed the failure, the judge said "no" to all four. The run finished without an error.

The model had not changed, and neither had the prompt template. Our evals ran when one of those two changed, so nobody had run the golden dataset, and when the failures went missing we did not know which change was responsible.

The root cause was the context builder, the code that assembles what the model sees. It is just as much a part of the intelligence system as the model and the prompt, but changing it never triggered our evals. When later feature work changed that code, the same model with the same instructions found most of the failures, and running the golden dataset against the context builder before the large run, four calls for under a dollar, would have caught the problem.

This is the core problem with LLM systems in CI. Code tests check the shape of the output, such as whether it is valid JSON with the right fields, but they do not check judgment, so a change to the code around a model can pass every test and still change what the model concludes. With coding agents, those changes happen all the time, often in a PR that was never meant to touch the model. We needed an eval gate that knows which files can change model behavior and refuses to merge them on green tests alone.

The principle: review evidence, not lines

Agents make changes cheap. We require current evidence before those changes reach production.

That sentence is the core of how we do LLMOps. LLMs make mistakes, both the coding agents that write our code and the models inside our product, and we do not try to stop that. Instead, we build an environment where mistakes are cheap, visible and caught before they reach a user, and four ideas run through everything below.

1. Review evidence, not lines. A human still decides, but on test results, eval scores, fingerprints and exact commit SHAs rather than on a 1,000-line diff, and security, architecture and interactions between changes still need a person.
2. People own the rules. The policy files that define our gates are small, and changing them needs review from an owner, such as a senior engineer or an ML engineer, while the gates apply those rules to every change.
3. Gates talk to agents. When a gate fails, its output says what changed, why it matters and how to fix it, so the coding agent can fix the problem itself. Most CI output is written for humans, but ours is written for the next agent turn.
4. Approval is bound to exact content. Every approval names a fingerprint or a SHA, so if one more line changes, the approval goes stale automatically and nobody has to remember to re-check.

Here is the whole path a pull request takes through the eval gate, before we go through it step by step.

Every new commit gets a new fingerprint, so a blocked PR goes back through the same path after the agent runs the eval or reverts the change.

How the eval gate works in CI

The eval gate is the most important idea in this post: a CI check that knows which files can change model behavior. We call ours the Intelligence Change Gate, which is why that name appears in its output below, and it works in five steps.

Step 1: declare what is protected. A policy file lists every path that can change model behavior, such as prompts, context builders, model settings and retrieval code, and groups them by the LLM component they affect. The policy calls each component a stage, for example monitoring.

{
  "rules": [
    {
      "id": "monitor-context",
      "paths": ["src/monitor/context_builder.py", "src/monitor/judge.py"],
      "stages": ["monitoring"]
    },
    {
      "id": "monitor-runtime-settings",
      "paths": ["src/settings.py"],
      "line_patterns": ["MONITOR_MODEL", "MONITOR_REASONING"],
      "stages": ["monitoring"]
    }
  ]
}

Notice the first rule: it protects a context builder, not only a prompt file, because that kind of code is what failed us in July. The line_patterns field matters because a settings file changes often, and we only want the gate to fire when a line that sets the monitor's model or reasoning effort changes, not when someone adds a database timeout.

Step 2: fingerprint the exact change. On every PR, the gate finds the protected files that changed, takes the diff of only those files, and hashes it with SHA-256 together with the affected stages and rules to produce the change fingerprint. If someone else changes the same protected file and you rebase, the fingerprint can change and you need fresh evidence. That is deliberate, because the code you measured is no longer exactly the code that will merge.

Step 3: require an eval report for that fingerprint. If a protected file changed, the PR must contain an eval report, a small JSON manifest that names the fingerprint, the golden dataset, the baseline, the candidate and the result for every affected stage:

{
  "schema_version": 1,
  "experiment_id": "monitor-context-builder-2026-10-02",
  "change_fingerprint": "<the 64-character fingerprint from CI>",
  "affected_stages": ["monitoring"],
  "dataset": { "id": "private-monitor-sample", "fingerprint": "<sha256>", "case_count": 50,
               "breakdown": { "known_positive": 20, "known_negative": 30 } },
  "baseline":  { "name": "context builder at develop",  "artifact_sha256": "<sha256>" },
  "candidate": { "name": "context builder in this PR", "artifact_sha256": "<sha256>" },
  "results": [
    { "stage": "monitoring", "kind": "quality", "metric": "verdict_agreement",
      "direction": "higher_is_better", "baseline": 0.94, "candidate": 0.96,
      "minimum": 0.9, "maximum_regression": 0.0, "passed": true }
  ],
  "decision": "pass",
  "why_better": "Same verdicts on all known positives, one fewer false alarm on negatives (47 to 48 correct of 50).",
  "limitations": ["Only one failure mode in the sample."]
}

Put simply, the eval gate makes an LLM eval a required check: a PR that touches a protected file cannot merge until an eval of that exact change passes, or an owner signs a break-glass override. The gate does not trust the passed field and recomputes the pass from the numbers. Raw data stays private, because the report only points to it by hash, so no customer data goes into the repository.

Step 4: let evidence go stale by itself. This is the part we like most. Because the fingerprint covers the exact protected diff, any edit to a protected file after the eval ran, whether by a person or a coding agent, changes the fingerprint, so the old report no longer matches and the gate fails again. Changes to docs and tests do not affect the fingerprint, so you can still fix a typo. Nobody has to remember that the evidence was for an older version. The gate remembers.

Step 5: keep a visible break-glass override. Sometimes you consciously accept a trade-off. Say your goal for a failure mode shifts from precision to recall: you would rather catch every real failure and accept more false positives. The eval will show precision dropping, and instead of tuning thresholds until that looks like a pass, you record it as a decision. For that, the gate allows an owner override. The override:

• is a separate file for that one PR number;
• names the exact fingerprint;
• needs a reason and a risk statement;
• must come from a listed approver;
• expires after seven days;
• shows in CI as owner_override, never as a pass.

For the precision-to-recall change, the override file looks like this:

{
  "schema_version": 1,
  "pull_request": 512,
  "change_fingerprint": "3f9c…e21a",
  "approved_by": "owner-github-handle",
  "approved_at": "2026-10-02T09:30:00Z",
  "reason": "We now want recall over precision for this failure mode, so the judge should flag more borderline cases even if some are false alarms.",
  "risk": "Reviewers will see more false alarms. We will watch the false-positive rate in monitoring and roll back if it doubles.",
  "acknowledgement": "I consciously accept merging this exact intelligence change without a passing experiment."
}

An override is a decision that someone owns, not evidence, and the gate never treats it as evidence. Our staging branch holds 43 eval reports, and owner overrides were used on 9 PRs.

What the coding agent sees. When the gate fails, it prints this:

Intelligence Change Gate
Result: FAIL
Decision: blocked
Affected stages: monitoring
Protected paths: src/monitor/context_builder.py
Matched rules: monitor-context
Change fingerprint: 3f9c…e21a
Failures:
  - Protected intelligence changed without a passing exact-diff experiment.
  - No changed experiment manifest matches this protected diff fingerprint
  - Owner override was not valid.
  - No changed exact-diff owner override exists at .github/intelligence-overrides/pr-512.json

Normal resolution: run the affected eval and commit a matching
manifest under evals/experiments/.

That output is a work order. The agent knows which stage to evaluate, which files caused the failure and which fingerprint to put in the eval report, and the two override lines only mean that no valid override exists either. Sometimes the agent finds that it never meant to touch that file at all, so it reverts that part and the gate passes with no eval, which is how the gate also catches accidental scope creep and misunderstood requests.

A second story: what a passing eval gate does not prove

Gates do not only stop bad changes. They also make good changes safe to try.

We tested a judge model that was about six times cheaper, and it passed our first check by finding every known failure in our golden dataset. In a wider comparison, though, it said "yes" far more often than our judge on one failure mode, because it had dropped part of the definition and flagged cases that were out of scope.

The first check missed this because our golden dataset had only positive examples. A dataset like that measures recall, whether the model finds the failures, but never precision, whether its "yes" can be trusted. A model that says "yes" too often cannot fail it.

The cheaper model did reach production later, after an eval with both known positive and known negative cases, and it cut our monitoring cost by about 83%. Its eval report lists its own limits: one failure mode had no positive example, so its recall is still untested. The gate did not design that sample. It made sure that evidence for the exact change existed, with its limits written next to it, but people decided what the evidence had to cover, because LLM changes fail in ways that one number does not show.

So we want to be exact about what the eval gate proves. It checks that:

• a protected file changed (by path, or by changed line for shared files);
• an eval report in the PR matches the exact protected diff;
• every affected stage has a quality result;
• the sample meets the minimum size the policy sets for that stage;
• the run is recent;
• each result meets the thresholds written in the report.

It cannot check that:

• the eval ran, or that the numbers are real. CI never sees the private data, and it only checks that the hashes are well formed.
• the thresholds are right. The author of the report writes them.
• the golden dataset covers the failure modes that matter. One quality result per stage can come from one failure mode.
• anything in a file the policy does not list.

That is why a person must still review each eval report, and for us that happens at the production approval, where the approval packet lists every report.

The LLMOps pipeline around the gate

The eval gate only works because of what surrounds it. Our gates sit in four layers (local, pull request, staging and production), and a change has to pass all four.

 Local                Pull request          Staging                  Production
 ───────────────      ─────────────────     ─────────────────────    ─────────────────────
 AGENTS.md rules      Product CI            Merge deploys staging    Only named PRs
 Agent skills         Regression evals      Never authorizes prod    LLM cost impact
 Same checks, local   Eval gate                                      One page of evidence
 One worktree/agent   Release-intent label                           Approval at exact SHA

Local

The cheapest mistake is the one a coding agent catches before it opens a PR. Our repository has an AGENTS.md at the root, which Claude Code, Codex and most other coding agents load before they start work, and it describes boundaries rather than coding style:

• develop is staging. main is production.
• "Merge PR #N" means merge into its declared base only. It never means "release to production".
• Changes to models, prompts, context builders, retrieval, monitoring or model-written reports are protected. Here is how to prove they are safe.
• Never push to main. Never bypass a failed guard.

These lines are short and absolute on purpose, because in our experience an agent follows a clear rule more reliably than a nuanced paragraph. Repeated procedures, such as a production release, live in agent skills in the repository, versioned with the code, so the agent loads the procedure instead of improvising it.

Every gate that runs in CI can also run on a laptop, so an agent can run the eval gate against its own branch before it pushes and fix a failure in the same session:

python tools/validate_intelligence_change.py --base origin/develop --head HEAD

Each agent also works in its own git worktree on its own branch, because parallel agents in one checkout overwrite each other's work, and this way a mistake stays inside that worktree.

Pull request

Every PR runs Product CI. Most of it is normal (backend tests, frontend and database contracts, and a check that the public website only ships files we listed), but two parts are specific to LLM systems: the eval gate above, and regression evals for the code around the model.

The eval gate asks for a live eval only when protected files change, while the regression evals run on every PR, whatever it touches:

python tools/run_evals.py --suite ci

They run the real product code for each stage against frozen, synthetic cases with recorded LLM inputs and outputs, and compare each metric with a reviewed baseline, so a critical failure or a regression fails the build without a single model call, which keeps them fast and free. They do not prove the model is good. They prove that the code around the model still behaves as it did, and that is the part coding agents break most often.

Protect the gate itself. An agent that can edit the gate can turn it off, so give the policy, the gate script, the workflows, the eval reports and the override folder an owner in CODEOWNERS. A CODEOWNERS file only requests reviews and does not require them, though, so on GitHub you also need to:

• turn on Require review from Code Owners in a branch protection rule or a ruleset;
• make the gate a required status check;
• give CODEOWNERS itself an owner, for example /.github/CODEOWNERS @owner.

One trap for small teams: GitHub does not let a PR author approve their own PR, so if your coding agents open PRs from your account, you cannot give the code-owner approval yourself, and the agents need an account of their own.

Release intent. Every PR to develop carries exactly one label, production-candidate or preview-only, and a PR with no label or with both fails a required check. The label is cheap, and it forces one question early: is this meant for users or not? A separate advisory job reports whether the patch would apply cleanly to current production, which is only advice but shows conflicts before anyone asks for a release.

Staging and production

A merge into develop deploys to staging, which is where mistakes are allowed, and a bad merge there costs a revert, not an incident. One rule matters more than any check here: a merge into staging never authorizes production. We never merge develop into main, so staging can hold several experiments and unfinished work at once without any of it moving to production as a group. Production is the one place where a human decision is required every time.

• Selective releases. The owner names the exact PRs to release, and a release branch starts from current main with only the squash commits of those PRs. If a patch does not apply cleanly, the release stops. A clean apply does not prove that nothing is missing, so the tests and regression evals run again on the assembled candidate as a second check, and they are not complete either.
• LLM cost impact. Every release declares whether it changes what we spend on model calls: none, decrease, increase or new-paid-path. For LLM systems, cost is part of correctness, and a change that doubles your bill is a regression even if every answer is right.
• One page of evidence. The coding agent that prepared the release posts an approval packet with the PR number, exact head SHA, source PRs, migrations, eval reports, configuration changes, LLM cost impact and test results. The owner reads that packet, not the diff.
• Approval bound to a SHA. The approval is a PR comment that names the exact commit, for example Approve production PR #502 at 1a6e515…, and a guard checks that the head is still that SHA, so any new commit invalidates it. Why not use GitHub's built-in dismissal of stale approvals? Because a PR author cannot approve their own PR, and our agents open PRs from the owner's account.

Is this slower than "merge and deploy"? A little, but the slow part is one person reading one page of evidence, and everything else is automatic. This release model fits a very small team, and while a larger organization would likely stay trunk-based, the LLM cost field and the SHA-bound approval still carry over.

What this costs, and what it does not catch

We want to be honest about the limits.

• It costs friction. A one-line edit to a context builder can need a live eval, which is sometimes annoying. The break-glass override exists for those moments, and it is visible on purpose.
• The evidence is only as good as the golden dataset. A report with 50 cases can miss a rare failure, a 0.02 change on 50 cases is a single verdict, and two judge runs can differ. The gate cannot check that the dataset represents your users.
• Evidence is self-reported. Our next step is eval reports produced by a trusted eval runner, with thresholds set in the policy instead of in the report.
• The policy can be incomplete. If a file can change model behavior and is not in the policy, the gate is blind to it, and we add the path when we find a gap. A prompt snapshot test on fixed inputs would catch changes from code the policy does not list, but we do not run one yet.
• It does not replace human review of security, architecture or interactions between changes.
• It does not replace production monitoring. Gates protect the change, and monitoring protects the user, so you need both.
• It works for a small team. We have not run this with fifty engineers, and while we think the ideas scale, we have not proven it.

Start with a golden dataset

You do not need all of this on day one. Start here:

1. Build a small golden dataset from real traces, with both positive and negative examples.
2. List the files that can change model behavior, including your context builders.
3. Add a required CI check that fails when those files change, unless the PR links an eval run.
4. Add an LLM cost impact line to your PR or release template.
5. Rewrite one CI failure message so a coding agent can act on it.

Set it up in your repository

We packaged the eval gate as an agent skill. Point Claude Code or Codex at your repository, ask it to set up eval gates, and the skill will:

1. find the files in your repository that can change model behavior, including context builders;
2. propose a policy file and ask you to confirm it;
3. add the gate script, the CI workflow and an eval report template;
4. add the rules to your AGENTS.md, so your coding agents know what to do when the gate fails;
5. protect the gate files with CODEOWNERS and tell you which branch protection settings to turn on;
6. test the gate on a throwaway branch.

The skill and the gate are open source: github.com/Zoroval/agent-gates

Here is the short checklist if you want to build it yourself:

• List every path that can change model behavior, by stage, including context builders.
• Fingerprint the protected diff, not the whole PR.
• Require an eval report that matches the fingerprint and covers every affected stage.
• Recompute pass and fail from the numbers, and have a person review the thresholds.
• Make break-glass overrides per PR, owned, expiring and never shown as a pass.
• Write every failure message for a coding agent to act on.
• Enforce code-owner review on the gate itself.
• Bind production approval to an exact SHA.

Why we care about this

Zoroval does the same thing for AI products that these gates do for our code. Your agent talks to your users every day, and when you change a prompt, swap a model or adjust a context builder, the real question is: did that change make your agent better or worse for the people using it?

Every gate in this post needs a golden dataset: real traces where someone confirmed a failure, and real traces where they confirmed there was none. Most teams do not have one, and that is what Zoroval builds. Your domain experts review real production traces and name the failure modes they see, and those become a golden dataset and LLM-as-a-judge monitors that run on your live traffic.

Our gates answer that question for one repository, and Zoroval answers it for your production traffic. If that is a question your team asks, we would like to talk.