Deterministic Evals for Coding Agents: Prove the Fix

Your coding agent says it fixed the retry bug. The diff looks reasonable, its new test passes, and an LLM reviewer gives the patch a high score. None of those results proves that a duplicate webhook can no longer charge the customer twice.

The missing piece is a deterministic eval that makes the failure happen on the old code, runs the same failure against the proposed fix, and lets measured behavior decide the verdict.

What is a deterministic eval engine for coding agents?

A deterministic eval engine for coding agents executes a defined invariant against real code and calculates the result with fixed logic. Given the same code, scenario, and observations, it returns the same verdict. A model can help discover a bug or propose a test, but it does not get to turn the final run green.

For a bug fix, the useful evaluation is differential:

1. Start with the current, still-broken working tree.
2. Apply the proposed patch to a separate copy.
3. Inject the same provider failure into both copies.
4. Run the same executable probe against both.
5. Require the old code to violate the invariant.
6. Require the patched code to hold the invariant.
7. Record the observations in a receipt.

FetchSandbox implements this pattern for third-party API integrations. It is a verification layer between the coding agent's claim and the pull request.

Why is an LLM judge not enough to verify a code fix?

An LLM judge is useful when the requirement is subjective: Is the explanation clear? Is the design unnecessarily complex? Does the patch appear maintainable?

An integration invariant is usually not subjective.

• One payment event should create one entitlement.
• A replayed webhook should not repeat a charge.
• A provider 409 should not be swallowed and reported as success.
• A lost response should not make a retry create the resource twice.
• A stale event should not overwrite newer local state.

For those properties, a probability or prose assessment is weaker than an observation. The relevant question is not whether another model likes the patch. It is whether the required state held after the failure actually ran.

Anthropic's guidance on evaluating AI agents makes the same grader distinction: use deterministic graders where correctness can be checked by code, and reserve model-based graders for criteria that require judgment.

That separation matters in production. If generation and final grading are both model decisions, the verification layer inherits model variance. If execution produces a stable signal and pure logic maps that signal to a verdict, CI can rely on it.

How does the exit-code flip decide whether a fix is proven?

FetchSandbox probes use a small exit-code contract:

• 0 means the invariant HELD
• 1 means the invariant was VIOLATED
• 2 or another harness error means INCONCLUSIVE
• a probe that did not run is also INCONCLUSIVE

Only one measured transition can produce a green verdict:

buggy tree   1  VIOLATED
fixed tree   0  HELD
result          PROVEN
green_allowed   true

Every other result declines or rejects the proof:

• 0 → 0: the supposed bug did not reproduce, so the patch is not proven
• 1 → 1: the bug reproduced, but the fix is incomplete
• either side inconclusive: the engine could not establish the claim
• self-reported before-and-after results: useful context, but never independently proven

This gate is deliberately boring. It is a pure function of the two measured runs. Reference simulations, agent confidence, a persuasive explanation, or a passing unrelated test cannot change green_allowed to true.

Why must the same failure run on old and fixed code?

Running only the patched code proves less than it appears to.

Suppose a coding agent adds a test for a duplicate webhook and the test passes. Several explanations are still possible:

• the test never triggered a duplicate delivery
• the handler never booted
• the mock skipped the provider behavior
• the assertion checked only the HTTP response
• the patch suppressed every side effect, including the one the customer paid for
• the bug was never present in the tested path

The old-code run is the negative control. It demonstrates that the scenario can detect the claimed bug in this repository. The fixed-code run then demonstrates that the same detector no longer finds it after the proposed change.

Without the first leg, “pass” can mean “the check is blind.” Without the second, there is no fix. The pair is what turns a test result into differential evidence.

How does FetchSandbox inject integration failures?

FetchSandbox runs integration code against a stateful sandbox of the services it talks to. A scenario can deliberately introduce behavior that happy-path mocks often omit, including duplicate webhook delivery, 409 conflict responses, retries, and observable repeated side effects.

Consider a payment webhook handler that deduplicates on a delivery ID instead of the provider's stable event ID. A retry arrives with a new delivery ID, so the application treats one event as two and grants or charges twice.

The executable invariant is not:

The handler returned 200.

It is:

One provider event creates one business side effect,
even when delivery is attempted more than once.

The probe drives the duplicate delivery, observes the application state, and exits 1 if the side effect accumulates. The proposed patch is applied to a copy, the same delivery sequence runs again, and the probe exits 0 only when the invariant holds.

FetchSandbox also has a zero-effect backstop for this class of proof. A patch that stops the duplicate by granting or charging nothing does not get credit for correctness.

Is there an LLM anywhere in the FetchSandbox eval path?

The final verdict is not an LLM opinion. Once both probe runs complete, fixed logic classifies their exit codes and decides whether the transition qualifies for green.

The scenario tier is a separate concern:

• curated scenarios use reviewed, hand-authored probes
• declared scenarios use an integration's explicit invariant definition
• a novel bug class can use an LLM to propose a candidate probe

A generated probe is not trusted because a model wrote it. It must first reproduce the failure on the buggy code, execute against both trees, state the invariant it checked, and survive qualification before it can be reused. If FetchSandbox cannot produce a probe that runs and reproduces the bug, the result stays unproven.

This is the important boundary: models may help propose what to execute; models do not decide whether the measured execution passed.

What does a coding agent send to prove_fix?

The prove_fix MCP tool needs the still-broken project and the proposed unified diff. Call it before applying the patch to the working tree.

find_bugs
  → fix_bug returns a proposed diff
  → prove_fix receives broken tree + diff
  → FetchSandbox applies diff to a copy
  → same scenario runs against both trees
  → deterministic gate returns proven, rejected, or unproven

A successful response includes machine-readable fields such as:

{
  "state": "proven",
  "green_allowed": true,
  "reproduced": true,
  "verified": true,
  "receipt_url": "https://fetchsandbox.com/runs/fix-...?flow=..."
}

If the agent has already applied the patch, the original failing state is gone from the workspace. Reverting mentally or writing a new fixed-only test is not equivalent. The tool needs the real broken tree to establish the first half of the proof.

What is in the proof receipt?

The receipt is the review artifact for the run. It records the scenario, the before-and-after outcome, the invariant, and the measured verdict at a shareable URL.

Here is a measured FetchSandbox proof receipt from a Paddle entitlement bug. The first proposed patch was rejected as incomplete. A later patch reproduced the bug on the old tree and held the invariant on the fixed tree, producing the required 1 → 0 transition.

A reviewer can paste the receipt into the pull request:

## Integration proof

- Failure: duplicate webhook repeats a paid side effect
- Invariant: one provider event creates one side effect
- Before patch: VIOLATED
- After patch: HELD
- Verdict: proven
- Receipt: https://fetchsandbox.com/runs/fix-...?flow=...

The receipt does not replace review. It removes one factual question from review: whether the named integration failure reproduced before the patch and stopped after it.

How does the proof fit into PR review and CI/CD?

Use the proof receipt during pull-request review, then keep the corresponding provider workflow in CI.

The two checks serve different moments:

• prove_fix evaluates a proposed patch against a reproduced bug and returns the before-and-after receipt
• fetchsandbox run --all --json runs the configured integration workflows in a pipeline and exits non-zero when a workflow fails

The current shipped workflow is to paste the prove_fix receipt into the PR manually. FetchSandbox does not yet post GitHub comments automatically. In CI, store the JSON workflow output as an artifact and make the job a required check when its coverage matches the branch's risk.

This creates a useful chain of evidence:

agent proposes code
  → sandbox reproduces the failure
  → patch flips VIOLATED to HELD
  → receipt travels with the PR
  → workflow remains gated in CI/CD

The agent remains fast. The reviewer gets a falsifiable result instead of a longer assurance.

Does FetchSandbox evaluate every kind of coding task?

No. FetchSandbox is focused on code that integrates with third-party APIs and on failure modes its scenarios and probes can observe. The measured real-code runner currently boots Node.js and Python projects.

It does not prove subjective code quality, migration safety, authorization design, or every requirement in a pull request. Those still need static checks, ordinary test suites, security review, and human judgment.

It also does not turn missing coverage into success. If the app cannot boot, the scenario cannot reproduce the bug, or the probe cannot judge the outcome, the verdict remains unproven.

That refusal is part of the product. A deterministic eval engine is valuable not because it always says yes, but because nothing except the required measured transition can make it say yes.

How do I use FetchSandbox from a coding agent?

Connect FetchSandbox MCP to Claude Code, Cursor, Claude Desktop, Codex, or another MCP-compatible client. The agent can discover integrations, run stateful workflows, inject failures, investigate a bug, propose a patch, and call prove_fix before editing the working tree.

For the broader setup, see agent API workflow testing and API integration testing in CI.

Questions about deterministic coding-agent evals

What is the difference between a coding-agent benchmark and a fix proof?

A benchmark compares agent performance across a task set. A fix proof evaluates one concrete claim in one repository: the named failure occurred before this patch and stopped after it under the same executable scenario.

Can the same coding agent generate the fix and request the proof?

Yes. Independence comes from the external scenario, real execution, and deterministic gate. The agent can request the run, but it cannot set green_allowed.

Why not trust a test that the coding agent wrote?

Agent-written tests can be useful, but they can repeat the same mistaken assumption as the patch. The stronger pattern includes a negative control: the probe must catch the bug on the old code before its pass on the new code counts.

What happens if the old code passes?

The result is not_reproduced, not proven. A 0 → 0 result shows that the probe did not establish the claimed before-and-after change.

What happens if the runner or probe crashes?

The run is inconclusive and green is blocked. A harness error is not evidence that the application violated or held the business invariant.

Can an LLM-generated probe produce a green result?

It can only contribute a candidate executable check. The candidate must run against real code, reproduce the old failure, and observe the fixed invariant. The deterministic gate, not the model, decides the result.