9/9 Agent Tests Passed. The Trace Oracle Flagged 4 Behavior Changes.

Agent-written tests are green because the patch wrote them. That is the entire problem in one sentence.

A patch that rewrites behavior produces a test suite that asserts the new behavior. The suite proves the patch is internally consistent. It says nothing about what the system promised before the patch arrived.

So I stopped asking "did the tests pass?" and started asking "what behavior changed?". The trace oracle answers that: replay a frozen corpus against the old and new builds, canonicalize the noise, diff the structured traces, and check every delta against an explicit allowlist.

In the reproducible demo below, an agent patch passes 9/9 unit tests. The oracle still flags four silent behavior changes.

Why the agent's tests are the wrong oracle

Unit tests and patches grow from the same diff. The assertions mirror the patch's intent, so they cannot see the patch's side effects.

A missing-key delete now returns success? The agent's tests never called delete on a missing key. An exit code changed from 2 to 1? The tests only assert "exit is non-zero".

A behavioral regression is a delta the patch did not intend. The reliable way to find it is to compare against something the patch did not write: the previous binary's observable trace.

The oracle in three pieces

The whole gate is three small components.

1. A property-generated seed corpus. Each seed is a short command sequence. I generate them with a property-based generator, then freeze the RNG seed so the corpus is reproducible. The corpus targets edges the unit tests skip: empty values, duplicate keys, missing keys, unknown commands.

2. Frozen fixtures. Before every replay I hash every fixture the binary touches. If a fixture drifted, the diff is invalid. The fixture hash and the corpus hash are pinned in the merge gate.

3. Canonicalization and a policy diff. Runtime noise (pid, timestamps, elapsed time) is redacted. Payloads are compared. An allowlist marks deltas that match the patch's stated intent. Everything else is flagged for a human.

The merge rule is short: flagged == 0, no fixture drift, and no quarantined test.

The demo

I use a toy key-value CLI so every number here is reproducible. The trace format is seed|step|command|payload. The full traces are ten lines each; the excerpts below show only the lines where the two builds differ.

golden.log:

A|2|set|key=a|value=2|status=0|warn=none
A|4|delete|key=missing|status=1|error=missing
A|5|list|order=insertion|count=1
B|1|set|key=e|value=|status=1|error=empty
B|2|get|key=e|hit=false
C|3|badcmd|status=2|error=unknown

patched.log:

A|2|set|key=a|value=2|status=0|warn=duplicate
A|4|delete|key=missing|status=0|error=none
A|5|list|order=sorted|count=1
B|1|set|key=e|value=|status=0
B|2|get|key=e|hit=true
C|3|badcmd|status=1|error=unknown

The agent's nine unit tests pass on both builds. They cover roundtrips, overwrite, delete-existing, and a non-zero exit for unknown commands. None of them covers delete-missing, duplicate-set, empty values, or the exact exit code. That is exactly where the behavioral diff lives.

The workflow is six steps:

1. Generate the seeds; freeze the generator RNG seed.
2. Hash every fixture the binary touches.
3. Run the old build; capture golden.log.
4. Apply the patch; run the new build; capture patched.log.
5. Run trace_oracle.py; classify the deltas against the policy.
6. Merge only when flagged == 0 and no fixture hash changed.

trace_oracle.py:

#!/usr/bin/env python3
"""trace_oracle.py — diff two CLI runs and classify behavioral deltas."""

import json
import re
import sys
from pathlib import Path

NOISE = {
    "pid": r"pid=\d+",
    "ts": r"ts=\d{4}-\d{2}-\d{2}T[\d:]+",
    "elapsed_ms": r"elapsed_ms=\d+",
}

def canonical(payload):
    for field, pattern in NOISE.items():
        payload = re.sub(pattern, f"{field}=<redacted>", payload)
    return payload

def load_trace(path):
    traces = {}
    for raw in Path(path).read_text().splitlines():
        if not raw.strip():
            continue
        seed, step, cmd, payload = raw.split("|", 3)
        traces.setdefault(seed, {})[int(step)] = (cmd, canonical(payload))
    return traces

def diff(old_path, new_path, policy_path):
    old = load_trace(old_path)
    new = load_trace(new_path)
    policy = json.loads(Path(policy_path).read_text())
    allowed = {(p["seed"], p["step"]) for p in policy["allowed"]}

    flagged = 0
    for seed in sorted(old):
        for step in sorted(old[seed]):
            old_cmd, old_payload = old[seed][step]
            new_cmd, new_payload = new[seed].get(step, ("<missing>", "<missing>"))
            if old_payload == new_payload:
                continue
            verdict = "ALLOWED" if (seed, step) in allowed else "FLAGGED"
            if verdict == "FLAGGED":
                flagged += 1
            print(f"[{verdict}] {seed} step={step} {old_cmd}: "
                  f"{old_payload} -> {new_payload}")
    return flagged

if __name__ == "__main__":
    if len(sys.argv) != 4:
        sys.exit("usage: trace_oracle.py golden.log patched.log policy.json")
    sys.exit(min(diff(sys.argv[1], sys.argv[2], sys.argv[3]), 255))

policy.json:

{
  "allowed": [
    { "seed": "A", "step": 4, "reason": "intent: delete is idempotent" },
    { "seed": "A", "step": 5, "reason": "intent: list sorts by default" }
  ]
}

Run it:

python trace_oracle.py golden.log patched.log policy.json

The report:

Seed	Step	Command	Delta	Verdict
A	2	set	warn=none → warn=duplicate	FLAGGED
A	4	delete	status=1 → status=0	ALLOWED
A	5	list	order=insertion → order=sorted	ALLOWED
B	1	set	error=empty → status=0	FLAGGED
B	2	get	hit=false → hit=true	FLAGGED
C	3	badcmd	status=2 → status=1	FLAGGED

Six deltas. Two match the patch's stated intent. Four are side effects: a duplicate-set warning that changes stderr contracts, empty values now accepted, reads after a failed write returning hits, and an exit-code semantics change.

Each of those four would ship under a green 9/9 suite. And each one independently breaks a downstream consumer: parsers on stderr, callers checking exit codes, applications relying on empty-value rejection.

The flake freeze

A flaky test poisons the whole gate. If the suite fails randomly, you cannot tell whether a delta came from the patch or from the environment.

The quarantine rule is numeric: a test that fails twice or more across 10 consecutive suite runs moves to quarantine.txt for 7 days. It leaves the merge gate the moment it is quarantined. During that week the trace oracle is the deterministic reference — if the oracle is green and the only failure is the quarantined test, the patch proceeds.

Property generators get the same treatment. Freeze their RNG seed, or the corpus is flaky by construction and the oracle diff becomes noise.

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The oracle itself is plain Python and two log files; it runs anywhere. MonkeyCode fits in two unglamorous steps of my setup. The free model tier drafts the first policy skeleton from a golden trace — a first-pass intent-versus-collateral classification that I correct by hand. The replay loop, one process spawn per seed across both builds, runs on the free server option instead of my laptop. Check the current free limits before scaling that loop to a large corpus; I am not quoting quotas here because they change.

Limits of the approach

The trace oracle needs deterministic, structured output. A GUI, a timing-sensitive path, or a raw syscall stream will not produce a stable diff. Races remain ThreadSanitizer's job, not this script's.

A patch that rewrites the trace format itself defeats the diff. You must regenerate the golden corpus, and the oracle cannot validate its own grammar change.

The policy file is a trust boundary. The same flaw that lets an agent write its own tests can let it write its own allowlist. Review the policy with the same suspicion you apply to the patch.

This is also overkill for a one-line patch in a well-covered module. A mutation audit is the cheaper instrument there.

The question, not the script

Start smaller than you think: thirty corpus lines, one noise rule, one policy file. The value is not the tool. The value is the question it forces — what behavior changed, not which tests passed.

If you are already running an agent-assisted workflow, that question is the one worth automating. MonkeyCode's free model access and free server can carry the boring parts of the replay matrix, but the corpus and the policy stay yours.