How I caught an AI-generated Python regression that the tests missed -hackathon project

#debugging #git #python #testing

The change was small, the diff looked sensible, and the existing tests passed.

Then I tried an empty list.

The original function returned []. The revised version returned None. Nothing immediately crashed, but code expecting a list had quietly received a different contract.

That gap is why I built Cross‑Examine.

Cross‑Examine is a verification harness for Python changes. Given a base Git revision and a proposed revision, it captures observed behavior from the base, then replays the same inputs against the changed code.

When behavior differs, it preserves a receipt:

Input: []
Base output: []
Head output: None
Verdict: BROKEN

The receipt includes the reproducing input, the exact command, and captured output from both revisions. It is meant to be something a developer can inspect and rerun, rather than a confidence score from a model.

The model has a deliberately limited role. GPT‑5.6 proposes behavior worth checking through a constrained schema, but it cannot produce a verdict. Deterministic execution runs the comparison, and a pure aggregation step decides the result. Cross‑Examine also runs bounded, derandomized Hypothesis examples to look for counterexamples beyond the initial proposal.

The idea is not that a tool can prove a PR correct. It is narrower: when an AI-generated change looks plausible and the tests are green, replay behavior that already existed and see whether the change still keeps its promises.

Cross‑Examine is currently Python-only, supports a bounded set of call shapes, and should run only on repositories you trust. It executes target code, so it is not a sandbox. Those limits are intentional. An honest UNVERIFIABLE result is better than an unearned “safe.”

I built it during OpenAI Build Week with Codex and GPT‑5.6. Codex helped implement the system. At runtime, GPT‑5.6 proposes checks, while deterministic code owns the verdict.

The repository includes an offline demo that recreates the empty-list regression without an API key:

https://github.com/stefbuilds/cross-examine

I’d be interested in where behavior replay would help in your workflow, and where you would be cautious about relying on it.