The latest articles on DEV Community by Stefanos Palyvos (@stefbuilds). https://dev.to/stefbuilds https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4074769%2F7696a20e-bbdb-4ae3-b0da-7ab45ac33e70.jpg https://dev.to/stefbuilds en Stefanos Palyvos Wed, 12 Aug 2026 12:10:40 +0000 https://dev.to/stefbuilds/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon-project-hhn https://dev.to/stefbuilds/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon-project-hhn <p>The change was small, the diff looked sensible, and the existing tests passed.</p> <p>Then I tried an empty list.</p> <p>The original function returned []. The revised version returned None. Nothing immediately crashed, but code expecting a list had quietly received a different contract.</p> <p>That gap is why I built Cross‑Examine.</p> <p>Cross‑Examine is a verification harness for Python changes. Given a base Git revision and a proposed revision, it captures observed behavior from the base, then replays the same inputs against the changed code.</p> <p>When behavior differs, it preserves a receipt:</p> <p>Input: []<br> Base output: []<br> Head output: None<br> Verdict: BROKEN</p> <p>The receipt includes the reproducing input, the exact command, and captured output from both revisions. It is meant to be something a developer can inspect and rerun, rather than a confidence score from a model.</p> <p>The model has a deliberately limited role. GPT‑5.6 proposes behavior worth checking through a constrained schema, but it cannot produce a verdict. Deterministic execution runs the comparison, and a pure aggregation step decides the result. Cross‑Examine also runs bounded, derandomized Hypothesis examples to look for counterexamples beyond the initial proposal.</p> <p>The idea is not that a tool can prove a PR correct. It is narrower: when an AI-generated change looks plausible and the tests are green, replay behavior that already existed and see whether the change still keeps its promises.</p> <p>Cross‑Examine is currently Python-only, supports a bounded set of call shapes, and should run only on repositories you trust. It executes target code, so it is not a sandbox. Those limits are intentional. An honest UNVERIFIABLE result is better than an unearned “safe.”</p> <p>I built it during OpenAI Build Week with Codex and GPT‑5.6. Codex helped implement the system. At runtime, GPT‑5.6 proposes checks, while deterministic code owns the verdict.</p> <p>The repository includes an offline demo that recreates the empty-list regression without an API key:</p> <p><a href="https://github.com/stefbuilds/cross-examine" rel="noopener noreferrer">https://github.com/stefbuilds/cross-examine</a></p> <p>I’d be interested in where behavior replay would help in your workflow, and where you would be cautious about relying on it.</p> <h1> Python #SoftwareTesting #DevTools #OpenSource #AI </h1> debugging git python testing