Reproduce Before You Patch: A First-Week Gate for AI Diffs

Green CI is not evidence. A red-to-green test transition is.

You just joined a repo. Your first PR arrives from an AI agent. The suite is green, so you merge it. Two days later the same bug returns in a different file.

This post gives you one gate to run before your first merge. It takes about ten minutes. It uses only the test runner you already have.

The current debate about AI-written code is mostly about trust. This gate turns trust into a measurement you can read in a terminal.

The problem with green CI in week one

CI tells you the repo was healthy at some point. It does not tell you the patch changed anything.

On a fresh clone you cannot tell the difference. You have no memory of which tests failed yesterday. Neither does the agent.

So capture the suite state before the patch, not after. That single ordering decision is the whole gate.

Step 1: Capture the pre-patch baseline

Create a branch at the patch point. Record the target test result and the full suite result.

git switch -c gate/baseline origin/main
TARGET="rejects expired session token" ./tools/gate.sh before

Here is the script. Treat it as a template, not a drop-in: swap npm test and npx jest for your runner.

#!/usr/bin/env bash
# tools/gate.sh -- template, adapt runner flags to your repo
set -uo pipefail
PHASE="${1:?usage: gate.sh before|after}"
TARGET="${TARGET:?set TARGET to the reproducing test name}"
mkdir -p .gate

# full suite first, so you always get a number
npm test --silent > ".gate/$PHASE.suite.txt" 2>&1
echo "suite=$?" >> ".gate/$PHASE.status"

# the one test that is supposed to reproduce the bug
npx jest -t "$TARGET" > ".gate/$PHASE.target.txt" 2>&1
echo "target=$?" >> ".gate/$PHASE.status"

cat ".gate/$PHASE.status"

Use set -uo pipefail and never -e. A failing test must not stop the script.

The baseline line must read target=1. If it reads target=0, stop there. You have not reproduced the bug, so nothing downstream means anything.

Step 2: Ask for the test, not the fix

Send the agent the issue text and the failing input. Ask for one test only. No production code yet.

A good reproducing test has three properties:

1. It fails on the baseline commit, for the reason in the ticket.
2. It names the user-visible symptom in its description.
3. It asserts on observable behavior, not on internal call counts.

Reject any test that mocks the function under test. Those pass for the wrong reason and hide the bug later.

Step 3: Freeze a small guard set

Pick five to ten existing tests in or near the changed files. These are your guards. They must stay green after the patch.

Do not let the agent edit them in the same PR. If a guard test genuinely needs a change, that is a separate PR with its own explanation.

# guards.txt -- one test name per line, reviewed by a human
rejects expired session token
rotates the token after refresh
logs out on missing header

Keep this list short. A guard set you cannot read in one screen is a guard set you will not check.

Step 4: Re-run after the patch and compare

Apply the agent's diff on a fresh branch. Re-run the same script with the same TARGET.

git switch gate/agent-patch
TARGET="rejects expired session token" ./tools/gate.sh after
paste .gate/before.status .gate/after.status

The output is two columns. The left is before, the right is after. You now have a table, not an opinion.

The decision table

Before: target	After: target	After: guards	Verdict
fail	pass	all pass	Accept. Red-to-green on one named behavior.
fail	fail	any	Reject. The patch does not change behavior.
pass	pass	all pass	Investigate. The test never reproduced the bug.
fail	pass	any fail	Reject. The patch traded one failure for another.
absent	—	—	Reject. There is no evidence to read.

Row three is the one juniors miss. A test that passed before the patch proves nothing, even if it is green now.

Where a hosted model fits in this loop

You need two things to run this gate on a scratch machine: a way to draft the reproducing test, and somewhere to run it. I used MonkeyCode for both while onboarding, through its free model access and its free server option.

That is a narrow role. The model drafts a test from the ticket text. I read it, trim it, and keep only the assertion that fails on the baseline. The verdict still comes from the script, never from the model.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free model access matters for one reason here: cost stops being the excuse for skipping the test. The free server option matters because a clean box gives you a clean baseline. Treat both as operator-stated availability and check the current terms yourself, since quotas and offers change.

Limitations

This gate assumes deterministic tests. Flaky suites produce false reds, and the whole table becomes noise.

It says nothing about untested behavior. A patch can pass every check and still be wrong in a path no test covers.

It is also weak on config, docs, and dependency bumps. Those diffs rarely have a reproducing test, so use a different review method there.

The scripts above are templates. They are not run against your repository, and the runner flags will differ.

Who should not use this

Skip this approach if your suite takes more than twenty minutes on a fresh clone. Run only the TARGET job instead, and note that you skipped the guards.

Skip it if the repo has no test runner at all. Write the harness first; a gate without a runner is just a ritual.

Skip it for generated files, lock files, and formatting-only patches. There is nothing behavioral to measure.

Your first-week checklist

1. Run gate.sh before and confirm target=1.
2. Ask the agent for one reproducing test, not a fix.
3. Freeze five to ten guard tests by hand.
4. Run gate.sh after and paste the two status files side by side.
5. Accept only on a red-to-green transition with guards intact.

If you want to try this on a throwaway box instead of your laptop, MonkeyCode offers free model access and a free server option, and the project terms live on its site.

One gate, run twice, tells you more than a green badge ever will.