Most AI refactor attempts fail at review, not at generation. The model writes plausible code. Nobody can prove the behavior stayed the same.
The fix is not a better prompt. It is a gate the patch must pass before a human reads it.
This article describes a loop with three parts. A frozen behavior corpus, a hard change budget, and a rejection log that feeds the next prompt. The artifact below is illustrative. It was not executed for this post, so treat the numbers as knobs, not results.
The loop in one paragraph
You freeze current behavior first. Then you let a model propose exactly one small patch. A script measures two things: did outputs drift, and did the patch exceed its size budget. Failing patches are rejected automatically and logged. The log becomes the constraint in the next prompt. This keeps the model useful while removing its ability to bluff.
Step 0: Prerequisites
- Git with a clean working tree. The gate diffs against
HEAD. - A corpus of inputs and recorded outputs. Even twelve cases beat zero.
- A deterministic entry point. If output is random, fix the seed first.
git status --porcelain # must print nothing
If that prints anything, commit or stash. The budget check in Step 3 counts lines against HEAD, and stale edits corrupt the count.
Step 1: Freeze the behavior you refuse to change
Write a corpus runner that prints stable, sorted JSON. Sorting matters. Dict ordering and set ordering create false drift.
# corpus.py -- illustrative, not executed
import json, random
from app.pricing import quote # the messy function under refactor
CASES = [
{"sku": "A1", "qty": 1, "tier": "basic"},
{"sku": "A1", "qty": 7, "tier": "basic"},
{"sku": "B2", "qty": 3, "tier": "pro", "coupon": "SAVE10"},
{"sku": "B2", "qty": 0, "tier": "pro"},
]
def snapshot() -> list:
out = []
for case in CASES:
random.seed(1234) # pin any internal randomness
try:
out.append({"in": case, "out": quote(**case)})
except Exception as exc:
out.append({"in": case, "raised": type(exc).__name__})
return out
if __name__ == "__main__":
print(json.dumps(snapshot(), sort_keys=True, indent=2))
Record exceptions as data. A refactor that turns a raise into a return value is a behavior change, even if it feels like an improvement.
python corpus.py > corpus/golden.json
Step 2: Set a change budget you can defend
A small patch is reviewable. A large patch is a rewrite wearing a refactor label.
# budget.py -- illustrative, not executed
BUDGET = {
"max_changed_lines": 25, # added + removed across tracked files
"max_new_files": 0,
"forbidden_paths": ["migrations/", "settings/", "conftest.py"],
}
Pick the numbers from your own review tolerance. If you cannot read a diff in ten minutes, the budget is too high.
Step 3: The gate script
The gate runs the corpus, compares to golden, measures the diff, and exits non-zero on any violation.
# gate.py -- illustrative, not executed
import json, pathlib, subprocess, sys
from budget import BUDGET
GOLDEN = pathlib.Path("corpus/golden.json")
def corpus_output() -> dict:
proc = subprocess.run(
[sys.executable, "corpus.py"], capture_output=True, text=True
)
if proc.returncode != 0:
sys.exit(f"corpus crashed:\n{proc.stderr}")
return json.loads(proc.stdout)
def diff_stats() -> tuple[int, list[str]]:
proc = subprocess.run(
["git", "diff", "--numstat", "HEAD"], capture_output=True, text=True
)
changed, files = 0, []
for line in proc.stdout.splitlines():
added, removed, path = line.split("\t")
files.append(path)
changed += int(added) + int(removed)
return changed, files
def main() -> int:
golden = json.loads(GOLDEN.read_text())
now = corpus_output()
reasons = []
if now != golden:
reasons.append("behavior drift detected")
changed, files = diff_stats()
if changed > BUDGET["max_changed_lines"]:
reasons.append(f"budget: {changed} lines > {BUDGET['max_changed_lines']}")
for path in files:
if any(path.startswith(p) for p in BUDGET["forbidden_paths"]):
reasons.append(f"forbidden path touched: {path}")
verdict = "REJECT" if reasons else "PASS"
print(json.dumps({"verdict": verdict, "lines": changed, "reasons": reasons}))
return 1 if reasons else 0
if __name__ == "__main__":
raise SystemExit(main())
Note what this gate cannot see. It does not judge style. It does not prove the new code is correct. It only proves the old contract survived and the patch stayed small.
Step 4: Trust the gate before you trust the model
Run the gate against a deliberately broken patch first.
python gate.py # expect PASS on clean tree
sed -i 's/qty \* price/qty * price + 1/' app/pricing.py # inject a bug
python gate.py # expect REJECT
If the broken patch passes, your corpus is too thin. Add cases until the mutation is caught. This mutation check is the only honest way to claim your harness works.
Step 5: Log rejections, then reuse them as constraints
Each rejection is a free prompt ingredient. Append one JSON line per attempt.
# log.py -- illustrative, not executed
import json, pathlib, time
LOG = pathlib.Path("corpus/rejections.jsonl")
def record(attempt: int, reasons: list[str], patch_sha: str) -> None:
row = {"ts": time.time(), "attempt": attempt, "reasons": reasons, "sha": patch_sha}
with LOG.open("a") as fh:
fh.write(json.dumps(row) + "\n")
Then the next prompt carries the constraint, not the complaint.
Previous attempt was rejected for: behavior drift on qty=7 tier=basic.
Propose one patch under 25 changed lines. Do not touch migrations/ or settings/.
Preserve all outputs in corpus/golden.json exactly.
Where free model access and a free server fit
This loop is chatty. It needs many small proposals, and each one burns a run. That is exactly the workload where metered access punishes exploration.
MonkeyCode, per the operator, offers free model access and a free server option. The free server matters here for a boring reason. Long gate runs and repeated proposals tie up a machine. Moving them off your laptop keeps the loop running while you work.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I am not claiming specific models, quotas, hardware, or uptime. Those are operator-supplied and not verified here. Check the project's own documentation before you plan around them. The project is open source, so you can read the code rather than trust a landing page.
One soft suggestion: run Step 4 against the gate before you route any model traffic through it. A harness you have not tested is a vibe with a shebang.
Decision table
| Situation | Use this loop? | Why |
|---|---|---|
| Function is messy but deterministic | Yes | Corpus is cheap, drift is detectable |
| Output includes timestamps or UUIDs | Yes, after normalizing | Raw values create false drift |
| Heavy concurrency side effects | Carefully | Corpus must assert on final state, not interleaving |
| No tests and no sample inputs | No, capture first | The gate has nothing to compare against |
| You want a full module rewrite | No | Budget will reject every attempt, correctly |
| Behavior is genuinely undefined | No | You cannot preserve what you cannot state |
Limitations
- Determinism is assumed. Seeded randomness works. Real I/O timing does not.
- The corpus only covers cases you thought of. Absence of drift is not proof of correctness.
- Line budgets are crude. A one-line change can still be catastrophic.
- Rejection reasons bias the next proposal. Too-specific constraints can make the model timid.
- The gate does not review naming, structure, or intent. A human still reads every PASS.
Who should skip this
Skip it if you cannot produce a stable corpus in an hour. Skip it if your team will rubber-stamp PASS verdicts. Skip it if the task is greenfield design, where there is no behavior to preserve.
Also skip it if you expect the model to choose the architecture. This loop constrains edits. It does not delegate judgment.
The smallest safe change is not the smallest change a model can write. It is the smallest change your evidence can defend.
Build the gate first. Then let the model in.
Top comments (0)