A Reproducible Bug-Fix Trial for AI Coding Assistants Using Disposable Git Worktrees

Harper Zhu — Wed, 05 Aug 2026 06:57:42 +0000

AI coding assistants are easy to demo and hard to compare. One model fixes a failing test but rewrites half the file. Another produces a beautiful patch that does not apply. A third passes the visible test and quietly breaks an edge case.

This article describes a small, repeatable workflow for evaluating whether an AI assistant can repair a real bug without trusting vibes. It uses disposable Git worktrees, a frozen failing test, constrained patch prompts, and a simple scorecard. It works with local models, hosted endpoints, or a free tier; the evaluation logic stays the same.

The problem: "it fixed the test" is not enough

A useful bug-fix trial needs to answer four questions:

Did the patch apply cleanly?
Did it make the failing test pass?
Did it break anything else in a relevant test subset?
Was the change small and reviewable enough to trust?

If the workflow does not isolate each attempt, results get polluted by previous edits, half-applied patches, or dependency drift.

Artifact: an isolated patch trial script

The script below uses git worktree so every model attempt starts from the same clean commit. It expects you to provide:

a failing test command, such as pytest tests/test_parser.py -q
a broader verification command, such as pytest tests -q
an environment variable AI_PATCH_CMD that reads a prompt on stdin and prints a unified diff to stdout

Save as scripts/ai_patch_trial.sh:

#!/usr/bin/env bash
set -euo pipefail

BASE_BRANCH="${1:-main}"
TEST_CMD="${2:-pytest tests/test_parser.py -q}"
VERIFY_CMD="${3:-pytest tests -q}"
TRIAL_ID="${4:-trial-$(date +%Y%m%d-%H%M%S)}"

ROOT="$(git rev-parse --show-toplevel)"
WORKTREES_DIR="$ROOT/.ai-trials"
TRIAL_DIR="$WORKTREES_DIR/$TRIAL_ID"
PATCH_FILE="$TRIAL_DIR/model.patch"
PROMPT_FILE="$TRIAL_DIR/prompt.md"
RESULT_FILE="$TRIAL_DIR/result.txt"

mkdir -p "$WORKTREES_DIR"

git worktree add --detach "$TRIAL_DIR" "$BASE_BRANCH" >/dev/null

cat > "$PROMPT_FILE" <<'EOF'
You are given a repository with a failing test.

Rules:
- Return only a unified diff patch.
- Do not explain the patch.
- Change the smallest amount of production code needed.
- Do not modify tests unless they are demonstrably wrong.
- Do not add dependencies.
- Preserve existing public behavior except for the bug being fixed.

Context:
EOF

{
  echo "- Failing command: $TEST_CMD"
  echo "- Verify command: $VERIFY_CMD"
  echo "- Relevant files:"
  git -C "$TRIAL_DIR" status --short
  echo
  echo "- Recent commit:"
  git -C "$TRIAL_DIR" log -1 --oneline
} >> "$PROMPT_FILE"

cd "$TRIAL_DIR"

echo "[trial] collecting baseline failure"
set +e
BASELINE_OUTPUT="$($TEST_CMD 2>&1)"
BASELINE_STATUS=$?
set -e

if [[ $BASELINE_STATUS -eq 0 ]]; then
  echo "Baseline test unexpectedly passed. Freeze a failing test before comparing models." | tee "$RESULT_FILE"
  exit 2
fi

{
  echo
  echo "Baseline failure output:"
  echo '```

text'
  echo "$BASELINE_OUTPUT"
  echo '

```'
} >> "$PROMPT_FILE"

echo "[trial] requesting patch from AI_PATCH_CMD"
if [[ -z "${AI_PATCH_CMD:-}" ]]; then
  echo "AI_PATCH_CMD is not set. Example: AI_PATCH_CMD='python scripts/ask_model.py'" | tee "$RESULT_FILE"
  exit 3
fi

bash -lc "$AI_PATCH_CMD" < "$PROMPT_FILE" > "$PATCH_FILE"

if [[ ! -s "$PATCH_FILE" ]]; then
  echo "Model returned an empty patch." | tee "$RESULT_FILE"
  exit 4
fi

echo "[trial] checking patch applicability"
if ! git apply --check "$PATCH_FILE" 2>"$TRIAL_DIR/apply-check.err"; then
  echo "Patch failed git apply --check." | tee "$RESULT_FILE"
  exit 5
fi

git apply "$PATCH_FILE"

echo "[trial] running targeted test"
set +e
TARGET_OUTPUT="$($TEST_CMD 2>&1)"
TARGET_STATUS=$?
set -e

echo "[trial] running verification subset"
set +e
VERIFY_OUTPUT="$($VERIFY_CMD 2>&1)"
VERIFY_STATUS=$?
set -e

CHANGED_FILES="$(git diff --name-only)"
CHANGED_LINES="$(git diff --numstat | awk '{adds += $1; dels += $2} END {print adds + dels}')"

{
  echo "trial_id=$TRIAL_ID"
  echo "baseline_status=$BASELINE_STATUS"
  echo "target_status=$TARGET_STATUS"
  echo "verify_status=$VERIFY_STATUS"
  echo "changed_lines=$CHANGED_LINES"
  echo "changed_files=$CHANGED_FILES"
  echo
  echo "target_output:"
  echo "$TARGET_OUTPUT"
  echo
  echo "verify_output:"
  echo "$VERIFY_OUTPUT"
} | tee "$RESULT_FILE"

echo "[trial] artifacts written to $TRIAL_DIR"

Make it executable:

chmod +x scripts/ai_patch_trial.sh

The key property is isolation: every attempt gets its own worktree under .ai-trials/, and the main checkout stays untouched.

A generic model adapter

Keep the model client separate from the trial harness. This avoids baking a vendor-specific assumption into the evaluation.

Save as scripts/ask_model.py:

#!/usr/bin/env python3
import os
import sys
import urllib.request
import json

API_URL = os.environ.get("AI_API_URL")
API_KEY = os.environ.get("AI_API_KEY", "")
MODEL = os.environ.get("AI_MODEL", "")

if not API_URL:
    raise SystemExit("Set AI_API_URL to an OpenAI-compatible chat completions endpoint.")

prompt = sys.stdin.read()

payload = {
    "model": MODEL or "default",
    "messages": [
        {"role": "system", "content": "Return only a unified diff patch."},
        {"role": "user", "content": prompt},
    ],
    "temperature": 0,
}

req = urllib.request.Request(
    API_URL,
    data=json.dumps(payload).encode("utf-8"),
    headers={
        "Content-Type": "application/json",
        **({"Authorization": f"Bearer {API_KEY}"} if API_KEY else {}),
    },
    method="POST",
)

with urllib.request.urlopen(req, timeout=120) as resp:
    data = json.loads(resp.read().decode("utf-8"))

content = data["choices"][0]["message"]["content"]
print(content)

This adapter is intentionally generic. If a provider exposes an OpenAI-compatible endpoint, point AI_API_URL at it. If it does not, replace ask_model.py while keeping ai_patch_trial.sh unchanged.

Where MonkeyCode can fit

MonkeyCode can be a useful backend for this kind of trial when the goal is to keep experimentation cost low: the operator-supplied availability claims for this article are that MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not assuming specific model names, quotas, uptime, hardware, context windows, or permanence. The safe way to use a free option here is as a replaceable endpoint behind AI_PATCH_CMD, not as a magic fixer. If MonkeyCode's server is OpenAI-compatible in your setup, the generic adapter may be enough; otherwise, write a small client and keep the scoring identical.

Score patches with a decision table

Do not reduce model quality to "pass" or "fail." A patch can pass a test and still be a bad review candidate.

Signal	Good	Warning	Reject
Patch applies	`git apply --check` passes	Needs manual rebase	Patch is prose, not diff
Target test	Failing test passes	Passes only after retries	Still fails
Broader subset	No regressions	Unrelated flaky failure	New deterministic failure
Diff size	Small, localized	Touches several files	Rewrites module boundaries
Test edits	None	Updates only invalid fixture	Weakens assertion to pass
Reviewability	Obvious intent	Needs domain context	Cannot explain why it works

A practical pass threshold for a first screen is:

target test passes
verification subset has no new deterministic failures
production diff is under a reviewable threshold you choose, such as 80 changed lines
no test assertion is weakened

Suggested trial protocol

Run at least five attempts per bug per model configuration:

export AI_API_URL="https://example.invalid/v1/chat/completions"
export AI_MODEL="replace-with-your-configured-model"
export AI_PATCH_CMD="python scripts/ask_model.py"

for i in 1 2 3 4 5; do
  ./scripts/ai_patch_trial.sh main \
    "pytest tests/test_parser.py -q" \
    "pytest tests -q" \
    "parser-bug-run-$i" || true
done

Then summarize manually or with a small script. Do not average away failures; record the failure classes separately. A model that produces two good patches and three non-applying diffs is different from one that produces five mediocre but applicable patches.

Limitations

A passing test does not prove correctness. It proves the observed failure changed.
Small benchmarks reward small bugs. Large architectural defects need design review, not patch gambling.
Free or shared endpoints may have changing latency, limits, or behavior. Re-run baseline checks before comparing results across days.
If the prompt includes sensitive code, do not send it to a hosted endpoint unless your project policy allows it.
Flaky suites can make an assistant look better or worse than it is. Quarantine flaky tests before drawing conclusions.

Who should not use this approach

Do not use this workflow as the only gate for security fixes, payment logic, data migrations, safety-critical behavior, or irreversible production changes. It is a screening tool for model-assisted bug fixing, not a substitute for review, property-based tests, threat modeling, or staged rollout.

Takeaway

The valuable pattern is not the specific endpoint. It is the separation of concerns: freeze a failing test, isolate each attempt, constrain the output to a reviewable diff, and score failure modes explicitly. If you try this with a free backend, keep the adapter replaceable and publish your rubric with the results; that is what makes the comparison credible instead of promotional.

DEV Community: Harper Zhu