DEV Community

Dakota Liu
Dakota Liu

Posted on

Cap the Diff. Freeze the Lockfile. Score From a Sidecar the Agent Cannot Touch.

#testing #ai #python #git

The agent does not get to grade its own homework. That is the whole article.

If the model can edit the tests, the lockfile, and the script that prints PASS, you do not have a gate. You have a diary. I want a kill-switch that lives outside the tree the agent can write, and I want it to fail closed on three boring signals: patch size, lockfile hash, and the test exit code.

Sound harsh? Good. Pretending the work is done is cheap now. Measuring it still is not.

What you will actually build

A fixture repo. A sidecar directory the agent never mounts. A patch-budget checker. A lockfile freeze. A timeout wrapper. A verification command after every stage, not a vibe check at the end.

You can run every command on a laptop. Parking the same loop on a free remote box so a hung job does not melt your fan is optional. The sidecar does not care where it runs.

This is a copy-paste walkthrough, not a production war story. I am not going to invent pass rates. If a step is red, stop. Do not skip ahead and “see if it works.”

Stage 0 — a repo small enough to be honest

Create a throwaway app. Keep the tests out of it. Yes, out of it. Why would the oracle live next to the code the model is allowed to rewrite?

mkdir -p /tmp/budget-demo/{app,scoreboard,incoming}
cd /tmp/budget-demo
Enter fullscreen mode Exit fullscreen mode

The app is unfinished on purpose.

# app/total.py
def total(rows):
    # TODO: skip blanks, sum the rest as ints
    return 0
Enter fullscreen mode Exit fullscreen mode 
# app/requirements.txt
# pin something so a "helpful" agent cannot "just add a package"
# this file is part of the freeze story even if it has no versions yet
Enter fullscreen mode Exit fullscreen mode

And a tiny lock stand-in we will hash. I am not hitting PyPI in this tutorial. A frozen manifest is enough to prove the idea.

# app/manifest.lock
total.py==unreleased
Enter fullscreen mode Exit fullscreen mode

Verification 0. The function is wrong. Stay there.

cd /tmp/budget-demo
PYTHONPATH=app python3 -c "from total import total; print(total(['1','','2']))"
# expect 0, which is the bug
Enter fullscreen mode Exit fullscreen mode

Did it print 0? Keep going. If you already “fixed” it by hand, you skipped the point of the gate.

Stage 1 — write the scoreboard where the agent cannot see it

The tests do not live in app/. They live in scoreboard/. That directory is not on any prompt, not in any mount, not in any diff the model is asked to produce.

# scoreboard/test_total.py
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "app"))
from total import total

def test_skips_blanks():
    assert total(["1", "", "2"]) == 3

def test_empty():
    assert total([]) == 0
Enter fullscreen mode Exit fullscreen mode

Run them against the broken app.

python3 -m pytest /tmp/budget-demo/scoreboard/test_total.py -q
# expect FAIL
Enter fullscreen mode Exit fullscreen mode

Verification 1. Red tests, and app/ does not contain them.

find /tmp/budget-demo/app -name 'test_*.py'
echo "empty find is correct"
Enter fullscreen mode Exit fullscreen mode

If that find prints a test file, you already lost. The agent can rewrite a neighbor. Can you feel how fast “just this one test” becomes the whole suite living inside the sandbox?

Stage 2 — cap the diff before pytest even starts

Unbounded patches are how “one helper” becomes a rewrite of your logging stack. I cap added plus removed lines. Not files. Lines. File allowlists are a different gate.

Here is a checker you can copy. Treat it as a fixture, not a security product.

#!/usr/bin/env python3
"""scoreboard/cap_diff.py — fail if the unified diff is too chatty."""
from __future__ import annotations
import sys
from pathlib import Path

MAX_CHANGED = 40  # added + removed; file headers ignored

def changed_lines(text: str) -> int:
    n = 0
    for line in text.splitlines():
        if line.startswith("+++") or line.startswith("---"):
            continue
        if line.startswith("+") or line.startswith("-"):
            n += 1
    return n

def main() -> int:
    raw = Path(sys.argv[1]).read_text(encoding="utf-8")
    n = changed_lines(raw)
    print(f"changed_lines={n} max={MAX_CHANGED}")
    if n == 0:
        print("empty patch; refuse")
        return 2
    if n > MAX_CHANGED:
        print("over budget; refuse")
        return 3
    return 0

if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Verification 2a. A fat patch must die.

python3 - <<'PY'
from pathlib import Path
p = Path("/tmp/budget-demo/incoming/pad.diff")
lines = ["--- a/total.py\n", "+++ b/total.py\n"]
lines += [f"+noise_{i}\n" for i in range(50)]
p.write_text("".join(lines), encoding="utf-8")
PY
python3 /tmp/budget-demo/scoreboard/cap_diff.py /tmp/budget-demo/incoming/pad.diff
echo exit:$?
# expect 3
Enter fullscreen mode Exit fullscreen mode

Verification 2b. A thin, on-ticket patch must live.

cat > /tmp/budget-demo/incoming/ok.diff <<'EOF'
--- a/total.py
+++ b/total.py
@@ -1,4 +1,9 @@
 def total(rows):
-    # TODO: skip blanks, sum the rest as ints
-    return 0
+    acc = 0
+    for row in rows:
+        if row == "":
+            continue
+        acc += int(row)
+    return acc
EOF
python3 /tmp/budget-demo/scoreboard/cap_diff.py /tmp/budget-demo/incoming/ok.diff
echo exit:$?
# expect 0
Enter fullscreen mode Exit fullscreen mode

Did the fat patch die and the thin one live? Then the budget is real. If both live, your counter is counting headers. Fix the counter. Do not raise MAX_CHANGED to make the demo pretty.

Stage 3 — freeze the lockfile with a hash, not a speech

Agents love to “clean up” manifests. I do not argue with them. I hash.

#!/usr/bin/env python3
"""scoreboard/freeze_lock.py — compare sha256 before and after apply."""
from __future__ import annotations
import hashlib
import sys
from pathlib import Path

def sha(p: Path) -> str:
    return hashlib.sha256(p.read_bytes()).hexdigest()

def main() -> int:
    before, after = Path(sys.argv[1]), Path(sys.argv[2])
    a, b = sha(before), sha(after)
    print(f"before={a}")
    print(f"after={b}")
    if a != b:
        print("lockfile moved; refuse")
        return 4
    return 0

if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Verification 3. Same bytes pass. One-line vandalism fails. Then you restore.

cp /tmp/budget-demo/app/manifest.lock /tmp/budget-demo/incoming/manifest.lock.before
python3 /tmp/budget-demo/scoreboard/freeze_lock.py \
  /tmp/budget-demo/incoming/manifest.lock.before \
  /tmp/budget-demo/app/manifest.lock
echo exit:$?
# expect 0

echo "total.py==hijacked" > /tmp/budget-demo/app/manifest.lock
python3 /tmp/budget-demo/scoreboard/freeze_lock.py \
  /tmp/budget-demo/incoming/manifest.lock.before \
  /tmp/budget-demo/app/manifest.lock
echo exit:$?
# expect 4

cp /tmp/budget-demo/incoming/manifest.lock.before /tmp/budget-demo/app/manifest.lock
Enter fullscreen mode Exit fullscreen mode

If a one-line edit of the lockfile still exits 0, you hashed the wrong path. Stop. Do not “see if tests pass anyway.” A green suite on a mutated lock is how supply-chain noise sneaks in while everyone watches the function body.

Stage 4 — apply in git, then score from the sidecar

The agent never runs pytest. You do. After git apply --check. After the budget. After the freeze. Hung jobs are not “still thinking.” They are leaks. timeout is the first process in the wrapper, not a comment in the prompt.

Initialize a base so apply has something to bite.

cd /tmp/budget-demo/app
git init -q
git add total.py manifest.lock requirements.txt
git -c user.email=dev@example.com -c user.name=demo commit -qm start
Enter fullscreen mode Exit fullscreen mode 
#!/usr/bin/env bash
# scoreboard/run_job.sh
set -euo pipefail
ROOT=/tmp/budget-demo
PATCH=${1:?patch file}

cd "$ROOT/app"
timeout 30s git apply --check "$PATCH"
python3 "$ROOT/scoreboard/cap_diff.py" "$PATCH"
cp "$ROOT/app/manifest.lock" "$ROOT/incoming/manifest.lock.before"
timeout 30s git apply "$PATCH"
python3 "$ROOT/scoreboard/freeze_lock.py" \
  "$ROOT/incoming/manifest.lock.before" \
  "$ROOT/app/manifest.lock"
timeout 30s python3 -m pytest "$ROOT/scoreboard/test_total.py" -q
echo "JOB_OK"
Enter fullscreen mode Exit fullscreen mode

Verification 4a. Confirm GNU timeout actually kills. If this does not print 124, you are on a timeout that is not the one you think it is.

timeout 1s python3 -c "import time; time.sleep(5)"; echo timeout_exit:$?
Enter fullscreen mode Exit fullscreen mode

Verification 4b. Thin patch goes green.

chmod +x /tmp/budget-demo/scoreboard/run_job.sh
/tmp/budget-demo/scoreboard/run_job.sh /tmp/budget-demo/incoming/ok.diff
echo exit:$?
# expect JOB_OK and 0
Enter fullscreen mode Exit fullscreen mode

Verification 4c. Reset. Fat patch must not stick.

cd /tmp/budget-demo/app && git checkout -- .
/tmp/budget-demo/scoreboard/run_job.sh /tmp/budget-demo/incoming/pad.diff
echo exit:$?
# expect non-zero
git -C /tmp/budget-demo/app diff --stat
# expect empty
Enter fullscreen mode Exit fullscreen mode

If the fat patch still mutates total.py, set -e is off, or you applied before the cap. Read the script again. Slowly. Would you merge a job that can write first and apologize later?

Stage 5 — where a free model and a free server actually fit

Up to here, no model was required. That is the point. The scoreboard stays useful if every product name in this file is deleted.

When I do want a model to draft the thin patch, I still do not give it the sidecar path. I give it app/total.py and a prompt file I hashed. The model returns a unified diff. The wrapper above is the only thing that can say yes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option are relevant for one mechanical reason: the timeout 30s kill-switch should not have to live on the laptop you are using to write the sidecar. Park the draft job on the free server. Keep scoreboard/ off the mount the agent sees. Bring the diff back. Score it locally if you want a second opinion.

I am not going to name models, quote a token budget, or pretend a server size. Those numbers go stale. The contract does not: cap the diff, freeze the lock, run pytest from outside, fail closed.

A prompt file you can hash before the job starts:

# incoming/prompt.txt
Return only a unified diff for app/total.py.
Implement total(rows) so blanks are skipped and the rest sum as ints.
Do not touch manifest.lock. Do not add files. Do not write tests.
Enter fullscreen mode Exit fullscreen mode 
sha256sum /tmp/budget-demo/incoming/prompt.txt
Enter fullscreen mode Exit fullscreen mode

If the prompt bytes change after you “just tweaked the wording,” you are debugging a different job. Hash it or you will lie to yourself later. Want to be extra mean? Refuse any model output that is not a unified diff. English recaps are not patches.

Decision table for this fixture

Signal Action
empty diff refuse, exit 2
changed_lines > MAX_CHANGED refuse, exit 3
manifest.lock hash drift refuse, exit 4
pytest non-zero refuse, script dies under set -e
timeout fires refuse, no retry inside the same wrapper
JOB_OK inspect the diff yourself anyway

That last row is not a joke. A twelve-line patch can still be wrong in a way this suite does not see. The sidecar is a floor, not a medal. Who is supposed to read the diff if not you?

Limitations

This does not stop a model from writing clever code that passes test_total and still explodes on real input. It does not replace review. It does not prove the prompt was good. git apply can fail on fuzzy context. I want that failure. I do not want a three-way merge “helping.”

MAX_CHANGED = 40 is arbitrary. Pick a number that matches the ticket, not a blog post. Hashing one lockfile does not freeze a supply chain. timeout does not kill grandchildren unless you set process-group options. pytest on one file is not your CI.

I also do not run this as root, and I do not let the job inherit AWS_* or GITHUB_TOKEN. Environment inheritance is a different hole. Do not widen this tutorial to cover it and then ship a leaky wrapper.

Who should not use this

Do not use a sidecar-outside-the-tree setup if your org already mounts the whole monorepo into the agent and you cannot change that. You will copy tests into app/tests “just for convenience” and the gate dies.

Do not use it if the task is an exploratory refactor with no lockfile and no test oracle. A line budget on a greenfield spike just makes you split one rewrite into five lying patches.

Do not park customer dumps, private keys, or unreleased product code on a shared free server. Local sidecar, local apply, no upload. The kill-switch is not worth that trade.

Stop when the fat patch dies

I will not add a dashboard. I will not parse the model’s English recap. Run stage 0 through 4 until the fat patch dies and the thin patch is the only thing that prints JOB_OK.

Then, if you want the draft step off your laptop, try MonkeyCode’s free model access on the free server with the sidecar left behind. That is the invitation. No countdown.

Top comments (0)