One Test Must Flip, One Must Not: A Pairing Gate for Agent Patches

A rate limiter let through one request too many at the edge of its window. No outage, no page, just a 429 that never fired for a client hammering an endpoint at exactly the boundary. I was the junior in that pairing session, and the patch I brought in the next morning came from a free coding model. The senior did not read the diff first. She asked three questions first.

Before going further: the session below is a reconstructed composite assembled from pairing notes, not a verbatim transcript. Treat the dialogue as illustrative. Treat the harness in the middle of this post as the reproducible part, because it is runnable and it is the actual artifact that survived the session.

The reproduction we started from

The bug lived in a sliding-window limiter. The eviction loop removed timestamps that were strictly older than the window and the size check used >, so a burst could squeak past the configured limit by one request at the boundary.

# ratelimit.py  (before the patch)
import time
from collections import deque

class SlidingWindow:
    def __init__(self, limit, window_s, clock=time.monotonic):
        self.limit = limit
        self.window_s = window_s
        self.clock = clock
        self._hits = deque()

    def allow(self):
        now = self.clock()
        while self._hits and now - self._hits[0] > self.window_s:
            self._hits.popleft()
        if len(self._hits) > self.limit:
            return False
        self._hits.append(now)
        return True

The injectable clock is the only reason this session went anywhere. Without it, every test would be a sleep-and-hope, and no agent patch would ever be provable.

Round one: the question that reshaped the task

The first senior question was not about code. It was: what does the spec say at exactly t = window?

Neither of us knew. That is not a small thing, because the wall-clock boundary decides which operator is correct. The senior wrote the answer on the whiteboard — window is half-open, [t - window, t) — and only then we looked at my patch.

Her second question was: what did you ask the model for?

I had asked it to "fix the rate limiter." The patch rewrote the class, added a threading.Lock, changed the constructor to accept a maxlen replay buffer, and renamed allow() to try_acquire(). Three of those four changes were unrelated to the bug, and one of them broke every caller in the repository.

The third question was the one I kept: which test will flip, and which test is not allowed to flip?

The gate: one flip test, one guard test

That question became the acceptance gate. Any agent patch must be evaluated against two tests, not one:

1. Flip test — a test that fails on the current code and must pass after the patch. It encodes the bug.
2. Guard test — a test that passes on the current code and must still pass afterward. It encodes the behavior you are not paying for.

The guard test is the part most people skip. Without it, a patch that "fixes" the limiter by disabling eviction entirely looks like a win.

# test_ratelimit_gate.py
from ratelimit import SlidingWindow

class FakeClock:
    def __init__(self): self.t = 0.0
    def __call__(self): return self.t


def test_flip_limit_is_enforced_at_boundary():
    """FAILS before the patch, must PASS after. Encodes the bug."""
    clock = FakeClock()
    rl = SlidingWindow(limit=3, window_s=10.0, clock=clock)
    for _ in range(3):
        assert rl.allow() is True
    # Fourth call inside the same window must be rejected.
    assert rl.allow() is False


def test_guard_window_expiry_unchanged():
    """PASSES before the patch, must STILL PASS after. Guards adjacent behavior."""
    clock = FakeClock()
    rl = SlidingWindow(limit=1, window_s=10.0, clock=clock)
    assert rl.allow() is True
    clock.t = 9.999
    assert rl.allow() is False
    clock.t = 10.0
    assert rl.allow() is True

Run the gate as a single command so the result is a fact, not an opinion:

#!/usr/bin/env bash
# gate.sh — run from a clean checkout of the branch under review
set -euo pipefail

spec="./test_ratelimit_gate.py::test_guard_window_expiry_unchanged"
flip="./test_ratelimit_gate.py::test_flip_limit_is_enforced_at_boundary"

# Guard must be green on the pre-patch tree.
if ! pytest -q "$spec" >/dev/null 2>&1; then
  echo "REJECT: guard test is already red before the patch" >&2; exit 2
fi

# Flip must be red on the pre-patch tree. If it is green, the test is wrong.
if pytest -q "$flip" >/dev/null 2>&1; then
  echo "REJECT: flip test does not reproduce the bug" >&2; exit 3
fi

echo "pre-patch state verified; apply the candidate diff now"

The gate refuses to evaluate a patch until the pre-patch state is proven. That single check killed two of the three patches I brought to the session. The first patch produced a flip test that was already green — I had written the test against the patched tree. The second touched __init__ and broke seven callers; the guard test caught nothing there, so we added a caller-collection test as a third gate.

Decision table: what goes to a free model, what stays with the pair

Task shape	Hand to a model?	Why
Write the flip test from a written spec	Yes	Cheap to verify: it must be red
Write the guard test	Partly	Models over-guard; the pair prunes to two assertions
Rename or reshape a public signature	No	Diff cost lands on callers you did not show it
Explain an unfamiliar module	Yes	Output is conversation, not a commit
Concurrency or clock behavior	No, until the clock is injectable	Unverifiable claims otherwise
Multi-file refactor with no tests	No	No gate exists to accept it

Where the free model and the free server actually fit

The model's useful contribution was the second opinion on the boundary spec, not the diff. I pasted the test file and the failing assertion, asked for counterexamples at t = window - epsilon and t = window, and got two cases I had not written. I then wrote them myself inside the gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project whose operators also provide hosted access. Per the operators, that access includes a free tier with 10 million tokens and a free server option. I have not independently audited either figure, and free-tier terms and capacity change, so verify the current limits on the project's own page before you plan a workflow around them. In this session the free server mattered for one narrow reason: the gate needs a clean checkout with the project's own Python and pytest versions, and a disposable remote box is easier to throw away than a laptop with three virtualenvs and a stale .pyc cache.

The pre-flight I ran on the remote box was four commands:

git status --porcelain            # must be empty, or abort
git rev-parse --short HEAD        # record the exact pre-patch commit
python -V && pytest --version     # record the toolchain the gate ran under
cp -r . /tmp/gate-$(date +%s)     # snapshot so the revert is one path, not one memory

That commit hash and toolchain line go into the pull request body. A gate result without the commit it ran against is not evidence.

Dead ends worth remembering

• Asking the model to "find the bug" before the spec was written. It produced three plausible bugs and no way to rank them. The spec came first, and then the model was useful.
• One big patch. The first candidate was 90 lines. We could not attribute any single failure to a single change, so we rejected it on that basis alone and asked for the smallest diff that could flip the test.
• Trusting a green flip test. A flip test that is green before the patch proves nothing about the bug. The gate now fails loudly on that condition instead of reporting success.
• Running the gate on a dirty tree. A leftover edit in the working directory made one guard test pass for the wrong reason. git status --porcelain is now step one.

Limitations, and who should not use this

This gate assumes you can express the bug as an executable test. If the defect is a race, a memory ceiling, or a UX judgment call, one flip test will not capture it.

Skip this approach if:

• The codebase has no runnable test harness and you will not build one this week.
• The change is a signature or interface refactor where every caller is a potential guard test.
• You are optimizing for throughput rather than for a defensible diff.
• The free server is your only environment and it cannot reproduce your production clock, locale, or hardware behavior. Do not accept a gate result from a box that differs from production in ways that matter to the assertion.

The decision we kept

The senior's rule survived the session intact: a patch is not reviewed until one test flips and one test does not. Everything else was negotiable — the model, the branch name, the box it ran on. The gate was the part we kept, and it is the only part of that morning I still run today.

If you want to try the second-opinion pass on a clean box, the project's free tier and free server option are a low-friction place to start. Read the current terms yourself rather than trusting any number in a blog post, including this one.