Green Locally, Red on the Server: A Two-Environment Gate for Agent Patches

A passing test run is a claim about one machine, not about the patch. Agent-generated patches produce those claims quickly and confidently, which means a single-environment gate is mostly measuring the machine. The workflow below runs one narrow test selection twice per environment, stores all four results as a single artifact, and treats divergence between environments as a distinct failure mode from flakiness.

The failure mode: environment-coupled green

Most patch gates ask "did the tests pass?" Very few ask "did they pass the same way in two places?" When an agent patch is graded only where it was written, these couplings survive review:

Coupling	Symptom after merge	Cheap detector
Case-insensitive filesystem	import Models works on macOS, breaks on Linux ext4	git ls-files vs the actual import path
Timezone / clock	a date-boundary assertion passes at UTC+8, fails under TZ=UTC	rerun with TZ=UTC
Locale collation	sort order of mixed-case keys changes under LC_ALL=C.UTF-8	pin the locale in one place
Parallelism	a session fixture passes with -p no:xdist, fails with -n auto	run twice with different worker counts
Working-tree state	a fixture file exists locally, is gitignored, and is absent in a clean clone	git clean -nxd before the run

None of these are exotic. All of them are invisible if the patch is only ever executed on the author's machine, and with agent-authored patches, "the author's machine" is exactly where the first green appears.

Three stages, one artifact

The strategy has three stages. The third one is the stage most teams never build.

1. Property checks decide what "correct" means.
2. Fixture locks decide what "the same input" means.
3. Parity replay decides where the claim was evaluated.

Stage 3 is what makes stages 1 and 2 falsifiable. A property check that only ever ran under one locale, one clock, and one filesystem is a hypothesis, not evidence.

Stage 1: property checks that survive a different machine

Assert invariants that do not depend on machine state, and make input ordering explicit. The example below uses a small ledger, but the shape applies to parsers, schedulers, and merge logic.

# tests/test_ledger_properties.py
import random

def replay(ops):
    ledger = Ledger()
    for op in ops:
        ledger.apply(op)
    return ledger.entries()

def test_replay_is_order_independent_for_commuting_ops():
    ops = [Deposit("acct-1", 100), Deposit("acct-2", 250)]
    baseline = replay(ops)
    for seed in range(8):
        shuffled = ops[:]
        random.Random(seed).shuffle(shuffled)
        assert replay(shuffled) == baseline

Three rules keep this property environment-stable:

• Never assert on wall-clock values. Inject the clock and compare durations, not timestamps.
• Never assert on dictionary or set iteration order. Sort with an explicit key, or compare sets when order is not part of the contract.
• Never assert on locale-dependent formatting. Format with an explicit locale, or compare parsed values.

If a property fails only under TZ=UTC, you have found environment coupling, not a flaky test.

Stage 2: pin the environment in one place

Scattered os.environ writes are how coupling gets introduced. Pin everything in a session fixture so the pins are reviewable in a single diff.

# tests/conftest.py
import os, random, time, pytest

@pytest.fixture(autouse=True, scope="session")
def pinned_environment():
    os.environ.update({
        "TZ": "UTC",
        "LC_ALL": "C.UTF-8",
        "PYTHONHASHSEED": "0",
        "PYTHONDONTWRITEBYTECODE": "1",
    })
    time.tzset()          # POSIX only
    random.seed(0)
    yield

Two honest caveats: time.tzset() does not exist on Windows, and PYTHONHASHSEED only fully takes effect when it is set before the interpreter starts. The parity stage is what catches that second one.

Fixtures an agent can edit need their own lock, otherwise "fix the fixture" becomes a way to make a patch pass:

# tests/test_fixture_lock.py
import hashlib, json, pathlib

LOCK = pathlib.Path("tests/fixtures.lock.json")

def test_fixtures_match_lock():
    locked = json.loads(LOCK.read_text())
    for rel, expected in locked.items():
        actual = hashlib.sha256(pathlib.Path(rel).read_bytes()).hexdigest()
        assert actual == expected, f"fixture changed: {rel}"

Refresh it deliberately with python -m tools.lock_fixtures --write, so the diff shows exactly which recordings moved.

Stage 3: the parity gate

This is the artifact. One selection, two targets, two attempts each. Adapt the runner flags to your test framework; the script assumes pytest-randomly for the seed flag.

#!/usr/bin/env python3
"""parity_gate.py - run one selection twice per target; refuse on divergence."""
import json, os, re, shlex, subprocess, sys

SELECTION = shlex.split(os.environ.get("PARITY_SELECTION", "tests/"))
ATTEMPTS = int(os.environ.get("PARITY_ATTEMPTS", "2"))
SEEDS = [os.environ.get("PARITY_SEED_A", "1"), os.environ.get("PARITY_SEED_B", "2")]
TARGETS = {
    "local": None,                                  # run in this process's cwd
    "server": os.environ.get("PARITY_SSH_TARGET"),  # e.g. user@host
}
SUMMARY = re.compile(r"(?P<passed>\d+) passed")

def argv_for(seed):
    return ["python", "-m", "pytest", "-q", "-p", "no:cacheprovider",
            f"--randomly-seed={seed}", *SELECTION]

def execute(target, argv):
    if target is None:
        cmd, timeout = argv, 1800
    else:
        cmd = ["ssh", "-o", "BatchMode=yes", target, "bash", "-lc", shlex.join(argv)]
        timeout = 1800
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
    match = SUMMARY.search(proc.stdout)
    return {
        "returncode": proc.returncode,
        "passed": int(match.group("passed")) if match else 0,
        "tail": proc.stdout[-400:],
    }

rows = []
for env_name, target in TARGETS.items():
    if target is False:
        continue
    for attempt, seed in enumerate(SEEDS, start=1):
        rows.append({"env": env_name, "attempt": attempt, "seed": seed,
                     **execute(target, argv_for(seed))})

with open("parity_report.json", "w") as handle:
    json.dump(rows, handle, indent=2)

by_env = {}
for row in rows:
    by_env.setdefault(row["env"], set()).add(row["returncode"])

unstable = {env for env, codes in by_env.items() if len(codes) > 1}
if unstable:
    print(f"NONDETERMINISM in {sorted(unstable)}: freeze with expiry, or fix", file=sys.stderr)
    sys.exit(2)
if len({next(iter(codes)) for codes in by_env.values()}) > 1:
    print("ENVIRONMENT COUPLING: targets disagree", file=sys.stderr)
    sys.exit(3)
print("parity ok")

Exit codes matter here. 2 means unstable, 3 means coupled. A CI job can block on both, but a human triages them differently.

Read the four results as a taxonomy, not a pass rate

Within local	Within server	Local vs server	Label	Action
stable pass	stable pass	same	consistent green	proceed
stable	stable	differ	environment coupling	fix or narrow the patch; never freeze
unstable	any	-	nondeterminism	freeze with expiry and evidence, or delete
unstable	unstable	-	nondeterminism everywhere	fix the test; freezing hides a real bug

The distinction is the whole point. A test that disagrees with itself is nondeterministic. A test that agrees with itself in two places but not between them is coupled, and freezing it will hide a genuine portability bug until production finds it.

Freezing with an expiry, not forever

A freeze ledger needs three fields to be honest: why, who owns it, and when it dies.

{
  "tests/test_export.py::test_streaming_order": {
    "owner": "@finley",
    "expires": "2026-10-06",
    "evidence": ["parity_report.json"],
    "reason": "third-party stream reorders on retry; see issue tracker"
  }
}

# tests/test_freeze_ledger.py
import datetime as dt, json, pathlib, pytest

LEDGER = pathlib.Path("tests/flake_freeze.json")
ENTRIES = json.loads(LEDGER.read_text())

@pytest.mark.parametrize("nodeid,meta", ENTRIES.items())
def test_freeze_is_alive(nodeid, meta):
    expires = dt.date.fromisoformat(meta["expires"])
    assert dt.date.today() <= expires, f"freeze expired: {nodeid} ({expires})"

An expired freeze fails the suite on purpose. That converts "we will look at it later" into a build break with a name attached. Entries enter the ledger only with the parity report attached, and only from the third row of the table above.

Where MonkeyCode's free model access and free server fit

The parity gate needs a second target, and for small teams the friction is rarely the test runner. It is having somewhere else to run it. MonkeyCode's free server option supplies that second target without provisioning and maintaining your own box, and the free model access is what makes the patch-generation plus first-pass triage loop cheap enough to run a narrow selection repeatedly instead of once. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it inside the same loop, not instead of it:

1. Narrow the selection from the diff surface: touched files map to their test paths.
2. Run the gate locally with the local target only, both attempts, until it is stable.
3. Point the second target at the free server. If it exposes a shell or an SSH target, set PARITY_SSH_TARGET and rerun. If it only runs commands through a UI, paste the same pytest command there and write the output into the same JSON schema so one report covers both targets.
4. Attach parity_report.json to the pull request as the evidence artifact.

Limitations and who should not use this

• Secrets. Do not ship credentials, tokens, or private fixtures to a shared or free environment. Replay only what is safe to run elsewhere.
• Cost is real. Two targets times two attempts is four runs. Keep the selection under a few minutes or the gate becomes the bottleneck.
• Not production parity. Two environments prove the claim is portable, not that it is correct in production. Architecture, kernel, and service-version differences remain.
• Hardware-bound tests. GPU work, licensed software, and tests hitting internal services cannot be replayed this way. Tag and exclude them explicitly rather than letting the gate skip them silently.
• Single-environment teams. If a second target is genuinely unavailable, stages 1 and 2 still help, but do not label the result "parity".

The code above is a reference implementation, not a benchmarked result; read the parsing and timeouts against your own runner before trusting the exit codes in CI. The gate itself is deliberately boring: pin the environment once, lock the fixtures, run twice in two places, and let exit code 3 stop the merge. If you want to try the two-target split without buying hardware, point the gate at MonkeyCode's free server and keep the selection small.