A Free-Server Pass Cannot Close a Local Fixture Fail

A free-server pass cannot close a local fixture fail. If those two runs disagree, the disagreement is a rejected pair, not a flake. A freeze, when one is allowed at all, stays inside the server class that actually failed, and it dies at a time written before the run.

This is a review sketch for agent-patch gates. The classifier has not been executed against a production suite. It reports no flake rate, no quota, and no hardware result.

The mix-up the gate exists to stop

An agent patch is a diff plus a claim that existing properties still hold. The cheap mistake is to run that diff in two places and let the greener place win. One place is a pinned local fixture. The other is a free server you did not size, warm, or keep.

Those runs can share a fixture hash and still not share a meaning. A local assertion failure is evidence about the patch on a fixture you can replay. A free-server pass is evidence about that runner, on that attempt, under whatever limit it happened to have. Using the second fact to erase the first fact stores a real miss under a flake label.

Resource exits make the mix-up worse. A killed process, a timeout, or an empty response body is not an assertion result. It must not become a freeze candidate, and it must not be retried by editing the sample card until the color changes.

Decision table

Apply the row that matches the stamps. Do not add a row in which any green closes the gate.

Local outcome	Free-server outcome	Fixture hash	Action
pass	pass	equal	accept as a candidate
assert_fail	pass	equal	reject; do not freeze
pass	assert_fail	equal	freeze only inside free-server if the test is allowlisted and unexpired
assert_fail	assert_fail	equal	reject; no freeze; fail is stable in both classes
any	resource_exit or empty_body	equal	discard the run; new run id; same sample card
any	any	different	reject; fixture identity broke

Row two is the one teams want to soften. Leave it hard. A later green on the free server still does not edit that row.

What a stamp records

The stamp is a JSON object written by the runner, not by the model that drafted the patch. Six fields are enough.

1. fixture_hash is the hash of fixture bytes plus the sample card. The patch is not part of the hash.
2. server_class is local-pinned or free-server. A third class waits until you can say what it does not guarantee.
3. sample_card fixes the property name, the case count, and the seed before either run starts.
4. outcome is pass, assert_fail, resource_exit, or empty_body.
5. run_id is unique per attempt. A retry writes a new id. It does not overwrite the old stamp.
6. freeze_expires_at is copied from the card for allowlisted tests. A green run must not extend it.

If a red result tempts you to raise the case count, write a new card and a new hash. That is a different experiment. It is not a completion of the failed one.

Example card, with an illustrative expiry integer rather than a product deadline:

{
  "property": "parser_round_trip",
  "cases": 32,
  "seed": 7,
  "freeze_expires_at": 1791590400
}

Numbered workflow

Keep the order. Skipping a step fails the gate.

1. Write the sample card and hash the fixture directory. Store both beside the patch under review.
2. Draft the candidate patch with whatever model access you already use. The draft is not allowed to rewrite the card after a red result.
3. Run local-pinned first. Map the process status into one outcome. Write local.json with a fresh run_id.
4. Run the same fixture hash on the free server. Write remote.json. If that runner is unavailable, stop. Do not clone the local stamp and relabel the class.
5. Classify the pair. Open a freeze file only when the function below returns freeze_inside_server_class.
6. Archive both stamps next to the junit file. A missing stamp means that class did not run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access fits step 2 as a way to draft a patch, and the free server option fits step 4 as the free-server class. That is plumbing for the two steps. It is not an oracle, and it does not relax the table.

Classifier you can run locally

The module is a proposal you can execute on your machine. It does not call a network. It has not been benchmarked.

import hashlib, json, unittest
from pathlib import Path

CLASSES = {"local-pinned", "free-server"}
NON_ASSERT = {"resource_exit", "empty_body"}

def fixture_hash(paths, card):
    h = hashlib.sha256()
    for p in sorted(map(Path, paths)):
        h.update(p.name.encode())
        h.update(p.read_bytes())
    h.update(json.dumps(card, sort_keys=True).encode())
    return h.hexdigest()

def outcome_from_exit(code, body_len, assertion_failed):
    # Convention for this gate, not a universal process standard.
    if code == 0 and body_len == 0:
        return "empty_body"
    if code in (124, 137):
        return "resource_exit"
    if assertion_failed:
        return "assert_fail"
    if code == 0:
        return "pass"
    return "resource_exit"

def classify(local, remote, allowlist, now):
    if local["fixture_hash"] != remote["fixture_hash"]:
        return "reject_fixture_mismatch"
    if local["server_class"] not in CLASSES or remote["server_class"] not in CLASSES:
        return "reject_bad_class"
    if local["server_class"] == remote["server_class"]:
        return "reject_unstamped_pair"
    if local["outcome"] in NON_ASSERT or remote["outcome"] in NON_ASSERT:
        return "reject_run_retry_new_id"
    if local["outcome"] == "assert_fail" and remote["outcome"] == "pass":
        return "reject_cross_class_pass"
    if local["outcome"] == "pass" and remote["outcome"] == "assert_fail":
        return _freeze(remote, allowlist, now)
    if local["outcome"] == "pass" and remote["outcome"] == "pass":
        return "accept_candidate"
    if local["outcome"] == "assert_fail" and remote["outcome"] == "assert_fail":
        return "reject_stable_fail"
    return "reject_unclassified"

def _freeze(run, allowlist, now):
    if run["test_id"] not in allowlist:
        return "reject_not_allowlisted"
    if now >= run.get("freeze_expires_at", 0):
        return "reject_freeze_expired"
    return "freeze_inside_server_class"

def row(**kw):
    base = {
        "fixture_hash": "abc",
        "test_id": "t1",
        "freeze_expires_at": 100,
        "outcome": "pass",
        "server_class": "local-pinned",
    }
    base.update(kw)
    return base

class GateTests(unittest.TestCase):
    def test_remote_pass_cannot_close_local_fail(self):
        got = classify(
            row(outcome="assert_fail", server_class="local-pinned"),
            row(outcome="pass", server_class="free-server"),
            {"t1"}, 10,
        )
        self.assertEqual(got, "reject_cross_class_pass")

    def test_resource_exit_is_not_a_freeze(self):
        got = classify(
            row(outcome="pass", server_class="local-pinned"),
            row(outcome="resource_exit", server_class="free-server"),
            {"t1"}, 10,
        )
        self.assertEqual(got, "reject_run_retry_new_id")

    def test_freeze_names_only_the_failing_class(self):
        got = classify(
            row(outcome="pass", server_class="local-pinned"),
            row(outcome="assert_fail", server_class="free-server"),
            {"t1"}, 10,
        )
        self.assertEqual(got, "freeze_inside_server_class")

    def test_empty_body_is_not_a_pass(self):
        self.assertEqual(outcome_from_exit(0, 0, False), "empty_body")

if __name__ == "__main__":
    unittest.main()

Save the file and run the four checks:

python class_stamp_gate.py -v

Four successes mean the table's dangerous confusions are rejected by this sketch. They do not mean your suite is clean.

Wire outcome_from_exit to your runner's real codes before you trust a stamp from either class. If your timeout command does not use exit 124, change the set. A wrong code turns a capacity problem into assert_fail, and that is how a freeze opens for the wrong reason.

Hashing is separate from classification. Point fixture_hash at the fixture directory and the card, then confirm both stamps carry the same digest before you call classify.

python -c "import class_stamp_gate as g, json; print(g.fixture_hash(['tests/fixtures/case.json'], json.load(open('sample_card.json'))))"

That command is an interface check. It assumes you already have those two paths. It is not a claim that a particular repository layout is required.

Limits of the two-class split

Two stamps cannot describe a flake that appears only on a third kind of machine. Do not cite this sketch as a flake rate. It never counts occurrences across a week, and it should not be extended with a fabricated percentage.

The allowlist is a human list. A wide list freezes too many tests, though only inside the class that failed. Clock skew can make freeze_expires_at lie if now comes from a different source than the card. Pass one clock into both.

Properties that call live network services are out of scope. A fixture hash does not pin a moving dependency. If the property needs the network, this gate will look decisive while the oracle moves underneath it.

Same-class pairs are rejected on purpose. Two local files, or two free-server files, are not a cross-class check. Relabeling a copy is the failure mode reject_unstamped_pair exists to catch.

Who should not use it

Skip this workflow if you have only one server class. The table needs two stamps, and a duplicated file will fail closed. Skip it if fixtures are still being edited while the run is in flight. The hash will move for reasons that have nothing to do with the patch.

Skip it if you need an audited compliance control. Nothing here is a certificate. Skip it if the free server is supposed to be the only oracle. That choice deletes row two, which is the row this gate is for.

Also skip it when the patch under review changed the fixture or the sample card. Re-hash, then start at step 3. Do not classify stamps that belong to the previous card.

Close

Keep the sample card fixed, stamp the server class, and let a freeze name only the class that failed. A free-server pass can support a candidate when the local pinned run passed as well. It cannot close a local fail, and a resource exit cannot stand in for either color.

If a local fixture hash is already in place, the free model access and free server option noted above can fill steps 2 and 4 on a draft patch. Step 3 still runs first, on the pinned class, with the card left untouched.