Pin Observed Behavior Before an Agent Patch Moves It

Agent patches should not merge on a green suite. Green only means the patch agreed with the tests that ran. The merge question is tighter: did observed outputs stay inside a frozen characterization envelope, did error paths gain properties the agent did not author, and did every flake enter quarantine with counts rather than a calendar date?

A passing CI job is a weak token. Agent-authored tests often restate the patch. Characterization records what the tree already does. Error-path properties record what it must refuse. Flake counts record whether the signal is stable enough to trust.

The failure mode this gate targets

Agent patches tend to preserve the happy path and rewrite the edges. Timeouts become retries. Invalid JSON becomes a default object. A missing file becomes an empty list. Each of those moves can keep example tests green while shifting production behavior.

Fixtures drift in the same window. A test that reads the clock, the network, or an unordered set will fail on one runner and pass on another. Treat that noise as a product regression and reviewers start ignoring the gate. Ignore the noise and the agent learns that flaky red is optional.

The workflow below is a proposal. It is not a field report. Commands and modules are examples you can run locally after you point them at your own binary.

Layer 1: freeze a characterization envelope

Characterization tests record current outputs for a pinned input set. They are not a specification of intent. They are a digest of observed behavior. An agent patch that changes an envelope entry must declare that entry id in the pull request. Undeclared drift is a reject.

Follow the steps in order.

1. Choose a directory of frozen inputs that do not require the network.
2. Run the current main binary against those inputs with a pinned clock, locale, and RNG seed.
3. Store stdout, stderr, exit code, and a canonical digest per case.
4. Commit the envelope as a reviewable artifact, not as a generated-only cache.

export TZ=UTC
export LANG=C
export LC_ALL=C
export PYTHONHASHSEED=0
python tools/characterize.py --bin ./app --cases tests/envelope/in --out tests/envelope/observed.json
git add tests/envelope/observed.json tests/envelope/in

The envelope file should be boring. Reviewers can diff it. Agents can rewrite it. The gate only accepts a rewrite when the PR lists the case ids that moved.

Canonical hashing is the whole pin. Sort object keys. Strip trailing whitespace. Reject raw json.dump output that depends on dict insertion order. A one-byte churn in whitespace should not look like a behavior change, and a real behavior change should not hide behind pretty-print settings.

# tools/characterize.py — proposal
from __future__ import annotations

import argparse, hashlib, json, os, subprocess, sys
from pathlib import Path

def digest_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def canonical_json(obj: object) -> bytes:
    return json.dumps(obj, sort_keys=True, separators=(',', ':')).encode('utf-8')

def run_case(binary: Path, source: Path, timeout: float) -> dict:
    completed = subprocess.run(
        [str(binary), str(source)],
        capture_output=True,
        timeout=timeout,
        env={**os.environ, 'TZ': 'UTC', 'LANG': 'C', 'PYTHONHASHSEED': '0'},
    )
    payload = {
        'id': source.stem,
        'exit_code': completed.returncode,
        'stdout_sha256': digest_bytes(completed.stdout),
        'stderr_sha256': digest_bytes(completed.stderr),
    }
    payload['digest'] = digest_bytes(canonical_json(payload))
    return payload

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument('--bin', type=Path, required=True)
    parser.add_argument('--cases', type=Path, required=True)
    parser.add_argument('--out', type=Path, required=True)
    parser.add_argument('--timeout', type=float, default=5.0)
    args = parser.parse_args()
    cases = sorted(p for p in args.cases.iterdir() if p.is_file())
    envelope = {
        'version': 1,
        'pythonhashseed': os.environ.get('PYTHONHASHSEED', ''),
        'cases': [run_case(args.bin, path, args.timeout) for path in cases],
    }
    args.out.write_bytes(canonical_json(envelope) + b'\n')
    return 0

if __name__ == '__main__':
    sys.exit(main())

Pin the interpreter hash seed before the binary starts. Otherwise two runs of the same unordered set can produce two envelopes and the gate becomes a source of flakes instead of a detector.

Layer 2: add error-path properties the agent did not write

Happy-path examples are cheap for an agent to satisfy. Error-path properties are not, if a human owns the oracle. The human writes predicates over refusals: bad UTF-8 is rejected, negative limits raise, duplicate keys do not silently collapse.

Keep those predicates in a tree the agent cannot edit in the same commit as production code. That split is the point. The patch may add helpers. It may not weaken the refusal.

Do not gold-file the error paragraph. Agents rewrite copy. Check that the process fails closed and that stdout stays empty when the input is invalid.

# tools/error_paths.py — proposal
from __future__ import annotations

import json
from pathlib import Path

class EnvelopeDrift(AssertionError):
    pass

def load_envelope(path: Path) -> dict:
    data = json.loads(path.read_text())
    if data.get('version') != 1:
        raise ValueError('unsupported envelope version')
    return data

def assert_undeclared_drift(old: dict, new: dict, declared: set[str]) -> None:
    old_cases = {c['id']: c['digest'] for c in old['cases']}
    new_cases = {c['id']: c['digest'] for c in new['cases']}
    moved = [
        case_id
        for case_id, digest in new_cases.items()
        if case_id in old_cases and old_cases[case_id] != digest and case_id not in declared
    ]
    missing = sorted(set(old_cases) - set(new_cases))
    if moved or missing:
        raise EnvelopeDrift(f'undeclared drift={moved} missing={missing}')

def refuse_invalid_utf8(run) -> None:
    result = run(b'\xff\xfe not text')
    assert result.exit_code != 0
    assert result.stdout == b''
    err = result.stderr.lower()
    assert b'decode' in err or b'utf' in err

def refuse_negative_limit(run) -> None:
    result = run(b'{"limit": -1}')
    assert result.exit_code != 0
    assert result.stdout == b''

def refuse_duplicate_keys(run) -> None:
    result = run(b'{"k": 1, "k": 2}')
    assert result.exit_code != 0
    assert result.stdout == b''

Wire run to the same pinned environment as characterize.py. If error-path tests use a different locale or an unpinned clock, a refusal can look like a flake, and the ledger will quarantine a real contract.

Layer 3: quarantine flakes with counts, not calendars

Calendar freezes teach the wrong lesson. A test that is red on 1 of 30 runs is not finished on day 14. It is still a biased coin. Record (pass, fail) pairs per test id and per fixture digest. Promote a test out of quarantine only after a minimum of independent runs against the same digest.

A simple ledger rule, labeled as a proposal:

• Fewer than 20 runs: not blocking. Evidence is too thin to be a merge token.
• fail / (pass + fail) > 0 with at least 20 runs: stay quarantined. Do not block merge on that id.
• fail == 0 with at least 30 runs on the same fixture digest: release into the blocking set.
• Any quarantined test the agent deletes: reject the patch. Deletion is not a fix.

# tools/flake_ledger.py — proposal
from __future__ import annotations

from dataclasses import dataclass

@dataclass(frozen=True)
class Record:
    test_id: str
    fixture_digest: str
    passes: int
    fails: int

def total(record: Record) -> int:
    return record.passes + record.fails

def blocking(record: Record) -> bool:
    n = total(record)
    if n < 30:
        return False
    return record.fails == 0

def agent_may_delete(test_id: str, quarantined: set[str]) -> bool:
    return test_id not in quarantined

blocking is conservative on purpose. Sparse data does not become a green vote. If the fixture digest changes, reset the counters. A new fixture is a new experiment.

Compute the fixture digest from the envelope inputs plus the error-path module, not from the agent patch. Otherwise the patch can reset its own flake history by touching a comment.

python - <<'PY'
from pathlib import Path
import hashlib, json
parts = []
for path in sorted(Path('tests/envelope/in').iterdir()):
    parts.append(path.read_bytes())
parts.append(Path('tools/error_paths.py').read_bytes())
print(hashlib.sha256(b''.join(parts)).hexdigest())
PY

Decision table for the merge token

Signal	Merge vote	Notes
Envelope digest unchanged	allow	Behavior pin held
Envelope digest changed, case ids declared	review	Human reads the delta
Envelope digest changed, undeclared	reject	Silent behavior move
Error-path property failed	reject	Refusal contract broken
Agent edited error-path module and production code together	reject	Split the commit
Flake ledger still has fails on a blocking test	reject	Signal is not stable
Agent deleted a quarantined test	reject	Deletion is not a fix
New tests only, envelope and error paths hold	allow	Suite grew without moving pins

Run the table as a command. Do not keep it as a chat checklist.

python tools/merge_token.py \
  --old tests/envelope/observed.json \
  --new /tmp/observed.json \
  --declared pr_declared_ids.txt \
  --ledger tests/flake_ledger.json \
  --error-paths tools/error_paths.py

pr_declared_ids.txt is a committed allow-list for that PR, one case id per line. A model-generated summary in the PR body is not a substitute. The gate reads the file. It does not parse prose.

Running the envelope off the laptop

Local runs contaminate the ledger. Laptop clocks, thermal throttling, and leftover VPN routes all create unique flakes. The characterization pass and the error-path pass need a hermetic worker: pinned packages, frozen time, no outbound network.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A practical split is to keep the envelope and the error-path module in git, then use MonkeyCode's free model access to propose extra characterization cases from a diff, and use the free server option to execute characterize.py plus error_paths.py away from the laptop. The models do not own the oracle. They only suggest case ids and input blobs. A reviewer accepts or discards each suggestion before the envelope is updated. The free server is relevant only as a remote place to run the same pinned commands. It does not replace the ledger, and it does not decide the merge.

If a hermetic runner already exists, keep it. The value is the pin, not the vendor.

Limitations

This gate does not prove functional completeness. An envelope can be thin. A property can miss a domain. A ledger can hide a rare race when the run count is small. None of those gaps is fixed by adding more agent-written examples.

It also handles intentional, large refactors poorly. A rename that changes every digest looks like mass drift. In that case, regenerate the envelope on an isolated branch, review it as its own change, and only then allow the agent to patch behavior.

Skip this workflow when the product is a UI with no stable textual output to pin, when tests must hit live third-party APIs, when the team cannot split oracle files from production commits, or when reviewers will not read envelope diffs. Those environments need a different token. This one assumes hermetic inputs and a human who will reject undeclared drift.

What to count next

Track three counters per week, not a model-quality dashboard: undeclared drift rejects, error-path rejects, and quarantined tests the agent tried to delete. If the first counter is zero, the envelope is either complete or ignored. If the third counter is high, deleting noise has become cheaper than fixing it. Both readings are actionable. Neither requires a published coding-model scoreboard.