An Incomplete Model Transcript Is Not a Passing Property Run

An incomplete model transcript is not a passing property run. If capture never stored a complete patch body, the suite has no oracle, and a zero exit from the runner is only evidence that a process ended. Score that trial incomplete. Do not promote it to green.

Agent diffs fail in ways a single snapshot misses. The patch can drop a test, weaken an assertion, or arrive truncated because the model stream died. Those are different defects.

This workflow gives each defect its own receipt: transcript, fixture lock, property verdict, and an optional flake freeze. The merge gate reads the verdict only after the first two receipts exist. Later columns do not repair earlier ones.

The code below is an unexecuted proposal. It is not a benchmark. It does not report a measured catch rate, latency, or token cost.

Three receipts, then a verdict

Keep the columns separate. A flake freeze answers a different question from a property miss, and a transport failure answers a different question from both.

Record	Written when	Fail-closed rule
Transcript receipt	Capture stores a non-empty diff and its SHA-256	Empty, short, non-diff, or missing body => incomplete
Fixture lock	Fixture files are hashed before properties run	Hash drift, or a zero file count => fixture_mismatch
Property verdict	Replay applies the frozen patch and runs named checks	Any miss => reject
Flake freeze	A known flake has an expiry, a reason, and a revision	Transport loss cannot open or extend it

review is the strongest automated outcome this ledger emits. It is not a merge approval. A human still reads the diff.

The ledger only stops a false green. It stops one when the model call, the fixtures, or the properties did not actually hold.

1. Capture once, then leave the model alone

Only the capture phase needs a model endpoint. Persist the raw body, the byte length, a content hash, and the request id if the server sent one. Compare the stored length with any length the server declared. Inequality means the stream ended early.

A body that is not a unified diff is also incomplete. Chatty prose, an empty object, and an error payload are not patches. Do not ask the property runner to interpret them.

Do not retry from inside the property runner. A second attempt, if the receipt policy allows one, is still capture. It gets a new id and a new file. The property phase never opens a socket to the model.

mkdir -p .agent-ledger/transcripts .agent-ledger/fixtures
python3 capture_patch.py \
  --prompt-file prompts/patch.md \
  --out .agent-ledger/transcripts/run-001.json
test -s .agent-ledger/transcripts/run-001.json

# Unexecuted proposal. Adapt the client; keep the receipt shape.
import hashlib, json, pathlib

def looks_like_diff(body: bytes) -> bool:
    text = body.lstrip()
    return text.startswith(b'diff --git ') or text.startswith(b'--- ')

def build_receipt(body: bytes, declared: int | None) -> dict:
    length_ok = declared is None or len(body) == declared
    complete = bool(body) and length_ok and looks_like_diff(body)
    return {
        'sha256': hashlib.sha256(body).hexdigest() if body else None,
        'bytes': len(body),
        'declared': declared,
        'complete': complete,
        'verdict': 'captured' if complete else 'incomplete',
    }

def write_receipt(path: str, body: bytes, declared: int | None) -> dict:
    rec = build_receipt(body, declared)
    pathlib.Path(path).write_text(json.dumps(rec, indent=2))
    if not rec['complete']:
        raise SystemExit(2)
    return rec

Short bodies are cheap to reject. They are expensive when a later step treats them as patches. Exit 2 here, before any fixture hash is trusted.

2. Lock fixtures before any property executes

Fixtures are inputs. They are not part of the agent diff. Hash every file under the fixture root after capture and before replay. If the patch also edits that directory, the lock breaks.

That break is fixture_mismatch, not a quiet new baseline. A lock file with count: 0 is not a lock. An empty directory proves nothing about the patch, so refuse it.

# Unexecuted proposal.
import hashlib, pathlib

def lock_fixtures(root: pathlib.Path) -> dict:
    rows = []
    for path in sorted(p for p in root.rglob('*') if p.is_file()):
        rows.append({
            'path': path.as_posix(),
            'sha256': hashlib.sha256(path.read_bytes()).hexdigest(),
        })
    if not rows:
        raise SystemExit('fixture root has no files')
    return {'count': len(rows), 'files': rows}

python3 lock_fixtures.py --root tests/fixtures \
  --out .agent-ledger/fixtures/run-001.lock.json
python3 lock_fixtures.py --root tests/fixtures \
  --compare .agent-ledger/fixtures/run-001.lock.json

Two comparisons matter: file count and digest equality. Either mismatch stops the trial. Do not update the lock from the agent patch and continue.

3. Replay properties against the frozen transcript

Apply the captured patch in a temporary worktree pinned to a known base revision. Run properties there. They read the receipt file. They do not call the model again.

That split is the control. A free endpoint can drop, throttle, or return a shorter body on a second call. If properties depend on that second call, a disconnect can erase a real miss or fabricate a pass.

Replay makes the disconnect a rerun of replay, not a new oracle. Capture stays closed unless its own receipt is missing.

# Unexecuted proposal. Each check returns a boolean.
def run_properties(tree: str) -> list[str]:
    checks = {
        'patch_applies': patch_applies(tree),
        'tests_not_deleted': no_test_files_removed(tree),
        'assertions_not_weakened': assertion_delta_non_negative(tree),
        'oracle_holds': domain_oracle(tree),
    }
    return [name for name, ok in checks.items() if not ok]

Use four named checks as the minimum. Do not add a row you cannot fail.

1. The patch applies on the pinned base revision.
2. No path under tests/ is deleted.
3. Assertion count in touched tests does not fall.
4. A domain oracle fails on a known bad fixture you committed earlier.

Write the fourth check yourself. Do not ask the same capture call to invent the oracle and the patch. A check that cannot fail is not a property. Leave it out of the table.

4. Classify in receipt order

Read transcript, then lock, then misses. Stop at the first hard failure. A green property log cannot override a missing receipt.

Transcript	Fixture lock	Properties	Verdict
incomplete	any	not run	incomplete
complete	mismatch or empty	not trusted	fixture_mismatch
complete	match	one or more misses	reject
complete	match	no misses	review

# Unexecuted proposal.
def classify(transcript: dict, lock_ok: bool, misses: list[str]) -> str:
    if not transcript.get('complete'):
        return 'incomplete'
    if not lock_ok:
        return 'fixture_mismatch'
    if misses:
        return 'reject'
    return 'review'

python3 classify_trial.py \
  --transcript .agent-ledger/transcripts/run-001.json \
  --lock .agent-ledger/fixtures/run-001.lock.json \
  --properties .agent-ledger/properties/run-001.json

Exit 0 only for review. Exit 2 for incomplete. Exit 3 for fixture_mismatch. Exit 4 for reject. Map every non-review status to a failed CI job.

Do not map incomplete to a skipped job. A skip is how a dead capture becomes a green badge on the pull request.

5. Keep the flake freeze on its own record

A flake freeze names a test id, an expiry, a reason, and the revision that observed the flake. It does not name the model endpoint. It is not a shared allowlist, and it is not copied into the agent diff.

# Unexecuted example. One observation, one expiry.
test_id: tests/test_queue.py::test_retry_window
expires: 2026-10-02
reason: nondeterministic timeout on the pinned runner image
revision: a1b2c3d
opens_on: observation_only

Four constraints keep this file from laundering other failures.

1. A property miss cannot create a row.
2. A transport error cannot extend expires.
3. The agent diff cannot carry the file.
4. After expiry, the test is back in the suite. No row means no freeze.

If capture failed, do not open a freeze to stabilize the red run. Mark the trial incomplete and capture again under a new id. The observation column stays untouched.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode, the open-source project named in this outreach, participates in two optional slots. It does not participate in the verdict rules. Capture can use its free model access, so one transcript can be recorded without a private endpoint. Replay can use its free server option, so the property commands have a runner that is separate from the model call.

Any client that writes a body to disk can fill the capture slot. Any runner that can apply a patch can fill the replay slot. The receipt order does not change when the client changes.

The outreach for this draft also describes a free token allowance on the order of 10 million tokens, together with that free server option. Treat the number as a campaign claim supplied for this article. It is not a measurement from the proposal above, and it is not a finding about permanence, model names, rate limits, or hardware.

Those details are outside this text. Read the current project documentation, then put the allowance you actually have into configuration. A hardcoded quota will lie the moment the terms change.

# Unexecuted. The classifier must not embed a quota.
if [ -z "$MODEL_TOKEN_BUDGET" ]; then
  echo "MODEL_TOKEN_BUDGET is required; set it from current project terms" >&2
  exit 1
fi
python3 capture_patch.py \
  --max-tokens "$MODEL_TOKEN_BUDGET" \
  --out .agent-ledger/transcripts/run-001.json

Spend model tokens in step 1 only. Steps 2 through 5 should record zero additional model tokens. If the property log shows a new completion id, replay is still coupled to the endpoint.

A disconnect can then rewrite the verdict. That coupling is the failure this ledger exists to prevent. Free access changes how capture is paid for. It does not relax a short body into a pass, and it does not turn a vanished runner into a skipped job.

Rerun replay when the runner dies after a complete receipt. Rerun capture only when the transcript receipt is missing or incomplete.

Limitations

The ledger blocks one family of false greens. It does not notice a wrong oracle. If domain_oracle encodes the defect, every complete transcript looks clean. Review the oracle as its own change, before you trust a week of review rows.

Hashing fixtures will not catch a malicious fixture you committed yourself, as long as the hash stays stable. Security-sensitive diffs still need review outside this table. The four checks are a floor, not a threat model.

The classifier has not been executed against a live endpoint here. No latency, cost, or catch-rate figure in this article should be read as data. There is none. Exit codes are a proposed contract for the scripts, not a result from a run.

A free server option is a place to run the replay commands. This article does not state an uptime target, a region, a machine size, or a retention period for it. If your policy requires those in writing, do not treat convenience access as a substitute.

Who should skip the approach

Skip it when you have no pinned base revision. patch_applies is noise on a moving tree. Pin the revision, then capture.

Skip it when the only checks are snapshots produced by the same model call that wrote the patch. You would be grading a document against itself. The oracle has to come from somewhere else.

Skip it when the suite must run inside a contracted environment with a stated availability target. Convenience access is not that contract. This draft does not claim an SLA for the free server option, and a missing SLA should stay missing rather than be inferred.

Teams that already archive model transcripts can keep their store. They still need the column split. The product name on the capture client is incidental. The order of receipts is the part worth copying.

Start with one property and one fixture directory if you try the split. Confirm the current free-model allowance and free-server terms in the MonkeyCode docs, export MODEL_TOKEN_BUDGET from that page, and fail the job on incomplete before you add checks.