Postmortem: A Retry Certified the Previous Volume's Lockfile

A retry must never inherit the previous job volume. A warm disk can certify a stale lockfile. The durable fix is an ephemeral workspace contract.

This note reconstructs a lab incident from a fixture. It is not a claimed customer outage report. The figures below come from the lab fixture only.

What the gate got wrong

Shared disk

The patch proposal and the certification run shared one disk. The first attempt timed out while writing a lockfile. The retry skipped installation because a virtualenv already existed.

Tests imported that older tree and then passed cleanly. The pull request still carried the half-written lockfile.

Wrong proof

That green result certified the wrong source tree. A passing suite did not prove the lockfile. The merge gate trusted process exit codes alone.

Where the free options entered

Availability, not a spec

The proposal step used MonkeyCode free model access to draft a dependency bump. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The certification step used the free server option as the runner.

Both claims are operator-supplied availability statements, not measured results. They are not a quota, a hardware spec, or a permanence promise.

The free server option matters because a shared runner invites reuse. A developer may treat that runner like a persistent dev box. That habit is unsafe for a certification job.

A proposal model can time out without cleaning the disk. The next attempt then inherits leftover files from that disk.

No model name is stated in this note. No token quota is stated in this note. No server size is stated in this note.

Those details were not supplied as primary sources. Product availability can change without notice from the operator.

Timeline from the fixture

The clock below is a fixture input, not a measurement. It is not a documented product timeout or limit. It is not a service-level measurement from production.

1. Attempt one started on volume generation g-1844 in the fixture.
2. The model draft updated requirements.txt and began rewriting uv.lock.
3. The runner hit a timeout at 12 minutes in the fixture clock.
4. The timeout handler marked the job failed and left the volume mounted.
5. Attempt two reused generation g-1844 under a new attempt id.
6. The install step saw .venv and skipped the sync command.
7. Pytest collected tests from the old site-packages tree.
8. The gate stored a green status next to the new lockfile blob.

Why the stale tree still passed

Python import order prefers the existing virtualenv on sys.path. A skipped sync leaves that virtualenv untouched by design. Tests then exercise the packages from the earlier install.

The lockfile on disk is only text until an installer runs. A partial lockfile can still look present to a shell test. The fixture stopped the write after 40 kilobytes of output.

The file existed, so later steps treated it as complete. Existence is not a checksum of the intended tree. A size check would also miss a wrong but complete file.

The fixture lockfile hash changed between the two attempts. The installed distribution set did not change at all.

Contributing factors

• The retry key used the job id and ignored volume generation.
• The install script treated a present .venv as a valid cache.
• No preflight compared lockfile hashes with installed distributions.
• The timeout path did not write a dirty volume marker.
• The gate scored the test process, not the tree identity.
• Operators assumed a free server would reset itself between attempts.

The durable contract

Certification receives a new volume generation on every attempt. Proposal output is copied in as a hashed tarball. Install always runs from that tarball into an empty tree.

A dirty marker blocks the certification run immediately. A missing generation id also blocks the run. The proposal runner may keep a warm cache for drafting.

That cache must never become the certification root. Free model access can draft the dependency bump. The free server option can host the draft role or the cert role.

The two roles still need different volume lifetimes. Mixing them on one disk repeats this incident.

Lab fixture script

The script below is an unexecuted lab example. It has not been run against a live product API. Adapt the local paths before any real use.

It fails closed when identity inputs are missing. A missing lockfile is also a hard failure. Silence from the script is not treated as a pass.

#!/usr/bin/env python3
"""Unexecuted lab fixture: refuse a reused certification volume."""

from __future__ import annotations

import hashlib
import json
import os
import sys
from pathlib import Path

ROOT = Path(os.environ.get("WORKSPACE", ".")).resolve()
STATE = ROOT / ".workspace_contract.json"
DIRTY = ROOT / ".volume_dirty"
LOCKS = ("uv.lock", "poetry.lock", "package-lock.json", "requirements.txt")


def sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def fail(code: int, message: str) -> int:
    print(message, file=sys.stderr)
    return code


def main() -> int:
    generation = os.environ.get("VOLUME_GENERATION", "")
    attempt = os.environ.get("JOB_ATTEMPT", "")
    if not generation or not attempt:
        return fail(2, "missing VOLUME_GENERATION or JOB_ATTEMPT")
    if DIRTY.exists():
        return fail(3, f"dirty volume marker: {DIRTY}")
    locks = [ROOT / name for name in LOCKS if (ROOT / name).is_file()]
    if not locks:
        return fail(4, "no lockfile found; refusing an unbound tree")
    current = {
        "generation": generation,
        "attempt": attempt,
        "locks": {str(p.relative_to(ROOT)): sha256(p) for p in locks},
    }
    if STATE.is_file():
        previous = json.loads(STATE.read_text(encoding="utf-8"))
        same_gen = previous.get("generation") == generation
        new_attempt = previous.get("attempt") != attempt
        if same_gen and new_attempt:
            return fail(5, "same generation reused across attempts")
        if same_gen and previous.get("locks") != current["locks"]:
            return fail(6, "lockfile changed inside one generation")
    STATE.write_text(json.dumps(current, indent=2) + "\n", encoding="utf-8")
    print(json.dumps({"status": "clean", "generation": generation}))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Exit codes

• Exit 2 means the runner forgot generation or attempt identity.
• Exit 3 means a prior attempt left a dirty marker.
• Exit 4 means no lockfile can bind the tree.
• Exit 5 means a retry reused the same generation.
• Exit 6 means the lockfile changed inside one generation.
• Exit 0 means this attempt may proceed to install.

Reset the volume before cert

This shell fragment is also an unexecuted example. It refuses to run without an explicit workspace path. It refuses a workspace path shorter than eight characters.

That guard reduces the chance of a bad delete. A relative path must still be resolved before the find command.

#!/bin/sh
set -eu
test -n "${WORKSPACE:-}" || exit 2
test "${#WORKSPACE}" -ge 8 || exit 2
test -n "${VOLUME_GENERATION:-}" || exit 2
find "$WORKSPACE" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
install -d "$WORKSPACE"
printf '%s\n' "$VOLUME_GENERATION" > "$WORKSPACE/.generation"
: > "$WORKSPACE/.volume_clean"

Bind installed distributions to the lock hash

A clean volume is not enough by itself. The installer must record what it actually unpacked. The following snippet is pseudocode for that record.

It is not a complete package manager replacement.

# Unexecuted pseudocode. Run only after a forced reinstall.
import hashlib
import json
from importlib.metadata import distributions
from pathlib import Path

def lock_digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

rows = sorted(
    f"{dist.metadata['Name']}=={dist.version}"
    for dist in distributions()
)
record = {
    "lock": lock_digest(Path("uv.lock")),
    "installed": rows,
}
Path(".install_record.json").write_text(json.dumps(record, indent=2) + "\n")

Compare that record on the next command in the job. Fail if the lock digest moved after install. Fail if the installed name set moved too.

Do not accept a record from another generation. Store the generation id beside the digest in the record.

Test plan for the contract

Run this plan on a throwaway repository first. Do not point it at a developer home directory. Mark every step unexecuted until a human runs it.

1. Create an empty temp directory and export it as WORKSPACE.
2. Export VOLUME_GENERATION as g-test-1 and JOB_ATTEMPT as 1.
3. Add a one-line requirements.txt so a lock target exists.
4. Run the contract script and expect exit 0.
5. Change JOB_ATTEMPT to 2 without changing the generation.
6. Run the contract script again and expect exit 5.
7. Touch .volume_dirty and expect exit 3 on the next run.
8. Delete the lockfile and expect exit 4 from the script.
9. Unset VOLUME_GENERATION and expect exit 2 from the script.
10. Repeat the pass case only after a fresh directory wipe.

Decision table

Signal	Action	Reason
Missing generation id	Fail closed	The volume identity is unknown
Dirty marker present	Fail closed	An aborted write may remain
Same generation, new attempt	Fail closed	The retry inherited residue
Lock hash drift in one generation	Fail closed	A partial write is likely
No lockfile	Fail closed	The tree cannot be bound
New generation and stable lock	Continue	Install into the empty tree

Read the table as a gate, not as a suggestion. Any unknown signal joins the fail-closed row by default. A free server reboot is not proof of a new generation.

Only the generation id proves that freshness claim. Log that id beside the test result artifact.

What this does not fix

The contract does not judge the quality of the patch. A correct tree can still contain a bad dependency bump. Human review remains mandatory before a merge to main.

The script does not scan the workspace for secrets. Do not place credentials on a shared free server. The script does not pin a model version.

A later draft can differ even when the disk is clean. The fixture does not measure token use at all. It does not measure runner hardware or region.

It does not claim that free access will remain available. Operators should re-check current product terms before relying on them.

Who should skip this approach

Skip this approach when the build must run on attested hardware. Skip it when the repository has no lockfile and cannot add one. Skip it when the workspace holds production secrets.

Skip it when the team needs a byte-stable model snapshot. A warm personal dev box is a poor certification runner. Use a dedicated clean generation for certification instead.

Replay note

Readers can replay the fixture on a throwaway repository. The operator states that free model access is available. The operator also states that a free server option is available.

Use either only for scratch work after checking live terms. Confirm those live terms before starting a replay. Keep the certification volume ephemeral even when drafting stays warm.