Postmortem: Two Free Workers Approved One Patch, and the Quorum Believed Them

A merge gate counted two green verdicts as two independent checks. They came from the same code path. Only one was real.

Conclusion first:

• The lease key was built from wall-clock seconds, not from work identity.
• A one-second clock drift produced two valid leases for one commit.
• The quorum counted verdict rows, so it counted the same evidence twice.
• The durable fix is a fencing token plus a stricter definition of independence.

This is a lab-reconstructed postmortem of that class of failure. The code below is runnable and the reproduction steps are deterministic.

Impact

The gate merged four patches in 38 minutes with single-source verification. One required a revert. No data was lost. The real cost was trust: the green badge stopped meaning anything.

Timeline (UTC)

• 09:11:58 — Merge queue enqueues PR 4417 at commit a91f3c2.
• 09:12:00 — Worker A computes lease key eval:4417:1760776320.
• 09:12:01 — Worker B computes eval:4417:1760776321. One second of drift.
• 09:12:01 — Both workers acquire a lease. Neither sees a conflict.
• 09:18:44 — Worker A posts a green verdict using runner image r-2026.09.
• 09:19:02 — Worker B posts a green verdict using cached runner image r-2026.07.
• 09:19:06 — Quorum counts 2 rows, merges the patch.
• 11:40:11 — Nightly suite fails on the merged behavior change.
• 12:05:30 — Revert lands. Gate frozen for 90 minutes.

What actually broke

The bug is three lines long. It is boring, which is why it survived review.

# BROKEN: key derived from wall clock, not from work identity
def broken_key(pr_id: int) -> str:
    return f"eval:{pr_id}:{int(time.time())}"

Two workers never collide on the key. They collide on the work. The lease protected nothing, because the lease never described the task.

The second bug hid behind the first

The quorum query counted rows:

-- BROKEN: counts evidence rows, not independent evidence
SELECT COUNT(*) FROM verdicts
WHERE pr_id = 4417 AND verdict = 'green';

Two rows. Two workers. Zero independence. Worker B ran a stale runner image, so its coverage overlapped Worker A's only partially.

Contributing factors

1. Clock-derived identity. Free-tier hosts restart often and drift. Wall-clock keys amplify that.
2. No fencing token. Worker B could write a verdict after Worker A's lease expired.
3. Quorum defined by count. The gate asked "how many greens" instead of "how many distinct evidences".
4. Retries were expected. High worker churn made duplicate execution normal, so nobody flagged it.
5. Monitoring watched failures only. Duplicate greens emitted no alert.

The durable fix

1. Key the lease to work identity

import hashlib

def eval_key(repo: str, commit: str, config_hash: str) -> str:
    material = f"{repo}:{commit}:{config_hash}"
    return hashlib.sha256(material.encode()).hexdigest()[:16]

Same commit, same config, same key. Always. A retry contends with the original lease instead of forking a new one.

2. Add a monotonic fencing token

import sqlite3, time

def acquire(conn: sqlite3.Connection, key: str, worker: str, ttl_s: int = 900):
    now = time.time()
    cur = conn.execute(
        "UPDATE leases SET worker = ?, token = token + 1, expires_at = ? "
        "WHERE key = ? AND expires_at < ?",
        (worker, now + ttl_s, key, now),
    )
    if cur.rowcount == 1:
        return conn.execute("SELECT token FROM leases WHERE key = ?", (key,)).fetchone()[0]
    try:
        conn.execute(
            "INSERT INTO leases(key, worker, token, expires_at) VALUES(?, ?, 1, ?)",
            (key, worker, now + ttl_s),
        )
        return 1
    except sqlite3.IntegrityError:
        return None

On Postgres, the same intent is one statement: INSERT ... ON CONFLICT (key) DO UPDATE SET token = leases.token + 1, expires_at = EXCLUDED.expires_at WHERE leases.expires_at < now() RETURNING token.

3. Reject stale writers

def record_verdict(conn, key, token, worker, verdict, seed, runner_digest):
    row = conn.execute("SELECT token, expires_at FROM leases WHERE key = ?", (key,)).fetchone()
    if row is None or row[0] != token:
        raise RuntimeError("stale fence: verdict rejected")
    conn.execute(
        "INSERT INTO verdicts(key, token, worker, verdict, seed, runner_digest, created_at) "
        "VALUES(?, ?, ?, ?, ?, ?, ?)",
        (key, token, worker, verdict, seed, runner_digest, time.time()),
    )

A worker that lost its lease can no longer contribute a green.

4. Redefine quorum as evidence diversity

SELECT COUNT(DISTINCT seed || '|' || runner_digest) AS independent_greens
FROM verdicts
WHERE key = :key
  AND verdict = 'green'
  AND token = (SELECT MAX(token) FROM leases WHERE key = :key);

The gate now demands two greens from different seeds and different runner digests. Two workers on identical inputs count as one.

Reproduction test plan

1. Spin two worker processes against one SQLite file.
2. Inject a two-second offset into worker B's clock (libfaketime or an injected clock() callable).
3. Revert to the broken key function. Observe two leases, two greens, one merged patch.
4. Apply the work-identity key. Observe one lease, one green, and a blocked quorum.
5. Assert the stale-fence path raises on every second run.
6. Add a CI check: any key with more than one active lease fails the build.

Step 6 is the cheap regression guard. It catches the next version of this bug before merge.

Where free-tier workers fit

Cheap second-opinion evals are exactly what free capacity is good for. The team ran its cross-seed check on free-tier workers using MonkeyCode's free model access and its free server option, both supplied by the project operator. The project advertises a free token allotment; confirm the current limits on the project page before sizing a pipeline around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free capacity raises concurrency. Concurrency raises duplicate-work risk. Free tiers make fencing mandatory, not optional.

When not to use this approach

• Single-worker pipelines. Fencing adds schema for a race that cannot occur.
• No durable store. In-memory leases vanish on restart and certify nothing.
• Hard latency SLOs. Shared free capacity is not a latency guarantee.
• Judgment-heavy review. Seed diversity does not make a subjective check objective.

Situation	Free-tier workers	Paid/dedicated workers
Cross-seed regression runs	Good fit	Overkill
Duplicate-sensitive merge gates	Needs fencing first	Needs fencing first
Latency-bound release checks	Poor fit	Good fit
Exploratory prompt evals	Good fit	Wasteful

Rollout checklist

1. Replace clock-derived keys with work-identity hashes.
2. Add the token column and backfill leases with token = 1.
3. Reject verdicts whose token is stale.
4. Change quorum to count DISTINCT seed, runner_digest.
5. Alert on more than one active lease per key.
6. Replay 30 days of merges against the new query. Expect fewer greens.

Fewer greens is the correct outcome. The old number was inflated by a clock.

If a gate ever reports two independent approvals, ask what made them independent. If the answer is "two rows", the gate is counting, not verifying.