Make Pass Rate a Quorumed Histogram Before Late Workers Flip the Gate

Last Tuesday I traced a canary that flipped after a recycled worker delivered one late score. The dashboard showed a 0.81 pass rate, then 0.74, then 0.81 again, with no model change. Have you ever shipped a planner because a dashboard mean looked stable while stragglers were still rewriting bins? I had already marked the gate green before I noticed the histogram was not closed.

The violating order was almost boring once I laid the events on a whiteboard. Worker W1 scored run r7 against digest dA and the coordinator published a live mean. Worker W2, recycled after a preemption, scored a stale replica of r7 against digest dB and the gate crossed the threshold backward. Why do we still store pass rate as a mutable scalar when the underlying set of scores is still open?

This piece is an architecture review of that control plane, not a production war story with invented SLOs. I treat the numbers in the fixture as labels inside a simulator, and I ask you to break them. If the invariant does not survive injected event orders, the dashboard is theater.

Declared assumptions

I am reviewing a batch evaluator for long-running agent workflows, not an online recommender with sliding windows. Each run has a content-addressed artifact, and workers may restart on shared or free compute without warning. The ship gate reads a single pass rate, and retries are at-least-once. I assume clocks are not trusted across machines, so wall time cannot close a batch. I also assume duplicate delivery is normal, because queues retry after ack loss. If your evaluator never preempts and never retries, this protocol is heavier than you need. Do you actually have that luxury on shared inference?

Constraints I will not hand-wave

Latency, throughput, correctness, and cost pull in four different directions on this path. A live mean is cheap to render and easy to explain to a reviewer who wants a single number. Waiting for every worker is correct until one worker never returns, which is the common case on preemptible capacity. Quorum without a digest check is fast and still wrong, because two scores can share a run id and describe different bytes. Cost stays bounded only if retries cannot steal slots from fresh runs. Which constraint do you actually want to be load-bearing when the batch is half dead?

I refuse to treat “the model got worse” as the first hypothesis. The first hypothesis is that the merge protocol accepted an event it should have fenced. Evaluation architecture fails in the control plane long before it fails in the decoder.

Data flow and failure domains

Here is the sequence I want the coordinator to enforce. Notice that the gate never reads a scalar that workers can still mutate.

sequenceDiagram
    participant Q as Eval queue
    participant W as Worker
    participant C as Coordinator
    participant H as Histogram log
    participant G as Ship gate

    Q->>W: lease run r7, epoch e, digest dA
    W->>C: score(r7, dA, e, bin=pass)
    C->>H: append if digest+epoch legal
    Note over C,H: watermark closed_count, not wall clock
    H->>G: derive pass_rate from closed bins
    W-->>C: late score(r7, dB, e_stale)
    C-->>H: reject stale epoch / digest mismatch

I split the system into four failure domains on purpose. Domain A is the queue, which can redeliver, delay, or drop an ack. Domain B is the worker, which can preempt, restart with a new epoch, or finish a stale replica. Domain C is the coordinator log, which must append monotonically and never rewrite a published bin. Domain D is the gate, which must refuse to read an open histogram. If you collapse C and D into one mutable counter, a single late write moves the business decision. Is that really an evaluation system, or a shared integer?

The invariant the common implementation drops

Invariant. A ship/no-ship pass rate is legal only when it is derived from a closed histogram of scores whose (run_id, artifact_digest) pairs are unique, whose worker epochs are still fenced in, and whose accepted count meets a declared quorum denominator. Late, duplicate, or digest-mismatched events must not move any bin that the gate can read. The denominator is closed_accepted_count, not “runs we hoped to finish.” If you divide by intended count while stragglers still arrive, you are mixing two different experiments.

The common implementation stores pass_rate = passes / n on a row and updates it in place. That row is not a protocol. It is a race.

Numbered protocol I would actually implement

1. Fence the worker, not the hostname. Issue (run_id, epoch, digest) as a single lease. A restarted process must bump epoch before it can speak. Hostnames lie after recycle, and I will not key correctness on them.

2. Address the artifact, not the prompt text. Hash the frozen workspace bytes before the worker starts. If the digest changes, it is a different run, even when the run id looks familiar. Would you let two judges vote on different files and still call it one sample?

3. Append bins, never mutate a mean. The log stores counts per bin. The mean is a pure function of a closed snapshot. Derived numbers can be cached, but the cache is not the source of truth.

4. Close on watermark plus quorum, never on wall clock. Declare N intended runs, Q minimum accepted scores, and a max in-flight lease set. When accepted >= Q and no legal epoch remains in flight, the histogram closes. Wall time is a timeout for leases, not a close signal for science.

5. Classify every rejected event. Stale epoch is a reject. Digest mismatch is a reject and a new run proposal. Duplicate (run_id, digest, epoch) is an idempotent ack. After close, everything is a reject. Compensation, if any, allocates a fresh run_id rather than rewriting history.

6. Put backpressure in the same contract. If retry storms fill the lease set, refuse new admissions until in-flight epochs drain. Otherwise retries starve fresh eval and your denominator silently shrinks. Have you measured how many of last week’s “model regressions” were actually queue occupancy?

Minimal simulator you can run locally

This fixture is a proposal, not a production service. I ran it as a single-process event pump so the merge rules are visible. You should inject orders, not admire the happy path.

from collections import defaultdict
from dataclasses import dataclass

PASS, FAIL = "pass", "fail"

@dataclass(frozen=True)
class Score:
    run_id: str
    digest: str
    epoch: int
    bin: str

class HistogramLog:
    def __init__(self, quorum: int):
        self.quorum = quorum
        self.fenced_epoch = {}
        self.accepted = {}  # run_id -> (digest, epoch, bin)
        self.bins = defaultdict(int)
        self.closed = False
        self.rejects = []

    def fence(self, run_id: str, epoch: int) -> None:
        prev = self.fenced_epoch.get(run_id, -1)
        if epoch <= prev:
            raise ValueError("epoch must increase to fence a restart")
        self.fenced_epoch[run_id] = epoch

    def apply(self, s: Score) -> str:
        if self.closed:
            self.rejects.append(("after_close", s))
            return "reject_closed"
        if self.fenced_epoch.get(s.run_id, -1) != s.epoch:
            self.rejects.append(("stale_epoch", s))
            return "reject_epoch"
        prior = self.accepted.get(s.run_id)
        if prior:
            digest, epoch, bin_ = prior
            if (digest, epoch, bin_) == (s.digest, s.epoch, s.bin):
                return "idempotent"
            self.rejects.append(("digest_or_dup_conflict", s))
            return "reject_conflict"
        self.accepted[s.run_id] = (s.digest, s.epoch, s.bin)
        self.bins[s.bin] += 1
        if len(self.accepted) >= self.quorum:
            self.closed = True
        return "accept"

    def pass_rate(self):
        n = len(self.accepted)
        if not self.closed or n == 0:
            raise RuntimeError("gate cannot read an open histogram")
        return self.bins[PASS] / n

def replay(events, quorum=3):
    log = HistogramLog(quorum=quorum)
    actions = []
    for kind, payload in events:
        if kind == "fence":
            log.fence(*payload)
            actions.append("fence")
        else:
            actions.append(log.apply(payload))
    return actions, log

A counterexample that should fail the naive scalar, and pass this log, looks like the following event order. Read it as a test, not as decoration.

events = [
    ("fence", ("r7", 1)),
    ("score", Score("r7", "dA", 1, PASS)),
    ("fence", ("r8", 1)),
    ("score", Score("r8", "dA", 1, PASS)),
    ("fence", ("r9", 1)),
    ("score", Score("r9", "dA", 1, FAIL)),
    # recycled worker tries to rewrite r7 with different bytes
    ("score", Score("r7", "dB", 1, FAIL)),
    # restart with a new epoch, still too late after close
    ("fence", ("r7", 2)),
    ("score", Score("r7", "dB", 2, FAIL)),
]

actions, log = replay(events, quorum=3)
assert actions[-2] == "reject_conflict"
assert actions[-1] == "reject_closed"
assert log.closed and abs(log.pass_rate() - 2 / 3) < 1e-9
print(actions, dict(log.bins), log.pass_rate())

Run it with a stock interpreter. If pass_rate() is callable before close, your gate is already lying. If a digest mismatch can decrement a bin, you do not have a histogram, you have a tug of war. I want the fixture to make that argument boringly mechanical.

Failure analysis I actually care about

Preemption is not a rare ops event on shared capacity; it is a first-class input. A recycled worker that keeps its old epoch is a split brain, because two processes can both believe they own r7. Duplicate delivery after a successful append must be idempotent, or your counts inflate and the denominator becomes fan-out. Digest mismatch after a partial write is a new experiment, and folding it into the old run id launders contamination into the mean. Close-then-late is the flip I opened with, and it is the one dashboards love, because the line moves after someone already screenshotted it. Which of those four do you currently classify, and which do you still file as “model variance”?

I also inject a fifth class: retry storms that occupy every lease. Backpressure has to live beside the histogram, or correctness wins a tiny closed set while freshness dies. A perfectly fenced batch of twelve retries is not a canary. It is a congested queue wearing a lab coat.

Tradeoff table

Merge policy	Latency to a number	Correctness under preemption	Cost under retries	What the gate can claim
Mutable scalar mean	Lowest	Poor; late writes flip the gate	Unbounded retries rewrite the same row	A screenshot, not a contract
Wait for every worker	Highest	Good until one worker vanishes	Idle capacity while stragglers sit	Completeness you will not get
Quorum without digest	Medium	False consensus on mixed bytes	Medium	A majority of the wrong artifact
Quorumed histogram plus epoch fence	Medium, bounded by Q	Rejects stale epochs and mixed bytes	Bounded if leases backpressure	A closed denominator you can defend

I would pick the last row for ship gates and keep the live scalar, if you must, as a non-binding progress widget. Mixing those two audiences on one chart is how 0.81 becomes 0.74 becomes a postmortem. Are you designing for operators who need a pulse, or for a gate that must not blink?

Where free shared compute actually belongs

I needed a scratch place to execute the fixture and a cheap model endpoint to generate dummy traces, without standing up a reserved cluster. I used MonkeyCode’s free model access and free server option for that scratch path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not asserting model names, quotas, hardware, duration, or permanence; those two availability options are the only product claims I am willing to stand on here. Shared free workers make preemption more likely, which is exactly why a mutable mean is the wrong gate. If you only ever run eval on pinned capacity, you can still use the log, but you will feel the protocol as overhead rather than as survival.

If you try the same fixture there, keep the product at the edge of the method. The invariant lives in the merge rules, not in the vendor logo. One pass through replay() on your laptop is enough to see the reject classes. A remote server is optional, and it should not become the argument.

Limitations, and who should not use this

This protocol is wrong for streaming product analytics that must fold late events into a sliding window by design. It is also not a substitute for power analysis, confidence intervals, or paired comparisons against a frozen baseline. Quorum Q can hide bias if the missing workers are not missing at random, and I did not model that selection effect here. Do not use a closed histogram as permission to skip artifact hashing. Do not use free shared servers for eval that cannot tolerate preemption even with fencing, such as hardware-in-the-loop jobs that cannot restart cleanly. If your audience needs incident dashboards and pager policies, that operational layer belongs elsewhere; this review stops at the merge contract.

The simulator is single process. It does not prove linearizability under a real log. Treat it as a property sketch you can promote into a replicated append-only store later.

What I would change next

I would stop exporting pass rate as a time series that updates in place. I would export closed snapshots with a snapshot id, a digest set, and an explicit denominator. I would add a property test that generates shuffled event orders instead of the one counterexample above. I would also split retry admission from score merge, because they fail independently and currently share one mental model. After that, I would canary the gate with three failure classes only: stale epoch, digest mismatch, and after-close delivery. If those three cannot move the snapshot, I will start arguing about models again. Until then, the model is not the suspect.

Readers can validate the conclusion without trusting me. Run the fixture. Shuffle the event list. Add a worker that reuses run_id with a new digest before close. If your current evaluator still returns a number, you found the bug the dashboard will not name.

Which event order still moves your pass rate after close, and should the system reject, replay, or compensate?