Canary Remote Agent Eval With Timeout Races and a Quorum Gate

I opened a design review with a timeline that should have blocked the merge, not flipped a dashboard. A remote eval runner went silent, so the control plane retried the same job elsewhere. The second replica returned a hard failure, and a human almost shipped an unnecessary rollback. Then a late success arrived from the first host, and last-write-wins painted the gate green.

Have you ever let the last arriving packet decide whether an agent change was actually safe? I keep reconstructing that race because shared eval hosts make the pattern ordinary, not exotic. The invariant I want is simple, and most harnesses I review fail to preserve it. A merge gate may advance only when a quorum hashes one transcript under a frozen snapshot.

Declared assumptions

I am reviewing an evaluation control plane rather than a production agent runtime, and that boundary is deliberate. The unit of work is one eval job, keyed by a content-addressed snapshot of prompts, tools, and fixtures. Replicas may run on machines you do not own, including a free remote server that can vanish. I assume at-least-once delivery of start and complete messages, and I refuse to trust wall clocks.

Why freeze the snapshot instead of trusting whatever prompt tree the runner happened to pull at start? A late replica that rebuilt the tree can pass a different exam than the one you gated. Does your harness even persist the snapshot digest beside the boolean pass bit in the gate record? If the answer is no, the rest of this review is already describing your incident.

Constraints that actually bind

I treat these as hard constraints, not style preferences, because they follow from the hosts you do not schedule.

1. You cannot treat a single remote completion as evidence, because that host is an independent failure domain.
2. You cannot cancel a silent replica with certainty, because a cancel message is just another unreliable gossip packet.
3. You cannot compare raw pass and fail bits across replicas, because two greens may still hash different transcripts.
4. You cannot wait forever for stragglers, because a merge queue has a finite admission window and humans will bypass it.

Those four constraints force a protocol, not a prettier status column on the dashboard. I want a quorum of hashed transcripts, a reject path for late replicas, and an explicit replay after the window. Anything weaker is last-write-wins with extra YAML.

Data flow I would actually ship

Here is the sequence I want the control plane to enforce, even when one replica lives on a machine you do not operate. The late pass is evidence about the worker, not a vote that can rewrite the gate.

sequenceDiagram
    participant Gate as MergeGate
    participant CP as EvalControlPlane
    participant S as SnapshotStore
    participant R1 as Replica1
    participant R2 as Replica2
    participant R3 as Replica3

    Gate->>CP: requestEval(changeId)
    CP->>S: freeze(snapshotDigest)
    CP->>R1: start(jobId, digest, window)
    CP->>R2: start(jobId, digest, window)
    CP->>R3: start(jobId, digest, window)
    R2-->>CP: complete(transcriptHash, fail)
    R3-->>CP: complete(transcriptHash, fail)
    CP->>Gate: reject(noQuorumPass)
    R1-->>CP: lateComplete(otherHash, pass)
    CP-->>R1: rejectReplica(outsideWindow)

Notice the late pass never mutates the gate after quorum has already spoken. The control plane records that packet as a counterexample for later replay design. If your worker still upserts eval_status = payload.status, you are implementing the bug with a REST client. Who owns the write to the gate in your diagram today?

Failure domains

I split the system into four domains because they fail independently and should not share a mutable row.

1. Snapshot store. If this lies, every replica studies a different exam, and quorum becomes theater.
2. Control plane log. If this drops complete messages, you either retry forever or you double-count identical votes.
3. Remote replica hosts. This includes laptops, borrowed CI tenants, and any free server you do not schedule.
4. Merge gate. If this reads a mutable status cell, a late packet can rewrite history after reviewers have moved on.

The remote host is not the eval system. It is an untrusted worker that may return a well-formed result after you already decided. Have you drawn that boundary on your own whiteboard, or does the runner still write straight into the gate?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A replica set still needs somewhere inexpensive to run untrusted workers, and MonkeyCode currently offers free model access and a free server option for that worker role. Those hosts remain a failure domain. They must not become the signing authority for a merge, even when the HTTP body looks perfectly green.

Minimal simulator you can run

I want an executable counterexample, not another architecture slide without an execution path. The fixture below is a proposal, and I am not attaching production timings or pass rates to it. Run it locally and watch last-write-wins accept a late pass that quorum would reject.

# eval_quorum_sim.py
# Proposal: timeout-race simulator for remote agent eval replicas.
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Tuple

@dataclass(frozen=True)
class ReplicaResult:
    replica_id: str
    window_id: int
    snapshot: str
    transcript_hash: str
    passed: bool
    arrived_at: int  # logical time, not wall clock

@dataclass
class ControlPlane:
    window_id: int = 1
    snapshot: str = "snap-a"
    quorum: int = 2
    window_end: int = 10
    seen: Dict[str, ReplicaResult] = field(default_factory=dict)
    log: List[ReplicaResult] = field(default_factory=list)
    gate: Optional[str] = None  # pass | fail | None

    def ingest(self, r: ReplicaResult) -> str:
        if r.window_id != self.window_id or r.snapshot != self.snapshot:
            return "reject_replica"
        if r.arrived_at > self.window_end:
            return "late_gossip"
        prior = self.seen.get(r.replica_id)
        if prior is not None:
            return "idempotent_duplicate"
        self.seen[r.replica_id] = r
        self.log.append(r)
        buckets: Dict[str, List[ReplicaResult]] = {}
        for vote in self.log:
            buckets.setdefault(vote.transcript_hash, []).append(vote)
        for digest, votes in buckets.items():
            if len(votes) >= self.quorum:
                self.gate = "pass" if votes[0].passed else "fail"
                return "quorum_decided"
        return "accepted_vote"

def last_write_wins(events: List[ReplicaResult]) -> str:
    return "pass" if events[-1].passed else "fail"

def demo_timeout_race() -> Tuple[str, Optional[str], List[str]]:
    cp = ControlPlane()
    events = [
        ReplicaResult("r2", 1, "snap-a", "h-fail", False, 4),
        ReplicaResult("r3", 1, "snap-a", "h-fail", False, 5),
        ReplicaResult("r1", 1, "snap-a", "h-pass", True, 12),  # late
    ]
    actions = [cp.ingest(e) for e in events]
    return last_write_wins(events), cp.gate, actions

if __name__ == "__main__":
    lww, gate, actions = demo_timeout_race()
    assert lww == "pass"
    assert gate == "fail"
    assert actions[-1] == "late_gossip"
    print("lww=", lww, "quorum_gate=", gate, "actions=", actions)

Run it with python eval_quorum_sim.py. You should see last-write-wins print pass while the quorum gate stays fail. That is the violating event order I opened with, reduced to three structs and a logical clock. If the assertion fails on your machine, the control plane model itself has drifted, and you should stop before adding network code.

Injected failures and testable properties

I would canary this harness with a short failure list before I trusted it on a real agent change. The point is not chaos for its own sake. The point is to prove the gate is monotonic after the admission window closes.

1. Timeout then late success. The gate must ignore the late replica and keep the quorum decision already recorded.
2. Digest drift. A replica that rebuilt prompts under a new snapshot must be rejected, even if it reports pass.
3. Duplicate complete. Two copies of the same replica vote must count as one vote, not as artificial majority.
4. Split transcripts. Two passes with different hashes are disagreement, not a green merge you can ship.

I would assert these properties in a unit test, not in a design document that nobody executes.

def test_late_pass_cannot_invert_fail():
    lww, gate, actions = demo_timeout_race()
    assert lww == "pass"
    assert gate == "fail"
    assert actions == ["quorum_decided", "accepted_vote", "late_gossip"] or \
           actions[0] in {"accepted_vote", "quorum_decided"}

def test_duplicate_replica_is_idempotent():
    cp = ControlPlane()
    a = ReplicaResult("r2", 1, "snap-a", "h-fail", False, 4)
    assert cp.ingest(a) in {"accepted_vote", "quorum_decided"}
    assert cp.ingest(a) == "idempotent_duplicate"

If your current CI only checks that eval_status is not null, you are testing occupancy, not agreement. Which of those four failures would your harness currently swallow without a sound?

Tradeoff table

Design	Latency to decision	Correctness under late replicas	Cost in replica hours	Operational burden
Freeze snapshot plus quorum hashes	Higher, waits for k of n	Preserves the invariant	Higher, extra workers	You must store transcripts
Single remote runner, last write	Lowest	Broken by the opening race	Lowest	Looks simple until merge day
Wait for every replica, including late ones	Unbounded	Can still mix snapshots	Wasteful	Humans bypass the wait
Majority pass bits, ignore transcripts	Medium	False agreement on different exams	Medium	Silent semantic drift

The denominator I care about is agreed transcript hashes per frozen snapshot, not HTTP 200 bodies per job id. An acceptance rule follows from that denominator. Merge only when at least two replicas share one digest and one transcript hash inside the admission window. Otherwise reject the change, then replay under a new window, or compensate by leaving the change unmerged.

Architecture review: what I would change next

If I inherited this control plane tomorrow, I would stop replicas from writing status rows directly into the gate. I would make the control plane log the only voter, and I would store transcript hashes as content-addressed blobs. I would also split admission from evidence collection, so a late replica can still be audited without moving the gate. That split is the change that kills last-write-wins without pretending you can cancel the universe.

Would I put a free remote host in the replica set at all? Yes, as a worker, never as a quorum of one. Free model access is useful when you want another model-backed eval path without standing up a private farm. A free server option is useful when you want a fourth failure domain that is not your laptop and not your only CI tenant. Neither one should sign a release, and neither one should be allowed to upsert the merge bit.

I would not use this approach for certified safety eval, for regulated audit trails, or for teams that already run dedicated deterministic runners. I would also not use a two-replica quorum if both replicas share a network, a disk, and a deploy pipeline. Shared fate is not diversity, and a duplicate process id is not a second failure domain. If your fixtures are flaky, you will quorum-fail forever, and that is the correct outcome rather than a reason to weaken the gate.

Quorum does not create a better model. It only stops you from treating a timeout race as a pass. Transcript hashing does not prove the agent is helpful. It proves replicas took the same exam under the same frozen snapshot. That is a smaller claim, and it is the one a merge gate can actually defend.

A path to validate the conclusion

1. Copy the simulator and replace snap-a with the digest your real prompt bundle would produce.
2. Inject a late pass after window_end and confirm the recorded gate does not invert.
3. Run two replicas against the same frozen snapshot and require matching transcript hashes before admission.
4. Record the rejected late replica in the log, then choose reject, replay, or compensate with the change still closed.

Which event order breaks the invariant, and should the system reject, replay, or compensate? If replica r1 lands a late pass after r2 and r3 already hashed a failure under the same snapshot, I want reject then replay under a new admission window, never a rewritten gate. What does your harness do with that packet today?