Make Eval Completion a Join Barrier Before Duplicate Tool Spend Rewinds Pass/Fail

I keep a lab fixture because shared eval loops lie in a very specific way. The dashboard flips a flaky agent case from red to green after the shared budget is already gone. The planner joined two tool results, stored pass, and then a slower worker delivered a duplicate retry. Does that retry still spend the token window, or do we pretend the case is already closed?

That event order is the counterexample I use whenever shared eval compute looks cheap enough to skip a protocol. I am not reviewing a prompt. I am reviewing whether completion is a join, or just the first HTTP 200 that made the notebook feel done.

The invariant the common loop fails to preserve

I want one property, and I want it testable without a production cluster. An eval case may commit pass or fail only after every tool attempt in its fan-out has joined or been rejected against a still-open admission window. Late deliveries after that window closes must not mutate the recorded verdict, and they must not spend budget that belongs to the next case. If your harness treats a worker 200 as completion, you already dropped the invariant. Have you ever marked a case green while a retry was still in flight on another core?

MCP-style tool fan-out makes the join set obvious, which is why I care about it now. The tools are not the architecture. The architecture is whether completion is closed before the next case is admitted.

Declared assumptions

I am reviewing a lab architecture, not a production planner, and every fixture below is labeled until you run it. Assumption one: tool workers are at-least-once, so duplicates are normal rather than exceptional. Assumption two: the scarce resource is a shared token budget on one eval host, not a private GPU map. Assumption three: verdicts live in an append-only log, and readers treat the latest committed verdict as truth. Assumption four: I will not invent wall-clock SLOs, model names, or load numbers I did not collect.

If any assumption is false in your shop, the join protocol is the wrong fence. Would you still commit on the first 200 if assumption one is the only honest one?

Constraints, data flow, and where the join lives

I treat the eval host as a tiny distributed system because the workers already are. The control plane admits a case, leases a token window, fans out tool calls, and must not commit until the join set is closed. The data plane stays dull on purpose: each attempt carries case_id, attempt_id, and window_id, and the join table keys on that triple. Why would you key only on case_id when retries exist? That is how most notebooks rewind pass/fail without noticing.

Here is the lab flow I want, drawn as a sequence rather than a slide.

sequenceDiagram
    participant Admit as Admission
    participant Log as VerdictLog
    participant W1 as ToolWorkerA
    participant W2 as ToolWorkerB
    Admit->>Log: open window(case, budget)
    Admit->>W1: call tool (case, attempt=1, window)
    Admit->>W2: call tool (case, attempt=2, window)
    W1-->>Admit: result attempt=1
    Admit->>Log: join(attempt=1)
    Note over W2,Admit: retry or late duplicate
    W2-->>Admit: result attempt=2 after close?
    Admit->>Log: reject if window closed
    Admit->>Log: commit verdict iff join set complete and window open

Notice the reject path is part of completion, not an error handler you bolt on later. If the late result lands after close, the system should reject, not replay, and not compensate by flipping the stored verdict. Compensation belongs to the product workflow under test, not to the harness that claims to measure it. Do you see how easy it is to mix those two planes?

Failure domains I actually care about

I split failures into four domains because they do not share a fix. Domain A is admission: the window opens with a budget that cannot cover the fan-out, so the case should never start. Domain B is in-flight duplication: two workers execute the same attempt id, or two attempt ids for one logical tool. Domain C is window expiry: remaining tokens hit zero while the join set is still open. Domain D is log rewind: a late writer appends a second verdict for a case that already committed.

Which domain is your green dashboard actually hiding? Shared free compute makes Domain C louder, because noisy neighbors steal tokens without stealing your process. That is a constraint on the join protocol, not an operations tutorial. If you cannot observe remaining budget as a first-class signal, you cannot close the window honestly. I would rather fail the case closed than let a late duplicate spend the next case's window.

Minimal state machine

I keep five states, and I refuse hidden sixth states in worker-local memory. NEW means admitted but no window. OPEN means a window id is live and the join set is incomplete. JOINED means every expected attempt has a terminal event inside the window. COMMITTED means a verdict was appended and further tool events must be rejected. ABORTED means the window closed early and the verdict is fail or inconclusive, never a silent pass.

Can a case move from COMMITTED back to OPEN? Not in this protocol, and that is the whole point.

NEW -> OPEN -> JOINED -> COMMITTED
              \-> ABORTED
OPEN -> ABORTED
COMMITTED -> (reject late events, stay COMMITTED)

A numbered lab you can actually run

I want you to validate the claim with a fixture, not with a slide about backpressure. Follow these steps on one machine.

1. Save the simulator below as eval_join.py and read the assertions before you edit anything.
2. Run python3 eval_join.py and confirm the late duplicate is rejected after commit.
3. Run python3 eval_join.py --inject duplicate_after_close and watch the next case keep its budget.
4. Run python3 eval_join.py --inject budget_exhausted_mid_join and confirm the verdict is ABORTED, not pass.
5. Change the join key to case_id only, rerun, and record which property breaks first.

If step five does not fail a property, your test is too weak. What event order did you forget?

Simulator (labeled fixture, not a production service)

#!/usr/bin/env python3
"""Eval join-barrier simulator. Fixture only; not a production planner."""
from __future__ import annotations

import argparse
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Set


@dataclass
class Event:
    case_id: str
    attempt_id: str
    window_id: str
    tokens: int
    kind: str  # result | duplicate | late


@dataclass
class Case:
    case_id: str
    expected: Set[str]
    window_id: Optional[str] = None
    state: str = "NEW"
    joined: Set[str] = field(default_factory=set)
    spent: int = 0
    verdict: Optional[str] = None


class EvalJoin:
    def __init__(self, budget: int) -> None:
        self.budget = budget
        self.remaining = budget
        self.cases: Dict[str, Case] = {}
        self.log: List[str] = []
        self.rejected: List[Event] = []

    def admit(self, case: Case, window_cost: int) -> None:
        if self.remaining < window_cost:
            case.state = "ABORTED"
            case.verdict = "inconclusive"
            self.cases[case.case_id] = case
            self.log.append(f"abort_admit {case.case_id}")
            return
        case.window_id = f"w-{case.case_id}"
        case.state = "OPEN"
        self.remaining -= window_cost
        case.spent += window_cost
        self.cases[case.case_id] = case
        self.log.append(f"open {case.case_id} {case.window_id}")

    def on_tool(self, ev: Event) -> None:
        case = self.cases[ev.case_id]
        if case.state in {"COMMITTED", "ABORTED"}:
            self.rejected.append(ev)
            self.log.append(f"reject_late {ev.case_id} {ev.attempt_id}")
            return
        if ev.window_id != case.window_id:
            self.rejected.append(ev)
            self.log.append(f"reject_window {ev.case_id} {ev.attempt_id}")
            return
        if ev.attempt_id in case.joined:
            self.rejected.append(ev)
            self.log.append(f"reject_dup {ev.case_id} {ev.attempt_id}")
            return
        if ev.attempt_id not in case.expected:
            self.rejected.append(ev)
            self.log.append(f"reject_unknown {ev.case_id} {ev.attempt_id}")
            return
        case.joined.add(ev.attempt_id)
        self.log.append(f"join {ev.case_id} {ev.attempt_id}")
        if case.joined == case.expected:
            case.state = "JOINED"
            self.commit(case, "pass")

    def expire(self, case_id: str) -> None:
        case = self.cases[case_id]
        if case.state == "OPEN":
            case.state = "ABORTED"
            case.verdict = "inconclusive"
            self.log.append(f"abort_expire {case_id}")

    def commit(self, case: Case, verdict: str) -> None:
        if case.state not in {"JOINED", "ABORTED"}:
            raise AssertionError("commit without join or abort")
        if case.state == "JOINED":
            case.state = "COMMITTED"
            case.verdict = verdict
            self.log.append(f"commit {case.case_id} {verdict}")


def scenario(inject: str) -> EvalJoin:
    harness = EvalJoin(budget=100)
    c1 = Case("case-1", expected={"a1", "a2"})
    c2 = Case("case-2", expected={"b1"})
    harness.admit(c1, window_cost=40)
    harness.on_tool(Event("case-1", "a1", "w-case-1", 10, "result"))
    if inject == "budget_exhausted_mid_join":
        harness.remaining = 0
        harness.expire("case-1")
        harness.on_tool(Event("case-1", "a2", "w-case-1", 10, "late"))
        harness.admit(c2, window_cost=40)
        return harness
    harness.on_tool(Event("case-1", "a2", "w-case-1", 10, "result"))
    late = Event("case-1", "a2", "w-case-1", 10, "duplicate")
    if inject == "duplicate_after_close":
        harness.on_tool(late)
        harness.admit(c2, window_cost=40)
        return harness
    harness.on_tool(late)
    return harness


def check(harness: EvalJoin, inject: str) -> None:
    c1 = harness.cases["case-1"]
    if inject == "duplicate_after_close":
        assert c1.state == "COMMITTED" and c1.verdict == "pass"
        assert any(e.kind == "duplicate" for e in harness.rejected)
        assert harness.cases["case-2"].state == "OPEN"
        assert harness.remaining == 20
    elif inject == "budget_exhausted_mid_join":
        assert c1.state == "ABORTED" and c1.verdict == "inconclusive"
        assert c1.verdict != "pass"
        assert harness.cases["case-2"].state == "ABORTED"
    else:
        assert c1.state == "COMMITTED"
        assert any("reject_dup" in line for line in harness.log)


if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--inject", default="duplicate_after_close")
    args = p.parse_args()
    h = scenario(args.inject)
    check(h, args.inject)
    print("ok", args.inject, h.log, "rejected", len(h.rejected))

Run it like this.

python3 eval_join.py --inject duplicate_after_close
python3 eval_join.py --inject budget_exhausted_mid_join

The first command should keep case-2's window intact after rejecting the duplicate. The second command should refuse to call a mid-join case pass once the shared budget is gone. If either assertion fails on your machine, the protocol is not implemented. Do not argue with the dashboard.

Testable properties and injected failures

I care about three properties, and I want them as predicates over the log. Property P1: after commit, no later join line exists for that case_id. Property P2: a rejected duplicate does not decrease remaining a second time. Property P3: pass never appears on an ABORTED case. Which property does your current harness skip because the happy path is green?

Injected failures I would keep in CI are duplicate after close, unknown attempt id, mismatched window id, admit when remaining is below window cost, and expire while one attempt is missing. I would not inject model-quality noise here, because that confuses planner bugs with harness bugs. The denominator for acceptance is one hundred seeded permutations of those five failures, and the rule is zero violations of P1 through P3. Anything weaker is a demo, not a gate.

Tradeoffs

Choice	Latency	Throughput	Correctness	Cost
Commit on first 200	Lowest	Highest	Breaks P1 on late retry	Cheap until the rewind
Join barrier, reject late	Wait for the slowest attempt	Caps in-flight cases by budget	Preserves P1–P3	Extra bookkeeping
Replay late events into a new window	Unbounded tail	Burns the next case	Rewinds verdicts	Hidden token spend
Compensate by flipping pass/fail	Looks responsive	Pollutes metrics	Mixes product and harness	Debug cost dominates

I pick the join barrier for eval, and I keep compensation inside the agent under test. Replay is the wrong default on a shared token pool. Would you rather wait for a missing attempt, or publish a green number you cannot replay?

Where a free shared server actually fits

I needed optional lab capacity that can replay this join log against a real tokenizer path without turning the review into a capacity paper. MonkeyCode is an open-source project that currently offers free model access, stated by the operator as on the order of ten million tokens, plus a free server option I would treat as a lab rather than a multi-tenant SLO. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would park the fixture there to replay rejects, not to claim any model is accurate, and I would still gate every case on the window instead of a worker 200.

If you want a scratch box for the simulator, that free server is enough to store the log and reject late events. I would not point a production planner at it. The protocol is the artifact; the server is only scaffolding around the same fixture.

Limitations, and who should not use this

This protocol does not give you model quality, prompt regression insight, or a tracing story. It will abort aggressively when the shared budget is tight, which is correct for a lab and hostile for an interactive product. Do not use it if you need exactly-once side effects in customer systems, because the workers are still at-least-once. Do not use it if eval cases share mutable world state without their own idempotency keys. Do not use it as an incident runbook; I am not covering dashboards, deploys, or on-call here.

I also did not measure tokens per case on a live model, and I will not invent a speedup. The acceptance rule is the three properties, not a throughput multiplier. If you cannot run the fixture, you cannot claim the architecture converges.

What I would change next

I would persist the join table in an append-only file before I trusted a second process. I would bind window_id to a hashed (case_id, expected_attempts, budget_epoch) so a restarted harness cannot reopen a committed window. I would add a property test that permutes event arrival instead of the two scripted injections above. I would still reject late events rather than compensating inside the harness. The next design review is whether inconclusive should block a release gate or merely quarantine the case.

Counterexample question

Which event order breaks the invariant: a duplicate a2 after commit, an a2 with a stale window_id, or an admit of case-2 while case-1 is still OPEN and remaining is already zero? Should the system reject, replay, or compensate? If your answer is replay because the tool is idempotent, you are optimizing the product path and abandoning the eval path. Run the fixture, inject that order, and only then decide whether the join barrier is too strict.