Make Judge Binding a Closed Contract Before Pool Rebalance Flips Pass/Fail

I start this architecture review with a canary that went red while candidate traces stayed byte-identical. The rubric file had not moved, and the planner graph was untouched. The judge prose had drifted overnight anyway. Have you ever signed a pass/fail bit that was authored by a critic you never bound?

That event order is the bug, not a mysterious model regression. The candidate finished under binding B1, the pool rebalanced, and the scorer ran as B2. The log still treated the row as one closed eval. I want that mix to be a protocol violation, not a flaky-model ticket.

The violating order

Here is the counterexample I keep on the whiteboard during a control-plane review. It is short, ugly, and enough to dissolve a release gate.

1. Enqueue records run_id=R, candidate_binding=C1, and judge_binding=J1.
2. The candidate stream completes, and the tool log is sealed for R.
3. The pool rebalances, and the next completion is routed to J2.
4. A score for R is appended with no fingerprint check against J1.

Did the system reject, replay, or compensate? In the common implementation it just commits. Why do we debug the model when the contract already dissolved?

Invariant

A persisted eval outcome is valid only when four fields close together: run_id, candidate_binding, judge_binding, and score_digest. If any field is rewritten after bind, the outcome is not the same object. I do not care that both critics are good enough for a demo. I care that the bit we ship is the bit we contracted.

Call the invariant closed judge binding. Once bind() returns, scoring traffic for that run_id must carry the same binding token, or the write is refused. Anything else is a different experiment wearing the original name.

Assumptions

I am reviewing a shared-pool eval service, not a single-process notebook. These assumptions stay in force for the rest of the design, and you should reject the conclusion if yours differ.

1. Candidate and judge may be scheduled on different workers, and those workers do not share memory.
2. The model pool can rebalance, preempt, or replace a replica between any two RPCs.
3. Score rows outlive the worker that wrote them, and downstream gates read those rows as truth.
4. Bindings are opaque tokens with an equality test; I am not assuming a stable vendor model name.
5. Readers will run the simulator below as the acceptance test, not a production SLO board.

If your judge is a deterministic program with no model pool, you can skip this contract. If you intentionally ensemble several critics, you need a different object: a quorum, not a singleton bind.

Constraints I will not relax

Latency, throughput, correctness, and cost pull in four directions here. I will not pretend they agree, and I will not hide the conflict behind a retry loop.

The control plane must keep enqueue cheap, because eval traffic is bursty around a release. It must not hide a pool swap inside a 200 from the scorer. It must not bill a second full candidate run unless replay was explicit. And it must survive shared-pool preemption without corrupting the pass bit. Which of those would you drop first? I will not drop correctness of the score row.

Data flow

The ledger, not the HTTP client, owns equality. Workers are allowed to die. The score log is not allowed to forget which critic signed the bit.

sequenceDiagram
    participant P as Planner
    participant B as BindingLedger
    participant C as CandidatePool
    participant J as JudgePool
    participant L as ScoreLog

    P->>B: bind(run, C1, J1)
    B-->>P: token T
    P->>C: complete(run, T)
    C-->>P: transcript
    Note over J: rebalance replaces J1 with J2
    P->>J: score(run, T, transcript)
    J-->>P: score plus observed_binding
    P->>B: compare(T, observed_binding)
    alt mismatch
        P->>L: reject or replay
    else match
        P->>L: commit(run, T, score)
    end

Notice the compare sits on the planner side of the log. A judge replica that cannot echo identity is not a scorer. It is an untrusted completion source, and I would fail that path closed.

Failure domains

I split the system into four domains so a rebalance cannot impersonate a rubric change. If you draw only one box labeled eval, you will keep filing model bugs forever.

Binding ledger. This is the source of truth for tokens. If it is down, enqueue must fail closed. A stale cache of bindings is a silent mix, which is worse than a hard error.

Candidate pool. Preemption here should fence the transcript, not the judge. A restarted candidate with the same token may replay work. It may not mint a new binding under the old run_id.

Judge pool. This is the dangerous domain on a shared pool. Rebalance, warm-start, and sticky-session loss all present as the scorer returned. I treat an identity mismatch as a transport error, not as a slightly different grade.

Score log. Append-only rows need the token in the primary key. A late score without a token is garbage, even when the numeric grade looks plausible. Would you merge that row because the dashboard went green?

Treat routing as a binding protocol

I want the implementation to look like a protocol, not a retry wrapper. Stickiness can reduce mismatch rate, but it is not a lock. Have you been using affinity as if it were a compare-and-set?

Here is the closed-contract path I would require before the service scales.

1. Mint. On enqueue, write (run_id, candidate_binding, judge_binding, epoch) in one compare-and-set.
2. Stamp. Every candidate and judge RPC carries the token; responses echo the observed identity.
3. Compare. Before append, the planner checks observed == bound. Inequality is not retried against a new replica.
4. Decide. Mismatch chooses reject, replay-with-rebind, or compensate; it does not fall through to commit.
5. Seal. A committed row stores the token beside the score digest so later audits can re-verify equality.

The missing piece in most harnesses is step three. People retry the scorer, get a warmer replica, and call that resilience. I call it a contract rewrite.

Minimal simulator

The fixture below is a proposal you can run locally. It is not a production service, and I am not claiming cluster measurements against it. Run it until the empty-log assertion bores you.

from dataclasses import dataclass


@dataclass(frozen=True)
class Binding:
    run_id: str
    candidate: str
    judge: str
    epoch: int


class ScoreLog:
    def __init__(self):
        self.rows = {}

    def commit(self, b: Binding, digest: str) -> None:
        key = (b.run_id, b.judge, b.epoch)
        if key in self.rows:
            raise ValueError('duplicate score')
        self.rows[key] = digest


class JudgeBinder:
    def __init__(self):
        self.bound = {}
        self.log = ScoreLog()
        self.pool_judge = 'J1'

    def bind(self, run_id: str, candidate: str, judge: str, epoch: int) -> Binding:
        b = Binding(run_id, candidate, judge, epoch)
        if run_id in self.bound:
            raise ValueError('already bound')
        self.bound[run_id] = b
        return b

    def rebalance(self, new_judge: str) -> None:
        self.pool_judge = new_judge

    def score(self, run_id: str, digest: str) -> str:
        b = self.bound[run_id]
        observed = Binding(b.run_id, b.candidate, self.pool_judge, b.epoch)
        if observed != b:
            return 'reject'
        self.log.commit(b, digest)
        return 'commit'


def test_rebalance_must_not_commit():
    h = JudgeBinder()
    h.bind('R', 'C1', 'J1', epoch=1)
    h.rebalance('J2')
    assert h.score('R', 'pass') == 'reject'
    assert h.log.rows == {}


def test_stable_pool_commits_once():
    h = JudgeBinder()
    h.bind('R', 'C1', 'J1', epoch=1)
    assert h.score('R', 'pass') == 'commit'
    try:
        h.score('R', 'pass')
        raise AssertionError('duplicate must fail')
    except ValueError:
        pass


if __name__ == '__main__':
    test_rebalance_must_not_commit()
    test_stable_pool_commits_once()
    print('ok')

Run it with python judge_binder.py. The interesting assertion is the empty log after rebalance. If your harness still writes a row, the invariant is already dead, and prompt edits will not revive it.

Injected failures I actually care about

I do not inject the model got worse. I inject order, because order is what a shared pool actually breaks. If your test suite only perturbs temperature, you are measuring the wrong failure class.

1. Rebalance after candidate completion, before score. Must reject or replay-with-rebind.
2. Double score with the same token. Second append must fail on the composite key.
3. Rebind with a new epoch, then score. Only the new epoch may commit; the old token is tombstoned.
4. Score that echoes a different candidate binding. Reject even if the judge matches, because mixed candidate identity is a different run.
5. Lost ack after a successful commit. Replay may read the row; it may not create a second digest.

If you cannot name which of those five your service currently mishandles, you do not have an eval architecture. You have a script that sometimes prints green.

Tradeoffs

Choice	What you keep	What you pay	When I would pick it
Reject on mismatch	Closed contract, cheap enqueue	More red canaries	Default for release gates
Replay candidate under a new bind	Eventual score	Double candidate cost, longer tail	Nightly research sweeps
Dual-judge quorum	Tolerance to one bad replica	Extra critic spend, tie-break rules	Safety-critical graders
Sticky routing only	Lower mismatch rate	Silent mixes when stickiness drops	Never, as the sole control
Compensate with a human review queue	Explicit uncertainty	Slow gates, review load	Tiny gold sets

Sticky routing is the tempting row. It looks free until the pool rebalances on a quiet night. I will take reject as the default, with replay as an explicit operator action. Quorum is a different protocol, and I will not smuggle it in through retries.

The denominator for any canary is committed rows that still match their bind token, not jobs that returned 200. An acceptance rule I would ship is simple: mismatch rate must be zero in the score log, and retries live in a separate replay counter. If you mix those counters, you will congratulate yourself for availability you did not keep.

Where a spare shared pool actually helps

I keep this simulator local until the protocol is boring. When I want the same fixture sitting next to a live shared pool, I need a spare worker that can die without taking the ledger with it. That is a scheduling constraint, not a reason to skip the compare.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is a reasonable place to park this harness while you inject rebalance and preemption. If you try it, run the five failures above before you trust a dashboard color; the protocol still owns the pass bit.

What I would change next

The binder above is a single-process sketch, and I would not ship it as the ledger. The next revision I want is a compare-and-set on a real log, with the token in the row key. I would also stamp epoch onto the candidate transcript so a late judge cannot score a preempted draft. I would not add another status page before that key exists. Would a prettier graph have caught J1 versus J2? Not if the row schema collapsed them.

I would also stop treating empty judge identity as a default. An unbound score path is how pool swaps sneak in. Fail closed at mint, or do not call it an eval service. The planner should refuse to advance when echo identity is missing, even if the numeric grade looks confident.

Limitations, and who should not use this

This contract assumes you persist scores and later treat them as gates. Ad-hoc chat experiments do not need it, and forcing a ledger there is ceremony. Multi-critic ensembles that want disagreement need a quorum object with declared members, not a singleton bind you keep swapping. If your pool cannot echo an observed identity, you cannot implement compare, and you should not fake it with a local config file.

I also will not claim this removes variance inside one bound model. It only stops you from attributing a routing mix to the model. Do not use reject-by-default if a missed nightly score is more expensive than a mixed critic. In that world, pick explicit replay, and pay for it in the budget line, not in corrupted history.

Validate the conclusion

You do not need my word. You need an order, a log, and a property that stays true after injection.

1. Bind a run and seal a transcript.
2. Change the pool identity without minting a new epoch.
3. Attempt score and watch the ledger.
4. Confirm the score log stayed empty, or a replay row was written under a new token.
5. Re-run with a stable pool and confirm a single commit.

If step 4 writes a row under the old token, the design does not converge. Fix the compare, not the prompt. Which event order still breaks closed judge binding in your tree, and should the system reject, replay, or compensate?