Make Transcript Append a Dual-Key Compare Before Preempted Workers Mix Eval Runs

Here is a violating event order I keep on the whiteboard during architecture reviews of agent eval harnesses. Run R1 starts a tool call on shared worker W, then a preemption notice arrives before the tool result is fenced. Worker W later resumes under run R2, and the late tool event from R1 is appended to R2's transcript as if causality still held. Does your scoreboard treat that as a successful tool use, or as a protocol violation?

Most harnesses I review treat resume as a convenience restart, and they silently mix two runs. The invariant the common implementation fails to preserve is not exactly-once tool execution in the abstract. Every transcript event must carry a closed envelope of run_id and lease_generation before any append is legal. Why do teams miss this during design review, when resume looks local and the failure is cross-run?

If either field disagrees, the event is poison for scoring, not a retry candidate you can reshape. This is a proposed architecture review with a labeled simulator, not a customer postmortem or metric dump. I will not invent production numbers, because the point is the event order, not a dashboard.

1. Declared assumptions

I am reviewing a shared eval pool, not a sticky accelerator reserved for a single candidate run. The assumptions below are load-bearing, and they change the protocol you should accept. If they do not match your world, the handshake I want is the wrong cost.

1. Workers are preemptible, including scavenged or free servers that can vanish in the middle of a step.
2. Tool results are delivered at least once, and they can arrive after the planner has already moved on.
3. Clocks are not a source of truth across workers, so wall time cannot fence causal order.
4. The scoreboard is the system of record for pass or fail, and mixed transcripts are invalid measurements.
5. Model identity may rotate between leases on a free pool, so a resume is not the same closed world.

If your eval never shares a worker and never resumes, this review is the wrong document for your constraints. Skip it and keep a simpler sticky-session model. Are you actually measuring one closed world, or just concatenating whatever the process still had in memory?

2. Data flow and sequence model

I want the happy path and the violating path in the same picture, because reviews that only draw success hide the invariant. The objects are a planner, a worker lease manager, a tool bus, and a scoreboard. Ask yourself which process is allowed to append, and under which key.

sequenceDiagram
    participant P as Planner
    participant L as LeaseManager
    participant W as Worker
    participant T as ToolBus
    participant S as Scoreboard

    P->>L: acquire(run=R1)
    L-->>W: lease(R1, gen=7)
    W->>T: tool_call(env={R1,7})
    Note over W: preemption
    L->>W: fence(R1, gen=7)
    P->>L: acquire(run=R2)
    L-->>W: lease(R2, gen=8)
    T-->>W: late_result(env={R1,7})
    alt naive resume
        W->>S: append(R2, late_result)
    else dual-key compare
        W->>S: reject(stale envelope)
    end

The naive branch is the counterexample that most just-retry-the-worker designs will accept without blinking. The dual-key branch is the protocol I want the harness to speak under preemption. If your sequence diagram has no reject arrow, you have not modeled resume. What does your current diagram do with that late result?

3. Failure domains

I split the system into four domains because a single worker-crashed ticket hides the real coupling. Domain A is the lease manager, which is the only writer of generation. Domain B is the worker runtime, which holds an uncommitted transcript buffer. Domain C is the tool bus, which is at-least-once and unordered relative to leases.

Domain D is the scoreboard, which must refuse mixed envelopes or the eval is not a measurement. Cross-talk happens at the seams, not inside a neatly drawn box. A preemption in B without a fence visible to C leaves a poison message in flight. Do you isolate those seams in tests, or only in architecture slides?

4. The resume handshake, in numbered steps

I would not add retries and call the review finished. I would treat resume as a compare-and-append protocol with an explicit reject. The steps below are the minimum I would demand before anyone scales the shared pool.

1. Mint a lease as the pair (run_id, generation), and never reuse generation for that run_id under any retry story.
2. Stamp every tool call and every partial transcript chunk with that pair before the bytes leave the worker.
3. On preemption, publish a fence for the old pair and drop the local uncommitted buffer instead of flushing it.
4. On resume, require a handshake: the worker presents the pair it still believes, and the manager returns live or stale.
5. Append to the scoreboard only with a dual-key compare on (run_id, generation), never with process liveness alone.
6. Dead-letter stale envelopes; do not replay them into a new lease, and do not compensate by guessing intent.
7. Close the eval only when the scoreboard has a join of fenced leases, not when the worker process is merely alive.

If step four is a log line instead of a decision, you do not have a handshake. You have hope dressed up as recovery. Would you accept that hope in a payment ledger, or only in an eval because the score feels softer?

5. Minimal simulator, labeled as a proposed fixture

This fixture is proposed and unexecuted in your environment until you run it yourself. It is small on purpose, because I want the invariant to fail in a few dozen steps. I do not want a cluster bring-up to hide a one-line compare.

from dataclasses import dataclass, field
from typing import List, Optional, Tuple

Envelope = Tuple[str, int]  # (run_id, generation)

@dataclass
class Lease:
    run_id: str
    generation: int
    live: bool = True

@dataclass
class Scoreboard:
    transcripts: dict = field(default_factory=dict)
    rejects: List[Envelope] = field(default_factory=list)

    def append(self, live: Lease, env: Envelope, payload: str) -> str:
        if not live.live or env != (live.run_id, live.generation):
            self.rejects.append(env)
            return "reject"
        self.transcripts.setdefault(live.run_id, []).append(payload)
        return "accept"

@dataclass
class Worker:
    lease: Optional[Lease] = None
    buf: List[tuple] = field(default_factory=list)

    def attach(self, lease: Lease) -> None:
        self.lease = lease
        self.buf.clear()

    def tool_call(self, payload: str) -> Envelope:
        assert self.lease and self.lease.live
        env = (self.lease.run_id, self.lease.generation)
        self.buf.append((env, payload))
        return env

    def resume_handshake(self, presented: Envelope, live: Lease) -> str:
        if presented != (live.run_id, live.generation):
            self.buf.clear()
            return "stale"
        self.lease = live
        return "live"

def simulate_crosstalk(dual_key: bool) -> dict:
    sb = Scoreboard()
    w = Worker()
    r1 = Lease("R1", 7)
    w.attach(r1)
    env = w.tool_call("tool:search")
    r1.live = False
    r2 = Lease("R2", 8)
    presented = env  # poisoned resume still holding R1
    decision = w.resume_handshake(presented, r2)
    if not dual_key:
        # naive path lies about the envelope and appends under R2
        result = sb.append(r2, ("R2", 8), "tool:search")
    else:
        result = sb.append(r2, env, "tool:search")
    mixed = any(
        "tool:search" in xs
        for rid, xs in sb.transcripts.items()
        if rid != "R1"
    )
    return {
        "decision": decision,
        "append": result,
        "mixed": mixed,
        "rejects": sb.rejects,
    }

Run the two branches as a property check, not as a demo screenshot someone might ignore. I care about the boolean mixed, not about a throughput number nobody measured on this fixture. Save it as test_resume_envelope.py and keep the assertions boring.

def test_naive_mixes_runs():
    out = simulate_crosstalk(dual_key=False)
    assert out["mixed"] is True

def test_dual_key_rejects_poison():
    out = simulate_crosstalk(dual_key=True)
    assert out["mixed"] is False
    assert out["append"] == "reject"
    assert out["rejects"] == [("R1", 7)]

python -m pytest test_resume_envelope.py -q

If the naive test does not fail on your harness, your harness is already lying about measurement. That is the entire point of the fixture. Can your current resume path fail this in under a second, or only after a human reads logs?

6. Injected failures and testable properties

I would inject three schedules before anyone argues about model quality on a shared pool. The first is preemption after tool_call and before the result envelope returns. The second is a late envelope after the generation has already bumped. The third is a double resume where two workers believe they hold generation eight.

Properties I would gate on, with an explicit denominator, look like this. P1 says every accepted append has envelope.run_id equal to the live lease run. P2 says every accepted append has envelope.generation equal to the live lease generation. P3 says mixed-transcript count is zero per one thousand injected preemption schedules.

P4 says stale envelopes are rejected or dead-lettered, never compensated by copying payload into a new run. The denominator is injected preemption schedules, not wall-clock hours and not tokens consumed. Acceptance is P3 equal to zero under the three schedules above. If you cannot state the denominator, you are not canarying a protocol.

7. Tradeoff table

I want the cost in the same row as the failure class, because correctness without a load model is just a preference. Sticky workers look clean until the free pool reclaims the box. Dual-key compare looks chatty until you price a polluted score.

Design	Correctness under preemption	Extra latency	Density on a shared pool	What I would not pretend it solves
Naive process resume	Mixes runs; scores stop being measurements	Lowest	Highest apparent	Cross-talk and closed-world eval
Sticky worker per run	Holds if the worker never dies	Medium	Poor when slots vanish	Preemption, which is in the contract
Dual-key compare-and-append	Rejects poison; eval stays closed-world	One handshake on resume	Recovers the slot	Semantic splice of an old prefix
Isolated machine per eval	Strong isolation	Highest	Worst	Short unit evals that never resume

I would pick dual-key compare for shared pools, and sticky workers only when preemption is outside the contract. Full isolation is a different product, not a resume protocol. Which row are you actually operating, when the worker image is free and therefore reclaimable?

8. Where a free shared pool actually matters

Shared, preemptible workers are not a hypothetical topology for this review, because spare capacity is how many eval pools actually run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option recreate that shared shape, which is enough to host this fixture. I am not attaching a model name, a quota, a hardware SKU, or a duration, because the handshake does not need them.

If you already have an isolated lab, run the same fixture there and ignore any vendor path entirely. If you want a scratch box, the free server option is sufficient to host the simulator process. The interesting work remains the dual-key compare, not the signup form.

9. What I would change next

Generation integers are a weak vector clock, and I would not stop there after the gate turns green. I would stamp each tool envelope with a parent hash of the last committed transcript prefix. A replay then cannot splice a semantically old payload into a live generation that merely shares an integer.

I would also split the scoreboard append into a two-phase intent, because dual writers still exist between the worker buffer and the log. Would I do that before the dual-key gate is green? No. First close the mixed-transcript hole, then raise the bar. The next review should start from a forged prefix, not from another process restart.

10. Limitations, and who should not use this

Do not adopt this handshake if every eval is a single process with no resume and no shared cache. You will pay a round trip for a failure class you cannot hit. Do not use the fixture as evidence of model quality, because it measures harness causality, not answers. Do not treat reject as compensate: copying a stale tool payload into a new run is how you launder poison.

And do not ask this review for dashboards, paging playbooks, or deploy scripts. Those are a different failure domain than the protocol, and they will not make a mixed transcript become a valid measurement. If your world is one laptop and one process, keep the sticky model and save the handshake for later.

Which event order still breaks the invariant after dual-key compare lands? A tool result that carries the live pair but a prefix from a previous world, because generation never saw the bytes. Should the system reject that append, replay the tool under a new generation, or compensate from the dead-letter? I would reject, then replay only with a fresh envelope, because anything else pretends the eval still measured one closed world.