Architecture Review: Keep Eval Provenance on the Control Plane, Not the Borrowed Runtime

Last week I ran a four-shard eval batch against a borrowed runtime, went to sleep, and read the ledger over coffee. One shard had three committed rows, another had none, and the number I had pasted into a doc was an average over attempts rather than over shards. Nothing in the pipeline had thrown an exception, which is exactly why it took me an hour to find. The current cycle of public discussion keeps asking whether AI output holds up under review; far less of it asks whether the harness that produced the number could be trusted at all.

So let me state the invariant I now refuse to ship without. An eval batch is interpretable only if every committed row is attributable to exactly one (job, shard, epoch, model_revision) tuple, and the aggregate's denominator is the shard count, not the row count. Everything below is a consequence of that sentence.

Declared assumptions

I am reviewing a specific class of system here, so the conclusions are only valid inside these assumptions.

1. The control plane (your CI job, your laptop, your scheduler) is the single writer for a job and the only component allowed to mint attempt identities.
2. A shard is sealed synchronously with respect to the commit that finalizes it; there is no window where the ledger holds an unsealed authoritative row.
3. The model endpoint is addressed through a floating alias unless the job spec pins a revision digest explicitly.
4. Preemption is a normal outcome rather than an error, and the runtime is never required to report it to you.
5. The batch aggregate is a mean over shards; if yours is a sum over tokens or a p95 over calls, the same argument applies with a different denominator.

Constraints, and where they come from

A free or preemptible runtime gives you three properties whether you asked for them or not: your work can be evicted at any point, your completion rate is not guaranteed, and the underlying model alias may resolve differently between two calls. Those are not defects, they are the contract you accepted in exchange for not paying for a queue. The architecture question is therefore not "how do I make the runtime reliable" but "what must be true on my side so that an unreliable runtime cannot corrupt a number."

Notice how much of the damage happens after the work is done. A preempted attempt is harmless if it publishes nothing; it becomes a correctness bug only when a late worker publishes a row that your ledger accepts. That is where the review should start, and it is why I moved identity and sealing to the control plane.

Data flow: who is allowed to name an attempt

sequenceDiagram
    participant CP as Control plane (single writer)
    participant R as Borrowed runtime (free / preemptible)
    participant M as Model endpoint (floating alias)
    participant L as Ledger (append-only)

    CP->>R: dispatch(key = job/shard/epoch)
    R->>M: resolve alias -> revision
    M-->>R: revision, grade
    R-->>CP: row(key, provenance, grade)
    CP->>L: commit(row)
    CP->>L: seal(shard, epoch)  %% only after the write-ahead ack
    Note over R,L: preemption may drop the row entirely,<br/>and the same attempt may publish again much later
    R-->>L: late row(same key, original or drifted revision)
    L-->>CP: committed | deduped | conflict | rejected_stale

The failure domains I actually observed

Failure domain	Symptom	What it corrupts	Containment
Preemption	attempt returns nothing	shard has no row, so the denominator silently shrinks	retry at a new epoch, never reuse the key
Late publication	row arrives after the shard sealed	an older attempt double-weights the shard	epoch gate evaluated at the ledger
Alias drift	same key, two revisions	two models mixed inside one aggregate	pin the revision per job, quarantine conflicts
Duplicate delivery	identical row twice	nothing, if writes are idempotent by key	first-write-wins per key
Clock skew	"last write wins" by timestamp	nondeterministic winner across replicas	order by control-plane epoch, not wall time

Build the fixture in five steps

The artifact below runs on the standard library alone, and it is a simulator rather than a distributed system. Run it with python3 borrowed_runtime_sim.py; every scenario is deterministic given a seed, so you can argue with the outcome instead of with my prose.

Step 1 — pin the contract. Identity comes from the control plane, and provenance travels with the row instead of being reconstructed later.

"""borrowed_runtime_sim.py — does a preemptible eval runtime preserve provenance?"""
from __future__ import annotations

import hashlib
import random
from dataclasses import dataclass


def sha(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:12]


@dataclass(frozen=True)
class AttemptKey:
    job_id: str
    shard: int
    epoch: int          # control-plane authority, strictly increasing per shard


@dataclass(frozen=True)
class Provenance:
    model_revision: str
    prompt_sha256: str
    gateway: str


@dataclass(frozen=True)
class Row:
    key: AttemptKey
    provenance: Provenance
    grade: float

Step 2 — make the ledger classify instead of merge. Four outcomes are possible, and only one of them is a commit. A conflict is not something to resolve by preferring the newer row; it is a signal that the batch must be re-run.

class Ledger:
    """First write per AttemptKey wins. Everything else is classified, never merged."""

    def __init__(self) -> None:
        self.rows: dict[AttemptKey, Row] = {}
        self.sealed: dict[int, int] = {}
        self.conflicts: list[tuple[Row, Row]] = []
        self.rejected_stale: list[AttemptKey] = []
        self.deduped = 0

    def seal(self, shard: int, epoch: int) -> None:
        self.sealed[shard] = max(epoch, self.sealed.get(shard, -1))

    def commit(self, row: Row, *, epoch_gate: bool) -> str:
        if epoch_gate and row.key.epoch < self.sealed.get(row.key.shard, -1):
            self.rejected_stale.append(row.key)
            return "rejected_stale"
        prior = self.rows.get(row.key)
        if prior is None:
            self.rows[row.key] = row
            return "committed"
        if prior == row:
            self.deduped += 1
            return "deduped"
        self.conflicts.append((prior, row))
        return "conflict"

Step 3 — stub the runtime with its real misbehaviours. The stub preempts, duplicates, publishes late, and resolves a floating alias on every call, including on the late one.

class BorrowedRuntime:
    """A free or preemptible executor: it vanishes, duplicates, and arrives late."""

    def __init__(self, revision, alt_revision, rng, cfg):
        self.revision = revision
        self.alt_revision = alt_revision
        self.rng = rng
        self.cfg = cfg
        self.late: list[Row] = []

    def _resolve(self) -> str:
        return self.alt_revision if self.rng.random() < self.cfg.p_alias_flip else self.revision

    def _row(self, key: AttemptKey, prompt: str) -> Row:
        revision = self._resolve()
        return Row(
            key,
            Provenance(revision, sha(prompt), "borrowed-runtime"),
            int(sha(f"{revision}:{prompt}")[:6], 16) / 0xFFFFFF,
        )

    def attempt(self, key: AttemptKey, prompt: str) -> list[Row]:
        if self.rng.random() < self.cfg.p_late:
            self.late.append(self._row(key, prompt))   # published after the seal
        if self.rng.random() < self.cfg.p_preempt:
            return []                                  # we time out and retry later
        row = self._row(key, prompt)
        return [row, row] if self.rng.random() < self.cfg.p_duplicate else [row]

Step 4 — let the control plane retry only unfinished shards. A shard that already holds an authoritative row is never re-dispatched, and the seal happens the moment the commit lands.

@dataclass
class Config:
    shards: int = 4
    epochs: int = 3
    p_preempt: float = 0.30
    p_duplicate: float = 0.20
    p_late: float = 0.25
    p_alias_flip: float = 0.0
    epoch_gate: bool = True
    bump_revision: bool = True


def run(cfg: Config, seed: int) -> Ledger:
    rng = random.Random(seed)
    ledger = Ledger()
    prompts = {s: f"eval-prompt-{s}" for s in range(cfg.shards)}
    carry: list[Row] = []

    for epoch in range(cfg.epochs):
        revision = f"rev-{epoch}" if cfg.bump_revision else "rev-pinned"
        runtime = BorrowedRuntime(revision, "rev-next", rng, cfg)
        for shard in range(cfg.shards):
            if any(r.key.shard == shard for r in ledger.rows.values()):
                continue                       # already authoritative, do not re-dispatch
            key = AttemptKey("job-1", shard, epoch)
            for row in runtime.attempt(key, prompts[shard]):
                if ledger.commit(row, epoch_gate=cfg.epoch_gate) == "committed":
                    ledger.seal(shard, epoch)
        carry.extend(runtime.late)

    for row in carry:                          # the tail: attempts that finished late
        ledger.commit(row, epoch_gate=cfg.epoch_gate)
    return ledger

Step 5 — audit with named properties. The audit returns the denominator explicitly, because a missing shard and a double-counted shard are different bugs that produce the same wrong mean.

def audit(ledger: Ledger, cfg: Config) -> dict:
    by_shard: dict[int, list[Row]] = {}
    for row in ledger.rows.values():
        by_shard.setdefault(row.key.shard, []).append(row)
    revisions = sorted({r.provenance.model_revision for rows in by_shard.values() for r in rows})
    return {
        "rows": len(ledger.rows),
        "shards_with_rows": len(by_shard),
        "missing_shards": cfg.shards - len(by_shard),
        "max_rows_per_shard": max((len(v) for v in by_shard.values()), default=0),
        "committed_revisions": revisions,
        "conflicts": len(ledger.conflicts),
        "rejected_stale": len(ledger.rejected_stale),
        "deduped": ledger.deduped,
    }


def verdict(a: dict) -> str:
    problems = []
    if a["missing_shards"]:
        problems.append("incomplete-batch")
    if a["max_rows_per_shard"] > 1:
        problems.append("denominator-drift")
    if a["conflicts"]:
        problems.append("unresolved-revision-mix")
    if len(a["committed_revisions"]) > 1:
        problems.append("multi-revision-aggregate")
    return ", ".join(problems) if problems else "interpretable"

Read the matrix, not the mean

The baseline property test zeroes out every noise term, so it must pass or the fixture itself is wrong.

# test_properties.py  (pytest)
from borrowed_runtime_sim import Config, audit, run


def test_pinned_gate_is_self_consistent():
    cfg = Config(epoch_gate=True, bump_revision=False, p_alias_flip=0.0,
                 p_preempt=0.0, p_late=0.0, p_duplicate=0.0)
    for seed in range(32):
        a = audit(run(cfg, seed), cfg)
        assert a["max_rows_per_shard"] == 1
        assert a["conflicts"] == 0
        assert len(a["committed_revisions"]) == 1
        assert a["missing_shards"] == 0

Then run the four scenarios and read the verdict field. Exact counts move with the seed, but the violation class should be deterministic given the flags, and that is the thing you are testing.

Scenario	Epoch gate	Revision handling	Designed violation	Required response
A, gate on, pinned	on	one pinned digest	none, except possible missing shards	publish only with missing_shards == 0
B, gate on, revision drifts	on	revision per retry epoch	multi-revision aggregate	split the report by revision before quoting it
C, gate off	off	revision per retry epoch	late row lands after the seal	fix the gate, do not re-weight
D, gate on, alias floats	on	resolved per call	same key, two revisions	re-run the batch, never average the conflict

Scenario D is the one that surprised me. The epoch gate is doing its job, the denominator stays clean, and the batch still is not interpretable, because a redelivered attempt resolved the alias to a different revision. Gating and pinning are two different controls, and you need both.

Where a free runtime fits, and where it does not

This is the part of the review that people skip. A free or preemptible runtime is not a cheaper version of a production endpoint; it is a different lifecycle, and it is only honest to use it where lost or duplicated work is a retry rather than a decision.

Workload	Fits a preemptible free runtime?	Why
Harness, ledger, and schema development	Yes	failures are the input to the test, not an accident
Smoke evals on a pinned revision	Usually	bounded retries; accept and report missing shards
Exploratory prompt sweeps	Yes	a lost row costs a retry, not a release gate
Release thresholds and regression gates	No	you do not control the revision or the completion rate
Latency or cost benchmarks	No	the denominator is not yours to define
Long-lived stateful runtimes	No, by construction	eviction is the contract, not an incident

For the first three rows I have been using MonkeyCode for exactly this reason: it exposes free model access plus a free server option, and the operator states a free allowance on the order of ten million tokens, so the cost of running a fixture matrix like the one above is not the thing that stops me. Check the current terms yourself rather than trusting my summary, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to try the matrix against a live endpoint instead of a stub, that free tier is a reasonable place to start, and the interesting part is not the grades at all; it is whether your ledger refuses the row it should refuse.

What I would change next

1. Move the seal into the same transaction as the commit, so there is no instant where a shard is finalized but not yet fenced against older epochs.
2. Put a revision digest into the job spec itself, then treat any row whose revision differs from the digest as uncommittable rather than as a conflict to investigate.
3. Add a compaction step that converts superseded rows into an explicit superseded_by chain, so a retried shard leaves an audit trail instead of a silent replacement.
4. Emit missing_shards in the report header, and make the report writer refuse to print an aggregate when that count is nonzero.

Tradeoffs I am accepting

The gate costs one extra piece of state per shard and a small amount of latency on the first commit, and it assumes a single-writer control plane. Strict pinning also removes a convenience you may actually want: you can no longer let a retry pick up a fresher model as a free upgrade, because that upgrade is precisely what makes the aggregate uninterpretable. The conflict path adds a manual re-run step that a lenient ledger would have hidden.

Limitations and who should not use this

The simulator models ordering and attribution, not consensus, and it says nothing about how you would fence two control-plane replicas. Do not adopt the epoch gate if you already have an endpoint with pinned revisions, exactly-once delivery, and a completion guarantee, because you would be adding ceremony to a problem you do not have. Do not adopt it for a batch with one shard and one attempt, where the denominator is trivially one. And make sure your aggregate story matches the gate; a gate that protects a per-shard mean is useless if the number you publish is a total over rows.

So here is the question I want answered from someone else's ledger. Which event order breaks your invariant first: a stale epoch committing after the seal, or a fresh attempt resolving an alias to a new revision? And when that happens, does your system reject the row, replay the shard, or compensate the aggregate, because those three answers lead to very different architectures.