Make Token Spend an Invariant Before a Preemptible Free Tier Replays Your Eval Batch

It was 03:12 and shard three of four had already finished 78% of its cases when the worker simply vanished. The driver had no heartbeat, no exit code, and no way to tell whether the last forty graded cases had ever been written anywhere. When the batch finally reported complete, my spend ledger showed roughly 1.4x the tokens I had planned, because two attempts had committed output for the same shard. Would your evaluation harness notice that, or would it just average the two answers and hand you a number you trust?

This is an architecture review of that failure, not a tutorial about model quality. I run cheap classification and grading passes on free model access, and I run the shard workers on a free server option, because both are exactly where prototyping belongs. The catch is that free capacity is preemptible by definition, so the correctness of your results depends on how you fence attempts, not on how good the model is.

Assumptions I am declaring up front

Before any invariant makes sense, the constraint set has to be explicit, otherwise the design is unfalsifiable.

1. Shard workers can be reclaimed without notice, and no shutdown hook is guaranteed to run.
2. Result delivery is at-least-once, so a late commit can arrive after a replacement attempt has already finished.
3. There is no shared clock between the driver, the workers, and the artifact store that is trustworthy enough for ordering.
4. Model calls dominate cost, so the ledger must be denominated in tokens rather than in wall-clock seconds.
5. Shards are independent units of work, and each shard must produce exactly one accepted artifact per run.
6. The free capacity I use is MonkeyCode's free model access for the cheap grading stage and its free server option for the shard workers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states a free token allowance on the order of ten million tokens; verify the current figure on the project page before you size a batch around it, because I am not treating that number as a permanent contract.

That last point matters more than it looks. If you assume a fixed allowance and the allowance changes, your batch does not fail loudly — it fails quietly by truncating, which is worse.

The event order that broke my accounting

Here is the counterexample in full, because the bug is only visible in the interleaving.

1. Driver reserves budget for (run=7, shard=3, epoch=0) and dispatches.
2. Worker 3 grades 39 of 50 cases and writes a checkpoint after case 30.
3. The host reclaims worker 3; the lease expires with no heartbeat renewal.
4. Driver decides the attempt is dead and dispatches (run=7, shard=3, epoch=1).
5. Worker 3′ grades all 50 cases and commits an artifact for shard 3.
6. The original worker resumes on a different host, finishes its last 11 cases, and commits a second artifact for shard 3.

Nothing in that sequence is a network partition, a Byzantine node, or an exotic clock problem. It is just a preempted process that came back, and any harness that keys results by shard_id alone will accept both writes. The invariant I was missing is simple to state: for a given run, the committed token spend per shard is a function of the highest epoch that produced an accepted artifact, and lower epochs must never commit.

Model: lease, epoch, and a monotonic spend ledger

I model the driver as a lease grantor and the ledger as a monotonic reservation table keyed by attempt identity.

sequenceDiagram
    participant D as Driver
    participant W as Shard worker (free tier)
    participant L as Spend ledger
    participant S as Artifact store
    D->>L: reserve(run=7, shard=3, epoch=0)
    L-->>D: RESERVED
    D->>W: dispatch(lease, input_digest)
    W->>W: grade cases, checkpoint every 30
    Note over W: preempted at 39/50
    D->>W: re-dispatch epoch=1
    W->>S: PUT artifact(epoch=1, input_digest)
    S-->>D: 201 Created
    D->>L: commit(run=7, shard=3, epoch=1)
    Note over W: epoch=0 host resumes
    W->>L: commit(run=7, shard=3, epoch=0)
    L-->>W: REJECTED (stale epoch)

The important detail is that the ledger rejects on two different grounds: a key that was never reserved, and a key whose epoch is below the highest committed epoch for that shard. Those are different failure classes and they deserve different counters, otherwise you cannot tell a replay from a forged attempt.

Step-by-step: building the fence

1. Key every attempt by (run_id, shard_id, epoch)

The epoch is granted by the driver, never by the worker, because a worker that can increment its own epoch can always win. Reservation happens before dispatch, so a budget overshoot is prevented rather than detected.

2. Reserve before the call and commit only after the artifact digest exists

A reservation is a promise, and a commit is a receipt, and conflating them lets a crashed attempt consume budget forever. I keep reservations with a lease expiry so a dead attempt's reservation is reclaimable, but the committed total is append-only.

3. Make every checkpoint carry the input digest

A checkpoint that only records case_index is not resumable, because the fixture may have changed between attempts. Storing the digest of the input slice means a resumed attempt can prove it is continuing the same work, and if it cannot, it restarts the shard instead of mixing two fixture versions.

4. Gate resumption on lease validity, not on liveness

Liveness checks lie under preemption, because a paused host looks alive from the outside. A worker should refuse to write when its lease has expired, even if it has a completed result in memory, and that refusal is the whole point.

A minimal fixture you can adapt

This is a reference sketch, not a benchmark; I have not executed this exact file, so treat it as a fixture to adapt and pin down with your own tests.

"""preempt_sim.py - reference fence for preemptible eval shards (sketch)."""
from dataclasses import dataclass, field

TOKENS_PER_CASE = 37          # declared assumption, not measured
CASES_PER_SHARD = 50
SHARDS = 4
BUDGET_TOKENS = SHARDS * CASES_PER_SHARD * TOKENS_PER_CASE  # 7,400


@dataclass
class SpendLedger:
    reserved: dict = field(default_factory=dict)   # key -> tokens
    committed: dict = field(default_factory=dict)  # key -> tokens
    highest_epoch: dict = field(default_factory=dict)  # shard -> epoch

    def reserve(self, run, shard, epoch, tokens):
        key = (run, shard, epoch)
        if key in self.reserved or key in self.committed:
            return False                  # replay of the same attempt
        if self.spent() + sum(self.reserved.values()) + tokens > BUDGET_TOKENS:
            return False                  # would overshoot the run ceiling
        self.reserved[key] = tokens
        return True

    def commit(self, run, shard, epoch, tokens):
        key = (run, shard, epoch)
        if key in self.committed:
            return "ALREADY_COMMITTED"
        if epoch < self.highest_epoch.get(shard, -1):
            return "STALE_EPOCH"           # a resurrected attempt lost the race
        if self.reserved.get(key) != tokens:
            return "UNRESERVED_SPEND"      # partial or forged attempt
        del self.reserved[key]
        self.committed[key] = tokens
        self.highest_epoch[shard] = epoch
        return "COMMITTED"

    def spent(self):
        return sum(self.committed.values())


def resume_is_safe(checkpoint, input_digest, lease_valid):
    if checkpoint is None:
        return True                        # nothing to trust yet
    if not lease_valid:
        return False                       # expired lease must not write
    return checkpoint["input_digest"] == input_digest

Drive it with a deterministic event queue: reserve, dispatch, deliver, preempt, re-dispatch, and then deliver a late commit from the dead epoch. If your ledger returns STALE_EPOCH on that last event, the fence holds for that interleaving.

Failure injection matrix

Injected failure	Observable symptom	Invariant attacked	Detection	Recovery action
Preemption at 78%	Lease expires, no exit code	Exactly-one artifact per shard	Heartbeat gap plus missing commit	Re-dispatch with epoch + 1
Duplicate delivery	Same artifact arrives twice	Spend is idempotent	ALREADY_COMMITTED counter	Reject, do not re-bill
Resurrected worker	Late commit from old epoch	Monotonic epoch per shard	STALE_EPOCH counter	Reject, log the host identity
Fixture change mid-run	Checkpoint digest mismatch	Result belongs to one input version	Digest compare on resume	Restart the shard, never merge
Clock skew of 90s	Lease appears valid after expiry	Lease validity gates writes	Driver-issued expiry, not worker clock	Shorten lease, renew earlier

Tradeoffs I would accept, and one I would not

Choice	What you gain	What you pay	When to pick it
Driver-issued epochs	Single ordering authority	Driver becomes a coordination point	Always, for at-least-once workers
Checkpoint every 30 cases	Cheap resume after preemption	Slightly more store writes	Shards longer than a few minutes
Reserve-before-dispatch	Budget cannot overshoot	Reservations must expire cleanly	Any tier with hard free limits
Restart on digest mismatch	No mixed-version artifacts	Wasted work	Fixtures change during a run
Worker self-epochs	Fewer round trips	Two writers can both win	Never, in my experience

Limitations and who should not use this

The overhead is real, and a single-worker batch that finishes in ninety seconds does not need a ledger at all. If your results are advisory rather than decision-bearing, a duplicate artifact is annoying but survivable, so skip the machinery. If your preemption rate is near zero on a dedicated host, the lease renewal traffic buys you very little. This design also assumes shards are independent, and a harness with cross-shard aggregation needs a second fence at the reduce step. Finally, my load model above is declared, not benchmarked, so replace those constants before you quote any number from it.

What I would change next

Two changes are on my list. First, I would move from shard-level epochs to token-level reservations inside a shard, so a preemption at 78% does not force a full re-grade of the remaining 22%. Second, I would add an explicit compensation path for the case where a stale artifact was already accepted by a downstream reducer, since rejection at the ledger is too late at that point. Both changes trade throughput for a narrower blast radius, and I would rather pay that than explain a duplicated grade in a release decision.

The acceptance rule I use is blunt: over 500 simulated preemptions, committed spend must stay under the run ceiling, each shard must end with exactly one accepted artifact, and every rejection must carry a classified reason. Which event order breaks your own ledger — a resurrected worker, a duplicate delivery, or a fixture that changed underneath a resume — and should your system reject, replay, or compensate when it does?

If you want to prototype this harness without paying for the first thousand attempts, MonkeyCode's free model access and free server option are a reasonable place to run the shard workers while you build the fence.