Two Clocks, One Ledger: Running an Eval Harness on Free Compute Without Losing Records

A scenario I have reviewed more than once — numbers invented here for illustration, not measured from anyone's deployment — goes like this. A team wants to know whether a new retrieval re-ranker beats the baseline across 320 held-out cases, so they run the sweep on a free server with free model access. Twenty minutes later the report says 78.4% for the candidate and 74.1% for the baseline, and everyone starts writing a changelog entry. Then someone counts the ledger: 312 records, not 320, and the eight missing ones were disproportionately candidate failures. Was the improvement real, or did the denominator quietly get healthier while nobody was looking?

That question is the whole article. Not "how do I get free compute", but how do you run an evaluation on two resources you do not own, whose lifetimes are governed by two independent clocks, and still trust the number you publish? I'll walk through the failure domains, a runnable harness that reproduces them, and the invariants I now require before I believe any free-tier eval verdict.

1. Name the two clocks before you write any code

The first clock is the lease: the free server exists for a window you do not control, and it can end without a graceful shutdown. The second clock is the model budget: free model access is granted per request or per window, and a call can be throttled, rejected, or slow regardless of what the server is doing. These clocks are unsynchronized, and the interesting failures live exactly where they disagree.

If your harness treats "server alive" as "job will finish", you have merged two failure domains into one assumption, which is the same class of mistake as treating a message broker as a database. The invariant I want is simple to state and surprisingly easy to break: every scheduled case must end in exactly one durable terminal record, or one explicit deferral — never in silence. Everything below exists to preserve that invariant across both clocks.

2. Assumptions I want declared up front

1. The free server's local disk and process memory are ephemeral: assume they vanish when the lease ends, with no flush and no warning.
2. The only durable store is the control plane — your laptop, a CI workspace, or a small volume you actually pay for.
3. Model calls can be throttled or rejected at any point, and a rejection is not a failure of the case under test.
4. Clock skew between control plane and server is nonzero, so deadlines must be computed from a monotonic source on one side only.
5. Re-running a case is allowed but must not create a second terminal record for the same logical unit of work.

If assumption 2 is false for you — if you have no durable control plane — stop here and fix that first, because no amount of retry logic substitutes for a place to write the outcome.

3. Where a record can die: the data flow

sequenceDiagram
    participant CP as Control plane (durable)
    participant S as Free server (lease)
    participant M as Free model access (budget)
    CP->>CP: append intent(case, attempt, key)
    CP->>S: dispatch case + deadline
    S->>M: call with remaining budget
    M-->>S: response or throttle
    S->>CP: commit terminal(key, output, passed)
    Note over S: lease may end after response, before commit
    CP->>CP: summary = terminal records only

Read the note in the middle carefully, because that is the counterexample the rest of this design defends against. A response that was computed but never committed is worse than a call that never happened: it burns budget, produces no evidence, and makes the surviving population non-random. Do you see why the summary arrow points at the control plane and not at the server? The summarizer must never need to ask a leased machine what happened.

4. Step 1 — Write the intent before you call anything

The harness below is a model, not a benchmark: it calls no real model, and its latencies are synthetic. Replace model_call with your client and the structure still holds. Everything is Python 3.11 standard library.

# two_clock_harness.py — models a server lease and a model budget separately.
import argparse, hashlib, json, os, pathlib, random, time

LEDGER = pathlib.Path("ledger.jsonl")
SAFETY_MARGIN_S = 8.0     # reserved for commit + teardown, tune per workload
MODEL_BUDGET_S = 25.0     # your assumption, not a product claim

def append(record: dict) -> None:
    line = json.dumps(record, sort_keys=True) + "\n"
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(line); fh.flush(); os.fsync(fh.fileno())

def case_key(case_id: str) -> str:
    return hashlib.sha256(case_id.encode()).hexdigest()[:16]

def read_ledger() -> list[dict]:
    if not LEDGER.exists():
        return []
    return [json.loads(l) for l in LEDGER.read_text().splitlines() if l.strip()]

def committed() -> dict[str, dict]:
    return {r["key"]: r for r in read_ledger() if r["state"] == "terminal"}

def model_call(case: dict, budget_s: float) -> dict:
    latency_ms = random.uniform(50.0, budget_s * 800.0)
    if latency_ms > budget_s * 1000.0:
        raise TimeoutError("model budget exhausted")
    time.sleep(0.001)  # scaled down so the demo finishes quickly
    out = case["prompt"][::-1]
    return {"output": out, "passed": len(out) % 3 != 0, "latency_ms": round(latency_ms, 1)}

The intent record is the write-ahead half of the protocol: it is durable, it carries the idempotency key, and it says "I am about to spend money on this case". Appending it first is what makes a crash visible as intent_without_commit instead of invisible as a missing row.

5. Step 2 — Derive the deadline, never inherit it

def run_case(case: dict, lease_deadline: float, inject: str | None) -> str:
    key = case_key(case["id"])
    if lease_deadline - time.monotonic() <= SAFETY_MARGIN_S:
        append({"key": key, "case": case["id"], "state": "deferred"})
        return "deferred"
    append({"key": key, "case": case["id"], "state": "intent", "attempt": 1})
    if inject == "kill_before_model":
        raise SystemExit("simulated lease expiry before model call")
    result = model_call(case, MODEL_BUDGET_S)
    if inject == "kill_after_model":
        raise SystemExit("simulated lease expiry after model response")
    append({"key": key, "case": case["id"], "state": "terminal", **result})
    return "ok"

The refusal branch matters as much as the commit branch. A case that cannot finish inside lease_deadline - SAFETY_MARGIN_S should be deferred by name, not started optimistically and abandoned, because a named deferral keeps the denominator honest while an abandonment corrupts it. What safety margin is right for your workload? Measure your p99 commit time and multiply generously; the margin is cheap and the bias is not.

6. Step 3 — Make resume a replay of uncommitted intents

def resume(cases: list[dict], lease_deadline: float, inject: str | None) -> None:
    done = committed()
    pending = [c for c in cases if case_key(c["id"]) not in done]
    for case in pending:
        run_case(case, lease_deadline, inject)

Resume keys off the durable ledger, so it is safe to run on a brand-new server with nothing in common with the one that died. The replay is idempotent at the ledger level because terminal records are keyed by content hash, but it is only cheap if your case execution is side-effect free — a re-ranker call is, an agent that writes files is not. Which of your eval cases mutate the world, and did you notice that before you made replay your recovery strategy?

7. Step 4 — Inject the three kills and read the denominator

rm -f ledger.jsonl
python two_clock_harness.py --cases 40 --lease-s 30 --inject none
python two_clock_harness.py --cases 40 --lease-s 30 --inject kill_after_model --resume
python two_clock_harness.py --cases 40 --lease-s 30 --inject kill_before_model --resume

Always report the denominator explicitly, and never let the scheduled count stand in for it:

scheduled = len(intents | deferred)
pass_rate = passed / len(terminal) if terminal else None
# report {scheduled, terminal, deferred, intent_without_commit, pass_rate}

The properties I actually assert in CI, as plain assert statements over the ledger file:

1. Totality: terminal | deferred == scheduled_intents after resume, with no key in neither set.
2. Uniqueness: no key appears in more than one terminal record.
3. Monotonic commit: a terminal record for key k is never followed by an intent for k.
4. Denominator stability: pass rate computed from terminal records drifts by less than 0.5pp between an uninterrupted run and a killed-then-resumed run of the same cases.
5. Deferral visibility: every lease-driven refusal produces a deferred record naming the case, never a retry loop.

Property 4 is the acceptance rule, and it is the one people skip. If your killed-and-resumed pass rate differs from the clean run by more than your tolerance, you have measured the failure of your harness, not the quality of your candidate. Run the injection twenty times before you trust a verdict — this is precisely the kind of repetition that free compute makes affordable.

8. Tradeoffs, stated as tradeoffs

Choice	Gain	Cost	Reject when
Large SAFETY_MARGIN_S	Fewer half-committed cases	Lower lease utilisation	Lease windows are shorter than a single call
Intent-before-call	Crashes stay visible	One extra durable write per case	Write path dominates runtime
Resume over rerun-all	Preserves budget	Replay must be idempotent	Cases mutate external state
Terminal-only denominator	No silent bias	Smaller n, wider intervals	You need every scheduled case counted
Deferral instead of retry	Bounded, honest accounting	Manual re-dispatch	You own the server and can just wait

No row in that table is universally right, and I would push back on anyone who presents one as a rule. The table exists so that the choice is written down next to the evidence it trades against.

9. Where free model access and a free server genuinely fit

This is the workflow where I have found the combination useful rather than decorative. MonkeyCode offers free model access and a free server option, which is exactly the shape of resource this harness is designed for: short-lived compute, budgeted inference, and a control plane you keep. Because the harness never trusts the server, an unexpected lease end costs one resume invocation instead of a corrupted report — and free access makes the twenty-run failure-injection loop above a default rather than a luxury.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats. Free allowances and machine specifications change, so read the current terms before you size a sweep rather than trusting any number you saw in a post — including this one, which deliberately quotes none. And confirm the project's licence and data-handling terms yourself before you send it evaluation data you would not want retained.

10. Who should not use this approach

Skip it if you need a durable server-local cache, if your cases take longer than a single lease window and cannot be checkpointed, if you require an availability commitment you can put in a contract, or if your evaluation data is regulated and the provider's retention terms are not something you can accept. This is a design for cheap, repeatable measurement under uncertainty; it is not a production control plane and it will not become one by adding retries.

11. What I would change next, and the question I'd ask you

The next change I would make is not more retries but a pre-flight feasibility probe: one throwaway call that measures p99 latency, multiplied by the batch size, compared against the lease window before any case is dispatched. That converts a whole class of late-run deferrals into an up-front refusal to start, which is far easier to reason about. The second change is a per-case budget split, so one pathologically slow case cannot consume the margin that protects the other thirty-nine.

So here is the counterexample question I will leave you with. Which event order breaks your invariant — a lease that ends after the response but before the commit, a throttle that arrives after you already recorded an intent, or a resume that replays a case whose side effects are not idempotent? And when it happens, should your system reject the case, replay it, or compensate for it? Write the answer down before you look at your pass rate, because after you look, you will be tempted to explain it away.