Interrupted Runs Belong in the Denominator

A benchmark score that quietly drops interrupted runs is not a conservative score; it is a score whose hardest cases were deleted before anyone counted them. Quota ceilings and short-lived servers turn interruption into a routine outcome rather than an exotic failure, so the reporting has to absorb it. The practical fix is bookkeeping: record how each run ended, count truncation as a failure in the headline number, and publish the completed-only rate as a secondary diagnostic. Everything below is a method, not a product review, and it stays useful if the tooling underneath is swapped out.

Truncation is not randomly distributed across a task set, which is exactly why dropping it distorts the result. Tasks that consume many tokens or trigger retries are the ones most likely to hit a ceiling, and those are typically the tasks a model finds difficult. Removing them raises the average in the same way that timing a runner's laps only when the runner finished would raise the recorded pace. A published number should therefore answer two questions at once: how often did work finish, and how did the finished work perform.

The artifact for this is a run ledger with four end states, written at the moment a run stops rather than reconstructed later from a log grep. COMPLETE means the harness reached its own termination condition; QUOTA_CUT and RESTART_CUT mean an external boundary or process lifetime ended the attempt; ERROR covers everything the harness itself could not classify. Keeping quota and restart separate matters, because one is a budget decision and the other is an infrastructure event.

# ledger.py — record how a run ended, not just whether it passed
from dataclasses import dataclass
from enum import Enum

class End(Enum):
    COMPLETE = "complete"
    QUOTA_CUT = "quota_cut"
    RESTART_CUT = "restart_cut"
    ERROR = "error"

@dataclass(frozen=True)
class Run:
    task_id: str
    seed: int
    ended: End
    passed: bool
    tokens_used: int
    run_id: str
    boot_id: str

Scoring reads that ledger and emits both rates plus the gap between them, which is the truncation exposure. A simple reporting rule keeps the headline honest without hiding the completed-only view: when truncation exceeds a pre-registered threshold, the strict score is the headline and the observed score is context. When truncation sits below the threshold, the observed score can lead, but the truncation rate still ships alongside it.

# score.py
from ledger import End

TRUNCATIONS = {End.QUOTA_CUT, End.RESTART_CUT}

def report(runs, threshold=0.10):
    n = len(runs)
    done = [r for r in runs if r.ended is End.COMPLETE]
    passed = [r for r in done if r.passed]
    truncated = [r for r in runs if r.ended in TRUNCATIONS]

    strict = len(passed) / n
    observed = len(passed) / len(done) if done else float("nan")
    rate = len(truncated) / n
    headline = strict if rate > threshold else observed

    return {
        "runs": n,
        "truncation_rate": round(rate, 3),
        "strict_score": round(strict, 3),
        "observed_score": round(observed, 3),
        "headline": round(headline, 3),
        "headline_basis": "strict" if rate > threshold else "observed",
    }

The output below is synthetic, produced from hand-written run logs to show the report shape; it is not a measurement of any product or model. Its only purpose is to show what an auditable line looks like when truncation crosses the threshold and the strict number takes over the headline.

$ python score.py runlog.jsonl --threshold 0.10
runs: 120 | truncation_rate: 0.183 | strict: 0.542 | observed: 0.663
headline: 0.542 (strict) — truncation above the 10% threshold

Interruption is also an opportunity to measure recovery, which is a separate quantity and should never be folded into the pass rate. A resume pass replays each truncated run from its last checkpoint under the same token budget envelope and records whether it eventually finished and passed. The recovery ratio is passed-resumed divided by resumed-attempts, and it answers a question the pass rate cannot: whether the harness loses work permanently or merely pauses it.

python -m harness.run \
  --tasks holdout.jsonl \
  --seed 7 \
  --token-budget 60000 \
  --checkpoint ./ckpt \
  --emit runlog.jsonl

python -m harness.resume \
  --runlog runlog.jsonl \
  --checkpoint ./ckpt \
  --emit runlog.resumed.jsonl

On short-lived machines, restart attribution needs one extra field, because a run that dies with the process leaves no trace beyond a missing final line. Writing the kernel boot identifier and the instance start time into the run record makes restarts distinguishable from quota stops during analysis. Injecting the boot id and start time once, at harness startup, is enough to attribute every later RESTART_CUT to a specific machine lifetime.

BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
STARTED=$(date -Is)
printf '{"run_id":"%s","boot_id":"%s","started":"%s"}\n' \
  "$RUN_ID" "$BOOT_ID" "$STARTED" >> /tmp/harness.env.json

This is where an environment with free model access and a free server option becomes genuinely useful for methodology work, rather than merely cheap. MonkeyCode is an open-source project whose free model access and free server option let a team rehearse this ledger on real quota and restart events before committing budget to a paid run. The operator advertises a free token allowance for evaluation; quotas, model availability, and server terms change, so confirm the current figures on the project's own pages rather than trusting a number quoted in an article.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A short decision table keeps the method from being applied where it adds nothing but paperwork. Teams with guaranteed uninterrupted capacity and no resume path should still record end states, but they can report a single rate and skip the resume pass entirely. Teams comparing two configurations must use the same threshold and the same budget envelope, or the comparison measures reporting choices instead of systems.

Situation	Headline metric	Resume pass needed
Quota-limited or ephemeral runs	Strict score with truncation rate	Yes, to separate pause from loss
Interruption rate under 1%	Observed score with truncation rate	Optional
Guaranteed capacity, no restarts	Single rate, end states still logged	No

The limitations are real and worth stating plainly. This bookkeeping cannot repair a harness that fails to write a ledger line when it dies, and it adds a small amount of engineering before the first useful number appears. It also says nothing about statistical confidence, which requires paired runs and reported intervals rather than a single strict figure. Anyone who only needs a rough internal signal, and who accepts that the signal is optimistic, should skip the strict denominator and say so in the memo.

The test of a published score is whether it survives one plain follow-up: what happened to the runs that never finished. A ledger that answers that question with counts, machine identifiers, and a resume result turns an awkward footnote into part of the method. That is the difference between a number that reports on a system and a number that advertises one.