A recovered retry is a transport event, not evidence that the model became cleaner on the same case. If the harness keeps only the surviving completion, the first failure disappears and the scorecard starts describing luck. The practical fix is an attempt ledger that freezes the first graded outcome before any later call is allowed to speak. Developers who already pin fixtures still miss this leak when a client library retries and returns only the last body.
Retry policy belongs in the client, while score ownership belongs in the harness, and those two decisions should not share a variable. A retry can be the right client behavior when a socket drops, and the wrong measurement behavior when that recovery replaces the row. The analogy is a lab notebook that keeps every titration, rather than a whiteboard that erases the failed reading after the beaker is refilled. Once the failed reading is gone, a weekly diff can report stability while the first-attempt miss rate climbs without a trace.
The ledger does not forbid a second call, and it does not treat every dropped connection as a content failure. It only refuses to let the second call occupy the slot that the first call already earned. Availability stays on an availability series, while content quality stays on the earliest generation the grader actually scored. Mixing those series is how a noisy afternoon becomes a false improvement by the time the report is exported.
Freeze ordinal one
The workflow starts before the model is called, with a case identifier, a fixture hash, and an attempt ordinal set to one. Each response, including a timeout or a transport error, appends an immutable row that names the fault class and the ordinal. A content row is graded once, then marked frozen, so a later pass can be stored beside it without becoming the published score. The scorecard function reads only ordinal one when that row is graded content, and it ignores every later success for that case.
Transport and timeout rows are excluded from the quality mean, because a missing completion is not a wrong completion. A later success after a timeout is stored for diagnosis, but it never enters the quality mean for that case. If every case needed a second attempt, a naive last success mean can still look fine while the ledger mean stays empty. That split is the regression signal: availability rising, quality absent or frozen, and no silent overwrite between the series.
Proposal you can run locally
The following Python is a proposal for that split, labeled unexecuted, and it is not a measured result from any hosted run. It records attempts, classifies faults, and publishes a quality score only when ordinal one is graded content. A tiny assertion then fails the build when a later perfect score would otherwise replace a weaker first reading. You can drop the module into a scratch repository without any network access, then point a client at it later.
# Attempt ledger proposal.
# Unexecuted example: not a measured run.
# Targets Python 3.11+ for builtin generics.
from dataclasses import dataclass, field
from enum import Enum
class Fault(str, Enum):
NONE = 'none'
TRANSPORT = 'transport'
TIMEOUT = 'timeout'
@dataclass(frozen=True)
class Attempt:
case_id: str
ordinal: int
fault: Fault
score: float | None
graded: bool
@dataclass
class Ledger:
rows: list[Attempt] = field(default_factory=list)
def record(self, row: Attempt) -> None:
clash = any(
r.case_id == row.case_id and r.ordinal == row.ordinal
for r in self.rows
)
if clash:
raise ValueError(f'duplicate ordinal for {row.case_id}')
self.rows.append(row)
def quality_score(self, case_id: str) -> float | None:
first = next(
(r for r in self.rows if r.case_id == case_id and r.ordinal == 1),
None,
)
if first is None or first.fault != Fault.NONE or not first.graded:
return None
return first.score
def availability_faults(self, case_id: str) -> int:
return sum(
1
for r in self.rows
if r.case_id == case_id and r.fault in (Fault.TRANSPORT, Fault.TIMEOUT)
)
def naive_last_success(rows: list[Attempt], case_id: str) -> float | None:
scored = [r.score for r in rows if r.case_id == case_id and r.score is not None]
return scored[-1] if scored else None
def test_retry_pass_does_not_replace_first_score() -> None:
ledger = Ledger()
ledger.record(Attempt('case-7', 1, Fault.NONE, 0.25, True))
ledger.record(Attempt('case-7', 2, Fault.NONE, 1.0, True))
assert ledger.quality_score('case-7') == 0.25
assert naive_last_success(ledger.rows, 'case-7') == 1.0
def test_timeout_retry_is_not_promoted() -> None:
ledger = Ledger()
ledger.record(Attempt('case-8', 1, Fault.TIMEOUT, None, False))
ledger.record(Attempt('case-8', 2, Fault.NONE, 1.0, True))
assert ledger.quality_score('case-8') is None
assert ledger.availability_faults('case-8') == 1
assert naive_last_success(ledger.rows, 'case-8') == 1.0
def demo() -> None:
ledger = Ledger()
ledger.record(Attempt('case-7', 1, Fault.NONE, 0.25, True))
ledger.record(Attempt('case-7', 2, Fault.NONE, 1.0, True))
ledger.record(Attempt('case-8', 1, Fault.TIMEOUT, None, False))
ledger.record(Attempt('case-8', 2, Fault.NONE, 1.0, True))
print('frozen', ledger.quality_score('case-7'), 'naive', naive_last_success(ledger.rows, 'case-7'))
print('timeout_case', ledger.quality_score('case-8'), 'naive', naive_last_success(ledger.rows, 'case-8'))
print('faults', ledger.availability_faults('case-8'))
if __name__ == '__main__':
demo()
Run the two local commands below before you trust any narrative about what the ledger actually protects. The commands assume pytest is installed beside the file, and neither command calls a model or a remote server. A passing test only shows that the scoring rule holds on the fixture data baked into the file. It does not show that your production client writes the ordinal before the retry policy fires.
python -m pytest attempt_ledger.py -q
python attempt_ledger.py
If the proposal behaves as written, the script prints a frozen 0.25 beside a naive 1.0, then None beside another naive 1.0. Those two contrasts are the bug the ledger is meant to make obvious, without any claim that a live model produced them. Treat a different printout as a failed transcription of the snippet, not as a finding about a hosted model.
Wire the client so the ordinal increments inside the retry loop, not after the loop returns a survivor. If the increment happens too late, the ledger will honestly store a lie, and the unit test will stay green. Log the fault class from the exception type you actually catch, rather than from a string search on the message body. Message text drifts across client versions, while a typed exception remains a stable key for the availability series.
What a free runner does not prove
A local pytest run proves the rule, but it does not exercise the path where a hosted runner times out and the SDK retries. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that client path without a local accelerator reservation. Those two availability claims are the only product facts used here, and they are not a quota, a hardware specification, or an uptime promise.
Confirm the current terms yourself before a nightly job depends on either option, because this draft does not pin duration or capacity. The ledger still matters on a free server, since shared capacity makes timeouts more likely and automatic retries more tempting. Treat a timeout on that server as an availability row, then decide in a separate policy whether a second attempt is worth spending. Do not let the second attempt's score flow into the column that a release review will later treat as truth.
A free endpoint is a reasonable place to rehearse the client, and a poor place to publish a comparative leaderboard. Nothing in this workflow identifies a model name, a token allotment, or a machine size, because those details were not verified for this draft. If a later note records which endpoint answered, store that note on the attempt row and leave the quality column numeric. Readers can join the note when they investigate a timeout, without letting the note rewrite the frozen mean.
The approach fails when the client swallows the first exception and returns a single response object with no attempt count. It also fails when the grader itself calls the model again, because that hidden call never receives an ordinal from the ledger. Teams that cannot retain raw attempt rows should not adopt the scorecard, since the frozen score cannot be audited after the job ends. Safety, medical, and legal evaluations should not use this sketch, because a frozen first score is not a hazard analysis.
Anyone who needs a contractual service level should not park a release gate on a free server option. Anyone who wants a benchmark against named models should stop here, because this article deliberately names none. The sample stores a numeric score so the control flow stays visible, and that placeholder is too weak for open-ended prose. Swap in your own grader only after the ordinal rule is tested, or the new grader will hide the leak you meant to catch.
A weekly report can print three numbers side by side without pretending that they describe a single quality metric. Those numbers are the first-attempt content mean, the availability fault rate, and the count of retry passes that were refused promotion. If the third number grows while the first stays flat, retries are doing public relations for a prompt that did not improve. That pattern is the silent regression this ledger is built to catch, and it leaves a row you can diff next week.
If you want a hosted place to exercise the retry path, MonkeyCode offers free model access and a free server option. Read the live terms before you depend on either, and keep the refusal rule even when the first hosted call succeeds. A dry run that cannot show its ordinal log is not evidence, even if the hosted response looks clean on the screen.
Top comments (0)