Stop on the Ledger: A Take-Home for Quota-Aware Clients

A strong inference client stops on a local ledger before a free endpoint spends another token. Hiring panels that score only the final completion miss the failure that matters once a shared pool runs dry. This packet grades the stop condition, the persisted usage record, and the typed outage path as one unit. The model answer is only evidence that wiring worked, and it is not the score itself.

What the packet measures

Free model access removes a credit-card step, and a free server removes a cluster step, yet neither gift removes a budget. A candidate who treats either path as unlimited will retry through exhaustion and hide the real cost. The exercise therefore asks for a small client that can target an operator-provided free endpoint without assuming that access lasts forever. Exact quotas, model identifiers, and server duration stay outside the packet and must be read from current project documentation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow below still stands if that product name is replaced by any compatible endpoint offering a bounded free path. Readers should confirm live limits in the current repository notes before they point a runner at that host. A missing limit in those notes is a reason to stay on the local stub, not a reason to guess a quota.

The assignment

The candidate receives a pinned fixture, a numeric budget, and a base URL that is supplied only at runtime. The client must refuse a call when the remaining ledger cannot cover that row's declared ceiling. After each accepted call, the client must persist provider-reported usage and subtract that total from the ledger. A transport failure, an HTTP 429, or a missing usage field must become a typed result rather than a silent retry.

Constraints that keep the review reproducible:

• The fixture file remains immutable, and the reviewer rechecks its checksum before accepting the recorded run.
• Secrets and endpoint keys arrive only through environment variables, never through the fixture or the ledger file.
• The default budget is small enough that the third fixture row must stop before its network call is built.
• Retries stay forbidden unless a documented transient code returns and the ledger still covers the full ceiling.
• The process exits non-zero when a required row is skipped for a reason other than budget or typed outage.

Command surface

The expected command surface stays narrow so grading does not depend on a web framework or a hosted runner. A reviewer should run the same three commands against the local stub before considering any live host. The live path stays optional because a free server can be absent, throttled, or catalog-shifted on review day. That optionality is a deliberate feature of the packet, not an incomplete part of the assignment.

python -m budget_gate run \
  --fixture fixtures/prompts.jsonl \
  --ledger var/ledger.json \
  --budget-tokens 180 \
  --base-url http://127.0.0.1:8080/v1 \
  --model "$MODEL_ID"

Candidates must pass the model identifier from current documentation into the flag, rather than hardcoding it. A hardcoded name fails this packet even when the local stub still returns a plain HTTP 200. The variable exists so a later smoke test can follow a catalog change without editing the client.

Row contract

Each fixture row maps to one action, one ledger effect, and one exit class, with no blended statuses allowed. The table is the contract a second reviewer uses when the narrative in the README disagrees with the code. A submission that prints a friendly message but writes a different status still fails the row.

Observed condition	Required action	Ledger effect	Exit class
Remaining tokens below the row ceiling	Do not dispatch	Unchanged	budget_stop
HTTP 200 with a total_tokens field	Record the completion id and usage	Subtract the reported total	ok
HTTP 200 without usage	Do not trust a local estimate	Unchanged	usage_missing
HTTP 429 or a connection reset	No retry loop; one retry only if policy and ledger both allow it	Unchanged until a later success	upstream_unavailable
HTTP 401 or HTTP 403	Stop the remaining batch	Unchanged	auth_rejected

Fixture arithmetic

The pinned fixture uses declared ceilings so the gate does not depend on a tokenizer package. Row one allows 40 completion tokens and a prompt ceiling of 30, for a gate value of 70. Row two repeats that same shape, and row three uses a gate value of 120. With a budget of 180, two stub calls that report 40 tokens each leave 100 remaining, which sits below 120, so the third row must stop locally.

{"id":"r1","prompt":"Summarize the ledger rule in one sentence.","max_tokens":40,"prompt_ceiling":30}
{"id":"r2","prompt":"Name three typed stop reasons without calling a network.","max_tokens":40,"prompt_ceiling":30}
{"id":"r3","prompt":"Explain why a missing usage field must not decrement spend.","max_tokens":80,"prompt_ceiling":40}

Proposal code for the gate

The module below is a proposal for reviewers and candidates, not a recorded benchmark and not an executed trace. It shows the pre-dispatch gate, the ledger write, and the typed results a panel should be able to find. Another language is acceptable when the same decisions remain visible in code and in the ledger file. Streaming responses, tool calls, and concurrent writers stay intentionally out of scope for this first packet.

#!/usr/bin/env python3
'''Proposal: local budget gate for a fixture runner. Not an executed benchmark.'''

from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

@dataclass(frozen=True)
class RowResult:
    row_id: str
    status: str
    spent: int

def load_ledger(path: Path, budget: int) -> dict[str, Any]:
    if path.exists():
        return json.loads(path.read_text(encoding='utf-8'))
    return {'budget': budget, 'spent': 0, 'events': []}

def remaining(ledger: dict[str, Any]) -> int:
    return int(ledger['budget']) - int(ledger['spent'])

def gate(ledger: dict[str, Any], ceiling: int) -> str | None:
    if ceiling < 1:
        return 'bad_ceiling'
    if remaining(ledger) < ceiling:
        return 'budget_stop'
    return None

def apply_usage(ledger: dict[str, Any], row_id: str, usage: dict[str, Any]) -> None:
    total = int(usage['total_tokens'])
    ledger['spent'] = int(ledger['spent']) + total
    ledger['events'].append({'row_id': row_id, 'total_tokens': total})

def post_json(base_url: str, payload: dict[str, Any], api_key: str) -> dict[str, Any]:
    body = json.dumps(payload).encode('utf-8')
    request = Request(
        base_url.rstrip('/') + '/chat/completions',
        data=body,
        headers={
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + api_key,
        },
        method='POST',
    )
    with urlopen(request, timeout=20) as response:
        return json.loads(response.read().decode('utf-8'))

def run_row(ledger, row, base_url, model, api_key) -> RowResult:
    ceiling = int(row['max_tokens']) + int(row.get('prompt_ceiling', 0))
    blocked = gate(ledger, ceiling)
    if blocked:
        return RowResult(row['id'], blocked, 0)
    payload = {
        'model': model,
        'messages': [{'role': 'user', 'content': row['prompt']}],
        'max_tokens': int(row['max_tokens']),
    }
    try:
        data = post_json(base_url, payload, api_key)
    except HTTPError as exc:
        status = 'auth_rejected' if exc.code in (401, 403) else 'upstream_unavailable'
        return RowResult(row['id'], status, 0)
    except URLError:
        return RowResult(row['id'], 'upstream_unavailable', 0)
    usage = data.get('usage') or {}
    if 'total_tokens' not in usage:
        return RowResult(row['id'], 'usage_missing', 0)
    apply_usage(ledger, row['id'], usage)
    return RowResult(row['id'], 'ok', int(usage['total_tokens']))

def save_ledger(path: Path, ledger: dict[str, Any]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(ledger, indent=2), encoding='utf-8')

Local stub and unexecuted check

A local stub returns scripted usage so the third row crosses the cap without touching a shared pool. The stub belongs in the packet because a live free server may be down during the review window. Separating those two paths is the teaching point, not a temporary workaround for missing credentials.

#!/usr/bin/env python3
'''Local stub proposal. Returns usage until FAIL_AFTER, then HTTP 429.'''

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json

FAIL_AFTER = 2
SEEN = {'n': 0}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        length = int(self.headers.get('Content-Length', '0'))
        payload = json.loads(self.rfile.read(length) or b'{}')
        SEEN['n'] += 1
        if SEEN['n'] > FAIL_AFTER:
            body = {'error': {'message': 'fixture throttle', 'type': 'rate_limit'}}
            raw = json.dumps(body).encode('utf-8')
            code = 429
        else:
            used = int(payload.get('max_tokens', 32))
            body = {'id': 'stub-%d' % SEEN['n'], 'usage': {'total_tokens': used}}
            raw = json.dumps(body).encode('utf-8')
            code = 200
        self.send_response(code)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

if __name__ == '__main__':
    ThreadingHTTPServer(('127.0.0.1', 8080), Handler).serve_forever()

The check below is a proposal as well, and this article does not claim that anyone executed it. It locks the arithmetic that a reviewer should reimplement in the test runner they already trust. A passing stub log for row three shows no request when spent tokens are already 80 against a 180 budget. A failing submission still opens a socket and then explains the overspend after the response returns.

def test_third_row_stops_before_dispatch():
    ledger = {'budget': 180, 'spent': 80, 'events': []}
    row = {'id': 'r3', 'max_tokens': 80, 'prompt_ceiling': 40}
    ceiling = int(row['max_tokens']) + int(row['prompt_ceiling'])
    assert ceiling == 120
    assert gate(ledger, ceiling) == 'budget_stop'

Rubric

Reviewers should score observable client behavior, not the prose quality of whatever completion text comes back. Each line is pass or fail, and a pass requires the behavior in the ledger file as well as in the exit code. A fluent README cannot replace a missing status string that the persisted ledger file should contain.

1. The pre-dispatch gate blocks a row whose ceiling exceeds remaining tokens, and the stub log shows no request for that row.
2. Reported total_tokens is the only quantity subtracted, and a missing usage object leaves the spent field unchanged.
3. HTTP 429 and connection errors map to upstream_unavailable without a retry loop that keeps calling the host.
4. HTTP 401 and HTTP 403 stop the batch as auth_rejected and never print, rotate, or store the key.
5. The fixture checksum is unchanged, and the model identifier comes from configuration rather than a source constant.
6. A second run against the same ledger does not double-spend rows that were already recorded as ok.

Failure modes the rubric catches

The rubric treats the following outcomes as failures because each one hides spend or blocks a second reviewer. They are design risks written for this packet, not measured incidents from a named production system. A demo that looks smooth on the first row can still fail every line in the list below.

• Estimating tokens locally and ignoring provider usage drifts as soon as the server tokenizer disagrees with the client.
• Retrying HTTP 429 until the process is killed turns a free-server throttle into a self-inflicted outage for everyone sharing the host.
• Treating a free host as always warm hangs the review on a live call when the packet already required a stub.
• Writing the API key into the ledger, the traceback, or the fixture fails the secret rule even when the gate itself is correct.
• Hardcoding a model name from an older README breaks the day the free catalog rotates, which this packet refuses to freeze.
• Scoring completion text against a hidden answer key measures the model and leaves the client behavior ungraded.
• Resetting the ledger on every invocation hides double-spend and makes the second-run check impossible to pass honestly.

Offline review procedure

The offline path is the grading path, and the live call is a later smoke test with a tiny budget. A reviewer starts the stub, runs the client, and pretty-prints the ledger before reading any completion text. No key is required for that path, which keeps the packet usable on a laptop that has no outbound network. If the ledger file is missing after the run, the submission fails before any status debate begins.

python stub_server.py
python -m budget_gate run \
  --fixture fixtures/prompts.jsonl \
  --ledger var/ledger.json \
  --budget-tokens 180 \
  --base-url http://127.0.0.1:8080/v1 \
  --model "$MODEL_ID"
python -m json.tool var/ledger.json

A correct ledger shows spent tokens at or below 180 and an event list whose length matches successful rows only. The third row status is budget_stop when the ceiling no longer fits, even if the stub would still have returned HTTP 200. If the stub is set to fail after two calls and the gate still had room, the status is upstream_unavailable instead. Those two outcomes must remain distinct strings, because collapsing them hides whether the client or the host stopped the work.

Where the free path fits

An operator-supplied free server is a reasonable optional smoke target after the stub is already green. Free model access is relevant because a candidate can read current notes, export a key, and confirm that the same client speaks that route. Neither availability claim in this packet includes a promised token count, a named model, a hardware shape, or a permanence guarantee. If the live route rejects the key or omits usage, the typed result is the correct outcome, and estimating tokens is not a fix.

Readers who want the optional smoke path can start from the MonkeyCode project notes for the current date. Those notes, not this article, are the source for the free model route, the free server instructions, and any current quota. The base URL flag should change only after the stub run passes and the ledger already shows a clean stop. That step checks route compatibility for the same client, and it does not replace the ledger tests above.

Limitations and non-goals

This packet does not measure answer quality, latency, or cost in currency, and it should not be cited as a capacity plan. It does not prove that a free tier will exist next month, because availability claims are not duration claims. The sample modules are unexecuted proposals, and they omit streaming, tool calls, and file locking on purpose. A local ceiling is a review control, not an invoice that a finance team can reconcile against a provider bill.

Who should skip this packet

The following groups should not stretch this packet past the evidence the ledger and stub actually contain.

• Interviewers who need to grade research quality of model prose should use a different rubric and a frozen evaluation set.
• Teams that cannot supply a fixture, a stub, and a second reviewer should not treat a live demo as this packet.
• Anyone who needs a certified spend number for finance should not read the local ledger as a billing export.
• Candidates asked only to wrap a vendor SDK, without a visible ledger, will not satisfy the pass lines above.

Closing criterion

The packet is complete when a second reviewer can delete the live URL, rerun the stub, and reach the same pass or fail. That property keeps the hiring decision on client discipline rather than on whoever had a warmer free host that morning. Free access can shorten setup for the optional smoke test, but the score still belongs to the stop ledger. A submission that cannot reproduce the stop offline is incomplete, regardless of how fluent its sample completion looked.