The Ledger Take-Home: An Interview Task That Grades Token Accounting, Not the Chatbot

Most AI feature interviews grade the demo, and the demo is the least transferable part of the work. A candidate who ships a chat box in ninety minutes has shown almost nothing about how the system behaves on the second day, when retries, rate limits, and partial failures start compounding. The task below inverts the usual weighting: the visible feature is trivial, and the deliverable that decides the hire is a run ledger that reconciles against the provider's own reported usage.

The exercise assumes free model access and a free server option, because an interview task that requires a paid account filters candidates on their credit card rather than their judgment. The reference setup uses MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Quotas and limits change, so the prompt tells candidates to read the current allowance from the project's own page instead of trusting a number copied into an assignment document.

Why accounting is the missing interview signal

Every team that ships an LLM-backed feature eventually discovers that the bill and the logs disagree. The gap usually comes from a retry loop that silently triples traffic, a cache keyed on the wrong field, or a client-side token estimate that nobody labeled as an estimate. None of those bugs show up in a demo, and all of them show up in a ledger within a day of real traffic.

Grading accounting also removes a large chunk of interviewer subjectivity. Either the candidate's ledger sums match the dashboard within a stated tolerance, or they do not, and either the candidate explained the delta or they hid it. That is a much cleaner signal than an opinion about code style.

The prompt candidates receive

Candidates get two hours and a written brief with no hidden requirements. The brief is deliberately narrow so the grading stays about discipline rather than scope.

Deliverables

1. A runner that accepts a support ticket on stdin and returns a structured triage record plus the raw model text.
2. An append-only runs.jsonl ledger with one line per run, containing run id, prompt hash, every attempt, status codes, retry delays, latency, and usage numbers.
3. A reconcile command that prints ledger totals next to the provider-reported totals and a signed delta.
4. A short LIMITS.md naming every number the candidate could not measure and how they labeled it.
5. A deployment on the free server option, with the first-request cold-start latency recorded as a ledger field rather than described in prose.

Constraints

• No dependency heavier than the Node standard library, so a reviewer can run it without an install step.
• The API key must never appear in the ledger, the logs, or a thrown error message.
• A run must terminate inside a stated wall-clock budget even when every attempt is throttled.

The rubric

The rubric is published to candidates before they start, which is the point: the task measures whether they read the requirements, not whether they guessed them.

Criterion	Points	Strong answer	Weak answer
Ledger completeness	25	Every attempt is a row, including abandoned ones	Only successful calls are logged
Reconciliation	25	Delta computed and explained, tolerance stated	Totals printed with no comparison
Retry semantics	20	429, 5xx, and timeout handled differently	One fixed sleep for every failure
Budget enforcement	15	Hard deadline in the loop, not in a wrapper	Relies on the caller to time out
Honest labeling	15	Unknown usage marked null, never 0	Estimates presented as measurements

A candidate can score well with an ugly CLI. That is intended, and it is worth telling them so in the brief, because otherwise strong candidates over-invest in formatting and under-invest in the part being graded.

Sample solution

The reference runner below is the artifact a reviewer compares against, not a target for byte-for-byte imitation. It is written for Node 20 and has zero dependencies; run it yourself before reusing it, since provider response shapes differ and the usage field names must be mapped once, in one place.

// ledger.mjs — Node 20+, zero dependencies. Reference solution; verify against your provider.
import { createHash, randomUUID } from "node:crypto";
import { appendFile } from "node:fs/promises";

const ENDPOINT = process.env.MODEL_ENDPOINT; // operator-supplied base URL
const KEY = process.env.MODEL_API_KEY;       // never logged, never written to the ledger
const LEDGER = process.env.LEDGER_PATH ?? "./runs.jsonl";

const sha = (s) => createHash("sha256").update(s).digest("hex").slice(0, 16);

function classify(r) {
  if (r.ok) return "done";
  if (r.status === 429) return "throttled";
  if (r.status >= 500) return "server";
  if (r.status === 0) return "timeout";
  return "client";
}

async function attempt(prompt, n) {
  const started = Date.now();
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), 20_000);
  try {
    const res = await fetch(ENDPOINT, {
      method: "POST",
      headers: { "content-type": "application/json", authorization: `Bearer ${KEY}` },
      body: JSON.stringify({ input: prompt }),
      signal: ac.signal,
    });
    const body = await res.json().catch(() => null);
    return {
      attempt: n,
      status: res.status,
      ok: res.ok,
      retryAfter: res.headers.get("retry-after"),
      usage: body?.usage ?? null, // null means unknown; never coerce to 0
      latencyMs: Date.now() - started,
    };
  } catch (err) {
    return { attempt: n, status: 0, ok: false, error: err.name, usage: null, latencyMs: Date.now() - started };
  } finally {
    clearTimeout(timer);
  }
}

export async function run(prompt, { maxAttempts = 4, budgetMs = 45_000 } = {}) {
  const runId = randomUUID();
  const promptHash = sha(prompt);
  const deadline = Date.now() + budgetMs;
  const attempts = [];

  for (let n = 1; n <= maxAttempts; n++) {
    if (Date.now() > deadline) { attempts.push({ attempt: n, skipped: "budget" }); break; }
    const r = await attempt(prompt, n);
    attempts.push(r);
    const verdict = classify(r);
    if (verdict === "done" || verdict === "client") break;
    const backoff = Math.min(8_000, 500 * 2 ** (n - 1)) + Math.floor(Math.random() * 250);
    const wait = r.retryAfter ? Number(r.retryAfter) * 1000 : backoff;
    await new Promise((resolve) => setTimeout(resolve, wait));
  }

  const record = {
    runId,
    promptHash,
    attempts,
    outcome: attempts.at(-1)?.ok ? "ok" : "failed",
    inputTokens: attempts.reduce((s, a) => s + (a.usage?.input_tokens ?? 0), 0),
    outputTokens: attempts.reduce((s, a) => s + (a.usage?.output_tokens ?? 0), 0),
    unknownUsageAttempts: attempts.filter((a) => !a.usage).length,
  };
  await appendFile(LEDGER, JSON.stringify(record) + "\n");
  return record;
}

The reconciliation step is where most submissions fall apart, because it requires the candidate to admit what they cannot see. A ledger that sums to a number nobody compares against is a diary, not a control.

// reconcile.mjs — node reconcile.mjs ./runs.jsonl <reportedInput> <reportedOutput>
import { readFile } from "node:fs/promises";

const rows = (await readFile(process.argv[2] ?? "./runs.jsonl", "utf8"))
  .trim().split("\n").filter(Boolean).map((line) => JSON.parse(line));

const sum = (key) => rows.reduce((s, r) => s + r[key], 0);
const reportedIn = Number(process.argv[3] ?? "NaN");
const reportedOut = Number(process.argv[4] ?? "NaN");

const table = {
  runs: rows.length,
  failedRuns: rows.filter((r) => r.outcome !== "ok").length,
  retriedRuns: rows.filter((r) => r.attempts.length > 1).length,
  attemptsTotal: rows.reduce((s, r) => s + r.attempts.length, 0),
  unknownUsageAttempts: sum("unknownUsageAttempts"),
  ledgerInputTokens: sum("inputTokens"),
  ledgerOutputTokens: sum("outputTokens"),
};
table.deltaInput = Number.isFinite(reportedIn) ? reportedIn - table.ledgerInputTokens : "unmeasured";
table.deltaOutput = Number.isFinite(reportedOut) ? reportedOut - table.ledgerOutputTokens : "unmeasured";
console.table(table);

What the sample deliberately omits

There is no cache, no prompt template registry, and no streaming path, because each of those adds a reconciliation surface the candidate would not have time to defend. Reviewers should say this out loud during the interview: scope exclusion is part of the grade, and a candidate who cuts a feature to protect the accounting is making the right trade.

A decision table for retry signals

The single most discriminating question in the review is what the candidate does with a given failure signal. The table below is the answer key a reviewer keeps open.

Signal	Correct response	Common mistake	How the rubric catches it
429 with Retry-After	Wait the stated interval, then resume	Fixed 200 ms sleep loop	Attempts spike with no wall-clock growth
429 without Retry-After	Exponential backoff with jitter, capped	Immediate immediate retry	Attempt count hits max in under a second
5xx	Bounded retries, ledger the failure	Treat as a prompt problem	No 5xx rows in the ledger at all
Timeout	Abort, log, retry once at most	Hang until the platform kills the process	Missing skipped: budget rows
4xx (auth or schema)	Fail fast, do not retry	Retry until the budget is gone	One bad request becomes four billed requests

Common failure modes

1. Estimating tokens and reporting them as measured. The classic version divides characters by four and puts the result in a field called tokens. The fix is a separate estimated: true flag, and the rubric should award points for adding it.
2. Caching on the ticket id instead of the prompt hash. Two tickets with identical bodies then cost two calls, or worse, a stale answer gets served for an edited ticket.
3. Counting a retried run once. The ledger records the run, but the bill records every attempt, so the delta grows with the retry rate and nobody notices until a spike.
4. Coercing missing usage to zero. A null usage block means unknown, and writing zero turns an unknown into a false measurement that propagates into every later total.
5. Leaking the key in an error message. A thrown fetch error that includes the request object will serialize the authorization header into the ledger on the first timeout.

Limitations and who should skip this

This task is wrong for teams that already have per-request usage in their production telemetry, because the candidate would be reimplementing a solved problem instead of demonstrating judgment. It is also a poor fit for a thirty-minute screen, since the reconciliation step alone needs time to design and defend.

Candidates on a provider that does not return usage in the response body should not be asked to produce exact numbers. In that setup the exercise changes shape: the deliverable becomes a labeled estimate plus a written explanation of the error bound, which is still gradeable but measures something different.

One practical note for the free server option: the first request after idle will be slow, and that latency belongs in the ledger as a field rather than in a README sentence. Reviewers should check that the candidate recorded it instead of quietly excluding warm-up requests from their averages.

If you want to run the reference pair before rewriting it for your own stack, start with the free model access and the free server option, and read the current allowance on the project's own page rather than any figure quoted in a post. Then change one thing at a time and watch the reconciliation delta move.