DEV Community: Riley Zhu

The Ledger Take-Home: An Interview Task That Grades Token Accounting, Not the Chatbot

Riley Zhu — Tue, 27 Oct 2026 13:25:36 +0000

Most AI feature interviews grade the demo, and the demo is the least transferable part of the work. A candidate who ships a chat box in ninety minutes has shown almost nothing about how the system behaves on the second day, when retries, rate limits, and partial failures start compounding. The task below inverts the usual weighting: the visible feature is trivial, and the deliverable that decides the hire is a run ledger that reconciles against the provider's own reported usage.

The exercise assumes free model access and a free server option, because an interview task that requires a paid account filters candidates on their credit card rather than their judgment. The reference setup uses MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Quotas and limits change, so the prompt tells candidates to read the current allowance from the project's own page instead of trusting a number copied into an assignment document.

Why accounting is the missing interview signal

Every team that ships an LLM-backed feature eventually discovers that the bill and the logs disagree. The gap usually comes from a retry loop that silently triples traffic, a cache keyed on the wrong field, or a client-side token estimate that nobody labeled as an estimate. None of those bugs show up in a demo, and all of them show up in a ledger within a day of real traffic.

Grading accounting also removes a large chunk of interviewer subjectivity. Either the candidate's ledger sums match the dashboard within a stated tolerance, or they do not, and either the candidate explained the delta or they hid it. That is a much cleaner signal than an opinion about code style.

The prompt candidates receive

Candidates get two hours and a written brief with no hidden requirements. The brief is deliberately narrow so the grading stays about discipline rather than scope.

Deliverables

A runner that accepts a support ticket on stdin and returns a structured triage record plus the raw model text.
An append-only runs.jsonl ledger with one line per run, containing run id, prompt hash, every attempt, status codes, retry delays, latency, and usage numbers.
A reconcile command that prints ledger totals next to the provider-reported totals and a signed delta.
A short LIMITS.md naming every number the candidate could not measure and how they labeled it.
A deployment on the free server option, with the first-request cold-start latency recorded as a ledger field rather than described in prose.

Constraints

No dependency heavier than the Node standard library, so a reviewer can run it without an install step.
The API key must never appear in the ledger, the logs, or a thrown error message.
A run must terminate inside a stated wall-clock budget even when every attempt is throttled.

The rubric

The rubric is published to candidates before they start, which is the point: the task measures whether they read the requirements, not whether they guessed them.

Criterion	Points	Strong answer	Weak answer
Ledger completeness	25	Every attempt is a row, including abandoned ones	Only successful calls are logged
Reconciliation	25	Delta computed and explained, tolerance stated	Totals printed with no comparison
Retry semantics	20	429, 5xx, and timeout handled differently	One fixed sleep for every failure
Budget enforcement	15	Hard deadline in the loop, not in a wrapper	Relies on the caller to time out
Honest labeling	15	Unknown usage marked `null`, never `0`	Estimates presented as measurements

A candidate can score well with an ugly CLI. That is intended, and it is worth telling them so in the brief, because otherwise strong candidates over-invest in formatting and under-invest in the part being graded.

Sample solution

The reference runner below is the artifact a reviewer compares against, not a target for byte-for-byte imitation. It is written for Node 20 and has zero dependencies; run it yourself before reusing it, since provider response shapes differ and the usage field names must be mapped once, in one place.

// ledger.mjs — Node 20+, zero dependencies. Reference solution; verify against your provider.
import { createHash, randomUUID } from "node:crypto";
import { appendFile } from "node:fs/promises";

const ENDPOINT = process.env.MODEL_ENDPOINT; // operator-supplied base URL
const KEY = process.env.MODEL_API_KEY;       // never logged, never written to the ledger
const LEDGER = process.env.LEDGER_PATH ?? "./runs.jsonl";

const sha = (s) => createHash("sha256").update(s).digest("hex").slice(0, 16);

function classify(r) {
  if (r.ok) return "done";
  if (r.status === 429) return "throttled";
  if (r.status >= 500) return "server";
  if (r.status === 0) return "timeout";
  return "client";
}

async function attempt(prompt, n) {
  const started = Date.now();
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), 20_000);
  try {
    const res = await fetch(ENDPOINT, {
      method: "POST",
      headers: { "content-type": "application/json", authorization: `Bearer ${KEY}` },
      body: JSON.stringify({ input: prompt }),
      signal: ac.signal,
    });
    const body = await res.json().catch(() => null);
    return {
      attempt: n,
      status: res.status,
      ok: res.ok,
      retryAfter: res.headers.get("retry-after"),
      usage: body?.usage ?? null, // null means unknown; never coerce to 0
      latencyMs: Date.now() - started,
    };
  } catch (err) {
    return { attempt: n, status: 0, ok: false, error: err.name, usage: null, latencyMs: Date.now() - started };
  } finally {
    clearTimeout(timer);
  }
}

export async function run(prompt, { maxAttempts = 4, budgetMs = 45_000 } = {}) {
  const runId = randomUUID();
  const promptHash = sha(prompt);
  const deadline = Date.now() + budgetMs;
  const attempts = [];

  for (let n = 1; n <= maxAttempts; n++) {
    if (Date.now() > deadline) { attempts.push({ attempt: n, skipped: "budget" }); break; }
    const r = await attempt(prompt, n);
    attempts.push(r);
    const verdict = classify(r);
    if (verdict === "done" || verdict === "client") break;
    const backoff = Math.min(8_000, 500 * 2 ** (n - 1)) + Math.floor(Math.random() * 250);
    const wait = r.retryAfter ? Number(r.retryAfter) * 1000 : backoff;
    await new Promise((resolve) => setTimeout(resolve, wait));
  }

  const record = {
    runId,
    promptHash,
    attempts,
    outcome: attempts.at(-1)?.ok ? "ok" : "failed",
    inputTokens: attempts.reduce((s, a) => s + (a.usage?.input_tokens ?? 0), 0),
    outputTokens: attempts.reduce((s, a) => s + (a.usage?.output_tokens ?? 0), 0),
    unknownUsageAttempts: attempts.filter((a) => !a.usage).length,
  };
  await appendFile(LEDGER, JSON.stringify(record) + "\n");
  return record;
}

The reconciliation step is where most submissions fall apart, because it requires the candidate to admit what they cannot see. A ledger that sums to a number nobody compares against is a diary, not a control.

// reconcile.mjs — node reconcile.mjs ./runs.jsonl <reportedInput> <reportedOutput>
import { readFile } from "node:fs/promises";

const rows = (await readFile(process.argv[2] ?? "./runs.jsonl", "utf8"))
  .trim().split("\n").filter(Boolean).map((line) => JSON.parse(line));

const sum = (key) => rows.reduce((s, r) => s + r[key], 0);
const reportedIn = Number(process.argv[3] ?? "NaN");
const reportedOut = Number(process.argv[4] ?? "NaN");

const table = {
  runs: rows.length,
  failedRuns: rows.filter((r) => r.outcome !== "ok").length,
  retriedRuns: rows.filter((r) => r.attempts.length > 1).length,
  attemptsTotal: rows.reduce((s, r) => s + r.attempts.length, 0),
  unknownUsageAttempts: sum("unknownUsageAttempts"),
  ledgerInputTokens: sum("inputTokens"),
  ledgerOutputTokens: sum("outputTokens"),
};
table.deltaInput = Number.isFinite(reportedIn) ? reportedIn - table.ledgerInputTokens : "unmeasured";
table.deltaOutput = Number.isFinite(reportedOut) ? reportedOut - table.ledgerOutputTokens : "unmeasured";
console.table(table);

What the sample deliberately omits

There is no cache, no prompt template registry, and no streaming path, because each of those adds a reconciliation surface the candidate would not have time to defend. Reviewers should say this out loud during the interview: scope exclusion is part of the grade, and a candidate who cuts a feature to protect the accounting is making the right trade.

A decision table for retry signals

The single most discriminating question in the review is what the candidate does with a given failure signal. The table below is the answer key a reviewer keeps open.

Signal	Correct response	Common mistake	How the rubric catches it
429 with `Retry-After`	Wait the stated interval, then resume	Fixed 200 ms sleep loop	Attempts spike with no wall-clock growth
429 without `Retry-After`	Exponential backoff with jitter, capped	Immediate immediate retry	Attempt count hits max in under a second
5xx	Bounded retries, ledger the failure	Treat as a prompt problem	No 5xx rows in the ledger at all
Timeout	Abort, log, retry once at most	Hang until the platform kills the process	Missing `skipped: budget` rows
4xx (auth or schema)	Fail fast, do not retry	Retry until the budget is gone	One bad request becomes four billed requests

Common failure modes

Estimating tokens and reporting them as measured. The classic version divides characters by four and puts the result in a field called tokens. The fix is a separate estimated: true flag, and the rubric should award points for adding it.
Caching on the ticket id instead of the prompt hash. Two tickets with identical bodies then cost two calls, or worse, a stale answer gets served for an edited ticket.
Counting a retried run once. The ledger records the run, but the bill records every attempt, so the delta grows with the retry rate and nobody notices until a spike.
Coercing missing usage to zero. A null usage block means unknown, and writing zero turns an unknown into a false measurement that propagates into every later total.
Leaking the key in an error message. A thrown fetch error that includes the request object will serialize the authorization header into the ledger on the first timeout.

Limitations and who should skip this

This task is wrong for teams that already have per-request usage in their production telemetry, because the candidate would be reimplementing a solved problem instead of demonstrating judgment. It is also a poor fit for a thirty-minute screen, since the reconciliation step alone needs time to design and defend.

Candidates on a provider that does not return usage in the response body should not be asked to produce exact numbers. In that setup the exercise changes shape: the deliverable becomes a labeled estimate plus a written explanation of the error bound, which is still gradeable but measures something different.

One practical note for the free server option: the first request after idle will be slow, and that latency belongs in the ledger as a field rather than in a README sentence. Reviewers should check that the candidate recorded it instead of quietly excluding warm-up requests from their averages.

If you want to run the reference pair before rewriting it for your own stack, start with the free model access and the free server option, and read the current allowance on the project's own page rather than any figure quoted in a post. Then change one thing at a time and watch the reconciliation delta move.

The Rejection File: A Take-Home That Grades the Patch the Candidate Refused to Ship

Riley Zhu — Mon, 26 Oct 2026 11:57:57 +0000

A take-home that grades only the final diff measures output volume; the stronger hiring signal sits in the proposals a candidate refused to ship. The Rejection File is a small exercise that asks for a working patch plus an auditable record of at least one model proposal the candidate rejected, together with a command that proves the refusal was correct. It fits in two to three hours, runs on free-tier infrastructure, and separates engineers who verify from engineers who relay.

The format responds to a DEV thread this week arguing that engineers should not become a proxy between a requester and an AI. That complaint is really about unverified relay: a proposal arrives, a patch leaves, and nobody checks the middle step. A rejection file makes the middle step the deliverable, so reviewers can inspect judgment instead of guessing at it.

The signal a diff-only take-home cannot see

Most take-homes reward candidates for producing code that passes hidden tests, which is a narrow measurement of one skill. A model can generate a passing patch from a decent prompt, so the passing patch no longer distinguishes much. What still distinguishes candidates is whether they noticed that two of the four proposals were quietly wrong.

Rejecting a proposal is cheap to claim and expensive to prove, which is exactly why it makes a good artifact. Anyone can write "I rejected the broad exception handler because it hides errors" in a README. Requiring a repro command that fails for the rejected proposal turns the claim into evidence a reviewer can replay in one terminal command.

The prompt, written for the candidate

The following text can be handed over verbatim; it assumes an isolated fixture repository with a small, real defect.

Fix the defect in src/. Any approach is acceptable as long as the existing test suite stays green.
Use an AI assistant for at least two candidate proposals. Record each proposal in rejections.json using the schema below.
Reject at least one proposal and supply a repro command that fails when your rejection is correct.
Keep the accepted diff under roughly forty changed lines unless the repository forces more; explain any overrun in the handoff note.
Write a half-page handoff note covering what you verified, what you assumed, and what you would check next.

The constraints matter more than the defect. Candidates who burn the whole repository into context tend to produce a plausible patch plus a rejection file full of unverifiable prose, which the rubric scores accordingly.

The rubric

Criterion	Weight	What earns full credit
Rejection evidence	30	Every rejection has a repro command that actually fails
Patch correctness	20	Full suite green, including tests the candidate did not write
Reproduction quality	15	One command, no manual setup steps, deterministic output
Scope discipline	15	Diff stays near the defect; unrelated refactors are absent
Budget discipline	10	Token spend is recorded and re-prompting loops are explained
Handoff note	10	Separates verified facts from assumptions without hedging

The weighting is deliberate. A candidate with a mediocre patch and three credible rejections generally scores higher than one with a clean patch and a rejection file that cannot be replayed, because the second candidate produced no inspectable judgment.

The artifact: a rejection file that can be audited

The schema is intentionally boring so that a script can check it.

{
  "acceptance": {
    "repro": "pytest -q tests/test_resume.py::test_truncated_stream"
  },
  "rejections": [
    {
      "id": "reject-stream-close",
      "proposal": "Close the stream inside the except branch and return None",
      "repro": "pytest -q tests/test_resume.py::test_truncated_stream",
      "expect_failure": true,
      "decision": "rejected: callers cannot distinguish truncation from empty input"
    },
    {
      "id": "reject-broad-except",
      "proposal": "Wrap the consumer loop in except Exception and retry once",
      "repro": "pytest -q tests/test_cancel.py::test_cancel_mid_stream",
      "expect_failure": true,
      "decision": "rejected: converts a loud failure into an invisible stall"
    }
  ]
}

The audit script below is a reference sketch rather than a benchmarked tool, and it should be run against your own fixture before it grades anyone. It replays each repro inside the candidate's working copy and checks that the exit code matches the claim, then confirms that the accepted patch passes its own repro.

# rejection_audit.py — replay every rejection and confirm the claim holds
import json, pathlib, subprocess, sys

WORK = pathlib.Path("work")

def run(cmd: str, timeout: int = 120) -> int:
    proc = subprocess.run(cmd, shell=True, cwd=WORK, timeout=timeout,
                          capture_output=True, text=True)
    return proc.returncode

def audit(path: str) -> int:
    data = json.loads(pathlib.Path(path).read_text())
    bad = 0
    for entry in data["rejections"]:
        code = run(entry["repro"])
        credible = (code != 0) == entry["expect_failure"]
        print(f'{entry["id"]:<26} exit={code:<3} credible={credible}')
        bad += 0 if credible else 1
    accepted = run(data["acceptance"]["repro"])
    print(f'{"accepted patch".ljust(26)} exit={accepted:<3} credible={accepted == 0}')
    return bad + (accepted != 0)

if __name__ == "__main__":
    sys.exit(1 if audit(sys.argv[1]) else 0)

A non-zero exit from the auditor is not a rejection of the candidate; it is a signal for the reviewer to open the named entry and read the reasoning. The script checks whether evidence exists, not whether the reasoning is sound.

Running the exercise without paying per candidate

The environment cost is the part that usually kills custom take-homes, because per-candidate model spend and hosted runners add up across a hiring loop. MonkeyCode's free model access and free server option make a dry run practical: the reviewer keeps the fixture repository and the audit script, and candidates get a working environment without a procurement conversation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability details come from the operator rather than from independent testing: free model access, a free server option, and a free token allotment quoted at ten million tokens at the time of writing. Those terms can change, so confirm the current limits before you put them in a candidate brief. The claim that matters for this exercise is narrower and easy to check yourself: a candidate can complete a two-hour patch-and-reject loop without a paid account.

git clone <your-fixture-repo> work
cd work && python -m venv .venv && . .venv/bin/activate
pip install -r requirements-dev.txt
python ../rejection_audit.py ../rejections.json

A worked sample

A strong submission produces output where every line is credible, and one weaker submission produces output where a single line is not.

$ python rejection_audit.py rejections.json
reject-stream-close        exit=1   credible=True
reject-broad-except        exit=1   credible=True
reject-retry-forever       exit=0   credible=False
accepted patch             exit=0   credible=True

The third entry is the interesting one. The candidate claimed to reject an infinite retry loop, but the supplied repro exits cleanly, so the rejection has no supporting evidence. In practice this is usually rejection theater: a proposal the candidate never seriously considered, documented after the fact to satisfy the prompt.

Reviewers should read the failing entry before scoring, because a credible rejection and an honest admission of a wrong repro look similar in the terminal. A candidate who writes "this repro passes and I no longer believe the rejection holds" is demonstrating better judgment than one who fabricates a failing command.

Common failure modes

Rejection theater. The rejected proposal is trivial, such as removing a comment, and the repro proves nothing about the defect.
Unreplayable evidence. The repro depends on a local file, an environment variable, or a manual step that the reviewer cannot reproduce.
Silent scope creep. A four-line defect becomes a two-hundred-line refactor, and the rejection file never explains why the extra surface was necessary.
Passing-by-accident patches. The suite is green because the candidate edited a test rather than the source, which the auditor catches only if fixtures are read-only.
Context dumping. Every re-prompt includes the entire repository, the budget is exhausted early, and the remaining decisions are made without evidence.
Narrated verification. The handoff note says "verified" without naming the command, the test, or the observed output that supports the word.

Mitigations are cheap: make fixture tests read-only in CI, cap the token allotment per candidate, and require one repro per rejection with no free-text substitute.

Limitations and who should skip this

The format needs a fixture repository with a genuine, small, test-detectable defect, and building that fixture takes a reviewer most of a day. Teams without a sandboxed runner should not adopt it, because replaying candidate commands on a shared host is a security problem rather than a convenience problem.

It also adds review time, roughly forty-five minutes per submission when the rejection file is read properly. Organizations that need same-day decisions, or roles where the work is not test-driven, get little from this exercise and should use a shorter structured interview instead. Finally, anyone who cannot confirm the current free-tier terms should describe the environment generically in the brief rather than quoting numbers that may have moved.

The trade is straightforward: a diff-only take-home measures what a candidate shipped, while a rejection file measures what a candidate was willing to stop. If you want to dry-run the harness before committing to it, run one internal submission through MonkeyCode's free server option and see whether the audit script survives your own repository first.

Claim Cards: A Take-Home That Grades What the Candidate Verified, Not What They Built

Riley Zhu — Tue, 20 Oct 2026 10:07:03 +0000

The short version

A take-home that grades the artifact rewards plausible output, and plausible output is now cheap. A take-home that grades the evidence attached to the artifact rewards engineering judgment, and that evidence is expensive to fake. This article packages the idea as a reusable interview task: the candidate ships a claim card for every assertion the build depended on, plus a checker that re-executes each claim and compares the result to what the card recorded. Candidate time lands around three to four hours; reviewer time lands around thirty minutes.

The design goal is narrow. The task should make the difference between checked and assumed visible in a diff, so that a reviewer can grade verification discipline without re-reading an entire repository. Everything below is a template that a team can copy, trim, and rerun.

Why output-only grading breaks down

Most take-homes ask for a small service, then grade it on structure, tests, and naming. Assistants have collapsed the cost of all three, which means the signal that used to separate candidates has decayed. A candidate who never ran the test suite can still deliver a tidy repository with a green-looking CI badge.

The failure is not dishonesty in most cases; it is unexamined confidence. A generated README says the importer streams rows, the code actually buffers, and nobody measured either. Claim cards attack exactly that gap, because each card must record the command that produced the belief and the exit code that came back.

A useful side effect appears during review. Two candidates can submit near-identical code while their card sets look nothing alike, and the card set is the more predictive document.

The prompt

Hand the candidate the block below verbatim, with the clock stated in writing.

Task: implement CSV import for the attached fixture set (about 200 lines).

You may use any assistant, any model, and any editor. You must also ship:

1. src/            working implementation
2. claims.yaml     one entry per assertion the build relies on
3. notes.md        <= 30 lines: what you refused to claim, and why

Rules:
- Do not modify verify.py; the reviewer runs it unmodified.
- Every entry marked verified must carry a command and the exit code you observed.
- Assertions you could not test go in as status: unverified. That is not a penalty.
- A claim you deleted because you could not test it is worth less than a claim you
  labelled unverified and explained.

The third rule carries most of the weight. Candidates arriving from output-only take-homes tend to either over-claim or silently drop the hard parts, and both behaviours are visible in the file.

The artifact: a claim card schema and a re-executable checker

Three fields make a card useful: the assertion in one sentence, the method that tests it, and the result the candidate actually saw. Anything softer turns the file into prose.

# claims.yaml (schema is the candidate's contract)
- id: C-01
  claim: "CSV files with a UTF-8 BOM import without an offset error"
  method: "python -m pytest tests/test_import.py -k bom -q"
  recorded_exit: 0
  status: verified
  evidence: "1 passed in 0.41s"

- id: C-02
  claim: "The importer streams rows instead of buffering the file"
  method: "no measurement performed"
  recorded_exit: null
  status: unverified
  evidence: "row count memory profile not taken"

The checker stays deliberately small, because a large checker becomes a second implementation the candidate has to debug. It re-runs each verified method, compares exit codes, and flags drift; it never executes a command the reviewer has not already seen in the file.

# verify.py — reference implementation, roughly 60 lines
import subprocess
import sys

import yaml


def run(cmd: str, timeout: int = 120) -> tuple[int, str]:
    proc = subprocess.run(
        cmd, shell=True, capture_output=True, text=True, timeout=timeout
    )
    return proc.returncode, (proc.stdout + proc.stderr)[-300:].strip()


def main(path: str = "claims.yaml") -> int:
    cards = yaml.safe_load(open(path, encoding="utf-8"))
    drifted = 0
    for card in cards:
        cid, status = card["id"], card.get("status")
        if status != "verified":
            print(f"{cid}  SKIP     {status}")
            continue
        code, tail = run(card["method"])
        ok = code == card["recorded_exit"]
        drifted += 0 if ok else 1
        print(f"{cid}  {'REPRODUCED' if ok else 'DRIFT':<10} exit={code}  {tail[:80]}")
    print(f"\ndrifted cards: {drifted}/{len(cards)}")
    return 1 if drifted else 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

The reference above is a labelled starting point rather than a hardened tool. It has no sandboxing, no per-card timeout budget, and no protection against a candidate shipping a conftest.py that lies. Reviewers who accept untrusted code should run it in a disposable container and pin the checker by hash before sending the prompt.

The rubric

Five dimensions, weighted so that a broken implementation with honest cards still outscores a polished implementation with fabricated ones.

Dimension	Weight	0	1	2
Claim coverage	25	few cards, mostly restating the README	cards cover the main paths	cards cover the main paths and one edge case the fixture hides
Reproducibility	25	methods are prose	some commands rerun cleanly	every verified card reruns on a clean checkout
Honest gaps	20	everything marked verified	a few unverified cards, unexplained	unverified cards named and bounded in `notes.md`
Checker integrity	15	`verify.py` edited	unmodified, but cards exploit loose matching	unmodified, and cards avoid commands that trivially pass
Implementation	15	does not import the fixtures	imports the happy path	imports happy path plus a malformed row

A reviewer can apply this table in about ten minutes per submission. The 20-point honesty axis is the one that consistently separates senior candidates from fluent ones, because fluent candidates have the most to gain from over-claiming.

A sample solution, excerpted

A strong submission on the CSV task typically ships five to eight cards, not thirty. One representative excerpt shows the behaviour worth hiring for: the candidate discovered that a buffering assumption did not hold, downgraded the card, then added a small measurement instead of arguing.

- id: C-02
  claim: "The importer streams rows instead of buffering the file"
  method: "python -m pytest tests/test_streaming.py -k peak_rss -q"
  recorded_exit: 1
  status: unverified
  evidence: "peak RSS grew linearly with fixture size; assumption failed"

- id: C-02b
  claim: "The importer buffers the whole file, acceptable below 50 MB"
  method: "python -m pytest tests/test_streaming.py -k peak_rss -q"
  recorded_exit: 1
  status: unverified
  evidence: "threshold stated from fixture sizes, not from a measured limit"

A card that records a failing exit code alongside status: unverified is a stronger signal than a passing suite, because it proves the candidate distinguishes observation from conclusion.

Calibrating the task with free model access and a free server

Interview tasks decay when the rubric has never seen a weak submission. MonkeyCode's free model access makes calibration cheap: generate three or four candidate attempts against the same prompt, keep them as an internal reference set, and score them with the table above before the first real interview. Because the free endpoint is enough for short generation runs, a team can refresh the reference set whenever the prompt changes without a procurement step. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free server option covers the second half of calibration. The checker needs a clean checkout to be meaningful, and a disposable preview environment is the natural place for it; a reviewer wires verify.py into that environment, runs the reference attempts, and records which ones drift. The workflow that matters is not the host itself but the habit of never trusting a card set that has only ever run on the candidate's laptop.

Two guardrails belong in the setup. Reference attempts stay internal, since shipping them to candidates converts the task into pattern matching. And the checker runs against reviewer-controlled infrastructure only, because a candidate-supplied Makefile plus a shell-executing checker is an arbitrary-code path into shared compute.

Limitations and who should not use this

The task costs more to author than a feature ticket, and it grades written reasoning, which disadvantages candidates who write English slowly. It also rewards candidates who already work in verification-heavy environments, so it should be paired with a live discussion where the reasoning is spoken rather than written.

Teams hiring for pure implementation speed should not use it, because the rubric spends 45 points on documentation behaviour. Teams without a disposable runner should not use the checker either; a claim-card task with an unrunnable checker teaches candidates that evidence is decorative.

Common failure modes

Cards as a changelog. Entries describe what was built instead of what was tested, and the checker reproduces nothing.
Trivially passing methods. python -c "import src" gets marked verified and inflates the reproducibility score.
Silent deletion. Hard claims vanish from claims.yaml with no note, which is worse than an honest unverified.
Checker drift. The candidate patches verify.py to skip a failing card, which the hash pin catches immediately.
Rubric inflation. Reviewers award 2 on coverage for a large card set whose methods do not rerun on a clean checkout.
Prompt leakage. An earlier candidate posts the fixture set publicly, and the calibration value collapses.

Closing note

The test of a take-home is whether it still separates candidates after assistants got good. Claim cards survive that shift because they grade a behaviour the assistant cannot perform on the candidate's behalf: deciding what deserves to be called true. Teams that want to try the calibration loop can start with the free model access and free server option, then keep whichever reference attempts still earn their place in the rubric.

The Schema Repair Loop: A Take-Home Where the First Reply Is Allowed to Be Wrong

Riley Zhu — Mon, 19 Oct 2026 08:47:11 +0000

Structured output is the first place a cheap or free model breaks, so the interesting engineering is not the first reply but the recovery path after it. A take-home that grades only the happy path rewards a candidate for picking a cooperative prompt and a lucky sample. A take-home that grades the repair loop rewards the candidate for assuming drift, measuring it, and failing closed when repair stops working. This packet defines the brief, a weighted rubric, a reference implementation, and the failure modes that separate the two groups.

The brief handed to the candidate

Build an endpoint that turns a free-text support ticket into a validated record. The record must match a supplied JSON schema. The model may return invalid output, and your job is to recover without human help.

Hard constraints keep the exercise comparable across candidates:

One generation function is injected; the candidate may not swap providers mid-run.
A validation failure may trigger at most two repair attempts before the call returns a typed failure.
The endpoint must log the attempt count and the exact validation errors for every request.
The test suite must run offline against recorded fixtures, with no network access.
Fenced Markdown, prose preambles, and trailing commentary must be handled explicitly, not by a catch-all regex.

The schema is deliberately small but not trivial: ticket_id (string), severity (enum of low, high), components (array of strings), and customer_impact (boolean). Candidates who hand-roll a validator and candidates who pull a library are both acceptable, as long as the failure messages are specific enough to feed back into a repair prompt.

What the take-home does not ask for

The brief never asks for a wrapper around a single vendor SDK, and it never asks for retry logic that hides errors. Reviewers should treat an elaborate retry backoff as a distraction from the graded question, which is prompt repair quality. A candidate who adds exponential backoff without attempt telemetry has optimized the wrong variable.

Rubric, weighted for review speed

Weight	Criterion	What earns full credit
40%	Repair loop correctness	Parse and validation failures are distinguished; attempts are capped; failure is typed
25%	Failure specificity	Error messages name the path and the expectation, so a repair prompt is actionable
20%	Offline test suite	Fixtures cover clean, fenced, truncated, and unrecoverable outputs
15%	Operational honesty	Logs show attempt counts and the final status without leaking raw customer text

Two reviewers can apply this table in about fifteen minutes. The 40% line is the only one that requires reading code closely, and it is the one that predicts on-call behavior.

Reference solution

The snippet below is a starting point for reviewers, not a benchmarked result. It uses a hand-rolled validator so the exercise stays dependency-free, and it treats parsing and validation as separate failure classes.

// validator.js — minimal subset: type, required, properties, items, enum
export function validate(schema, value, path = '$') {
  const errors = [];
  const actual = Array.isArray(value) ? 'array' : typeof value;

  if (schema.type && actual !== schema.type) {
    return [`${path}: expected ${schema.type}, received ${actual}`];
  }
  if (schema.enum && !schema.enum.includes(value)) {
    errors.push(`${path}: expected one of ${schema.enum.join(' | ')}, received ${JSON.stringify(value)}`);
  }
  if (schema.type === 'object') {
    for (const key of schema.required ?? []) {
      if (!(key in value)) errors.push(`${path}.${key}: required field missing`);
    }
    for (const [key, sub] of Object.entries(schema.properties ?? {})) {
      if (key in value) errors.push(...validate(sub, value[key], `${path}.${key}`));
    }
  }
  if (schema.type === 'array' && schema.items) {
    value.forEach((item, index) => errors.push(...validate(schema.items, item, `${path}[${index}]`)));
  }
  return errors;
}

The loop keeps the task text stable and replaces only the error block, which keeps token use roughly flat across attempts.

// repair-loop.js
import { validate } from './validator.js';

const MAX_REPAIRS = 2;

export async function extractRecord({ generate, schema, task }) {
  const trace = [];
  let prompt = task;

  for (let attempt = 1; attempt <= MAX_REPAIRS + 1; attempt++) {
    const raw = await generate(prompt);
    const candidate = parseModelJson(raw);

    if (!candidate.ok) {
      trace.push({ attempt, stage: 'parse', errors: [candidate.error] });
      prompt = `${task}\n\nReturn a single JSON object. No prose, no code fences, no trailing text.`;
      continue;
    }

    const errors = validate(schema, candidate.value);
    trace.push({ attempt, stage: 'validate', errors });
    if (errors.length === 0) return { ok: true, value: candidate.value, attempts: attempt, trace };

    prompt = `${task}\n\nThe previous object failed validation:\n- ${errors.join('\n- ')}\nReturn a corrected JSON object only.`;
  }

  return { ok: false, error: 'validation_budget_exhausted', attempts: MAX_REPAIRS + 1, trace };
}

function parseModelJson(raw) {
  const text = String(raw ?? '').trim().replace(/^```
{% endraw %}
(?:json)?\s*|\s*
{% raw %}
```$/g, '');
  if (!text) return { ok: false, error: 'empty response' };
  try {
    return { ok: true, value: JSON.parse(text) };
  } catch (error) {
    return { ok: false, error: `json parse failed: ${error.message}` };
  }
}

Fixtures the suite must contain

clean.json — a valid object on the first attempt; the loop must return attempts: 1.
fenced.md — the same object wrapped in a JSON code fence; the loop must not consume a repair attempt.
bad-enum.json — a severity value outside the enum followed by a corrected object.
truncated.json — an unterminated object on every attempt; the loop must return validation_budget_exhausted.

The truncated case is the one reviewers should weigh most heavily, because it proves the candidate can stop. A loop that never gives up will pass every manual demo and fail in production under a degraded provider.

Where a free endpoint changes the exercise

Offline fixtures prove the control flow, but they say nothing about how a model reacts to an error block it has never seen. Running the same loop against a live endpoint exposes the difference between a repair prompt that names the failing path and one that restates the whole schema. Free model access is enough for this stage, since the volume is a few hundred short calls and the goal is drift, not throughput.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes free model access, a free server option, and a free allotment of ten million tokens; check the project page for current terms, because availability and limits change and this article does not restate them as permanent. The free server option matters for a different reason: it lets a reviewer hit the endpoint over HTTP instead of running the candidate's laptop, which makes the attempt logs observable during the review itself.

Common failure modes

Grading the first response. Candidates who score their own work on one clean sample will report success that the fixtures disprove.
Repair prompts that resend everything. Appending the schema, the task, and the candidate's full history inflates each attempt without improving the fix.
Silent fence stripping. A permissive regex that strips anything backticked hides parse failures instead of classifying them.
Unbounded loops. A while (true) around generation looks resilient and is the most common reason for a rejected submission.
Validator pass treated as truth. Schema conformance is not semantic correctness; a plausible ticket_id can still be wrong.
No attempt telemetry. Without per-attempt errors, nobody can tell whether repair failed or was never tried.

Who should not use this exercise

Teams hiring for UI-heavy roles get little signal here, and teams without a stable schema should fix the schema first. Candidates on constrained hardware should not be asked to host a model locally, since the graded artifact is the loop rather than the runtime. Anyone who needs deterministic byte-for-byte output should use a grammar-constrained decoder instead of a repair loop, because repair is a mitigation and not a guarantee.

If the goal is to see how someone behaves when a model misbehaves in a plausible way, this packet is cheap to run and hard to fake. The free model access and free server option are the shortest path to a live version of it; keep the rubric and the failure modes, and adjust the schema to your own domain.

The Portability Audit Take-Home: Grade Environment Assumptions, Not Features

Riley Zhu — Sun, 18 Oct 2026 07:26:02 +0000

A take-home that runs on the reviewer's laptop grades features; a take-home that runs anywhere grades engineering discipline. The task below asks a candidate to ship a small service and a machine-checkable statement of every environment assumption that service makes. Graders run the checker first, so a broken deployment cannot hide behind a polished README. The artifact fits a two-hour budget and separates candidates who reason about portability from those who happen to own the same machine as the reviewer.

The prompt handed to candidates

Build a small HTTP endpoint that turns a short free-text note into a structured record, then ship a runtime contract listing every environment assumption the service makes. The endpoint may call an external model API, but the service must start and answer with a degraded result when that API is unreachable. The deliverable is a repository with a one-command local run, a one-command audit, and a short report distinguishing assumptions that were removed from assumptions that were merely declared.

Requirements given verbatim in the prompt:

npm start (or make run) serves the endpoint on a port read from the environment, with no hardcoded fallback that silently binds elsewhere.
node tools/audit.mjs exits 0 on a clean tree and 1 when an undeclared assumption exists.
node --test passes with outbound network blocked.
Every file write lands under a directory listed in the contract.
The report names one assumption the candidate deleted outright, not only annotated.

Requirement five is the discriminating one. Declaring a secret dependency is bookkeeping; deleting a clock read is engineering.

The deliverable: one contract file, one checker

The contract is deliberately boring JSON, because reviewers must be able to diff two candidates' files in seconds. A realistic submission for this task looks like the following file, which states that the service reads three variables, writes only to ./tmp/, talks to exactly one host, and receives its clock and randomness by injection.

{
  "env": ["PORT", "MODEL_BASE_URL", "MODEL_API_KEY"],
  "hosts": ["model.example-host.invalid"],
  "writes": ["./tmp/"],
  "clock": "injected",
  "random": "injected"
}

The checker walks first-party source, matches a small rule set, and compares matches against the contract. It reports two classes of problem: findings that are not declared, and declarations that are never used. The second class matters because a contract padded with speculative variables tells the reviewer nothing about the running system.

#!/usr/bin/env node
// tools/audit.mjs — Node 20+, no dependencies.
import { readFile, readdir } from "node:fs/promises";
import { join, extname } from "node:path";

const ROOTS = ["src", "tools"];
const EXTS = new Set([".js", ".mjs", ".cjs", ".ts", ".py"]);
const SKIP = new Set(["node_modules", ".git", "coverage"]);

const RULES = [
  { kind: "env",    id: "env.static",  re: /process\.env\.([A-Z][A-Z0-9_]*)/g, capture: 1 },
  { kind: "hosts",  id: "net.literal", re: /https?:\/\/([a-z0-9.-]+)/gi,        capture: 1 },
  { kind: "writes", id: "fs.write",    re: /\b(?:writeFile|appendFile|mkdir|createWriteStream)\s*\(/g, capture: 0 },
  { kind: "clock",  id: "clock.now",   re: /\bDate\.now\s*\(/g,                capture: 0 },
  { kind: "random", id: "random.math", re: /\bMath\.random\s*\(/g,             capture: 0 },
];

async function walk(dir, out = []) {
  for (const entry of await readdir(dir, { withFileTypes: true })) {
    if (SKIP.has(entry.name)) continue;
    const p = join(dir, entry.name);
    if (entry.isDirectory()) await walk(p, out);
    else if (EXTS.has(extname(entry.name))) out.push(p);
  }
  return out;
}

const contract = JSON.parse(await readFile("runtime.contract.json", "utf8"));
const declared = {
  env: new Set(contract.env ?? []),
  hosts: new Set(contract.hosts ?? []),
  writes: new Set(contract.writes ?? []),
  clock: new Set(contract.clock === "injected" ? ["<injected>"] : []),
  random: new Set(contract.random === "injected" ? ["<injected>"] : []),
};

const files = (await Promise.all(ROOTS.map((r) => walk(r)))).flat();
const findings = [];

for (const file of files) {
  const text = await readFile(file, "utf8");
  for (const rule of RULES) {
    for (const m of text.matchAll(rule.re)) {
      const value = rule.capture ? m[rule.capture] : "<injected>";
      const line = text.slice(0, m.index).split("\n").length;
      findings.push({ file, line, kind: rule.kind, rule: rule.id, value });
    }
  }
}

const errors = [];
for (const f of findings) {
  if (f.kind === "clock" || f.kind === "random") {
    if (declared[f.kind].size === 0) {
      errors.push({ ...f, why: `${f.kind} used but not declared as injected` });
    }
    continue;
  }
  const ok = [...declared[f.kind]].some((d) => f.value.startsWith(d));
  if (!ok) errors.push({ ...f, why: `${f.kind} "${f.value}" missing from runtime.contract.json` });
}

const used = new Set(findings.map((f) => `${f.kind}:${f.value}`));
const warnings = [...declared.env]
  .filter((e) => !used.has(`env:${e}`))
  .map((e) => ({ kind: "env", value: e, why: "declared but never read" }));

console.log(JSON.stringify({ scanned: files.length, findings: findings.length, errors, warnings }, null, 2));
process.exit(errors.length ? 1 : 0);

The paired test proves the degraded path exists rather than asserting that it was intended. Pointing the base URL at a closed loopback port is the simplest way to force a fast connection refusal inside CI.

// test/degraded.test.mjs — run with: node --test
import test from "node:test";
import assert from "node:assert/strict";

test("answers with a degraded record when the model host refuses", async () => {
  process.env.MODEL_BASE_URL = "http://127.0.0.1:9"; // closed port, refuses immediately
  const { handle } = await import("../src/handler.mjs");
  const res = await handle({ text: "ship the audit on friday" });
  assert.equal(res.status, "degraded");
  assert.ok(res.record.title.length > 0);
  assert.equal(res.record.confidence, "unstructured");
});

Reviewers run three commands, in this order, and stop at the first failure:

node tools/audit.mjs; echo "audit_exit=$?"
node --test
network_block node --test   # or: docker run --rm --network none -v "$PWD":/app -w /app node:20 node --test

The middle command proves the suite does not depend on the outer network. The third proves the service does not either.

The rubric

Scores are weighted toward the deterministic parts, because those are the parts a hiring decision can defend later.

Weight	Criterion	Passing signal
20	Contract accuracy	Every finding declared; no speculative entries left behind
25	Checker behaviour	Undeclared write or host produces exit `1` with a file and line
20	Degraded path	Endpoint returns a typed degraded record, never a hang or a 500
15	Test portability	Suite passes with egress blocked
10	Deletion report	Names one assumption removed, with the diff
10	One-command run	Fresh clone to answering endpoint without manual edits

Anything below sixty is usually a candidate who wrote a working feature and skipped the contract entirely. That is a legitimate result, not a trick.

Common failure modes

Regex auditing that misses dynamic reads. process.env[${name}_KEY] never matches a static rule. The honest fix is to route all environment access through one module and add the checker rule env.dynamic, flagged as a warning that requires a comment.
Auditing node_modules. Scans that descend into dependencies produce hundreds of findings and a checker nobody keeps green. Restrict roots to first-party directories.
Hardcoding an allowlist to make the build pass. A contract containing every host in the repository is a suppression file, not a declaration. Reviewers should diff contract size against feature size.
Hidden wall-clock reads. Date.now() inside a token expiry check makes tests order-dependent. Injection through a clock parameter removes the read rather than documenting it.
Writes escaping the declared directory. Log files landing in the repo root break the second run in a read-only container. Use an explicit temp path resolved once at startup.
A degraded path that is really a crash. Returning 500 with a message is not degradation. The record must still satisfy its own schema, with a confidence field that tells the caller what it got.

Failure modes five and six tend to appear together, because both come from treating the environment as something that will behave.

Where free model access and a free server genuinely fit

Two parts of this task can use a hosted model without weakening the grading contract. The first is triage: ambiguous findings such as a dynamic environment read need a human-readable explanation, and a model is good at drafting one. The second is demonstration: a public endpoint shows the degraded path without asking a reviewer to install anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode advertises a free tier that includes model access, announced at the time of writing as ten million tokens, plus a free server option. Treat both numbers as vendor claims and re-check the current terms before planning a hiring round around them, because free-tier limits are the part of any announcement most likely to change. The safe pattern is to use either resource only off the grading path, with a recorded fixture between them.

# Model output is recorded once, then replayed; grading never calls the network.
node tools/triage.mjs --audit audit.json --record fixtures/triage.json
node tools/triage.mjs --audit audit.json --replay fixtures/triage.json --require-exit 0

Limitations are worth stating plainly. A recorded triage file goes stale as soon as the checker rules change, so it must be regenerated deliberately rather than silently. A free server is a demo surface, not a grading environment; the exit code must come from the local checker. Model judgment never gates the build, because a nondeterministic reviewer cannot defend a hiring decision.

This exercise is a poor fit for cohorts under three hours, for roles where deployment belongs to a platform team, and for reviewers who cannot pin a fixture before scoring. In those settings the same ideas are better tested in a live session with a prepared repository.

What a strong submission sounds like

A strong submission says: the audit found four undeclared assumptions, I removed the clock read, I declared the two environment variables, and I narrowed file writes to a temp directory resolved at startup. A weak submission says the service works locally. If this prompt is being rehearsed before a real hiring round, the free tier and free server option are a low-cost place to dry-run it once before committing paid infrastructure.

The Cold-Start Budget Packet: A Take-Home Where the Free Server Is Allowed to Vanish

Riley Zhu — Sat, 17 Oct 2026 06:26:00 +0000

A take-home that never simulates a host restart measures almost nothing about cost discipline. The packet below asks one question: after the machine disappears mid-run, does the candidate's harness still know what it spent, and does it refuse to spend more than it can account for? Everything else in the exercise, including the model calls themselves, is scaffolding around that single property.

This version is written for interviewers who hand out agent or LLM-harness tasks, and for candidates who want to know what a strong submission looks like before they open an editor. It assumes free-tier model access and a free server are available to whoever runs the packet, which is exactly the situation where hard budgets and restart behaviour stop being theoretical.

Why the constraint belongs in the interview

A paid API key with a generous ceiling hides the two decisions that separate a careful harness from a fragile one. The first decision is where the spend record is written relative to the network call, and the second is what happens to that record when the process dies between them. Candidates who have only worked against unmetered internal gateways tend to answer both questions by accident rather than by design.

Free tiers make the failure legible without asking anyone to burn money. They also introduce realistic instability: a shared host can be recycled, and a client that has never checkpointed will silently re-pay for every call it already made. The packet turns that instability into a graded requirement instead of an unpleasant surprise in week two.

The prompt handed to the candidate

The candidate receives a repository containing three JSON tasks, a stub transport that returns canned text, and a README. The written brief is deliberately short and numbered, because long briefs let candidates negotiate with ambiguity instead of designing for it.

Implement runTask(task, transport) so that the total token spend never exceeds the cap in MAX_TOKENS.
Persist enough state that a second process invocation reuses earlier results instead of re-calling the transport.
Make an interrupted call visible as spent budget after restart, even though it produced no usable output.
Fail loudly with a stable error code when the cap cannot accommodate the next call.
Ship a test that kills the transport mid-call and proves points 2 and 3 with assertions, not prose.

The candidate may run this against any provider that offers free model access, or against the included stub if no network is reachable. A free server option is acceptable as the execution host, provided the harness survives that host being restarted; the brief says explicitly that the host is assumed to be disposable.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The rubric, out of twenty points

The scoring table is published to the candidate up front, because hidden rubrics reward guessing and punish engineering. Reviewers score each row independently and record the file and line that justify the score.

Points	Requirement	What earns full credit
5	Budget correctness	Spend is summed from a durable log, and the cap check happens before the call
5	Restart behaviour	Re-running after a kill reuses committed output and does not re-call
4	Interrupted-call accounting	An unmatched reservation counts as spent after restart
3	Error taxonomy	Distinct codes for `budget_exhausted`, `transport_failed`, and `invalid_task`
3	Test quality	A test kills the transport and asserts both reuse and burned spend

Two submission styles score badly on this table even when the demo looks good. An in-memory counter passes a single happy-path run and fails row one the moment the process restarts. A try/catch around everything passes row three only in appearance, because a swallowed error turns an unknown spend into a silent one.

A reference solution: reserve, commit, resume

The shape below is the implementation an interviewer can keep private and compare against. It uses one append-only JSONL file, writes a reservation before the call, and writes a commit after it, so an interruption leaves a witness rather than a gap. The transport is injected, which keeps the packet runnable offline.

// budget-runner.mjs — Node 20+, zero dependencies
import { createHash } from 'node:crypto';
import { appendFileSync, existsSync, readFileSync } from 'node:fs';

const LEDGER = process.env.LEDGER ?? 'ledger.jsonl';
const CAP = Number(process.env.MAX_TOKENS ?? 20000);

const taskKey = (task) =>
  `${task.id}:${createHash('sha256').update(task.prompt).digest('hex').slice(0, 12)}`;

function readLedger(path) {
  if (!existsSync(path)) return { spent: 0, committed: new Map(), rows: [] };
  const rows = readFileSync(path, 'utf8').split('\n').filter(Boolean).map((l) => JSON.parse(l));
  const committed = new Map(); // committed.get(key) -> output
  const open = new Map();      // reservation id -> tokens
  let spent = 0;
  for (const row of rows) {
    if (row.type === 'reserve') open.set(row.id, row.tokens);
    if (row.type === 'commit') {
      open.delete(row.id);
      committed.set(row.key, row.output);
      spent += row.tokens;
    }
  }
  for (const tokens of open.values()) spent += tokens; // burned by an interrupted call
  return { spent, committed, rows };
}

export async function runTask(task, transport) {
  const key = taskKey(task);
  const { spent, committed, rows } = readLedger(LEDGER);
  if (committed.has(key)) return { output: committed.get(key), source: 'ledger' };

  const estimate = transport.estimate(task.prompt);
  if (spent + estimate > CAP) {
    throw new Error(`budget_exhausted spent=${spent} need=${estimate} cap=${CAP}`);
  }

  const id = `${key}#${rows.filter((r) => r.type === 'reserve').length + 1}`;
  appendFileSync(LEDGER, JSON.stringify({ type: 'reserve', id, key, tokens: estimate }) + '\n');

  const res = await transport.call(task.prompt); // the process may die here

  appendFileSync(
    LEDGER,
    JSON.stringify({ type: 'commit', id, key, tokens: res.tokens, output: res.text }) + '\n'
  );
  return { output: res.text, source: 'call' };
}

The reservation is an estimate and the commit carries the measured count, which keeps the accounting conservative without pretending the client can predict usage exactly. A candidate who explains that trade-off in the README, rather than silently picking one number, should score higher on row four than one who hard-codes a constant.

The restart test that decides the review

A single test file separates submissions that reasoned about interruption from submissions that reasoned about the happy path. The stub throws on the first call, simulating a recycled host, and the assertions cover reuse, burned spend, and the cap refusal that follows.

// restart.test.mjs — run with: node --test
import test from 'node:test';
import assert from 'node:assert/strict';
import { rmSync } from 'node:fs';
import { runTask } from './budget-runner.mjs';

const stub = (crashOnCall) => {
  let calls = 0;
  return {
    estimate: () => 100,
    async call() {
      calls += 1;
      if (calls === crashOnCall) throw new Error('transport_failed');
      return { text: `answer-${calls}`, tokens: 100 };
    },
  };
};

test('a killed call stays visible in the budget', async () => {
  process.env.LEDGER = 'test-ledger.jsonl';
  process.env.MAX_TOKENS = '250';
  rmSync('test-ledger.jsonl', { force: true });

  await assert.rejects(() => runTask({ id: 't1', prompt: 'summarise the diff' }, stub(1)));

  const retry = await runTask({ id: 't1', prompt: 'summarise the diff' }, stub(99));
  assert.equal(retry.source, 'call');

  await assert.rejects(
    () => runTask({ id: 't2', prompt: 'draft the changelog' }, stub(99)),
    /budget_exhausted/
  );
});

Run node --test restart.test.mjs and read the ledger afterwards. A correct submission shows three rows: an unmatched reservation, a commit, and a refusal, with the refusal caused by the burned reservation rather than by an arbitrary counter. A broken submission shows a clean commit and no memory of the first attempt, which is the exact bug that turns a free tier into a surprise invoice on a paid one.

Failure modes that recur in review

Counters in memory. The budget resets with the process, so a restart loop spends without limit while every individual run looks compliant.
Reserving after the call. The cheapest possible mistake to make and the most expensive to debug, because the only record of an interrupted call is its absence.
Index-keyed caches. Results stored by array position silently mismatch once tasks are reordered or filtered, and the test still passes.
Blanket error handling. Catching every exception and continuing converts a known failure into an unknown spend, which no ledger can repair later.
Backoff against a cap. Retrying a budget refusal with exponential delay wastes wall time without ever making the call admissible.
Live dependencies in CI. Pointing tests at a shared host makes the suite flaky and hides whether the candidate's own restart logic works.

Each of these maps to a rubric row, which is why the table is worth publishing: the candidate can self-review against the same list the reviewer uses.

Limitations and who should not use this packet

A single-writer JSONL ledger is not concurrency-safe, so the packet is unsuitable for teams that want to grade multi-process throughput or distributed scheduling. Token estimation is approximate by nature, and a strict cap will occasionally refuse a call that would have fit, which is an acceptable trade for determinism in an interview.

Organisations that require private data residency, contractual SLAs, or audited retention should run the transport stub only and keep candidate code entirely local. Anyone evaluating prompt-engineering taste rather than harness engineering should pick a different exercise, because this one grades bookkeeping and recovery, not phrasing.

Finally, treat every availability claim as perishable. Free allowances, host lifetimes, and API surface change without notice, so the README should instruct candidates to confirm the current terms in the provider's documentation on the day they run the packet, and should state plainly that the exercise does not depend on any of those numbers staying fixed.

Where the free tier actually helps

The reason this packet is runnable by anyone is boring and useful: MonkeyCode is an open-source project that offers free model access and a free server option, so an interviewer can hand a candidate a path to real calls without provisioning a paid key first. That matters for the restart test, because a disposable host is precisely the environment the rubric describes, and instructions to "assume the machine can vanish" stop being hypothetical.

The honest boundary is that free access is a convenience for exercises and prototypes, not a capacity plan. If your evaluation needs guaranteed uptime, fixed model versions, or data-handling commitments, the packet should still use the stub and the real provider choice should be made separately.

If you want to try the exercise end to end, the free model access and free server option are enough to get one candidate through the full loop, and the ledger will tell you more about their engineering than the demo ever will.

The Disconnect Test: A Take-Home Where Cancelling Upstream Is the Real Requirement

Riley Zhu — Fri, 16 Oct 2026 05:29:50 +0000

A streaming proxy take-home rarely fails on the happy path, because almost every candidate can pipe bytes from an upstream endpoint into an HTTP response. The graded behaviour appears when the client disappears halfway through the stream and the upstream request has to die with it. Most submissions leak that upstream call, keep generating tokens that nobody will read, and never notice, because their own test suite exercises only the successful case. This article gives the prompt, a reference solution, a weighted rubric, and the failure modes worth grading.

The prompt handed to candidates

The exercise stays intentionally small: one route, one upstream dependency, one process. Candidates receive the following brief and roughly two hours to return code plus a short written note.

Implement POST /chat as a streaming proxy in front of an OpenAI-compatible /v1/chat/completions endpoint.

Stream the upstream body to the client without buffering the whole response.

Abort the upstream request when the client disconnects.

Retry only before the first byte reaches the client, never afterwards.

Use an idle timeout rather than a total-duration timeout.

Emit one structured log line per request with status, bytes, and outcome.

Ship a test that proves the cancellation path without network access.

The third and fourth bullets carry most of the signal, because they encode a specific mental model of stream lifetime. A candidate who treats the upstream call as a fire-and-forget side effect will pass everything else and fail those two. The deliverable is deliberately narrow so that graders compare designs rather than feature counts.

Why cancellation is the graded behaviour

Every leaked upstream request keeps a socket open, holds a slot on a shared endpoint, and continues generating tokens that no user will read. On a metered endpoint that leak shows up directly on the invoice, and on a shared free endpoint it degrades the experience for everyone else using the same capacity. Cancellation is therefore not a nicety but the difference between a proxy that scales and one that quietly accumulates debt.

The same reasoning explains the no-retry-after-first-byte rule. Once a token has reached the client, a retry produces a second stream spliced into the first, which surfaces as duplicated or interleaved text. Graders look for that duplication because it is one of the few failures a reviewer can spot from outside the process.

The one-line test that separates candidates

Two counters on a fake upstream settle the question faster than any code review. Start a stream, abort the client after a couple hundred milliseconds, then read the counters back.

curl -s localhost:8787/__stats   # {"started":2,"completed":1,"aborted":1}

An aborted count that matches the number of killed clients means the design is sound at the process boundary. A zero there means the upstream call outlived its consumer, and the rest of the submission is mostly commentary.

Reference solution: Node 20, no dependencies

The reference uses the built-in fetch, one AbortController per upstream attempt, and a deterministic fake upstream so the tests never touch a real endpoint. It is a spec-complete sketch rather than a polished service, so run it locally before you grade anyone with it.

Step 1: a deterministic fake upstream

// fake-upstream.mjs
import http from "node:http";

const state = { started: 0, completed: 0, aborted: 0 };

http.createServer((req, res) => {
  if (req.url === "/__stats") {
    res.writeHead(200, { "content-type": "application/json" });
    res.end(JSON.stringify(state));
    return;
  }
  if (req.url !== "/v1/chat/completions") {
    res.writeHead(404).end();
    return;
  }

  state.started++;
  let finished = false;
  res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });

  let i = 0;
  const timer = setInterval(() => {
    if (i === 40) {                       // 40 chunks every 25 ms => ~1 s stream
      clearInterval(timer);
      finished = true;
      state.completed++;
      res.write("data: [DONE]\n\n");
      res.end();
      return;
    }
    res.write(`data: {"choices":[{"delta":{"content":"tok${i} "}}]}\n\n`);
    i++;
  }, 25);

  res.on("close", () => {                 // fires for a clean end and for a client abort
    if (finished) return;
    clearInterval(timer);
    state.aborted++;
  });
}).listen(8787, () => console.log("fake upstream on :8787"));

Step 2: the proxy

// proxy.mjs
import http from "node:http";

const UPSTREAM = process.env.UPSTREAM ?? "http://127.0.0.1:8787";
const IDLE_TIMEOUT_MS = 2000;
const MAX_RETRIES = 2;

const readBody = (req) => new Promise((resolve, reject) => {
  const chunks = [];
  req.on("data", (c) => chunks.push(c));
  req.on("end", () => resolve(Buffer.concat(chunks)));
  req.on("error", reject);
});
const once = (emitter, event) => new Promise((r) => emitter.once(event, r));
const backoff = (n) => new Promise((r) => setTimeout(r, 50 * 2 ** n));

http.createServer(async (req, res) => {
  if (req.method !== "POST" || req.url !== "/chat") return void res.writeHead(404).end();

  const body = await readBody(req);
  const clientGone = new AbortController();
  // Bind the upstream lifetime to the response socket, not to the request body.
  res.on("close", () => { if (!res.writableEnded) clientGone.abort(new Error("client_gone")); });

  for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
    const upstream = new AbortController();
    const onClientGone = () => upstream.abort(new Error("client_gone"));
    clientGone.signal.addEventListener("abort", onClientGone, { once: true });

    const idle = setTimeout(() => upstream.abort(new Error("idle_timeout")), IDLE_TIMEOUT_MS);
    let firstByteSent = false;

    try {
      const up = await fetch(`${UPSTREAM}/v1/chat/completions`, {
        method: "POST",
        headers: { "content-type": "application/json", accept: "text/event-stream" },
        body,
        signal: upstream.signal,
      });

      if (up.status >= 500 && attempt < MAX_RETRIES) {
        clearTimeout(idle);
        await backoff(attempt);
        continue;
      }

      res.writeHead(up.status, {
        "content-type": up.headers.get("content-type") ?? "text/event-stream",
        "cache-control": "no-cache",
      });

      for await (const chunk of up.body) {
        idle.refresh();                                   // idle, not wall-clock
        firstByteSent = true;
        if (!res.write(chunk)) await once(res, "drain");   // respect backpressure
      }
      clearTimeout(idle);
      res.end();
      return;
    } catch (err) {
      clearTimeout(idle);
      if (firstByteSent || clientGone.signal.aborted) return void res.end();
      if (attempt === MAX_RETRIES) return void res.writeHead(502).end("upstream_unavailable");
      await backoff(attempt);
    } finally {
      clientGone.signal.removeEventListener("abort", onClientGone);
    }
  }
}).listen(3000, () => console.log("proxy on :3000"));

Two details in that file deserve explicit attention. The cancellation listener attaches to res, because req emits close when the request body finishes uploading and would abort every healthy stream. The retry guard checks firstByteSent, because a retry after the first chunk is a correctness bug rather than a resilience feature.

Step 3: the assertion script

node fake-upstream.mjs & UP=$!
node proxy.mjs &        PX=$!
sleep 0.3

# Happy path: the stream must reach [DONE] and count as completed.
curl -sN -X POST localhost:3000/chat -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hi"}]}' -o /dev/null
sleep 0.15

# Disconnect test: abort 200 ms into a roughly one-second stream.
curl -sN -X POST localhost:3000/chat -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hi"}]}' -o /dev/null --max-time 0.2 || true
sleep 0.15

curl -s localhost:8787/__stats
# expected: {"started":2,"completed":1,"aborted":1}
kill $UP $PX

The rubric

#	Criterion	Weight	Pass condition
1	Upstream cancelled on client disconnect	25	`aborted` increments within a few hundred milliseconds of the client abort
2	No retry after the first byte	20	no duplicated `tok` fragments appear in the client output
3	Backpressure respected	15	the proxy awaits `drain` instead of buffering the whole stream
4	Idle timeout, not a total timeout	10	a slow but live stream survives longer than the timeout window
5	Bounded retries before the first byte	10	repeated 5xx leads to at most two retries, then a 502
6	Deterministic test included	10	the suite passes with the fake upstream and no network access
7	Written trade-off note	10	the candidate names one real limitation of their own design

Criterion one carries the largest weight by design, and criterion seven breaks the tie between two submissions that both pass the harness. Neither is a style preference, since both map onto production incidents that a streaming service will eventually experience.

Common failure modes

Listening for disconnect on the request instead of the response. req.on("close") fires when the request body has been fully read, so every healthy stream looks abandoned and real users get dropped mid-answer.
Cancelling the client but not the upstream. The response socket closes and the handler returns, while the fetch promise keeps its connection open until the upstream finishes on its own schedule.
Retrying after partial delivery. A 500 that arrives mid-stream triggers a second attempt, and the client sees two interleaved token sequences with no way to tell them apart.
Buffering the whole upstream body before writing. This passes the cancellation test and fails the product, because time to first byte equals full generation time and memory grows with response length.
Ignoring the return value of res.write(). Without awaiting drain, a slow consumer causes unbounded buffering inside the process, which looks like a memory leak rather than a backpressure bug.
Timing out on total duration. A hard wall-clock limit kills legitimate long generations while leaving genuinely stuck streams alive until the limit finally expires.
Testing against a real endpoint. Output drifts, the suite becomes flaky, and the cancellation assertion ends up depending on network timing that no reviewer can reproduce.

Where free model access and a free server fit the exercise

The harness above is deliberately offline, but reviewers want a URL they can open and candidates need an endpoint that costs nothing to experiment against. The MonkeyCode project, which the operator of this account describes as open source, is presented as offering free model access and a free server option for exactly this kind of small deployment. The operator states an allowance on the order of ten million tokens together with a free server, so pointing UPSTREAM at that endpoint and running proxy.mjs on the free instance makes the demo reachable without a card on file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats belong next to that recommendation. Free tiers change over time, so confirm the current allowance and terms on the project's own page instead of trusting a figure quoted in any blog post, including this one. A free allowance is also sized for harnesses and demos rather than load testing, so keep the fake upstream as the assertion target and treat the live endpoint as a smoke test.

A decision table

Situation	Fake upstream	Free endpoint plus free server
Contract tests for cancellation and backpressure	Required	Optional
A demo URL a reviewer can open	Not possible	Useful
Comparing model output quality across weeks	Not applicable	Unsuitable, outputs drift
Sustained production traffic	Not applicable	Unsuitable
Measuring your own container's cold-start latency	Not applicable	Workable
An interview take-home that must cost nothing to grade	Required	Useful for the demo step

Limitations and who should skip this

The reference solution is a sketch rather than a hardened service, since it has no authentication, no per-client rate limiting, and no request-size cap.
The fake upstream asserts transport behaviour only, so it says nothing about whether a model's answers are useful, safe, or on-brand.
Teams with data-residency or audit obligations should not route interview material through any third-party endpoint, free or paid.
Candidates who cannot share a public URL can submit the offline harness alone, because the cancellation test still carries the grade.

If you want a live endpoint behind the proxy during review, the free access described above covers a harness of this size, while the fake upstream remains the source of truth for pass and fail.

Replay the Patch, Not the Chat: A Take-Home for Agent Sessions

Riley Zhu — Thu, 15 Oct 2026 11:32:21 +0000

The deliverable is a patch pipeline, not a transcript

AI coding agents produce persuasive sessions and unreproducible results. A candidate who submits a chat log has demonstrated nothing that a reviewer can re-run. The exercise below asks for something narrower and far more useful: a recorded tool trace plus a replay harness that rebuilds the final file states on a clean checkout.

Two constraints keep the task honest and comparable across candidates. The first is that no paid inference and no paid compute may appear in the submitted workflow, because a take-home should not depend on a candidate's credit card. The second is that every claimed fix must survive three independent replays from the pinned fixture, or it counts as unverified.

The interview brief

Scenario

A reviewer hands the candidate a small repository with one failing regression test and a written description of the desired behavior. The candidate may use an AI coding agent freely to reach a fix, but must hand back evidence rather than narration.

Constraints

The agent session must run against free model access, so the environment is reproducible for reviewers without a budget.
Any server-side step must run on a free server option rather than a candidate-provisioned machine.
The fixture is pinned by commit SHA, and the trace must record that SHA alongside every tool call.
No secrets, tokens, or proprietary paths may appear in the submitted trace.

Deliverables

trace/toolcalls.jsonl — one JSON object per tool call, in execution order.
harness/ — recorder, replay runner, and determinism scorer.
tests/ — at minimum one test that fails before the fix and passes after it.
README.md — what was fixed, what was assumed, and what the harness cannot prove.

MonkeyCode is relevant here mainly as the cheapest way to satisfy the first two constraints. Its free model access gives the exploration loop a home, and its free server option removes the need for a candidate-owned box. The allowance and the available models change over time, so verify the current terms before you write them into a rubric.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the trace must contain

A trace is only replayable if each entry describes an effect on the filesystem or a process invocation. Conversational text belongs in a separate log and never in the trace.

Each line should carry these fields:

kind — one of read, write, exec, test.
payload — the concrete arguments, such as a target path and full content, or an argv array.
files — a map of touched paths to short content digests at that moment.
fixture_sha — the pinned commit the call was made against.

# harness/record.py
import hashlib, json, pathlib, time

TRACE = pathlib.Path("trace/toolcalls.jsonl")

def digest(p):
    return hashlib.sha256(pathlib.Path(p).read_bytes()).hexdigest()[:16]

def record(kind, payload, fixture_sha, files=()):
    entry = {
        "t": round(time.time(), 3),
        "kind": kind,
        "fixture_sha": fixture_sha,
        "payload": payload,
        "files": {str(f): digest(f) for f in files},
    }
    with TRACE.open("a") as fh:
        fh.write(json.dumps(entry, sort_keys=True) + "\n")
    return entry

The rubric

Weight	Criterion	What the reviewer inspects
20%	Trace completeness	Every write and exec has a corresponding entry
20%	Reproducibility	Three replays yield identical file digests
15%	Failure surfacing	Non-zero exit and a diff when replay diverges
15%	Constraint discipline	No paid inference or paid compute in the path
15%	Test quality	A failing test exists before the patch is applied
15%	Written limits	README states what the harness does not prove

A submission that scores well on reproducibility but omits the failing test is still a weak submission. The rubric deliberately rewards the smallest defensible fix, not the largest plausible refactor.

A reference solution

The layout below is deliberately small enough to review in one sitting. It separates recording, replay, and scoring so each stage can fail independently.

agent-handover/
  fixture/                 # pinned checkout
  trace/toolcalls.jsonl
  trace/fixture.sha
  harness/record.py
  harness/replay.py
  harness/score.py
  tests/test_regression.py
  README.md

# harness/replay.py
import json, pathlib, shutil, subprocess, sys

def load_trace(path):
    lines = pathlib.Path(path).read_text().splitlines()
    return [json.loads(line) for line in lines if line.strip()]

def apply_entry(entry, workdir):
    if entry["kind"] != "write":
        return
    target = pathlib.Path(workdir) / entry["payload"]["path"]
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(entry["payload"]["content"])

def run_commands(trace, workdir):
    for entry in trace:
        if entry["kind"] == "exec":
            subprocess.run(entry["payload"]["argv"], cwd=workdir, check=True)

def main(trace_path, out_dir):
    trace = load_trace(trace_path)
    shutil.rmtree(out_dir, ignore_errors=True)
    shutil.copytree("fixture/", out_dir)
    for entry in trace:
        apply_entry(entry, out_dir)
    run_commands(trace, out_dir)

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

# harness/score.py
import hashlib, pathlib, sys

def digest(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()

def collect(root):
    root = pathlib.Path(root)
    return {str(p.relative_to(root)): digest(p)
            for p in root.rglob("*") if p.is_file() and ".git" not in p.parts}

def score(runs):
    snaps = [collect(r) for r in runs]
    keys = set.intersection(*(set(s) for s in snaps))
    stable = [k for k in keys if len({s[k] for s in snaps}) == 1]
    return len(stable) / max(len(keys), 1)

if __name__ == "__main__":
    value = score(sys.argv[1:])
    print(f"determinism score: {value:.2f}")
    sys.exit(0 if value == 1.0 else 1)

The run sequence is three replays plus a score, and the score doubles as the CI gate.

git -C fixture rev-parse HEAD > trace/fixture.sha
python harness/replay.py trace/toolcalls.jsonl replay/1
python harness/replay.py trace/toolcalls.jsonl replay/2
python harness/replay.py trace/toolcalls.jsonl replay/3
python harness/score.py replay/1 replay/2 replay/3

Normalize before you compare

Raw traces are noisy for reasons that have nothing to do with the fix. Normalize these sources of drift first, or the determinism score will be meaningless.

Absolute paths and timestamps, which differ per machine and per run.
Model sampling, which changes wording but should not change final file content.
Tool ordering, when two writes touch disjoint files and have no dependency.
Network responses, which should be captured once and replayed from disk.

The rule of thumb is simple: compare semantic effects on files, not the narrative that produced them.

Common failure modes

Recording prose instead of tool calls, which makes replay impossible and the trace unreadable.
Comparing chat transcripts across runs and calling that verification, which proves only that the model was verbose.
Pinning the fixture commit while leaving environment variables unpinned, so a fix passes locally and fails in review.
Reporting a partial score as success; a determinism score of 0.7 means at least one artifact is not reproducible.
Applying the patch by hand before replay, which hides the exact step the harness was meant to verify.
Writing no failing test first, which leaves the reviewer unable to distinguish a fix from a coincidence.
Copying credentials into the trace, which turns a tidy take-home into an incident report.

Limitations and who should skip this

This harness verifies that a patch is reproducible, not that it is correct. Flaky tests, clock-dependent behavior, and hardware-specific bugs will still escape it, and the scorer reports stability rather than quality. Teams with strict regulatory requirements should not route proprietary source through any hosted model tier, free or otherwise.

It is also a poor fit for agents whose value lies in long-horizon planning across many repositories, because a single pinned fixture cannot express that work. If the deliverable is a design document rather than a patch, this rubric will reward the wrong behavior.

Closing

The point of the exercise is that reviewers should be able to run one command and see the same repository state the candidate saw. Everything else, including the model and the machine, is replaceable. If you want to try the brief yourself, the free model access and free server option on MonkeyCode are enough to run the exploration loop, and python harness/score.py will tell you honestly whether you are done.

Stop on the Ledger: A Take-Home for Quota-Aware Clients

Riley Zhu — Tue, 13 Oct 2026 02:32:26 +0000

A strong inference client stops on a local ledger before a free endpoint spends another token. Hiring panels that score only the final completion miss the failure that matters once a shared pool runs dry. This packet grades the stop condition, the persisted usage record, and the typed outage path as one unit. The model answer is only evidence that wiring worked, and it is not the score itself.

What the packet measures

Free model access removes a credit-card step, and a free server removes a cluster step, yet neither gift removes a budget. A candidate who treats either path as unlimited will retry through exhaustion and hide the real cost. The exercise therefore asks for a small client that can target an operator-provided free endpoint without assuming that access lasts forever. Exact quotas, model identifiers, and server duration stay outside the packet and must be read from current project documentation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow below still stands if that product name is replaced by any compatible endpoint offering a bounded free path. Readers should confirm live limits in the current repository notes before they point a runner at that host. A missing limit in those notes is a reason to stay on the local stub, not a reason to guess a quota.

The assignment

The candidate receives a pinned fixture, a numeric budget, and a base URL that is supplied only at runtime. The client must refuse a call when the remaining ledger cannot cover that row's declared ceiling. After each accepted call, the client must persist provider-reported usage and subtract that total from the ledger. A transport failure, an HTTP 429, or a missing usage field must become a typed result rather than a silent retry.

Constraints that keep the review reproducible:

The fixture file remains immutable, and the reviewer rechecks its checksum before accepting the recorded run.
Secrets and endpoint keys arrive only through environment variables, never through the fixture or the ledger file.
The default budget is small enough that the third fixture row must stop before its network call is built.
Retries stay forbidden unless a documented transient code returns and the ledger still covers the full ceiling.
The process exits non-zero when a required row is skipped for a reason other than budget or typed outage.

Command surface

The expected command surface stays narrow so grading does not depend on a web framework or a hosted runner. A reviewer should run the same three commands against the local stub before considering any live host. The live path stays optional because a free server can be absent, throttled, or catalog-shifted on review day. That optionality is a deliberate feature of the packet, not an incomplete part of the assignment.

python -m budget_gate run \
  --fixture fixtures/prompts.jsonl \
  --ledger var/ledger.json \
  --budget-tokens 180 \
  --base-url http://127.0.0.1:8080/v1 \
  --model "$MODEL_ID"

Candidates must pass the model identifier from current documentation into the flag, rather than hardcoding it. A hardcoded name fails this packet even when the local stub still returns a plain HTTP 200. The variable exists so a later smoke test can follow a catalog change without editing the client.

Row contract

Each fixture row maps to one action, one ledger effect, and one exit class, with no blended statuses allowed. The table is the contract a second reviewer uses when the narrative in the README disagrees with the code. A submission that prints a friendly message but writes a different status still fails the row.

Observed condition	Required action	Ledger effect	Exit class
Remaining tokens below the row ceiling	Do not dispatch	Unchanged	`budget_stop`
HTTP 200 with a total_tokens field	Record the completion id and usage	Subtract the reported total	`ok`
HTTP 200 without usage	Do not trust a local estimate	Unchanged	`usage_missing`
HTTP 429 or a connection reset	No retry loop; one retry only if policy and ledger both allow it	Unchanged until a later success	`upstream_unavailable`
HTTP 401 or HTTP 403	Stop the remaining batch	Unchanged	`auth_rejected`

Fixture arithmetic

The pinned fixture uses declared ceilings so the gate does not depend on a tokenizer package. Row one allows 40 completion tokens and a prompt ceiling of 30, for a gate value of 70. Row two repeats that same shape, and row three uses a gate value of 120. With a budget of 180, two stub calls that report 40 tokens each leave 100 remaining, which sits below 120, so the third row must stop locally.

{"id":"r1","prompt":"Summarize the ledger rule in one sentence.","max_tokens":40,"prompt_ceiling":30}
{"id":"r2","prompt":"Name three typed stop reasons without calling a network.","max_tokens":40,"prompt_ceiling":30}
{"id":"r3","prompt":"Explain why a missing usage field must not decrement spend.","max_tokens":80,"prompt_ceiling":40}

Proposal code for the gate

The module below is a proposal for reviewers and candidates, not a recorded benchmark and not an executed trace. It shows the pre-dispatch gate, the ledger write, and the typed results a panel should be able to find. Another language is acceptable when the same decisions remain visible in code and in the ledger file. Streaming responses, tool calls, and concurrent writers stay intentionally out of scope for this first packet.

#!/usr/bin/env python3
'''Proposal: local budget gate for a fixture runner. Not an executed benchmark.'''

from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

@dataclass(frozen=True)
class RowResult:
    row_id: str
    status: str
    spent: int

def load_ledger(path: Path, budget: int) -> dict[str, Any]:
    if path.exists():
        return json.loads(path.read_text(encoding='utf-8'))
    return {'budget': budget, 'spent': 0, 'events': []}

def remaining(ledger: dict[str, Any]) -> int:
    return int(ledger['budget']) - int(ledger['spent'])

def gate(ledger: dict[str, Any], ceiling: int) -> str | None:
    if ceiling < 1:
        return 'bad_ceiling'
    if remaining(ledger) < ceiling:
        return 'budget_stop'
    return None

def apply_usage(ledger: dict[str, Any], row_id: str, usage: dict[str, Any]) -> None:
    total = int(usage['total_tokens'])
    ledger['spent'] = int(ledger['spent']) + total
    ledger['events'].append({'row_id': row_id, 'total_tokens': total})

def post_json(base_url: str, payload: dict[str, Any], api_key: str) -> dict[str, Any]:
    body = json.dumps(payload).encode('utf-8')
    request = Request(
        base_url.rstrip('/') + '/chat/completions',
        data=body,
        headers={
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + api_key,
        },
        method='POST',
    )
    with urlopen(request, timeout=20) as response:
        return json.loads(response.read().decode('utf-8'))

def run_row(ledger, row, base_url, model, api_key) -> RowResult:
    ceiling = int(row['max_tokens']) + int(row.get('prompt_ceiling', 0))
    blocked = gate(ledger, ceiling)
    if blocked:
        return RowResult(row['id'], blocked, 0)
    payload = {
        'model': model,
        'messages': [{'role': 'user', 'content': row['prompt']}],
        'max_tokens': int(row['max_tokens']),
    }
    try:
        data = post_json(base_url, payload, api_key)
    except HTTPError as exc:
        status = 'auth_rejected' if exc.code in (401, 403) else 'upstream_unavailable'
        return RowResult(row['id'], status, 0)
    except URLError:
        return RowResult(row['id'], 'upstream_unavailable', 0)
    usage = data.get('usage') or {}
    if 'total_tokens' not in usage:
        return RowResult(row['id'], 'usage_missing', 0)
    apply_usage(ledger, row['id'], usage)
    return RowResult(row['id'], 'ok', int(usage['total_tokens']))

def save_ledger(path: Path, ledger: dict[str, Any]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(ledger, indent=2), encoding='utf-8')

Local stub and unexecuted check

A local stub returns scripted usage so the third row crosses the cap without touching a shared pool. The stub belongs in the packet because a live free server may be down during the review window. Separating those two paths is the teaching point, not a temporary workaround for missing credentials.

#!/usr/bin/env python3
'''Local stub proposal. Returns usage until FAIL_AFTER, then HTTP 429.'''

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json

FAIL_AFTER = 2
SEEN = {'n': 0}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        length = int(self.headers.get('Content-Length', '0'))
        payload = json.loads(self.rfile.read(length) or b'{}')
        SEEN['n'] += 1
        if SEEN['n'] > FAIL_AFTER:
            body = {'error': {'message': 'fixture throttle', 'type': 'rate_limit'}}
            raw = json.dumps(body).encode('utf-8')
            code = 429
        else:
            used = int(payload.get('max_tokens', 32))
            body = {'id': 'stub-%d' % SEEN['n'], 'usage': {'total_tokens': used}}
            raw = json.dumps(body).encode('utf-8')
            code = 200
        self.send_response(code)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

if __name__ == '__main__':
    ThreadingHTTPServer(('127.0.0.1', 8080), Handler).serve_forever()

The check below is a proposal as well, and this article does not claim that anyone executed it. It locks the arithmetic that a reviewer should reimplement in the test runner they already trust. A passing stub log for row three shows no request when spent tokens are already 80 against a 180 budget. A failing submission still opens a socket and then explains the overspend after the response returns.

def test_third_row_stops_before_dispatch():
    ledger = {'budget': 180, 'spent': 80, 'events': []}
    row = {'id': 'r3', 'max_tokens': 80, 'prompt_ceiling': 40}
    ceiling = int(row['max_tokens']) + int(row['prompt_ceiling'])
    assert ceiling == 120
    assert gate(ledger, ceiling) == 'budget_stop'

Rubric

Reviewers should score observable client behavior, not the prose quality of whatever completion text comes back. Each line is pass or fail, and a pass requires the behavior in the ledger file as well as in the exit code. A fluent README cannot replace a missing status string that the persisted ledger file should contain.

The pre-dispatch gate blocks a row whose ceiling exceeds remaining tokens, and the stub log shows no request for that row.
Reported total_tokens is the only quantity subtracted, and a missing usage object leaves the spent field unchanged.
HTTP 429 and connection errors map to upstream_unavailable without a retry loop that keeps calling the host.
HTTP 401 and HTTP 403 stop the batch as auth_rejected and never print, rotate, or store the key.
The fixture checksum is unchanged, and the model identifier comes from configuration rather than a source constant.
A second run against the same ledger does not double-spend rows that were already recorded as ok.

Failure modes the rubric catches

The rubric treats the following outcomes as failures because each one hides spend or blocks a second reviewer. They are design risks written for this packet, not measured incidents from a named production system. A demo that looks smooth on the first row can still fail every line in the list below.

Estimating tokens locally and ignoring provider usage drifts as soon as the server tokenizer disagrees with the client.
Retrying HTTP 429 until the process is killed turns a free-server throttle into a self-inflicted outage for everyone sharing the host.
Treating a free host as always warm hangs the review on a live call when the packet already required a stub.
Writing the API key into the ledger, the traceback, or the fixture fails the secret rule even when the gate itself is correct.
Hardcoding a model name from an older README breaks the day the free catalog rotates, which this packet refuses to freeze.
Scoring completion text against a hidden answer key measures the model and leaves the client behavior ungraded.
Resetting the ledger on every invocation hides double-spend and makes the second-run check impossible to pass honestly.

Offline review procedure

The offline path is the grading path, and the live call is a later smoke test with a tiny budget. A reviewer starts the stub, runs the client, and pretty-prints the ledger before reading any completion text. No key is required for that path, which keeps the packet usable on a laptop that has no outbound network. If the ledger file is missing after the run, the submission fails before any status debate begins.

python stub_server.py
python -m budget_gate run \
  --fixture fixtures/prompts.jsonl \
  --ledger var/ledger.json \
  --budget-tokens 180 \
  --base-url http://127.0.0.1:8080/v1 \
  --model "$MODEL_ID"
python -m json.tool var/ledger.json

A correct ledger shows spent tokens at or below 180 and an event list whose length matches successful rows only. The third row status is budget_stop when the ceiling no longer fits, even if the stub would still have returned HTTP 200. If the stub is set to fail after two calls and the gate still had room, the status is upstream_unavailable instead. Those two outcomes must remain distinct strings, because collapsing them hides whether the client or the host stopped the work.

Where the free path fits

An operator-supplied free server is a reasonable optional smoke target after the stub is already green. Free model access is relevant because a candidate can read current notes, export a key, and confirm that the same client speaks that route. Neither availability claim in this packet includes a promised token count, a named model, a hardware shape, or a permanence guarantee. If the live route rejects the key or omits usage, the typed result is the correct outcome, and estimating tokens is not a fix.

Readers who want the optional smoke path can start from the MonkeyCode project notes for the current date. Those notes, not this article, are the source for the free model route, the free server instructions, and any current quota. The base URL flag should change only after the stub run passes and the ledger already shows a clean stop. That step checks route compatibility for the same client, and it does not replace the ledger tests above.

Limitations and non-goals

This packet does not measure answer quality, latency, or cost in currency, and it should not be cited as a capacity plan. It does not prove that a free tier will exist next month, because availability claims are not duration claims. The sample modules are unexecuted proposals, and they omit streaming, tool calls, and file locking on purpose. A local ceiling is a review control, not an invoice that a finance team can reconcile against a provider bill.

Who should skip this packet

The following groups should not stretch this packet past the evidence the ledger and stub actually contain.

Interviewers who need to grade research quality of model prose should use a different rubric and a frozen evaluation set.
Teams that cannot supply a fixture, a stub, and a second reviewer should not treat a live demo as this packet.
Anyone who needs a certified spend number for finance should not read the local ledger as a billing export.
Candidates asked only to wrap a vendor SDK, without a visible ledger, will not satisfy the pass lines above.

Closing criterion

The packet is complete when a second reviewer can delete the live URL, rerun the stub, and reach the same pass or fail. That property keeps the hiring decision on client discipline rather than on whoever had a warmer free host that morning. Free access can shorten setup for the optional smoke test, but the score still belongs to the stop ledger. A submission that cannot reproduce the stop offline is incomplete, regardless of how fluent its sample completion looked.

Pin the Fixture, Not the Vendor: A Take-Home for Swap-Safe Eval Runners

Riley Zhu — Sun, 11 Oct 2026 22:02:28 +0000

A second reviewer cannot trust a take-home that only passes on one laptop and one model string. The score that survives a handoff is the fixture hash, a shared decision schema, and invariants that do not name a vendor. This packet asks a candidate to build a small evaluation runner for a frozen support-triage task. The runner must pass locally with a stub, then accept an optional live endpoint without edits to the assertions.

Free model access and a free hosted fixture matter here only because they let a grader replay the same bytes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims stay operator-supplied for this exercise, and they are not treated as measured capacity. No token quota, hardware profile, region list, or permanence promise appears here, because no primary source dated this draft supplied those figures.

The defect under test

Swap drift is the defect this packet isolates, and it shows up when a suite is green only on the draft-day endpoint. A candidate's checks then fail after the base URL, the model id, or the fixture host changes. Reviewers often mislabel that failure as an outage, even though the assertions were coupled to one vendor envelope. This packet leaves aside retries, outbox timing, and sticky flags, because those stories already have their own grading sheets.

The practical cost appears when hiring loops try to compare candidates without a shared document or a shared port. Two reviewers cannot score fairly when each demo depends on a private key and a laptop process that sleeps. A hosted fixture with a recorded hash removes that excuse for the document that both reviewers must read. A swappable adapter removes the matching excuse for the completion path, which keeps the handoff off one machine.

What the candidate receives

The prompt should stay on one page and should name files rather than a preferred brand. Candidates receive the six constraints below, together with a written note that any live call stays optional. A missing key must skip the network path and must still produce a complete stub report. The page should not mention a numeric allowance, because that number is not part of the contract under test.

Prompt constraints

Load the local fixture.json file, call one adapter, and write report.json with one decision object.
Read MODEL_BASE_URL, MODEL_API_KEY, and FIXTURE_URL only from the process environment, never from committed files.
Store the SHA-256 of the fixture bytes, and fail closed when fetched bytes differ from the pinned hash.
Enforce three invariants: citations use only fixture ids, unknown fields are dropped, and one idempotency key yields one decision id.
Ship a stub mode that needs no network, and a hosted mode that fetches the same fixture document.
State every claim left unverified, including allowance, retention, and region, instead of writing those claims into tests.

A candidate may point the live path at MonkeyCode's operator-stated free model route and may publish the fixture through its free server option. The grader must still pass when that live path is skipped for lack of a key. A numeric allowance must not appear in an assertion, a fixture field, or a comment that later tests treat as fact. When current project notes describe that route, the candidate cites them in the README and omits secrets.

Rubric for the reviewer

Behavior remains the only scoring surface that reviewers accept for this slice of the hiring loop. Prose about a product does not add points, and a green screenshot without a hash does not count. Reviewers should mark the live path as skipped whenever that path did not actually run in the session. They should not upgrade a stub-only pass into a written statement about any particular model route.

Score sheet rules

Check	Pass condition	Fail condition
Fixture pin	Report stores SHA-256 of the exact bytes	Assertion searches for a model id
Adapter swap	Stub and live modes share one schema	Live mode uses a second parser
Scope fence	Unknown fields are rejected or dropped	Extra fields are copied into the decision
Idempotency	Same key keeps the same decision id	Each call mints a fresh id
Secret handling	Key is read from the environment only	Key or token is written into the report
Claim hygiene	Unverified allowances are omitted	A fixed quota is asserted as fact

Four passing checks still count as a useful signal within this narrow slice of take-home work. A missing fixture pin is an automatic fail, because every other check can look green against the wrong document. The table is a grading aid, not a claim that any hosted route stayed up during a measured window.

Proposed sample, not a recorded run

The following Python is a proposed sample that reviewers may adapt inside their own hiring packet. It was not executed for this article, and it is not a latency or cost benchmark. The live call stays optional so that the stub path remains the deterministic grade for reviewers. Candidates who add a real HTTP client should keep the same decide function for both modes.

import hashlib, json, os, urllib.request
from pathlib import Path

def load_fixture(url: str | None, local: Path) -> bytes:
    if not url:
        return local.read_bytes()
    with urllib.request.urlopen(url, timeout=10) as resp:
        return resp.read()

def decide(fixture: dict, completion: str) -> dict:
    allowed = {item["id"] for item in fixture["items"]}
    ids = [tok.removeprefix("id:") for tok in completion.split() if tok.startswith("id:")]
    extra = set(fixture.get("echo_fields", [])) - {"id", "title"}
    if extra:
        return {"ok": False, "reason": "unknown_field", "decision_id": fixture["key"]}
    if any(i not in allowed for i in ids):
        return {"ok": False, "reason": "out_of_scope", "ids": ids, "decision_id": fixture["key"]}
    return {"ok": True, "reason": "in_scope", "ids": ids, "decision_id": fixture["key"]}

def run(base: Path) -> dict:
    raw = load_fixture(os.environ.get("FIXTURE_URL"), base / "fixture.json")
    pinned = os.environ.get("FIXTURE_SHA256")
    digest = hashlib.sha256(raw).hexdigest()
    if pinned and pinned != digest:
        raise SystemExit("fixture pin mismatch")
    fixture = json.loads(raw)
    completion = os.environ.get("STUB_COMPLETION", "use id:item-1 only")
    report = {
        "fixture_sha256": digest,
        "live_configured": bool(os.environ.get("MODEL_BASE_URL")),
        "decision": decide(fixture, completion),
    }
    (base / "report.json").write_text(json.dumps(report, indent=2))
    return report

if __name__ == "__main__":
    run(Path("."))

A matching fixture should stay small and stable, with no extra whitespace before the pin is recorded. Reviewers hash these exact bytes, and they should not reformat the file before pinning it. The sample completion cites item-1, and that identifier is the only one the fixture marks as allowed.

{
  "key": "case-17",
  "items": [
    {"id": "item-1", "title": "reset token flow"}
  ]
}

Commands a reviewer can reuse

Stub path

These commands stay illustrative, and they assume that both files already sit in the working directory. They do not call a live vendor, and they do not require a paid seat or a named model. A reviewer runs the pin check first, then the out-of-scope check, and only then considers a hosted URL.

python -m json.tool fixture.json >/dev/null
export FIXTURE_SHA256=$(python -c "import hashlib,pathlib; print(hashlib.sha256(pathlib.Path('fixture.json').read_bytes()).hexdigest())")
STUB_COMPLETION="use id:item-1 only" python runner.py
python -c "import json,os; r=json.load(open('report.json')); assert r['fixture_sha256']==os.environ['FIXTURE_SHA256']"
STUB_COMPLETION="use id:item-9" python runner.py
python -c "import json; r=json.load(open('report.json')); assert r['decision']['reason']=='out_of_scope'"

Hosted path

A hosted fixture check should fail closed when its URL is wrong or otherwise unreachable today. The invalid host below is deliberate, and a cached report from yesterday must not hide that failure. Replacing that URL remains a reviewer step, and it must never rewrite the assertion text itself.

FIXTURE_URL="https://example.invalid/fixture.json" python runner.py
echo "exit=$?"

If an operator has currently documented a free server, the reviewer fetches the fixture, recomputes the hash, and stores both values on the score sheet. Test names should stay generic so they do not embed a product name or a numeric quota. The same score sheet notes whether MODEL_BASE_URL was set, and it never prints the key.

Hidden cases to plant after the draft

The written prompt implies the cases below, but it should not reveal the exact hidden inputs. That gap is the place where swap drift usually hides during a candidate's quick local self-check. Graders should add those hidden cases only after the candidate has frozen every submitted file.

A trailing newline in the fixture must change the hash and must fail the pin check.
A completion citing id:item-9 must return out_of_scope when only item-1 exists in the file.
A second run with the same fixture key must reuse decision_id rather than allocate another one.
An empty MODEL_API_KEY must skip the live call and must not log the literal None value.
A fixture that carries echo_fields outside id and title must not copy those fields into report.json.
A live envelope with a different wrapper must still normalize to the same decision schema as the stub.

Candidates who snapshot one lucky completion will fail the hidden id case on the second run. Candidates who treat a free URL as a permanent dependency will fail the pin when that host is absent. Availability during the review window is an operator statement, not a property the suite should hardcode. The hidden list therefore functions as a contract test, not as a probe of answer style.

Failure modes before any model debate

Three ordinary contract failures show up long before reviewers argue about generated answer quality at all. The first pins a model id inside a unit test, so a free route with another id cannot pass. The second fetches the fixture once, then replaces it with a paraphrase, so reviewers no longer share bytes. The third writes an unverified allowance into a comment or assertion, which turns a changing commercial detail into a false invariant.

A fourth failure is a quiet adapter split between the local stub and the live response parser. The stub returns a plain string, while the live parser expects one vendor envelope and no other shape. The suite stays green only on the stub, and the candidate reports the swap as an infrastructure outage. The rubric should mark that split as an adapter defect rather than as evidence about route uptime.

A fifth failure prints the API key into report.json while the candidate chases a debug log. Secret handling fails in that case even when every invariant holds and the hash matches. Reviewers should delete that report from shared storage and then ask for a clean rerun. That rerun still uses the same fixture pin, so the candidate cannot quietly swap documents between attempts.

Who should not use this packet

This exercise does not rank any models, measure latency, or estimate spend on a live route. It does not permit hiring-packet text, customer records, or secrets to be published on a public fixture host. Teams that require a named licensed model or an air-gapped network should keep only the stub path. A formal vendor security review is also outside this packet, so those teams should skip the hosted path.

Operators should hand each candidate the current project documentation on the morning of the hiring loop. If that documentation disagrees with this packet, the documentation wins and the prompt should be patched. The packet only requires a swappable route and fixture bytes that remain fetchable during the review window. It should not be reused as a press claim about capacity, uptime, or a standing token grant.

Closing the review

The reviewer hosts the fixture, records the URL with the hash, and runs the stub path first. The live path runs only when a key is present and the endpoint answers within the exercise window. The score sheet must clearly record whether the live path was skipped during that review session. Readers who need this split can follow current operator notes on free model access and a free server, with assertions left on the hash.

Sent Before Ack: A Kill-Window Fixture for an Outbox Take-Home

Riley Zhu — Sat, 10 Oct 2026 18:43:45 +0000

A status write that lands before broker acknowledgement is a data-loss bug, and a happy-path print will not catch it. This packet asks a candidate to expose that window with a kill hook, then recover the row without double-applying the payload. The exercise stays small on purpose, because a hiring loop should grade one state machine rather than a platform rewrite. A free check host can run the fixture, but a green process exit is not a durability claim.

What this packet decides

The packet decides whether the candidate can separate three facts that assistant drafts often blur together. Those facts are write order inside one process, what a later process observes after a crash, and what an in-memory double may prove. Candidates who collapse those facts into a claim that the sample looked fine do not pass, even when the surrounding prose is clean.

Interviewers should hand out the buggy worker, the in-memory broker, and a one-page prompt, and they should withhold the fixed state machine. Withholding the fix matters, because the point is to see whether the candidate derives a distinct in-flight state. They should also withhold invented incident stories, since colorful production anecdotes are not evidence in this packet. The grader later compares the candidate fixture output with the expected stranded row, rather than scoring tone.

The candidate prompt

The prompt below is the artifact a coordinator can paste into a private repository for a timed exercise. It names the constraint, the forbidden shortcuts, and the required observation, and it does not ask for a slide. A blog post or a broker product comparison would measure writing taste, which is not the decision this packet makes.

You are given buggy_worker.py and an in-memory broker double.
The worker marks an outbox row sent before publish returns.
Add a kill hook that stops the process after that status write.
Show that the row is sent while the broker message list is empty.
Propose a state machine that can resume after that crash.
Publish must be idempotent on the row id.
Do not add a real network broker.
Do not treat a clean run, without the kill hook, as a pass.
Time box: 90 minutes. Submit the test file and a short note.

The time box is a hiring constraint, not a claim about how long a careful production fix should take. Coordinators who need a deeper broker exercise should choose a different packet with a real cluster. This packet isolates the early commit, the resume path, and the idempotent key, and it stops there.

Fixture the grader actually runs

The teaching fixture uses a hook instead of a real signal, so a reviewer can run it inside one process. A later optional step can start a subprocess and send a signal, but that step is not required to see the bug. The code below is a proposal for the packet, and it is meant to be executed as a unit test. It should not be imported into a running service, because the hook raises SystemExit on purpose during the tick.

from dataclasses import dataclass

@dataclass
class Row:
    id: str
    payload: str
    status: str
    attempts: int = 0

class MemoryBroker:
    def __init__(self):
        self.messages = []
        self.seen = set()

    def publish(self, key: str, body: str) -> None:
        if key in self.seen:
            return
        self.seen.add(key)
        self.messages.append(f"{key}:{body}")

class BuggyWorker:
    def __init__(self, rows, broker, kill_after_status=False):
        self.rows = rows
        self.broker = broker
        self.kill_after_status = kill_after_status

    def tick(self) -> None:
        for row in list(self.rows.values()):
            if row.status != "pending":
                continue
            row.status = "sent"
            row.attempts += 1
            if self.kill_after_status:
                raise SystemExit("killed after status write")
            self.broker.publish(row.id, row.payload)

def show_buggy() -> None:
    rows = {"m1": Row("m1", "hello", "pending")}
    broker = MemoryBroker()
    worker = BuggyWorker(rows, broker, kill_after_status=True)
    try:
        worker.tick()
    except SystemExit as exc:
        print(exc)
    print(rows["m1"].status, broker.messages)

The listing is an unexecuted teaching proposal for the packet, and a coordinator should run it before treating the assertions as confirmed. A local failure of the import path is a setup problem, not evidence about the outbox state machine. The expected diagnosis print is the exit text, then the word sent, then an empty list.

Diagnosis command

The matching tests make the failure observable without a log scrape or a manual console session. The first test expects a sent row and an empty message list after the hook fires. The second test belongs to the sample solution and should fail when it is pointed at the buggy worker.

def test_kill_window_strands_sent_row():
    rows = {"m1": Row("m1", "hello", "pending")}
    broker = MemoryBroker()
    worker = BuggyWorker(rows, broker, kill_after_status=True)
    try:
        worker.tick()
    except SystemExit:
        pass
    assert rows["m1"].status == "sent"
    assert broker.messages == []

Graders can run the file with one ordinary command from a clean checkout on a machine that has Python available. The command does not depend on a particular model vendor, a private token, or a hosted console. A failing first test is the pass condition for the diagnosis half of this packet, before any patch is discussed.

python -m pytest -q test_outbox_takehome.py
python -c "from outbox_takehome import show_buggy; show_buggy()"

A candidate who edits the assertion until the suite goes green has not found the bug. The grader should diff the submitted test against the prompt, and should not stop at the process exit code. A green exit after a deleted hook is a fail, even when the remaining assertions describe the happy path accurately.

Rubric

Scores stay binary on purpose, because partial credit invites essay padding during a short hiring exercise. Each row is something the submitted test or the short note must show without extra interpretation. A pass on one row does not borrow credit from a neighboring row, which keeps a fluent note from hiding a skipped hook.

Check	Pass looks like	Fail looks like
Kill hook after the status write	Row is `sent` and the broker list is empty	Only the happy path is executed
Resume state	An in-flight status is distinct from `sent`	An exception handler leaves `sent` in place
Second tick	One message, then `sent`	The second tick skips the row forever
Idempotent key	A replay does not append a second body	Duplicate publish is renamed and left in place
Scope	The patch stays inside the worker and the test	A new queue product or cloud account appears
Evidence note	The note states what the double cannot prove	The note claims disk durability or multi-host fencing

A candidate can pass the table and still be a weak hire for reasons this packet does not measure. The table does not measure system-design breadth, on-call judgment, or writing aimed at a non-engineering audience. It measures whether this early-commit bug can be made to fail in a way another engineer can rerun.

Sample solution sketch

The sample solution sets an in-flight status before the broker call and writes sent only after publish returns successfully. A crash during that in-flight status leaves a row that the next tick must claim again. The broker double ignores a repeated key, which models idempotent produce rather than exactly-once delivery across a cluster.

In-flight worker

class LeaseWorker:
    def __init__(self, rows, broker, kill_after_status=False):
        self.rows = rows
        self.broker = broker
        self.kill_after_status = kill_after_status

    def tick(self) -> None:
        for row in list(self.rows.values()):
            if row.status not in ("pending", "publishing"):
                continue
            row.status = "publishing"
            row.attempts += 1
            if self.kill_after_status and row.attempts == 1:
                raise SystemExit("killed while publishing")
            self.broker.publish(row.id, row.payload)
            row.status = "sent"

The recovery assertion is the other half of the artifact, and it should be present in the submission. After the first tick dies, the row must not say sent, and the message list must still be empty. After the hook is cleared, one message exists and the status catches up to sent on that same row.

Assertions the note must include

def test_resume_publishes_once():
    rows = {"m1": Row("m1", "hello", "pending")}
    broker = MemoryBroker()
    worker = LeaseWorker(rows, broker, kill_after_status=True)
    try:
        worker.tick()
    except SystemExit:
        pass
    assert rows["m1"].status == "publishing"
    assert broker.messages == []
    worker.kill_after_status = False
    worker.tick()
    assert broker.messages == ["m1:hello"]
    assert rows["m1"].status == "sent"

def test_replay_does_not_duplicate():
    rows = {"m1": Row("m1", "hello", "publishing")}
    broker = MemoryBroker()
    broker.publish("m1", "hello")
    worker = LeaseWorker(rows, broker, kill_after_status=False)
    worker.tick()
    worker.tick()
    assert broker.messages == ["m1:hello"]
    assert rows["m1"].status == "sent"

This sketch does not implement lease expiry, a fencing token, or a dead-letter cap for poison payloads. Those controls belong in a follow-up prompt when the open role actually owns a long-running worker fleet. Naming them as out of scope is a better note than silently adding untested machinery to the patch. The idempotent key covers the dual window, where publish succeeded and the status write never landed.

Failure modes that still look polished

Several submissions read well and still lose the message or append it twice under a replay. The list below is what graders should mark before they comment on sentence style or confidence. These misses show up in the diff, which is why the grader reads assertions before adjectives.

A try/except around publish logs the error and leaves the row sent, which hides the loss from every later tick.
A retry loop runs without a maximum attempt or a dead-letter status, so one poison payload can block the worker.
A timestamp is labeled a lease, but nothing stops an expired worker from publishing after a newer worker claimed the row.
The note accepts the assistant draft because a manual print matched the payload, and the kill hook was never executed.
The kill hook is removed so the suite passes on a clean machine, and the note describes that deletion as the fix.
The row is deleted before acknowledgement, which removes the only local copy of the payload if publish then fails.

Polished wording does not offset a missing assertion, and graders should quote that assertion or its absence. That habit keeps the written score about the state machine rather than about how certain the note sounded. A second reader should be able to rerun the quoted test without asking what the candidate meant.

How the hour is scored

The coordinator clones the private repo and runs the diagnosis test before opening the candidate note, so the fixture result comes first.
The grader records the status string and the message list, and treats a missing kill hook as a fail even if other tests pass.
The grader then reads the proposed state names and checks that sent is written only after a successful publish return.
The grader replays the same row id and confirms the message list length stays at one after the second successful tick.
The grader marks any durability claim that the in-memory double cannot support, including disk flush and multi-host fencing.

This order keeps the review mechanical, which matters when several candidates submit notes that sound similar. Similar notes are expected if the same free model access sketched them, so the differing artifact is the test diff. A coordinator who cannot explain the stranded row should not score the packet, regardless of seniority.

Where a draft host and a check host fit

Assistant drafts are useful for the first worker file and for a candidate note, and they are a poor source of the verdict. The task brief describes MonkeyCode as an open-source project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. License terms and current limits should be confirmed in the project repository before a hiring loop relies on either option.

This article does not name models, token quotas, hardware, or a duration, because those details change and were not re-checked against a primary source. The workflow worth teaching is that split, not a preference for one vendor console over another. One step uses free model access to sketch the buggy worker and a short review note for the candidate packet. A second step uses the free server option only as a check host for the pinned fixture and its expected failure.

The score still comes from the rubric table, including the explicit note about what the double cannot prove. If either option is unavailable, the same files run on any machine with Python and pytest installed. That fallback is required, because a hiring packet must not depend on a private console staying up. Coordinators who want that split can check MonkeyCode's current free model access and free server docs, then keep the fixture in git.

Limits and who should skip it

The in-memory double does not model disk flushes, a network partition, or duplicate delivery from a real broker. A green run on a free server shows that this fixture behaved on that host, and it shows nothing about production latency. Free model access and a free server option are availability claims for drafting and checking, not a service level agreement. Numeric quotas, hardware size, and retention were left out on purpose, so this post does not freeze a limit that may already have changed.

Teams hiring for production broker ownership should not use this packet as the only exercise, because the double hides cluster failure. Loops that score confidence, slide quality, or how closely a note matches an assistant's tone should skip it. Anyone who would treat a free-tier check host as proof of durability should not adopt the workflow described here. Candidates without time to run a test should not be judged on this packet, because the artifact is the failing assertion.

The useful outcome is a stranded row that another engineer can reproduce, followed by a resume path that publishes once. Everything else written in the note remains optional color around that pair of results from the fixture. A hiring loop that can show those two results has enough evidence to score this packet and to stop.

Hidden Truncation Cases for a Structured-Output Hiring Packet

Riley Zhu — Fri, 09 Oct 2026 15:05:28 +0000

A structured-output hiring packet should pass only when a deterministic checker accepts valid tool calls and rejects truncated or over-broad payloads. Model confidence, a successful local demo, and a polished explanation are not evidence that the boundary holds under hidden cases. Graders freeze fixtures before any model runs, then execute those cases on a shared runtime that candidates cannot silently upgrade. Optional drafting help may suggest extra cases, but the score belongs to the checker and the frozen fixture contract.

What the packet measures

The packet measures whether a candidate can separate untrusted model text from a tool request that is safe to execute. It does not measure taste, prompting speed, or the ability to narrate a debugging story after the fact. A passing submission rejects incomplete JSON, unknown tool names, unexpected keys, and paths that escape the assigned workspace. A failing submission may still look impressive when a reviewer grades the prose instead of the hidden fixture results.

Three boundaries carry the design, and each one should stay visible in the candidate prompt and the rubric. The checker owns acceptance, the fixture file owns the expected reasons, and the shared runtime owns the execution environment. A model may propose a payload, but it may not loosen the schema, rename a failure reason, or skip a case. That split keeps the hiring signal stable when model output changes between one cohort and the next.

The prompt to hand over

The candidate prompt should be short enough to read in one sitting and strict enough to grade without a meeting. It names the allowed tools, the exact key set, and the rule that extra keys are failures rather than hints. It also states that no tool runs until the checker returns an accept decision for that single payload. Hidden fixtures remain with the grader and cover truncation, repeated call identifiers, and paths that leave the workspace.

Hand over language close to the following prompt, and attach the public fixture file in the same archive. State which command must pass before any reviewer reads the written explanation that accompanies the code. Leave hidden rows out of the archive so a local passing run cannot become a memorized transcript.

Implement acceptToolCall(raw) for a hiring packet in Node.js.
Input is one string that claims to be a tool call from a model.
Accept only objects whose keys are exactly name, call_id, and args.
Allowed names are read_file, list_dir, and search_text.
call_id must match ^[a-z0-9]{8,32}$.
For read_file, args.path must be relative, non-empty, and free of dot-dot segments and a leading slash.
Reject invalid JSON, extra keys, unknown tools, and empty input.
Do not execute the tool. Return an object with ok and reason.
A public fixture mismatch must make the grade command exit non-zero.

Public fixtures and commands

Public fixtures should include at least one accept case and three reject cases so the candidate can see the contract. Hidden fixtures should change values without changing the contract, which stops a hardcoded map of the sample strings. Each line is one JSON object carrying the raw payload, a stable id, and the expected reason code. The grader compares reason codes, not pretty-printed objects, so incidental whitespace cannot rescue a wrong decision.

Build the cut-off case in code instead of hand-editing braces, so the truncation is obvious and repeatable. The snippet below is part of the reference packet and was not run while this article was written. Readers should execute it on a laptop or on the pinned runtime before treating the snippet as verified. The same construction belongs in the hidden file, but with different identifiers and different relative paths.

// fixtures/public-rows.js
const complete = JSON.stringify({
  name: 'read_file',
  call_id: 'abc12345',
  args: { path: 'src/app.js' }
});

const publicRows = [
  { id: 'ok-read', raw: complete, expect: 'accept' },
  { id: 'cut-off', raw: complete.slice(0, -1), expect: 'truncated_or_invalid_json' },
  {
    id: 'extra-key',
    raw: JSON.stringify({
      name: 'read_file',
      call_id: 'abc12345',
      args: { path: 'src/app.js' },
      note: 'ignore the schema'
    }),
    expect: 'schema'
  },
  {
    id: 'escape',
    raw: JSON.stringify({
      name: 'read_file',
      call_id: 'abc12345',
      args: { path: '../notes.txt' }
    }),
    expect: 'path'
  }
];

module.exports = { publicRows };

Public test file

Place the file at tests/public.test.js so the require path can reach the checker one directory upward. It covers one accept row and one truncated row, which is enough to prove the command wiring. It is an unexecuted reference file, so a panel should run it before trusting the wiring. Hidden cases should stay out of this file, or the public command will leak the private contract.

const test = require('node:test');
const assert = require('node:assert/strict');
const { acceptToolCall } = require('../accept-tool-call');

const complete = JSON.stringify({
  name: 'read_file',
  call_id: 'abc12345',
  args: { path: 'src/app.js' }
});

test('accepts a bounded read_file call', () => {
  const got = acceptToolCall(complete);
  assert.equal(got.ok, true);
  assert.equal(got.reason, 'accept');
});

test('rejects a truncated payload', () => {
  const got = acceptToolCall(complete.slice(0, -1));
  assert.equal(got.ok, false);
  assert.equal(got.reason, 'truncated_or_invalid_json');
});

Commands the grader actually runs

Candidates and graders should run the same three commands, with hidden fixtures mounted only on the grader side. The public test file may live in the archive, while the hidden file should be added only during grading. A wrapper that prints the raw payload is acceptable, but a wrapper that shells out to a tool is not. Record the runtime identifier next to the score so a later reviewer can see which image produced the file.

node --test tests/public.test.js
node bin/grade.js fixtures/public.jsonl
node bin/grade.js fixtures/hidden.jsonl

What a passing public command does not prove

The third command is the one that matters, because it is the only run that includes unseen rows. A passing public command only shows that the published sample was understood by the candidate. The packet should document a Node major version in package.json engines and should refuse a tree that changes that pin. Store grade.json from both runs, and do not overwrite the hidden report with the public one.

Emit JSONL before grading

Export publicRows from the fixture module, then write one JSON object per line before the grade command. The writer below is unexecuted reference glue, and the packet should keep that file at bin/emit-public.js. Run it once when fixtures change, and commit the JSONL so candidates do not need the writer.

const fs = require('fs');
const { publicRows } = require('../fixtures/public-rows');
const newline = String.fromCharCode(10);
const text = publicRows.map((row) => JSON.stringify(row)).join(newline) + newline;
fs.writeFileSync('fixtures/public.jsonl', text);

Sample checker

Reference accept function

The module below is an unexecuted reference solution, not a measured production component or a latency claim. It parses once, rejects non-objects, requires an exact key set, and then applies tool-specific argument rules. Path checks stay conservative because the take-home is about refusal, not about building a full filesystem sandbox. Teams that need stronger isolation should add an operating-system sandbox outside this function rather than trusting string filters alone.

// accept-tool-call.js
const ALLOWED = new Set(['read_file', 'list_dir', 'search_text']);
const KEYS = ['args', 'call_id', 'name'];

function acceptToolCall(raw) {
  if (typeof raw !== 'string' || raw.length === 0 || raw.length > 8192) {
    return { ok: false, reason: 'size' };
  }
  let parsed;
  try {
    parsed = JSON.parse(raw);
  } catch (_err) {
    return { ok: false, reason: 'truncated_or_invalid_json' };
  }
  if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
    return { ok: false, reason: 'shape' };
  }
  const keys = Object.keys(parsed).sort();
  if (keys.join(',') !== KEYS.join(',')) {
    return { ok: false, reason: 'schema' };
  }
  if (!ALLOWED.has(parsed.name)) {
    return { ok: false, reason: 'tool_name' };
  }
  if (!/^[a-z0-9]{8,32}$/.test(parsed.call_id)) {
    return { ok: false, reason: 'call_id' };
  }
  if (!parsed.args || typeof parsed.args !== 'object' || Array.isArray(parsed.args)) {
    return { ok: false, reason: 'args' };
  }
  if (parsed.name === 'read_file') {
    const filePath = parsed.args.path;
    const badPath = typeof filePath !== 'string'
      || filePath.length === 0
      || filePath.includes('..')
      || filePath.startsWith('/');
    if (badPath) return { ok: false, reason: 'path' };
  }
  return { ok: true, reason: 'accept' };
}

module.exports = { acceptToolCall };

Illustrative grader

A small grader script keeps the human reviewer out of the comparison loop for every fixture row. It reads JSONL, calls the implementation, and writes a score file that another person can review later. The script below is illustrative glue for the packet, and it assumes the checker file sits beside the bin directory. Panels should replace the require path when they ask candidates to submit a different filename than the sample.

// bin/grade.js
const fs = require('fs');
const path = require('path');
const { acceptToolCall } = require(path.join(__dirname, '..', 'accept-tool-call'));

const file = process.argv[2];
if (!file) {
  console.error('usage: node bin/grade.js fixtures/public.jsonl');
  process.exit(2);
}

const newline = String.fromCharCode(10);
const lines = fs.readFileSync(file, 'utf8').trim().split(newline);
const misses = [];
for (const line of lines) {
  if (!line.trim()) continue;
  const row = JSON.parse(line);
  const got = acceptToolCall(row.raw);
  if (got.reason !== row.expect) {
    misses.push({ id: row.id, expected: row.expect, got: got.reason });
  }
}

const kept = lines.filter((line) => line.trim());
const report = {
  fixture: file,
  total: kept.length,
  failed: misses.length,
  misses
};
fs.writeFileSync('grade.json', JSON.stringify(report, null, 2) + newline);
console.log('failed=' + misses.length + ' total=' + report.total);
process.exit(misses.length === 0 ? 0 : 1);

Rubric

Grade the submitted artifact rather than the interview conversation that happens after the coding window closes. The table below is a starting rubric that a hiring panel can edit before the packet goes out. Points should be assigned from the grade file, not from a recollection of how confident the candidate sounded. Publish the same rubric with the prompt so candidates can see the gates before they write code.

Check	Pass condition	Points
Public fixture match	Every public row returns the expected reason	20
Hidden truncation	Cut-off JSON returns truncated_or_invalid_json	15
Exact keys	Extra or missing keys return schema	15
Tool allowlist	Unknown names return tool_name and do not run	15
Path bound	Parent segments and absolute paths return path	15
No execution	The graded process never reads args.path from disk	10
Runtime pin	grade.json was produced on the pinned shared runtime	10

A score of 70 or above can advance only when the no-execution check and the runtime pin both passed. Those two rows function as gates rather than decorative columns inside an otherwise flexible scoring table. A candidate who scores well by reading files anyway has missed the safety property this packet exists to detect. Partial credit on style should not reopen a failed gate once the grade file is written.

Failure modes the hidden set should catch

Several misses show up often when model output is treated as data that is already structured and safe. Naming those misses in the rubric lets reviewers stay consistent across a full week of submissions. The list below concerns grading practice rather than a contest to collect clever payloads from candidates. Reviewers should mark the matching row in the grade file instead of inventing a new reason during the debrief.

A parser that rescues truncated JSON by appending braces will pass a demo and fail the cut-off row.
A schema that ignores unknown keys will accept a note field that tries to change the assigned task.
A handler that reads args.path before acceptToolCall returns has already crossed the safety boundary under test.
A solution hardcoded to the four public strings will fail as soon as the hidden identifiers change.
A passing run on a newer Node release can hide a syntax or Unicode difference from the pinned runtime.
A write-up that pastes fixture rows into a model, including secrets, creates a second leak the packet should forbid.

The last miss deserves a written rule in the candidate prompt and in the grader checklist. Candidates may use a model to brainstorm case shapes, but they may not upload hidden fixtures or employer code. Credential-shaped strings belong in neither the brainstorming prompt nor the fixture archive that leaves the company. The grader should reject a submission that asks any model to rewrite expected reasons for hidden rows.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two availability claims are used as operator-supplied context only: free model access, and a free server option. This article does not state model names, token quotas, hardware sizes, time limits, or permanence of either option. Those terms must be checked in current product documentation before a hiring panel relies on them.

Free model access is useful while the grader drafts extra reject cases for the hidden file. The author can request malformed shapes, keep the useful ones, and freeze them into JSONL by hand. The model is not the judge of those frozen lines, even when a later reply sounds more confident. Once a line is frozen, no assistant may change its expected reason without a human editing the fixture.

A free server option is useful when every candidate and the grader need the same Node runtime. The practical workflow stays narrow, and the packet should list it as required steps rather than tips. Pin the major version, clone the packet, run the public command, and keep hidden fixtures on the grader account only. If that option is unavailable or differs from the documented image, pause instead of accepting laptop runs.

Availability is an operational convenience, not a benchmark result and not a promise that the option will stay unchanged. The article remains usable without any particular vendor, because any assistant and any shared runner can preserve fixtures and a runtime pin. What matters is the freeze step, the hidden file, and the refusal to execute before acceptance. Product convenience should not become an unexamined dependency inside the hiring loop or the score file.

Steps for one cohort

Freeze the allowed tool names and the exact key set in the prompt before asking any assistant for sample payloads.
Draft reject shapes with an assistant if desired, then copy only human-reviewed lines into the hidden JSONL file.
Run the public fixture command on the pinned runtime and store the grade file beside the cohort notes.
Run hidden fixtures from the grader account and treat a failed gate as a stop rather than a debate.

Who should skip this packet

This packet is a poor fit for roles that center visual design, product copy, or infrastructure work without a tool boundary. It is also a poor fit when the company cannot keep hidden fixtures private or cannot provide a shared runtime. Panels that want to grade communication style should use a separate exercise, because this one hides that signal on purpose. Teams without time to refresh fixtures should not reuse one public file for many cohorts, since candidates may memorize it.

The string path checks are a teaching control, not a complete isolation story for a production tool runner. A later system can still do harm if some other service ignores the accept decision and runs the path. The packet therefore forbids execution inside the graded process and leaves sandbox design to a separate review. A perfect score here is not permission to connect the same function to production credentials or customer data.

After the score file

A hiring loop that freezes fixtures, pins the runtime, and scores reason codes will keep its meaning when model output shifts. The reference checker above is enough to start that loop on a single afternoon without a new platform. Panels should still read current runtime terms before they depend on any free server, and they should refresh hidden rows before each cohort. Readers drafting fixtures with MonkeyCode should confirm current free-access and free-server terms, then run the pinned grader only after that check.