DEV Community: Riley Zhu

Sent Before Ack: A Kill-Window Fixture for an Outbox Take-Home

Riley Zhu — Sat, 10 Oct 2026 18:43:45 +0000

A status write that lands before broker acknowledgement is a data-loss bug, and a happy-path print will not catch it. This packet asks a candidate to expose that window with a kill hook, then recover the row without double-applying the payload. The exercise stays small on purpose, because a hiring loop should grade one state machine rather than a platform rewrite. A free check host can run the fixture, but a green process exit is not a durability claim.

What this packet decides

The packet decides whether the candidate can separate three facts that assistant drafts often blur together. Those facts are write order inside one process, what a later process observes after a crash, and what an in-memory double may prove. Candidates who collapse those facts into a claim that the sample looked fine do not pass, even when the surrounding prose is clean.

Interviewers should hand out the buggy worker, the in-memory broker, and a one-page prompt, and they should withhold the fixed state machine. Withholding the fix matters, because the point is to see whether the candidate derives a distinct in-flight state. They should also withhold invented incident stories, since colorful production anecdotes are not evidence in this packet. The grader later compares the candidate fixture output with the expected stranded row, rather than scoring tone.

The candidate prompt

The prompt below is the artifact a coordinator can paste into a private repository for a timed exercise. It names the constraint, the forbidden shortcuts, and the required observation, and it does not ask for a slide. A blog post or a broker product comparison would measure writing taste, which is not the decision this packet makes.

You are given buggy_worker.py and an in-memory broker double.
The worker marks an outbox row sent before publish returns.
Add a kill hook that stops the process after that status write.
Show that the row is sent while the broker message list is empty.
Propose a state machine that can resume after that crash.
Publish must be idempotent on the row id.
Do not add a real network broker.
Do not treat a clean run, without the kill hook, as a pass.
Time box: 90 minutes. Submit the test file and a short note.

The time box is a hiring constraint, not a claim about how long a careful production fix should take. Coordinators who need a deeper broker exercise should choose a different packet with a real cluster. This packet isolates the early commit, the resume path, and the idempotent key, and it stops there.

Fixture the grader actually runs

The teaching fixture uses a hook instead of a real signal, so a reviewer can run it inside one process. A later optional step can start a subprocess and send a signal, but that step is not required to see the bug. The code below is a proposal for the packet, and it is meant to be executed as a unit test. It should not be imported into a running service, because the hook raises SystemExit on purpose during the tick.

from dataclasses import dataclass

@dataclass
class Row:
    id: str
    payload: str
    status: str
    attempts: int = 0

class MemoryBroker:
    def __init__(self):
        self.messages = []
        self.seen = set()

    def publish(self, key: str, body: str) -> None:
        if key in self.seen:
            return
        self.seen.add(key)
        self.messages.append(f"{key}:{body}")

class BuggyWorker:
    def __init__(self, rows, broker, kill_after_status=False):
        self.rows = rows
        self.broker = broker
        self.kill_after_status = kill_after_status

    def tick(self) -> None:
        for row in list(self.rows.values()):
            if row.status != "pending":
                continue
            row.status = "sent"
            row.attempts += 1
            if self.kill_after_status:
                raise SystemExit("killed after status write")
            self.broker.publish(row.id, row.payload)

def show_buggy() -> None:
    rows = {"m1": Row("m1", "hello", "pending")}
    broker = MemoryBroker()
    worker = BuggyWorker(rows, broker, kill_after_status=True)
    try:
        worker.tick()
    except SystemExit as exc:
        print(exc)
    print(rows["m1"].status, broker.messages)

The listing is an unexecuted teaching proposal for the packet, and a coordinator should run it before treating the assertions as confirmed. A local failure of the import path is a setup problem, not evidence about the outbox state machine. The expected diagnosis print is the exit text, then the word sent, then an empty list.

Diagnosis command

The matching tests make the failure observable without a log scrape or a manual console session. The first test expects a sent row and an empty message list after the hook fires. The second test belongs to the sample solution and should fail when it is pointed at the buggy worker.

def test_kill_window_strands_sent_row():
    rows = {"m1": Row("m1", "hello", "pending")}
    broker = MemoryBroker()
    worker = BuggyWorker(rows, broker, kill_after_status=True)
    try:
        worker.tick()
    except SystemExit:
        pass
    assert rows["m1"].status == "sent"
    assert broker.messages == []

Graders can run the file with one ordinary command from a clean checkout on a machine that has Python available. The command does not depend on a particular model vendor, a private token, or a hosted console. A failing first test is the pass condition for the diagnosis half of this packet, before any patch is discussed.

python -m pytest -q test_outbox_takehome.py
python -c "from outbox_takehome import show_buggy; show_buggy()"

A candidate who edits the assertion until the suite goes green has not found the bug. The grader should diff the submitted test against the prompt, and should not stop at the process exit code. A green exit after a deleted hook is a fail, even when the remaining assertions describe the happy path accurately.

Rubric

Scores stay binary on purpose, because partial credit invites essay padding during a short hiring exercise. Each row is something the submitted test or the short note must show without extra interpretation. A pass on one row does not borrow credit from a neighboring row, which keeps a fluent note from hiding a skipped hook.

Check	Pass looks like	Fail looks like
Kill hook after the status write	Row is `sent` and the broker list is empty	Only the happy path is executed
Resume state	An in-flight status is distinct from `sent`	An exception handler leaves `sent` in place
Second tick	One message, then `sent`	The second tick skips the row forever
Idempotent key	A replay does not append a second body	Duplicate publish is renamed and left in place
Scope	The patch stays inside the worker and the test	A new queue product or cloud account appears
Evidence note	The note states what the double cannot prove	The note claims disk durability or multi-host fencing

A candidate can pass the table and still be a weak hire for reasons this packet does not measure. The table does not measure system-design breadth, on-call judgment, or writing aimed at a non-engineering audience. It measures whether this early-commit bug can be made to fail in a way another engineer can rerun.

Sample solution sketch

The sample solution sets an in-flight status before the broker call and writes sent only after publish returns successfully. A crash during that in-flight status leaves a row that the next tick must claim again. The broker double ignores a repeated key, which models idempotent produce rather than exactly-once delivery across a cluster.

In-flight worker

class LeaseWorker:
    def __init__(self, rows, broker, kill_after_status=False):
        self.rows = rows
        self.broker = broker
        self.kill_after_status = kill_after_status

    def tick(self) -> None:
        for row in list(self.rows.values()):
            if row.status not in ("pending", "publishing"):
                continue
            row.status = "publishing"
            row.attempts += 1
            if self.kill_after_status and row.attempts == 1:
                raise SystemExit("killed while publishing")
            self.broker.publish(row.id, row.payload)
            row.status = "sent"

The recovery assertion is the other half of the artifact, and it should be present in the submission. After the first tick dies, the row must not say sent, and the message list must still be empty. After the hook is cleared, one message exists and the status catches up to sent on that same row.

Assertions the note must include

def test_resume_publishes_once():
    rows = {"m1": Row("m1", "hello", "pending")}
    broker = MemoryBroker()
    worker = LeaseWorker(rows, broker, kill_after_status=True)
    try:
        worker.tick()
    except SystemExit:
        pass
    assert rows["m1"].status == "publishing"
    assert broker.messages == []
    worker.kill_after_status = False
    worker.tick()
    assert broker.messages == ["m1:hello"]
    assert rows["m1"].status == "sent"

def test_replay_does_not_duplicate():
    rows = {"m1": Row("m1", "hello", "publishing")}
    broker = MemoryBroker()
    broker.publish("m1", "hello")
    worker = LeaseWorker(rows, broker, kill_after_status=False)
    worker.tick()
    worker.tick()
    assert broker.messages == ["m1:hello"]
    assert rows["m1"].status == "sent"

This sketch does not implement lease expiry, a fencing token, or a dead-letter cap for poison payloads. Those controls belong in a follow-up prompt when the open role actually owns a long-running worker fleet. Naming them as out of scope is a better note than silently adding untested machinery to the patch. The idempotent key covers the dual window, where publish succeeded and the status write never landed.

Failure modes that still look polished

Several submissions read well and still lose the message or append it twice under a replay. The list below is what graders should mark before they comment on sentence style or confidence. These misses show up in the diff, which is why the grader reads assertions before adjectives.

A try/except around publish logs the error and leaves the row sent, which hides the loss from every later tick.
A retry loop runs without a maximum attempt or a dead-letter status, so one poison payload can block the worker.
A timestamp is labeled a lease, but nothing stops an expired worker from publishing after a newer worker claimed the row.
The note accepts the assistant draft because a manual print matched the payload, and the kill hook was never executed.
The kill hook is removed so the suite passes on a clean machine, and the note describes that deletion as the fix.
The row is deleted before acknowledgement, which removes the only local copy of the payload if publish then fails.

Polished wording does not offset a missing assertion, and graders should quote that assertion or its absence. That habit keeps the written score about the state machine rather than about how certain the note sounded. A second reader should be able to rerun the quoted test without asking what the candidate meant.

How the hour is scored

The coordinator clones the private repo and runs the diagnosis test before opening the candidate note, so the fixture result comes first.
The grader records the status string and the message list, and treats a missing kill hook as a fail even if other tests pass.
The grader then reads the proposed state names and checks that sent is written only after a successful publish return.
The grader replays the same row id and confirms the message list length stays at one after the second successful tick.
The grader marks any durability claim that the in-memory double cannot support, including disk flush and multi-host fencing.

This order keeps the review mechanical, which matters when several candidates submit notes that sound similar. Similar notes are expected if the same free model access sketched them, so the differing artifact is the test diff. A coordinator who cannot explain the stranded row should not score the packet, regardless of seniority.

Where a draft host and a check host fit

Assistant drafts are useful for the first worker file and for a candidate note, and they are a poor source of the verdict. The task brief describes MonkeyCode as an open-source project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. License terms and current limits should be confirmed in the project repository before a hiring loop relies on either option.

This article does not name models, token quotas, hardware, or a duration, because those details change and were not re-checked against a primary source. The workflow worth teaching is that split, not a preference for one vendor console over another. One step uses free model access to sketch the buggy worker and a short review note for the candidate packet. A second step uses the free server option only as a check host for the pinned fixture and its expected failure.

The score still comes from the rubric table, including the explicit note about what the double cannot prove. If either option is unavailable, the same files run on any machine with Python and pytest installed. That fallback is required, because a hiring packet must not depend on a private console staying up. Coordinators who want that split can check MonkeyCode's current free model access and free server docs, then keep the fixture in git.

Limits and who should skip it

The in-memory double does not model disk flushes, a network partition, or duplicate delivery from a real broker. A green run on a free server shows that this fixture behaved on that host, and it shows nothing about production latency. Free model access and a free server option are availability claims for drafting and checking, not a service level agreement. Numeric quotas, hardware size, and retention were left out on purpose, so this post does not freeze a limit that may already have changed.

Teams hiring for production broker ownership should not use this packet as the only exercise, because the double hides cluster failure. Loops that score confidence, slide quality, or how closely a note matches an assistant's tone should skip it. Anyone who would treat a free-tier check host as proof of durability should not adopt the workflow described here. Candidates without time to run a test should not be judged on this packet, because the artifact is the failing assertion.

The useful outcome is a stranded row that another engineer can reproduce, followed by a resume path that publishes once. Everything else written in the note remains optional color around that pair of results from the fixture. A hiring loop that can show those two results has enough evidence to score this packet and to stop.

Hidden Truncation Cases for a Structured-Output Hiring Packet

Riley Zhu — Fri, 09 Oct 2026 15:05:28 +0000

A structured-output hiring packet should pass only when a deterministic checker accepts valid tool calls and rejects truncated or over-broad payloads. Model confidence, a successful local demo, and a polished explanation are not evidence that the boundary holds under hidden cases. Graders freeze fixtures before any model runs, then execute those cases on a shared runtime that candidates cannot silently upgrade. Optional drafting help may suggest extra cases, but the score belongs to the checker and the frozen fixture contract.

What the packet measures

The packet measures whether a candidate can separate untrusted model text from a tool request that is safe to execute. It does not measure taste, prompting speed, or the ability to narrate a debugging story after the fact. A passing submission rejects incomplete JSON, unknown tool names, unexpected keys, and paths that escape the assigned workspace. A failing submission may still look impressive when a reviewer grades the prose instead of the hidden fixture results.

Three boundaries carry the design, and each one should stay visible in the candidate prompt and the rubric. The checker owns acceptance, the fixture file owns the expected reasons, and the shared runtime owns the execution environment. A model may propose a payload, but it may not loosen the schema, rename a failure reason, or skip a case. That split keeps the hiring signal stable when model output changes between one cohort and the next.

The prompt to hand over

The candidate prompt should be short enough to read in one sitting and strict enough to grade without a meeting. It names the allowed tools, the exact key set, and the rule that extra keys are failures rather than hints. It also states that no tool runs until the checker returns an accept decision for that single payload. Hidden fixtures remain with the grader and cover truncation, repeated call identifiers, and paths that leave the workspace.

Hand over language close to the following prompt, and attach the public fixture file in the same archive. State which command must pass before any reviewer reads the written explanation that accompanies the code. Leave hidden rows out of the archive so a local passing run cannot become a memorized transcript.

Implement acceptToolCall(raw) for a hiring packet in Node.js.
Input is one string that claims to be a tool call from a model.
Accept only objects whose keys are exactly name, call_id, and args.
Allowed names are read_file, list_dir, and search_text.
call_id must match ^[a-z0-9]{8,32}$.
For read_file, args.path must be relative, non-empty, and free of dot-dot segments and a leading slash.
Reject invalid JSON, extra keys, unknown tools, and empty input.
Do not execute the tool. Return an object with ok and reason.
A public fixture mismatch must make the grade command exit non-zero.

Public fixtures and commands

Public fixtures should include at least one accept case and three reject cases so the candidate can see the contract. Hidden fixtures should change values without changing the contract, which stops a hardcoded map of the sample strings. Each line is one JSON object carrying the raw payload, a stable id, and the expected reason code. The grader compares reason codes, not pretty-printed objects, so incidental whitespace cannot rescue a wrong decision.

Build the cut-off case in code instead of hand-editing braces, so the truncation is obvious and repeatable. The snippet below is part of the reference packet and was not run while this article was written. Readers should execute it on a laptop or on the pinned runtime before treating the snippet as verified. The same construction belongs in the hidden file, but with different identifiers and different relative paths.

// fixtures/public-rows.js
const complete = JSON.stringify({
  name: 'read_file',
  call_id: 'abc12345',
  args: { path: 'src/app.js' }
});

const publicRows = [
  { id: 'ok-read', raw: complete, expect: 'accept' },
  { id: 'cut-off', raw: complete.slice(0, -1), expect: 'truncated_or_invalid_json' },
  {
    id: 'extra-key',
    raw: JSON.stringify({
      name: 'read_file',
      call_id: 'abc12345',
      args: { path: 'src/app.js' },
      note: 'ignore the schema'
    }),
    expect: 'schema'
  },
  {
    id: 'escape',
    raw: JSON.stringify({
      name: 'read_file',
      call_id: 'abc12345',
      args: { path: '../notes.txt' }
    }),
    expect: 'path'
  }
];

module.exports = { publicRows };

Public test file

Place the file at tests/public.test.js so the require path can reach the checker one directory upward. It covers one accept row and one truncated row, which is enough to prove the command wiring. It is an unexecuted reference file, so a panel should run it before trusting the wiring. Hidden cases should stay out of this file, or the public command will leak the private contract.

const test = require('node:test');
const assert = require('node:assert/strict');
const { acceptToolCall } = require('../accept-tool-call');

const complete = JSON.stringify({
  name: 'read_file',
  call_id: 'abc12345',
  args: { path: 'src/app.js' }
});

test('accepts a bounded read_file call', () => {
  const got = acceptToolCall(complete);
  assert.equal(got.ok, true);
  assert.equal(got.reason, 'accept');
});

test('rejects a truncated payload', () => {
  const got = acceptToolCall(complete.slice(0, -1));
  assert.equal(got.ok, false);
  assert.equal(got.reason, 'truncated_or_invalid_json');
});

Commands the grader actually runs

Candidates and graders should run the same three commands, with hidden fixtures mounted only on the grader side. The public test file may live in the archive, while the hidden file should be added only during grading. A wrapper that prints the raw payload is acceptable, but a wrapper that shells out to a tool is not. Record the runtime identifier next to the score so a later reviewer can see which image produced the file.

node --test tests/public.test.js
node bin/grade.js fixtures/public.jsonl
node bin/grade.js fixtures/hidden.jsonl

What a passing public command does not prove

The third command is the one that matters, because it is the only run that includes unseen rows. A passing public command only shows that the published sample was understood by the candidate. The packet should document a Node major version in package.json engines and should refuse a tree that changes that pin. Store grade.json from both runs, and do not overwrite the hidden report with the public one.

Emit JSONL before grading

Export publicRows from the fixture module, then write one JSON object per line before the grade command. The writer below is unexecuted reference glue, and the packet should keep that file at bin/emit-public.js. Run it once when fixtures change, and commit the JSONL so candidates do not need the writer.

const fs = require('fs');
const { publicRows } = require('../fixtures/public-rows');
const newline = String.fromCharCode(10);
const text = publicRows.map((row) => JSON.stringify(row)).join(newline) + newline;
fs.writeFileSync('fixtures/public.jsonl', text);

Sample checker

Reference accept function

The module below is an unexecuted reference solution, not a measured production component or a latency claim. It parses once, rejects non-objects, requires an exact key set, and then applies tool-specific argument rules. Path checks stay conservative because the take-home is about refusal, not about building a full filesystem sandbox. Teams that need stronger isolation should add an operating-system sandbox outside this function rather than trusting string filters alone.

// accept-tool-call.js
const ALLOWED = new Set(['read_file', 'list_dir', 'search_text']);
const KEYS = ['args', 'call_id', 'name'];

function acceptToolCall(raw) {
  if (typeof raw !== 'string' || raw.length === 0 || raw.length > 8192) {
    return { ok: false, reason: 'size' };
  }
  let parsed;
  try {
    parsed = JSON.parse(raw);
  } catch (_err) {
    return { ok: false, reason: 'truncated_or_invalid_json' };
  }
  if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
    return { ok: false, reason: 'shape' };
  }
  const keys = Object.keys(parsed).sort();
  if (keys.join(',') !== KEYS.join(',')) {
    return { ok: false, reason: 'schema' };
  }
  if (!ALLOWED.has(parsed.name)) {
    return { ok: false, reason: 'tool_name' };
  }
  if (!/^[a-z0-9]{8,32}$/.test(parsed.call_id)) {
    return { ok: false, reason: 'call_id' };
  }
  if (!parsed.args || typeof parsed.args !== 'object' || Array.isArray(parsed.args)) {
    return { ok: false, reason: 'args' };
  }
  if (parsed.name === 'read_file') {
    const filePath = parsed.args.path;
    const badPath = typeof filePath !== 'string'
      || filePath.length === 0
      || filePath.includes('..')
      || filePath.startsWith('/');
    if (badPath) return { ok: false, reason: 'path' };
  }
  return { ok: true, reason: 'accept' };
}

module.exports = { acceptToolCall };

Illustrative grader

A small grader script keeps the human reviewer out of the comparison loop for every fixture row. It reads JSONL, calls the implementation, and writes a score file that another person can review later. The script below is illustrative glue for the packet, and it assumes the checker file sits beside the bin directory. Panels should replace the require path when they ask candidates to submit a different filename than the sample.

// bin/grade.js
const fs = require('fs');
const path = require('path');
const { acceptToolCall } = require(path.join(__dirname, '..', 'accept-tool-call'));

const file = process.argv[2];
if (!file) {
  console.error('usage: node bin/grade.js fixtures/public.jsonl');
  process.exit(2);
}

const newline = String.fromCharCode(10);
const lines = fs.readFileSync(file, 'utf8').trim().split(newline);
const misses = [];
for (const line of lines) {
  if (!line.trim()) continue;
  const row = JSON.parse(line);
  const got = acceptToolCall(row.raw);
  if (got.reason !== row.expect) {
    misses.push({ id: row.id, expected: row.expect, got: got.reason });
  }
}

const kept = lines.filter((line) => line.trim());
const report = {
  fixture: file,
  total: kept.length,
  failed: misses.length,
  misses
};
fs.writeFileSync('grade.json', JSON.stringify(report, null, 2) + newline);
console.log('failed=' + misses.length + ' total=' + report.total);
process.exit(misses.length === 0 ? 0 : 1);

Rubric

Grade the submitted artifact rather than the interview conversation that happens after the coding window closes. The table below is a starting rubric that a hiring panel can edit before the packet goes out. Points should be assigned from the grade file, not from a recollection of how confident the candidate sounded. Publish the same rubric with the prompt so candidates can see the gates before they write code.

Check	Pass condition	Points
Public fixture match	Every public row returns the expected reason	20
Hidden truncation	Cut-off JSON returns truncated_or_invalid_json	15
Exact keys	Extra or missing keys return schema	15
Tool allowlist	Unknown names return tool_name and do not run	15
Path bound	Parent segments and absolute paths return path	15
No execution	The graded process never reads args.path from disk	10
Runtime pin	grade.json was produced on the pinned shared runtime	10

A score of 70 or above can advance only when the no-execution check and the runtime pin both passed. Those two rows function as gates rather than decorative columns inside an otherwise flexible scoring table. A candidate who scores well by reading files anyway has missed the safety property this packet exists to detect. Partial credit on style should not reopen a failed gate once the grade file is written.

Failure modes the hidden set should catch

Several misses show up often when model output is treated as data that is already structured and safe. Naming those misses in the rubric lets reviewers stay consistent across a full week of submissions. The list below concerns grading practice rather than a contest to collect clever payloads from candidates. Reviewers should mark the matching row in the grade file instead of inventing a new reason during the debrief.

A parser that rescues truncated JSON by appending braces will pass a demo and fail the cut-off row.
A schema that ignores unknown keys will accept a note field that tries to change the assigned task.
A handler that reads args.path before acceptToolCall returns has already crossed the safety boundary under test.
A solution hardcoded to the four public strings will fail as soon as the hidden identifiers change.
A passing run on a newer Node release can hide a syntax or Unicode difference from the pinned runtime.
A write-up that pastes fixture rows into a model, including secrets, creates a second leak the packet should forbid.

The last miss deserves a written rule in the candidate prompt and in the grader checklist. Candidates may use a model to brainstorm case shapes, but they may not upload hidden fixtures or employer code. Credential-shaped strings belong in neither the brainstorming prompt nor the fixture archive that leaves the company. The grader should reject a submission that asks any model to rewrite expected reasons for hidden rows.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two availability claims are used as operator-supplied context only: free model access, and a free server option. This article does not state model names, token quotas, hardware sizes, time limits, or permanence of either option. Those terms must be checked in current product documentation before a hiring panel relies on them.

Free model access is useful while the grader drafts extra reject cases for the hidden file. The author can request malformed shapes, keep the useful ones, and freeze them into JSONL by hand. The model is not the judge of those frozen lines, even when a later reply sounds more confident. Once a line is frozen, no assistant may change its expected reason without a human editing the fixture.

A free server option is useful when every candidate and the grader need the same Node runtime. The practical workflow stays narrow, and the packet should list it as required steps rather than tips. Pin the major version, clone the packet, run the public command, and keep hidden fixtures on the grader account only. If that option is unavailable or differs from the documented image, pause instead of accepting laptop runs.

Availability is an operational convenience, not a benchmark result and not a promise that the option will stay unchanged. The article remains usable without any particular vendor, because any assistant and any shared runner can preserve fixtures and a runtime pin. What matters is the freeze step, the hidden file, and the refusal to execute before acceptance. Product convenience should not become an unexamined dependency inside the hiring loop or the score file.

Steps for one cohort

Freeze the allowed tool names and the exact key set in the prompt before asking any assistant for sample payloads.
Draft reject shapes with an assistant if desired, then copy only human-reviewed lines into the hidden JSONL file.
Run the public fixture command on the pinned runtime and store the grade file beside the cohort notes.
Run hidden fixtures from the grader account and treat a failed gate as a stop rather than a debate.

Who should skip this packet

This packet is a poor fit for roles that center visual design, product copy, or infrastructure work without a tool boundary. It is also a poor fit when the company cannot keep hidden fixtures private or cannot provide a shared runtime. Panels that want to grade communication style should use a separate exercise, because this one hides that signal on purpose. Teams without time to refresh fixtures should not reuse one public file for many cohorts, since candidates may memorize it.

The string path checks are a teaching control, not a complete isolation story for a production tool runner. A later system can still do harm if some other service ignores the accept decision and runs the path. The packet therefore forbids execution inside the graded process and leaves sandbox design to a separate review. A perfect score here is not permission to connect the same function to production credentials or customer data.

After the score file

A hiring loop that freezes fixtures, pins the runtime, and scores reason codes will keep its meaning when model output shifts. The reference checker above is enough to start that loop on a single afternoon without a new platform. Panels should still read current runtime terms before they depend on any free server, and they should refresh hidden rows before each cohort. Readers drafting fixtures with MonkeyCode should confirm current free-access and free-server terms, then run the pinned grader only after that check.

Grade the Fence Before the Green Diff: A Scope-Control Take-Home

Riley Zhu — Thu, 08 Oct 2026 10:48:19 +0000

A coding-agent take-home should score scope control before it scores polish, speed, or a green local demo. Candidates who cannot keep a narrow ticket fenced will ship adjacent edits that no reviewer requested. This packet gives interviewers a prompt, a rubric, a path probe, and a list of misses that still look competent. Free model access and a disposable server can host the rehearsal, but neither one certifies production behavior.

Why this packet exists

Hiring loops already see agents rename files, upgrade dependencies, and rewrite tests while they fix a single comment. Those extra edits often compile, so a green exit code hides the defect that the reviewer actually needed to catch. The defect is an expanded change set, not a syntax error or a missing semicolon in the requested line. Interviewers need a repeatable way to separate a careful operator from a candidate who accepts every agent suggestion.

Public developer discussion often mixes portfolio demos, vibe-coded sites, and agent events into one noisy stream. None of that noise replaces a scoped review of the diff the candidate is willing to merge. Earlier exercises associated with this account already covered delivery predicates, replay safety, latency gates, and untrusted exit codes. This packet stays on a different fault, which is an agent that widens a documentation ticket into unrelated project edits.

The prompt to hand the candidate

Interviewers should hand the candidate a tiny repository and a written ticket, then forbid edits on any production system. The ticket should request one documentation correction in a single allowed path, with no implied permission to clean the tree. The planted agent transcript should also propose a dependency bump, a workflow change, and a broad test rewrite. The candidate must accept, reject, or split those proposals, and each choice needs a written reason tied to risk.

Constraints that keep grading mechanical

The prompt should state constraints so grading does not collapse into taste, seniority, or familiarity with a particular editor. Allowed work is the documentation fix plus a probe that fails when the diff leaves the named allowlist. Disallowed work includes new services, secret handling, calls toward private hosts, and claims about unverified model quotas. A free model session and a free server may serve only as scratch space, and unverified limits must be recorded as unknown.

Ask for four artifacts, and reject any packet that omits one of them without a documented blocker. The artifacts are a decision note, a diff or patch series, a probe script, and a short miss log. A time box keeps unlimited polishing from hiding a missing fence behind extra prose and screenshots. Sample code inside the interviewer guide should be labeled as an unexecuted reference, not as a scored benchmark.

How to plant the fixture

Interviewers should commit the toy repository before the exercise and record that commit hash in the prompt. The planted transcript can live in a checked-in fixture so every candidate sees the same three out-of-scope proposals. The interview must not ask the model to invent attacks or to reach systems outside the toy tree. The candidate job is classification and fencing, not a live contest against an unrestricted coding model.

Rubric

Score the packet on observable behavior, not on the confidence or polish of the accompanying write-up. Partial credit applies only when the candidate documents the miss before the interviewer discovers it. A silent extra file is a fail even when the requested documentation sentence is factually correct. A beautiful README that never runs the probe is also a fail, because the fence was never exercised.

Allowlist fidelity. The accepted diff touches only paths named in the ticket, unless the note justifies a probe file that the prompt explicitly allows.
Rejection quality. Each rejected agent proposal names the risk, the missing evidence, and the safer next step in a separate ticket.
Fail-closed probe. The probe exits non-zero when the diff is missing, the allowlist is empty, or an unexpected path appears.
Environment honesty. The note states which claims came from the operator and which limits were not verified, including quota, hardware, and duration.
No secret spill. The transcript and server notes contain no credentials, customer data, or private endpoints from a real system.

Decision table

The decision table below keeps graders aligned when two interviewers read the same packet on different days. Pass rows require both a correct diff boundary and an honest statement about what the environment did not prove. Fail rows cover expansions that compile, quota claims that were never sourced, and secrets placed on a shared server. Graders should mark the row that matches the evidence, rather than averaging a vibe across the whole packet.

Evidence in the packet	Grade	Reason
Diff stays on allowlisted paths, and the probe exits non-zero on a planted extra file	Pass	The fence is specified and actually exercised
A dependency bump is accepted inside the documentation ticket	Fail	The ticket was widened without a separate change
The write-up states a token quota, machine size, or expiry that the packet did not source	Fail	Environment facts were invented
Free-server notes include a credential or a private endpoint	Fail	Scratch space was treated as a secret store
The probe is missing, but the decision note lists that blocker before grading	Partial	The miss is visible and not laundered into a pass

Sample probe

The following Python is an unexecuted proposal for interviewers, not a result from a measured or hosted run. It reads a unified diff on standard input and an allowlist file, then exits non-zero on any unexpected path. Candidates may rewrite it, provided they keep the fail-closed branches for a missing file, an empty list, and an empty diff. Interviewers should not treat a clean run of this sample as a benchmark of any hosted model.

#!/usr/bin/env python3
'''Unexecuted reference probe: fail if a diff leaves the allowlist.'''
import pathlib
import sys

def changed_paths(diff_text):
    paths = []
    for line in diff_text.splitlines():
        if line.startswith('+++ b/'):
            path = line[6:]
            if path != '/dev/null':
                paths.append(path)
    return paths

def main():
    if len(sys.argv) != 2:
        print('usage: scope_fence.py ALLOWLIST < diff', file=sys.stderr)
        return 2
    allow_file = pathlib.Path(sys.argv[1])
    if not allow_file.is_file():
        print('missing allowlist', file=sys.stderr)
        return 2
    allowed = {p.strip() for p in allow_file.read_text().splitlines() if p.strip()}
    if not allowed:
        print('empty allowlist', file=sys.stderr)
        return 2
    diff_text = sys.stdin.read()
    if not diff_text.strip():
        print('empty diff', file=sys.stderr)
        return 2
    unexpected = [p for p in changed_paths(diff_text) if p not in allowed]
    if unexpected:
        print('unexpected paths:', ', '.join(unexpected), file=sys.stderr)
        return 1
    print('scope ok')
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

Allowlist contents

A minimal allowlist for this ticket can live in a plain text file placed beside the probe. The probe file itself belongs on that list only because the prompt explicitly allowed a new tool path. Documentation edits outside the named billing note should fail even when the added prose looks helpful to a reader. Interviewers who change the ticket must change the allowlist in the same commit, or the fixture will grade the wrong boundary.

docs/billing-note.md
tools/scope_fence.py

Commands for the interviewer

These shell illustrations show the expected branch, and they are not a published performance result from any hosted run. The first command should print a scope confirmation when the allowlist contains that documentation path and the diff is otherwise empty. The second command should exit non-zero because the package manifest was never named in the ticket or the allowlist. Interviewers should run both commands against the candidate script, not against an unseen remote repository or a production host.

printf '%s\n' '+++ b/docs/billing-note.md' | python3 tools/scope_fence.py scope.allow
printf '%s\n' '+++ b/package.json' | python3 tools/scope_fence.py scope.allow

An unexecuted decision note can look like the fragment below, and it exists only to show the expected tone. It is a template, not a candidate submitted answer, and it must not be pasted into a score sheet as evidence. Each line maps one path or one unknown to a single decision, which keeps the later grading pass mechanical. Interviewers should replace the paths when their toy ticket uses different filenames or a different probe location.

Accept docs/billing-note.md because the ticket names that path.
Reject package.json because a dependency bump needs its own advisory and ticket.
Reject .github/workflows/ci.yml because workflow edits are outside the allowlist.
Unknown: token quota, server size, and session duration were not verified.

Common failure modes

Several misses recur when candidates treat the agent as an author instead of a junior patch proposer. Each miss can look productive in a portfolio screenshot, which is why the probe has to be mechanical. The list is not exhaustive, but it covers the expansions that most often sneak past a friendly reading. A strong packet rejects these moves in writing and may schedule a separate ticket for real follow-up work.

The candidate accepts a lockfile or dependency bump because the agent called the edit a security cleanup, without a separate advisory.
The candidate adds continuous integration, formatting, and a workflow file, then describes that expansion as harmless drive-by quality.
The probe shells out to a formatter that rewrites the tree, so the fence checks a moving target instead of the submitted diff.
The write-up cites a free token allotment, a machine size, or an uptime promise that this packet never verified against a primary source.
The candidate pastes repository secrets into a free server notebook so the agent can inspect a failure that the toy repo does not contain.
The decision note argues from taste, and it never maps each rejected hunk to a concrete risk, a missing fact, or a safer follow-up ticket.

Where a free session fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator brief describes MonkeyCode as an open-source project and supplies two availability claims for this draft. Those claims are free model access and a free server option, which can host the toy repository for a shared rehearsal. This article does not treat numeric quotas, model names, hardware sizes, durations, or permanence as verified facts.

Readers should read the current project terms before relying on either availability claim in a live hiring loop. A bounded rehearsal still has a clear operational shape that does not depend on a particular vendor dashboard. The candidate runs the agent against the toy repository, exports the diff, and feeds that diff to the probe. The free server, if used, should hold only the toy repository and the probe, with no production credentials mounted.

What the rehearsal must not assume

If the session ends early, the packet should record the stop and grade the written fence rather than invent a completed run. Unknown budget metadata is a reason to stop the session, not a reason to guess a quota or a hardware profile. The same pattern stays useful if every product name is removed from the interviewer guide and the candidate notes. Any editor with a local diff and a Python interpreter can host the probe without a remote account.

The product mention matters only when the interviewer wants a shared scratch server that candidates need not provision themselves. That convenience does not transfer test coverage, on-call ownership, or a security review from the toy tree to production. Teams should keep the hiring record in their existing system and delete the scratch server when the exercise ends. A public write-up may describe the method, as this one does, without reproducing a candidate submitted solution.

Who should not use this approach

This packet is a poor fit for several real hiring needs, and forcing it into those loops will waste interviewer time. Teams hiring for distributed systems, kernel work, or data modeling need a domain task, not a documentation fence. Roles that require supervised production access should not be simulated on an unverified free server or a shared notebook. Interviewers who cannot read a diff should not outsource the score to an agent summary or a green status badge.

Candidates must not be told that a free allotment is permanent, unlimited, or equal for every account and region. Security exercises that ask models to attack a named organization are outside this packet and should not be improvised here. The sample parser also misses renames, binary patches, and commits that never emit a standard new-file diff header. Pair the script with a name-only diff when the repository is local, and do not call the sample a supply-chain control.

Closing

Scope is the first hiring signal in an agent-assisted change, and a short allowlist probe makes that signal cheap to grade. Interviewers can hand out the prompt, require the four artifacts, and fail packets that widen the ticket or invent environment facts. Readers who want a disposable bench for the toy repository can review MonkeyCode current free model access and free server option. They should confirm the live terms before a candidate depends on either option for a timed exercise.

Grade the Missing Sticky Store: A Feature-Flag Review Take-Home

Riley Zhu — Thu, 08 Oct 2026 08:14:15 +0000

A hiring panel should treat a green diff from a short-context assistant as incomplete evidence, because the missing module often holds the real invariant. This packet asks a candidate to review a feature-flag patch that looked finished on a disposable two-file server. The review must name each hidden check that server never ran and must end in a reject or revise decision. Panels that already publish take-home packets can place this exercise beside a coding task without turning the interview into a product demo.

Why a short sandbox changes the review

Public portfolio repos now ship quickly, and many of those changes begin in an assistant that only saw the files an author pasted. A disposable server run can show that a snippet boots while still omitting the store, the kill switch, and the audit path production calls. Interviewers who grade only the visible diff will favor candidates who trust a narrow passing run over candidates who map constraints. This packet makes that narrowness part of the specification, so the candidate scores the boundary rather than the coding style.

The exercise stays fair when every candidate receives the same sealed fixture, the same prompt, and the same ninety-minute time box. It does not require a paid model seat, a private monorepo, or a shared cloud account for the panel. It also does not ask the candidate to attack a live system, scrape hidden endpoints, or expand the fixture with outside data. The work is a review of supplied files plus a small local probe that runs on a laptop inside one folder.

The candidate prompt

Give the candidate the brief below without edits, and attach the fixture in the next section as the only allowed corpus. The brief tells the candidate that an assistant read only flags.js and flags.test.js, then reported a passing test from a disposable server. Hidden files in the packet were not mounted for that run, and the candidate must decide whether the patch may merge. The review is capped at four hundred words, must cite file names, and must not rewrite the service or add dependencies.

State that network calls are out of scope and that tool use is allowed only inside the supplied folder. Pasting the fixture into an external chat is a process failure when the team forbids third-party retention of interview material. That rule matters once the fixture grows to include realistic identifiers, even if the first version uses only synthetic keys. Record the prohibition in the same brief so every candidate sees the data rule before opening any file.

A candidate reviews a patch that an assistant produced after reading only flags.js and flags.test.js on a disposable server. That server reported a passing test, but hidden files in this packet were never mounted for the assistant run. The review stays within four hundred words, states a merge decision, and cites each file plus the predicate that citation protects. Rewrites, new dependencies, probes outside this folder, and external pastes of the packet are all process or scope failures.

Fixture the assistant did not fully see

The visible patch replaces a static boolean with a percentage bucket, and it looks complete if the reviewer never opens the store. The assistant could hash a user id, compare the bucket with a percent, and return an enabled flag without reading any other module. The hidden file in the candidate packet keeps a sticky assignment and a kill switch that the new resolver never calls. A one-file test can stay green while both invariants fail, which is the exact trap the rubric is built to catch.

Visible resolver

// flags.js — visible to the assistant
const { createHash } = require('crypto');

function bucket(userId, percent) {
  const hex = createHash('sha256').update(String(userId)).digest('hex');
  const n = parseInt(hex.slice(0, 8), 16) % 100;
  return n < percent;
}

function resolveFlag(userId, flag, config) {
  const percent = config[flag] ?? 0;
  return { flag, enabled: bucket(userId, percent) };
}

module.exports = { resolveFlag, bucket };

// flags.test.js — the only test the assistant ran
const test = require('node:test');
const assert = require('node:assert/strict');
const { resolveFlag } = require('./flags');

test('full percentage enables the bucketed user', () => {
  const result = resolveFlag('u1', 'beta', { beta: 100 });
  assert.equal(result.enabled, true);
  assert.equal(result.flag, 'beta');
});

Hidden invariants

// assignmentStore.js — hidden from the assistant run
class AssignmentStore {
  constructor(rows) {
    this.rows = new Map(rows);
  }

  get(userId, flag) {
    return this.rows.get(`${userId}:${flag}`) ?? null;
  }
}

function killSwitchOn(config, flag) {
  return Boolean(config.killed && config.killed[flag]);
}

module.exports = { AssignmentStore, killSwitchOn };

Local probe

The probe is a proposed local check for this packet, not a production harness and not a benchmark. It loads the sticky row and the kill switch, then compares both with the patch output. Interviewers should treat a printed miss list as a fixture self-check, not as the candidate score. Candidates may extend the probe, but the written review still has to name the predicates in prose.

// probe.js — run locally; do not present the output as a benchmark
const { resolveFlag } = require('./flags');
const { AssignmentStore, killSwitchOn } = require('./assignmentStore');

const store = new AssignmentStore([['u1:beta', false]]);
const config = { beta: 100, killed: { beta: true } };

const sticky = store.get('u1', 'beta');
const fromPatch = resolveFlag('u1', 'beta', config).enabled;
const killed = killSwitchOn(config, 'beta');

const misses = [];
if (sticky === false && fromPatch === true) misses.push('sticky-override');
if (killed && fromPatch === true) misses.push('kill-switch');

const decision = misses.length ? 'reject' : 'accept';
console.log(JSON.stringify({ decision, misses }));

Running node probe.js in that folder prints a reject decision with both miss names when the patch ignores the store and the switch. The command is a reproducibility check for the interviewer, not a score, and it is not a latency benchmark. A candidate who only pastes that JSON without explaining the predicates has not finished the written review. Interviewers should run the probe once on a clean machine before the first candidate and should fix relative paths if the layout differs.

Rubric

Score the written review, not the assistant that drafted the patch, and use the same four rows for every candidate. Sticky assignment is worth up to three points and requires a clear statement that an existing false row wins over a new hash bucket. A missing row may fall through to the percentage, and the review should say that fall-through is allowed only after the sticky lookup. Kill switch is worth up to three points and requires a killed flag to stay disabled even at full percentage.

The review must also note that the visible patch never reads the killed map, so the helper in the hidden file is currently dead. Evidence boundary is worth up to two points and requires an explicit refusal to treat the two-file passing run as merge proof. Decision hygiene is worth up to two points and requires a reject or revise call with file citations and no drive-by rewrite. A total of seven or higher passes, and a six may proceed only when the miss is wording rather than a dropped invariant.

Do not average this score with a live-coding score from a different exercise, because the constructs are not interchangeable. The packet measures review judgment under a stated context limit, which is a separate hiring signal from implementation speed. Panels should publish the four rows with the prompt so candidates are not graded against a hidden taste rule. Keep the point values stable across a hiring cycle, and change them only between cycles with a written note.

Required resolution order

Read the kill switch and force the flag disabled when the killed map contains the flag name.
Read the sticky store and return the stored boolean when a row exists for that user and flag.
Fall through to the percentage bucket only when the kill switch is off and no sticky row exists.
Treat a two-file passing test as non-evidence until a probe imports the hidden module and fails on both misses.

Check	If true	If false or missing
Kill switch for the flag	Return disabled and stop	Continue to the sticky lookup
Sticky row for user and flag	Return the stored boolean and stop	Continue to the percentage bucket
Percentage present	Return whether the hash bucket falls inside that percent	Return disabled

Sample review that should pass

The following review is a proposed answer key for interviewer calibration, not a transcript from a real candidate. It should not be published as evidence that any person sat this test or that any team adopted the packet. The sample cites both files, names both predicates, and refuses the green run without claiming a timing result or a customer incident. Interviewers should accept equivalent structure when the prose differs, provided the decision and the citations stay intact.

The patch in flags.js computes a stable bucket and returns the enabled bit from that bucket alone. The assignment store already holds a sticky false value for the synthetic user and the beta flag, and the new resolver never calls get. A user held at disabled would flip on whenever the percentage is high, which breaks the sticky contract the store exists to protect. The kill-switch helper reads the killed map for beta, but resolveFlag never consults that helper at any branch.

A killed flag therefore still enables when the percentage is one hundred, even though the hidden module already encodes the stop. The assistant reported a green flags test after a two-file run, and that run cannot import the store or the switch. That report is not merge evidence, so the decision is reject until the kill switch, the sticky row, and the bucket resolve in that order. Add a local probe that fails when either hidden predicate flips, and keep that probe inside the supplied folder.

Common failure modes

Candidates miss this packet in a small set of repeating ways, and naming those ways in the debrief keeps the panel off taste debates. The first cluster is a competency miss, where the candidate argues about the hash and never mentions the unread store. The second cluster is a process miss, where the candidate treats the disposable server as production or pastes the packet into a forbidden chat. Mark competency misses and process misses separately, and fail the packet on a process miss when a written data rule exists.

Treating the hash function as the defect, and proposing a different hash, while the sticky row and the kill switch remain unread.
Accepting the patch because the visible test exits zero, which repeats the assistant evidence boundary instead of challenging it.
Rewriting the resolver inside the review, which spends the time box on unrequested code and hides whether the invariants were seen.
Calling the disposable server production-like without listing mounted files, which turns a convenience into a false environment claim.
Pasting the packet, including any later realistic identifiers, into a hosted chat after the brief forbade third-party retention.

Reproducing the narrow run

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant, and the only availability claims used here are free model access and a free server option. An interviewer can reproduce the short-context run on that disposable server, mount only the two visible files, and keep the hidden files off the run. Free access lets a panel try the same fixture without buying a seat for every interviewer who wants to see the narrow session. Those claims do not name a model, a quota, hardware, or duration, and current documentation must be checked first.

Clone the fixture into a fresh directory, and start the free server option only when the panel wants the original two-file assistant session. Do not mount the assignment store for that session, and keep the candidate review on the laptop rather than in the hosted chat. Record the date the terms were checked, because free access is an operator-supplied current option and not a permanence promise. If the documented offer has changed, drop the hosted step and run the probe locally, since the rubric does not depend on the host.

Readers who want a calibration pass can open the MonkeyCode project and confirm the current free access terms before mounting any files. Compare the assistant note with the local probe output, and treat that comparison as a panel aid rather than a candidate requirement. That single check is enough, and this article does not ask readers to sign up under a deadline or to prefer one assistant over another.

Limitations and who should skip this

This packet does not measure system design, people management, or security research, and it should not be the only signal in a hire. Teams that ban hosted models should skip the assistant step and still use the local probe, the prompt, and the rubric. Teams that need every candidate to hit an identical model build should not depend on a free tier whose routing may change. Roles that never touch feature flags will find the domain distracting, and a neighboring packet in their actual stack would be fairer.

Do not load real user identifiers, customer code, or secrets into the fixture or into any hosted run used to reproduce the assistant. The probe is a proposed example, and this article does not present it as an executed benchmark or as a measured pass rate. No hiring outcomes, view counts, or customer stories are claimed, because none were supplied for this account. Interviewers should run the probe on their own machines before the first scheduled candidate and should retire the fixture if a path breaks.

What the hiring file should keep

A short-context patch can be locally consistent and still wrong, and the hiring packet should score that gap directly. The prompt, the two-file boundary, the hidden store, the probe, and the ten-point rubric are enough to run the exercise in one sitting. Free model access and a free server can reproduce the narrow run when the panel verifies current terms and keeps hidden files unmounted. The candidate review, not the assistant green note, is the artifact that belongs in the hiring file.

Last-Unit Checkout as a Hiring Packet: Predicate, Probe, and Misses

Riley Zhu — Thu, 24 Sep 2026 21:31:05 +0000

A concurrent checkout packet should reject any patch that guards stock with a process-local lock. Two separate worker processes can read one remaining unit and both persist a paid order. The packet hides that second worker behind a deliberately slow payment handler and a delayed commit. A conditional update on the stock row, not a local mutex, is the behavior the rubric rewards.

Why a single-process test lies

An in-memory lock looks correct when the whole suite shares one interpreter and one event loop. The hidden probe starts a second process against the same database and the same sku. Both processes read a remaining quantity of one unit before either process writes its order. Only a predicate on the update statement can make one of those two writes fail.

Hiring loops often stop at the happy path because a single green test feels like coverage. That habit is especially expensive when a coding agent rewrites the handler and the reviewer never leaves one process. The packet in this article treats the second process as the real exam, not as an optional stretch goal. A candidate who cannot explain the lost update should not pass on naming or style alone.

Assignment prompt

The prompt should stay short so the candidate spends time on the race, not on product flavor. It asks for a function that reserves stock and records an order inside one database transaction. It forbids solutions that only work inside a single process or inside a single test thread. It also asks for a test that still fails if that process-local constraint is later removed.

The wording below is only a proposed prompt, not a transcript taken from a live interview loop. Interviewers can paste it into a private gist or a short-lived repository with the fixture already committed. The candidate, whether a human or an agent, should change only the reservation function and its tests. Hidden probe files should stay read-only so a later cleanup pass cannot delete the planted race.

Implement reserve_and_order(conn, sku, qty, request_id).
Decrement on_hand only when the row still has enough units.
Insert exactly one order row for request_id when the decrement succeeds.
Roll back both writes when either write fails.
Do not rely on a lock that lives only inside this process.
Include a test that fails if two callers can oversell one unit.

Fixture the candidate does not edit

The fixture inserts one sku with a single unit and a version column set to zero. The supplied helper pauses the first worker after its select and before its stock write. That pause lasts long enough for the second process to observe the same unchanged stock snapshot. Candidates who delete the pause to quiet the suite should lose credit on the rubric.

The order table carries a unique constraint on request_id, which catches a naive retry that inserts twice. That constraint does not catch two different request identifiers selling the same last stock unit. Interviewers should say that distinction out loud during the readout, because agents often treat uniqueness of the request as uniqueness of the stock. The fixture therefore issues two distinct request identifiers so that the stock race stays clearly visible.

Rubric that ignores a lonely green test

Reviewers should score the two-process probe before they score names, comments, or other formatting choices. A single-process happy-path test remains necessary and is still insufficient as a clear final hire signal. Reviewers should use the checks below as a pass-fail table rather than as a loose impression.

Pass-fail table

Check	Pass condition	Fail condition
Happy path	One order exists and on_hand is zero	Exception, missing order, or leftover stock
Two-process probe	Exactly one order and no negative stock	Two orders, or on_hand below zero
Write predicate	Database rejects the stale decrement	Python computes the next quantity and writes it
Lock scope	Safety holds across two processes	threading.Lock, a global dict, or a file lock on one host only
Transaction shape	Reserve and order commit or roll back together	Order inserted after a failed reserve, or reserve kept after a failed insert
Conflict result	Caller receives a conflict, not a fake success	Exception swallowed, then a success value returned
Retry behavior	Same request_id does not create a second order	Blind retry inserts another order or double-decrements

A patch can pass the first row and fail the second row, and that split is the point of the packet. Interviewers should record which specific rubric row failed rather than collapsing the result into one note. Numeric weights are optional here, but a failed two-process row should block a hire signal. Comments on naming style should wait until after the two-process probe has turned fully green.

Reference solution

The following snippet is an unexecuted example for reviewers, not a claim that a particular vendor ran it. It uses a conditional update and then inserts the order only when that update returns a row. The version column advances so a later writer can detect a stale read of other fields. The unique key on request_id remains only a replay backstop, not the actual stock guard.

def reserve_and_order(conn, sku, qty, request_id):
    if qty <= 0:
        raise ValueError('qty must be positive')
    with conn:
        row = conn.execute(
            '''
            UPDATE inventory
               SET on_hand = on_hand - ?,
                   version = version + 1
             WHERE sku = ?
               AND on_hand >= ?
            RETURNING on_hand, version
            ''',
            (qty, sku, qty),
        ).fetchone()
        if row is None:
            raise StockConflict(sku)
        conn.execute(
            '''
            INSERT INTO orders (request_id, sku, qty)
            VALUES (?, ?, ?)
            ''',
            (request_id, sku, qty),
        )
    return 'reserved'

The quantity predicate is the actual fence, because two transactions cannot both satisfy it for the last unit. That returning clause tells the caller whether this particular transaction has actually won the stock write. A missing row means another worker already took the unit, so the order insert must not run. Wrapping both statements in one transaction keeps a failed insert from leaving a silent decrement behind.

Reviewers who prefer an explicit version compare can add a version predicate when the application already loaded that version. That form is stricter when other columns change, and it is unnecessary for a pure decrement of on_hand. The sample omits the compare so candidates are not punished for a correct quantity predicate. Either form fails the rubric if the new quantity is calculated in process memory and written blindly.

Commands for the two-process probe

Run the narrow schema test first so a broken table fails before the race even starts. Then run the probe that forks two workers against one database file or one server database. The example commands below are proposed, and they assume a local checkout with pytest and a small harness. Teams should confirm the printed row counts before they trust a local adaptation of the harness.

python -m pytest -q tests/test_reserve.py::test_single_buyer
python reserve_probe.py --workers 2 --sku SKU-1 --qty 1 --pause-ms 200

What the probe must print

The probe should print one reserved result and one stock conflict, then exit non-zero if the order count is not exactly one. A zero exit code with two inserted orders is a failed packet even when both child processes exit cleanly. Interviewers should keep the database file after the run so the readout can show the rows. Deleting that database file before the readout would hide the only row-level evidence that still matters.

SELECT on_hand, version FROM inventory WHERE sku = 'SKU-1';
SELECT request_id, qty FROM orders WHERE sku = 'SKU-1';

What the queries must show

Expected values are a zero on-hand balance, a version that advanced once, and a single order row. Two separate order rows mean that the quantity predicate was missing or applied much too late. A negative on-hand balance means the writer used a blind assignment from a stale Python integer. Those two failing outcomes should be copied into the written score note without any later softening.

Failure modes worth scoring

Patterns that look finished

A threading lock makes the single-process test pass, while the second process still sells the last unit.
The handler reads the balance, subtracts in Python, and writes that integer back without a guarding predicate.
The order insert runs before the code inspects the update row count, so a conflict still leaves a paid row.
A broad retry repeats the insert under a new request identifier and decrements stock a second time.
The conflict exception is swallowed, and the function returns success to a caller that will ship the item.
A cleanup pass deletes the comment that states why the predicate must live in the update, and the next edit drops the clause.

The last mode is a clarity failure rather than a concurrency failure, and it still predicts the next regression. Agents that equate clean code with fewer lines often remove the sentence that documents the lost update. The rubric can award a separate note for that deletion without letting it outweigh a wrong write. Keep the comment if it states an invariant the test does not spell out in nearby prose.

Where a free model pass fits this packet

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can draft several candidate patches from one prompt for the rubric to score. MonkeyCode's free server option can host the database and the two worker processes so the race is not simulated inside one notebook kernel. Those two options are availability claims supplied for this workflow, not measured limits, named models, or a fixed promise. The packet stays usable on any laptop if that hosted option is declined or later unavailable.

What to send and what to withhold

Reviewers should paste the prompt and the fixture, not production connection strings, customer rows, or live payment keys. Generated patches still need the two-process probe, because a fluent explanation of lost updates can wrap a blind write. Treat the model output as an untrusted candidate submission that still needs a readable diff review. A free server is only a convenience for the probe, not evidence about a larger catalog or another isolation level.

Readers who already run a hiring loop can copy the table and the probe without any hosted tool. Readers who want a scratch server plus free model access for those drafts can try MonkeyCode for this narrow workflow. That trial does not replace the rubric, and it should not become the reason a weak patch is accepted. A declined or expired trial changes none of the pass conditions already listed in the table above.

Limits and poor fits

This packet does not prove isolation behavior under a broad anomaly suite, and it does not model a warehouse with many skus. It will mislead teams who need a multi-region inventory design or a formal proof of linearizability. Candidates who were told to use an existing queue lease should not be scored against this checkout prompt. People evaluating a payment capture, a tax engine, or a consensus protocol should write a different packet.

SQLite running in rollback-journal mode can serialize writers so tightly that a sloppy patch appears safe. Reviewers who choose SQLite should enable WAL mode and still run two processes, or point both workers at one server database. A file lock on a single host fails the lock-scope row even when the local probe looks calm. None of these commands were executed for this article, so teams should confirm the row counts themselves.

What to keep from the readout

The hire signal is a conditional write that survives two processes, plus a caller-visible conflict when the predicate misses. Everything else inside the submitted diff should be treated as commentary around that one stored result. A green exit code from the single-buyer test is not a pass, and a tidy function that returns true after an oversell is a fail. Interviewers who keep the score table next to the order query will make that distinction in one readout.

At-Least-Once Webhooks as a Take-Home: Rubric, Hidden Replays, and Failure Modes

Riley Zhu — Tue, 22 Sep 2026 15:30:26 +0000

Webhook take-homes expose a reliability gap that ordinary green unit tests rarely catch in otherwise polished candidate submissions. At-least-once delivery is the default producer contract, yet many solutions acknowledge the HTTP request before durable idempotent storage finishes. The packet below treats early 2xx responses, missing signature checks, and unordered event replays as automatic scoring failures. Interviewers can score a human candidate or an AI coding agent with the same fixtures, rubric, and hidden replay schedule.

Why this packet exists

Payment and provisioning webhooks arrive more than once, sometimes minutes apart, and occasionally in reverse sequence order. A handler that inserts one row and returns 200 still fails when a concurrent replay creates a second captured charge. Signature checks that run after JSON parsing invite body tampering that trusted fixtures in unit tests never actually exercise. The assignment therefore hides replay, reordering, and mild clock skew inside the scorer rather than the public prompt.

Tool-calling agents fail this style of task for a different but related reason than typical backend candidates. Many agents copy a framework tutorial, return JSON, and stop once a happy-path curl command prints a 200 status. They rarely add a unique index or a crash-restart case unless the public prompt names those controls explicitly. The packet is written to keep those controls in the hidden suite so the public prompt stays short.

Public prompt for the candidate

The candidate receives a small HTTP service skeleton, a documented HMAC header, and a local SQLite file path. The service must accept POST /webhooks/payments, verify the signature, and apply each event at most once. Public notes state that the producer retries on timeouts and on any status outside the 2xx range. The candidate may add tables, indexes, and middleware, but must not rename headers or alter the JSON event shape.

Wire contract

The public README shows one request shape and forbids any additional required headers beyond the three listed. The signature header uses a sha256= prefix, and the body must be hashed exactly as received on the wire. Candidates who pretty-print JSON before verification will fail the hidden tamper and reorder cases together.

POST /webhooks/payments HTTP/1.1
Content-Type: application/json
X-Webhook-Id: evt_01HZX4K2
X-Webhook-Timestamp: 1690000000
X-Webhook-Signature: sha256=d7c1...

{"event_id":"evt_01HZX4K2","type":"payment.captured","amount_cents":4099,"currency":"USD","order_id":"ord_9k2","sequence":3}

Public rules

The README lists six rules and nothing else about concurrency, disk fsync, or replay storms. Candidates who ask for extra rules during the interview should be pointed back to those six bullets. Anything about WAL mode, busy timeout, or kill -9 stays in the hidden scorer by design. The starter commands are intentionally boring so the work sits in handler correctness rather than tooling.

The signature is HMAC-SHA256 over the timestamp, a dot, and the raw body, using WEBHOOK_SECRET from the environment.
The handler must reject timestamps older than five minutes and timestamps more than thirty seconds in the future.
The JSON event_id is the idempotency key and must equal the X-Webhook-Id header value.
For one order_id, a later sequence must not mutate balances before every earlier sequence exists.
Duplicate delivery of an already applied event must return 2xx and must not change stored amounts.
The process must remain correct after a crash that occurs before the producer issues its next retry.

Starter commands in the repository look like the following block.

export WEBHOOK_SECRET=test_secret_do_not_reuse
export WEBHOOK_DB=./webhook.db
python -m pip install -r requirements.txt
python app.py

Rubric used by the hidden scorer

The scorer awards one hundred points and rejects a submission that returns 2xx for an invalid signature. Partial credit exists for durable dedupe without sequence gating, but signature failures are not negotiable. Timeouts that still persist the event can pass the crash case if a later retry becomes a no-op. Returning 409 for a true duplicate is a fail because the producer will retry forever.

Area	Points	Pass condition
Signature and raw body	25	Invalid HMAC, truncated body, and header/body id mismatch all yield non-2xx
Timestamp window	10	Skewed clocks outside the documented window yield non-2xx
Durable idempotency	25	Two concurrent replays produce one applied row after process restart
Sequence gating	20	Sequence 3 cannot apply before sequences 1 and 2 for the same order
Duplicate ACK contract	10	Second delivery of an applied event returns 2xx with unchanged totals
Crash before ACK	10	Process kill after the write but before the response still yields a single apply

Decision table for HTTP status choices

Status codes are part of the idempotency contract, not an afterthought for the web framework default handler. The table below is the scoring key for duplicates, gaps, and auth failures in this packet. Candidates who collapse every error into 500 will cause unnecessary retries and can still pass durable storage checks. Candidates who return 2xx on signature failure fail the packet regardless of database contents.

Situation	Mutate store	Status family	Producer next step
Valid new event, sequences complete	Yes, once	2xx	Stop
Valid duplicate of an applied event	No	2xx	Stop
Valid event with a sequence gap	No	409 or 503	Retry later
Bad HMAC or id mismatch	No	401 or 400	Do not retry as-is
Timestamp outside the window	No	401	Do not retry as-is
Unique-index race on insert	No extra row	2xx	Stop

Hidden fixtures the public prompt never lists

The scorer runs after the candidate declares the server ready on port 8080. It does not use the candidate's unit tests as evidence of concurrent replay safety. Each case below is a separate process restart with a copied database file. Interviewers should treat the YAML schedule as confidential material, not as extra README flavor text.

The replay pair posts the same captured event twice in parallel with identical signatures and timestamps.
The reorder pair sends sequence 3 first, then sequence 1, then sequence 2, each with a valid signature.
The tamper pair computes a valid signature, then changes one digit in amount_cents before the POST.
The clock pair sets the timestamp sixty seconds in the future relative to the scorer host clock.
The crash pair closes the TCP connection after the first response byte if the row is not yet visible.
The cross-order pair uses two different order_id values that must not block each other during sequence checks.

A compact driver for the replay pair looks like this labeled reference.

# Reference scorer fragment. This is an unexecuted example, not a published benchmark.
import hashlib, hmac, json, os, threading, urllib.request

SECRET = os.environ["WEBHOOK_SECRET"]
BODY = json.dumps({
    "event_id": "evt_01HZX4K2",
    "type": "payment.captured",
    "amount_cents": 4099,
    "currency": "USD",
    "order_id": "ord_9k2",
    "sequence": 1,
}, separators=(",", ":")).encode()

def post_once():
    ts = "1690000000"
    mac = hmac.new(SECRET.encode(), ts.encode() + b"." + BODY, hashlib.sha256).hexdigest()
    req = urllib.request.Request(
        "http://127.0.0.1:8080/webhooks/payments",
        data=BODY,
        headers={
            "Content-Type": "application/json",
            "X-Webhook-Id": "evt_01HZX4K2",
            "X-Webhook-Timestamp": ts,
            "X-Webhook-Signature": f"sha256={mac}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req) as resp:
        return resp.status

threads = [threading.Thread(target=post_once) for _ in range(2)]
for t in threads:
    t.start()
for t in threads:
    t.join()

Reference solution labeled as a sample

The sample below is a teaching sketch for reviewers, not a claim that this exact process ran in production traffic. It verifies the HMAC against the raw bytes, then opens an immediate SQLite transaction, then inserts the event under a unique index. Sequence gating uses a count of prior rows for the same order rather than an in-memory map. The HTTP status is chosen only after the commit returns.

# Sample solution for reviewers. Treat as labeled reference code, not live metrics.
import hashlib
import hmac
import json
import os
import sqlite3
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

SECRET = os.environ["WEBHOOK_SECRET"].encode()
DB_PATH = os.environ.get("WEBHOOK_DB", "webhook.db")

def db():
    conn = sqlite3.connect(DB_PATH, timeout=5, isolation_level=None)
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute("PRAGMA busy_timeout=5000")
    conn.execute(
        """
        CREATE TABLE IF NOT EXISTS events (
          event_id TEXT PRIMARY KEY,
          order_id TEXT NOT NULL,
          sequence INTEGER NOT NULL,
          amount_cents INTEGER NOT NULL,
          applied INTEGER NOT NULL
        )
        """
    )
    conn.execute(
        "CREATE UNIQUE INDEX IF NOT EXISTS order_seq ON events(order_id, sequence)"
    )
    return conn

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        if self.path != "/webhooks/payments":
            self.send_error(404)
            return
        length = int(self.headers.get("Content-Length", "0"))
        raw = self.rfile.read(length)
        header_id = self.headers.get("X-Webhook-Id", "")
        ts = self.headers.get("X-Webhook-Timestamp", "")
        sig = self.headers.get("X-Webhook-Signature", "")
        try:
            ts_i = int(ts)
        except ValueError:
            self._reply(400, b"bad timestamp")
            return
        now = int(time.time())
        if ts_i < now - 300 or ts_i > now + 30:
            self._reply(401, b"timestamp window")
            return
        expected = "sha256=" + hmac.new(
            SECRET, f"{ts}.".encode() + raw, hashlib.sha256
        ).hexdigest()
        if not hmac.compare_digest(expected, sig or ""):
            self._reply(401, b"bad signature")
            return
        try:
            payload = json.loads(raw.decode("utf-8"))
        except json.JSONDecodeError:
            self._reply(400, b"bad json")
            return
        if payload.get("event_id") != header_id:
            self._reply(400, b"id mismatch")
            return
        order_id = payload["order_id"]
        sequence = int(payload["sequence"])
        amount = int(payload["amount_cents"])
        conn = db()
        try:
            conn.execute("BEGIN IMMEDIATE")
            row = conn.execute(
                "SELECT applied FROM events WHERE event_id = ?", (header_id,)
            ).fetchone()
            if row:
                conn.execute("COMMIT")
                self._reply(200, b"duplicate")
                return
            prior = conn.execute(
                "SELECT COUNT(*) FROM events WHERE order_id = ? AND sequence < ? AND applied = 1",
                (order_id, sequence),
            ).fetchone()[0]
            if prior != sequence - 1:
                conn.execute("ROLLBACK")
                self._reply(409, b"out of order")
                return
            conn.execute(
                "INSERT INTO events(event_id, order_id, sequence, amount_cents, applied) VALUES (?,?,?,?,1)",
                (header_id, order_id, sequence, amount),
            )
            conn.execute("COMMIT")
            self._reply(200, b"applied")
        except sqlite3.IntegrityError:
            conn.execute("ROLLBACK")
            self._reply(200, b"duplicate")
        finally:
            conn.close()

    def _reply(self, code, body):
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

Reviewers should notice three deliberate choices in that sketch before they treat it as a scoring baseline. HMAC comparison uses compare_digest on the full sha256= prefix rather than a parsed hex string with equality. The unique index turns a race between two inserts into a handled integrity error that still returns 2xx. Sequence 409 is reserved for gaps, not for duplicates, so the producer retries only when waiting can make progress.

Common failure modes the scorer records

Most rejected packets share a small set of implementation patterns rather than rare or exotic protocol mistakes. The list below is useful as a debrief document after the timed interview window closes. Interviewers should map each failing fixture to one pattern instead of arguing about framework taste. Agents and humans tend to land on the same eight bugs when the public prompt stays short.

The handler returns 200 from the web framework and writes SQLite on a background task that the tests never await.
The handler parses JSON first, then signs the re-serialized object, so key order changes break valid producer signatures.
The handler stores seen event ids in a process-local set that disappears on restart and admits a second capture.
The handler applies sequence 3 when sequences 1 and 2 are missing, then cannot repair the order total.
The handler returns 409 for an already applied duplicate, which traps the producer in an infinite retry loop.
The handler uses equality on hex digests, or compares after lowercasing only one side of a mixed-case header.
The handler accepts a timestamp as a float string and silently truncates, which widens the window beyond five minutes.
The handler locks per event_id in memory and still double-applies across two worker processes on one database file.

Local commands for the interviewer

Interviewers should run the hidden suite from a second checkout so candidates cannot read the replay schedule by accident. The commands below assume the candidate server is already bound to 127.0.0.1 on port 8080. A separate checkout also prevents an agent from grepping scorer.py during an open-book take-home. The interviewer should record HTTP status codes and the SQLite dump for the debrief notes.

export WEBHOOK_SECRET=test_secret_do_not_reuse
python scorer.py --base-url http://127.0.0.1:8080 --db ./webhook.db --cases hidden.yaml
sqlite3 ./webhook.db 'SELECT order_id, sequence, amount_cents, applied FROM events ORDER BY 1,2;'

A passing database after the reorder case contains sequences 1, 2, and 3 exactly once for ord_9k2. A failing database contains sequence 3 alone, or two rows that incorrectly share the same event_id. Interviewers should keep the YAML case file out of the candidate tarball and should rotate WEBHOOK_SECRET per session. The sqlite3 query is the fastest way to explain a failure during the live debrief without opening application logs.

Where hosted free model access fits this workflow

A local runner remains the source of truth for fixtures, HMAC secrets, and the hidden replay schedule used in scoring. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Teams without spare dedicated hardware can run the same public prompt against MonkeyCode free model access. The free server option is enough for that step after the local hidden tests already fail. No model name, quota, duration, or benchmark is claimed here beyond that free model access and a free server option exist.

The useful workflow is to paste the public prompt, collect a patch, then execute the hidden scorer on the interviewer's machine. Agents that only print a handler without a unique index fail the concurrent replay case in that second step. That split keeps HMAC secrets off the remote side and keeps the hidden rubric honest during scoring.

Limitations and who should not use this packet

This packet is a teaching filter for webhook idempotency, not a complete payments platform or a security audit. It does not cover rotating secrets, body size limits, replay caches with TTL, or multi-region conflict resolution. SQLite busy handling is enough for the take-home and is the wrong isolation story for a multi-primary production datastore. Reviewers should treat the sample server as a scoring aid rather than a template for public internet exposure.

Skip this assignment when the role does not own HTTP consumers or durable webhook processing at all. Skip it for junior screens that only need routing and JSON parsing, because the hidden suite will fail almost every submission. Skip it when the interviewer cannot run the scorer locally, because remote demos without the replay file prove almost nothing. The packet also wastes time when the team already has a production idempotency library with an enforced review checklist.

The same structure still works when the candidate is an AI agent rather than a person in an IDE. The public prompt stays short, and the hidden suite stays private during the timed interview window. The debrief then lists concrete failure modes instead of a vague complaint about prompt quality. Interviewers who want a next iteration can add signed timestamp rotation without changing the rubric shape.

Mean Latency Is Not a Pass: A Timeout-Aware Load-Check Take-Home

Riley Zhu — Mon, 21 Sep 2026 13:50:24 +0000

Hiring screens that accept a green mean-latency number routinely miss bimodal services that timeout under a stated budget. This proposed take-home packet treats measurement design as the deliverable, not as an optional afterthought. The fixture is a small Node.js HTTP service with a hidden timeout cliff, plus a checker that must fail closed. The listings below are a proposed reference for local reproduction and are not production benchmark results.

The packet exists because generated clients often loop one request, print an arithmetic mean, and declare the API healthy. That workflow resembles a vanity dashboard more than an SLO, and it now appears frequently in agent-authored take-homes. A service can answer most calls in a few milliseconds and still drop a tail that exceeds the product budget. Mean latency then looks calm while callers retry, pile on, and turn a small stall into an outage.

The failure this packet isolates

The grader should score three claims together, because a fast happy path is not the same as a passing check.

The service stays correct under overlapping clients, including invalid JSON and missing fields.
Each request has a deadline, and a late HTTP 200 counts as a failure against the budget.
Reported numbers include percentiles, timeout rate, and error rate, not only a mean.

A mean-only checker can bless a handler that is fast on ninety-five percent of calls and over budget on the rest. Closed-loop tests make the lie worse by waiting for each response before issuing the next call. That waiting hides lost capacity, which load testers have long described as coordinated omission. This packet therefore grades the measurement method as strictly as the service implementation.

Why the mean hides a cliff

Consider a bimodal handler under a five-hundred-millisecond budget. Ninety-five percent of requests return in about ten milliseconds, while five percent sleep long enough to miss the budget. The arithmetic mean can still land near one hundred fifty milliseconds, which looks acceptable on a summary slide. Percentile views and timeout counts tell a different story, because the slow class already violated the budget. Treating those late two-hundred responses as successes then trains both humans and agents to ship a cliff.

The second-phase trick is to run the candidate checker against two servers that share a contract. A clean server must produce pass: true. A stalled fixture must produce pass: false even when its mean still looks modest. Checkers that never abort, never compute percentiles, or never count timeouts will pass the wrong server.

Take-home prompt for the candidate or agent

Time box: 75 minutes. Stack: Node.js 18 or later, standard library only. Constraint: no extra packages, no cloud accounts, bind only to 127.0.0.1.

Prompt text to paste:

Build two programs in one directory.

server.js listens on 127.0.0.1:8080 and handles POST /checkout with body { "cartId": string, "items": integer }. Valid carts return 200 and { "ok": true, "totalItems": <n> }. Invalid JSON or missing fields return 400 with { "ok": false }. Unknown paths return 404. The process must remain up under concurrent requests.
loadcheck.js issues one hundred twenty timed requests to that endpoint with concurrency eight, after a twenty-request warmup that is discarded. Each request must abort at 500ms. Print one JSON object and then exit 0 on pass or 2 on fail.

Required JSON shape:

{
  "meanMs": 0,
  "p50Ms": 0,
  "p95Ms": 0,
  "p99Ms": 0,
  "timeoutRate": 0,
  "errorRate": 0,
  "pass": true
}

pass may be true only when timeoutRate is 0, errorRate is 0, and p95Ms is at most 500. A low mean cannot override a high p95 or a nonzero timeout rate. Timed-out calls must be counted in timeoutRate and included in the percentile sample at 500ms, so a tail cannot vanish from p95.

Hidden grader note, not shown to the candidate: keep a fixture server that stalls every twentieth request by 800ms. Run the candidate checker against that fixture and against a clean server. The checker must fail the fixture and pass the clean process.

Rubric

Score out of 12. A packet that only prints a mean cannot pass, even if server.js is tidy.

Score	Signal	What the grader actually runs
0–2	Server contract	Valid POST returns 200; bad JSON returns 400; process survives overlap
0–3	Deadlines	Each call aborts at 500ms; hung sockets do not stall the checker
0–3	Statistics	Warmup discarded; p50/p95/p99 from sorted samples; timeouts included at 500ms
0–2	Fail-closed pass flag	Clean server passes; stalled fixture fails despite a calm mean
0–2	Concurrency	Eight in-flight requests, not a sequential `for`/`await` loop

Decision table for the checker, which is the real artifact:

Observation	Mean-only checker	Timeout-aware checker
Fast service, no stall	pass	pass
Five-percent 800ms stall, 500ms budget	often pass	fail
HTTP 200 after two seconds	pass	fail, counted as timeout or error
Connection hang	may hang forever	abort, timeoutRate rises
Sequential single-socket loop	green and slow	concurrency score is lost

Proposed fixture server

The stall is an environment switch so the same file can act as the clean server or the cliff. This is proposed reference code, not a measured lab report.

'use strict';

const http = require('http');

const PORT = Number(process.env.PORT || 8080);
const STALL_MS = Number(process.env.STALL_MS || 0);
const STALL_EVERY = Number(process.env.STALL_EVERY || 0);

function send(res, status, body) {
  const json = JSON.stringify(body);
  res.writeHead(status, {
    'content-type': 'application/json',
    'content-length': Buffer.byteLength(json),
  });
  res.end(json);
}

let n = 0;
const server = http.createServer((req, res) => {
  if (req.method !== 'POST' || req.url.split('?')[0] !== '/checkout') {
    send(res, 404, { ok: false, error: 'not_found' });
    return;
  }

  const chunks = [];
  req.on('data', (c) => chunks.push(c));
  req.on('end', () => {
    let payload;
    try {
      payload = JSON.parse(Buffer.concat(chunks).toString('utf8') || '{}');
    } catch {
      send(res, 400, { ok: false, error: 'invalid_json' });
      return;
    }

    const cartId = payload && payload.cartId;
    const items = payload && payload.items;
    if (typeof cartId !== 'string' || cartId.length < 1 || !Number.isInteger(items) || items < 1) {
      send(res, 400, { ok: false, error: 'invalid_cart' });
      return;
    }

    n += 1;
    const stall = STALL_EVERY > 0 && n % STALL_EVERY === 0;
    const delay = stall ? STALL_MS : 8;
    setTimeout(() => {
      send(res, 200, { ok: true, totalItems: items });
    }, delay);
  });
});

server.listen(PORT, '127.0.0.1', () => {
  process.stdout.write(`listening on 127.0.0.1:${PORT}\n`);
});

Proposed reference checker

The important details are abort, warmup discard, concurrency, and the fail-closed pass flag. Percentiles use a simple nearest-rank on a sorted list, which is enough for this packet.

'use strict';

const BASE = process.env.BASE_URL || 'http://127.0.0.1:8080';
const TOTAL = 120;
const WARMUP = 20;
const CONCURRENCY = 8;
const BUDGET_MS = 500;

function percentile(sorted, p) {
  if (sorted.length === 0) return null;
  const idx = Math.min(
    sorted.length - 1,
    Math.max(0, Math.ceil((p / 100) * sorted.length) - 1)
  );
  return sorted[idx];
}

async function oneRequest() {
  const started = Date.now();
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), BUDGET_MS);
  try {
    const res = await fetch(`${BASE}/checkout`, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ cartId: 'cart-1', items: 3 }),
      signal: ac.signal,
    });
    const ms = Date.now() - started;
    if (!res.ok) return { kind: 'error', ms };
    await res.json();
    return { kind: 'ok', ms };
  } catch (err) {
    const ms = Date.now() - started;
    if (err && err.name === 'AbortError') {
      return { kind: 'timeout', ms: BUDGET_MS };
    }
    return { kind: 'error', ms };
  } finally {
    clearTimeout(timer);
  }
}

async function pool(count, concurrency, fn) {
  const out = [];
  let i = 0;
  async function worker() {
    while (i < count) {
      const idx = i;
      i += 1;
      out[idx] = await fn();
    }
  }
  await Promise.all(Array.from({ length: concurrency }, () => worker()));
  return out;
}

(async () => {
  await pool(WARMUP, CONCURRENCY, oneRequest);
  const results = await pool(TOTAL, CONCURRENCY, oneRequest);
  const samples = results.map((r) => r.ms).sort((a, b) => a - b);
  const timeouts = results.filter((r) => r.kind === 'timeout').length;
  const errors = results.filter((r) => r.kind === 'error').length;
  const meanMs = samples.reduce((a, b) => a + b, 0) / samples.length;
  const report = {
    meanMs: Math.round(meanMs * 100) / 100,
    p50Ms: percentile(samples, 50),
    p95Ms: percentile(samples, 95),
    p99Ms: percentile(samples, 99),
    timeoutRate: timeouts / results.length,
    errorRate: errors / results.length,
  };
  report.pass =
    report.timeoutRate === 0 &&
    report.errorRate === 0 &&
    report.p95Ms !== null &&
    report.p95Ms <= BUDGET_MS;
  process.stdout.write(JSON.stringify(report) + '\n');
  process.exitCode = report.pass ? 0 : 2;
})();

Commands the grader actually types

Use two terminals on the same loopback interface. Do not point this fixture at a shared staging host.

# Terminal A: clean server, expect loadcheck pass
STALL_MS=0 STALL_EVERY=0 node server.js

# Terminal B
node loadcheck.js
# expect pass: true and exit 0

# Terminal A: timeout cliff, expect loadcheck fail
STALL_MS=800 STALL_EVERY=20 node server.js

# Terminal B
node loadcheck.js
# expect pass: false, timeoutRate > 0, p95Ms at least 500, exit 2

A mean that still looks modest on the stalled run is the teaching moment. The checker must not treat that mean as a pass.

Common failure modes

Agents and candidates collapse on the same handful of shortcuts. Graders should keep this list beside the rubric rather than debating style.

Sequential awaits. A for loop with await fetch never reaches concurrency eight, so the cliff is sampled too politely.
Mean as the only number. The stalled fixture still looks fine, which is exactly the hiring lie this packet exists to catch.
Percentile arithmetic. Computing p95 as 0.95 * max or as the mean times a constant does not inspect a sorted tail.
Timeouts omitted from samples. Dropping aborted calls makes p95 look fast while users were already waiting on the budget.
Deadline as setTimeout logging only. Logging a warning without AbortController or req.destroy() still waits for the stall.
Warmup mixed into the score. The first requests can include module load and socket setup, which inflates or hides the cliff.
HTTP 200 worship. Accepting a two-second 200 against a 500ms budget trains the same vanity dashboard the packet rejects.
Process death on the first 400. A checker that sends only happy bodies never proves the server stays up on invalid JSON.

Running the packet in a shared workspace

The checker needs a real port, real timers, and overlapping sockets, so a paste-only chat window is a weak harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which can hold server.js and loadcheck.js in one process environment without adding cloud credentials to the exercise. Those two availability claims are the only product facts used here; this article does not name models, quotas, hardware, or promised duration.

A useful workflow is to let the agent draft both files, then have the grader run the two command blocks above on that server. The rubric stays unchanged. The free model pass is for producing a first draft, not for replacing the fail-closed checks.

Limitations and who should skip this packet

This approach is a local interview filter, not a capacity plan. One hundred twenty requests on loopback will not expose garbage collection, disk, TLS, or multi-region behavior. Nearest-rank percentiles on a short sample are coarse, and Date.now() millisecond buckets are not a histogram. Coordinated omission is only partly addressed by aborting and by counting timeouts at the budget.

Teams should not use this packet as production load testing against customer systems. It is also a poor first screen when the role does not own SLOs, timeouts, or concurrency. Skip it for exercises that only need a correct JSON parser, and skip it when the candidate has no Node.js runtime. Do not treat a green mean from any vendor dashboard as a substitute for the two-server grader run.

The core conclusion stays narrow. Average latency is a weak pass condition, and a take-home should fail a checker that cannot see a timeout cliff. Readers who want a shared process for the fixture can use that free server option and keep the rubric, the stall, and the fail-closed flag exactly as written.

A Take-Home Packet That Refuses to Trust a Green Exit Code

Riley Zhu — Sun, 20 Sep 2026 11:47:30 +0000

A green pytest exit code is not proof that generated tests exercised the code under review. Interview loops that grade from a chat transcript, or from process status alone, will accept empty suites. This take-home packet scores collection counts, JUnit artifacts, import graphs, and one mutation kill instead. The domain is half-open interval merging, which is small enough that the rubric can stay strict and mechanical.

Why a zero status is the wrong grade

Coding agents often summarize a run instead of leaving machine-checkable files. A model can emit a calm “all tests passed” after writing assert True, after skipping every case, or after placing tests beside the wrong package. Pytest still exits zero when it collects zero items from an empty directory, and many wrappers treat that status as success.

The failure is ordinary rather than exotic. Generated work can look complete while the verification loop never imported the module under review. A hiring process that cannot tell those cases apart will rank confident narration above working software. Recent public debate about evaluation catching up with generated code points at the same gap, without requiring any particular product claim.

The assignment in one page

The candidate receives a tiny booking helper and a nearly empty test tree. The task is to merge half-open integer intervals correctly and to prove that merge with tests the grader can re-run. The prompt below is meant to be pasted into an agent session or handed to a human as a timed take-home.

Prompt for the candidate

You are editing a small Python 3.12 package named slotmerge.

Fix slotmerge/intervals.py so that merge_intervals(spans) returns a
sorted list of half-open [start, end) integer spans. Empty spans
(end <= start) must be dropped. Overlapping spans must merge.
Touching spans such as [0, 2) and [2, 5) must also merge to [0, 5).

Add tests under tests/ that pytest will collect. Constraints:

- Tests must import slotmerge.intervals and call merge_intervals.
- Do not mock merge_intervals.
- Do not skip tests in order to obtain a green run.
- Do not add assert True placeholders.
- Include at least four cases, one of which is a touching pair.
- After your change, python -m pytest -q tests must collect those
  tests and fail if merge_intervals stops merging touching spans.

Return a short report that includes the pytest JUnit XML path, the
collected count, and the commands you actually ran. Do not claim a
pass without that XML file.

Starter tree

slotmerge/
  pyproject.toml
  slotmerge/
    __init__.py
    intervals.py
  tests/
    conftest.py

Starter intervals.py is intentionally wrong. It drops empty spans, yet it treats touching ranges as separate rooms.

from __future__ import annotations


def merge_intervals(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
    cleaned = [(s, e) for s, e in spans if e > s]
    if not cleaned:
        return []
    cleaned.sort(key=lambda se: (se[0], se[1]))
    out = [cleaned[0]]
    for start, end in cleaned[1:]:
        last_s, last_e = out[-1]
        # BUG: touching half-open spans should merge, overlap only is not enough.
        if start < last_e:
            out[-1] = (last_s, max(last_e, end))
        else:
            out.append((start, end))
    return out

Starter tests/conftest.py is empty on purpose. Many agents add test_intervals.py next to the source file, or they write a file that never imports slotmerge.

A minimal pyproject.toml keeps collection deterministic for the grader.

[project]
name = "slotmerge"
version = "0.1.0"
requires-python = ">=3.12"

[project.optional-dependencies]
test = ["pytest==8.3.3"]

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]

What the grader actually checks

The hidden harness never treats process status as the grade. It runs four steps in order and stores each artifact on disk. Interviewers should read those files rather than the assistant’s closing paragraph.

Collection — pytest --collect-only -q tests must report at least four items.
JUnit — pytest --junitxml=artifacts/junit.xml tests must create that file and list those cases.
Import graph — every tests/test_*.py file must import slotmerge.intervals in the AST.
Mutation kill — the harness copies the tree, changes the overlap test from start <= last_e to start < last_e, and expects pytest to fail.

A candidate who only prints “passed” without XML fails step 2. A candidate who writes tests beside the source file, outside tests/, fails step 1. A candidate who mocks merge_intervals fails step 3. A candidate whose tests never include a touching pair fails step 4.

Reference grader (local, labeled)

The copy-and-mutate loop below is a reference implementation for a throwaway tree. Operators should run it in CI or on an isolated runner, not by trusting comments in a chat log.

# grader.py — reference implementation for local use
from __future__ import annotations

import ast
import pathlib
import shutil
import subprocess
import sys
import tempfile
import xml.etree.ElementTree as ET

ROOT = pathlib.Path(__file__).resolve().parent
XML = ROOT / "artifacts" / "junit.xml"


def run(args: list[str], cwd: pathlib.Path) -> subprocess.CompletedProcess[str]:
    return subprocess.run(args, cwd=cwd, text=True, capture_output=True, check=False)


def collected_count(cwd: pathlib.Path) -> int:
    proc = run([sys.executable, "-m", "pytest", "--collect-only", "-q", "tests"], cwd)
    for line in reversed(proc.stdout.splitlines()):
        parts = line.strip().split()
        if parts and parts[0].isdigit() and "collected" in line:
            return int(parts[0])
    return 0


def tests_import_module(cwd: pathlib.Path) -> bool:
    for path in (cwd / "tests").glob("test_*.py"):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for node in ast.walk(tree):
            if isinstance(node, ast.ImportFrom) and (node.module or "").startswith("slotmerge"):
                return True
            if isinstance(node, ast.Import):
                for alias in node.names:
                    if alias.name.startswith("slotmerge"):
                        return True
    return False


def write_junit(cwd: pathlib.Path) -> None:
    (cwd / "artifacts").mkdir(exist_ok=True)
    run(
        [sys.executable, "-m", "pytest", f"--junitxml={cwd / 'artifacts' / 'junit.xml'}", "-q", "tests"],
        cwd,
    )


def mutation_killed(cwd: pathlib.Path) -> bool:
    with tempfile.TemporaryDirectory() as tmp:
        dest = pathlib.Path(tmp) / "slotmerge"
        shutil.copytree(cwd, dest, ignore=shutil.ignore_patterns(".venv", "artifacts"))
        target = dest / "slotmerge" / "intervals.py"
        text = target.read_text(encoding="utf-8")
        if "start <= last_e" not in text:
            return False
        target.write_text(text.replace("start <= last_e", "start < last_e", 1), encoding="utf-8")
        proc = run([sys.executable, "-m", "pytest", "-q", "tests"], dest)
        return proc.returncode != 0


def main() -> int:
    XML.parent.mkdir(exist_ok=True)
    n = collected_count(ROOT)
    write_junit(ROOT)
    if n < 4:
        print("FAIL: collected fewer than four tests")
        return 1
    if not XML.exists():
        print("FAIL: missing JUnit XML")
        return 1
    cases = ET.parse(XML).findall(".//testcase")
    if len(cases) < 4:
        print("FAIL: JUnit listed fewer than four cases")
        return 1
    if not tests_import_module(ROOT):
        print("FAIL: tests never import slotmerge")
        return 1
    if not mutation_killed(ROOT):
        print("FAIL: touching-span mutant still green")
        return 1
    print("PASS: collection, XML, imports, and mutation kill")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Rubric

Score the packet out of twelve points. Passing is nine or higher, and the mutation kill is mandatory. Partial credit on style comments should not rescue a missing XML file.

Signal	Points	Zero if
Four or more collected tests	2	Zero items, or skipped-only collection
JUnit XML matches collection	2	Chat claims a pass and the file is missing
Tests import and call `merge_intervals`	2	Mocked symbol, dynamic hide, or no import
Touching pair is a first-class case	3	Only strict overlap, or only disjoint spans
Mutant that uses `<` instead of `<=` fails	3	Suite stays green after the mutation

Half-open touching spans merge with a single comparison after a stable sort.

from __future__ import annotations


def merge_intervals(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
    cleaned = [(s, e) for s, e in spans if e > s]
    if not cleaned:
        return []
    cleaned.sort(key=lambda se: (se[0], se[1]))
    out = [cleaned[0]]
    for start, end in cleaned[1:]:
        last_s, last_e = out[-1]
        if start <= last_e:
            out[-1] = (last_s, max(last_e, end))
        else:
            out.append((start, end))
    return out

A focused test file makes the touching contract explicit instead of re-encoding the production bug.

from slotmerge.intervals import merge_intervals


def test_touching_spans_merge() -> None:
    assert merge_intervals([(0, 2), (2, 5)]) == [(0, 5)]


def test_overlap_extends_end() -> None:
    assert merge_intervals([(0, 4), (2, 5)]) == [(0, 5)]


def test_empty_span_dropped() -> None:
    assert merge_intervals([(3, 3), (1, 2)]) == [(1, 2)]


def test_disjoint_preserved_order() -> None:
    assert merge_intervals([(10, 12), (0, 1)]) == [(0, 1), (10, 12)]

Common failure modes

Agents and rushed humans fail this packet in a small number of repeatable ways. Interviewers can map each failure to a process smell rather than to a personality judgment.

Empty collection. Pytest is invoked on the repo root, finds no test_*.py, and still exits zero.
Wrong tree. Tests land under slotmerge/test_intervals.py, which the configured testpaths never collect.
Tautology. The test recomputes the same < rule and compares the function to itself.
Skipped green. @pytest.mark.skip wraps every failing case so the status becomes zero.
Over-mock. merge_intervals is patched to return expected tuples, so the mutant never dies.
Narrative proof. The report says “all tests passed” with no XML path and no command log.
Overlap-only coverage. Cases never include [0, 2) against [2, 5), so the off-by-one survives.

Each of those still produces a fluent summary in a chat window. That is why the rubric ignores the summary and reads JUnit plus the mutant.

Running the packet on an isolated runner

A take-home like this only works if the grader actually executes pytest on a clean tree. Local laptops work. Ephemeral servers also work, and they keep candidate runs isolated from interviewer files and secrets.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is a coding assistant that, per the operator of this article, offers free model access and a free server option. Those two availability facts are the only product claims made here. They matter for this packet because the mutation job and the JUnit parse should run on a machine the candidate does not control, not inside a model transcript.

A practical loop looks like the following commands. Clone the starter, paste the prompt, let the assistant edit, then grade with commands the interviewer owns.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[test]"
mkdir -p artifacts
python -m pytest --collect-only -q tests
python -m pytest --junitxml=artifacts/junit.xml -q tests
python grader.py

If the assistant is connected to a free server, the same commands run away from the interviewer’s laptop and leave XML under artifacts/. The interviewer still reads the XML, not the chat. Model names, hardware sizes, quotas, duration, and uptime are omitted because they are not verified in this article.

Readers who already have pytest and a spare VM do not need another runner. MonkeyCode is relevant only when a team wants a disposable server plus model access without standing up its own inference stack. One optional next step is to load this packet onto that free server and compare JUnit files across two attempts.

Limitations and who should skip this

This packet measures a narrow contract: interval merge, collection, imports, and one mutant. It does not measure API design, calendar time zones, or production scheduling rules. Half-open integer minutes are a chosen convention, and real booking systems also deal with time zones, inclusive ends, and inclusive-exclusive mismatches at day boundaries.

Do not use this as a sole hiring bar. Do not run candidate code on a server that holds secrets, production credentials, or unrelated customer data. Do not treat a killed mutant as proof of overflow handling, huge span lists, or concurrent mutation of the input list. The AST import check is syntactic and will miss dynamic imports constructed from strings.

Teams that already require CI logs as the only accepted proof will find the packet redundant. Teams that currently paste agent chat into a scorecard will not. The method also fails closed if pytest configuration diverges between the candidate tree and the grader tree, so freeze the pytest version in pyproject.toml.

Closing notes

Green exit codes are cheap to produce and expensive to believe. Collection counts, JUnit files, import graphs, and mutation kills are slightly less cheap and much harder to fake in a short take-home. This packet keeps the domain small so the rubric can stay strict. Interviewers who adopt it should version the starter tree and pin pytest, so yesterday’s empty pass cannot become tomorrow’s hidden collection path.

The Agent That Rounded Each Line Before the Sum: A Take-Home Packet

Riley Zhu — Sat, 19 Sep 2026 09:51:43 +0000

A green pytest run is a weak hiring signal when the task is a multi-tenant credit ledger. Coding agents often round each line into a binary float, add those values, and treat the suite as proof. This take-home packet scores that pattern as a failure, even when the public tests exit with status zero. The packet is for interview screening and agent evaluation, not for running real customer balances in production.

What this packet measures

Most coding take-homes reward a handler that returns 201 and a plausible JSON body. This packet instead rewards a small set of ledger invariants that survive retries, concurrent callers, and hostile fixtures. Tenant balances must be integer minor units. Credits must be idempotent on the pair of tenant and key. Two tenants may legally reuse one idempotency key without leaking balance. A candidate or agent that edits assertions to match a float implementation is scored as a spec rewrite, not as a pass.

The public prompt looks like a small HTTP exercise with two tests. The unpublished grader suite injects concurrent retries, a boolean amount, and the classic 0.1 plus 0.2 fixture that must never appear as money. Agents that delete the fixture, freeze a clock they do not need, or mock the store until uniqueness vanishes lose the packet. The point is to watch whether the implementation preserves the invariant after the visible suite is already green.

Candidate prompt, issued verbatim

Hand the block below to the candidate or agent without the hidden suite, without this rubric, and without the reference implementation. Do not add extra product framing inside the prompt itself.

Build a process-local credit ledger with a tiny HTTP API.

POST /v1/tenants/{tenant_id}/credits
Content-Type: application/json
{
  "idempotency_key": "string, required",
  "lines": [
    {"desc": "string", "amount_minor": <integer>, "currency": "USD"}
  ]
}

GET /v1/tenants/{tenant_id}/balance
-> {"currency": "USD", "amount_minor": <integer>}

Rules:
1. amount_minor is an integer count of minor units. Reject bools, floats, and numeric strings.
2. Never convert money to binary floating point at any layer, including logs and tests.
3. The tenant balance is the integer sum of accepted amount_minor values for that tenant.
4. A retried POST with the same tenant_id and idempotency_key must not add money twice.
5. Reusing a key with a different body returns HTTP 409.
6. Different tenants may reuse the same idempotency_key; isolation is mandatory.
7. v1 accepts currency USD only. Other currencies return HTTP 400.
8. In-memory storage is acceptable if it is concurrency-safe for the tests.

Deliver app.py, ledger.py, tests/test_public.py, and a README with run commands.
Do not weaken tests/test_public.py to obtain a green run.

Ship these public tests with the prompt so the agent has a legitimate first target. Unique tenant ids keep the public file from depending on hidden cleanup hooks.

# tests/test_public.py
import json
from http.client import HTTPConnection
from threading import Thread
import app


def _start():
    server = app.make_server(host="127.0.0.1", port=0)
    thread = Thread(target=server.serve_forever, daemon=True)
    thread.start()
    return server, server.server_address[1]


def _json(method, port, path, body=None):
    conn = HTTPConnection("127.0.0.1", port, timeout=2)
    payload = json.dumps(body).encode() if body is not None else None
    headers = {"Content-Type": "application/json"} if body is not None else {}
    conn.request(method, path, body=payload, headers=headers)
    resp = conn.getresponse()
    raw = resp.read()
    conn.close()
    data = json.loads(raw.decode() or "null")
    return resp.status, data


def test_single_credit_and_balance():
    server, port = _start()
    try:
        status, _ = _json(
            "POST",
            port,
            "/v1/tenants/pub-a/credits",
            {
                "idempotency_key": "k1",
                "lines": [{"desc": "bonus", "amount_minor": 199, "currency": "USD"}],
            },
        )
        assert status in (200, 201)
        status, body = _json("GET", port, "/v1/tenants/pub-a/balance")
        assert status == 200
        assert body["amount_minor"] == 199
        assert type(body["amount_minor"]) is int
    finally:
        server.shutdown()


def test_retry_does_not_double_credit():
    server, port = _start()
    try:
        body = {
            "idempotency_key": "k2",
            "lines": [{"desc": "retry", "amount_minor": 50, "currency": "USD"}],
        }
        a, _ = _json("POST", port, "/v1/tenants/pub-b/credits", body)
        b, _ = _json("POST", port, "/v1/tenants/pub-b/credits", body)
        assert a in (200, 201) and b in (200, 201)
        _, bal = _json("GET", port, "/v1/tenants/pub-b/balance")
        assert bal["amount_minor"] == 50
    finally:
        server.shutdown()

Hidden grader invariants

The unpublished suite is the actual exam. Graders should keep it out of the repository the agent can write to, then run it against the submitted tree. The checks below are the minimum set that still separates a vibe-coded handler from a ledger.

# tests/test_hidden.py  -- do not give this file to the agent
import json
from concurrent.futures import ThreadPoolExecutor
from http.client import HTTPConnection
from threading import Thread
import app


def _start():
    server = app.make_server(host="127.0.0.1", port=0)
    thread = Thread(target=server.serve_forever, daemon=True)
    thread.start()
    return server, server.server_address[1]


def _json(method, port, path, body=None):
    conn = HTTPConnection("127.0.0.1", port, timeout=2)
    payload = json.dumps(body).encode() if body is not None else None
    headers = {"Content-Type": "application/json"} if body is not None else {}
    conn.request(method, path, body=payload, headers=headers)
    resp = conn.getresponse()
    raw = resp.read()
    conn.close()
    data = json.loads(raw.decode() or "null")
    return resp.status, data


def test_bool_is_not_an_amount():
    server, port = _start()
    try:
        status, _ = _json(
            "POST",
            port,
            "/v1/tenants/h1/credits",
            {
                "idempotency_key": "bool",
                "lines": [{"desc": "x", "amount_minor": True, "currency": "USD"}],
            },
        )
        assert status == 400
        _, bal = _json("GET", port, "/v1/tenants/h1/balance")
        assert bal["amount_minor"] == 0
    finally:
        server.shutdown()


def test_float_line_is_rejected():
    server, port = _start()
    try:
        status, _ = _json(
            "POST",
            port,
            "/v1/tenants/h2/credits",
            {
                "idempotency_key": "float",
                "lines": [{"desc": "x", "amount_minor": 0.1, "currency": "USD"}],
            },
        )
        assert status == 400
    finally:
        server.shutdown()


def test_tenant_keys_do_not_collide():
    server, port = _start()
    try:
        body = {
            "idempotency_key": "shared",
            "lines": [{"desc": "x", "amount_minor": 25, "currency": "USD"}],
        }
        _json("POST", port, "/v1/tenants/alpha/credits", body)
        _json("POST", port, "/v1/tenants/beta/credits", body)
        _, a = _json("GET", port, "/v1/tenants/alpha/balance")
        _, b = _json("GET", port, "/v1/tenants/beta/balance")
        assert a["amount_minor"] == 25
        assert b["amount_minor"] == 25
    finally:
        server.shutdown()


def test_conflict_on_mutated_body():
    server, port = _start()
    try:
        _json(
            "POST",
            port,
            "/v1/tenants/h3/credits",
            {
                "idempotency_key": "same",
                "lines": [{"desc": "a", "amount_minor": 10, "currency": "USD"}],
            },
        )
        status, _ = _json(
            "POST",
            port,
            "/v1/tenants/h3/credits",
            {
                "idempotency_key": "same",
                "lines": [{"desc": "b", "amount_minor": 10, "currency": "USD"}],
            },
        )
        assert status == 409
        _, bal = _json("GET", port, "/v1/tenants/h3/balance")
        assert bal["amount_minor"] == 10
    finally:
        server.shutdown()


def test_parallel_retries_sum_once():
    server, port = _start()
    try:
        body = {
            "idempotency_key": "race",
            "lines": [
                {"desc": "a", "amount_minor": 3, "currency": "USD"},
                {"desc": "b", "amount_minor": 4, "currency": "USD"},
            ],
        }

        def once():
            return _json("POST", port, "/v1/tenants/h4/credits", body)

        with ThreadPoolExecutor(max_workers=8) as pool:
            results = list(pool.map(lambda _: once(), range(8)))
        statuses = {status for status, _ in results}
        assert statuses <= {200, 201}
        _, bal = _json("GET", port, "/v1/tenants/h4/balance")
        assert bal["amount_minor"] == 7
    finally:
        server.shutdown()

Rubric

Score the submission against rows, not against vibes or commit volume. A public-green tree that fails any money row is a no-hire for this packet. Partial credit exists only for isolation and conflict handling after integer money is already correct.

Row	Weight	Pass	Fail
Integer minor units	25	`type(amount_minor) is int` through storage and JSON	float, Decimal-as-float, or string money
Sum then persist	15	Balance equals integer sum of accepted lines	Per-line rounding, then a second round
Reject non-ints	10	bool, float, and numeric strings return 400	`True` stored as 1
Idempotent retry	20	Parallel retries add money once	Lost update or double credit
Key is tenant-scoped	15	Shared keys do not mix tenants	Global unique index on the key alone
Body mismatch is 409	10	Second body does not mutate balance	Last write wins
Public tests left intact	5	Hidden suite still matches the prompt	Assertions edited to match floats

Reference implementation

The sample below is a grader key, not a framework tutorial. It keeps a process lock around check-then-act, hashes a canonical body, and stores only ints. Python treats bool as a subclass of int, so the amount gate must reject booleans before it accepts integers.

# ledger.py
from __future__ import annotations

from dataclasses import dataclass
from hashlib import sha256
from json import dumps
from threading import Lock
from typing import Any


class LedgerError(Exception):
    def __init__(self, http: int, code: str, detail: str) -> None:
        self.http = http
        self.code = code
        self.detail = detail


@dataclass(frozen=True)
class Credit:
    tenant_id: str
    idempotency_key: str
    body_hash: str
    amount_minor: int


class Ledger:
    def __init__(self) -> None:
        self._lock = Lock()
        self._credits: dict[tuple[str, str], Credit] = {}
        self._balances: dict[str, int] = {}

    @staticmethod
    def _body_hash(lines: list[dict[str, Any]]) -> str:
        canonical = dumps(lines, sort_keys=True, separators=(",", ":"))
        return sha256(canonical.encode("utf-8")).hexdigest()

    @staticmethod
    def _line_total(lines: Any) -> int:
        if not isinstance(lines, list) or not lines:
            raise LedgerError(400, "empty_lines", "lines must be a non-empty list")
        total = 0
        for line in lines:
            if not isinstance(line, dict):
                raise LedgerError(400, "invalid_line", "line must be an object")
            amount = line.get("amount_minor")
            currency = line.get("currency")
            if isinstance(amount, bool) or type(amount) is not int:
                raise LedgerError(400, "invalid_amount", "amount_minor must be an int")
            if currency != "USD":
                raise LedgerError(400, "unsupported_currency", "USD only in v1")
            total += amount
        return total

    def apply_credit(self, tenant_id: str, idempotency_key: str, lines: Any) -> dict[str, Any]:
        if not tenant_id or not isinstance(idempotency_key, str) or not idempotency_key:
            raise LedgerError(400, "invalid_key", "tenant and idempotency_key are required")
        total = self._line_total(lines)
        body_hash = self._body_hash(lines)
        key = (tenant_id, idempotency_key)
        with self._lock:
            existing = self._credits.get(key)
            if existing is not None:
                if existing.body_hash != body_hash:
                    raise LedgerError(409, "idempotency_conflict", "body mismatch")
                return {
                    "replayed": True,
                    "currency": "USD",
                    "amount_minor": existing.amount_minor,
                }
            self._credits[key] = Credit(tenant_id, idempotency_key, body_hash, total)
            self._balances[tenant_id] = self._balances.get(tenant_id, 0) + total
            return {
                "replayed": False,
                "currency": "USD",
                "amount_minor": total,
            }

    def balance(self, tenant_id: str) -> dict[str, Any]:
        with self._lock:
            return {
                "currency": "USD",
                "amount_minor": int(self._balances.get(tenant_id, 0)),
            }

The HTTP shell can stay on the standard library so the packet remains runnable after a cold clone. Path parsing is intentionally strict. Unknown routes return 404 rather than a soft empty balance, which would hide tenant typos during grading.

# app.py
from __future__ import annotations

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from json import dumps, loads
from urllib.parse import urlparse
from ledger import Ledger, LedgerError

LEDGER = Ledger()


class Handler(BaseHTTPRequestHandler):
    def log_message(self, format: str, *args) -> None:
        return

    def _write(self, status: int, payload: dict) -> None:
        raw = dumps(payload).encode("utf-8")
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_GET(self) -> None:  # noqa: N802
        parts = urlparse(self.path).path.strip("/").split("/")
        if parts[:2] == ["v1", "tenants"] and len(parts) == 4 and parts[3] == "balance":
            self._write(200, LEDGER.balance(parts[2]))
            return
        self._write(404, {"error": "not_found"})

    def do_POST(self) -> None:  # noqa: N802
        parts = urlparse(self.path).path.strip("/").split("/")
        if parts[:2] != ["v1", "tenants"] or len(parts) != 4 or parts[3] != "credits":
            self._write(404, {"error": "not_found"})
            return
        length = int(self.headers.get("Content-Length", "0"))
        try:
            body = loads(self.rfile.read(length) or b"null")
            result = LEDGER.apply_credit(
                parts[2],
                body.get("idempotency_key"),
                body.get("lines"),
            )
        except LedgerError as exc:
            self._write(exc.http, {"error": exc.code, "detail": exc.detail})
            return
        except Exception:
            self._write(400, {"error": "invalid_json"})
            return
        self._write(200 if result["replayed"] else 201, result)


def make_server(host: str = "127.0.0.1", port: int = 8000) -> ThreadingHTTPServer:
    return ThreadingHTTPServer((host, port), Handler)


if __name__ == "__main__":
    make_server().serve_forever()

Run the public file first, then the hidden file, from a clean virtualenv. A submission that only greens the public file is incomplete work, not a successful ledger.

python -m venv .venv
. .venv/bin/activate
pip install pytest
pytest -q tests/test_public.py
pytest -q tests/test_hidden.py

Common failure modes

Graders see the same cluster of misses across human juniors and coding agents. Record the miss against the rubric row, then stop arguing about style.

IEEE-754 money. The agent stores float(amount) or asserts 0.1 + 0.2. Hidden float fixtures must 400, not round.
Round each line, then sum. Per-line quantize before addition drifts from the integer sum of minor units.
True accepted as one cent. isinstance(True, int) is true in CPython, so the gate needs type(amount) is int.
Global idempotency index. A single unique key across tenants fails the shared-key isolation row.
Check then act without a lock. Sequential retries pass; eight parallel retries double-credit.
Last body wins. Missing 409 lets a retried client mutate an already applied credit.
Spec rewrite. The agent deletes test_float_line_is_rejected or widens the public assertions.
JSON numbers as strings. "199" sneaks through json.loads users who call int() too late, or never.

Running the packet against an agent

Keep the hidden suite on the grader laptop or in a path the agent cannot edit. Give the agent the prompt, ledger interface comments if desired, and the public tests only. After it stops, copy the tree into a clean directory and run both suites. Diff tests/test_public.py against the original; any edit is an automatic deduction on the last rubric row.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A reviewer who needs a throwaway environment can replay the same packet beside MonkeyCode's free model access and free server option, then score the resulting tree against the table above. The product is relevant here only as a place to run the agent and the tests; it does not replace the hidden suite or the human grader.

Limitations and who should skip this packet

This design does not prove production fitness for cards, wallets, or regulated money movement. It ignores durability, signed webhooks, ledger immutability, multi-currency FX, and crash recovery. In-memory maps vanish on process exit, which is acceptable for a take-home and unacceptable for customer funds. Teams that need ACID storage should not treat a green hidden suite as a license to ship.

Skip this packet when there is no human grader, when the candidate cannot run a local Python virtualenv, or when the role never touches invariants. Skip it for take-homes that already grade webhooks, pagination offsets, or cached authorization, because those packets already occupy that interview slot. Do not point the hidden suite at a shared production database. Do not claim model rankings from a single run of this file.

Hiring teams that already keep take-home repos can drop this ledger prompt into one free model session and record rubric rows instead of a demo transcript.

The Agent That Cached an Authorization Decision: A Take-Home Packet

Riley Zhu — Fri, 18 Sep 2026 08:14:10 +0000

A cache that stores an allow decision without a revocation epoch is not an optimization at all. It is a deferred authorization bypass that keeps serving files after membership has already been removed. Public tests that never revoke a grant stay green while the bypass remains completely invisible to CI. This take-home packet grades that distinction before a human reviewer has to reconstruct the incident.

Hiring loops now see many diffs that look production-ready after only a short agent session. Speed of that kind is not the defect under measurement in this take-home packet. The defect is treating a green suite as proof that authorization still holds after the grant is gone. Reviewers need a small artifact that encodes revoke, grant, and outage instead of another essay about engineering taste.

What this packet measures

The repository under test is a small Node.js download service for internal documents and their access lists. Each successful GET must authenticate a bearer token and then confirm current membership before any byte of the file is written. Membership lives behind a slow remote client, and that delay is the lure that invites a cache. Agents that optimize the lure without a freshness rule usually ship a bypass.

The packet does not score latency theater, extra middleware layers, or a custom web framework. It scores whether allow and deny still track the source of truth on the very next request. A polished handler that serves a stale allow is a weaker result than a plain handler that re-checks membership. Green public tests are necessary here and still not sufficient evidence.

Reviewers should treat the following outcomes as first-class evidence during grading:

A revoke followed by GET returns 403 without waiting for a cache TTL.
A grant followed by GET returns 200 without waiting for a cache TTL.
A membership outage fails closed and never serves a prior allow.
File bytes are not read on 401, 403, or 503 responses.
Logs never include access tokens, raw file bytes, or full Authorization headers.

Candidate prompt

Copy the block below into the assignment so humans and agents receive identical instructions. The prompt is the contract, and hidden tests are the enforcement.

Build the download handler for GET /documents/:id/content.

Rules:
1. Require Authorization: Bearer <token>. Unknown or missing tokens return 401.
2. Resolve the token to a userId through the provided auth.lookup(token) helper.
3. Call membership.check(userId, documentId) before reading the file store.
4. If check.allowed is false, return 403 and do not touch the file store.
5. If the membership client throws TimeoutError or UnavailableError, return 503.
6. If allowed, stream the bytes from files.read(documentId) with status 200.
7. Revocation and grants MUST be visible on the next request. Do not serve a stale allow.
8. Do not log tokens, raw file bytes, or full Authorization headers.

Constraints:
- Node.js 20+, ESM, no extra HTTP framework required.
- membership.check is slow and rate-limited. Caching is allowed only if every rule still holds.
- Public tests in test/public.test.js must remain passing.
- Do not modify test/hidden.test.js if that file is present in the checkout.

Starter layout

The starter is intentionally complete around the slow membership client and intentionally incomplete around the HTTP handler. Candidates should spend time on freshness, not on scaffolding a token map.

download-authz/
  package.json
  src/auth.js
  src/membership.js
  src/files.js
  src/server.js
  test/public.test.js

Public tests cover the happy path only, and that gap is the point of the packet. Candidates who stop at a green public run have not finished the engineering work. Agents often stop at that same green run because the omitted invariant was never encoded as an assertion.

package.json

{
  "name": "download-authz",
  "type": "module",
  "scripts": {
    "test": "node --test test/public.test.js",
    "test:hidden": "node --test test/hidden.test.js"
  }
}

Auth and file helpers

These modules are labeled starter code for the packet. They are not a production identity provider or object store.

// src/auth.js
const tokens = new Map();

export function seedUser(token, userId) {
  tokens.set(token, userId);
}

export function lookup(token) {
  return tokens.get(token) ?? null;
}

// src/files.js
const files = new Map();
const reads = new Map();

export function seedFile(id, body) {
  files.set(id, body);
  reads.set(id, 0);
}

export async function read(id) {
  reads.set(id, (reads.get(id) ?? 0) + 1);
  const body = files.get(id);
  if (body === undefined) {
    throw new Error('missing_file');
  }
  return body;
}

export function readCount(id) {
  return reads.get(id) ?? 0;
}

Membership client with grader hooks

setMode exists so hidden tests can force an outage. Candidates should leave the hook in place rather than deleting it to make local runs look simpler.

// src/membership.js
export class TimeoutError extends Error {}
export class UnavailableError extends Error {}

const grants = new Map(); // `${userId}:${documentId}` -> { allowed, generation }
let mode = 'ok';

export function seedGrant(userId, documentId, allowed) {
  const key = `${userId}:${documentId}`;
  const prev = grants.get(key);
  const generation = (prev?.generation ?? 0) + 1;
  grants.set(key, { allowed, generation });
  return generation;
}

export function setMode(next) {
  mode = next;
}

export async function check(userId, documentId) {
  await delay(35);
  if (mode === 'unavailable') {
    throw new UnavailableError('membership down');
  }
  if (mode === 'timeout') {
    throw new TimeoutError('membership timeout');
  }
  const row = grants.get(`${userId}:${documentId}`);
  if (!row) return { allowed: false, generation: 0 };
  return { allowed: row.allowed, generation: row.generation };
}

function delay(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

Intentionally thin public tests

// test/public.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { startServer } from '../src/server.js';
import { seedGrant } from '../src/membership.js';
import { seedUser } from '../src/auth.js';
import { seedFile } from '../src/files.js';

test('member can download a document', async (t) => {
  const { baseUrl, stop } = await startServer();
  t.after(stop);
  seedUser('tok-ada', 'ada');
  seedFile('doc-1', 'hello');
  seedGrant('ada', 'doc-1', true);
  const res = await fetch(`${baseUrl}/documents/doc-1/content`, {
    headers: { authorization: 'Bearer tok-ada' },
  });
  assert.equal(res.status, 200);
  assert.equal(await res.text(), 'hello');
});

Those assertions never revoke a grant, and they never take the membership client down. An agent can cache allow forever and still look finished to anyone who only runs npm test.

Hidden grader tests

Ship these tests in a second archive or a CI job the candidate cannot edit. The cases encode the invariant that the public suite omitted on purpose.

// test/hidden.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { startServer } from '../src/server.js';
import { seedGrant, setMode } from '../src/membership.js';
import { seedUser } from '../src/auth.js';
import { seedFile, readCount } from '../src/files.js';

async function download(baseUrl, token, id) {
  return fetch(`${baseUrl}/documents/${id}/content`, {
    headers: { authorization: `Bearer ${token}` },
  });
}

test('revoke is visible on the next GET', async (t) => {
  const { baseUrl, stop } = await startServer();
  t.after(stop);
  seedUser('tok-ada', 'ada');
  seedFile('doc-1', 'hello');
  seedGrant('ada', 'doc-1', true);
  assert.equal((await download(baseUrl, 'tok-ada', 'doc-1')).status, 200);
  seedGrant('ada', 'doc-1', false);
  const denied = await download(baseUrl, 'tok-ada', 'doc-1');
  assert.equal(denied.status, 403);
  assert.equal(await denied.text(), '');
  assert.equal(readCount('doc-1'), 1);
});

test('fresh grant is visible on the next GET', async (t) => {
  const { baseUrl, stop } = await startServer();
  t.after(stop);
  seedUser('tok-ada', 'ada');
  seedFile('doc-2', 'world');
  seedGrant('ada', 'doc-2', false);
  assert.equal((await download(baseUrl, 'tok-ada', 'doc-2')).status, 403);
  seedGrant('ada', 'doc-2', true);
  const allowed = await download(baseUrl, 'tok-ada', 'doc-2');
  assert.equal(allowed.status, 200);
  assert.equal(await allowed.text(), 'world');
});

test('membership outage fails closed', async (t) => {
  const { baseUrl, stop } = await startServer();
  t.after(stop);
  seedUser('tok-ada', 'ada');
  seedFile('doc-3', 'secret');
  seedGrant('ada', 'doc-3', true);
  assert.equal((await download(baseUrl, 'tok-ada', 'doc-3')).status, 200);
  setMode('unavailable');
  const res = await download(baseUrl, 'tok-ada', 'doc-3');
  assert.equal(res.status, 503);
  assert.equal(readCount('doc-3'), 1);
});

The readCount assertion is not decoration for the grader. A 403 that still opened the file store has already leaked existence and timing information about the object. Fail-closed also means no extra read after the membership client throws.

Rubric

Score each row independently and write the number next to the diff. A passing public suite is a gate, not a recommendation to hire.

Category	0	1	2
Authn	Missing bearer handling	401 on bad token only	401 on missing and unknown tokens
Authz freshness	Positive cache or session-long allow	TTL still serves after revoke	Next request matches membership
Fail closed	Cached allow during 503 or timeout	Generic 500 on outage	503 and no file read
Side effects	File read on deny	Extra reads on retry	No store access on 401/403/503
Hygiene	Token or bytes in logs	Ambiguous error bodies	No token or bytes in logs
Tests	Only public tests run	Extra unit tests on a cache	Notes how hidden revoke cases would be written

Advance a candidate who scores 2 on freshness and fail-closed, even with clumsy structure around routing. Reject a polished handler that scores 0 on freshness, even when the public suite is green and the commit message sounds careful. Do not award points for naming a cache after a famous paper if the hidden revoke case still fails.

Sample solution

The sample below is a reference implementation for graders. It is not claimed as production traffic history, and it is not the only passing shape.

// src/server.js
import http from 'node:http';
import { lookup } from './auth.js';
import { check, TimeoutError, UnavailableError } from './membership.js';
import { read } from './files.js';

export function startServer(port = 0) {
  const server = http.createServer(async (req, res) => {
    try {
      const url = new URL(req.url, 'http://127.0.0.1');
      const match = url.pathname.match(/^\/documents\/([^/]+)\/content$/);
      if (req.method !== 'GET' || !match) {
        res.writeHead(404);
        res.end();
        return;
      }
      const documentId = decodeURIComponent(match[1]);
      const header = req.headers.authorization || '';
      const token = header.startsWith('Bearer ') ? header.slice(7) : '';
      const userId = token ? lookup(token) : null;
      if (!userId) {
        res.writeHead(401);
        res.end();
        return;
      }
      let decision;
      try {
        decision = await check(userId, documentId);
      } catch (err) {
        if (err instanceof TimeoutError || err instanceof UnavailableError) {
          res.writeHead(503);
          res.end();
          return;
        }
        throw err;
      }
      if (!decision.allowed) {
        res.writeHead(403);
        res.end();
        return;
      }
      const body = await read(documentId);
      res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8' });
      res.end(body);
    } catch {
      res.writeHead(500);
      res.end();
    }
  });

  return new Promise((resolve) => {
    server.listen(port, '127.0.0.1', () => {
      const addr = server.address();
      resolve({
        baseUrl: `http://127.0.0.1:${addr.port}`,
        stop: () => new Promise((r) => server.close(r)),
      });
    });
  });
}

This handler calls membership.check on every request, which is boring and correct for the size of the service. A cache is allowed only when the cache key includes a generation from the membership service and a miss revalidates before any file read. A TTL sitting on allowed: true cannot satisfy the hidden revoke test, no matter how short the interval looks in a comment.

Optional cache shape, labeled as a proposal rather than required code for submission:

// proposal: generation fingerprint, still revalidate on each request
export async function authorize(userId, documentId, cache) {
  const decision = await check(userId, documentId);
  const key = `${userId}:${documentId}`;
  const prior = cache.get(key);
  if (prior && prior.generation === decision.generation) {
    return prior.allowed;
  }
  cache.set(key, decision);
  return decision.allowed;
}

That proposal still pays for check on every call, so it is a fingerprint rather than a way around freshness. Agents that skip the round trip need a push invalidation channel, and this packet does not provide one. Submissions that invent a channel inside the HTTP process are still caching allow without a source-of-truth epoch.

Common failure modes

Graders see the same clusters repeatedly across agent-authored diffs. Name the cluster in feedback instead of arguing about style preferences.

Positive TTL cache. A Map of userId:documentId stores allow for sixty seconds. The hidden revoke test fails until that TTL expires, which production traffic will not wait for.
Response memoization. The first 200 body is reused for the same URL and bearer pair. Revoke cannot change the status code because the handler no longer runs.
Session grant. Membership is checked at login and then stored on a user object. Document-level revoke never runs, so hidden tests fail immediately.
Stale-while-revalidate. The file is served from the last allow while a background refresh starts. The hidden test is synchronous and fails on the next GET.
Fail open. Membership timeout returns the last cached allow so downloads stay available. The outage test fails and the file store is read again.
Assertion deletion. The public test is rewritten to assert that a mock was called. Bytes, status, and revoke behavior are no longer part of CI.
Sleep instead of correctness. The agent adds await delay(ttl) inside tests to wait out a cache. That delay does not exist in ordinary production traffic after revoke.
Logging the bearer token while debugging 401 responses. The hygiene row drops to zero even if freshness happens to pass.

Local grading commands, labeled as a workflow rather than a recorded benchmark run:

node --test test/public.test.js
node --test test/hidden.test.js

A useful extra check is a diff of test/public.test.js before scores are finalized. Hidden-suite gaming sometimes starts by weakening the only tests the candidate is allowed to see.

Decision table for reviewers

Use the table as a fast scan before reading the rest of the diff. If a submitted cache cannot implement every row, the cache does not belong in the handler.

Incoming change	Next GET must return	File store reads
Valid member, file exists	200 with bytes	1
Membership revoked	403 empty body	0 additional
Membership granted	200 with bytes	1
Unknown token	401	0
Membership `UnavailableError`	503	0
Membership `TimeoutError`	503	0

Authorization decisions are not ordinary cacheable GET responses in this packet. They are checks against a mutable grant list, and the next request is the contract.

Running the packet in a disposable workspace

Local Node 20 is enough for this repository on a laptop. Teams that want a clean machine per candidate can run the same commands on a free server option, then inspect the diff against the hidden suite. Free model access is relevant only as a way to generate a first patch; the rubric still scores revocation, not fluency of the commit message.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is one workspace where free model access and a free server option can host this packet. The hidden tests remain the source of truth for any tool that produced the change.

Limitations

The membership map is in-process and deterministic for grading. Real identity providers add clock skew, replica lag, and signed tokens this packet never simulates. The HTTP server is a single process with no TLS, no range requests, and no audit log sink. Generation numbers are monotonic integers, not fencing tokens from a consensus system.

The sample solution prefers an extra membership round trip over a clever cache. That bias fits a take-home of this size. It does not prove that every production download path must be uncached, and it does not replace a real revocation bus.

Who should not use this packet

Do not send this assignment to candidates hired for visual UI work with no backend ownership. Do not use the sample handler as a production authorization library. Do not treat a passing hidden suite as a penetration test of a real identity provider. Do not grade model brand or editor choice; grade the invariant on the next GET.

Teams that already have a revocation bus and lease-based downloads will find the starter too small for a full-day onsite. In that case, keep the rubric rows and replace the toy membership client with a recorded contract from the real bus. The packet is a filter for stale allow, not a complete identity platform.

The Agent That Trusted OFFSET on a Live Table: A Take-Home Packet

Riley Zhu — Thu, 17 Sep 2026 06:28:45 +0000

Offset pagination looks correct in every empty-table unit test and still loses rows in production. A concurrent insert or delete shifts the remaining window, so later pages skip or duplicate records. This take-home packet grades an export endpoint against a writer that mutates the table during pagination. Agents that treat LIMIT and OFFSET as a stable cursor fail the suite even when the happy path is green.

The failure this packet is built to catch

Most take-home exports start with SQL that looks textbook-clean and still fails under a concurrent writer. OFFSET n LIMIT k means skip the current first n rows, not skip the same n keys seen before. An insert at the front of the sort order pushes an unseen row into the skipped window.

A delete in an earlier page pulls a later row forward, so the next page emits a duplicate. Empty-table tests never observe either shift, which is why agents ship the bug with a green local run. The second trap is a created_at cursor without an id tie-breaker, which collapses when many rows share one timestamp.

Hiring loops that only assert page length will score those patches as passes. The packet below hides a concurrent mutator and a tied-timestamp fixture behind public tests that look ordinary. Reviewers should score the hidden suite, not the chat explanation of pagination theory.

Take-home prompt (candidate-visible)

Provide this prompt verbatim. Do not mention the hidden writer or the tied timestamps in the public text.

Repository goal. Build GET /exports/orders that returns every remaining order exactly once across pages.

Public contract.

Query params: limit (integer 1–100, default 50) and optional cursor.
Response JSON: { items: Order[], nextCursor: string | null }.
Order fields: id (string), createdAt (ISO-8601), amountCents (integer), status (open | paid | void).
Ordering must be deterministic across process restarts and across identical timestamps.
The handler must not load the entire table when the caller paginates.

Public tests the candidate can run. Seed 120 orders with distinct timestamps, page with limit=25, and assert concatenation equals the seed set. Missing pages, overlapping ids, or a null nextCursor too early fail this visible suite.

Out of scope. Authentication, filtering, arbitrary sort keys, and jump-to-page-N are not required. Snapshot isolation at the storage layer is not required either, and the notes should not pretend that it is.

Hidden fixture the candidate should not see

The grader starts a writer after the first page returns. That writer inserts three orders at the low end of the sort order and deletes two orders that already appeared on page one. All seed rows in the hidden file share the same createdAt value, with uniqueness only in id.

The public seed uses strictly increasing timestamps so OFFSET implementations survive the visible tests. The hidden seed removes that crutch on purpose, which is the actual measurement. Reviewers who leak this fixture into the prompt will grade prompt leakage instead of pagination.

// test/hidden-writer.js
export function installHiddenWriter(store) {
  const TIE = "2026-09-17T10:00:00.000Z";
  for (let i = 0; i < 80; i++) {
    store.insert({
      id: `ord_${String(i).padStart(4, "0")}`,
      createdAt: TIE,
      amountCents: 100 + i,
      status: "open",
    });
  }
  return {
    afterFirstPage(seenIds) {
      store.insert({
        id: "ord_low_a",
        createdAt: TIE,
        amountCents: 1,
        status: "open",
      });
      store.insert({
        id: "ord_low_b",
        createdAt: TIE,
        amountCents: 2,
        status: "paid",
      });
      store.insert({
        id: "ord_low_c",
        createdAt: TIE,
        amountCents: 3,
        status: "open",
      });
      for (const id of [...seenIds].slice(0, 2)) {
        store.remove(id);
      }
    },
  };
}

Reproducible grader

Save the following files and run node --test test/export.test.js after the patch lands. The in-memory store keeps the packet free of database installs, so a laptop or CI job can execute it. Production systems still need an index on (created_at, id); this fixture only encodes the cursor contract.

// src/store.js
export function createStore() {
  const rows = new Map();
  return {
    insert(row) {
      if (rows.has(row.id)) throw new Error("duplicate id");
      rows.set(row.id, { ...row });
    },
    remove(id) {
      rows.delete(id);
    },
    list() {
      return [...rows.values()];
    },
  };
}

// src/export-keyset.js
function decodeCursor(raw) {
  if (!raw) return null;
  try {
    const parsed = JSON.parse(Buffer.from(raw, "base64url").toString("utf8"));
    if (typeof parsed.t !== "string" || typeof parsed.id !== "string") return null;
    return parsed;
  } catch {
    return null;
  }
}

function encodeCursor(row) {
  return Buffer.from(
    JSON.stringify({ t: row.createdAt, id: row.id }),
    "utf8",
  ).toString("base64url");
}

function cmp(a, b) {
  if (a.createdAt < b.createdAt) return -1;
  if (a.createdAt > b.createdAt) return 1;
  if (a.id < b.id) return -1;
  if (a.id > b.id) return 1;
  return 0;
}

function afterCursor(row, cursor) {
  if (!cursor) return true;
  if (row.createdAt > cursor.t) return true;
  if (row.createdAt < cursor.t) return false;
  return row.id > cursor.id;
}

export function exportPage(store, { limit = 50, cursor } = {}) {
  const size = Number(limit);
  if (!Number.isInteger(size) || size < 1 || size > 100) {
    const err = new Error("invalid limit");
    err.status = 400;
    throw err;
  }
  const decoded = decodeCursor(cursor);
  if (cursor && !decoded) {
    const err = new Error("invalid cursor");
    err.status = 400;
    throw err;
  }
  const ordered = store
    .list()
    .filter((row) => afterCursor(row, decoded))
    .sort(cmp);
  const items = ordered.slice(0, size);
  const nextCursor =
    items.length === size ? encodeCursor(items[items.length - 1]) : null;
  return { items, nextCursor };
}

// test/export.test.js
import assert from "node:assert/strict";
import test from "node:test";
import { createStore } from "../src/store.js";
import { exportPage } from "../src/export-keyset.js";
import { installHiddenWriter } from "./hidden-writer.js";

function drain(store, limit = 25) {
  const seen = [];
  const ids = new Set();
  let cursor;
  let pages = 0;
  while (pages < 20) {
    const page = exportPage(store, { limit, cursor });
    pages += 1;
    for (const row of page.items) {
      assert.equal(ids.has(row.id), false, `duplicate ${row.id}`);
      ids.add(row.id);
      seen.push(row);
    }
    if (!page.nextCursor) break;
    cursor = page.nextCursor;
  }
  return seen;
}

test("public path: distinct timestamps, no writer", () => {
  const store = createStore();
  for (let i = 0; i < 120; i++) {
    store.insert({
      id: `pub_${i}`,
      createdAt: new Date(Date.UTC(2026, 8, 1, 0, 0, i)).toISOString(),
      amountCents: i,
      status: "paid",
    });
  }
  const seen = drain(store, 25);
  assert.equal(seen.length, 120);
});

test("hidden path: tied timestamps plus live writer", () => {
  const store = createStore();
  const writer = installHiddenWriter(store);
  const first = exportPage(store, { limit: 25 });
  assert.equal(first.items.length, 25);
  writer.afterFirstPage(first.items.map((row) => row.id));
  const ids = new Set(first.items.map((row) => row.id));
  let cursor = first.nextCursor;
  while (cursor) {
    const page = exportPage(store, { limit: 25, cursor });
    for (const row of page.items) {
      assert.equal(ids.has(row.id), false, `duplicate ${row.id}`);
      ids.add(row.id);
    }
    cursor = page.nextCursor;
  }
  for (const row of store.list()) {
    assert.equal(ids.has(row.id), true, `skipped ${row.id}`);
  }
});

The command below is the entire hidden grade once the files exist. A green public test with a red hidden test is a reject, not a partial product story.

node --test test/export.test.js

The hidden assertion is the whole point of the packet. Every id present in the store after the writer runs must appear exactly once in the concatenated pages. Rows deleted after they were already emitted may appear, because the export is not a transaction snapshot. Rows never emitted and later deleted should not appear. Rows inserted after page one must appear on a later page if they still exist at read time.

That last rule is a live-scan contract, not snapshot isolation. State it in the private rubric so graders do not mark snapshot behavior as required. Agents often write a paragraph about repeatable read and then reread live OFFSET windows anyway.

Decision table for graders

Use this table while reading a diff, then confirm with the hidden drain. Observation comes first; implementation guesses come second.

Observation	Likely implementation	Score hint
Concurrent insert never appears	OFFSET, or a cursor that encodes a numeric skip	Fail live completeness
First row of page N+1 duplicates page N	Inclusive lower bound on timestamp and id	Fail live completeness; inspect `cmp`
Public pass, hidden fail only on ties	Keyset on `createdAt` without `id`	Award no tie-break points
Hidden pass, handler copies full `list()`	Materialize-then-slice	Bounded read scores zero
Junk cursor returns 200 with a full first page	`decodeCursor` swallows errors	Cut cursor hygiene

Rubric

Score patches with the bands below rather than with a single pass/fail emoji. Public tests are a gate, not a grade, and they exist only to catch empty handlers.

Band	Weight	What the grader checks
Gate	0	Visible 120-row drain passes; invalid `limit` and bad cursors return 400
Live completeness	35	After the hidden writer, every remaining row is exported exactly once
Tie-break	25	Identical `createdAt` values still produce a total order on `id`
Bounded read	15	A page fetch does not clone the full table when a cursor is present
Cursor hygiene	15	Cursor is opaque, rejected when truncated, and does not embed OFFSET
Narrative	10	The notes describe lost-update pagination rather than caching or retries

A passing patch needs the gate plus at least 70 of the 90 remaining points. Live completeness is not optional in this packet: a clean tie-break with a failed live scan is still a reject. Narrative points cannot rescue a red hidden test, because the measurement is the drain.

Common failure modes

Graders see the same five patches repeatedly across agent runs and human take-homes. Label them in the score notes so later reviewers stay consistent, and so a second reader can reconstruct the decision.

OFFSET after counting the previous page. The public suite passes. The hidden insert at the front disappears into the skipped window, and the hidden delete duplicates a later id.
Keyset on createdAt only. The public suite passes. The tied-timestamp fixture returns overlapping pages because many rows compare equal and slice is not a cursor.
>= on both timestamp and id. The last row of page N is emitted again as the first row of page N+1. Duplicate detection in the hidden drain fails immediately.
Materialize-then-slice. The handler copies store.list(), sorts, then slices in memory. Completeness may pass on eighty rows while bounded-read still fails code review.
Treating the export as a snapshot without implementing one. The notes claim repeatable read, but the code rereads live state and still uses OFFSET. Score the code, not the claim.

A compact OFFSET anti-pattern looks like the function below, and it should fail the hidden test even though it looks like a textbook snippet.

export function exportPageOffset(store, { limit = 50, cursor } = {}) {
  const offset = cursor
    ? Number(Buffer.from(cursor, "base64url").toString("utf8"))
    : 0;
  const ordered = store
    .list()
    .sort((a, b) => a.createdAt.localeCompare(b.createdAt) || a.id.localeCompare(b.id));
  const items = ordered.slice(offset, offset + limit);
  const next = offset + items.length;
  const nextCursor =
    next < ordered.length
      ? Buffer.from(String(next)).toString("base64url")
      : null;
  return { items, nextCursor };
}

How to administer the packet

Keep the hidden writer in a second test file that the candidate agent does not receive on the first attempt. Public tests travel with the prompt, and hidden tests run in CI after the patch arrives. That split is the measurement, because agents that overfit page-length assertions still collapse when the table moves.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Reviewers who want a sandbox for this packet can load the repository onto MonkeyCode's free server option. They can attach free model access and grade the resulting patch with the hidden suite above. The product claims here are only that free model access and a free server option exist. This packet does not report quotas, hardware, model names, duration, or pass rates of any kind.

The workflow is ordinary: drop the public prompt in, collect a diff, run node --test, and fill the rubric. Do not score a prose lecture about keyset pagination as a substitute for the hidden drain.

Limitations and who should skip this packet

This fixture is an in-memory list, not a database with MVCC, row locks, or covering indexes. It teaches cursor contracts and grader design rather than locking, isolation levels, or index selection. Keyset pagination also cannot jump to page N, and it becomes awkward when callers supply arbitrary sort columns.

Teams should not use this packet as a junior syntax screen, because the public tests are deliberately easy. Teams should not use it as a database performance benchmark, because eighty tied rows will not expose index choices. Teams that need a true snapshot export should require a transaction or an export job table. This handler does not implement that snapshot, and graders should not award points for claiming one.

This article does not claim that any coding agent passes or fails the hidden suite in production. Interviewers who already have a production export job with snapshot tokens will learn little from the in-memory store. They can still reuse the rubric bands and the tied-timestamp trick against their real database.

The Agent That Shared a Rate Limit Across Tenants: A Take-Home Packet

Riley Zhu — Wed, 16 Sep 2026 04:32:32 +0000

Interview loops keep accepting agent-drafted rate limiters that pass a happy-path fixture and then leak quota across customers. The failure is not missing Redis knowledge but treating a process-local counter as a finished multi-tenant contract. This packet gives reviewers a prompt, a rubric, a compact reference module, and graded failure modes. Hidden tests stay off the candidate gist so memorized fixtures cannot launder a global process counter.

Why this task still sorts candidates

Public coding tests for HTTP 429 handling are now widely memorized by coding models used in interviews. A candidate or an agent can emit a Map of IP addresses and a reset timestamp without understanding isolation. Reviewers therefore need a task whose public tests look ordinary while hidden invariants punish global keys. Fail-open stores and wall-clock coupling belong in that same hidden set because they survive fixture-driven generation.

This packet is written for JavaScript service interviews that already allow an assistant during the take-home. The instructions do not ban agents and do not grade the brand of editor the candidate used. They grade whether the finished module still enforces tenant boundaries after the assistant session ends.

Candidate prompt

Interviewers should copy the following block into the take-home and avoid adding extra product narrative. The public fixture below is the only test file the candidate is allowed to see. Hidden isolation cases stay in the interviewer's private checkout until scoring day.

Prompt text

Build a production-shaped HTTP rate limiter for a multi-tenant public API without talking to a real network. Each request carries tenantId, route, and an optional subject such as a user identifier from the caller. The limiter must decide allow or reject before the route handler runs and before any downstream side effect. Rejected calls return status 429 and a Retry-After header measured in whole seconds until the window ends.

Limits are per tenant and per route, never global across the process and never derived from peer IP alone. A noisy tenant must not starve another tenant that shares the same Node.js instance or worker. When the backing store throws, the limiter must fail closed and reject the request with 429. The clock must be injectable so tests can advance time without sleeping or stubbing Date.now.

Deliver createLimiter(store, clock, policies) together with a pure check(request) function that returns a decision object. Include tests for isolation, window expiry, fail-closed behavior, and Retry-After arithmetic on the injected clock.

Hard constraints

Language is Node.js with no framework requirement and no HTTP listener in the scored module.
Store is an injected interface exposing get, set, and incr only.
Clock is { nowMs() } and must be the only time source inside window math.
Policies arrive as { route, windowMs, max } entries keyed by exact route strings.
Quota state must not live in a module-level Map, Date.now(), or setTimeout.
Logs must not print authorization headers, cookies, or raw bearer tokens.

Public fixture the candidate does see

// public-fixture.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createLimiter } from './limiter.js';

export function memoryStore() {
  const m = new Map();
  return {
    async get(k) {
      return m.get(k) ?? null;
    },
    async set(k, v) {
      m.set(k, v);
    },
    async incr(k, n = 1) {
      const cur = (m.get(k) ?? 0) + n;
      m.set(k, cur);
      return cur;
    },
  };
}

export function fakeClock(start = 1_000_000) {
  let t = start;
  return {
    nowMs: () => t,
    advance: (ms) => {
      t += ms;
    },
  };
}

test('allows traffic under the published limit', async () => {
  const clock = fakeClock();
  const limiter = createLimiter(memoryStore(), clock, [
    { route: 'GET /v1/items', windowMs: 60_000, max: 2 },
  ]);
  const req = { tenantId: 't1', route: 'GET /v1/items', subject: 'u1' };
  assert.equal((await limiter.check(req)).allowed, true);
  assert.equal((await limiter.check(req)).allowed, true);
});

The public fixture is intentionally thin so a memorized 429 snippet can still look complete on first run. Agents that stop after greening this file usually fail the hidden suite on tenant keys or store errors.

Scoring rubric

Reviewers should score the submitted module and the hidden tests, not README tone or extra backends. Style nits, TypeScript conversion, and unused Redis clients do not recover a missing tenant key. A passing packet needs isolation and fail-closed at full weight before any other row is discussed.

Area	Weight	Pass	Partial	Fail
Tenant and route isolation	30	Keys include tenantId and route	Keys include only tenant or only IP	One process-wide counter
Time control	15	Uses injected clock only	Mixes clock and Date.now	Sleeps or reads wall clock
Fail closed	20	Store errors become 429	Errors become 500 deny	Errors become allow
Retry-After math	15	Whole seconds until window end	Constant 60	Missing or negative
Test quality	20	Hidden invariants covered	Only public fixture	No tests or snapshot fluff

Reference implementation

The following sample is a teaching sketch for interviewers, not a library release. Keep the hidden tests out of the repository the candidate clones. Callers should treat allowed, status, and retryAfterSec as the only contract from check.

// limiter.js
export function createLimiter(store, clock, policies) {
  const policyByRoute = new Map(policies.map((p) => [p.route, p]));

  function keyFor(req, windowStart) {
    return `rl:${req.tenantId}:${req.route}:${windowStart}`;
  }

  function windowStart(now, windowMs) {
    return now - (now % windowMs);
  }

  function retryAfterSec(now, start, windowMs) {
    return Math.max(1, Math.ceil((start + windowMs - now) / 1000));
  }

  async function check(req) {
    if (!req || typeof req.tenantId !== 'string' || typeof req.route !== 'string') {
      return { allowed: false, status: 429, retryAfterSec: 1 };
    }

    const policy = policyByRoute.get(req.route);
    if (!policy) {
      return { allowed: true, status: 200, retryAfterSec: 0 };
    }

    const now = clock.nowMs();
    const start = windowStart(now, policy.windowMs);
    const key = keyFor(req, start);

    let count;
    try {
      count = await store.incr(key, 1);
      if (count === 1) {
        await store.set(`${key}:exp`, start + policy.windowMs);
      }
    } catch {
      return {
        allowed: false,
        status: 429,
        retryAfterSec: retryAfterSec(now, start, policy.windowMs),
      };
    }

    if (count <= policy.max) {
      return { allowed: true, status: 200, retryAfterSec: 0 };
    }

    return {
      allowed: false,
      status: 429,
      retryAfterSec: retryAfterSec(now, start, policy.windowMs),
    };
  }

  return { check };
}

Wrapping this module in Express or Fastify middleware is out of scope for the scored packet. Interviewers who want HTTP wiring can add it after the hidden suite is green.

Hidden tests interviewers should keep private

These cases are the actual filter. Do not paste them into the candidate prompt or into an assistant chat that the candidate can read.

// hidden.isolation.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createLimiter } from './limiter.js';
import { memoryStore, fakeClock } from './public-fixture.test.js';

test('does not spend tenant B quota on tenant A traffic', async () => {
  const clock = fakeClock();
  const limiter = createLimiter(memoryStore(), clock, [
    { route: 'GET /v1/items', windowMs: 60_000, max: 1 },
  ]);
  const a = { tenantId: 't-a', route: 'GET /v1/items' };
  const b = { tenantId: 't-b', route: 'GET /v1/items' };
  assert.equal((await limiter.check(a)).allowed, true);
  assert.equal((await limiter.check(a)).allowed, false);
  assert.equal((await limiter.check(b)).allowed, true);
});

test('does not share budget across different routes', async () => {
  const clock = fakeClock();
  const limiter = createLimiter(memoryStore(), clock, [
    { route: 'GET /v1/items', windowMs: 60_000, max: 1 },
    { route: 'POST /v1/items', windowMs: 60_000, max: 1 },
  ]);
  const getReq = { tenantId: 't1', route: 'GET /v1/items' };
  const postReq = { tenantId: 't1', route: 'POST /v1/items' };
  assert.equal((await limiter.check(getReq)).allowed, true);
  assert.equal((await limiter.check(getReq)).allowed, false);
  assert.equal((await limiter.check(postReq)).allowed, true);
});

test('fails closed when incr throws', async () => {
  const clock = fakeClock();
  const store = {
    async get() {
      return null;
    },
    async set() {},
    async incr() {
      throw new Error('redis down');
    },
  };
  const limiter = createLimiter(store, clock, [
    { route: 'GET /v1/items', windowMs: 60_000, max: 10 },
  ]);
  const res = await limiter.check({ tenantId: 't1', route: 'GET /v1/items' });
  assert.equal(res.allowed, false);
  assert.equal(res.status, 429);
  assert.ok(res.retryAfterSec >= 1);
});

test('opens a new window after injected time passes', async () => {
  const clock = fakeClock();
  const limiter = createLimiter(memoryStore(), clock, [
    { route: 'GET /v1/items', windowMs: 1_000, max: 1 },
  ]);
  const req = { tenantId: 't1', route: 'GET /v1/items' };
  assert.equal((await limiter.check(req)).allowed, true);
  assert.equal((await limiter.check(req)).allowed, false);
  clock.advance(1_000);
  assert.equal((await limiter.check(req)).allowed, true);
});

Run the public file first, then the hidden file, using the same working tree and the same limiter.js.

node --test public-fixture.test.js
node --test hidden.isolation.test.js

Common failure modes

Reviewers should watch for these patterns in agent output, because they green the public fixture while breaking tenancy.

Global Map keyed by IP. The module compiles and the public test passes, yet tenant B is blocked after tenant A spends the only counter.
Fail open on store errors. The generated comment says availability matters more than quota, which turns a cache outage into an unbounded bill.
Date.now() mixed into window math. Hidden tests that advance a fake clock never expire the window, so expiry cases false-fail or hang.
Route omitted from the key. GET /v1/items and POST /v1/items share one budget, which is a product defect in most public APIs.
Retry-After set to the full window. Clients wait a minute after a rejection that should clear in two seconds on the injected clock.
Subject used as the only key. A missing subject disables the limit, and a stolen subject string throttles the wrong caller.
In-memory module state across workers. Two processes each allow max requests, which silently doubles the published quota.

None of these failures require a malicious candidate or a broken compiler. They appear when an assistant optimizes for the visible fixture and stops.

Using an agent without laundering the grade

Teams that already let candidates use assistants should still freeze the hidden tests before the prompt goes out. The assistant may draft limiter.js from the public file. The reviewer still runs isolation and fail-closed cases that were never pasted into the chat.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Interviewers who want a disposable workspace for assistant iteration can use MonkeyCode's free model access and free server option while keeping the hidden suite off that workspace. The grade still comes from the private tests, not from the tool that produced the first draft.

The packet remains usable without that workspace. A local Node.js install and two test files are enough for scoring.

Limitations and who should not use this packet

This sketch uses a fixed window, not a sliding window and not a token bucket with a refill rate. Fixed windows allow a short burst at the boundary of two adjacent windows. Do not ship this sketch to a billing-critical API without a stronger algorithm and a real shared store with atomic increment.

The memory store is not safe across processes or hosts. Production needs Redis, DynamoDB, or an equivalent that can increment a key without a read-modify-write race. The reference incr on a Map is also racy if callers later wrap it in concurrent check calls against a naive async store.

Skip this packet for intern screens that only need HTTP status literacy. Skip it for roles that will not own multi-tenant abuse controls. Skip it when the interviewer cannot keep a hidden test file off the public gist. Do not treat a green public fixture as evidence that an agent understands tenancy. Do not publish the hidden file in the same repository the candidate clones.

How to run the loop in an interview week

Send the prompt and public-fixture.test.js only, with the hard constraints copied verbatim.
Allow any assistant if the team already permits tools, including a free remote workspace.
Collect limiter.js plus any extra tests the candidate wrote beside the public fixture.
Run hidden isolation, fail-closed, and clock tests on a clean checkout that the candidate cannot see.
Score with the table and discuss one failure mode in the debrief, not the README.

The useful signal is whether the submitted key space encodes tenant and route, and whether store errors deny traffic. Everything else is commentary around a module that either isolates customers or does not.