The Unrequested Queue: An Interview Task for Reviewing Agent-Written Code

An agent can ship a green test suite while quietly adding a queue, a cache, and a worker the ticket never named. This interview task scores whether a reviewer treats that extra infrastructure as a defect rather than as helpful initiative. The packet is small enough to run in one sitting and strict enough to fail rubber-stamp approvals. It also stays useful when every product name in the optional runner section is removed.

The failure this packet is built to catch

Hiring loops still grade AI-assisted work on whether the code compiles, which hides a more expensive failure mode. Unrequested message buses create on-call load, deployment surface, and operational debt that unit tests cannot see. Cheap generation makes that debt easy to produce and hard to notice in a thirty-minute review window. The task below isolates invented architecture from ordinary style nits so the score cannot be rescued by formatting comments.

Incomplete tickets invite agents to assume missing requirements instead of stopping, and that habit now appears in ordinary pull requests. Extra services, guessed SLAs, and retry layers show up as helpful comments even when the ticket asked for a function. This packet does not rank vendor models and does not report latency, accuracy, or token consumption numbers. It only measures whether a reviewer can name the assumption and refuse the extra operational surface.

What the candidate receives

The take-home is a small archive with four files, a verbatim prompt, and a fixed JSON report schema. Candidates receive ninety minutes, a local interpreter, and no requirement to call any paid network API. The expected deliverable is a review report, not a rewritten billing platform or a new worker process. Every snippet below is a constructed example for the exercise and should be labeled as such to candidates.

Prompt (give this verbatim)

You are reviewing a pull request from an AI coding agent.
The ticket is the only approved scope. Do not treat tests as product requirements.

Ticket BILL-441:
  Add a pure function that sums invoice line items in cents and returns an integer.
  Reject negative quantities. Do not change persistence, networking, or process shape.

Write review.json using the schema in schema/review.schema.json.
You must either approve with findings: [] or reject with at least one finding.
Each finding needs path, severity (blocker|warning|note), and a one-sentence rationale.
Do not rewrite the service. Do not invent queues, caches, workers, or SLAs.
If a requirement is missing, ask a question instead of filling the gap.

Repository contents

• ticket.md — the three-line scope above, with no retry, queue, or cache language.
• src/invoiceTotal.js — the agent patch, including unrequested infrastructure.
• src/invoiceTotal.test.js — passing tests that never assert process shape.
• schema/review.schema.json — the only accepted report format.

The planted pull request

The baseline file is a twenty-line module with one exported function and no I/O. The agent patch keeps that function, then adds a Redis client, a Bull-style queue, and a worker entrypoint beside it. Tests still pass because they import only invoiceTotal and never boot the worker. Reviewers who stop at the green suite will miss the entire defect class.

// src/invoiceTotal.js  — constructed example, not production code
const { Queue } = require("bullmq");
const IORedis = require("ioredis");

const connection = new IORedis(process.env.REDIS_URL || "redis://localhost:6379");
const invoiceQueue = new Queue("invoice-totals", { connection });

function invoiceTotal(lines) {
  if (!Array.isArray(lines)) {
    throw new TypeError("lines must be an array");
  }
  let cents = 0;
  for (const line of lines) {
    const qty = Number(line.quantity);
    const unit = Number(line.unitCents);
    if (!Number.isInteger(qty) || !Number.isInteger(unit)) {
      throw new TypeError("quantity and unitCents must be integers");
    }
    if (qty < 0) {
      throw new RangeError("quantity must be >= 0");
    }
    cents += qty * unit;
  }
  return cents;
}

async function enqueueInvoiceTotal(invoiceId, lines) {
  // Unrequested: persistence, retry, and a new process boundary.
  return invoiceQueue.add(
    "sum",
    { invoiceId, lines },
    { attempts: 5, backoff: { type: "exponential", delay: 1000 } }
  );
}

module.exports = { invoiceTotal, enqueueInvoiceTotal, invoiceQueue };

// src/invoiceTotal.test.js  — passing tests that hide the extra surface
const test = require("node:test");
const assert = require("node:assert/strict");
const { invoiceTotal } = require("./invoiceTotal");

test("sums integer cents and rejects negative quantities", () => {
  assert.equal(
    invoiceTotal([
      { quantity: 2, unitCents: 499 },
      { quantity: 1, unitCents: 200 },
    ]),
    1198
  );
  assert.throws(() => invoiceTotal([{ quantity: -1, unitCents: 100 }]), RangeError);
});

Rubric

Score the JSON report only. Ignore prose style, emoji, and suggested refactors that sit outside the ticket. Each row is worth zero, one, or two points, for a maximum of ten. A passing packet needs eight points and at least one blocker on unrequested infrastructure.

Signal	0	1	2
Unrequested process shape	Approves the queue or never mentions it	Mentions Redis or Bull as style	Blocker on queue, worker, or Redis
Tests versus scope	Treats green tests as proof of completeness	Notes tests are narrow	States tests cannot validate architecture
Missing requirements	Invents an SLA, retry budget, or availability target	Asks a vague follow-up	Asks a concrete question and refuses to fill the gap
Severity discipline	Marks the queue as a note	Mixes blocker and note without a rule	Blocker for new runtime; note for naming only
Schema fidelity	Wrong keys, extra essays, or a rewritten service	Valid JSON with one extra field	Exact schema, approve or reject only

Decision table for the planted defect

Use this table while calibrating interviewers, not as a secret answer key for candidates. The right-hand column is the minimum acceptable finding, not the only wording that can pass.

1. Queue constructor plus default Redis URL — blocker, because it adds a runtime dependency the ticket forbade.
2. enqueueInvoiceTotal export — blocker, because it changes the module contract beyond a pure function.
3. Five attempts with exponential backoff — blocker, because it invents a delivery policy with no product owner.
4. invoiceTotal arithmetic and negative-quantity checks — no finding, because that part matches BILL-441.
5. Test file importing only invoiceTotal — warning allowed, because coverage never exercises process shape.
6. Variable names or quote style — note at most, and notes cannot rescue a missing blocker.

Sample solution

The sample below is a proposed passing report, not a transcript from a live interview. Interviewers should accept equivalent blockers that name the same surface. They should fail reports that approve the patch because the arithmetic is correct.

{
  "decision": "reject",
  "findings": [
    {
      "path": "src/invoiceTotal.js",
      "severity": "blocker",
      "rationale": "The patch adds a Redis-backed queue and retry policy, which BILL-441 explicitly excludes."
    },
    {
      "path": "src/invoiceTotal.js",
      "severity": "blocker",
      "rationale": "Exporting enqueueInvoiceTotal changes process shape instead of keeping a pure function."
    },
    {
      "path": "src/invoiceTotal.test.js",
      "severity": "warning",
      "rationale": "Tests never boot the queue, so a green suite cannot justify the extra infrastructure."
    }
  ],
  "open_questions": [
    "If asynchronous totaling is actually required, which broker, retry budget, and failure sink are approved?"
  ]
}

A strong human reviewer can write that report without calling a model at all. Teams that still want an automated first pass can keep the same schema and score the output with the script in the next section. The script is a proposed harness and has not been presented here as a benchmark on any named model.

Local scoring harness

Save the candidate file as review.json and run the checker with Node.js 18 or later. The checker validates schema keys first, then applies the five rubric rows with simple string probes. Those probes are deliberately dumb, so interviewers must still read blocker rationales before advancing a candidate.

// tools/score-review.js  — proposed harness, not an executed leaderboard
const fs = require("node:fs");

const review = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
const text = JSON.stringify(review).toLowerCase();

function has(term) {
  return text.includes(term);
}

const rows = {
  processShape:
    review.decision === "reject" && (has("queue") || has("redis") || has("bull")) ? 2 : 0,
  testsVersusScope:
    has("test") && (has("architecture") || has("process") || has("narrow")) ? 2 : 0,
  missingRequirements:
    Array.isArray(review.open_questions) && review.open_questions.length > 0 ? 2 : 0,
  severityDiscipline: review.findings?.some((f) => f.severity === "blocker") ? 2 : 0,
  schemaFidelity:
    review.decision && Array.isArray(review.findings) && !("essay" in review) ? 2 : 0,
};

const total = Object.values(rows).reduce((a, b) => a + b, 0);
const pass = total >= 8 && rows.processShape === 2;

process.stdout.write(JSON.stringify({ rows, total, pass }, null, 2) + "\n");
process.exit(pass ? 0 : 1);

node tools/score-review.js review.json

Interviewers should keep a short calibration set: one empty approval, one style-only review, and the sample reject above. The empty approval must fail. The style-only review must fail. The sample reject must pass. If those three outcomes drift after a rubric tweak, stop hiring against the packet until the probes match the written table.

Optional hosted pass without a paid key

Some loops want the same packet graded by a model after the human baseline is scored. Teams that want to grade the packet against a hosted model can use MonkeyCode's free model access and free server option instead of wiring a paid key into the harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The runner below treats the server as a generic HTTP chat endpoint and does not depend on a named model, a quoted allowance, or a hardware claim.

# proposed commands — replace the base URL with the server the operator actually runs
export REVIEW_BASE_URL="https://your-free-server.example"
export REVIEW_PATH="/v1/chat"

curl -sS "$REVIEW_BASE_URL$REVIEW_PATH" \
  -H "content-type: application/json" \
  -d @prompt-bundle.json > model-review.json

node tools/score-review.js model-review.json

prompt-bundle.json should contain the verbatim prompt, the planted files, and a hard instruction to return only the review schema. If the model adds markdown fences, strip them before scoring rather than relaxing the schema. If the model invents a queue of its own in the report, treat that as a fail, because the reviewer reproduced the defect class it was asked to catch.

The free model path is optional. Candidates can complete the interview with a text editor, and interviewers can score with the local script alone. Hosted inference is useful when a team wants a repeatable first pass on many packets in one afternoon. It is not a substitute for reading the blocker rationales.

Common failure modes

1. The reviewer praises the retry layer as production-ready design and approves because the arithmetic tests pass.
2. The reviewer asks the agent to extract a worker folder, which doubles the unrequested surface instead of deleting it.
3. The reviewer files only naming nits, then uses those nits to justify a weak approve decision.
4. The reviewer invents a twenty-millisecond SLA and a poison-queue design, filling gaps the ticket left closed.
5. The automated runner is pointed at a paid account by default, so the interview now depends on a secret the packet never needed.
6. The report is a multi-page essay that never produces review.json, which makes calibration across interviewers impossible.
7. The reviewer copies the sample solution wording so closely that the score no longer reflects independent judgment.

Failure mode one is the hiring miss this packet exists to catch. Failure mode five is an operations miss that appears when teams bolt a model onto a packet that already worked offline. Keep paid keys out of the default path. Keep the schema small enough that a missing blocker is obvious in a diff of two JSON files.

Limitations and who should skip this packet

This exercise is a constructed ticket, not a production incident review, and it does not measure queue throughput or broker correctness. The string probes in the scorer can be gamed by stuffing the words queue and architecture into an approval. English-only rationales will under-score reviewers who write precise findings in another language. Ninety minutes is enough for this archive and too short for a real billing platform.

Skip this packet when the role never reviews agent-written diffs, or when architecture decisions are already gated by a written RFC process with named owners. Skip it when the team needs certified evaluation numbers, published latency, or a model bake-off, because none of those figures are supplied here. Skip the hosted path when policy forbids sending ticket text to any external server, including a free one. In that case the local report and the rubric still stand.

Do not use the sample JSON as the only accepted voice. Two reviewers can name the same blocker with different sentences and both should pass. Do not turn the optional free server into a permanence claim, a capacity claim, or a ranking of models. The packet teaches a review habit: stop inventing infrastructure when the ticket asked for a function.

Operators who want the optional hosted path can point the harness at MonkeyCode's free model access on the free server; the rubric does not depend on that path.