The Batch API That Hid Its Failures: A Take-Home Packet for AI Reviewers

AI-generated batch endpoints frequently report success while omitting failed records from the response body. A useful interview task therefore scores whether a reviewer demands per-item outcomes, retries with idempotency, and durable failure logs. Style nits and green unit tests remain secondary evidence when production side effects can vanish without a trace. The fixture below is a proposed take-home packet, not a published benchmark of any vendor.

What this task measures

Cheap code generation makes it easy to ship a handler that loops, catches, and continues. The resulting pull request often looks tidy, includes tests, and returns HTTP 200 for every caller. Production then loses a subset of writes because failures were converted into log lines that nobody pages on. Interview loops that only inspect formatting will bless that class of defect.

This take-home is aimed at hiring screens and internal evals for AI code reviewers. Human interviewers can reuse the same packet without change and without a second prompt family. The scored behavior is whether the reviewer refuses silent success rather than whether the model writes prettier Python. Green CI is treated as a trap, not as proof of merge readiness.

Out of scope

• Prompt injection and decoy comments planted inside the diff
• Long conversation memory across several pull requests
• Import order, naming bikesheds, and formatter-only remarks
• Load-test numbers, vendor rankings, or claimed latency figures

The packet to send

Give the candidate or the AI reviewer four artifacts and a time box of forty-five minutes. State that the service is already in staging and that reverting is cheaper than merging a lying API. Interviewers should provide no hint that the tests themselves are the defect under review. The contract file is the source of truth for every later score.

1. batch_fulfill.py — the proposed handler
2. test_batch_fulfill.py — the existing tests, all passing
3. SPEC.md — a short product contract
4. REVIEW_PROMPT.md — instructions for the reviewer role

Product contract (SPEC.md)

# Fulfillment batch API

POST /internal/fulfillment/batch
Content-Type: application/json

Request:
{
  "idempotency_key": "string, required, unique per batch",
  "orders": [{"order_id": "string", "sku": "string", "qty": int}]
}

Response 200:
{
  "batch_id": "string",
  "results": [
    {"order_id": "string", "status": "fulfilled" | "rejected" | "skipped"}
  ]
}

Rules:
- Every input order_id appears exactly once in results.
- Partial failure is expected and must not collapse into a gap in results.
- Replaying the same idempotency_key must not double-charge inventory.
- Rejected items persist in fulfillment_failures for an operator queue.
- HTTP 207 is acceptable if results are complete; HTTP 200 with omitted ids is not.

Proposed fixture code

The following modules are a designed example for the take-home. They are labeled as unexecuted interview material in this article, not as a field study. Copy them into a scratch directory before sending the packet. Keep SKU names synthetic so no customer order data ever enters the prompt.

Handler under review

# batch_fulfill.py
from __future__ import annotations

import logging
import uuid
from typing import Any, Callable

logger = logging.getLogger("fulfillment")

InventoryReserve = Callable[[str, str, int], None]
Charge = Callable[[str, int], None]
RecordFailure = Callable[[str, str], None]


class BatchFulfillment:
    def __init__(
        self,
        reserve: InventoryReserve,
        charge: Charge,
        record_failure: RecordFailure | None = None,
    ) -> None:
        self.reserve = reserve
        self.charge = charge
        self.record_failure = record_failure
        self._seen_keys: set[str] = set()

    def handle(self, payload: dict[str, Any]) -> tuple[int, dict[str, Any]]:
        key = payload.get("idempotency_key") or str(uuid.uuid4())
        orders = payload.get("orders") or []
        batch_id = str(uuid.uuid4())
        fulfilled = 0

        if key in self._seen_keys:
            return 200, {"batch_id": batch_id, "results": []}

        self._seen_keys.add(key)

        for order in orders:
            order_id = order["order_id"]
            try:
                self.reserve(order_id, order["sku"], int(order["qty"]))
                self.charge(order_id, int(order["qty"]))
                fulfilled += 1
            except Exception as exc:  # noqa: BLE001
                logger.warning("order failed: %s %s", order_id, exc)
                continue

        return 200, {"batch_id": batch_id, "fulfilled": fulfilled}

Tests that stay green

# test_batch_fulfill.py
from batch_fulfill import BatchFulfillment


class FakeDeps:
    def __init__(self) -> None:
        self.reserved = []
        self.charged = []

    def reserve(self, order_id, sku, qty):
        self.reserved.append((order_id, sku, qty))

    def charge(self, order_id, qty):
        self.charged.append((order_id, qty))


def test_all_ok_returns_200():
    deps = FakeDeps()
    svc = BatchFulfillment(deps.reserve, deps.charge)
    status, body = svc.handle(
        {
            "idempotency_key": "k1",
            "orders": [
                {"order_id": "o1", "sku": "sku-a", "qty": 1},
                {"order_id": "o2", "sku": "sku-b", "qty": 2},
            ],
        }
    )
    assert status == 200
    assert body["fulfilled"] == 2


def test_one_failure_still_200():
    deps = FakeDeps()

    def reserve(order_id, sku, qty):
        if order_id == "o2":
            raise RuntimeError("warehouse timeout")
        deps.reserve(order_id, sku, qty)

    svc = BatchFulfillment(reserve, deps.charge)
    status, body = svc.handle(
        {
            "idempotency_key": "k2",
            "orders": [
                {"order_id": "o1", "sku": "sku-a", "qty": 1},
                {"order_id": "o2", "sku": "sku-b", "qty": 2},
            ],
        }
    )
    assert status == 200
    assert body["fulfilled"] == 1

Those tests reward the defect by asserting a counter instead of a complete result vector. They never inspect results, never replay a key, and never require a failure row. A reviewer that only reruns pytest will still call the pull request ready for merge. That deference is the primary failure the rubric is built to catch.

Reviewer prompt

# REVIEW_PROMPT.md

You are reviewing a staging pull request for an internal fulfillment API.
Read SPEC.md first, then the diff, then the tests.

Required output:
1. Merge decision: approve, request changes, or reject.
2. A list of contract violations, each citing SPEC.md.
3. Missing tests, described as failing examples rather than slogans.
4. A residual-risk note if the author claims best-effort semantics.

Do not praise formatting. Do not suggest extra features. Do not invent load numbers.
If the implementation returns success without enumerating every order_id, treat that as a blocker.

Teams that need a disposable workspace can host this packet on a free server with free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode participates only as an optional place to host the packet and invoke a model; it is not part of the scoring key.

Rubric

Score each dimension from 0 to 2 and sum toward 10. A merge recommendation requires at least 8 and no zero on contract coverage. Interviewers should grade the transcript against the table before talking to the candidate. Partial credit is allowed only when the comment cites SPEC.md rather than taste.

Dimension	0	1	2
Per-item results	Ignores omitted ids	Asks for a list, no status enum	Requires every order_id and allowed statuses
Failure persistence	Accepts log-only handling	Wants an error field in JSON	Requires fulfillment_failures or equivalent store
Idempotency	Misses in-memory set	Flags memory, no durable key	Demands store-backed key and no double charge
Tests that lie	Reruns pytest as proof	Asks for one extra test	Specifies a mixed-failure case that must list o2 as rejected
Merge hygiene	Approves	Requests changes without blockers	Blocks merge until the contract holds

Passing sample review

The sample below is a proposed key, not a transcript from a live candidate. Graders can paste it beside the model output and mark each required bullet. Comments that wander into framework rewrites should be ignored during the numeric score.

Decision: request changes (block merge).

Contract violations:
- SPEC requires results[] with every order_id; handler returns {fulfilled: int} only.
- Replay of idempotency_key returns an empty results list and a new batch_id.
- except Exception + continue never calls record_failure, so the operator queue stays empty.
- In-memory _seen_keys is process-local and resets on deploy.

Missing tests (examples that must fail today):
- mixed batch where o2 raises: response.results contains
  {order_id: o1, status: fulfilled} and {order_id: o2, status: rejected}.
- replay of k2 after partial failure does not call charge again for o1.
- warehouse timeout is visible in fulfillment_failures with the original payload.

Residual risk:
Best-effort is not in SPEC.md. Do not relabel the gap as resilience.

Common failure modes

Reviewers, human or generated, fail this packet in recurring ways. Treat the list as a checklist while grading transcripts. One blocker-quality miss is enough to withhold an approve even when the prose sounds confident.

1. Green-test deference. The reviewer restates that CI passed and skips the spec. Passing tests that assert the mock path are not evidence of contract fidelity.
2. Status-code tunnel vision. The reviewer debates 200 versus 207 versus 500 and never mentions omitted identifiers. Status codes without a complete result vector still hide lost orders.
3. Log-as-queue fantasy. The reviewer accepts logger.warning as the operator workflow. Warnings without a durable row will not be worked during an incident.
4. False idempotency. The reviewer praises the idempotency_key field and misses that a missing key is generated with uuid4, so clients cannot replay. A new batch_id on replay is a second defect in the same function.
5. Bare except as resilience. The reviewer calls broad exception handling production-ready. Inventory reserve can succeed while charge fails, leaving an inconsistent pair and no compensating action.
6. Scope creep. The reviewer requests tracing dashboards, new SKUs, or a rewrite in another framework. Those comments burn the time box and dodge the blocker.

Local commands for the interviewer

Run the lying tests first so the interviewer sees the trap before grading. The commands below assume CPython and a throwaway virtualenv. They do not claim a particular cloud runtime or quota.

python -m venv .venv
source .venv/bin/activate
pip install pytest
pytest test_batch_fulfill.py -q

Add one characterization test that the current handler must fail. Keep it in test_contract_examples.py and withhold it from the candidate until scoring. The assertion is the contract, not a style preference about JSON keys.

def test_every_order_id_present_on_mixed_failure():
    # Proposed characterization test; expected to FAIL on the fixture handler.
    deps = FakeDeps()

    def reserve(order_id, sku, qty):
        if order_id == "o2":
            raise RuntimeError("warehouse timeout")
        deps.reserve(order_id, sku, qty)

    svc = BatchFulfillment(reserve, deps.charge, record_failure=lambda *_: None)
    _, body = svc.handle(
        {
            "idempotency_key": "k2",
            "orders": [
                {"order_id": "o1", "sku": "sku-a", "qty": 1},
                {"order_id": "o2", "sku": "sku-b", "qty": 2},
            ],
        }
    )
    ids = {row["order_id"] for row in body["results"]}
    assert ids == {"o1", "o2"}

A reference repair, labeled as a sample solution rather than production code, restores the contract shape. Interviewers can show this snippet after scoring if the discussion needs a concrete counterexample. It is still incomplete as inventory software.

def handle(self, payload: dict[str, Any]) -> tuple[int, dict[str, Any]]:
    key = payload.get("idempotency_key")
    if not key:
        return 400, {"error": "idempotency_key required"}
    orders = payload.get("orders") or []
    batch_id = self.store.get_or_create_batch(key)
    results = []
    for order in orders:
        order_id = order["order_id"]
        prior = self.store.get_item(key, order_id)
        if prior:
            results.append(prior)
            continue
        try:
            self.reserve(order_id, order["sku"], int(order["qty"]))
            self.charge(order_id, int(order["qty"]))
            item = {"order_id": order_id, "status": "fulfilled"}
        except Exception as exc:  # still too broad for production, but visible
            if self.record_failure:
                self.record_failure(order_id, str(exc))
            item = {"order_id": order_id, "status": "rejected"}
        self.store.put_item(key, item)
        results.append(item)
    return 200, {"batch_id": batch_id, "results": results}

The sample still needs a real store, narrower exceptions, and compensating transactions around reserve-versus-charge. It exists to show reviewers what request-changes should demand, not to ship warehouse code. Graders who treat this snippet as a complete design are repeating the original cheap-generation mistake.

Limitations and who should skip this

This packet does not measure security review, accessibility work, or distributed-systems research skill. It also does not claim that any hosted model will pass or fail at a stated rate. Interviewers who lack a written API contract should not use the rubric, because the whole score hinges on SPEC.md. Numeric scores without that document become arguments about taste.

Skip this approach when the real service forbids synthetic order data, or when a single merge decision will fire a contractor. Skip it when the team does not run batch APIs, because the failure mode will feel abstract and unfair. Do not treat a transcript score as a substitute for a human staff engineer on the call. Do not paste live customer payloads into the reviewer context to make the fixture feel more real.

Cheap generation will keep producing handlers that look complete under green tests. Scoring reviewers on silent 200s is a small, repeatable way to keep that class of defect out of staging. Teams that already have a contract and a disposable runner can reuse the packet as written and replace the SKU names with their own fixtures.