Grading At-Least-Once Delivery: A Webhook Ingestion Take-Home With Rubric and Reference Implementation

The fastest way to judge whether an agent-written service is trustworthy is to give it a task whose correctness cannot be inferred from the happy path. Webhook ingestion fits that requirement, because the interesting decisions involve duplicates, restarts, and concurrency rather than one successful request. A submission that returns 200 for the first delivery will look correct in a demo and lose money in production.

This packet contains a 90-minute prompt, a weighted rubric, a runnable reference solution, and the failure modes that show up most often in submissions. It is written for a reviewer who must compare several agent-generated pull requests and needs a repeatable score instead of a gut feeling.

The prompt

Give the candidate one contract and no architectural hints.

Task: Build a service that ingests order webhooks, exposes GET /orders/:id, and reports how many times the order state changed.

Delivery contract:

• Delivery is at-least-once, so the same event may arrive any number of times.
• Duplicates may arrive up to 24 hours later, re-serialized with different key order or whitespace.
• Concurrent duplicates are possible, including from separate processes.
• The process may be restarted at any time without warning.
• Every 2xx response is treated as permanent success by the sender, while anything else triggers another attempt.

Required endpoints:

1. POST /webhooks/orders applies the state change at most once and returns a stable body for duplicates.
2. GET /orders/:id returns the current order plus stateChanges, the number of times the change was applied.
3. GET /health provides a cheap readiness check for the grader.

Deliberately omit guidance about dedupe keys, storage, and locking, because those choices are the measurement.

The rubric

Dimension	Weight	Full marks	Instant zero
Dedupe key derivation	20	Explicit event id when present, canonical hash otherwise	Raw body hash only
Exactly-once side effect	25	Key insert and state change commit in one transaction guarded by a unique index	Check-then-insert in application code
Durability across restart	15	Dedupe state survives process death	In-memory Set or Map
HTTP and retry semantics	15	200 for duplicates with the original response body	409, 500, or a body that changes between attempts
Tests that would catch the above	15	Concurrent duplicates and a restart are covered against real storage	Only a happy-path test with a mocked database
Operational notes	10	Retention window, table growth, and a duplicate-rate metric	No mention of what happens after a year

Scores below 60 usually indicate the submission would lose events or double-apply them under ordinary retry behavior. Scores above 85 require that the candidate explains why the unique index, not the application check, is the actual guarantee.

Reference solution

SQLite is the smallest storage engine that provides real transactions and a unique index, so it keeps the task honest without requiring infrastructure. The schema stores the dedupe record and the order in the same transaction, which makes the side effect atomic with the acknowledgement.

CREATE TABLE IF NOT EXISTS webhook_events (
  event_key   TEXT PRIMARY KEY,
  received_at TEXT NOT NULL,
  body        TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS orders (
  order_id     TEXT PRIMARY KEY,
  event_key    TEXT NOT NULL REFERENCES webhook_events(event_key),
  status       TEXT NOT NULL,
  amount_cents INTEGER NOT NULL
);

Canonicalization is required because duplicates get re-serialized by proxies and queues. Sorting object keys recursively produces a stable byte sequence before hashing.

import { createHash } from "node:crypto";

const canonicalize = (value: unknown): string => {
  if (Array.isArray(value)) return `[${value.map(canonicalize).join(",")}]`;
  if (value && typeof value === "object") {
    const entries = Object.entries(value as Record<string, unknown>)
      .sort(([a], [b]) => a.localeCompare(b));
    return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${canonicalize(v)}`).join(",")}}`;
  }
  return JSON.stringify(value ?? null);
};

export const deriveKey = (raw: string, headers: Record<string, string | undefined>) => {
  const explicit = headers["x-event-id"];
  if (explicit) return `id:${explicit}`;
  const parsed = JSON.parse(raw);
  const digest = createHash("sha256").update(canonicalize(parsed)).digest("hex");
  return `sha256:${digest}`;
};

The ingestion path relies on ON CONFLICT DO NOTHING and on the returned row count, so two concurrent writers cannot both apply the change.

const ingest = db.transaction((eventKey: string, order: Order) => {
  const inserted = db.prepare(
    `INSERT INTO webhook_events (event_key, received_at, body)
     VALUES (?, ?, ?) ON CONFLICT(event_key) DO NOTHING`
  ).run(eventKey, new Date().toISOString(), JSON.stringify({ ok: true, orderId: order.id }));

  if (inserted.changes === 0) {
    const prior = db.prepare("SELECT body FROM webhook_events WHERE event_key = ?")
      .get(eventKey) as { body: string };
    return prior.body;
  }

  db.prepare(
    `INSERT INTO orders (order_id, event_key, status, amount_cents)
     VALUES (?, ?, ?, ?)`
  ).run(order.id, eventKey, order.status, order.amountCents);

  return JSON.stringify({ ok: true, orderId: order.id });
});

The HTTP layer must read the raw body, because parsing and re-encoding before hashing destroys byte-level stability for clients that omit an event id.

app.post("/webhooks/orders", express.raw({ type: "application/json" }), (req, res) => {
  const raw = req.body.toString("utf8");
  const order = JSON.parse(raw) as Order;
  const body = ingest(deriveKey(raw, req.headers), order);
  res.status(200).type("application/json").send(body);
});

A grader can then assert the guarantee directly instead of reading the code.

seq 1 50 | xargs -P 16 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST localhost:3000/webhooks/orders \
  -H 'content-type: application/json' -H 'x-event-id: evt_load_1' \
  -d '{"id":"o_load","status":"paid","amountCents":100}'

curl -s localhost:3000/orders/o_load | jq -e '.stateChanges == 1'

Re-sending the same logical event without the x-event-id header, but with reordered keys, is the second assertion that separates canonical hashing from naive body hashing.

Running a submission in a disposable environment

Candidate submissions should never execute on a laptop or a shared runner that holds production credentials, because the grader starts arbitrary code and follows arbitrary install scripts. A scratch environment solves that, and the workflow is short: clone the repository, install with npm ci, start the service on a non-default port, run the assertions above, then destroy the environment.

MonkeyCode's operator describes a free server option and free model access, including a stated allowance of 10 million tokens, which is enough to run a reference agent and a grader loop without provisioning cloud spend first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free-tier quotas, available models, and server specifications are set by the operator and can change, so verify the current terms before wiring the environment into a paid pipeline.

The free environment is suitable for disposable grading, and it is not suitable for storing candidate code that must persist, for holding real webhook payloads, or for anything with an availability commitment. Ephemeral storage and shared capacity are the expected trade-offs, and secrets belong in environment variables that are created per run.

Common failure modes

1. Hashing the raw body. Duplicates that arrive through a re-serializing proxy produce a different digest and get applied twice.
2. In-memory deduplication. A Set passes the demo and loses every key on restart, which the contract explicitly allows.
3. Check-then-insert. Two concurrent deliveries both read "not seen" and both write the order row; only a unique index closes the window.
4. Deduplicating after the side effect. The charge or email is sent before the key is recorded, so a crash between the two produces a double effect.
5. Returning 409 for duplicates. The sender treats any non-2xx response as failure and retries indefinitely, which amplifies the duplicate storm.
6. Unbounded dedupe tables. The submission has no retention policy, so the table grows with every retry forever.
7. Ordering assumptions. Arrival order is treated as causality, so a late duplicate rewrites the order back to an older status.
8. Mocked storage in tests. The unit tests pass because the mock never enforces the unique constraint that production relies on.

Each failure mode is worth naming explicitly in the review notes, because the candidate's response to the finding is often more informative than the original code.

Calibration anchors

A junior submission typically implements deduplication, but places it in the wrong layer and cannot explain the failure window. A mid-level submission uses a unique constraint correctly and adds tests for concurrency, yet skips retention planning and restart coverage. A senior submission states the guarantee in terms of the storage engine, documents the retention window, emits a duplicate-rate metric, and identifies what the design deliberately does not handle.

Reviewers should also read the agent transcript rather than only the diff. A candidate who accepted an incorrect suggestion without checking it will usually repeat that behavior on the team.

Who should not use this approach

Teams without a written delivery contract should fix that first, because the rubric measures conformance to a contract that must exist. Roles that never touch stateful services, such as pure front-end positions, gain little from this packet. Anyone planning to reuse a submission as production code should stop, since the task is designed to expose gaps rather than to ship.

If the workflow is useful, the free environment is available for running a grader against a submitted service, and the operator is interested in which failure mode your model produced first.