At-Least-Once Webhooks as a Take-Home: Rubric, Hidden Replays, and Failure Modes

Webhook take-homes expose a reliability gap that ordinary green unit tests rarely catch in otherwise polished candidate submissions. At-least-once delivery is the default producer contract, yet many solutions acknowledge the HTTP request before durable idempotent storage finishes. The packet below treats early 2xx responses, missing signature checks, and unordered event replays as automatic scoring failures. Interviewers can score a human candidate or an AI coding agent with the same fixtures, rubric, and hidden replay schedule.

Why this packet exists

Payment and provisioning webhooks arrive more than once, sometimes minutes apart, and occasionally in reverse sequence order. A handler that inserts one row and returns 200 still fails when a concurrent replay creates a second captured charge. Signature checks that run after JSON parsing invite body tampering that trusted fixtures in unit tests never actually exercise. The assignment therefore hides replay, reordering, and mild clock skew inside the scorer rather than the public prompt.

Tool-calling agents fail this style of task for a different but related reason than typical backend candidates. Many agents copy a framework tutorial, return JSON, and stop once a happy-path curl command prints a 200 status. They rarely add a unique index or a crash-restart case unless the public prompt names those controls explicitly. The packet is written to keep those controls in the hidden suite so the public prompt stays short.

Public prompt for the candidate

The candidate receives a small HTTP service skeleton, a documented HMAC header, and a local SQLite file path. The service must accept POST /webhooks/payments, verify the signature, and apply each event at most once. Public notes state that the producer retries on timeouts and on any status outside the 2xx range. The candidate may add tables, indexes, and middleware, but must not rename headers or alter the JSON event shape.

Wire contract

The public README shows one request shape and forbids any additional required headers beyond the three listed. The signature header uses a sha256= prefix, and the body must be hashed exactly as received on the wire. Candidates who pretty-print JSON before verification will fail the hidden tamper and reorder cases together.

POST /webhooks/payments HTTP/1.1
Content-Type: application/json
X-Webhook-Id: evt_01HZX4K2
X-Webhook-Timestamp: 1690000000
X-Webhook-Signature: sha256=d7c1...

{"event_id":"evt_01HZX4K2","type":"payment.captured","amount_cents":4099,"currency":"USD","order_id":"ord_9k2","sequence":3}

Public rules

The README lists six rules and nothing else about concurrency, disk fsync, or replay storms. Candidates who ask for extra rules during the interview should be pointed back to those six bullets. Anything about WAL mode, busy timeout, or kill -9 stays in the hidden scorer by design. The starter commands are intentionally boring so the work sits in handler correctness rather than tooling.

1. The signature is HMAC-SHA256 over the timestamp, a dot, and the raw body, using WEBHOOK_SECRET from the environment.
2. The handler must reject timestamps older than five minutes and timestamps more than thirty seconds in the future.
3. The JSON event_id is the idempotency key and must equal the X-Webhook-Id header value.
4. For one order_id, a later sequence must not mutate balances before every earlier sequence exists.
5. Duplicate delivery of an already applied event must return 2xx and must not change stored amounts.
6. The process must remain correct after a crash that occurs before the producer issues its next retry.

Starter commands in the repository look like the following block.

export WEBHOOK_SECRET=test_secret_do_not_reuse
export WEBHOOK_DB=./webhook.db
python -m pip install -r requirements.txt
python app.py

Rubric used by the hidden scorer

The scorer awards one hundred points and rejects a submission that returns 2xx for an invalid signature. Partial credit exists for durable dedupe without sequence gating, but signature failures are not negotiable. Timeouts that still persist the event can pass the crash case if a later retry becomes a no-op. Returning 409 for a true duplicate is a fail because the producer will retry forever.

Area	Points	Pass condition
Signature and raw body	25	Invalid HMAC, truncated body, and header/body id mismatch all yield non-2xx
Timestamp window	10	Skewed clocks outside the documented window yield non-2xx
Durable idempotency	25	Two concurrent replays produce one applied row after process restart
Sequence gating	20	Sequence 3 cannot apply before sequences 1 and 2 for the same order
Duplicate ACK contract	10	Second delivery of an applied event returns 2xx with unchanged totals
Crash before ACK	10	Process kill after the write but before the response still yields a single apply

Decision table for HTTP status choices

Status codes are part of the idempotency contract, not an afterthought for the web framework default handler. The table below is the scoring key for duplicates, gaps, and auth failures in this packet. Candidates who collapse every error into 500 will cause unnecessary retries and can still pass durable storage checks. Candidates who return 2xx on signature failure fail the packet regardless of database contents.

Situation	Mutate store	Status family	Producer next step
Valid new event, sequences complete	Yes, once	2xx	Stop
Valid duplicate of an applied event	No	2xx	Stop
Valid event with a sequence gap	No	409 or 503	Retry later
Bad HMAC or id mismatch	No	401 or 400	Do not retry as-is
Timestamp outside the window	No	401	Do not retry as-is
Unique-index race on insert	No extra row	2xx	Stop

Hidden fixtures the public prompt never lists

The scorer runs after the candidate declares the server ready on port 8080. It does not use the candidate's unit tests as evidence of concurrent replay safety. Each case below is a separate process restart with a copied database file. Interviewers should treat the YAML schedule as confidential material, not as extra README flavor text.

• The replay pair posts the same captured event twice in parallel with identical signatures and timestamps.
• The reorder pair sends sequence 3 first, then sequence 1, then sequence 2, each with a valid signature.
• The tamper pair computes a valid signature, then changes one digit in amount_cents before the POST.
• The clock pair sets the timestamp sixty seconds in the future relative to the scorer host clock.
• The crash pair closes the TCP connection after the first response byte if the row is not yet visible.
• The cross-order pair uses two different order_id values that must not block each other during sequence checks.

A compact driver for the replay pair looks like this labeled reference.

# Reference scorer fragment. This is an unexecuted example, not a published benchmark.
import hashlib, hmac, json, os, threading, urllib.request

SECRET = os.environ["WEBHOOK_SECRET"]
BODY = json.dumps({
    "event_id": "evt_01HZX4K2",
    "type": "payment.captured",
    "amount_cents": 4099,
    "currency": "USD",
    "order_id": "ord_9k2",
    "sequence": 1,
}, separators=(",", ":")).encode()

def post_once():
    ts = "1690000000"
    mac = hmac.new(SECRET.encode(), ts.encode() + b"." + BODY, hashlib.sha256).hexdigest()
    req = urllib.request.Request(
        "http://127.0.0.1:8080/webhooks/payments",
        data=BODY,
        headers={
            "Content-Type": "application/json",
            "X-Webhook-Id": "evt_01HZX4K2",
            "X-Webhook-Timestamp": ts,
            "X-Webhook-Signature": f"sha256={mac}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req) as resp:
        return resp.status

threads = [threading.Thread(target=post_once) for _ in range(2)]
for t in threads:
    t.start()
for t in threads:
    t.join()

Reference solution labeled as a sample

The sample below is a teaching sketch for reviewers, not a claim that this exact process ran in production traffic. It verifies the HMAC against the raw bytes, then opens an immediate SQLite transaction, then inserts the event under a unique index. Sequence gating uses a count of prior rows for the same order rather than an in-memory map. The HTTP status is chosen only after the commit returns.

# Sample solution for reviewers. Treat as labeled reference code, not live metrics.
import hashlib
import hmac
import json
import os
import sqlite3
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

SECRET = os.environ["WEBHOOK_SECRET"].encode()
DB_PATH = os.environ.get("WEBHOOK_DB", "webhook.db")

def db():
    conn = sqlite3.connect(DB_PATH, timeout=5, isolation_level=None)
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute("PRAGMA busy_timeout=5000")
    conn.execute(
        """
        CREATE TABLE IF NOT EXISTS events (
          event_id TEXT PRIMARY KEY,
          order_id TEXT NOT NULL,
          sequence INTEGER NOT NULL,
          amount_cents INTEGER NOT NULL,
          applied INTEGER NOT NULL
        )
        """
    )
    conn.execute(
        "CREATE UNIQUE INDEX IF NOT EXISTS order_seq ON events(order_id, sequence)"
    )
    return conn

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        if self.path != "/webhooks/payments":
            self.send_error(404)
            return
        length = int(self.headers.get("Content-Length", "0"))
        raw = self.rfile.read(length)
        header_id = self.headers.get("X-Webhook-Id", "")
        ts = self.headers.get("X-Webhook-Timestamp", "")
        sig = self.headers.get("X-Webhook-Signature", "")
        try:
            ts_i = int(ts)
        except ValueError:
            self._reply(400, b"bad timestamp")
            return
        now = int(time.time())
        if ts_i < now - 300 or ts_i > now + 30:
            self._reply(401, b"timestamp window")
            return
        expected = "sha256=" + hmac.new(
            SECRET, f"{ts}.".encode() + raw, hashlib.sha256
        ).hexdigest()
        if not hmac.compare_digest(expected, sig or ""):
            self._reply(401, b"bad signature")
            return
        try:
            payload = json.loads(raw.decode("utf-8"))
        except json.JSONDecodeError:
            self._reply(400, b"bad json")
            return
        if payload.get("event_id") != header_id:
            self._reply(400, b"id mismatch")
            return
        order_id = payload["order_id"]
        sequence = int(payload["sequence"])
        amount = int(payload["amount_cents"])
        conn = db()
        try:
            conn.execute("BEGIN IMMEDIATE")
            row = conn.execute(
                "SELECT applied FROM events WHERE event_id = ?", (header_id,)
            ).fetchone()
            if row:
                conn.execute("COMMIT")
                self._reply(200, b"duplicate")
                return
            prior = conn.execute(
                "SELECT COUNT(*) FROM events WHERE order_id = ? AND sequence < ? AND applied = 1",
                (order_id, sequence),
            ).fetchone()[0]
            if prior != sequence - 1:
                conn.execute("ROLLBACK")
                self._reply(409, b"out of order")
                return
            conn.execute(
                "INSERT INTO events(event_id, order_id, sequence, amount_cents, applied) VALUES (?,?,?,?,1)",
                (header_id, order_id, sequence, amount),
            )
            conn.execute("COMMIT")
            self._reply(200, b"applied")
        except sqlite3.IntegrityError:
            conn.execute("ROLLBACK")
            self._reply(200, b"duplicate")
        finally:
            conn.close()

    def _reply(self, code, body):
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

Reviewers should notice three deliberate choices in that sketch before they treat it as a scoring baseline. HMAC comparison uses compare_digest on the full sha256= prefix rather than a parsed hex string with equality. The unique index turns a race between two inserts into a handled integrity error that still returns 2xx. Sequence 409 is reserved for gaps, not for duplicates, so the producer retries only when waiting can make progress.

Common failure modes the scorer records

Most rejected packets share a small set of implementation patterns rather than rare or exotic protocol mistakes. The list below is useful as a debrief document after the timed interview window closes. Interviewers should map each failing fixture to one pattern instead of arguing about framework taste. Agents and humans tend to land on the same eight bugs when the public prompt stays short.

1. The handler returns 200 from the web framework and writes SQLite on a background task that the tests never await.
2. The handler parses JSON first, then signs the re-serialized object, so key order changes break valid producer signatures.
3. The handler stores seen event ids in a process-local set that disappears on restart and admits a second capture.
4. The handler applies sequence 3 when sequences 1 and 2 are missing, then cannot repair the order total.
5. The handler returns 409 for an already applied duplicate, which traps the producer in an infinite retry loop.
6. The handler uses equality on hex digests, or compares after lowercasing only one side of a mixed-case header.
7. The handler accepts a timestamp as a float string and silently truncates, which widens the window beyond five minutes.
8. The handler locks per event_id in memory and still double-applies across two worker processes on one database file.

Local commands for the interviewer

Interviewers should run the hidden suite from a second checkout so candidates cannot read the replay schedule by accident. The commands below assume the candidate server is already bound to 127.0.0.1 on port 8080. A separate checkout also prevents an agent from grepping scorer.py during an open-book take-home. The interviewer should record HTTP status codes and the SQLite dump for the debrief notes.

export WEBHOOK_SECRET=test_secret_do_not_reuse
python scorer.py --base-url http://127.0.0.1:8080 --db ./webhook.db --cases hidden.yaml
sqlite3 ./webhook.db 'SELECT order_id, sequence, amount_cents, applied FROM events ORDER BY 1,2;'

A passing database after the reorder case contains sequences 1, 2, and 3 exactly once for ord_9k2. A failing database contains sequence 3 alone, or two rows that incorrectly share the same event_id. Interviewers should keep the YAML case file out of the candidate tarball and should rotate WEBHOOK_SECRET per session. The sqlite3 query is the fastest way to explain a failure during the live debrief without opening application logs.

Where hosted free model access fits this workflow

A local runner remains the source of truth for fixtures, HMAC secrets, and the hidden replay schedule used in scoring. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Teams without spare dedicated hardware can run the same public prompt against MonkeyCode free model access. The free server option is enough for that step after the local hidden tests already fail. No model name, quota, duration, or benchmark is claimed here beyond that free model access and a free server option exist.

The useful workflow is to paste the public prompt, collect a patch, then execute the hidden scorer on the interviewer's machine. Agents that only print a handler without a unique index fail the concurrent replay case in that second step. That split keeps HMAC secrets off the remote side and keeps the hidden rubric honest during scoring.

Limitations and who should not use this packet

This packet is a teaching filter for webhook idempotency, not a complete payments platform or a security audit. It does not cover rotating secrets, body size limits, replay caches with TTL, or multi-region conflict resolution. SQLite busy handling is enough for the take-home and is the wrong isolation story for a multi-primary production datastore. Reviewers should treat the sample server as a scoring aid rather than a template for public internet exposure.

Skip this assignment when the role does not own HTTP consumers or durable webhook processing at all. Skip it for junior screens that only need routing and JSON parsing, because the hidden suite will fail almost every submission. Skip it when the interviewer cannot run the scorer locally, because remote demos without the replay file prove almost nothing. The packet also wastes time when the team already has a production idempotency library with an enforced review checklist.

The same structure still works when the candidate is an AI agent rather than a person in an IDE. The public prompt stays short, and the hidden suite stays private during the timed interview window. The debrief then lists concrete failure modes instead of a vague complaint about prompt quality. Interviewers who want a next iteration can add signed timestamp rotation without changing the rubric shape.