Testing webhook retries deterministically: a fault sequence per Idempotency-Key

Two weeks ago I wrote about a webhook receiver that can be slow and flaky on purpose. A reader left a comment that changed how I think about it: "a 10% error rate tells me my sender retries. It doesn't tell me the retry is correct, and I can't put 10% in CI." He was right. The bug I actually care about is a specific one:

1. My service sends the webhook.
2. The receiver gets it, processes it, and the connection dies before my service reads the 200.
3. My service retries with the same idempotency key.
4. Does the receiver create a second order, or does it recognise the key and say "already done"?

No amount of random failure reproduces step 2 on demand. So the endpoint now has a third mode, and it's the one I'd use first.

A list of steps, walked per key

You give the endpoint a JSON list of steps. Every distinct Idempotency-Key (or whatever header you name) walks the list in order: the first call with a key gets step one, the second call with the same key gets step two, a new key starts again from step one.

{
  "key_header": "Idempotency-Key",
  "on_end": "repeat_last",
  "steps": [
    { "action": "drop" },
    { "action": "respond", "status": 200, "body": "{\"received\":true}" }
  ]
}

Three actions:

• respond: a status, a body and an optional delay_ms, each defaulting to the endpoint's normal settings
• drop: log the request, then cut the connection without delivering a body
• timeout: hold the connection for delay_ms (up to 30 seconds), then cut it

The example above is the duplicate-side-effect test. First delivery: logged, connection cut. Retry with the same key: 200. If your sender did the right thing you see exactly two requests in the log with the same key, and one order in your database.

on_end decides what happens after the last step. The default repeats the last step forever, which is what you want for a correct sender: once it has its 200 it stops, and if it retries anyway it keeps getting 200. loop starts over from step one, for soak tests.

Some things I got wrong the first time

The key is hashed before it's stored. You pick the header, so the value might be an order number, an email, anything. The counter table only needs to tell keys apart, so it keeps a hash. You also can't pick Authorization or Cookie as the key header; the form refuses.

No key means a shared counter. If a sender doesn't send the header at all, every call shares one sequence. That's usually a bug in the sender, and the request log makes it visible.

Counters reset when you save the endpoint with a different sequence, and there's a "Reset sequence" button plus DELETE /v1/endpoints/{id}/sequence for the start of a CI job. Counters nobody has touched for 7 days are dropped.

Every response carries X-Supercrontab-Sequence-Step, so when a test fails you can see which step the sender was on.

The honest limit

The endpoints run on Cloudflare Workers. There I can cut a connection in the middle of the body, but not before the status line goes out. So drop looks like a truncated response: curl exits with code 18, Python raises IncompleteRead, Go's body read returns an error. It is not a TCP reset. For every HTTP client I've tried, that lands in the same "the request failed, retry it" path, which is the path under test. If your client treats a truncated 200 as success, that itself is a finding.

Auth still runs first. A call with a bad token gets 401 and does not consume a step, so a sender that's fumbling credentials can't accidentally eat the drop you were waiting to see.

What it costs

Nothing on the free plan: 3 endpoints, 30 requests a minute, 500 a day, up to 20 steps per sequence. A sequence with a timeout step holds a connection for up to 30 seconds, which counts as one request.

The API reference for it is at https://supercrontab.com/docs#sequence and the form is at https://supercrontab.com/webhook-tester. If there's a step type you're missing (a random pick between two steps came up already), say so below.