DEV Community: Emma

Testing webhook retries deterministically: a fault sequence per Idempotency-Key

Emma — Fri, 18 Sep 2026 02:07:58 +0000

Two weeks ago I wrote about a webhook receiver that can be slow and flaky on purpose. A reader left a comment that changed how I think about it: "a 10% error rate tells me my sender retries. It doesn't tell me the retry is correct, and I can't put 10% in CI." He was right. The bug I actually care about is a specific one:

My service sends the webhook.
The receiver gets it, processes it, and the connection dies before my service reads the 200.
My service retries with the same idempotency key.
Does the receiver create a second order, or does it recognise the key and say "already done"?

No amount of random failure reproduces step 2 on demand. So the endpoint now has a third mode, and it's the one I'd use first.

A list of steps, walked per key

You give the endpoint a JSON list of steps. Every distinct Idempotency-Key (or whatever header you name) walks the list in order: the first call with a key gets step one, the second call with the same key gets step two, a new key starts again from step one.

{
  "key_header": "Idempotency-Key",
  "on_end": "repeat_last",
  "steps": [
    { "action": "drop" },
    { "action": "respond", "status": 200, "body": "{\"received\":true}" }
  ]
}

Three actions:

respond: a status, a body and an optional delay_ms, each defaulting to the endpoint's normal settings
drop: log the request, then cut the connection without delivering a body
timeout: hold the connection for delay_ms (up to 30 seconds), then cut it

The example above is the duplicate-side-effect test. First delivery: logged, connection cut. Retry with the same key: 200. If your sender did the right thing you see exactly two requests in the log with the same key, and one order in your database.

on_end decides what happens after the last step. The default repeats the last step forever, which is what you want for a correct sender: once it has its 200 it stops, and if it retries anyway it keeps getting 200. loop starts over from step one, for soak tests.

Some things I got wrong the first time

The key is hashed before it's stored. You pick the header, so the value might be an order number, an email, anything. The counter table only needs to tell keys apart, so it keeps a hash. You also can't pick Authorization or Cookie as the key header; the form refuses.

No key means a shared counter. If a sender doesn't send the header at all, every call shares one sequence. That's usually a bug in the sender, and the request log makes it visible.

Counters reset when you save the endpoint with a different sequence, and there's a "Reset sequence" button plus DELETE /v1/endpoints/{id}/sequence for the start of a CI job. Counters nobody has touched for 7 days are dropped.

Every response carries X-Supercrontab-Sequence-Step, so when a test fails you can see which step the sender was on.

The honest limit

The endpoints run on Cloudflare Workers. There I can cut a connection in the middle of the body, but not before the status line goes out. So drop looks like a truncated response: curl exits with code 18, Python raises IncompleteRead, Go's body read returns an error. It is not a TCP reset. For every HTTP client I've tried, that lands in the same "the request failed, retry it" path, which is the path under test. If your client treats a truncated 200 as success, that itself is a finding.

Auth still runs first. A call with a bad token gets 401 and does not consume a step, so a sender that's fumbling credentials can't accidentally eat the drop you were waiting to see.

What it costs

Nothing on the free plan: 3 endpoints, 30 requests a minute, 500 a day, up to 20 steps per sequence. A sequence with a timeout step holds a connection for up to 30 seconds, which counts as one request.

The API reference for it is at https://supercrontab.com/docs#sequence and the form is at https://supercrontab.com/webhook-tester. If there's a step type you're missing (a random pick between two steps came up already), say so below.

I wanted a webhook receiver that could be slow and flaky on purpose, so I made one

Emma — Thu, 17 Sep 2026 15:27:06 +0000

There are plenty of "paste this URL and see what arrives" webhook testers, and they're fine for the first half of the job: is my service sending the right headers and body? What none of them did for me was the second half: what does my service do when the receiver is having a bad day? Times out. Returns 500. Works four times out of five.

That second half is where the expensive bugs live. Duplicate orders because a retry fired after a slow 200. Alerts that never went off because nobody tested the 500 path. So I built a receiver that can misbehave on request.

The boring half: see what you sent

You create an endpoint (takes about a minute: https://supercrontab.com/webhook-tester), get a URL like https://yourslug.supercrontab.com/, point your service at it. Every request shows up in a log with the method, path, headers, body (up to 16 KB), the caller's IP and the status you answered with. Authorization, Cookie and API key headers are redacted in the log, on purpose, because I didn't want to run a service that stores other people's tokens.

Sub-paths and trailing slashes work, so /orders/42/ reaches the same endpoint and you see the path in the log.

The useful half: make it misbehave

Same form, a few more fields:

status code, so 202 for "accepted" or 500 when you want to see your alerting fire
delay in milliseconds, to find out whether your sender times out and what it does after
error rate, so one call in five fails, which is what a flaky partner actually looks like
auth: Basic, Bearer, an API key header, or an HMAC-SHA256 signature over the body

My default setup is "202, 3 seconds, 10% errors". If your sender gets through a day of that with correct retries and no duplicated side effects, you're in decent shape.

The HMAC one caught a real bug for me: I was signing the pretty-printed JSON and sending the compact one. The endpoint answered 401 and I stared at it for ten minutes before I understood why.

The third mode: a scripted fault sequence

A random error rate tells you whether your sender retries at all. It doesn't reproduce one specific bug on demand, and you can't put "10% of the time" in CI. A reader of this post asked for deterministic fault sequences keyed by an idempotency key, and that turned out to be the right shape, so it's in.

You give the endpoint a list of steps. Each Idempotency-Key (or any header you name) walks the steps in order: call one gets step one, call two gets step two. Three actions:

respond: a status and body, optionally after a delay
drop: log the request, then cut the connection before the body is delivered
timeout: hold for up to 30 seconds, then cut

The one I use most reproduces the classic duplicate side-effect bug: the first delivery is logged and the connection is cut, the retry with the same key gets 200.

{ "steps": [ { "action": "drop" }, { "action": "respond", "status": 200 } ] }

After the last step the endpoint repeats it, so a correct sender stays green; a new key starts again from step one. There's a "Reset sequence" button and a DELETE /v1/endpoints/{id}/sequence call for the start of a CI job.

One honest limit: the endpoints run on Cloudflare Workers, and there I can cut the connection mid-body but not before the status line. So drop looks like a truncated response (curl exit 18, Python IncompleteRead, Go body read error) rather than a reset. For most senders that's the same failure path.

Closing the loop

The same account has a cron scheduler, so you can schedule a job to call your endpoint every 15 minutes with a Bearer token and watch the job's run history and the endpoint's log agree with each other. I leave one running as a canary.

Why it needs an account

Anonymous receivers get used as drop boxes for stuff nobody should be storing. Tying every receiver to an account keeps the logs private, keeps the service clean, and lets you delete everything when you're done. The free plan has 3 endpoints, 30 requests a minute, 500 a day, and doesn't expire.

If you want to try only the response side first, without an account, there are 112 static demo endpoints (8 formats x 14 status codes) at https://supercrontab.com/mock, for example https://demo-json-503.supercrontab.com/.

Two things I still want: replaying a logged request to another URL, and exporting a log as a HAR. If you'd use either, tell me which first.

112 mock endpoints that return whatever HTTP status you ask for (no signup)

Emma — Thu, 17 Sep 2026 15:27:02 +0000

Last month I was writing a client for a partner API and wanted to check what happens when they return a 503. Simple question. The honest answer was "I don't know, I'd have to fake it", and faking it meant either monkeypatching fetch or spinning up an express server that returns 503, which I've done about forty times in my life and never kept.

So this time I kept it, and put it on a domain:

curl -i https://demo-json-503.supercrontab.com/

HTTP/2 503
content-type: application/json

{"ok":true,"id":42,"items":["alpha","beta","gamma"]}

The hostname is the whole API. demo-{format}-{status}.supercrontab.com, where format is one of json, xml, txt, csv, html, yaml, rss, js and status is one of 200, 201, 204, 400, 401, 403, 404, 409, 422, 429, 500, 502, 503, 504. That's 112 URLs. CORS is wide open so you can hit them from a browser tab too.

There's an index page with the sample body, curl and fetch snippets for each one, and a note about whether a client should retry that particular status: https://supercrontab.com/mock

Things I've actually used them for this week

Retry logic, mostly. Point the client at demo-json-503, watch it back off and retry. Then point it at demo-json-400 and make sure it doesn't retry, because a 400 won't get better on its own. The second case is where I've seen the most bugs in real code, including my own.

The CSV and XML ones are nice for parser tests. A tiny well-formed body, so you can have one integration test that touches the network and the rest can be unit tests with the same fixture.

And error screens. fetch('https://demo-json-429.supercrontab.com/') from the frontend, see the rate-limit banner, done. No backend changes, no feature flag.

If the static ones aren't enough

They're static on purpose, but sometimes you need a URL that returns your body, or is slow, or fails only some of the time. With a free account you get an endpoint at yourslug.supercrontab.com where you set the status, the content type, the body (up to 64 KB), a delay in ms, and an error rate. 10% error rate is a surprisingly good imitation of a flaky vendor.

You can also put auth on it (Basic, Bearer, an API key header, or HMAC over the body) if you want to check that your client actually sends credentials the way you think it does. And it logs every request it receives, headers and body, so it doubles as a webhook receiver.

Free plan limits: 3 endpoints, 30 requests a minute, 500 a day. Enough for tests, useless for production traffic, which is the idea.

httpbin?

I still use httpbin, it's great. It just doesn't do non-JSON formats, and it doesn't give you a URL with your own body and delay that a teammate or a CI job can hit tomorrow. That's the gap.

If there's a status code or format you need and it's not there, say so in the comments, adding one is a five-line change.