DEV Community

Jay_Stride
Jay_Stride

Posted on

Sequential Retries Pass, Concurrent Duplicates Fail: Black-Box Testing for Idempotency-Key Races

#idempotency #api #testing #go

IdemCheck

Sequential retries can pass while concurrent requests sharing one Idempotency-Key still produce more than one logical result. A retry loop sends a request, waits for the response, then sends the next one, so the second request finds the stored result. A burst does not wait, and both requests do the work.

Sequential retries: PASS
Concurrent duplicates: FAIL
Same Idempotency-Key ??multiple logical results
Enter fullscreen mode Exit fullscreen mode

The usual implementation looks like this:

check key
  ??do work
  ??create resource
  ??store result
Enter fullscreen mode Exit fullscreen mode

Two concurrent requests pass the check before either stores its result. Both create a resource, and the caller ends up with two order_id values for one intended operation.

Two columns: sequential retries return order_id 812 twice and pass; three concurrent requests return 812, 810 and 817 and fail

What IdemCheck does

IdemCheck is a small Go CLI for black-box HTTP testing: it sends requests and compares responses, which is all you have when the API belongs to a partner or vendor. It runs same-key concurrency on purpose, then replays the original request and checks whether everything converged on one observable result. The comparison covers status codes and JSON structure, and the report names the field that diverged, for example $.order_id.

BURST ??SETTLE ??REPLAY ??VERDICT
Enter fullscreen mode Exit fullscreen mode

Flow: BURST of same-key requests, SETTLE wait, REPLAY with the same key, VERDICT asking whether results converged

BURST releases ten requests sharing one key, SETTLE waits 250ms, REPLAY sends the original request again, and VERDICT reports whether the responses converged. Verdicts are PASS, FAIL, ERROR, and INCONCLUSIVE, mapped to exit codes 0, 1, 2, 3, so a CI job can fail on a real violation and tell it apart from a broken run.

Ten concurrent requests on one key is the default. --trials repeats the burst with a fresh key when the race window is narrow, --concurrency raises the count, and header credentials are redacted from the report.

This run uses the two demo servers in the repository: the unsafe one checks the key, sleeps, inserts, then stores it, and the safe one locks per key.

Demo: against the unsafe API the sequential retries pass and the concurrent burst fails with differing order IDs, then the safe API passes

Install 

go install github.com/hyukvoid/idemcheck/cmd/idemcheck@v1.0.0
Enter fullscreen mode Exit fullscreen mode

Then point it at an environment you're authorized to test:

idemcheck test \
  --url https://staging.example.com/orders \
  --body-file request.json \
  -H "Authorization: Bearer $TOKEN" \
  --allow-remote
Enter fullscreen mode Exit fullscreen mode

Loopback targets work without flags. Remote targets need --allow-remote, which prints a warning before it sends duplicates.

curl, k6, and vegeta can send these requests and report latency. IdemCheck reads the responses under one key and decides whether they agree.

Honest limits

A one-off script can test a simple case. IdemCheck exists to package the same-key burst, replay, response comparison, verdicts, and CI behavior together. The checks are written once and run on every endpoint.

A PASS means no violation was observed through HTTP. It does not prove hidden database writes, messages, emails, payment captures, or other internal side effects happened exactly once.

Feedback

I'd like to hear from engineers working with payments, orders, webhooks, background jobs, and retry-heavy APIs. Does this match the races you have actually hit, and what does a run like this miss on your endpoints? If a verdict disagrees with what you see in production, that gap is the useful bug report. Open an issue or a PR:

https://github.com/hyukvoid/idemcheck

Top comments (0)