We wired up a new billing webhook last sprint. Local tests passed. Postman collection green. Merged on Friday.
Monday morning staging logged hundreds of 401 invalid signature from the provider's retry queue. The handler was maybe fifty lines. The secret lived in an env var copied straight from the vendor dashboard.
I replayed the same payload in Postman with the same X-Signature header. Verification passed. Staging failed on requests that looked identical.
The gap wasn't the HMAC math on paper. It was which bytes we hashed.
Our API gateway parsed JSON before the route handler ran. By the time we read req.body, it was a JavaScript object serialized again — keys in a different order, sometimes a trailing newline gone. The provider had signed the raw request body from the wire. We were hashing a re-exported string.
Everything in logs looked fine: one-line JSON, correct algorithm in the header, secret without obvious typos. That's what made it annoying. I spent an hour convinced the vendor had rotated keys without telling us.
What I check now on webhook PRs:
Raw body first. If the framework touches JSON before your verifier runs, find where to hook the unparsed buffer. Signature schemes don't care about your ORM.
Secret encoding. Dashboards often show base64-looking strings. Is the env var the decoded key bytes, or the base64 text itself? Wrong layer and HMAC still "works" in a scratch script — with the wrong input.
One failing payload, viewed cleanly. I copy the exact body and signature into separate tabs so I'm not squinting at escaped quotes wrapped in a narrow terminal.
Browser pages I keep for these checks — one job each, local processing, no account for a two-minute look:
https://devcove.dev/en/tools/json-formatter/
https://devcove.dev/en/tools/base64/
https://devcove.dev/en/tools/hash-generator/
I maintain a small browser toolkit called DevCove for exactly these chores when I'm already in DevTools on another tab. curl --data-binary and a server-side replay fixture still win for the real fix; this is for ruling out "wrong bytes" vs "wrong secret" before you open a ticket with the vendor.
Half the outcome was code: verify on raw body, add one canned replay test. The other half was stopping the assumption that Postman parity means production parity.
If you maintain signed webhooks — where do you usually lose the plot? Middleware reordering the body, secret encoding, or something else entirely?
Top comments (0)