Short answer: verify the signature against the exact raw request bytes, and put the route that receives the webhook before any JSON parser. Keep a bounded copy of those bytes for the leaked-key drill, then record the verification decision and key version in an audit log. Re-serializing req.body after Express parses it is too late; whitespace, key order, and number formatting can all change.
This matters in a marketplace because a forged “payment captured” event can release a seller payout. During a leaked-key drill, the useful question is not only “did the request fail?” It is “can we prove which bytes were checked, with which key, and who changed the rule?”
What the deployment changed
The usual regression is a middleware order change. A global express.json() runs first, turns the stream into an object, and leaves the verifier with no canonical byte sequence. The application still sees valid fields, so health checks pass while every HMAC comparison fails in production.
There are two independent checks. First, parse the signature header and reject an old timestamp or an unknown key id. Second, compute HMAC over the untouched bytes and compare digests in constant time. A valid JSON object is not evidence that its original wire representation is available.
One short rule: bytes first.
For an Express service, isolate the webhook route and capture the body there. In Python, the same boundary looks like this (the control flow is portable to JavaScript):
import hashlib
import hmac
import time
def verify_webhook(raw_body: bytes, header: str, secrets: dict[str, bytes], now: int) -> str:
# Header format: t=unix_seconds,v1=hex_digest,k=key_version
fields = dict(item.split("=", 1) for item in header.split(","))
timestamp = int(fields["t"])
key_version = fields["k"]
if abs(now - timestamp) > 300:
raise ValueError("stale webhook")
secret = secrets.get(key_version)
if secret is None:
raise ValueError("unknown key version")
signed = f"{timestamp}.".encode("ascii") + raw_body
expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
if not hmac.compare_digest(expected, fields["v1"]):
raise ValueError("invalid signature")
return key_version
In Express, the equivalent implementation should use the parser’s verify hook to retain buf, or mount a route-specific raw parser before the global parser. Do not call JSON.stringify(req.body) and hope it reproduces the sender’s bytes. It usually will not.
How should Express preserve the body for a webhook signature check after deploy?
Mounting order is the fix that survives a redeploy. A minimal layout is:
from flask import Flask, request
app = Flask(__name__)
@app.post("/webhooks/marketplace")
def marketplace_webhook():
raw = request.get_data(cache=True)
# Verify raw before accessing request.get_json().
return {"accepted": True}
The snippet uses a generic Python HTTP boundary to make the invariant visible: read and retain bytes, verify, then parse. In Node, configure express.raw({ type: "application/json" }) on this route and place it before express.json(). If a framework adapter has already consumed the stream, change the adapter configuration or add an ingress that preserves bytes; a later middleware cannot reconstruct them.
Test the deployed route with a payload containing escaped Unicode, 1.0, duplicate-looking whitespace, and a changed property order. The verifier should accept the exact fixture and reject a semantically equivalent re-serialization. I once spent an afternoon comparing parsed objects that were equal while their byte strings differed by one newline. The 401s were correct; the test was wrong.
A leaked-key drill is an audit exercise, not a retry exercise
When a signing key leaks, rotate to a new version, keep the old version only for the documented overlap window, and mark every verification with the key version used. The drill should produce an evidence trail:
| Evidence | Why it matters | Retention choice |
|---|---|---|
| Request id, timestamp, digest result | Reconstructs the decision | Keep for the incident and review period |
| Key version, rotation actor, policy revision | Proves access control changed | Keep with the change record |
| Raw body hash, not the full payload by default | Correlates bytes without copying customer data | Keep longer than payload content |
| Replay outcome and idempotency key | Shows whether a captured event was applied twice | Keep through settlement disputes |
Store secrets in a managed secret store with least-privilege reads and rotation procedures. OWASP recommends inventory, access control, rotation, and monitoring as one lifecycle; a signature check that cannot be tied to those records is hard to defend during an incident.
Retention has a cost. Keeping every raw payload forever expands privacy exposure and storage bills, so I retain a cryptographic hash and metadata after the short forensic window. The catch is that a hash cannot answer a later question about a malformed field; for high-value payouts, retain encrypted payloads under a separate, time-limited policy.
Failure modes that look like crypto failures
Clock skew can invalidate an otherwise correct digest when the timestamp tolerance is too tight. Header parsing can select the wrong key version during rotation. A proxy can transparently decompress or transcode a body before the application sees it. None of these should be “fixed” by disabling verification.
Replay protection belongs beside signature verification. Require a bounded timestamp, persist an idempotency key, and make the settlement update transactional. Return a fast 2xx only after the event is durably queued or applied; otherwise the sender’s retry can race the first attempt. Log reasons such as stale, unknown_key, and digest_mismatch separately, without logging the secret or full authorization header.
Use negative tests as a release gate: one byte changed, one header removed, a reused idempotency key, and a timestamp six minutes old. Your mileage may vary on the exact tolerance because provider clocks and delivery latency differ; choose it from measured latency and document the decision.
Choosing an implementation boundary
A route-local raw parser is the least complex option for a small Express service. A framework-level capture hook is preferable when many signed routes share policy. An edge gateway can verify centrally, but then application teams must still receive an audit event that includes the key version and body hash. Self-hosting the verifier gives control over retention and network placement, while a hosted gateway can reduce operational work; neither removes the need to test parser order after every deploy.
This approach is not suitable when a downstream service needs the original payload but the gateway discards it, or when legal retention rules require field-level deletion that an encrypted blob cannot provide. In those cases, keep verification at the service that owns the data and pass a signed, minimal event downstream. Stick with a simpler route-local design when there are only one or two webhook types; the extra gateway hop is harder to audit than it is to operate.
References
- https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
- https://expressjs.com/en/api.html#express.raw
- https://nodejs.org/api/crypto.html#crypto_timingSafeEqual_a_b
- https://www.rfc-editor.org/rfc/rfc2104
Top comments (0)