I Chased a Webhook HMAC for 48 Hours. json.dumps Was Not the Payload.

Have you ever watched a webhook retry for two days while every local check printed MATCHED? I had a provider portal full of green deliveries and a worker that still rejected every signature. The dashboard claimed success, my logs claimed invalid_hmac, and I could not tell which side was lying.

This is a field notebook, not a victory lap, because the useful part is the dead ends. I will walk through what I tried, what broke, and the tiny replay I now keep in git. If you verify callbacks that sign the raw HTTP body, you have probably stepped on this rake too.

Hour 0–6: Why did two hex strings look identical?

I started where everyone starts, by printing both digests and squinting at the terminal. They looked identical until I stopped trusting my eyes and counted every character twice. One digest was sixty-four hex chars; the other was sixty-four hex chars plus a trailing newline.

That newline was a gift, because it was not the real bug at all. I stripped it, reran the worker, and still got invalid_hmac from the same captured payload. Have you noticed how a tiny logging bug can hide a bigger one for an entire afternoon?

I wrote this down in the notebook before I changed another line of code:

• Provider header: X-Signature: sha256=<hex>
• My check: hmac.new(secret, body, sha256).hexdigest()
• Local fixture: a Python dict I passed through json.dumps

The fixture was the landmine sitting in plain sight. I just did not know it yet, which is the embarrassing part. Hex in a log is not evidence. Length plus repr() is evidence, and I did not take it.

Hour 6–18: I blamed the secret, then the header format

I rotated the shared secret twice, because that is the move that feels responsible under pressure. Both sides showed the same prefix in the admin UI, so I assumed the bytes matched. Did they actually match in the running process, or only in the dotenv file I kept rereading?

Then I chased header formats like they were the whole story. Some providers send raw hex. Some send sha256=. Some send a comma-separated list of t= and v1= pieces. I wrote three parsers and a pile of unit tests that all passed against fixtures I had authored myself.

The parser I kept

Here is the parser I kept, labeled as a worked example rather than production gospel for every gateway.

def parse_signature_header(value: str) -> bytes:
    """Extract the first sha256 hex digest from a provider signature header."""
    value = value.strip()
    if value.startswith("sha256="):
        value = value[len("sha256="):]
    parts = {}
    if "=" in value and "," in value:
        for piece in value.split(","):
            if "=" in piece:
                key, raw = piece.split("=", 1)
                parts[key.strip()] = raw.strip()
        value = parts.get("v1", value)
    return bytes.fromhex(value)

The tests were green. The live traffic was not. What does a green unit test prove when the fixture never left my laptop?

I also dumped environment variables, container args, and the process list, looking for a second secret. There was no second secret. There was a second body, invented by my own serializer after the HTTP server had already parsed JSON.

Hour 18–36: A generated signer made the dumps prettier

Around hour eighteen I asked a coding model to write a verifier from the provider's public docs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access for that first draft of the signature verifier. Then I ran the result on their free server option so laptop packages could not contaminate the replay.

The draft looked confident, and it compiled, which is a dangerous combination at hour twenty. It also rebuilt the payload with json.dumps, which is how you invent a body the provider never signed in the first place.

# Worked example: the helper that wasted a day.
import hashlib
import hmac
import json

def sign_rebuilt(secret: str, payload: dict) -> str:
    serialized = json.dumps(payload)  # default separators insert spaces
    return hmac.new(
        secret.encode("utf-8"),
        serialized.encode("utf-8"),
        hashlib.sha256,
    ).hexdigest()

Default json.dumps emits {"event": "ping"} with a space after the colon, every single time. Many providers sign {"event":"ping"} with no extra whitespace, because their examples were copied from a Go encoder. Key order is another trap, because a Go map or a Ruby hash will not preserve the order your Python dict happened to have on Tuesday.

I also lost the original bytes in the web framework, which still makes me wince. The handler called await request.json(), which parses the body and then hides the exact octet stream. You cannot verify a signature over a body you no longer have. Why do we keep doing that in README demos?

The clean box finally disagreed with my laptop

On the clean server the failure reproduced immediately, which was the first honest signal all day. Locally I had a pytest fixture that dumped dicts into pretty JSON. Remotely I replayed a raw HTTP capture from the gateway access log. Only one of those objects is the payload the HMAC actually covered.

Would I skip the model next time? No. I would skip trusting its serializer. The free model access was useful for the header parser. The free server option was useful because it did not include my local conftest helpers. Neither one read the provider's canonicalization rules for me, and I should not have expected that shortcut.

Hour 36–48: Capture raw bytes, then compare digest bytes

The fix is boring, which is why I resisted it until the second night. Store the secret as bytes. Read the raw body once. Hash those bytes. Compare with hmac.compare_digest, not == on strings you have pretty-printed into chat.

import hashlib
import hmac

def verify_raw_body(secret: bytes, raw_body: bytes, header_value: str) -> bool:
    """Verify a hex HMAC-SHA256 over the exact request body bytes."""
    try:
        expected = parse_signature_header(header_value)
    except ValueError:
        return False
    digest = hmac.new(secret, raw_body, hashlib.sha256).digest()
    if len(expected) != len(digest):
        return False
    return hmac.compare_digest(digest, expected)

If a provider documents canonical JSON instead of raw bytes, you still should not call default json.dumps. You need their rules, written down in a comment, then locked in a test that fails when spaces appear. Otherwise the next generated helper will clean up the serializer and break production at lunch.

Decision table I wish I had at hour three

Provider rule	Wrong Python reflex	What to lock in a test
Sign raw body bytes	json.dumps(await request.json())	Replay a captured raw_body fixture
Canonical JSON, sorted keys	Default dumps with spaces	separators=(",", ":"), sort_keys=True
Hex in a sha256= header	Compare the full header string	Parse, then compare_digest
Timestamp in t=	Ignore t= or trust it blindly	Reject stale t= with documented skew
Multiple v1 signatures	Require the first one only	Accept if any advertised digest matches

I now refuse to merge a verifier without this replay sitting next to it. It is not elegant. It is short enough that I cannot hide, which is the whole point of a field note.

A reproducible replay you can run

Save this as test_webhook_hmac.py and run python -m pytest test_webhook_hmac.py -q. I am labeling it a lab fixture, not a claim about any vendor's current production behavior or about traffic volumes I did not measure.

import hashlib
import hmac
import json

SECRET = b"test-secret-do-not-use"

def sign(raw_body: bytes) -> str:
    return hmac.new(SECRET, raw_body, hashlib.sha256).hexdigest()

def test_raw_body_matches_provider_bytes():
    raw = b'{"event":"ping","id":"abc"}'
    header = "sha256=" + sign(raw)
    digest = bytes.fromhex(header.split("=", 1)[1])
    actual = hmac.new(SECRET, raw, hashlib.sha256).digest()
    assert hmac.compare_digest(actual, digest)

def test_default_json_dumps_is_not_the_payload():
    raw = b'{"event":"ping","id":"abc"}'
    rebuilt = json.dumps({"event": "ping", "id": "abc"}).encode("utf-8")
    assert rebuilt != raw
    assert sign(rebuilt) != sign(raw)

def test_sorted_compact_dumps_can_match_if_that_is_the_contract():
    payload = {"id": "abc", "event": "ping"}
    canonical = json.dumps(
        payload, separators=(",", ":"), sort_keys=True
    ).encode("utf-8")
    assert canonical == b'{"event":"ping","id":"abc"}'
    assert sign(canonical) == sign(b'{"event":"ping","id":"abc"}')

What should you try, in order, the next time the hex looks close enough to ignore?

1. Dump len(raw_body) and the first sixty bytes as repr(raw_body[:60]).
2. Dump the header with repr(), so a trailing newline or carriage return cannot hide.
3. Never log secrets, and never log full signatures in shared channels.
4. Compare json.dumps(parsed).encode() against raw_body; they will differ.
5. Move the replay to a machine that does not have your local fixtures installed.

That fifth step is why a clean remote box mattered more than another secret rotation. My laptop had a helper that re-serialized every fixture as pretty JSON, and I had stopped seeing the helper because it was mine.

What broke, and what I would repeat

What broke was not HMAC as an algorithm, and it was not the provider's clock either. HMAC did exactly what the textbooks said it would do with the bytes I handed it. What broke was my belief that a Python dict is a wire payload, plus a generated helper that made that belief look tidy and shippable.

I would repeat the clean-box replay before I rotate another secret in a hurry. I would repeat golden files of raw request bodies, checked in next to the verifier, with repr() in the assertion message. I would repeat asking a model for a first draft, then deleting the json.dumps line before I run anything against live traffic.

I would not repeat pretty-printing signatures in logs that other people can search later. I would not repeat unit tests that sign the same dumps they later verify, because that is a hall of mirrors. I would not repeat calling request.json() before request.body() in a framework that caches the stream and then pretends you still have it.

Limitations, and who should not copy this blindly

This notebook is a debugging workflow for HMAC-SHA256 over HTTP bodies, written after one long incident. It is not a full webhook security design, and it should not be copy-pasted into a payments service without review. I did not cover key rotation, IP allowlists, replay windows beyond a simple t= check, or algorithm agility when a vendor advertises several digest versions.

Do not use compare_digest as theater while you still rebuild the body from a parsed dict. Do not treat a coding model as a source of canonicalization rules, even when the draft looks complete. Docs win. Captured bytes win. The model is a draft intern who has never seen your gateway, your proxy, or your pretty printer.

Skip this approach if you cannot read the raw socket body in your stack without a plugin. Skip it if a compliance team must approve cryptographic code and you need a reviewed library, not a gist from a field notebook. Skip it if the provider signs a constructed string like timestamp + "." + body and you have not copied that exact concatenation into a failing test first.

Proxies that re-encode JSON will also break raw-body signatures, and that is outside this notebook. Gzip at the edge, UTF-8 BOM insertion, and form-versus-JSON confusion are sibling bugs with the same symptom. If your capture and the provider's capture disagree on length, stop hashing and start comparing bytes with hexdump.