The HMAC Matched in the REPL. The Framework Had Already Parsed the Body.

Have you ever watched a webhook verifier succeed in a scratch buffer and then fail against a real sender? I just spent forty-eight hours inside that gap, and the gap was not cryptography at all. It was the distance between bytes on the wire and the object my handler hashed. Why do we keep hashing a reconstructed dictionary when the sender signed a specific sequence of octets?

This write-up is field notes, not a vendor postmortem and not a benchmark. I am recording what I tried, what broke, and what I would run again. The only product detail I am treating as given is that free model access and a free server option exist for a coding workspace. Everything else below is a stdlib reproduction you can run without trusting my laptop.

Hour 0: staging returned 401, the REPL returned True

I had a tiny callback that checked X-Signature-256 against a shared secret. Staging rejected every ping, yet pasting the JSON into a Python shell printed True. Same secret, same keys, same pretty payload, two answers. That is the kind of split that makes you doubt HMAC itself, which is almost never the actual problem.

The handler looked innocent. It parsed JSON, dumped JSON, and hashed the dump. I even used hmac.compare_digest, so I felt briefly responsible. Then I asked the obvious question out loud: what if the sender never saw my dump?

import hashlib
import hmac
import json
import os

SECRET = os.environ["WEBHOOK_SECRET"].encode("utf-8")

def verify_parsed(payload: dict, header: str) -> bool:
    body = json.dumps(payload, separators=(",", ":")).encode("utf-8")
    digest = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(digest, header.removeprefix("sha256="))

That function is a trap with good posture. Compact separators make it look canonical. Insertion-ordered dicts in modern CPython make it look stable. None of that matters if the producer signed a trailing newline, a space after a colon, or a UTF-8 escape you will never emit.

Hours 1–6: I let a clean box take the first hit

Local Flask reloaders, browser proxies, and pretty printers have opinions about bodies. I wanted a process that would not “help.” Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and free server option as a clean room so my laptop's debug middleware could not rewrite the socket.

The models drafted verifiers quickly. That speed is useful and also how you get a green test that hashes the model's own json.dumps. I banned the session from editing files under testdata/. Golden bytes are evidence. Generated assertions are not.

What I actually ran on the box looked like this.

python3 -V
printf '%s' '{"ok":true}' | xxd
printf '%s\n' '{"ok":true}' | xxd

Those two xxd dumps already disagree by one 0a. If your secret is test-secret, the SHA-256 HMAC hex of the no-newline form is not the HMAC of the newline form. Ask yourself which one your framework stored after json.loads.

Hours 6–18: three “fixes” that made tests greener and production worse

I kept a punch list because the session loved deleting symptoms.

1. Sort the keys, then dump again. Real senders do not sort unless their spec says so.
2. Switch indent=None to indent=2 so the fixture is readable. Readability is not a signature scheme.
3. Hash str(payload) because it printed nicely in the REPL. It is not UTF-8 JSON, and it is not stable across types.

Each patch produced a unit test that passed against a helper the same patch had just defined. That is a closed loop, not verification. Have you noticed how often an agent names the helper canonical_json right after inventing the canonical form?

What broke in the actual route was ruder than style. A stdlib demo made it obvious.

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        raw = self.rfile.read(length)
        parsed = json.loads(raw)
        header = self.headers.get("X-Signature-256", "")
        ok_raw = verify_raw(raw, header)
        ok_parsed = verify_parsed(parsed, header)
        self.send_response(200 if ok_raw else 401)
        self.end_headers()
        msg = f"raw={ok_raw} parsed={ok_parsed} n={len(raw)}\n"
        self.wfile.write(msg.encode("ascii"))

verify_raw hashes raw and never asks JSON for permission. verify_parsed rebuilds a body the sender did not sign. Curl against that process is the whole experiment.

SECRET='test-secret'
BODY='{"ok":true}'
SIG=$(printf '%s' "$BODY" | python3 -c 'import sys,os,hmac,hashlib; s=os.environ["SECRET"].encode(); b=sys.stdin.buffer.read(); print(hmac.new(s,b,hashlib.sha256).hexdigest())')
SECRET="$SECRET" python3 server.py &
curl -sS -D - -H "X-Signature-256: sha256=${SIG}" --data-binary "$BODY" http://127.0.0.1:8080/
curl -sS -D - -H "X-Signature-256: sha256=${SIG}" --data "$BODY" http://127.0.0.1:8080/

--data may append a newline depending on how you quote it. --data-binary does not try to be helpful. One of those curls returned raw=True parsed=False. That single line paid the rent for the next night.

Hours 18–30: the header was not a naked hex digest either

Even after I stopped dumping dictionaries, the REPL still lied once. I had compared the digest to the full header string, prefix included. compare_digest then compared abc... with sha256=abc... and returned False with a straight face. No exception, no warning, just a boolean that looks like cryptography.

The other lie was character encoding on the secret file. I had loaded the secret with open(path).read().encode() and sometimes inherited a trailing newline from the file. The sender used the trimmed secret. Do you hash the password you meant, or the password plus \n that echo left behind?

def load_secret(path: str) -> bytes:
    text = open(path, encoding="utf-8").read()
    return text.strip("\n").encode("utf-8")

def verify_raw(raw_body: bytes, header: str) -> bool:
    if not header.startswith("sha256="):
        return False
    given = header[7:]
    if len(given) != 64:
        return False
    digest = hmac.new(SECRET, raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(digest, given)

I still refuse to parse the body before this function returns. Parsing is a business concern. Signature check is a bytes concern. Mixing them is how a pretty-printer becomes an authentication bypass in your test suite, even when production stays closed.

Hours 30–48: freeze the wire, then freeze the test

I wrote golden files as raw bytes and a pytest that is not allowed to reconstruct them. If you cannot explain a failure with xxd testdata/ping.bin, you do not have a signature bug yet.

# test_webhook_raw.py
from pathlib import Path
import hmac
import hashlib
import os

GOLD = Path(__file__).parent / "testdata"
SECRET = b"test-secret"

def sign(raw: bytes) -> str:
    return "sha256=" + hmac.new(SECRET, raw, hashlib.sha256).hexdigest()

def test_exact_bytes_match():
    raw = (GOLD / "ping.bin").read_bytes()
    assert verify_raw(raw, sign(raw)) is True

def test_trailing_newline_is_a_different_message():
    raw = (GOLD / "ping.bin").read_bytes()
    assert raw.endswith(b"\n") is False
    assert verify_raw(raw + b"\n", sign(raw)) is False

def test_rejson_is_not_evidence():
    raw = (GOLD / "ping.bin").read_bytes()
    import json
    rebuilt = json.dumps(json.loads(raw), separators=(",", ":")).encode()
    # This assertion documents the trap. It must stay False on this fixture.
    assert rebuilt != raw or verify_raw(rebuilt, sign(raw))

Create the fixture once, by hand, from a captured request. I used curl -sS -o testdata/ping.bin against a throwaway echo server, then signed that file. After that, models could propose code. They could not rotate the bytes to make a hash line up.

Decision table I wish I had taped to the monitor

• Need a clean listener without laptop middleware? Use a throwaway remote process and curl --data-binary.
• Need candidate implementations fast? Free model access is fine if golden files are read-only.
• Sender signed raw octets? Hash request body bytes, never json.dumps of a parsed object.
• Header has a scheme prefix? Strip it in one place and reject unknown schemes.
• Secret came from a file? Strip a single trailing newline, then encode UTF-8 explicitly.
• Agent offers sort_keys=True for “stability”? Only accept it if the published spec requires that exact canonicalization.
• Tests import the verifier and also rebuild the body? Those tests are hashing themselves.

Limitations, and who should not copy this loop

This workflow assumes a shared-secret HMAC and a header you control in tests. It does not cover rotating key ids, timestamp windows, or asymmetric signatures. I am not claiming timing numbers, model names, hardware, or quotas for the free server, because I did not measure those here.

Do not use a public free box for live customer secrets, production key material, or payloads that contain personal data. A clean room is not a vault. Copy synthetic goldens, not the incident. If your provider already documents a canonical string to sign, follow that document instead of inventing separators=(",", ":") because a REPL looked tidy.

Also skip this approach if you cannot pin the exact captured body. Without ping.bin, you are negotiating with folklore. I would not run forty-eight hours of model retries on a payload I cannot xxd.

What I would repeat tomorrow

I would capture one real body first, even if it is three bytes of JSON. I would sign that file with a tiny Python one-liner and store both the bytes and the header in git. I would point the handler at raw_body only. I would keep reconstruction tests that are expected to fail, so a future patch cannot “simplify” them back into json.dumps.

Would I still use a free remote process? Yes, as a listener that does not share my shell history or my HTTP proxy. Would I let the model edit testdata/? No. The useful loop is boring on purpose: freeze the wire, hash the wire, then discuss code. Everything else is a REPL that wants to be helpful, and helpful is how my 401 lasted two days.