Building a Service for Failures You Can't See

ALATPay, my first SDK, was about getting a thing to work. The feedback loop was
tight: call the API, get a 401 or a 200, fix the code, try again. Every failure
announced itself.

Webhook Relay was the opposite. It's a service built entirely around failures
that don't announce themselves. A webhook that was sent and didn't arrive leaves
no trace on either side. A webhook that arrived twice looks identical to one
that arrived once. A webhook with a forged signature looks identical to a
legitimate one if you don't verify.

Building for invisible failures is a different kind of engineering than building
for visible ones. This is what I learned doing it.

---

The problem nobody thinks about until money goes missing

If you accept payments — via Paystack, ALATPay, Flutterwave, anything — you
eventually have to handle webhooks. A webhook is a message the payment provider
sends to your server saying "someone paid." Your server receives it and marks
the order as paid.

The first version of this is trivial. You write an endpoint, you receive JSON,
you do something with it. It works in testing and for the first week.

Then the failure modes start.

The server was down when the webhook was sent. The provider tried, failed,
retried three times, gave up. The customer paid. Your server never knew. The
order sits unfulfilled.

The same webhook arrived twice. The provider didn't get your 200 response
in time (network blip, or your server was slow), so it retried. Now you've
marked the order as paid twice, sent the customer two confirmation emails, and
possibly refunded them once by accident.

Someone sent a fake webhook. There's a URL on your site that says "hit this
to mark an order as paid." It has no authentication. Anyone who knows the
pattern can hit it for any order ID and get free stuff.

The webhook arrived, your server crashed processing it, and nothing
happened. No error message, no retry, no record. The webhook is gone. The
customer's money is somewhere in a bank's clearing system.

None of these announce themselves. The customer's payment went through. From
the customer's side, everything worked. From your side, it didn't, but you
don't know that until they call you and ask why they didn't get their thing.

Webhook Relay exists to make these failures visible and recoverable. Which
means every design decision in it is about a failure mode that, without the
design, would be silent.

---

The three decisions that shaped everything

Decision 1: store the webhook before doing anything with it.

The naive version is: receive webhook, process webhook, return 200. That's what
most tutorials show. It's also the version that loses money.

The version I built stores the webhook first, returns 200 immediately, and
processes it in the background. The "processing" could fail; the storage
couldn't. So even if processing fails, the webhook isn't lost — it's sitting in
the database, waiting for a retry.

This added about 200 lines of code (the worker, the retry queue, the status
state machine) and it's the whole reason the service exists. The naive version
would have been a five-file FastAPI app. This version has twelve.

The question I had to answer for myself was: "If I had to choose between
losing a webhook and processing a webhook twice, which would I choose?"
Processing it twice is annoying but fixable (see Decision 2). Losing it is not
fixable — the customer paid and the money is gone from your records. So the
storage-first design is the only correct choice, and it's the one that costs
more code.

Decision 2: idempotency by (provider, event_id).

If the same webhook arrives twice, I don't want to process it twice. So the
service stores (provider, event_id) as a unique pair. The second attempt to
insert that pair throws an IntegrityError, the request returns 202 with
"status": "duplicate", and the original processing continues untouched.

The subtle part: this only works if the provider's event_id is actually
unique per event. Most providers document this. Some don't guarantee it. If a
provider reuses event IDs (rare but it happens), this design will reject a
legitimate webhook as a duplicate, and you've lost data in the opposite
direction.

I chose to trust the provider's event_id because every alternative is worse.
Hashing the raw body works until two different events have identical payloads
(very unlikely but possible). Hashing (event_id, amount, timestamp) works
until a legitimate retry arrives with a slightly different timestamp. In the end
you have to trust something the provider sends, and event_id is the thing
they've built the retry system around.

Decision 3: verify the signature on every request, using constant-time
comparison.

Every webhook must be signed. The provider uses a shared secret to compute an
HMAC of the request body, sends it as a header, and your server recomputes the
same HMAC and compares. If they match, the webhook is real. If they don't, it's
either a forgery or a body that was tampered with in transit.

The naive comparison is if expected == provided. That's wrong, and the reason
it's wrong is one of those things that sounds paranoid until you see it
exploited.

== on strings short-circuits — it stops comparing at the first character that
doesn't match. So if the attacker sends a signature where the first 31
characters are correct and the 32nd is wrong, == takes measurably longer to
return false than if the attacker had sent 32 wrong characters. By timing
enough requests, they can learn the correct signature one character at a time.

The fix is hmac.compare_digest(), which always takes the same amount of time
regardless of where the comparison fails. It's one function, and it's the
difference between a secure signature check and a very slow brute-force.

I learned this from a blog post years before I needed it. This was the first
time I actually needed it. That's how it works — you learn a thing in the
abstract, and then a year later you discover the specific moment it applies.

---

The test that failed for a week and the reason it was worth it

The signature verification has a test suite. About half of the 28 tests in the
project are signature tests: valid signature, wrong secret, tampered body, empty
signature, wrong-length signature, timing comparison.

The timing comparison test is the one I'm proudest of, and it's the one that
took longest to get right. It's not a test that checks a specific value — it's a
test that checks the shape of the comparison. It verifies that the function
doesn't leak information through timing, and testing for timing leaks is a
different kind of testing than asserting equals.

The version I first wrote compared the function's execution time against a
threshold. If the comparison took less than X milliseconds, the test passed.
That test passed on my laptop and failed on CI, because CI ran slower and the
baseline was different. I spent three days convinced that the timing-attack
protection was broken when the test was just wrong.

The fix was to stop trying to measure milliseconds and start testing the
property that actually matters: the function returns the same result for the
same input regardless of how much of the input is correct. That's testable
without timing. You compare a correct signature against a partial signature and
verify both return False, then verify a correct signature returns True.
That's not proof of constant-time behavior, but it's proof that the function
behaves the way the constant-time comparison does. And it's stable across CI
runners with different speeds.

The lesson: tests that measure time are tests that fail for the wrong reason.
You want tests that measure the property the timing protects against. Timing is
an implementation detail; security is the property.

---

The CI failure that almost wasn't

Two weeks after the first deploy, CI started failing on Python 3.10. Python 3.11
and 3.12 passed. 3.10 failed at the pip install step.

The error was:

ERROR: Could not find a version that satisfies the requirement
pydantic-settings==2.16.0

The reason: pydantic-settings version 2.16.0 requires Python 3.11 or higher.
My requirements.txt pinned that exact version. On Python 3.10, pip couldn't
find a matching version, so it errored out before the tests even ran.

I'd had a matrix of Python 3.10, 3.11, 3.12 because "supporting multiple
versions is good." I hadn't thought about whether the dependencies supported
multiple versions. They didn't, not all of them, not at the versions I'd pinned.

I had two options:

1. Pin pydantic-settings to 2.15.0, which supports Python 3.10, and keep the three-version matrix.
2. Drop Python 3.10 from the matrix, keep pydantic-settings at 2.16.0, and test on 3.11 and 3.12 only.

I chose option 2. Python 3.10 was released in 2021 and reaches end-of-life in
October 2026. By the time you're reading this, it may already be EOL. Pinning an
outdated dependency to keep testing on an outdated interpreter is technically
more code coverage but less useful coverage — it tests a configuration nobody
should be running.

The lesson: testing many versions is only useful if all of them are versions
someone should actually use. A red CI badge for a version of Python that no
longer gets security patches is worse than a green badge with a smaller matrix.

The commit that fixed this is a one-line change to requirements.txt or a
one-line change to .github/workflows/ci.yml, depending on which option you
choose. Two weeks of red CI for a one-line fix, because I hadn't read the error
closely enough — it said exactly what was wrong, and I spent a day not reading
it.

---

The bug I found because I couldn't see it working

Before deploying, I ran the tests locally and they all passed. I ran uvicorn
main:app --reload and hit /health. It returned {"status": "ok"}. I deployed
to Render, the deploy succeeded, and I stopped paying attention.

Three days later, I hit /health again and it was down. Render's free tier had
put the instance to sleep after 15 minutes of inactivity, and the first request
after sleep took 30 seconds to wake it up. The 30-second wait had timed out in
my browser, so the page looked broken.

There was no bug. The service was working exactly as designed — it just wasn't
designed for the free tier's sleep behavior. Adding a wake up endpoint, or
scheduling a keepalive request every 10 minutes, or moving to a paid tier
without sleeping, would each have solved a different version of the problem.

I did none of those. I accepted that the first request after inactivity takes 30
seconds, and I wrote it into the README so a visitor doesn't think the service
is broken. Then I moved on.

The lesson isn't "always keep your service warm." The lesson is: "the service
is working" and "the service appears to be working" are different things, and
the gap between them is what "production" means. Every deploy that succeeds is
a starting point, not a finish line. What happens the next time you look at it
is the actual test.

---

What it doesn't do

The README lists the limitations, and I'll list the most important ones here:

It stores every webhook as raw JSON. A high-volume merchant with 10,000
webhooks a day would fill up the SQLite database in a week. Postgres would
handle it; I used SQLite because it was simpler. That's a real limitation, and
it's the first thing I'd change if this were going to handle real traffic.

The retry logic has no circuit breaker. If a downstream processing endpoint
is down, the worker will retry every event, fail every retry, and exhaust its
attempts in a cascade. A circuit breaker would detect the downstream failure and
pause retries until it recovers. I didn't build one. The retry logic retries
faster than the downstream recovers, which is the opposite of what a good retry
system should do.

The HMAC secret is a single string per provider. Real systems need key
rotation — the ability to accept old and new signatures during a transition
window. My service accepts one secret. When the merchant rotates their key, every
in-flight webhook fails. The fix is obvious (accept a list of valid keys, try
each) but not built.

There's no admin UI. Events are inspected by querying the SQLite database
directly, or by hitting /webhooks/events. For a demo, that's fine. For a real
service, it's not.

No rate limiting. A malicious actor can flood the endpoint with fake
webhooks and exhaust the retry queue. HMAC signatures protect against forgery,
but not against volume.

Each of those is a v2 feature that I could add. None of them makes the current
version broken — the current version does what its README says it does. But
each of them is a real difference between "this works in a demo" and "you can
point a payment provider at this."

---

What it was worth

Webhook Relay is, so far, the most "real" service I've built. It's the one that
looks the most like something a company would actually use — because the problem
it solves (losing money to invisible webhook failures) is a problem that
companies actually have.

It has zero users. I've never had a merchant deploy it. Nobody has opened an
issue.

The reason I think it's worth the effort anyway isn't the users. It's the fact
that it's the first time I built something where the failure mode was the point
of the software, not a side effect of it. Every design decision in the service
was made in response to something that would otherwise be silent. That's a
different mode of thinking than building something that does a visible thing,
and it's the mode I want to be good at.

The other thing it was worth: it's the project that taught me that "reliability"
is not an adjective. It's a set of specific design decisions — store first, retry
with backoff, deduplicate, verify signatures with constant-time comparison — each
of which exists because someone, somewhere, lost money that didn't announce
itself. Building the service is how I learned what those decisions are for.

---

If you're building one

Four things I'd tell anyone building their first webhook receiver:

1. Store the webhook before doing anything with it. The processing can fail.
The storage shouldn't. Once the raw webhook is in a durable store, you can retry
processing as many times as you want without losing the original message.

2. Return 200 immediately, then process in a worker. The provider doesn't
care whether your processing succeeded — they care whether you received the
webhook. Returning 200 tells them "I got it, stop retrying." Processing can take
as long as it needs.

3. Use constant-time comparison for signature verification. hmac.compare_digest(), not ==. The difference is invisible in testing and material in security.

4. Test the property, not the implementation. Tests that measure milliseconds fail on slow CI. Tests that check "the function returns the correct boolean for every input shape" work anywhere. Test what the code guarantees, not what it happens to do on your laptop.

---

The link

Webhook Relay is at
github.com/Agenda-X-PR/webhook-relay,
with a live demo at
webhook-relay-xxvu.onrender.com/docs.
MIT licensed, 28 tests, CI on Python 3.11–3.13.

If you're building a payment integration, the README is the actual guide. The
article is the story of how the guide came to exist. Read whichever one is more
useful to you.

This is the third piece in a series about the projects I've shipped. The first
was why Excel fails at payment reconciliation. The second was
reconciling a Paystack export against a bank statement. The
fourth will be about the payments detection platform, when I get to
it.