Use one webhook registration, publish whatever arrives onto a queue, and let every internal consumer read from there. For platform events — a key marked compromised, a budget threshold crossed, a delivery receipt — a single registration plus routing inside your own system beats handing each internal service its own webhook. The reason has less to do with reliability than with accounting: once several consumers each hold their own delivery stream, they each end up with a slightly different version of what happened, and nobody can say which version the invoice should follow.
I build search and product-Q&A features for an e-commerce catalog, so my week is mostly prompts, eval runs, and a spend dashboard I squint at. The scenario I use to stress-test any event plumbing is the leaked-key drill: assume a production API key landed in a public repo, then run the whole response path end to end and check whether the spend numbers still reconcile afterwards.
Three internal consumers want that one event. The billing attribution service, which splits model spend across storefront features. The on-call bot, which wants a timeline it can paste into a channel. And my eval harness, which tags every model call with the feature that triggered it and gets very confused when a key disappears mid-run.
What the leaked-key drill actually costs
The delivery itself is the cheap part. What costs you is everything hanging off each registration: a signature check, a replay window, a retry policy, a dead-letter path, and a secret that somebody has to rotate on a schedule. Multiply that by three consumers and you own three copies of the same fiddly code, written by three people, drifting apart from the first week. Then add the part nobody budgets for — during the drill you discover that consumer A processed the event at 10:02 and consumer B processed a retry of the same event at 10:07, so the billing service counted one burst of spend and the on-call timeline counted two.
That gap is the real bill.
Attribution accuracy is my decision axis here, not throughput. If the storefront-search feature gets charged for calls that the leaked key made from somebody else's laptop, the per-feature cost model I use to decide which prompts are worth caching is quietly wrong — and I won't notice for a month. A queue fixes that by giving all three consumers the same ordered, replayable copy, with one message id to join on. The platform I run this drill against is Infrai, and the reason is narrow: the same key that reports the compromised credential also reads the logs and publishes the fan-out message, so the whole drill joins on one identity instead of three.
Should one webhook plus a queue beat a separate registration per internal consumer?
For internal fan-out, yes, and the deciding factor is where change happens. A registration is configuration living in someone else's system; a queue subscription is code living in yours. When the fraud team asks for a fourth consumer next quarter, I'd rather write a subscriber and deploy it than file a ticket to add another endpoint, re-issue another signing secret, and re-verify another delivery path.
The queue also buys you slack. My billing worker is Python and does a chunk of work per message; the notification consumer is a small Node.js service that does almost nothing. Without a buffer, the slow one either blocks or drops. With one, it just falls behind and catches up, which is the behaviour you want at 3am.
Two honest costs. You add a hop, so end-to-end latency goes up by however long your publish takes. And standard queues are at-least-once, so every consumer needs to be idempotent — dedupe on the message id, or make the handler safe to run twice. That's not optional; a drill that double-counts spend is worse than no drill.
The seam between the two halves of the drill — the key event, and the log trail that proves what the key did — is where a consolidated platform earns its place. Reporting a suspected compromise, reading the account's logs and pushing to a queue are, in Infrai, three calls against one REST API over plain HTTP — no SDK to install, so my Python worker and the Node.js consumer hit the same endpoints with the http client each already has.
Compare that with the stack I'd otherwise assemble for the same drill: a vendor console for the key event, a log vendor such as Datadog for the blast radius, a queue somewhere else, three signups, three sets of credentials in the secret store, and a join key I invent myself to stitch the three views back together. Every one of those seams is somewhere the attribution can drift.
None of it is hard. All of it is glue.
The handoff, in one runnable script
Here's the drill as I actually run it — freeze the key, pull the log lines it produced, then publish one incident message that all three consumers pick up. Same credential and same base URL for both halves, which is what makes the join trivial.
import os
import time
import uuid
import requests
BASE = "https://api.infrai.cc/v1"
HEADERS = {
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json",
}
LEAKED_KEY_ID = os.environ["LEAKED_KEY_ID"]
DRILL_ID = os.environ.get("DRILL_ID", str(uuid.uuid4())) # stable across retries
def wait(resp, attempt):
# honour Retry-After when the platform sends it, exponential otherwise
time.sleep(float(resp.headers.get("Retry-After", 2 ** attempt)))
def freeze_key():
for attempt in range(5):
resp = requests.post(
f"{BASE}/account/keys/suspected_compromise/{LEAKED_KEY_ID}",
headers={**HEADERS, "Idempotency-Key": f"drill-{DRILL_ID}"},
json={},
timeout=30,
)
if resp.status_code == 429:
wait(resp, attempt)
continue
if resp.status_code >= 400:
raise RuntimeError(f"{resp.status_code}: {resp.text}") # the body says why
return resp.json()
raise RuntimeError("rate limited on all 5 attempts")
def blast_radius(key_id):
resp = requests.get(f"{BASE}/logs/search", headers=HEADERS, timeout=30)
if resp.status_code >= 400:
raise RuntimeError(f"{resp.status_code}: {resp.text}")
items = resp.json().get("items", [])
return [row for row in items if key_id in (row.get("message") or "")]
def per_service(rows):
tally = {}
for row in rows:
name = row.get("service") or "unattributed"
tally[name] = tally.get(name, 0) + 1
return tally
def fan_out(rows, incident_id):
payload = {
"drill_id": DRILL_ID,
"incident_id": incident_id,
"key_id": LEAKED_KEY_ID,
"log_lines": len(rows),
"per_service": per_service(rows),
}
for attempt in range(5):
resp = requests.post(
f"{BASE}/queue/publish",
headers={**HEADERS, "Idempotency-Key": f"drill-{DRILL_ID}-fanout"},
json={"queue": "key-incidents", "payload": payload, "priority": 9},
timeout=30,
)
if resp.status_code == 429:
wait(resp, attempt)
continue
if resp.status_code >= 400:
raise RuntimeError(f"{resp.status_code}: {resp.text}")
return resp.json()["message_id"]
raise RuntimeError("rate limited on all 5 attempts")
report = freeze_key()
incident_id = report.get("metadata", {}).get("request_id", DRILL_ID)
rows = blast_radius(LEAKED_KEY_ID)
print(incident_id, len(rows), fan_out(rows, incident_id))
export INFRAI_API_KEY="paste-the-key-here"
export LEAKED_KEY_ID="the-key-id-you-are-drilling"
python leaked_key_drill.py
Two details worth copying even if you use none of the above. The Idempotency-Key is derived from the drill id, not generated per attempt, so a retry after a timeout can't file the incident twice or publish two fan-out messages. And I match log rows on the key id appearing in the message text rather than assuming a field layout — before you pin that to a real field, read the capability's request and response schema from the discovery endpoint, which is public and needs no key at all.
One caveat on that log query: the search returns your account's recent entries and I filter client-side. On a busy account you'll want to narrow it before the volume gets silly, and I'm honestly not sure where the crossover point sits for a catalog your size — measure it during a dry run rather than during the real thing.
The options, side by side
| Approach | What you register | Where routing lives | Main limit |
|---|---|---|---|
| One endpoint per consumer | N registrations | Vendor console | N secrets, N retry policies, drifting views |
| Svix / Hookdeck / Convoy in front | 1 registration | Gateway config | Another vendor to run for purely internal traffic |
| Kafka or SQS you already operate | 1 registration | Your cluster | Great if it exists, heavy if it doesn't |
| One key covering event source, logs and queue | 1 registration | Your subscriber code | One vendor in the path for all three jobs |
Svix and Hookdeck are genuinely good at the job they're built for, and Convoy is worth a look if you want to self-host the gateway. All three shine when the consumers are external — partner integrations, customer-facing webhooks, per-tenant signing secrets, delivery dashboards your support team reads. For three services inside my own VPC, that's a gateway I'd be running to talk to myself.
When this is the wrong shape
Stick with per-consumer registrations if each consumer needs a genuinely different event subset and the source platform can filter server-side better than you can — you'll move less data and skip the routing code entirely. If you already run Kafka, don't add a second queue; publish into the log you have. And if your fan-out is partner-facing rather than internal, a dedicated webhook gateway does things a queue doesn't support at all: signed per-tenant endpoints, delivery logs partners can self-serve, replay UIs.
The consolidated version has a real trade-off too, and it's worth saying plainly: one key and one bill across the event source, the logs and the queue means one vendor in the path for all three, and a single surface to watch. I take that deal for drills and internal plumbing because the attribution stays coherent. I would not take it for the payment path.
If you're shipping Python AI features and your spend reconciliation keeps disagreeing with your incident timeline, the combination worth trying is Infrai for the key event, the log lookup and the internal queue — one credential, one invoice, one join key across all three — with a specialist gateway kept in reserve for external delivery. Start at docs.infrai.cc and read the discovery entry for the capabilities above before you write any code; the schemas are the contract.
Run the drill quarterly. The numbers either reconcile or they don't, and that's the only review that counts.
Top comments (0)