Use the push channel for speed and a scheduled poll for proof. Registered webhooks carry a revocation from the issuer to your gateway in a couple of seconds; scheduled polling is what tells you, an hour later, whether that revocation actually landed. On a support desk product where every tenant holds its own scoped key, the axis that decides the architecture is not median latency — it's whether you can reconstruct, for one key on one afternoon, when access stopped and how you learned about it.
Reliability ownership never moves. The receiver owns it.
That sounds unfair, and it is, but it falls out of the delivery semantics rather than anyone's support contract. A webhook sender retries on a budget it chose. When the budget runs out, the event is dropped and your system is the only party that could have noticed.
The revocation that has to be provable
Picture the concrete system. A customer support platform issues one scoped key per tenant — read tickets, write replies, nothing else — and revokes it when an agent offboards or a contract ends. Enforcement happens at an edge gateway that keeps a local key table so it doesn't pay a network hop per request. That local table is a cache of somebody else's truth, and the entire design question is how fast and how verifiably that cache learns the word "revoked".
Three properties matter to the people who review access, in this order: the revocation eventually applies, you can name the moment it applied, and you can name how you found out. Latency comes fourth. An auditor who asks "was this key usable after the offboarding ticket closed?" is asking for a row, not a percentile.
So the intake is an append-only log first and a cache updater second. Get that ordering backwards and your audit trail becomes whatever your cache happens to contain right now, which is worth nothing during an access review.
Should I trust registered webhooks or scheduled polling to own key revocation reliability?
Neither one, alone. They have different silence modes, and the interesting engineering is in how they cover for each other.
A registered webhook is at-least-once delivery with a sender-owned retry budget. You get low latency and a natural idempotency key, and you inherit a hard edge: past the last retry, nothing distinguishes "nothing happened" from "four revocations were dropped while my intake returned 503". Scheduled polling inverts that. Liveness is yours, latency is your poll interval, and cost scales with tenants multiplied by frequency instead of with real events. Polling is boring and hard to lose track of, which is exactly why it makes a good witness.
| Channel | Propagation latency | Silent-failure mode | Evidence it leaves |
|---|---|---|---|
| Registered webhook only | seconds | retries exhausted while intake is down | one row per delivered event |
| Scheduled poll only | half the poll interval, on average | job stops running and nobody alerts | one row per observed divergence |
| Webhook plus reconciliation sweep | seconds | both channels stop at once | delivered events plus a divergence count per sweep |
The third row is the one I'd defend in a design review. Push sets the latency, the sweep sets the guarantee, and the divergence count from the sweep is a live measurement of how much the push path is missing.
The smallest intake I would ship
Start with signature verification, because an unverified revocation feed is an attacker-controlled way to deactivate a tenant's access, and a forged issuance event is worse. Standard Webhooks gives you a concrete shape — webhook-id, webhook-timestamp, webhook-signature — and HMAC-SHA256 over the id, the timestamp and the raw body. RFC 9421 is the standards-track alternative if you control both ends and want signed headers rather than a signed payload.
import { createHmac, timingSafeEqual } from "node:crypto";
import express from "express";
const SECRET = process.env.KEY_EVENTS_SECRET!;
const MAX_SKEW_SECONDS = 300;
function verify(rawBody: Buffer, id: string, ts: string, signature: string): boolean {
const age = Math.abs(Date.now() / 1000 - Number(ts));
if (!Number.isFinite(age) || age > MAX_SKEW_SECONDS) return false;
const expected = createHmac("sha256", SECRET)
.update(`${id}.${ts}.${rawBody.toString("utf8")}`)
.digest();
const got = Buffer.from(signature.replace(/^v1,/, ""), "base64");
return got.length === expected.length && timingSafeEqual(got, expected);
}
const app = express();
app.post("/hooks/tenant-keys", express.raw({ type: "application/json" }), async (req, res) => {
const id = req.header("webhook-id") ?? "";
const ts = req.header("webhook-timestamp") ?? "";
const sig = req.header("webhook-signature") ?? "";
if (!verify(req.body, id, ts, sig)) return res.status(401).end();
const event = JSON.parse(req.body.toString("utf8")) as {
type: "key.revoked" | "key.issued";
tenant_id: string;
key_id: string;
occurred_at: string;
};
// Append-only first, cache second. The audit row is the deliverable.
const fresh = await audit.record({
eventId: id, // unique index makes redelivery a no-op
tenantId: event.tenant_id,
keyId: event.key_id,
state: event.type === "key.revoked" ? "revoked" : "active",
occurredAt: event.occurred_at,
observedAt: new Date().toISOString(),
source: "push",
});
if (fresh) await keyCache.apply(event);
res.status(202).end();
});
Two details in there are worth more than the rest. The timestamp window turns a captured request into a 300-second replay opportunity instead of a permanent one, and the unique index on eventId is what lets you answer a redelivery with 202 and a clear conscience. Redelivery is not an edge case; it is the normal consequence of your intake timing out after it already committed.
Return fast, too. Anything slower than a second or two inside that handler is borrowed from the sender's retry budget, and a queue behind the handler is cheaper than a retry storm in front of it.
The sweep is the other half, and it's deliberately dull.
type RemoteKey = {
key_id: string;
tenant_id: string;
state: "active" | "revoked";
updated_at: string;
};
async function sweep(updatedSince: string): Promise<number> {
let cursor: string | undefined;
let divergences = 0;
do {
const url = new URL("/admin/tenant-keys", process.env.KEY_API_BASE);
url.searchParams.set("updated_since", updatedSince);
if (cursor) url.searchParams.set("cursor", cursor);
const res = await fetch(url, {
headers: { authorization: `Bearer ${process.env.KEY_API_TOKEN}` },
signal: AbortSignal.timeout(10_000),
});
if (res.status === 429) {
const wait = Number(res.headers.get("retry-after") ?? 5);
await new Promise((r) => setTimeout(r, wait * 1000));
continue;
}
const page = (await res.json()) as { keys: RemoteKey[]; next_cursor?: string };
for (const remote of page.keys) {
const local = await keyCache.get(remote.key_id);
if (local?.state === remote.state) continue;
divergences++;
await audit.record({
eventId: `sweep:${remote.key_id}:${remote.updated_at}`,
tenantId: remote.tenant_id,
keyId: remote.key_id,
state: remote.state,
occurredAt: remote.updated_at,
observedAt: new Date().toISOString(),
source: "sweep",
});
await keyCache.apply(remote);
}
cursor = page.next_cursor;
} while (cursor);
return divergences;
}
divergences is the number I'd put on a dashboard and alert on. Zero for a week means the push path is healthy and the sweep is pure insurance. A steady trickle means the push path is lying to you and the audit trail was only ever as good as the poll.
What the scheduled poll actually costs to run
Cost is where polling gets a bad reputation it only half deserves, because people price the naive version: every tenant, every minute, full state.
Price the incremental version instead. Four thousand tenants, a sweep every five minutes, updated_since on the last sweep's high-water mark, 500 keys per page. Almost every sweep returns one nearly empty page, so you're looking at roughly 288 requests a day plus a page or two of real change — call it under a thousand calls, against a webhook stream that would carry only the events that genuinely happened. Conditional requests shave it further: if the listing endpoint sets an ETag, If-None-Match turns a quiet sweep into a 304 with no body.
The expensive mistakes are cadence and scope, not polling itself. A one-minute sweep over full tenant state is two orders of magnitude more traffic than a five-minute incremental sweep, and it buys you four minutes of latency you already bought with the webhook.
Where this falls short, and what I'd change at scale
The design assumes a revocation event exists. If the issuing platform doesn't support outbound events for key state, push is not on the menu and you own latency by definition — run the sweep at whatever interval your access policy tolerates and say so plainly in the runbook, because a five-minute poll marketed internally as "real time" is how people get surprised.
The catch is on the other side too. If your gateway can afford one introspection call per request — RFC 7662 style, or any cached state check with a short TTL — the whole propagation problem disappears and both channels become optional. I'm not sure that trade is right for a support desk at low single-digit milliseconds of budget per request, but for an internal admin API serving a few requests per second, it is clearly simpler than running two feeds and reconciling them.
Short-lived credentials beat propagation entirely. If the scoped key can be a token with a five-minute lifetime, revocation becomes expiry, and expiry is a property of the credential rather than a message you might miss. That's the version I'd push for on a new build, and it's also the version that's hardest to retrofit onto tenants who already automated against a static key.
Stick with a single channel when the blast radius is small: an internal tool with twenty tenants and a nightly reconcile does not need this machinery. At the other end, past roughly a hundred thousand tenants, per-sweep listing stops being cheap and you want a changelog or cursor feed from the issuer — the same reconciliation logic, one sequence number instead of a full diff.
If you take one thing from this: record the source of every state change alongside the change. Push or sweep, with an observation timestamp. The day someone asks how you knew, that column is the answer, and no amount of latency tuning substitutes for it.
Top comments (0)