Building Alerting Pipelines for Instant Delivery SLAs on Digital-Goods Stores

When you sell instant-delivery digital goods — gift cards, game codes, mobile top-ups — "delivery time" isn't a shipping estimate. It's a promise measured in seconds. I work on the tooling behind digital storefronts like Kadokey, where redemption codes have to land in a customer's inbox almost immediately after payment. Here's how we think about monitoring that promise, and the alerting pipeline patterns that generalize to any digital-delivery system.

Define the SLA in measurable events

"Instant" is vague. We instrumented four timestamps per order:

1. payment_captured_at — money secured, delivery clock starts.
2. fulfillment_started_at — a worker picked up the job.
3. code_reserved_at — inventory allocated, code marked sold.
4. customer_notified_at — email/webhook/push actually handed to the delivery provider.

The headline SLA is the p95 of (customer_notified_at − payment_captured_at). Steps 1–3 are queue and reservation time; the fourth step is where most variance lives because it's the boundary with third-party email/SMS providers.

Alert on burn rate, not raw latency

A single slow order is noise; a sustained regression is an incident. We track an "SLA error budget": the share of orders allowed to exceed the target (say, 5% beyond 120 seconds) over a rolling window. Alerts fire on burn rate — how fast the budget is being consumed — with two tiers:

• Fast-burn: 2% of the budget consumed in 5 minutes → page someone now.
• Slow-burn: 100% of the budget consumed over 6 hours → ticket for the next business day.

This is the SRE alerting-on-SLOs pattern applied to delivery latency, and it kills the "one spike at 3am pages the whole team" problem.

Instrument the fulfillment queue, not just the web app

Most digital-goods architectures are queue-shaped: checkout enqueues a fulfillment job; workers reserve codes and trigger notifications. We export four queue metrics:

• Depth and age of oldest message — the earliest sign of trouble. Age-of-oldest beats depth because a stuck head-of-line message shows up immediately.
• Consumer lag per worker — identifies a poisoned worker or a bad deploy.
• Dead-letter rate — failed reservations. A spike here usually means an upstream inventory or supplier API changed.
• Retry count distribution — distinguishes a single flaky supplier (uniform retries) from a systematic failure (retry counts climbing across the board).

Separate internal latency from provider latency

When notification is slow, the first question is always "us or them?" We capture the provider's accepted timestamp when available (SES, SendGrid, Twilio all return one) and compute provider latency separately from our internal pipeline latency. An alert that says "internal pipeline healthy, provider p95 up 4x" saves twenty minutes of misdirected debugging — and it's the evidence you need when filing a provider incident report.

Keep a delivery canary

We run a synthetic "canary order" every few minutes against a sandboxed supplier endpoint — it buys a test SKU, measures end-to-end time, then voids it. The canary exercises the exact path customers use: payment capture, reservation, notification. It catches configuration drift (expired API keys, rotated webhook secrets) before real orders do. False-positive canaries are a tuning cost, but one undetected supplier outage pays for a year of tuning.

Dashboards people actually read during incidents

Three panels, in order:

1. Current p50/p95/p99 delivery latency vs. target.
2. Error-budget remaining (fast/slow burn).
3. Queue age-of-oldest plus dead-letter rate.

Everything else is drill-down. During an incident, cognitive load is the bottleneck; a dashboard with forty graphs is a dashboard nobody reads.

Practical takeaways

• Instrument four timestamps; make the SLA about the customer's experience (notification), not your internal steps.
• Alert on error-budget burn rate, not single slow orders.
• Track age-of-oldest-message and dead-letter rate on fulfillment queues.
• Separate your latency from your providers' latency.
• Run a canary order on the real customer path.

Getting a code into a customer's hands in under a minute, reliably, is an engineering problem with real business consequences — every minute of delay past expectation correlates with support tickets and refund requests. For stores operating at the scale of Kadokey's catalog, these pipelines are what make "instant delivery" an operational reality rather than a marketing line.