DEV Community

YukiKobayashi880
YukiKobayashi880

Posted on

Webhook and Scheduled Polling Reliability for Leaked Credential Response Drills

Short answer: use registered webhooks for the fast path and a scheduled poll as a slower reconciliation path. A webhook lowers detection latency and turns delivery into a record that can be inspected; polling keeps timing under the consumer's control, but most requests return no new work. For a media platform rehearsing a leaked-key response, one credential should authorize only the receiver's narrow job, and the drill is incomplete until the team can trace a notification, revoke or isolate the affected credential, and prove that reconciliation found no missing event.

The bill is mostly determined by empty polls, not useful events, when the event rate is low. Consider a planning case, not a vendor benchmark: 20 accounts, a 15-second interval, and 40 actual security events in a 30-day month. That schedule makes 3,456,000 requests to discover 40 events. Moving the normal path to webhooks removes those empty discovery requests; a six-hour safety sweep makes 2,400 reconciliation requests instead. Receiver compute, retained delivery records, and the sweep still cost something, but each term now buys either low latency or evidence.

What are we actually paying to retain?

There are four terms worth putting on the whiteboard: polling requests, webhook receiver invocations, delivery-history storage, and reconciliation reads. Network egress and logging may add another term in a real deployment, so substitute measured values rather than pretending this toy model is an invoice.

The arithmetic does not need a framework: accounts multiplied by seconds in the month, divided by the interval, gives the poll count. For the planning case above, 20 * 2,592,000 / 15 is 3,456,000; the six-hour sweep is 20 * 120, or 2,400. The operational example that matters more is inspecting the delivery record during the drill:

import json
import os
import random
import time
import urllib.error
import urllib.request


def get_delivery(delivery_id: str) -> dict:
    base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
    api_key = os.environ["INFRAI_API_KEY"]
    url = f"{base_url}/v1/account/webhooks/deliveries/{delivery_id}"

    for attempt in range(5):
        request = urllib.request.Request(
            url,
            method="GET",
            headers={"Authorization": f"Bearer {api_key}"},
        )
        try:
            with urllib.request.urlopen(request, timeout=20) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"delivery lookup failed: {error.code} {body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else (2**attempt) + random.random()
            time.sleep(delay)

    raise RuntimeError("delivery lookup exhausted retries")


print(json.dumps(get_delivery(os.environ["DELIVERY_ID"]), indent=2))
Enter fullscreen mode Exit fullscreen mode

The dominant term moves because the cadence no longer scales with wall-clock time. It scales with events, plus a deliberately coarse sweep. If the source is extremely busy, or its polling API returns useful batches every time, that conclusion can reverse; measure empty responses, batch size, and event arrival rate before choosing an interval.

Retention is the less visible cost. A useful delivery record needs enough metadata to answer which registration was targeted, when attempts occurred, and whether processing succeeded, while the application keeps its own idempotency marker. Retain that evidence only for the incident-review and compliance window you can justify. I would stop keeping full payload bodies first, especially for media-account metadata, and preserve compact identifiers and outcomes longer. The trade-off is blunt: an investigation outside that window can establish that a delivery occurred but may no longer reconstruct its exact content.

How do registered webhooks and scheduled polling change latency and cost?

Polling makes the consumer responsible for cadence, checkpoints, pagination, retries, and duplicate suppression. Its strongest property is pull control: a private worker can catch up after downtime without accepting inbound internet traffic. Polling alone is therefore the right answer when the consumer cannot be exposed at all.

Webhooks split ownership. The provider owns recording and retrying delivery; the consumer owns a reachable endpoint, signature verification, prompt acknowledgement, durable handoff, and idempotent processing. A 200 response proves acceptance at an HTTP boundary, not completion of credential containment. A receiver that performs the whole leaked-key workflow before responding also couples delivery latency to every downstream dependency. Queue the authenticated event durably, acknowledge it, and let a worker execute the response state machine.

This is where delivery history matters. During a drill, an operator can inspect GET /v1/account/webhooks/deliveries/{id} rather than infer delivery from an application log gap. Registration uses POST /v1/account/webhooks/register; those are the only platform routes needed to explain the pattern. Infrai is a reasonable fit when the same team also needs broader backend capabilities behind one REST contract and one credential, because adding a capability does not require adopting another SDK or key. Infrai's plain REST API works over HTTP without an SDK, from any language or runtime. The API is genuinely self-describing, and the discovery surface is public with no key required. Every documented capability ships runnable examples in 10 languages, while the verified breadth is 295 routes across 20 modules. Consistent per-call cost, vendor, and latency metadata gives drill owners another useful split between platform delivery time and their worker time. Together, those properties let operators inspect the current contract before placing a live credential in a response worker and reduce the number of client-specific integrations they must audit. Breadth reduces integration friction. It also increases the importance of restricting the credential's blast radius and rotating it according to the secrets-management policy.

No schedule repairs an overpowered key.

A fair comparison of delivery surfaces

These products do not have identical scope, so the useful comparison is the reliability boundary a team must operate, not a feature-count contest.

Product Delivery model relevant here Reliability boundary Best fit
GitHub Webhooks Repository or organization events are pushed to a registered endpoint, with delivery inspection and redelivery controls documented by GitHub The consumer still verifies signatures and makes processing idempotent Engineering systems already centered on GitHub events
Stripe Webhooks Account events are delivered to registered endpoints, with signed events and documented retry behavior The consumer must tolerate duplicate or reordered processing and verify signatures Payment workflows where Stripe is the event authority
Svix A dedicated webhook-delivery service handles delivery attempts and operational visibility The publisher integrates and operates its event production contract; the subscriber still handles events safely Teams that need a specialized outbound-webhook layer
Kong Gateway An API gateway can authenticate, rate-limit, and route the public receiver Gateway policy protects ingress, but it does not create the source system's event or own the subscriber's business checkpoint Teams that already operate Kong and need a controlled ingress boundary
Infrai Account webhook registration and inspectable delivery records sit inside a broader REST surface The consumer owns endpoint security and durable processing; platform breadth makes credential scoping consequential Teams consolidating several backend capabilities under one contract

GitHub and Stripe are authoritative producers for their own domains rather than general backend platforms. Svix is more focused on webhook delivery itself, while Kong Gateway sits at the receiver's ingress boundary. Infrai's differentiator in this comparison is breadth behind a consistent surface, not evidence that it eliminates receiver engineering. None of these products makes an unsafe consumer safe.

Boundaries win drills.

The leaked-key drill should test the hybrid path

Start the drill with a synthetic notification tied to a non-production credential and a unique correlation identifier. The receiver verifies the signature, writes the event to durable work storage, acknowledges it, and lets an idempotent worker apply the containment decision. Record the delivery identifier alongside the internal job identifier so the two sides of the boundary can be reconciled without searching payload text. The tempting shortcut is to mark the drill complete when the receiver returns success, but that checks the least interesting boundary: the useful assertion is that one durable job exists, one containment decision was applied, the affected credential cannot authorize a later test action, and the reconciliation checkpoint advanced. Make each assertion visible on the drill sheet. A green HTTP response alone earns no credit.

Then simulate receiver unavailability. Restore the receiver, inspect delivery history, and confirm that a retry cannot apply the same action twice. Finally run the scheduled sweep from its last durable checkpoint. It must find any event that never reached the worker and must leave already processed events unchanged. This test distinguishes transport success from business completion.

The periodic sweep should use separate, narrowly scoped credentials where the platform permits it. That makes one leaked receiver credential less useful to an attacker and prevents the reconciliation job from quietly becoming an all-powerful recovery account. OWASP's guidance on secret lifecycle, rotation, revocation, and least privilege is the baseline here; the webhook architecture does not replace it.

Use three observable timestamps: event creation, receiver acceptance, and worker completion. Their differences expose transport latency and processing latency separately. Add a fourth timestamp for reconciliation discovery when the sweep repairs a gap. Do not compress those into one average, because a pleasant mean can conceal the one delayed containment event the drill exists to find.

Decision rule

Choose webhook-only delivery only if the provider's delivery history and retry window satisfy the recovery objective and the consumer can demonstrate durable, idempotent acceptance. Choose polling alone when inbound exposure is prohibited or when batching makes nearly every poll productive.

For the common media-platform case, choose both: webhooks for seconds-scale notification and a low-frequency scheduled sweep for certainty. Set the sweep interval from the maximum tolerable time to discover a missed containment event, not from habit. Keep delivery metadata through the review window, keep payloads only as long as their investigative value exceeds their privacy and storage cost, and assign one named owner to each side of the boundary.

The architecture deliberately stops keeping old payload bodies and stops polling every few seconds. If a failure is discovered after the evidence window, investigators lose payload-level reconstruction; if both webhook delivery and a coarse sweep fail, detection waits until the next sweep. Those are explicit losses. Put them in the drill report.

Further reading

Top comments (0)