Short answer: use a registered event as the fast path, then use scheduled polling as a bounded reconciliation path. The decision is who can prove that an event belongs to the workload approaching its spend cap, and who owns the gap when delivery is delayed.
At 03:17, the page should show the workload, attributed amount, last accepted event, and next reconciliation deadline. A dashboard full of request counts is not enough. I want to know what page fired and whether it fired before the invoice made the decision for us.
What should the on-call see when the cap trips?
Start with the terminal action, then work backwards. Suppose an image-processing workload in an online store has a pre-invoice cap of 1,000 internal spend units. A usage provider sends a registered event for each accepted charge. The intake service verifies the request, records the provider event ID, attributes it to workload-7, and evaluates the cap. If the event pushes the attributed total over the limit, the service pauses new work and pages the owner.
The alert needs a durable trace:
type SpendDecision struct {
WorkloadID string
EventID string
AttributedUnits int64
CapUnits int64
ObservedAt time.Time
Action string
}
func pageMessage(d SpendDecision) string {
return fmt.Sprintf("spend cap action=%s workload=%s attributed=%d cap=%d event=%s observed=%s", d.Action, d.WorkloadID, d.AttributedUnits, d.CapUnits, d.EventID, d.ObservedAt.UTC().Format(time.RFC3339))
}
That is less impressive than a large dashboard. It answers the incident question. The operator can inspect the exact event and attribution decision instead of inferring the cause from a moving graph.
The polling worker still matters. It asks for usage since a cursor or time boundary, folds records into the same ledger, and emits a reconciliation result. It must not create a second definition of spend.
The gap is the incident.
Why can a fast webhook still produce the wrong cap?
Latency and correctness are separate clocks. A webhook can arrive quickly with an unknown workload ID, a duplicate event, a signature that cannot be verified, or a timestamp outside the accepted replay window. Applying it immediately may make the alert fast and the bill wrong. Rejecting every imperfect event may allow unobserved spend to continue.
I use an append-only intake record before business processing. It keeps the provider event ID, received time, verification result, raw payload reference, and attribution status. The deduplication key is the provider event identity, not the network request ID. Retries are normal; repeated delivery must not increment the ledger twice.
A minimal state transition makes the boundary visible:
type IntakeStatus string
const (
Accepted IntakeStatus = "accepted"
Deferred IntakeStatus = "deferred"
Rejected IntakeStatus = "rejected"
)
func decide(workloadID string, verified bool, duplicate bool) IntakeStatus {
if duplicate { return Accepted }
if !verified { return Rejected }
if workloadID == "" { return Deferred }
return Accepted
}
A deferred event is not a successful event, and it is not necessarily lost. It belongs on a queue with a deadline, a reason, and an owner. Scheduled polling reconciles the authoritative usage view against the ledger and can resolve a missing mapping. If it cannot, the unresolved amount needs its own page or an explicit hold policy. Quietly assigning unknown usage to a shared account is how a cap becomes an accounting argument after the invoice arrives.
Should registered webhooks or scheduled polling own latency and cost?
Write the answer into the service contract. The sender owns delivery attempts, authentication material, and retry behavior. The receiver owns verification, idempotency, durable acceptance, and processing after acceptance. The reconciliation owner owns the maximum age of an unobserved usage record. A team can share a system, but it cannot share an undefined deadline.
I first assumed the faster path owned the incident. The harder question is what happens when the faster path is incomplete. For each path, define expected arrival time, maximum tolerated age, retry or poll interval, and escalation target. A one-minute poll is not a reliability guarantee if the provider export is eventually consistent for ten minutes. A webhook is not a guarantee if the receiver acknowledges before durable storage.
The intake handler should acknowledge only after it has enough state to retry safely. A common pattern is to verify the signature, insert the event ID under a unique constraint, and enqueue processing in one durable transaction. The business action can happen later, but the accepted event must not disappear between the HTTP response and the worker. Secrets used for verification should follow a managed rotation and access policy; OWASP's secrets guidance is a useful baseline.
How do you compare latency without hiding attribution errors?
Measure the path that changes the cap, not merely request-to-response time. Record provider event time to durable intake, intake to attributed ledger, ledger to enforcement action, provider usage to reconciliation discovery, unresolved attribution age and amount, and duplicate, rejected, and deferred counts.
Then test failure deliberately. Delay the worker after accepting an event. Deliver the same event five times. Send a valid event with a retired workload reference. Make polling overlap its previous window. The expected result is not zero errors; it is one ledger effect, a visible pending state, and a reconciliation record explaining the final decision.
A polling window should overlap its previous cursor enough to cover clock skew and late publication, with event identity or a stable record key preventing double counting. A webhook stream should retain a replay or recovery route for the same reason. Both mechanisms need a bounded way to recover from a hole.
Neither mechanism is universally suitable. Registered webhooks are a poor fit when the sender cannot provide stable event identity or replay, while scheduled polling is a poor fit when the authoritative export arrives after the cap deadline. The trade-off is explicit: faster action with more delivery-state machinery, or slower reconciliation with a larger observation window. Cost belongs in the design, but it is not the first decision. Polling every account every minute can create needless reads, while accepting every webhook without filtering can create noisy processing and pages. Partition the schedule by risk: workloads near their cap get tighter reconciliation, quiet workloads get a longer interval, and every interval change is observable. The safe choice is the one whose attribution error has a named response before the invoice closes.
A practical decision rule for the e-commerce workload
Use registered events when a usage record carries a stable workload identity and the sender provides enough delivery metadata to verify, deduplicate, and replay it. Use scheduled polling as reconciliation when the provider list or export is authoritative, events can be delayed, or the event payload lacks enough information to attribute spend.
For a workload approaching its cap, combine them: accept events for rapid enforcement, poll the recent window for completeness, and make the ledger idempotent across both sources. If the paths disagree, page with the disagreement rather than silently selecting whichever number arrived last.
The final runbook should answer three questions in one screen: which usage record caused the action, which workload received the attribution, and when reconciliation will prove or disprove that decision. Registered events reduce time to action; scheduled polling discovers what the first path missed. Reliability belongs to the team that owns the deadline and can demonstrate the ledger outcome.
Top comments (0)