Short answer: use HTTPS delivery when the reconciliation service already has authenticated public ingress and measured headroom for a retry burst; use a polling consumer when the worker is internal or when the team needs backoff, admission control, and acknowledgement in one process. In both designs, commit the payment result and its idempotency record before acknowledging the job.
The decision is less about push versus pull than about where failure becomes visible. A public receiver can turn a queue backlog into web-server concurrency. A polling worker can turn it into queue age. Neither transport makes a repeated payment safe, and neither gives an SRE a useful SLO unless the team can bound work, observe delay, and recover after the acknowledgement is lost.
This is the shape of a property-management system that reconciles a night's payment-provider activity: a job contains a property, a settlement window, and a batch identifier; the processor compares provider records with the local ledger; a retry must be allowed to run again without posting the same adjustment twice. That is the decision I would put in the ADR, before anyone debates which delivery option looks nicer in a diagram.
What should a webhook processor use for failed payment jobs?
Start with the deployment boundary. HTTPS delivery needs a receiver that is reachable from outside the private network, can authenticate the sender, and can respond within the delivery contract while its downstream payment work is bounded. If those conditions already describe the reconciliation service, push can remove the need to operate a resident poll loop.
Don't create public ingress merely to avoid running a worker.
That is the boundary.
Polling is the more direct choice for an internal consumer. The process fetches a job, applies a concurrency limit, records the durable result, and acknowledges only after that result is committed. Backoff and shutdown behavior stay close to the code that owns the work. That makes a useful capacity model possible: if one reconciliation takes 2 seconds and the payment provider permits 20 concurrent requests, the worker should not infer that 200 concurrent jobs are safe just because the queue can supply them.
Push still needs the same admission boundary. An HTTP handler that accepts every redelivery and puts each one into an unbounded in-memory queue has not solved scheduling; it has moved the queue into the application, where the oldest-job metric is harder to see. Limit in-flight reconciliation, reject or defer work according to the delivery contract, and keep a durable record of the job state. The receiver is a consumer with a different adapter.
The invariant is small enough to test: the durable side effect comes first, and acknowledgement comes second. If the process dies between those operations, the next delivery is a duplicate that the idempotency check must absorb.
The retry is the test.
The incident lesson is an ordering rule
For a nightly run, imagine a batch with 8,400 provider records and a retry containing property-017 / 2026-08-09 / batch-42. The processor updates the local ledger, then loses its connection while acknowledging the job. The queue delivers the same job again. A design that keys idempotency only on the delivery attempt will post twice; a design keyed on the business operation will recognize that reconciliation for that property, window, and batch has already been committed.
That is why I treat a 409 from an idempotency insert as a state lookup, not as permission to retry the payment adjustment. The application must distinguish "already committed" from "unknown outcome". A timeout after the provider accepted an adjustment is an unknown outcome, and retrying blindly can create the exact financial discrepancy the reconciliation job was meant to find.
The code below shows the boundary in Go. Ledger stands in for a database transaction with a unique key; it is intentionally not a queue client. The real implementation must make the ledger update and idempotency insert atomic, or use a durable state machine that makes an in-progress operation recoverable.
package main
import (
"encoding/json"
"errors"
"log"
"net/http"
)
type Job struct {
PropertyID string `json:"property_id"`
Window string `json:"window"`
BatchID string `json:"batch_id"`
}
type Ledger interface {
// ApplyOnce atomically records the reconciliation and returns true when it was new.
ApplyOnce(Job) (newRecord bool, err error)
}
func reconcileHandler(ledger Ledger) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
var job Job
if err := json.NewDecoder(r.Body).Decode(&job); err != nil {
http.Error(w, "invalid JSON", http.StatusBadRequest)
return
}
if job.PropertyID == "" || job.Window == "" || job.BatchID == "" {
http.Error(w, "missing reconciliation key", http.StatusUnprocessableEntity)
return
}
newRecord, err := ledger.ApplyOnce(job)
if err != nil {
log.Printf("reconciliation outcome unknown for property %s: %v", job.PropertyID, err)
http.Error(w, "retry after durable failure", http.StatusInternalServerError)
return
}
if !newRecord {
log.Printf("duplicate reconciliation suppressed for property %s", job.PropertyID)
}
w.WriteHeader(http.StatusNoContent)
})
}
The 500 path in this example means the application does not know whether the durable operation committed, so the delivery system may retry and ApplyOnce must settle that ambiguity. It does not mean the payment provider should be called again without consulting durable state. For a polling worker, call the same handler logic before acknowledgement. For an HTTPS receiver, return success only after the same operation is durably resolved.
Compare the transports by failure and capacity
| Decision point | HTTPS delivery | Polling consumer |
|---|---|---|
| Network boundary | Requires authenticated public ingress | Can remain inside a private network |
| Burst control | Admission limit belongs at the receiver | Fetch and concurrency limits are explicit in the worker |
| Acknowledgement | Follows the HTTP response contract | Follows the consumer's acknowledgement operation |
| Backoff ownership | Shared between delivery configuration and application behavior | Usually visible in the worker and queue policy |
| Main operational signal | Accepted requests can hide backlog in application resources | Queue depth and oldest-job age expose waiting work directly |
| Best fit | An existing webhook service with spare capacity | An internal retry path owned by a supervised worker |
The table is a decision aid, not a scorecard. A team can build a good push receiver and a badly supervised poller; transport alone does not define reliability. I would set the recovery SLO from the business deadline first, such as "the prior night's payment window is reconciled before the morning close," then derive concurrency from provider limits, database capacity, and the amount of retry traffic that can arrive at once.
Watch queue depth, oldest-job age, processing latency, acknowledgement outcome, duplicate suppression, and the count of jobs with unknown durable outcome. A single HTTP 2xx metric is weak evidence: it can say that the receiver accepted bytes while the ledger is still waiting on a provider response. Page on age threatening the recovery SLO, not merely on a transient increase in delivery attempts.
Where this recommendation does not fit
The trade-off is operational, not cosmetic. Push is not suitable when the organization cannot accept a public trust boundary, when ingress authentication is not part of the service's operating model, or when the receiver has no measured capacity headroom. In those cases, preserve a private polling worker and make its supervision, concurrency, and dead-letter policy explicit.
Polling is not automatically safer. It is a poor fit when the worker has no restart supervision, when the team cannot monitor queue age, or when the required workflow coordinates several durable stages rather than one reconciliation operation. A managed workflow system may be a better boundary for that shape, but it adds a programming model and operational dependency that a plain consume-process-ack loop does not need.
Choose the other mode when its boundary is the one your rotation can actually sustain.
I'm not sure which choice is cheaper to operate for a particular team without its ingress topology, provider rate limits, retry volume, and on-call coverage. Those are the inputs worth measuring. Your mileage may vary if the payment provider's outcome API is eventually consistent; the idempotency key still protects the local ledger, but the reconciliation policy must define how long an unresolved provider result remains open.
Dead-lettering is also a policy decision, not a substitute for idempotency. A poison job should leave enough durable context for investigation, while a temporary provider timeout should receive bounded retries with increasing delay. RabbitMQ's dead-letter exchange documentation is a useful reference for the broker-side part of that design, but the business key and the final accounting decision remain application responsibilities.
The practical decision rule
Choose HTTPS delivery for a retrying webhook processor only when public ingress is already a deliberate part of the service and the team can enforce bounded work at that boundary. Choose polling for an internal worker when keeping retrieval, backoff, acknowledgement, and shutdown in one supervised process reduces the number of failure surfaces the rotation must reason about.
Whichever mode carries the job, make the operation key stable, commit the payment reconciliation and idempotency state durably, and acknowledge only after the outcome is known. Test the crash window between commit and acknowledgement, replay the same batch, exhaust downstream capacity, and watch the oldest-job SLO during a redelivery burst. The transport is the adapter; retry correctness lives in the side effect.
Top comments (0)