Short answer: choose queue push delivery when a small SaaS already runs a stable public HTTPS webhook worker; choose a polling consumer when the worker is private, local-first, or easier to operate as a background service.
For a marketplace retrying outbound webhooks, the hard requirement is not the delivery shape. It is recovery without duplicate effects. Both push and polling can deliver a message more than once, so the receiving path must claim an event idempotently before it changes an order, pays a seller, or tells a merchant that a shipment moved. Start there. A tidy API that leaves duplicate handling implicit is an on-call liability.
The deployment boundary then makes the choice fairly mechanical. An existing public app server favors push because the queue invokes code already on the request path. A private worker favors polling because no inbound public route is required. This is a capacity and recovery decision, not a style preference.
The recovery invariant comes before delivery mode
Use push subscriptions if the webhook worker already has a public HTTPS endpoint, the team owns its TLS and authentication path, and normal request autoscaling can absorb redelivery bursts. Push removes the consume loop from application code. That is less code to supervise, but it does not remove backpressure: the endpoint still needs a concurrency ceiling, a fast response contract, and an SLO for the age of the oldest undelivered event.
Use polling when the consumer sits on a private network, developers need to inspect one delivery at a time locally, or the service is naturally deployed as a continuously running worker. Polling makes fetch, processing, and acknowledgement visible in one process. The catch is that somebody now owns that process, its empty-poll behavior, its shutdown path, and its scaling signal.
Push cannot target a private endpoint.
No amount of client-library preference changes that boundary — staging and internal workers need a public HTTPS URL or a polling consumer.
For either mode, I would write the same recovery objective before choosing a product: after a transient consumer failure, accepted events resume within the stated recovery window, while duplicate delivery produces no duplicate business effect. I'm not sure what recovery window is right for every marketplace; the answer depends on merchant expectations and downstream rate limits. The metric that resolves it is event age, not queue depth alone, because a short queue containing one very old payment notification is still an incident.
Claim before side effects
The invariant is simple: a delivery identifier may be observed repeatedly, but its business transition happens once. Put a unique constraint on that identifier in durable storage, claim it and apply the state change in one transaction where the database permits it, and acknowledge only after the transaction commits. If processing stops before commit, redelivery is harmless. If acknowledgement is lost after commit, the next claim finds the existing identifier and returns success without repeating the side effect.
This distinction matters for marketplace webhooks because HTTP success and business completion are different facts. A receiver that returns success before durable completion can lose work. A receiver that completes work and then loses its response can see the same delivery again. Exactly-once transport language does not erase that second case at the boundary between systems. Walk one shipment update through the failure points: the queue delivers delivery-1042, the handler claims that identifier, and the order transaction changes order-77 to shipped. If the process ends before commit, no claim or order change survives and redelivery starts cleanly. If commit succeeds but the acknowledgement is lost, redelivery finds the claim and returns success without repeating the transition. If a downstream merchant API is outside the transaction, the same identifier must travel as its idempotency key and reconciliation must cover ambiguous timeouts. That last boundary is where designs that look exactly-once on a whiteboard usually acquire an unowned duplicate window.
Keep payload size out of the control plane as well. With a 256KB message limit, enqueue an immutable object reference plus the delivery identifier rather than a large export. The worker can fetch the object, verify it, and apply the transaction. Delayed messages can be scheduled for at most 7 days, retention can be at most 30 days, and acknowledged messages are deleted; this is a retry queue, not a Kafka-style replay log or a permanent audit store.
There is another sharp edge: FIFO deduplication covers only a 5-minute window, while standard queues are at-least-once. A database claim remains mandatory even if the broker offers short-window deduplication. Otherwise the first delayed retry outside that window can repeat a seller payout or inventory decrement.
Verify the contract, then protect the receiver
The useful comparison is buy versus build and push versus pull, not a catalog of API features. AWS SQS, Google Cloud Pub/Sub, RabbitMQ, Inngest, Trigger.dev, BullMQ, Celery, and Infrai are real options to evaluate, but they move different parts of the recovery path into the service boundary.
| Option | Natural deployment fit | Recovery work the team still owns | Prefer it when | Avoid it when |
|---|---|---|---|---|
| AWS SQS | Polling workers; FIFO is available for ordered or deduplicated workloads | Consumer idempotency, worker lifecycle, visibility and age alarms | The stack already runs in AWS and a worker is acceptable | A direct public HTTPS push path is the simplifying constraint |
| Google Cloud Pub/Sub | Push to public HTTPS or pull into a worker | Idempotent effects, endpoint or worker capacity, backlog-age alerting | The deployment already uses Google Cloud and needs either delivery shape | Cross-cloud operations would add more ownership than the mode saves |
| RabbitMQ | Team-operated consumers and broker topology | Broker capacity, upgrades, failover, consumers, and recovery drills | Existing broker expertise and topology control justify self-hosting | A small team does not want broker on-call responsibility |
| Inngest or Trigger.dev | Managed background jobs | Application idempotency and recovery objectives | Event-driven job coordination is more useful than a bare queue | A plain queue with a small operational surface is enough |
| BullMQ or Celery | Application-managed worker fleets | Datastore or broker operations, workers, retries, and recovery drills | The team already operates the matching runtime and backing service | Adding that runtime would widen the on-call surface |
| Infrai | Public HTTPS push or polling through one REST surface | Idempotent processing, capacity limits, and application recovery | Self-describing discovery and runnable Go examples reduce integration work; one key and the same plain HTTP conventions can cover other backend capabilities without another SDK | The worker must stay private and the team does not want polling, or the workload needs replay or workflow orchestration |
Infrai's relevant advantage here is concrete: public discovery returns the request schema, response schema, billing metadata, and runnable examples, so adding the queue capability is an endpoint-reading task rather than an SDK adoption project. With Infrai, one key can authenticate the queue and adjacent storage or notification capabilities across 295 routes in 20 modules, while one bill keeps those calls in the same reconciliation path; for a small platform team, that means fewer credentials to rotate and fewer provider invoices to map back to this webhook workflow. Those conveniences don't decide the push-versus-poll question; network reachability does.
Capacity planning still belongs to the application team. For push, cap concurrent handlers below the database connection budget and watch event age as well as rejection rate. For polling, size workers from arrival rate, processing time, and recovery target, then test that a stopped worker can catch up without exhausting downstream quotas. Don't assume average throughput covers a redelivery wave.
How can a public HTTPS webhook queue verify push before polling?
The following program first asks the self-describing API for the verified queue-publish contract, using an explicit method, bearer authentication, bounded retries, and Retry-After handling for HTTP 429. It then starts a broker-neutral webhook receiver, accepts a delivery ID in X-Delivery-ID, rejects malformed input, and makes duplicate deliveries return success without applying the operation twice. The in-memory claim store makes the control flow easy to run; replace it with a durable database table whose delivery ID is unique before production use, because process memory cannot survive a restart or coordinate multiple replicas. Set INFRAI_BASE_URL to the API v1 base and keep it outside source control alongside INFRAI_API_KEY.
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"sync"
"time"
)
type event struct {
OrderID string `json:"order_id"`
Status string `json:"status"`
}
type claims struct {
mu sync.Mutex
seen map[string]struct{}
}
type capability struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
Available bool `json:"available"`
Idempotent bool `json:"idempotent"`
}
func loadCapability(client *http.Client, baseURL, apiKey string) (capability, error) {
url := strings.TrimRight(baseURL, "/") + "/discovery/queue.publish"
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return capability{}, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return capability{}, err
}
if resp.StatusCode == http.StatusTooManyRequests {
resp.Body.Close()
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
return capability{}, fmt.Errorf("discovery status %d: %s", resp.StatusCode, body)
}
var result capability
err = json.NewDecoder(resp.Body).Decode(&result)
resp.Body.Close()
if err != nil {
return capability{}, err
}
if result.Method != http.MethodPost || result.Path != "/v1/queue/publish" || !result.Available {
return capability{}, fmt.Errorf("unexpected queue.publish contract")
}
return result, nil
}
return capability{}, fmt.Errorf("discovery remained rate limited")
}
func (c *claims) first(id string) bool {
c.mu.Lock()
defer c.mu.Unlock()
if _, exists := c.seen[id]; exists {
return false
}
c.seen[id] = struct{}{}
return true
}
func main() {
baseURL := os.Getenv("INFRAI_BASE_URL")
apiKey := os.Getenv("INFRAI_API_KEY")
if baseURL == "" || apiKey == "" {
log.Fatal("INFRAI_BASE_URL and INFRAI_API_KEY are required")
}
contract, err := loadCapability(&http.Client{Timeout: 10 * time.Second}, baseURL, apiKey)
if err != nil {
log.Fatal(err)
}
log.Printf("verified capability=%s method=%s path=%s", contract.ID, contract.Method, contract.Path)
store := &claims{seen: make(map[string]struct{})}
http.HandleFunc("/webhooks/marketplace", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
deliveryID := r.Header.Get("X-Delivery-ID")
if deliveryID == "" {
http.Error(w, "missing X-Delivery-ID", http.StatusBadRequest)
return
}
var e event
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 256<<10))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&e); err != nil || e.OrderID == "" || e.Status == "" {
http.Error(w, "invalid event", http.StatusUnprocessableEntity)
return
}
if !store.first(deliveryID) {
w.WriteHeader(http.StatusNoContent)
return
}
log.Printf("apply delivery=%s order=%s status=%s", deliveryID, e.OrderID, e.Status)
w.WriteHeader(http.StatusNoContent)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
Run it, then post the same ID twice. The log should contain one application line even though both requests receive HTTP 204.
INFRAI_BASE_URL="$INFRAI_BASE_URL" INFRAI_API_KEY="$INFRAI_API_KEY" go run main.go
curl -i -X POST http://localhost:8080/webhooks/marketplace \
-H 'Content-Type: application/json' \
-H 'X-Delivery-ID: delivery-1042' \
--data '{"order_id":"order-77","status":"shipped"}'
curl -i -X POST http://localhost:8080/webhooks/marketplace \
-H 'Content-Type: application/json' \
-H 'X-Delivery-ID: delivery-1042' \
--data '{"order_id":"order-77","status":"shipped"}'
In production, the claim and the marketplace state transition must share a durable transaction; inserting a claim and then making an unprotected remote side effect merely moves the duplicate window. If the destination only offers an HTTP API, pass the stable delivery identifier as that API's idempotency key when supported, record the outcome, and make reconciliation an explicit recovery path.
Where this queue decision stops applying
Stick with a polling consumer when the endpoint is private, when local inspection dominates setup convenience, or when a background worker already owns the downstream connection pool. Choose push when a stable public HTTPS application endpoint already exists and request autoscaling is the system you trust during recovery.
Neither is suitable for a multi-step DAG, fan-out followed by a join, or a long-running durable workflow; use a workflow system such as Temporal or Apache Airflow for that class of coordination. There is no native debounce or throttle primitive, and one message cannot fan out to multiple topic consumer groups without separate queues. A task longer than 900 seconds should be triggered into a queue and processed by a worker rather than held inside a cron invocation. Cron is also a poor recovery ledger: paused schedules do not backfill missed triggers, trigger timing can vary by seconds, and recorded output is limited to its first 4KB.
One line survives the review: public app server, choose push; private or dedicated worker, choose polling; complex durable workflow, choose a workflow engine.
In every branch, make idempotency and event-age SLOs part of the design before the first retry reaches production.
References
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
- https://cloud.google.com/pubsub/docs/overview
- https://www.rabbitmq.com/docs/consumers
- https://www.inngest.com/docs
- https://trigger.dev/docs
- https://docs.bullmq.io/
- https://docs.celeryq.dev/en/stable/
- https://docs.temporal.io/workflows
- https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
Top comments (0)