Short answer: Choose polling when one small team needs private, controlled execution; choose HTTPS push when delivery latency matters enough to justify a public receiver; choose a queue subscription when sustained fan-out and explicit backpressure matter more than the fewest moving parts. For a fintech SaaS sending shipment updates to subscribers in Europe and the US, the default I would operate is a durable queue with regional consumers and an idempotent delivery boundary. It gives up some setup simplicity to keep latency, concurrency, and retry pressure visible.
I've been paged by both missed scheduled jobs and duplicate deliveries. That history produces one invariant: scheduling decides when work becomes eligible, but a durable handoff plus an idempotent receiver decides whether the shipment update survives retries. Treat those as separate state transitions.
Duplicates still happen.
How should a small SaaS choose public HTTPS webhook push, queue subscribe, or polling?
Start with the failure mode the team is actually equipped to own. A public HTTPS push receiver can be the shortest path from an eligible task to a subscriber notification. It also becomes an internet-facing production service: authentication, replay defense, rate limits, deployment overlap, and traffic bursts all arrive on that endpoint. Low idle cost and low delivery delay are attractive, but the receiver must turn a burst into bounded work instead of letting request concurrency become the queue.
A queue subscription places a durable buffer between scheduling and delivery. Consumers can take work at a rate their dependencies tolerate, and queue depth makes backlog concrete. This is usually the cleanest operational shape for shipment fan-out because one shipment version may create many subscriber deliveries. The catch is that the team now owns consumer scaling, poison-message policy, retention decisions, and a clear acknowledgement rule. It isn't the easiest setup for a service that sends a handful of non-urgent updates each day.
A polling worker keeps the worker private and gives it direct control over batch size and cadence. For a small SaaS with relaxed latency targets, that can be the easiest deployment: one worker asks for eligible jobs, leases a bounded batch, processes it, then sleeps. The cost is paid in empty reads, database or queue load, and delay up to the polling interval. Tightening the interval cuts delay while increasing idle work. That's the latency-versus-cost dial in its least disguised form.
| Delivery shape | Best fit | Operational catch | Latency/cost tendency |
|---|---|---|---|
| Public HTTPS push | Sparse events that need prompt delivery | Public endpoint and burst control | Low idle work; receiver capacity must absorb bursts |
| Queue subscription | Sustained or bursty fan-out | Consumer fleet and acknowledgement policy | Fast while capacity is available; buffer adds infrastructure |
| Polling worker | Low volume with relaxed deadlines | Empty polls and interval delay | Longer interval lowers idle work but raises delay |
For Europe and the US, don't turn geography into one transatlantic critical path by accident. Keep an authoritative shipment version, create one delivery record per subscriber, and route each record to the region responsible for that subscriber. Whether data is allowed to cross regions is a legal and product requirement, not something an SRE should infer from a cloud diagram. I'm not sure which residency boundary applies to your customers; a written classification from counsel and security resolves that uncertainty.
The incident lesson is an idempotency boundary, not a retry count
A retry policy without a stable operation identity converts temporary uncertainty into duplicate side effects. The useful identity in this system is not merely a queue message ID. Use business fields that remain stable when the same work is republished: shipment ID, subscriber ID, and shipment version. The tuple means, "deliver this version of this shipment to this subscriber." A later shipment version is new work; a redelivery of the same version is not.
The long paragraph matters because the awkward failure window sits between systems. A worker can lease a delivery, send the HTTPS request, and lose its process before recording completion. On retry, its local state cannot prove whether the subscriber accepted the first request. Holding a database transaction open across the network doesn't repair that ambiguity and creates a different operational risk. The subscriber-facing contract therefore needs the same idempotency key, while the worker ledger uses a lease so an abandoned attempt becomes eligible again. A successful subscriber response is followed by completion in the ledger; a timeout or rate-limit response releases the lease for a later attempt. If the remote subscriber cannot deduplicate, the honest guarantee is at-least-once attempts with possible duplicate effects. Don't label that exactly once.
Keep the boundary boring.
The acknowledgement point follows from that model. A consumer acknowledges only after the durable delivery state is complete. It does not acknowledge when work starts, and it does not retry indefinitely inside one process. When a subscriber returns an overload signal such as HTTP 429, schedule the next eligible attempt with bounded backoff and jitter, then free the worker slot. Invalid or permanently unauthorized destinations need a terminal state and an operator-visible reason rather than a hot retry loop.
A preventative Go delivery path
The interfaces below avoid tying the core rule to a queue vendor. Claim must atomically create or renew a bounded lease for the operation key. Send forwards that same key to the subscriber. A crash after the remote side effect but before Complete is safe only when the subscriber also treats the key idempotently.
package delivery
import (
"context"
"errors"
"fmt"
"time"
)
var ErrLeaseHeld = errors.New("delivery lease held")
type Update struct {
ShipmentID string
SubscriberID string
Version int64
Payload []byte
}
func (u Update) Key() string {
return fmt.Sprintf("shipment:%s:subscriber:%s:version:%d",
u.ShipmentID, u.SubscriberID, u.Version)
}
type Ledger interface {
Claim(ctx context.Context, key string, lease time.Duration) (bool, error)
Complete(ctx context.Context, key string) error
Release(ctx context.Context, key string, nextAttempt time.Time, reason string) error
}
type Sender interface {
Send(ctx context.Context, update Update, idempotencyKey string) error
}
type Handler struct {
Ledger Ledger
Sender Sender
Lease time.Duration
Delay func(attempt int) time.Duration
}
func (h Handler) Deliver(ctx context.Context, update Update, attempt int) error {
key := update.Key()
claimed, err := h.Ledger.Claim(ctx, key, h.Lease)
if err != nil {
return fmt.Errorf("claim %s: %w", key, err)
}
if !claimed {
return ErrLeaseHeld
}
if err := h.Sender.Send(ctx, update, key); err != nil {
next := time.Now().Add(h.Delay(attempt))
if releaseErr := h.Ledger.Release(ctx, key, next, err.Error()); releaseErr != nil {
return fmt.Errorf("send failed: %v; release failed: %w", err, releaseErr)
}
return fmt.Errorf("send %s: %w", key, err)
}
if err := h.Ledger.Complete(ctx, key); err != nil {
return fmt.Errorf("complete %s: %w", key, err)
}
return nil
}
The queue adapter around Deliver has one sharp rule: acknowledge on nil; otherwise leave or reschedule the message according to the ledger's next-attempt time. ErrLeaseHeld should not trigger concurrent work on the same operation. Exact adapter behavior depends on the queue's visibility and acknowledgement model, so verify it against the primary documentation rather than assuming all subscriptions expose the same controls.
Scheduling, deployment, and observability must share one state model
Delayed work starts as data. Store eligible_at, the stable operation key, attempt count, lease expiry, and terminal outcome. A scheduler promotes eligible records to the delivery queue; it does not perform subscriber I/O itself. That separation lets a delayed task remain inspectable during a deploy and stops a slow subscriber from consuming the scheduler's capacity.
Deploy consumers with overlap only after proving that the lease and idempotency paths work under concurrency. The minimum test suite races two claims for one key, kills a worker after Send but before Complete, advances a lease past expiry, and republishes the same operation. A separate load test should create one shipment update with a realistic subscriber fan-out and confirm that concurrency limits protect the slowest dependency. No benchmark number transfers cleanly between teams — payload size, subscriber response time, and regional network paths dominate it.
Observe the state transitions, not just process health. Queue age answers whether customers are waiting; queue depth without age can exaggerate a fast-moving burst. Track eligible-to-start delay by region, delivery completion latency, attempts per operation key, terminal failures by reason, active leases, and lease expirations. Alert on customer-impacting delay and a sustained failure ratio. A single retry is evidence for a trace, not necessarily a page.
Cost needs the same workload-shaped treatment. Estimate scheduled records, fan-out deliveries, empty poll reads, retained bytes, regional transfer, worker idle time, and peak concurrency. Then replay a representative day through each design. Public push often removes empty polls, polling makes the idle tax explicit, and subscriptions trade that tax for managed buffering plus active consumers. Your mileage may vary because subscriber distribution and burst shape move the answer more than an architecture label does.
When should you choose a different delivery shape?
Stick with a polling worker when delivery can wait for the polling interval, volume is low, and keeping workers private reduces more operational burden than empty reads create. Increase the interval until the observed eligible-to-start delay approaches the product objective, then leave headroom for backlog recovery. This is a good constraint, not a second-class architecture.
Use public HTTPS push when events are sparse, prompt delivery is important, and the team already operates authenticated public receivers with admission control. It is not suitable when a burst can exceed receiver capacity and there is no durable buffer before business processing. A push endpoint should validate and durably accept quickly; long subscriber work belongs behind its own queue.
Choose a queue subscription when fan-out is sustained or bursty, independent scaling matters, and operators need backlog visibility. Don't choose it merely because queues sound more serious. If the team cannot define acknowledgement, retention, terminal failure, and replay procedures, the extra component has created ambiguity rather than reliability.
The final decision rule is operational: select the least complex shape that meets the measured delivery deadline while preserving a durable handoff and an end-to-end idempotency key. Revisit it when fan-out, regional boundaries, or subscriber latency changes. The transport may change; the invariant should not.
Top comments (0)