Short answer: for a weekly digest sent to active developer-tool customers, choose the delivery shape that makes operational recovery explicit. Use a public HTTPS push subscription when an existing webhook processor can authenticate, persist an attempt, and return quickly; use a polling consumer when the processor is private or when the application must own leasing and retry timing. In both cases, the durable boundary is an idempotent customer-send record plus an audit trail, not the transport.
This is an architecture decision record about recovery, not a contest between queue interfaces. A digest may be generated once and delivered twice, because a worker can lose its connection after the provider accepts the request but before the acknowledgement arrives. The business result still must be one digest per customer and period. That means the system needs a uniqueness rule such as (customer_id, digest_week), an attempt record for every delivery, and a reconciliation path that can distinguish “not tried,” “accepted,” “timed out,” and “already applied.” Exactly-once effects are an application property built on at-least-once delivery.
What should a webhook processor record when polling consumers retry failed jobs?
The first review question is not latency. It is: who can restart work after a partial failure?
With push, an external delivery service reaches a public HTTPS handler. The handler verifies the request, records the delivery identifier, performs bounded work, and acknowledges only after the digest state is committed. A handler that returns success before its database transaction is durable creates a recovery hole: the delivery system may stop retrying while the customer still has no recorded send.
With polling, a worker reaches the queue from an outbound network connection, leases a message, commits the idempotent send record, and acknowledges last. The worker is easier to keep private, but its liveness, lease duration, shutdown behavior, and retry schedule become your responsibility. That is a reasonable trade when security policy forbids public ingress or when the retry decision depends on application state that changes between attempts.
The weekly schedule belongs outside this decision. A scheduler should enqueue a digest job with a stable period identifier; it should not perform the customer send itself. If the scheduler runs again after a deploy, the uniqueness constraint turns the repeated enqueue into an observable duplicate rather than a second customer-facing effect.
Compare push versus polling recovery boundaries
Compare the options by failure ownership. The table is intentionally narrower than a feature checklist.
| Option | Network shape | Recovery owner | Suitable when | Main limitation |
|---|---|---|---|---|
| Public HTTPS push subscription | Delivery service calls your endpoint | Handler and delivery policy | The webhook processor is already public and independently monitored | Public ingress, authentication, and burst capacity are part of the queue design |
| Polling consumer | Worker calls the queue | Worker, lease policy, and acknowledgement logic | The processor must stay private or needs application-controlled timing | A healthy process and queue metrics must be maintained |
| Broker dead-letter route | Broker moves rejected messages | Broker topology plus an operator | A team already operates a broker and needs a clear redrive boundary | Dead letters do not replace business reconciliation or idempotency |
Neither option prevents duplicates. A timeout after the email provider accepted a request is indistinguishable from a timeout before acceptance unless the provider offers a durable idempotency key. Generate that key from the customer and digest period, store it with the attempt, and use the same key on every retry. Record the response classification separately from the business result; a transport success is evidence about delivery, not proof that the customer read the message.
This is where compliance concerns become practical. An audit trail should answer who or what scheduled the digest, which customer-period key was processed, which attempt number ran, what external request identifier was used, and why the next retry was or was not scheduled. Logs alone are a poor substitute because retention, redaction, and sampling policies can remove the very evidence needed during reconciliation.
Code path for a repeatable customer send
The critical path should be deliberately uneventful: receive, validate, claim, apply, record, acknowledge. The claim and application must be safe to repeat. Here is a generic polling shape; the queue adapter is intentionally separate from the customer-send transaction.
package main
import (
"context"
"database/sql"
"fmt"
)
type Job struct {
ID string
CustomerID string
Period string
}
type Queue interface {
Receive(context.Context) (Job, error)
Ack(context.Context, string) error
Retry(context.Context, string, string) error
}
func handle(ctx context.Context, db *sql.DB, q Queue, send func(context.Context, Job, string) error) error {
job, err := q.Receive(ctx)
if err != nil {
return err
}
key := fmt.Sprintf("%s:%s", job.CustomerID, job.Period)
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
// The unique customer-period key makes a redelivery harmless.
result, err := tx.ExecContext(ctx,
`INSERT INTO digest_sends (customer_id, period, status)
VALUES (?, ?, 'claimed') ON CONFLICT DO NOTHING`,
job.CustomerID, job.Period)
if err != nil {
return err
}
if n, _ := result.RowsAffected(); n == 0 {
return q.Ack(ctx, job.ID)
}
if err := tx.Commit(); err != nil {
return err
}
if err := send(ctx, job, key); err != nil {
_ = q.Retry(ctx, job.ID, err.Error())
return err
}
if _, err := db.ExecContext(ctx,
`UPDATE digest_sends SET status = 'sent' WHERE customer_id = ? AND period = ?`,
job.CustomerID, job.Period); err != nil {
return err
}
return q.Ack(ctx, job.ID)
}
The example leaves one important policy visible: claiming before the external send can create a stuck claimed row if the process dies. That is not a reason to remove the claim. Add a lease expiry and a reconciliation job that can safely re-open stale claims, while retaining the original attempt history. The alternative is to send first and write afterward, which makes duplicate sends more likely when the write fails. There is no transaction spanning your database and an arbitrary mail or webhook provider, so the design must expose this boundary instead of pretending it is atomic.
Three words: ack last.
Privacy constraints on private ingress
The rejected option in this ADR is public push. Select polling when the webhook processor has no permitted public ingress, when a private network is a hard requirement, or when an operator needs to pause leasing during reconciliation. A polling worker is also the clearer choice when the retry schedule depends on a customer’s current subscription state and the application must inspect that state before re-enqueueing.
Push is the better operational shape when the processor already has public HTTPS ingress, can reject unauthenticated requests before changing state, and has independent alarms for delivery silence and handler saturation. It reduces the need to run a continuously pulling process, but it does not remove capacity planning, deduplication, or audit requirements.
The catch is recovery ownership. Push is not suitable when nobody owns the public endpoint or when the delivery policy cannot be inspected during an incident; stick with a private polling worker until those controls exist. A broker with dead-letter exchanges can also be a sound choice for a team that already operates that topology, although dead-lettering still needs an application-level redrive rule and a ledger of outcomes.
Test reconciliation as the schedule's second clock
Test the failure boundaries, not only the happy path. A useful test kills the worker after the external provider accepts the idempotency key, then checks that a redelivery creates one customer-period effect and two auditable attempts. Another pauses the consumer beyond its lease, starts a second consumer, and verifies that the claim rule prevents concurrent business effects. A third replays an old scheduler message after the period has closed and confirms that the system records the decision without sending a second digest.
Metrics should make recovery visible: age of the oldest pending job, attempts by outcome, stale claims, acknowledgement latency, and the count of customer-period conflicts. Alerting on queue depth alone misses a silent public endpoint and a worker that is acknowledging messages while its final status update fails.
I'm not sure a queue dashboard, by itself, can answer every audit question; retention and export behavior vary, and the question should be verified before approval. Keep the durable evidence in an application-owned attempt table, then use the queue's metrics as operational signals. Your mileage may vary on retry intervals, but the decision rule should remain stable: select the ingress that gives the team a clear, testable recovery owner.
Top comments (0)