DEV Community

robertmiller4179
robertmiller4179

Posted on

Fintech Shipment Updates: Daily Report Email Recovery for Failed Sends, Queue or Cron Rerun?

Short answer: for a fintech shipment update that fans out to many subscribers, start one daily report run on a schedule, enqueue one delivery per subscriber, and retry at the delivery boundary; reserve a full cron rerun for a missing or untrustworthy run, not for a handful of failed sends.

That choice keeps latency and cost visible. A queue can release slow recipients while fast ones finish, but it adds a worker, durable state, and an operator path for dead letters. A scheduler-only design is cheaper to operate when replaying the whole audience is harmless. The useful question is how much duplicate delivery and recovery time the business can tolerate.

I've been paged for missed jobs and duplicate deliveries. The runbook lesson is plain: give every shipment notification a stable delivery identity before discussing retry counts.

The incident lesson: the report run is not the delivery

Picture a daily report containing shipment updates for merchants and their subscribers. The report is generated correctly. During fan-out, one email provider response is slow and another send is rejected temporarily. Most messages are accepted. The job still gets marked failed because the handler waited for the whole batch.

The tempting repair is to rerun cron. That restores activity, but it also reconstructs the audience and may resend messages that already succeeded. A per-recipient queue retry narrows the blast radius: the failed delivery remains the unit of recovery while completed deliveries stay completed.

The failure boundary matters more than the failure label. Suppose the report run contains 18,000 shipment updates and the worker has sent 17,962 emails when its provider connection starts timing out. A batch-level status of failed tells an on-call engineer almost nothing: 17,962 messages may already be accepted, some may be in flight, and 38 may never have reached the provider. If the worker stored only a cron execution ID, a rerun has to rediscover that uncertainty while the provider sees another burst. If it stored one delivery record per subscriber, the recovery job can select the 38 unfinished records, leave accepted records alone, and separately inspect the in-flight set according to the provider's idempotency rules. That extra bookkeeping is work, but it converts a midnight guess into a bounded operation with a measurable end state: every expected delivery is sent, suppressed, or waiting for a human decision in the DLQ.

This is the invariant I want in a postmortem: a calendar run identifies the intended work, while a delivery record identifies the external side effect. The record needs a key such as report-date:shipment-update:subscriber-id:template-version. It should also retain the provider outcome, attempt number, and timestamps. Queue state alone cannot answer whether a message was accepted before a worker crashed.

At-least-once delivery changes the design. A worker can receive the same message again after sending and before acknowledging it. That is normal queue behavior, not evidence that the scheduler made two decisions. The send log must therefore make the second attempt a no-op when the delivery is already complete.

One identity. One owner.

How can a fintech team retry failed email sends with a queue before rerunning cron?

Split the workflow into three stages. The scheduler creates a report run for an explicit business date. The run writes or upserts delivery records and publishes small messages containing delivery IDs. Workers load the current record, decide if the send is still eligible, call the email provider, and persist the result before acknowledging the message.

The message should not be the only copy of the rendered report. Keep the payload small and put large content in a durable application-owned store, or regenerate it from a versioned report reference. This makes redrive inspectable and prevents a dead-letter operator from guessing which report a recipient was meant to receive.

Retry classification deserves more care than a single if err != nil. A temporary rate limit or transport timeout can be retried with capped exponential backoff and jitter. An invalid address, revoked subscription, or rejected template is a terminal outcome that belongs in the DLQ with a reason. A worker crash after the provider accepted the email is the dangerous middle case: the next delivery must consult the idempotency record before sending again.

The queue should own delay and attempt accounting. The application should own business identity and the decision to suppress a duplicate. A DLQ is an inspection lane, not a recycling bin. Redrive only after an operator can explain the terminal reason and confirm that the external side effect is still safe.

Where does cron rerun still make sense?

Cron rerun is appropriate when the scheduler never created a trustworthy run record, the report was not generated, or the entire batch is intentionally replayable. It is also a reasonable first implementation for a small internal audience where duplicate mail has no material consequence and there is no need for per-subscriber latency isolation.

The catch is that a rerun is a poor repair for partial fan-out. It has no natural boundary between accepted and rejected sends unless the application adds one, at which point it is recreating a delivery ledger anyway. A blind rerun can also turn an email provider slowdown into a larger burst, increasing both recovery time and downstream pressure.

Do not use a queue just because queues sound operationally mature. For a low-volume report, a single scheduled process with a durable send log and bounded in-process retries may have lower total cost and fewer things to page on. Stick with cron plus application retries when full replay is safe and the team can inspect the send log. Add a queue and DLQ when individual delivery latency, backpressure, or selective recovery matters.

Your mileage may vary: the right boundary depends on provider limits, subscriber count, and the business meaning of a duplicate notification. Measure those inputs in a staging load test before choosing worker concurrency.

What should the Go worker prove before it acknowledges a send?

The acknowledgement order is the part I would test in a failure-injection drill. Claim the delivery, send once, record completion, then acknowledge. If the process dies between the provider call and the completion write, the record remains recoverable and the next attempt must follow the provider's idempotency contract or suppress the send based on a confirmed external result. Never acknowledge before the durable state transition.

Here is a small, provider-neutral model of the state machine. It deliberately uses an in-memory map so the transitions are visible; production code needs a transactional durable store and a queue adapter.

package main

import (
    "errors"
    "fmt"
)

var errTemporary = errors.New("temporary provider response")

type deliveryStore struct {
    state map[string]string
}

func (s *deliveryStore) claim(id string) bool {
    if s.state[id] == "sending" || s.state[id] == "sent" {
        return false
    }
    s.state[id] = "sending"
    return true
}

func (s *deliveryStore) complete(id string) { s.state[id] = "sent" }
func (s *deliveryStore) release(id string)  { delete(s.state, id) }

type mailer struct{ attempts int }

func (m *mailer) send(string) error {
    m.attempts++
    if m.attempts == 1 {
        return errTemporary
    }
    return nil
}

func process(id string, store *deliveryStore, mail *mailer) string {
    if !store.claim(id) {
        return "ack duplicate"
    }
    if err := mail.send(id); err != nil {
        if errors.Is(err, errTemporary) {
            store.release(id)
            return "retry with backoff"
        }
        return "move to DLQ"
    }
    store.complete(id)
    return "ack sent"
}

func main() {
    store := &deliveryStore{state: map[string]string{}}
    mail := &mailer{}
    id := "2026-08-10:shipment-update:subscriber-42:v3"
    fmt.Println(process(id, store, mail))
    fmt.Println(process(id, store, mail))
    fmt.Println(process(id, store, mail))
}
Enter fullscreen mode Exit fullscreen mode

The output is a retry, a completed send, and an acknowledged duplicate. The example does not claim that an in-memory claim solves a distributed send race. It shows the contract to preserve when the store becomes transactional: a delivery must not move from sending to sent invisibly, and an acknowledgement must follow the durable completion.

Which recovery design fits the latency-versus-cost decision?

Use the smallest architecture that makes the failure boundary explicit.

Design Latency behavior Cost and operational trade-off
Scheduled process with bounded retries Simple for small batches; one slow send can delay the run Few moving parts, but full-run recovery is blunt
Schedule plus per-delivery queue Fast recipients finish independently; backpressure can be controlled Requires workers, queue metrics, retry policy, and a DLQ
Queue plus durable send ledger Selective redrive and duplicate suppression are explicit More storage and state transitions to test and operate
Full workflow orchestrator Useful for report generation, approvals, and dependent stages Too much machinery when each email is an independent delivery

Alert on age of the oldest pending delivery, retry rate, DLQ growth, and the gap between a scheduled run and its expected recipient count. Queue depth alone misses a queue that is draining slowly and a send log that is recording duplicates. Include the report date and delivery ID in logs so an on-call engineer can trace one subscriber without replaying the batch.

The decision rule is operational: choose the queue when selective recovery reduces expected incident work enough to justify its worker and state costs. Choose a cron rerun when the audience is small, replay is safe, and the send log is already sufficient. Either way, document who may redrive a DLQ item and what evidence they need first.

References

Top comments (0)