TL;DR: Treat a bulk notification as many independently advancing recipient deliveries, never as one queue item with one success flag. For each generated student report, persist an immutable dispatch intent, one email attempt, and any SMS notice attempt; accept callbacks or bounded polling as evidence, but translate them into your own monotonic state machine. A stuck queue then becomes a precise question: which recipient lacks which expected evidence by which deadline?
The important trade-off is evidence versus immediacy. Waiting for every downstream network to reach a terminal state makes workers fragile, while marking the batch complete after submission hides partial failure. Use asynchronous dispatch with a durable per-recipient ledger: submission ends the worker's synchronous responsibility; reconciliation owns everything after it.
How should batch event notifications handle partial email and SMS failure?
A batch-level sent field collapses events with different meanings. The application may have rendered an attachment, handed an email message to an SMTP relay, submitted an SMS notice, received a delivery-status notification, or observed a permanent rejection. SMTP permits delayed delivery and defines transient and permanent reply codes. Acceptance is evidence of transfer, not evidence that a person received or read the report.
This distinction matters for an education report. The attachment can contain education-record information, and FERPA governs access to education records for covered institutions. The design question is therefore not “did the job return 200?” but “can an auditor reconstruct what content was authorized, addressed, submitted, and resolved without placing sensitive report data in logs?”
Model the unit of work as (report_id, recipient_id, channel, content_revision). A class of 240 recipients produces at least 240 independently reconcilable email deliveries; SMS notices, when policy permits them, are separate rows rather than a fallback bit. This creates more records. It also prevents one malformed address from holding 239 valid deliveries hostage.
Do not store attachment bytes in the event log. Store a content digest, template revision, authorization decision identifier, destination fingerprint, and an object reference governed by the report retention policy. Mustache variables are HTML-escaped by default, while triple braces and ampersands render unescaped values. Record the template revision and prohibit unescaped student-controlled values.
Implement the evidence ledger in Go
The dispatch intent should be immutable. Attempts are append-only children. Current status is a projection rebuilt from attempts and observations, much as a financial balance is reconstructed from entries rather than trusted as an unexplained mutable number.
| State | Evidence required | Retry decision |
|---|---|---|
planned |
Authorized report and destination snapshot | Dispatch once |
submitted |
Downstream acceptance ID and timestamp | Reconcile; do not immediately resend |
deferred |
Transient result or expired evidence deadline | Retry with a new attempt number |
delivered |
Channel-specific delivery observation | Terminal for automation |
failed |
Permanent rejection or exhausted policy | Terminal; create an operator task |
suppressed |
Consent or policy forbids dispatch | Terminal without submission |
Three rules keep the projection honest. Terminal observations cannot move backward when a delayed callback arrives. Every transition retains the source observation and receipt time. An idempotency key identifies the intent, while an attempt number identifies a particular submission. Reusing one identifier for both makes a harmless replay indistinguishable from a deliberate retry.
package delivery
import (
"crypto/sha256"
"encoding/hex"
"fmt"
)
type Intent struct {
ReportID, RecipientID, Channel, ContentRevision string
}
type Status string
const (
Planned Status = "planned"
Submitted Status = "submitted"
Deferred Status = "deferred"
Delivered Status = "delivered"
Failed Status = "failed"
Suppressed Status = "suppressed"
)
func IntentKey(i Intent) string {
material := fmt.Sprintf("%s\x00%s\x00%s\x00%s",
i.ReportID, i.RecipientID, i.Channel, i.ContentRevision)
sum := sha256.Sum256([]byte(material))
return hex.EncodeToString(sum[:])
}
func CanAdvance(from, to Status) bool {
if from == Delivered || from == Failed || from == Suppressed {
return false
}
allowed := map[Status]map[Status]bool{
Planned: {Submitted: true, Suppressed: true, Failed: true},
Submitted: {Delivered: true, Deferred: true, Failed: true},
Deferred: {Submitted: true, Failed: true, Suppressed: true},
}
return allowed[from][to]
}
Enforce uniqueness on the intent key and (intent_key, attempt_number). Commit an attempt before external submission, record the downstream identifier afterward, and leave ambiguous outcomes for reconciliation. A process can die after the external system accepts a message but before its identifier is stored. No local transaction makes that network call atomic. Use idempotent submission when available, then look up or reconcile before retry; claiming exactly-once delivery across that boundary would be false precision.
A troubleshooting worksheet for the stuck queue
“Stuck” needs a clock and an expected next event. An old planned row indicates local scheduling trouble. An open attempt without a submission result indicates an ambiguous handoff. A submitted row beyond its evidence deadline belongs to reconciliation, not automatic retransmission. A deferred row whose next_attempt_at has passed is retryable. These are four faults even if a dashboard paints all of them yellow.
Do not resend yet.
Start with recipient rows, not the batch header. Group them by state, attempt count, and age bucket, then sample the immutable observations behind each group. Check destination normalization, consent decisions, attachment generation, payload-size policy, relay response classes, callback authentication failures, and reconciliation lag. Keep report titles, student names, addresses, phone numbers, and attachment contents out of metric labels. A keyed destination fingerprint can correlate restricted logs without turning observability storage into another student-record system.
Count facts.
A useful service-level measure is the age of the oldest row awaiting an expected transition, partitioned by channel and state. Queue depth alone misleads: ten thousand newly planned notices may be healthy, while one submitted report without an observation beyond its deadline may represent unresolved evidence. Alert on the latter and include the intent key, never the raw destination.
Callbacks should be the normal evidence path when supported. Polling remains valuable for reconciliation: use a bounded schedule, stop on terminal states, add jitter, and persist the last checked time. Deduplicate repeated observations by stable event identity. Event time plus transition rules prevents late observations from regressing status.
Email and SMS require different interpretations. An SMTP delivery-status notification follows RFC 3464, but its absence is not affirmative delivery proof. An SMS network receipt, where available, is transport evidence rather than proof that the intended guardian read the notice. Record the literal evidence, its source, and the mapping rule; do not rename every positive signal read.
Email authentication and sender practices also belong in diagnosis because deliverability can fail outside the queue. Yahoo's published bulk-sender requirements cover authentication, complaint rates, and unsubscribe behavior. Those requirements do not establish delivery of a particular report, but their signals can explain a population-wide change without corrupting the recipient ledger.
Recovery matrix: preserve the audit consequence
Bulk replay is fast and difficult to defend: successful recipients may receive duplicate attachments, and a second send obscures which attempt satisfied the original intent. Prefer the narrowest action justified by evidence.
Ambiguity is a state.
| Observation | Action | Evidence preserved |
|---|---|---|
| No attempt was claimed | Requeue the original intent | Same intent, first attempt |
| Handoff is ambiguous | Reconcile before retry | Uncertainty remains explicit |
| Transient rejection | Schedule a numbered retry | Rejection and later attempt |
| Permanent rejection | Stop and open correction workflow | No repeated send to a bad destination |
| Email delivered; SMS unresolved | Reconcile SMS independently | Separate channel histories |
Put retry ceilings, evidence deadlines, legal retention, and escalation ownership in versioned configuration. Compliance teams decide what evidence and retention the institution requires; engineers encode those decisions. Do not infer a universal retention period from FERPA, because institutional obligations and other applicable rules vary.
Testing should target disorder. Feed duplicate observations, reverse their arrival order, crash between attempt creation and submission result, expire a worker lease, and make one recipient fail among 240 successes. The invariant is stronger than “the job eventually finishes”: no recipient gets another submission without a recorded decision, no terminal state regresses, and every displayed status traces to stored evidence.
A three-stage migration without replaying old reports
Begin in shadow mode. Populate intent and observation records for new report runs while the existing dispatcher remains authoritative, then compare recipient counts and projections without sending from the new path. Move one cohort to ledger-controlled retries, with a kill switch that stops new submissions but leaves reconciliation running. Finally, derive the batch header from recipient rows and remove any control that resends an entire batch.
Backfill historical batches only as historical facts: label provenance as imported and mark unknown outcomes unknown. Do not manufacture recipient results from a batch-level sent flag. Retry only the recipient-channel intent with retryable evidence; reconcile ambiguity; escalate permanent failure. That rule turns a vague stuck queue into a bounded accounting problem and leaves an audit trail fit for operational and compliance review.
Sources
References:
Top comments (0)