UlyssesBlack2385

Posted on Oct 10

Go Healthtech Event Notifications — Delayed Status Governs Email-to-SMS Fallback

#go #notifications #healthtech

TL;DR: When a healthtech marketplace has no delivery webhook, keep mail polling inside a narrow Go adapter and let one durable coordinator decide whether a new-order notice may fall back to text. A timeout means the result is unknown, not failed. Page on an order approaching its notification deadline, and put the order, current channel states, last successful poll, and permitted operator action in the alert.

At 03:00, the useful page is seller_order_notice_at_risk: order ord_7F2A, mail unknown, text not_started, deadline in 83 seconds, last successful status read 47 seconds ago. The useless page is "mail requests are slow." It leaves the on-call to discover the affected workflow, decide whether a duplicate notice is worse than a late one, and locate the component allowed to act.

This is an integration-boundary problem before it is a retry problem. The application should not learn every transport's status vocabulary, and the poller should not acquire permission to contact a seller by another channel merely because one read timed out. The coordinator owns policy; adapters own transport uncertainty. That division keeps a missing webhook from spreading special cases through order handling, scheduling, alerts, and recovery.

How should delayed email status trigger SMS fallback notifications?

Start with the action. The operator should be able to inspect one durable notification record, see whether the order still needs attention, and either allow the existing recovery policy to continue or pause it for a documented business reason. They should not be asked to infer delivery from a latency graph or manually send a text outside the workflow.

Work backward one step. Before the deadline page, a lower-urgency signal should report that the reconciliation worker has stopped advancing due records, or that the oldest unresolved order is consuming its reserved recovery window. An HTTP error counter is diagnostic context. It is not the customer-impact signal, because a high error rate can recover inside the deadline while a single stuck record can age quietly under an acceptable average.

What page fired?

That question disciplines the instrumentation. Emit a structured event on each coordinator decision with the order ID, attempt ID, channel, normalized result, decision reason, policy version, next check time, and remaining deadline. Exclude seller contact details from metrics and logs. The durable row is the audit record; metrics show cohorts, and traces connect a dispatch to later status reads.

The earlier warning and the page need different actions. The warning sends an engineer toward worker health and status-read failures. The page points to endangered orders and the recovery runbook. Don't combine them: the result fires often, explains little, and eventually gets treated as background noise.

Put the awkward integration behind one boundary

The smallest useful adapter can submit a logical attempt and reconcile its result. It must not choose the next channel, alter an order deadline, or hide an ambiguous timeout by returning a permanent failure. Those choices belong above the transport boundary, where they can be reviewed with marketplace policy and tested without a live email or SMS account.

package notice

import (
    "context"
    "time"
)

type Result string

const (
    Pending   Result = "pending"
    Delivered Result = "delivered"
    Rejected  Result = "rejected"
    Unknown   Result = "unknown"
)

type Submission struct {
    AttemptKey string
    OrderID    string
    Deadline   time.Time
}

type Receipt struct {
    ExternalID string
    Result     Result
}

type Transport interface {
    Submit(context.Context, Submission) (Receipt, error)
    Reconcile(context.Context, Receipt) (Result, error)
}

The AttemptKey is stable for a logical channel attempt and reused when the caller cannot tell whether submission succeeded. A network timeout only proves that the caller did not receive a complete answer. RFC 9110 distinguishes idempotent request semantics from the outcome of any particular connection, which is why retry behavior has to be designed deliberately rather than inferred from a socket error.

No guessing.

The coordinator persists the order event, policy version, deadline, channel attempt, and next reconciliation time before network work begins. A compare-and-swap version or database transaction guards every decision that starts the text path. Two workers can both wake after a lease expires; only one may claim escalation from the current state. This guard is part of the integration contract, because an adapter that encourages callers to treat an ambiguous submission as a clean failure makes duplicate contact likely under exactly the degraded conditions where the fallback is exercised.

Budget uncertainty instead of multiplying retries

Use separate clocks for a single request, the mail reconciliation window, and the end-to-end order-notification deadline. One timeout setting cannot express all three. For an illustrative five-minute internal deadline, a team might schedule status checks after 5, 15, 35, and 75 seconds with jitter, then stop early enough to preserve its tested text-dispatch budget. These are example policy values, not delivery guarantees; production values must come from the marketplace's required response window and observed integration behavior.

The decision code should stay boring:

package notice

import "time"

type Workflow struct {
    MailResult    Result
    TextClaimed   bool
    NextPollAt    time.Time
    DeadlineAt    time.Time
    TextReserve   time.Duration
}

func DueAction(w Workflow, now time.Time) string {
    if w.MailResult == Delivered || w.MailResult == Rejected {
        return "stop_mail"
    }
    if !now.Before(w.DeadlineAt.Add(-w.TextReserve)) {
        return "evaluate_text_policy"
    }
    if !now.Before(w.NextPollAt) {
        return "reconcile_mail"
    }
    return "wait"
}

evaluate_text_policy is intentionally not send_text. Reload the order, confirm that it still requires seller action, verify channel eligibility and consent, then atomically claim the text attempt. A late mail result remains evidence; it does not erase the claim or start another notification. Channel-local retries also stay local: a transient status-read failure can be retried within the mail budget, a permanent rejection ends that attempt, and a text failure does not restart mail.

Do not use a mail-open event as proof that the seller acknowledged the order. Apple explains that Mail Privacy Protection prevents senders from learning Mail activity and can download remote content in the background. If the workflow requires human acknowledgement, use an explicit marketplace action such as accepting the order.

Test the boundary before choosing the integration

Integration effort is not the number of SDK calls in a happy-path example. It is the amount of provider-specific behavior that escapes into the coordinator, plus the work required to reproduce uncertain outcomes before production. An integration is easier to own when every remote state maps to the four internal results, submission can be reconciled after a timeout, and raw responses can be retained outside high-cardinality metrics for diagnosis.

Use a fake clock and a scripted transport to test acceptance followed by repeated pending results, a submission timeout followed by delivered reconciliation, permanent rejection, cancellation before escalation, two workers competing for the text claim, and late mail delivery after that claim. Then run focused end-to-end checks against the real integrations in a non-production environment. The interface test proves coordinator behavior. It cannot prove how a remote service interprets an idempotency token or exposes a delayed status.

This is also where the no-webhook constraint should be challenged. If a transport offers authenticated push events that can be durably ingested and deduplicated, they can shorten reconciliation delay; a slower polling sweep is still useful for missed events. For a small, low-urgency workload, one indexed database scan may be less operational machinery than thousands of independent timers. I'd choose the scan until its measured delay threatens the deadline. The correct boundary survives either scheduler.

Deployment should be staged by policy version, with the version stored on every attempt. Watch unresolved age, transitions by reason, reconciliation-worker progress, and orders that reached more than one channel. Rollback changes the policy for new decisions without rewriting completed history. This is a clearer operational test than admiring a dashboard: can an engineer explain, from one record, why the text path was claimed?

A sensitive threshold creates two kinds of incident

Set the fallback boundary too early and sellers receive duplicate notices, text attempts rise, and the on-call is paged for mail that would have reconciled normally. Set it too late and the dashboard stays calm while the remaining recovery window collapses. Replay historical transition timing without contact data against candidate policies, then review both threatened deadlines and multi-channel outcomes; provider latency alone cannot choose the trade-off.

There is no honest zero-noise threshold. The alert should fire late enough that ordinary reconciliation has room to work, but early enough that the documented recovery action can still finish. Record the threshold as reviewed configuration, keep an operational margin, and require every page to name the endangered order and the next permitted action. Otherwise the integration has moved uncertainty onto the person holding the pager.

That cost is real. A threshold that produces repeated non-actionable pages trains the response team to discount the signal, while a threshold tuned only for quietness hides the order that needed attention. The right outcome is not the prettiest graph; it is a seller-notification workflow whose uncertain state is contained, inspectable, and recoverable before the business deadline.

DEV Community