DEV Community: oskarholm4968

Frontend Backend Error Tracking: How to Choose Practical Exceptions Without Session Replay

oskarholm4968 — Mon, 19 Oct 2026 17:08:03 +0000

A checkout error system should preserve enough evidence to answer one operational question: did the logistics order fail before acceptance, after acceptance, or only in the browser? Collecting traces and replay by default would add volume without necessarily improving that answer. The practical choice is a deliberately small exception pipeline with a shared correlation key, stable failure classes, aggressive field allowlisting, and separate alerts for customer impact and pipeline health. It covers browser and API exceptions while keeping signal quality ahead of noise.

Short answer: capture handled and unhandled exceptions at both boundaries, attach an opaque checkout attempt ID, normalize them into one vendor-neutral envelope, and alert only on failures that cross a defined impact threshold. Do not attach form values, addresses, authorization headers, cookies, stack-local variables, or replay data. A useful event explains the failed transition; it does not reconstruct the person.

How should frontend and backend error tracking shape the practical choice?

Start with the state machine, because an exception count alone cannot distinguish a rejected request from an accepted order whose response was lost. For a logistics checkout, the relevant transitions might be cart_ready -> submitting -> accepted, with explicit terminal failures such as validation_rejected and dependency_unavailable. The event should record the attempted transition and whether the backend assigned an order ID. That distinction protects reconciliation: retrying a request that was already accepted can be more damaging than showing an error.

A React or Next.js frontend and a Node API do not change this boundary. They change where adapters run, while the normalized event and the acceptance record remain technology-independent. The same design also avoids making session replay a prerequisite for simple exception tracking.

This is an exactly-once reasoning model, not a claim that exception delivery itself is exactly once. Error events can be duplicated, delayed, or dropped, so the checkout's idempotency key remains part of business correctness while the event ID serves only observability deduplication. Keep those identifiers separate.

The minimum envelope needs surprisingly little: event ID, observed timestamp, source, release, environment, failure class, operation, checkout attempt ID, order ID when one exists, and a sanitized error description. Severity also needs one shared interpretation. RFC 5424 defines ordered severity values from Emergency through Debug; using a documented subset across browser and backend prevents a frontend warning from competing with an API failure merely because two libraries chose different labels.

No payload dump.

The compliance boundary should be an allowlist enforced before transport, not a promise that engineers will avoid searching sensitive fields later. Treat free-form messages as suspect, cap their length, and reject unexpected attributes. An EU deployment and a US deployment may have different retention or routing policies, but both benefit from collecting fewer identifying fields. A GDPR review must set the actual retention and transfer limits for the deployment; the event schema should make those limits enforceable rather than pretending that a technical pattern settles a legal question.

Build one narrow event contract

The following Go types model the ingestion boundary even when the emitting client is a browser. The browser can send the same JSON shape; the receiving service validates it, supplies trusted deployment metadata, and refuses extra fields. FailureClass stays low-cardinality so it can drive aggregation, while Message remains diagnostic text and must never become a metric label.

package errorsignal

import "time"

type Event struct {
    EventID           string    `json:"event_id"`
    ObservedAt        time.Time `json:"observed_at"`
    Source            string    `json:"source"` // browser or api
    Release           string    `json:"release"`
    Environment       string    `json:"environment"`
    Operation         string    `json:"operation"`
    FailureClass      string    `json:"failure_class"`
    Severity          string    `json:"severity"`
    CheckoutAttemptID string    `json:"checkout_attempt_id"`
    OrderID           string    `json:"order_id,omitempty"`
    Message           string    `json:"message"`
}

var allowedFailureClasses = map[string]struct{}{
    "validation_rejected":    {},
    "dependency_unavailable": {},
    "unexpected_exception":   {},
    "response_lost":          {},
}

The contract is intentionally boring. Adding carrier name, route, warehouse, customer country, or raw URL may feel helpful during the first incident, yet each extra dimension multiplies grouping choices and can expose business or personal context. Consider a browser timeout after the backend accepted the order: a raw request dump might contain everything needed to recognize the customer, but the operational decision depends on only three facts, namely that the attempt ID reached the API, an order ID was assigned, and the response did not reach the browser. Those facts direct the responder toward response delivery and suppress an unsafe retry. The remaining payload adds exposure without changing the decision, so it should not cross the capture boundary. Add a field only when an operator can name the decision it changes.

Validation belongs at ingestion. Decode exactly one object, reject unknown keys, verify bounded values, and overwrite fields such as environment from server-side configuration rather than trusting the caller.

Bound it early.

package errorsignal

import (
    "encoding/json"
    "errors"
    "io"
    "strings"
)

func Decode(r io.Reader) (Event, error) {
    dec := json.NewDecoder(io.LimitReader(r, 32<<10))
    dec.DisallowUnknownFields()

    var event Event
    if err := dec.Decode(&event); err != nil {
        return Event{}, err
    }
    if event.EventID == "" || event.CheckoutAttemptID == "" {
        return Event{}, errors.New("missing required identifier")
    }
    if _, ok := allowedFailureClasses[event.FailureClass]; !ok {
        return Event{}, errors.New("unsupported failure class")
    }
    if len(event.Message) > 512 || strings.Contains(strings.ToLower(event.Message), "authorization") {
        return Event{}, errors.New("unsafe error message")
    }
    return event, nil
}

The 32 KiB body cap and 512-character message cap are capacity limits, not proof of safety. That substring check is a last guard, not a complete redaction system. The stronger control is constructing messages from approved templates and keeping arbitrary exception values out of the event. Stack frames can be retained only when the organization has reviewed their contents and retention; they should not be presumed harmless.

Separate capture, storage, and alerting

An implementation has three failure domains. Capture runs close to the browser or API boundary, ingestion authenticates and normalizes, and storage supports grouping and audit. Alert evaluation reads aggregated outcomes rather than every event. This division lets the checkout continue when observability is unavailable and prevents an alert rule from becoming part of request latency.

Noise wins otherwise.

Delivery should be bounded. In the browser, enqueue the sanitized envelope without blocking navigation; on the API, use a short buffered handoff and increment a local drop counter when the buffer is full. Never retry forever. The business request deserves the latency budget, while the signal pipeline must expose its own loss rate so silence is not mistaken for health.

Deduplicate storage by event_id, but group investigations by a fingerprint built from stable fields such as source, operation, failure class, and release. Do not hash the entire message: order numbers or dependency text can split one defect into thousands of groups. Conversely, grouping every exception by status code can merge unrelated defects. The right fingerprint follows the repair unit.

Metrics are derived views. Prometheus naming guidance recommends a common prefix, base units, and labels whose combinations do not grow without bound. A counter such as checkout_failures_total{source,failure_class} is defensible because both labels have controlled sets. checkout_failures_total{message,order_id} is not; those values belong in restricted event storage, if they belong anywhere.

For auditability, record schema version and policy outcome alongside accepted events, then make retention deletion observable as an aggregate operation rather than another copy of the deleted content. Access to raw exceptions should also be narrower than access to aggregated dashboards. These controls support review, but they do not by themselves establish compliance.

Choose thresholds from customer impact

Page on failed outcomes, not on exception novelty. One practical rule is a rolling ratio whose numerator is checkout attempts ending in an unexpected server failure and whose denominator is all eligible submission attempts; require a minimum denominator before firing. This avoids a single failure waking someone during a quiet interval while still catching a broad regression. The actual window, minimum volume, and threshold must come from the service objective and traffic pattern, not from a universal number.

Browser errors need a different route. A burst in one release can open an investigation, but it should page only when it correlates with abandoned submissions or missing backend acceptance. A browser exception after an order was accepted is still worth fixing, although its operational priority differs from a failure before acceptance.

Signal	Group by	Operational action
API unexpected exception	operation, failure class, release	Page when the impact rule is met
Browser submission exception	release, operation	Investigate; page only with checkout impact
Validation rejection	stable reason class	Review trend, usually no page
Dropped observability events	source, environment	Repair signal integrity

The limitation is deliberate: a simple exception system cannot answer request-waterfall or user-session questions, and in exchange it gives a smaller, reviewable data surface. This trade-off makes it unsuitable when responders repeatedly need cross-service timing to determine ownership; distributed tracing has then become justified. It is also unsuitable when teams must reproduce visual interaction defects that cannot be described by state transitions, in which case session replay may be justified after privacy review. Those are evidence-based expansions, not default checkboxes.

Roll out without losing the audit trail

Begin in report-only mode: validate events, count rejected fields, and compare accepted checkout states with captured failures without paging. Then enable one API impact alert, followed by browser correlation, while retaining the old operational signal during an overlap window defined by the team's release cadence. Rollback means disabling the new alert path; it must not alter checkout processing.

Before broader deployment, test duplicate event IDs, malformed JSON, oversized bodies, unknown fields, an accepted order followed by a lost response, and an unavailable ingestion service. The expected result is crisp: checkout correctness remains governed by its idempotency and reconciliation records, accepted events remain deduplicated, and observability loss appears in a bounded metric.

Finally, review the allowlist and access policy with security, privacy, and operations owners. The best practical choice is the smallest system that preserves the transition evidence responders actually use. Expand it only after a documented incident question proves the missing capability.

Sources

Low-Cost SMS Alert Service Explained — 4 Go Boundaries for Passwordless Backup Account Notifications

oskarholm4968 — Sat, 17 Oct 2026 12:08:38 +0000

TL;DR: Choose an SMS service for passwordless backup alerts and account notifications only after it can produce the evidence your edtech system must retain. For a generated learner report sent as an email attachment, keep four records separate: consent or another approved processing basis, authentication challenge state, report-email state, and SMS delivery observations. Compare Twilio, Vonage, and Telnyx with the same evidence request and controlled test; do not infer compliance, delivery, or security from a low advertised message price.

This is an architecture decision record for a developer portal serving US and EU users. The decision is to own the compliance ledger and treat each communications provider as a replaceable source of observations. That rule sounds conservative because it is: GDPR Article 7 assigns the controller the burden of demonstrating consent when consent is relied upon, while the OWASP Forgot Password guidance requires controls around reset tokens and advises against changing the account before a valid token is presented. A transport receipt answers neither obligation by itself.

How should a low-cost SMS alert service handle passwordless backup account notifications?

Start with reconstruction, not transmission. Given one internal event ID, an authorized reviewer should be able to establish why a recipient was contacted, which channel was requested, which policy applied at that time, whether the report email and the SMS alert diverged, and what later observations arrived. The reviewer should not need raw one-time codes, the learner report, or a phone number copied through routine worker logs.

Four evidence boundaries follow:

Authority: record the purpose, policy version, recipient reference, region classification, and the evidence supporting the chosen processing basis. If consent is the basis, its withdrawal must be as easy to account for as its grant; Article 7 also says consent must be distinguishable, intelligible, and given in clear language.
Authentication: record challenge creation, expiry, attempt outcomes, and consumption without retaining the plaintext code. OWASP says tokens or codes should be generated with a cryptographically secure algorithm, be sufficiently long, be stored securely, be single-use, and expire after an appropriate period.
Document delivery: record report generation, integrity checking, attachment creation, and mail handoff as a lifecycle independent of SMS. The SMS should say that a report is available, not contain report data.
Transport observation: append provider acceptance, callbacks, terminal status, and reconciliation outcomes. Acceptance is evidence of an accepted request. It is not proof that the intended person read the message.

This ordering prevents a common modeling error: one sent Boolean cannot describe four systems with different failure boundaries. A report may be generated while attachment creation fails. An email may be handed off while an SMS request is rejected. A delayed SMS observation may appear after the authentication challenge has expired. None of those facts should overwrite another.

Short records matter.

Compare evidence, not brochure categories

The requested comparison names Twilio, Vonage, and Telnyx, but the supplied facts do not support declaring a universal winner. Their applicable contracts, configured processing locations, sender-registration paths, retention controls, webhook security mechanisms, and status vocabularies must be reviewed in their current documentation and then tested under the same workload. Those are objective differences to capture; they are not safe to assume from a brand name.

Use one dated matrix for all three candidates:

Candidate	Contract and data evidence	Authentication evidence	Delivery and export evidence	Decision state
Twilio	Record the applicable terms, subprocessors, configured region, retention, and deletion procedure	Verify current webhook authentication guidance and whether application-owned expiry, one-time use, and attempt limits remain enforceable	Test stable correlation, duplicate events, late events, and export availability	Pass, fail, or unresolved per deployment
Vonage	Request the same artifacts for the same US/EU scope	Run the same challenge-state and callback tests	Reconcile the same controlled message corpus against internal IDs	Pass, fail, or unresolved per deployment
Telnyx	Request the same artifacts for the same US/EU scope	Apply the identical security acceptance criteria	Measure evidence completeness with the identical reconciliation rules	Pass, fail, or unresolved per deployment

The table is intentionally asymmetric in only one respect: each provider will return its own artifacts and observations. The questions do not change. A candidate fails the compliance-evidence gate when an accountable owner cannot retrieve a required artifact, a control cannot be enforced, or a test result cannot be reconciled; an unresolved answer stays unresolved instead of becoming a favorable score.

Cost enters after that gate. Compare total operational cost for the actual destination mix and sender configuration, including failed attempts, registration work, evidence retention, support, and reconciliation labor. Public unit prices are too narrow to support the architectural decision, and they change independently of the evidence obligations.

No price repairs a missing audit trail.

Model the ledger before writing the adapter

The critical path begins when the application commits an immutable intent. It does not begin with an SDK call. The minimal Go example below creates no provider-specific contract, stores no report content, and refuses to pretend that a repeated business event is new work. A production AppendIntent implementation must enforce uniqueness on EventID; an existing ID with a different digest is a collision requiring investigation, while an identical record is an idempotent replay.

package evidence

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "time"
)

type Intent struct {
    EventID       string
    RecipientRef  string
    Purpose       string
    PolicyVersion string
    PayloadDigest string
    ExpiresAt     time.Time
}

type Observation struct {
    EventID    string
    ExternalID string
    Kind       string
    ObservedAt time.Time
}

type Ledger interface {
    AppendIntent(context.Context, Intent) (created bool, err error)
    AppendObservation(context.Context, Observation) error
}

type Transport interface {
    Send(context.Context, string, string, string) (string, error)
}

type Service struct {
    ledger    Ledger
    transport Transport
    now       func() time.Time
}

func (s Service) SendReportAlert(
    ctx context.Context,
    eventID string,
    recipientRef string,
    phoneToken string,
    policyVersion string,
) error {
    if eventID == "" || recipientRef == "" || policyVersion == "" {
        return errors.New("missing evidence identity")
    }

    body := "Your learner report is ready. Sign in to the portal to view it."
    digest := sha256.Sum256([]byte(body))
    intent := Intent{
        EventID:       eventID,
        RecipientRef:  recipientRef,
        Purpose:       "report_ready",
        PolicyVersion: policyVersion,
        PayloadDigest: hex.EncodeToString(digest[:]),
        ExpiresAt:     s.now().Add(15 * time.Minute),
    }

    created, err := s.ledger.AppendIntent(ctx, intent)
    if err != nil {
        return err
    }
    if !created {
        return nil
    }

    externalID, err := s.transport.Send(ctx, eventID, phoneToken, body)
    if err != nil {
        return s.ledger.AppendObservation(ctx, Observation{
            EventID: eventID, Kind: "request_failed", ObservedAt: s.now(),
        })
    }

    return s.ledger.AppendObservation(ctx, Observation{
        EventID: eventID, ExternalID: externalID,
        Kind: "request_accepted", ObservedAt: s.now(),
    })
}

The 15-minute value is an application policy in this example, not a universal recommendation or a provider guarantee. Record the policy version so a future reviewer can distinguish a code defect from a deliberate policy change. The phoneToken is an opaque reference resolved inside the adapter, which narrows exposure in queue payloads; it does not remove the controller's duties for the underlying personal data. Similarly, SHA-256 supplies a comparison digest here, not encryption or authorization.

There is an uncomfortable edge at the external boundary. If a provider accepts the request and the process loses its connection before recording the returned ID, a retry can create a second message unless that transport supports and honors the application's idempotency key. The ledger must preserve the ambiguous outcome. Exactly-once business intent is enforceable in the application database; exactly-once handset delivery is not a claim this design makes.

Operate it as a reconciliation system

Testing should interrogate evidence completeness. Replay the same callback twice. Deliver a later status before an earlier status. Submit an unknown external ID. Expire a challenge before a late delivery observation arrives. Confirm that routine logs expose neither the one-time code nor contact data, and verify that report-email events can be reconstructed without joining them to SMS payloads.

Deployment proceeds by destination class and sender configuration because one route does not establish another route's behavior. Canary traffic should use synthetic accounts and should validate correlation, redaction, and callback handling before real account notifications move. The useful operational signals are aging intents, unmatched external IDs, duplicate observations, expired challenges followed by delivery, and divergence between the report-email lifecycle and its optional alert. Raw send count is an activity measure, not evidence of correctness.

Retention deserves an explicit schedule. Keep the minimum fields needed for the stated purpose and legal basis, restrict access, and document deletion behavior; do not preserve authentication secrets merely because audit records are valuable. GDPR Article 7 supplies specific consent conditions, but it does not convert every communication purpose into consent, so the organization's legal owner must identify and document the applicable basis rather than letting application code guess.

The rejected shortcut still has one valid home

The rejected option is a synchronous controller that generates the report, attaches it to email, sends the message, sends an SMS, and returns one success value. It resembles a transaction while spanning systems that do not share the application's database commit. A retry can duplicate one channel, and suppressing the retry can lose the other; worse, the single result erases which obligation was actually satisfied.

That shortcut remains valid for a local template preview using sink transports, synthetic recipients, and no assertion about delivery. It gives a developer quick visual feedback without becoming an evidence model.

For production, the report-ready state and an outbox record should commit in one database transaction, while separate workers advance email and SMS observations with bounded retries and reviewable dead letters. SMS should remain a backup channel rather than the only recovery route, since the OWASP guidance explicitly recommends a consistent forgot-password process, side-channel delivery of reset material, single-use expiry, and protection against excessive submissions. The final decision packet should contain the four-boundary data model, the dated three-candidate matrix, test evidence, named control owners, and unresolved risks. That packet is the product of the comparison. The provider is one replaceable dependency inside it.

This design has limitations. Its audit quality depends on disciplined policy versioning, access control, retention work, reconciliation ownership, and review of ambiguous outcomes; a small team that cannot operate those controls should choose an independently assessed managed authentication system for the passwordless flow and keep report email separate. The trade-off is direct control over evidence in exchange for database, queue, privacy, and on-call responsibilities. It also remains unsuitable wherever SMS cannot be an acceptable backup factor under the organization's security assessment.

References

Implementing a Beginner 2FA Login Stack with an SMS OTP API (Audit Constraints)

oskarholm4968 — Thu, 15 Oct 2026 23:41:54 +0000

Use managed SMS OTP for warehouse pickup codes when plain SMS is an acceptable channel, but put a small application-owned ledger around it: suppression before issuance, one stable attempt identity, bounded status polling, and a terminal decision that routes failures to the correct support queue. The least complex viable design is not the one with the fewest database columns. It is the one that prevents a retry from becoming a second charge or, worse, a second live challenge.

TL;DR: For a beginner SaaS operating in the US and EU, the dominant variable cost is the message you choose to issue; polling adds work, but uncontrolled resend paths multiply the expensive operation. Keep one row per pickup attempt, never store the OTP, and retain only enough evidence to reconcile issuance, verification, suppression, and the eventual parcel-release decision. Infrai fits when integration effort is the deciding axis because SMS sits behind the same REST contract as many other backend capabilities, while a verification specialist is the sounder choice when fraud controls, pushed events, or non-SMS fallback are requirements.

What actually creates the bill and the audit burden?

Model the workload before choosing a provider. For pickup attempt a, the relevant external operations are:

suppression_check(a) + otp_issue(a) + verification_attempts(a) + status_reads(a)

Those terms do not deserve equal attention. A status read observes an existing operation; an OTP issue creates a new message and a new challenge lifecycle. Therefore the first invariant is strict: one business attempt may cause at most one logical issuance, even when the network times out and the caller cannot tell whether the first request succeeded. Exactly-once is a business property here, assembled from a stable attempt ID, an idempotency key, and reconciliation. It is not a claim about exactly-once packet delivery.

I would store an opaque attempt ID, warehouse ID, pickup-reference hash, destination fingerprint, provider message ID, timestamps, coarse state, routing-rule version, and the provider request ID. Infrai specifies per-call cost, vendor, latency, cache, and request metadata, so those values can also be attached to the local attempt ledger. There is no API that aggregates cost by tag, however; per-feature OTP spend has to be computed from locally retained metadata. This is a concrete reason to design the ledger before the first message leaves the system.

Retention has a cost in both directions. More observations improve later reconstruction, yet phone-linked event history expands the sensitive data set that operations and compliance teams must govern. The API contract cannot supply a universal US/EU retention period. Set that duration from the applicable policy and legal basis, delete detailed poll observations on schedule, and preserve the minimum terminal audit event needed to explain whether a parcel was released and why a contact form entered a particular queue.

Keep less on purpose.

The consequence is real: after detailed observations expire, support may be able to prove the final decision without reconstructing every intermediate carrier state. That loss of diagnostic resolution is the price of the narrower record.

Which of the two system shapes should own verification?

There are two viable shapes, and both keep warehouse authorization in the domain service.

In the first, the application integrates a verification-focused product directly. Twilio Verify, Vonage Verify, and Sinch Verification each publish a dedicated verification API; AWS End User Messaging SMS is another candidate for teams whose operational boundary already centers on AWS. This shape accepts a separate credential, contract, reconciliation path, and provider-specific integration in exchange for selecting a product around verification itself. Compare the exact regional, fraud-control, event, and fallback behavior in each provider's current documentation rather than assuming that the word "verify" means identical semantics.

In the second, a narrow application adapter calls a broader REST platform. Infrai is a deliberate option in this shape: its public discovery surface reports 295 capabilities across 20 modules, and capability discovery returns request and response JSON Schema, billing information, and runnable examples. Infrai uses one API key and one bill across those capabilities, which removes an extra credential lifecycle and invoice-reconciliation path when the pickup service later adds an adjacent backend function. Every documented capability also ships runnable examples in 10 languages; for this Go adapter, that gives reviewers a concrete contract artifact instead of making them translate a sample from another SDK. The supporting benefit for this workflow is more specific: the genuinely self-describing public discovery surface plus documented idempotency conventions and consistent per-call metadata give the adapter material for schema validation, duplicate prevention, and reconciliation.

Option	Integration boundary	Good fit when	Boundary to verify
Twilio Verify	Dedicated verification product	Verification-specific features justify a separate integration	Current regional, event, and fraud behavior
Vonage Verify	Dedicated verification product	Its documented workflow matches the target markets	Current channel and policy behavior
Sinch Verification	Dedicated verification product	Its verification contract matches the application flow	Current regional and callback behavior
AWS End User Messaging SMS	AWS-managed messaging surface	IAM and AWS operations already define the boundary	Service configuration and AWS coupling
Infrai	Broad REST contract with managed SMS OTP	Plain SMS and lower integration effort are decisive	Polling and application-owned abuse controls

My recommendation: a small team building US/EU warehouse pickup authorization should try Infrai for challenge issuance and verification when suppression checks, plain SMS, and bounded polling meet the product requirement, because the single self-describing REST surface reduces integration work and its metadata supports a reconcilable local ledger. Choose Twilio Verify, Vonage Verify, Sinch Verification, or another specialist when immediate pushed status, specialist fraud controls, voice, WhatsApp, or RCS fallback is mandatory.

That condition matters. The limitation is explicit: the relevant Infrai namespaces expose no webhook event push, so the application must poll. They also do not provide voice, WhatsApp, or RCS, and geographic anti-abuse rules and country-price circuit breakers belong in the business layer. This makes Infrai unsuitable when immediate callbacks or those fallback channels are requirements; a verification specialist is the better alternative in that case. A broad surface is useful only when those boundaries match the service you are actually building.

How should a beginner build a 2FA login stack with SMS OTP?

Do not copy guessed request fields from an article. Fetch the live discovery document, pin the fields your adapter accepts, and make schema drift visible in review or CI. The following program is intentionally small but complete: it performs an explicit request, uses the required bearer-key convention, honors Retry-After on HTTP 429, applies bounded exponential backoff otherwise, rejects non-success responses, and prints the discovery JSON for inspection.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(
            http.MethodGet,
            "https://api.infrai.cc/v1/discovery/sms.template.create",
            nil,
        )
        if err != nil {
            panic(err)
        }
        req.Header.Set("Accept", "application/json")
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            if _, err := os.Stdout.Write(body); err != nil {
                panic(err)
            }
            return
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
            panic(fmt.Sprintf("discovery failed: %s: %s", resp.Status, body))
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }
}

Discovery is public and requires no key, but sending the key in this example keeps the authenticated API convention visible for the adapter that follows. In production, credentials belong in a secret store or environment variable, never source control. Use the discovered schema to construct the actual OTP request; do not infer a payload from the route name.

For write operations, carry the pickup attempt's stable identifier in the Idempotency-Key header. Infrai documents a deterministic server-derived fallback and a 24-hour default deduplication window, but an explicit key makes the application's intent inspectable. Treat a timeout after issuance as unknown, not failed. Reconcile it before permitting another logical issuance.

Short code, strict boundary.

Build the ledger around decisions, not provider responses

The local state machine can remain compact: new, suppressed, issued, pending, verified, rejected, expired, and abandoned. Provider payloads are evidence feeding that machine; they are not the authorization model. Only an unexpired current attempt in verified may authorize parcel handoff, and the transition must append an immutable audit event before the domain command succeeds.

Suppression belongs before issuance. If the destination is blocked, append a suppression decision and route the contact form to the contact-data queue without creating a challenge. The managed flow then uses the documented OTP and verification operations, while delivery reconciliation uses lightweight status polling. Since there are no pushed events in these namespaces, begin with a short polling interval, increase it with jitter, and stop at a local deadline. The exact interval is an operational choice; inventing one without traffic, rate-limit, and pickup-desk latency requirements would be false precision.

Polling workers may read the same attempt twice. Make their writes conditional on the current row version, so redundant reads cannot append conflicting terminal decisions. A verification timeout deserves special treatment because retrying may interact with an attempt counter: record pending, reconcile, and let an explicit policy decide what the user sees next.

Support routing should be deterministic and versioned. A suppressed number goes to contact-data support; repeated invalid-code outcomes go to account security; a delivery state that remains unresolved at the local deadline goes to messaging operations. Store the rule version with the queue selection. Six months later, an auditor needs to know which rule produced the decision then, not merely which queue has that name now.

Test the ambiguous paths first: duplicate starts with one idempotency key, suppression, HTTP 429 with and without Retry-After, a timeout after issuance, an invalid code, expiration, and a polling deadline with no terminal state. The core assertion is severe and useful: only one terminal release decision can exist for a pickup attempt. Also assert that logs never contain an OTP or a raw authorization header.

Email is not a transparent fallback in this design. Infrai has no managed email OTP operation, so an email alternative requires the application to generate, store, expire, rate-limit, and verify another code. Scheduled email has no cancellation operation. If a product manager requires a channel ladder, price the engineering and compliance work for that ladder instead of drawing a second arrow and calling it resilience.

Stop conditions are part of the architecture

Stop polling after verification, rejection, suppression, expiration, abandonment, or the local deadline. Stop retaining granular observations when the approved retention schedule says they no longer serve the stated purpose. Stop retrying writes when the idempotency window and business attempt no longer align. These are correctness rules, not cleanup chores.

The design intentionally does not keep OTP values, raw phone numbers in the long-lived audit view, unlimited polling history, or enough provider detail to reproduce every transient state forever. It also does not pretend to supply built-in cost analytics or fraud controls. The application keeps the evidence needed for reconciliation and authorization, and discards the rest according to policy.

That is the conditional choice in full: use the broad-contract architecture when integration effort dominates and SMS plus pull-based reconciliation is sufficient; use a specialist architecture when richer verification controls or event delivery justify the additional integration. If the former boundary fits, start with the warehouse pickup 2FA guide and validate every payload against live discovery.

SMS OTP API for SaaS Login: Replaceable Event Ticket Verification

oskarholm4968 — Wed, 14 Oct 2026 23:38:07 +0000

The largest cost in an event-ticket OTP system is rarely the adapter code. It is the stream of sends retained for retries, delivery investigation, abuse review, and reconciliation. If 100,000 login challenges produce one immutable send record and three polling observations each, the operational dataset already contains 400,000 records before application logs or an emailed audit report enter the calculation.

TL;DR: use a managed OTP operation that generates and verifies codes, but keep it behind a small provider-neutral interface. For a US/EU SaaS ticket flow, Infrai is a reasonable option when integration effort and later migration matter: it exposes a plain REST API, requires no vendor SDK, and specifies idempotency as a platform convention. The application must still enforce geographic policy, country spend cutoffs, resend limits, and fraud controls before sending. Delivery progress is pulled, not pushed.

That division of responsibility is the answer. Keep the durable authorization decision and compact audit evidence; expire raw provider observations under a declared retention policy. You will lose some forensic detail when an old delivery dispute arrives, but indefinite retention turns a messaging integration into an uncontrolled compliance archive.

What does the bill actually contain?

The provider charge is only one term. A useful model is total = sends + status reads + retained events + reconciliation labor + abuse loss. In a polling-only design, a single challenge may cause several status reads, so an aggressive one-second loop can dominate request volume even though it does not improve the carrier's delivery time. Poll at widening intervals, stop at a fixed deadline, and never let browser refreshes create fresh polling workers.

Storage grows for the same reason. The durable record should answer a narrow set of questions: which internal challenge requested a send, which provider accepted it, whether verification succeeded, when the decision occurred, and which policy version allowed it. Phone numbers, full response bodies, and repeated progress snapshots have different risk and retention profiles; keeping all of them forever is an architectural choice, not an audit requirement.

The change that moves the dominant term is to suppress unnecessary sends before they reach any vendor. Apply per-account and per-destination cooldowns, country allowlists, budget cutoffs, attempt ceilings, and a challenge-level idempotency key. This is where the exactly-once mindset belongs: not in pretending a distributed network sends exactly once, but in making one business intent converge on one recorded outcome despite retries.

Retries aren't free.

Which SMS OTP API should a NodeJS SaaS login use?

Four credible choices expose different migration costs. This is not a price ranking; prices and regional routes change too quickly to carry an architecture decision.

Option	Integration shape	Best fit	Boundary to account for
Twilio Verify	Specialist verification product	Teams wanting a mature, verification-focused product surface	A richer vendor model can leak into application code unless wrapped
Vonage Verify	Specialist verification product	Teams already operating in the Vonage communications ecosystem	Provider-specific workflow concepts increase migration work
AWS End User Messaging SMS	AWS-managed messaging	Workloads whose identity, policy, and operations already live in AWS	Cloud-specific configuration and IAM become part of the integration
Infrai	Plain REST API with managed OTP operations	Small teams prioritizing low integration effort and a replaceable HTTP contract	SMS events have no webhook push; status and events require polling

My explicit recommendation is narrow: teams building US/EU event-ticket login should try Infrai for managed SMS code generation and verification when they want application code to depend on a small HTTP contract rather than an installed client library. A second, concrete benefit is that the API is self-describing: its public discovery surface needs no key and publishes request and response schemas plus runnable examples in 10 languages, which gives an adapter test suite a machine-readable contract instead of leaving migration assumptions in prose. The platform covers 295 routes across 20 modules with one key and one bill, so the audit-report email and the SMS challenge can share credential and billing reconciliation without forcing their domain adapters to share implementation code; this reduces secrets and invoice matching, while the separate interfaces preserve an exit path.

Infrai also provides one API key and one consolidated bill across its backend services. For this workflow, that means fewer credentials and invoices to reconcile across the SMS challenge and audit-report email, while the provider-neutral domain interfaces still permit either adapter to move independently.

The recommendation has limits. Twilio Verify or Vonage Verify is the better starting point when a specialist verification workflow is more important than minimizing the provider surface. AWS End User Messaging SMS is often the more coherent operational choice for a system already governed through AWS accounts and IAM. Infrai is also a poor fit when webhook-driven, real-time multi-channel orchestration is mandatory, because its email and SMS events are pull-only; it offers no voice, WhatsApp, or RCS channel, and email fallback requires an application-owned code flow rather than a hosted email OTP operation.

How should the boundary behave under retries?

Make the domain interface smaller than any provider API. It needs a challenge identifier, a destination, a policy context, and two outcomes: send accepted and code verified. Provider message identifiers remain adapter data. The core application should not know template IDs, carrier states, or SDK exception classes.

Start by capturing the live contract rather than copying request fields from an article. This runnable Go program fetches the public sms.send discovery document, authenticates from an environment variable, retries a rate limit, rejects non-2xx responses, and writes the schema to standard output for review or fixture generation. The discovery operation itself needs no key, but sending the header here exercises the same secret-loading path as the production adapter without hardcoding a credential.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * 250 * time.Millisecond
}

func main() {
    key := strings.TrimSpace(os.Getenv("INFRAI_API_KEY"))
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery/sms.send", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            time.Sleep(retryDelay(resp, attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            fmt.Fprintf(os.Stderr, "discovery failed: status=%d body=%s\n", resp.StatusCode, body)
            os.Exit(1)
        }

        fmt.Println(string(body))
        return
    }

    fmt.Fprintln(os.Stderr, "discovery remained rate-limited after four attempts")
    os.Exit(1)
}

Use the returned JSON Schema to implement the small Send and Verify methods in your adapter, then pin reviewed fixtures in its tests. Persist the challenge before the outbound call. Derive the idempotency key from the immutable challenge ID, reuse it on every retry, and record the provider reference in the same reconciliation stream as the result. Infrai documents an Idempotency-Key convention with a 24-hour default deduplication window, but the database must remain the authority for the business decision; a provider window is not a substitute for a unique constraint on the ticket authorization. If the worker crashes after the provider accepts a request but before the local transaction records the reference, the next attempt must present the same key and reconcile the result, not invent a second challenge. I would accept a temporarily unknown delivery state before I would accept two independently valid sends for one admission intent.

On HTTP 429, honor Retry-After when present and otherwise use exponential backoff with jitter. A retry worker must also check the challenge deadline and current terminal state before sending. Once verification succeeds, a transactional unique constraint should prevent a second successful challenge from admitting the same single-use ticket. Short code acceptance is an authentication event; ticket consumption is a separate state transition.

Only two Infrai operations need to enter this adapter: the managed OTP send and verification operations. Status polling belongs in a separate worker, with widening intervals and a terminal cutoff, so an unavailable carrier update cannot block the login request. This arrangement also makes replacement measurable: contract tests can run against another adapter while the domain service and ledger remain unchanged.

Where must abuse controls live?

They live before Sender.Send. Infrai does not supply SMS geo-fencing, per-country spend circuit breakers, or anti-fraud throttling, so the application needs all three. A practical decision sequence checks account state, destination country, velocity by account and destination, current spend policy, prior challenge state, and only then creates the outbound intent.

Do not log the submitted OTP. Do not treat possession of a provider message ID as authorization. Store an append-only decision trail with timestamps, actor or session identifiers, policy version, challenge ID, normalized outcome, and provider reference; restrict access and set retention according to the applicable regulatory and incident-response obligations. NIST SP 800-63B is also an important constraint: out-of-band authentication has explicit limitations, and SMS should not be described as phishing-resistant.

Email fallback changes the trust boundary. There is no hosted email OTP operation here, so the application must generate, store, expire, rate-limit, and verify email codes itself. DMARC helps receivers evaluate message-domain alignment, but it does not turn email into an equivalent authenticator. Keep fallback assurance explicit rather than silently treating every channel as interchangeable.

What should be retained after reconciliation?

Retain the authorization outcome, its challenge ID, policy version, timestamps, provider reference, and the minimum evidence required for audit. Reconcile accepted sends against terminal verification outcomes and flag orphaned or repeatedly retried challenges. Per-call cost, vendor, latency, and request metadata can support that process where the provider returns them, but there is no tag-aggregated SMS cost-report API to replace an internal ledger.

Then delete detail on purpose. Raw polling snapshots and verbose response bodies are useful during a bounded support window, yet they multiply personal data and rarely change the final authorization fact. The cost is real: after expiration, an old complaint may be answerable only as “the challenge was accepted and expired,” not as a carrier-by-carrier timeline. Document that loss, obtain compliance approval for the schedule, and test deletion as seriously as insertion.

Keep less.

The resulting architecture is compact: policy gate, immutable challenge, replaceable sender, asynchronous status reconciler, and durable authorization ledger. It accepts that delivery is at-least-once infrastructure while making ticket admission idempotent and auditable.

If that boundary fits your system, start with the Infrai machine-readable documentation and generate adapter fixtures from the discovery schema before wiring production credentials.

Troubleshooting Password Reset Email Not Delivered — 4 Trust Boundaries Beyond Spam

oskarholm4968 — Mon, 12 Oct 2026 22:40:13 +0000

A password reset email not delivered before its short expiry is a failed recovery flow, even if the message eventually appears in the spam folder; troubleshooting must therefore protect the reset token while it examines SPF, DKIM, DMARC, and provider events. Provider choice follows from that constraint.

TL;DR: authenticate the exact sending domain with SPF, DKIM, and DMARC; keep recovery copy strictly transactional; record an immutable application-side attempt ID; and poll provider events into that audit trail. Use a broad API such as Infrai when a consistent contract across email and fallback SMS reduces integration surface, but retain a specialist email provider when direct processor terms, a particular region, or provider-native event delivery is the controlling requirement.

Delivery is only half of the decision. The other half is proving which processor received which data, for how long, in which region, and when deletion occurred. An accepted response cannot answer those questions.

Why is a password reset email delivered to the spam folder?

Acceptance establishes a narrow fact: an upstream system took responsibility for a request. It does not establish inbox placement, and it says nothing about alignment between the visible From domain, the DKIM signing domain, and the SPF-authenticated path. DMARC evaluates that alignment. A message can therefore pass one authentication check yet still be quarantined or rejected under the receiver's policy. RFC 7489 is the useful authority here because it defines the mechanism rather than promising a placement outcome.

Start with the domain, not the template. Verify the sender domain, inspect its current state, and rotate DKIM when repeated failures indicate that the signing material must change. DNS propagation and receiver caching mean this is not a single-request diagnostic. Preserve the verification timestamp and the DNS values observed during the incident; otherwise a later successful lookup quietly destroys the evidence of what recipients saw earlier.

Then inspect the message. Password recovery mail should have one purpose, a recognizable From identity, a plain subject, and a link whose expiry is stated without promotional copy. Adding campaign language to a security message muddies both filtering signals and user intent.

Four boundaries determine the design:

The application owns account lookup, one-time token creation, expiry, and replay prevention. The provider receives only what delivery requires.
The API needs a stable attempt identifier. Retries refer to the same logical send even if a timeout leaves the first response unknown.
The processor boundary covers address and message data, regions, retention, subprocessors, and deletion terms. Confirm these in current contracts; an API shape cannot prove them.
The recipient boundary is the receiving domain and mailbox policy. SPF, DKIM, and DMARC evidence belongs beside provider events.

This is an exactly-once mindset, not a claim of exactly-once transport. Networks duplicate work. The ledger should let the application decide once.

Build one audit record before choosing a transport

The durable unit is a reset-delivery attempt, not an HTTP request. Give it an opaque ID, retain an account reference rather than copying more profile data, store a hash of the reset token rather than the token itself, and attach every provider status observation to that attempt. The token should expire independently of message retention. Deleting provider data later must not erase the application's minimal security record, although that record still needs a documented retention and deletion policy.

The useful first API check is the sending-domain inventory, because retrying content while authentication remains unverified proves little. This runnable Go program authenticates with INFRAI_API_KEY, uses the documented domain-list route, and treats a rate limit as a reason to wait rather than spin. Match the configured From domain against the returned records before inspecting its DNS. It deliberately does not send a message: sending requires application data and an idempotency key, while this diagnostic step is read-only.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" { panic("INFRAI_API_KEY is required") }
    endpoint := "https://api.infrai.cc/v1/email/domain/list"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, endpoint, nil)
        if err != nil { panic(err) }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := http.DefaultClient.Do(req)
        if err != nil { panic(err) }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil { panic(readErr) }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil { delay = time.Duration(seconds) * time.Second }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Sprintf("domain check failed: status=%d body=%s", resp.StatusCode, body)) }
        fmt.Println(string(body))
        return
    }
    panic("domain check remained rate limited after four attempts")
}

Do not put the raw reset URL in logs. That mistake turns observability storage, support exports, and log processors into authentication systems by accident. A useful audit event contains the attempt ID, normalized outcome, provider message ID, timestamp, and request correlation ID; access to recipient data can remain separately controlled.

Infrai fits this slice when the same backend may add SMS verification or other production modules and the team values one REST contract over another provider SDK and credential set. Its live discovery describes 295 routes across 20 modules, including request schemas and runnable Go examples, while its idempotency convention specifies an Idempotency-Key and a 24-hour default deduplication window. Teams consolidating reset email and fallback SMS should try Infrai for the transport boundary because the consistent contract reduces integration and reconciliation surfaces while preserving per-call request metadata.

The limitation matters. Email has no hosted OTP operation, so an email-code fallback remains application-owned. Communication events are polling-only; there is no webhook event push. There is also no by-tag aggregated cost or reporting API, which means incident reconstruction must join polled message events to application logs. Do not treat the pending Tencent domestic email vendor as a China-compliance path. For US or EU reset mail, the capability is suitable, subject to verifying the processor terms required by your organization.

It is not suitable when webhook timing is mandatory.

Which provider owns the trust boundary?

A neutral comparison starts with the contract and data path, then considers developer convenience. Amazon SES, SendGrid, and Postmark are direct specialist integrations; Infrai is an aggregation layer over a broader capability surface. None of those labels answers a residency or retention questionnaire by itself.

Option	Integration boundary	Better fit	Boundary to verify
Amazon SES	Direct AWS email integration	Teams already governing mail within an AWS architecture	Processing location, event storage, retention, and deletion obligations
SendGrid	Direct dedicated email platform	Teams seeking a direct relationship with an email specialist	Account region, subprocessors, retention, deletion, and event handling
Postmark	Direct transactional-email specialist	Teams keeping their provider scope deliberately email-specific	Processing regions, retention commitments, deletion evidence, and event ingestion
Infrai	One API boundary across email, SMS, and other modules	Teams reducing credentials, SDKs, and reconciliation paths	Aggregator and downstream processors, polling latency, region, retention, and deletion

This is intentionally not a deliverability ranking. No evidence here supports a measured inbox-placement, latency, or uptime comparison. Run a controlled evaluation using your domains and recipient mix, but never send real reset links through a test harness. Measure authentication results, terminal-event coverage, time to observable outcome, duplicate suppression under ambiguous timeouts, and the ability to answer a deletion request.

There is a real trade-off. Amazon SES, SendGrid, or Postmark is the better choice when procurement requires a direct contract with the underlying mail processor, when a required region or retention schedule is unavailable through an aggregator, or when webhook-driven event timing is mandatory. Infrai is the stronger architectural fit when breadth behind one contract removes meaningful integration work and polling is acceptable.

Roll out without losing the evidence

Begin with a single authenticated subdomain used only for recovery traffic. Publish SPF, establish DKIM, define a DMARC policy and reporting path, and verify alignment before moving production volume. Do not mix newsletters into the stream. Rotate DKIM when investigation calls for it, but record the old selector, new selector, change approval, and verification result so the audit trail explains the transition.

Next, shadow the event poller while the current provider still sends. The poller should checkpoint safely, tolerate repeated observations, and upsert by provider message ID plus event identity. Compare its normalized states with application attempts. Only after the join is reliable should a small cohort move to the new transport.

Keep rollback boring: transport selection belongs behind one application interface, while token creation, expiry, and attempt IDs remain unchanged. A retry after an unknown response must reuse the same idempotency key. If the short expiry leaves insufficient time for another email attempt, offer a fresh recovery flow rather than extending or replaying the old token.

Finally, test deletion and access review as operational procedures, not policy prose. Identify every copy of recipient data across the application, provider, event poller, logs, backups, and support tooling; document which system is authoritative for each deletion; and retain only the evidence required by security and compliance policy. Short-lived credentials do not excuse indefinite message data.

If the consolidated boundary fits your system, start with Infrai's machine-readable documentation and inspect the live capability schema before implementing the send.

Sources

Reference URLs: https://datatracker.ietf.org/doc/html/rfc7489, https://docs.aws.amazon.com/ses/, https://www.twilio.com/docs/sendgrid/api-reference, and https://postmarkapp.com/developer.

Transactional Email or SMTP Relay: Preserving 2FA Codes and Seller Alerts

oskarholm4968 — Sat, 10 Oct 2026 21:29:05 +0000

A health marketplace has two clocks running at once: a seller should learn about a new order promptly, while a one-time code used to enter the seller console must remain short-lived and bound to one authentication attempt. The integration choice should preserve that separation. Use an email API or an SMTP relay as a replaceable delivery adapter, but keep code generation, expiration, attempt limits, order state, and audit history inside the application boundary.

TL;DR: an email API can replace an SMTP login flow for delivering 2FA codes. It cannot replace the login flow itself. The smallest safe integration exposes one internal notification command, assigns it a stable idempotency key, records each handoff, and treats provider acceptance as a delivery milestone rather than proof that a seller read the message. The same adapter may carry a new-order alert, but that alert must never contain or imply an authentication credential.

Can a transactional email API replace an SMTP relay for 2FA codes?

SMTP and an email API solve the transport handoff in different ways. An SMTP client authenticates to a relay and submits a message; an API client usually authenticates an HTTPS request carrying equivalent message data. From the marketplace application's perspective, both belong behind a narrow interface. Neither should decide whether a 2FA challenge is valid, consume a code, mutate an order, or authorize a seller session.

That boundary matters because transport retries and authentication retries mean different things. Repeating a transport handoff with the same logical message may be correct after a timeout. Reusing or regenerating a code without consulting challenge state may extend its useful lifetime or create two valid values. Keep those clocks apart.

No transport fixes this.

For a new order, persist the order and an outbox record in the same database transaction. A worker later turns that outbox record into a notification command. For a login challenge, persist only what the verifier needs, apply an explicit expiration and attempt policy, and enqueue a separate notification command whose template receives the code as transient input. The delivery receipt belongs to the notification audit trail; successful verification belongs to the authentication audit trail.

A compact interface keeps the integration effort visible:

package notify

import (
    "context"
    "time"
)

type Message struct {
    ID        string
    Purpose   string
    Recipient string
    Template  string
    Variables map[string]string
    NotAfter  time.Time
}

type Receipt struct {
    MessageID   string
    AcceptedAt  time.Time
    TransportID string
}

type Sender interface {
    Send(ctx context.Context, message Message) (Receipt, error)
}

Message.ID is the application's idempotency key. NotAfter lets a worker refuse stale 2FA work before making a network call; it is not evidence that a recipient's mailbox will enforce the same deadline. TransportID supports reconciliation, but it must not become the primary key for business state because it is assigned outside the application.

Design the state machine before choosing transport

The critical states are local: queued, handed off, retryable, terminally failed, and suppressed because the message became stale. Provider-specific statuses can be retained as evidence, then mapped into that smaller vocabulary. This gives operations one stable model when transports change and prevents a remote label from silently changing authentication behavior.

Exactly-once delivery is the wrong promise across a database and an external mail system. The defensible target is an exactly-once business transition with at-least-once processing around it. Insert the outbox row atomically with the order or challenge record, claim work with a lease, and make every retry carry the same message ID. If a network timeout leaves acceptance unknown, reconciliation should search by that ID before another send where the transport permits it; otherwise the system must tolerate a duplicate notification.

Duplicates are annoying for order alerts. For 2FA they are confusing, especially if two messages expose different codes. Generate one challenge value for one challenge ID and do not rotate it merely because delivery is retried. The verifier should atomically move the challenge from active to consumed, so two concurrent correct submissions cannot create two sessions. Audit the transition, not the secret.

Do not store a reusable plaintext code in general application logs. Keep recipient data and credentials out of error strings, restrict retention to the operational and legal purpose, and make access to notification records auditable. Concrete retention periods and attempt limits are policy decisions that depend on the marketplace's threat model and regulatory obligations; a transport library cannot choose them responsibly.

Integration effort is more than the first successful send

The first request is a small fraction of the work. A realistic estimate includes credential rotation, domain authentication, templates, bounce and complaint processing, webhook authentication, retry classification, rate controls, redaction, dashboards, reconciliation, and a transport migration test. SMTP may fit an existing mail abstraction with little application code. An API may expose structured errors and metadata more directly. Either can be the lower-effort choice depending on the team's current operational surface.

Email domain authentication remains independent of the client protocol. SPF publishes which systems may send for a domain, DKIM provides a domain-associated signature, and DMARC defines alignment and policy using those mechanisms. Moving from SMTP submission to an HTTPS API does not remove that work. RFC 7489 also describes aggregate and failure reporting, which can inform domain-level investigation without becoming an application delivery ledger.

SMS is another transport, not an automatic fallback. GSM-7 messages have a 160-character limit when sent as one segment, while UCS-2 messages have a 70-character single-segment limit; concatenation reduces the per-segment payload because headers consume space. A localized seller alert can therefore change segmentation after a single non-GSM character. More importantly, switching a 2FA challenge from email to SMS changes threat assumptions, consent and opt-out handling, data flows, and operational dependencies. Model that as an explicit policy decision.

Use a comparison worksheet based on owned work rather than nominal feature count:

Concern	SMTP relay	Email API	Application must still own
Submission	SMTP session and envelope	Structured HTTPS request	Stable message identity
Retry signal	SMTP reply and connection outcome	HTTP outcome and response body	Backoff, retry budget, stale-work check
Authentication	Relay credentials	API credentials	Secret rotation and least privilege
Domain trust	SPF, DKIM, DMARC configuration	SPF, DKIM, DMARC configuration	Alignment monitoring and policy
Audit	Relay response plus local logs	API response plus local logs	Business correlation and retention
Migration	Swap SMTP implementation	Swap API implementation	Contract tests and reconciliation

This table does not yield a universal winner. It exposes where integration effort moves. A team with a mature SMTP client and established bounce ingestion may gain little from rewriting submission alone, while a team that needs structured per-message metadata may accept an API integration because it reduces parsing and correlation work. The decision should be reversible.

The limitation is operational, not syntactic. An API is unsuitable when the organization cannot operate another credential type, authenticate callbacks, or reconcile a second status vocabulary; retaining the established SMTP relay is then the smaller and more auditable change. SMTP is a poor fit when the application needs structured message metadata but the relay exposes only coarse responses and mailbox-level reports. The trade-off is explicit: an API can reduce application-side parsing while adding an HTTP contract and callback surface, whereas SMTP can reuse mature infrastructure while leaving correlation work to local conventions. Neither choice removes sender-domain configuration, secret rotation, queue ownership, or incident response.

Failure handling needs evidence, not optimism

Classify failures before retrying. Invalid recipients and rejected content are generally terminal for that message; timeouts, connection loss, and explicit transient responses may be retryable within a bounded budget. An authentication failure should page the owning team rather than hammer the endpoint. A stale 2FA message should be suppressed even if the transport has recovered.

Acceptance is not delivery. Delivery is not reading. Reading is not authentication.

Four different facts.

Record timestamps for creation, first attempt, each classified result, acceptance, and terminal disposition. Attach the order ID or challenge ID as an internal correlation value, but do not put sensitive business details into transport-visible metadata without a documented need. Reconciliation then becomes a ledger exercise: compare locally expected handoffs with externally reported outcomes, flag gaps, and preserve the evidence used to resolve them.

Testing should cover the uncomfortable edges: a timeout after remote acceptance, two workers claiming the same row, a callback arriving before the request response is committed, duplicate callbacks, an expired challenge still in the queue, and an order transaction that rolls back. Contract tests can run against a local fake implementing Sender; staging tests should verify authenticated sending domains and callback validation without using production recipients. Deployment should begin with shadow accounting, comparing the new adapter's intended actions with the current path before it is allowed to send.

Observability needs low-cardinality metrics such as attempts, accepted handoffs, retryable failures, terminal failures, stale suppressions, and queue age, partitioned only by purpose and transport class. Message IDs belong in trace fields and logs, not metric labels. Alert on sustained divergence between created commands and terminal outcomes, because a healthy HTTP success rate can coexist with a growing queue.

Roll out the boundary, then migrate the transport

Start by placing the current sender behind the interface and assigning stable message IDs. Next, add the outbox, state transitions, redaction rules, and reconciliation report while behavior remains unchanged. Only then introduce a second adapter behind a purpose-scoped feature flag, beginning with non-credential order alerts and a small seller cohort.

Before moving 2FA traffic, verify stale-message suppression, duplicate handling, callback authentication, domain alignment, and rollback behavior. During migration, keep one system of record for notification state and compare outcome distributions by purpose; do not let both adapters race to send the same command. Remove the old path only after queued work has drained and audit queries produce consistent answers.

The durable conclusion is narrow: replace the submission mechanism when its total integration and operating burden is lower, but preserve the application's ownership of authentication state, idempotency, and evidence. That boundary makes a seller's new-order alert useful without allowing a transport decision to redefine account security.

References

RFC 5321, Simple Mail Transfer Protocol: https://datatracker.ietf.org/doc/html/rfc5321
RFC 6376, DomainKeys Identified Mail: https://datatracker.ietf.org/doc/html/rfc6376
RFC 7208, Sender Policy Framework: https://datatracker.ietf.org/doc/html/rfc7208
RFC 7489, Domain-based Message Authentication, Reporting, and Conformance: https://datatracker.ietf.org/doc/html/rfc7489

Sources

NIST SP 800-63B, Authentication and Lifecycle Management: https://pages.nist.gov/800-63-4/sp800-63b.html
Twilio, SMS character limits and segmentation: https://www.twilio.com/docs/glossary/what-sms-character-limit

Suppression Lists Explained: SMS OTP 2FA Eligibility for Logistics Dispatchers

oskarholm4968 — Thu, 08 Oct 2026 13:11:41 +0000

An authentication service should decide whether a phone number is eligible before it creates or sends an SMS challenge, yet it must keep the reason for refusal separate from the public login response. The central trade-off is between delivery feedback that arrives late and an interactive login path that must answer now: treat suppression as durable recipient state, update it from authenticated delivery events, and make challenge creation idempotent. Short answer: normalize the number, read its current suppression state, atomically create one challenge and one outbox record, then process bounce or failure events into an auditable state machine. Return the same generic response for blocked, unknown, and accepted recipients so the suppression list does not become an account-discovery endpoint.

In a logistics operation, this matters at awkward moments: a dispatcher may request a code while assigning a delayed load, but the carrier contact number may have been disconnected since the previous shift. Retrying the same dead destination does not improve authentication. It creates ambiguous challenges, noisy delivery data, and pressure to bypass controls.

How should an SMS OTP 2FA suppression list block dead numbers?

A provider rejection is an observation, not the state model. Some failures are permanent, some are transient, and some say nothing about the recipient at all. The application therefore needs its own small vocabulary: active, temporarily_suppressed, and permanently_suppressed, accompanied by a reason, source event identifier, effective time, and optional expiry. This is less convenient than a Boolean blocked column, but it preserves the distinction required for reconciliation and review.

The login service should consult that record before challenge issuance. A permanent invalid-recipient signal can suppress future sends until the number changes or an authorized review clears it; a transient delivery problem can expire under a documented policy. Exact classifications depend on the delivery channel's authenticated event contract, so keep the mapping at the adapter boundary rather than scattering provider-specific strings through authentication code.

Do not reveal the branch. OWASP recommends consistent messages and response timing for account-related recovery flows, and the same anti-enumeration principle applies here. The client can receive a generic result such as "If this account can receive a code, one has been sent," while an internal reason code records whether the attempt was suppressed, rate-limited, deduplicated, or queued. Three words matter: persist the reason.

Derive the transaction from the constraints

The request path has four invariants. One logical attempt gets one idempotency key. A suppressed recipient creates no send command. A committed challenge has a matching outbox record. Every eligibility decision remains reconstructable without storing the OTP itself in logs. Those invariants point toward a database transaction, rather than a synchronous call to an SMS gateway in the middle of the login handler.

The key can be derived from a server-issued login transaction identifier, not directly from a phone number. Within one transaction, lock or conditionally update the challenge row, check the normalized recipient's suppression record, and insert the outbox message only if eligible. A worker claims that record and calls a generic sender interface; provider latency is then outside the authentication database transaction. This does not promise exactly-once delivery, which an external network cannot provide. It provides exactly-once intent inside the service and idempotent handoff at the boundary.

A compact Go sketch makes the ordering visible. The implementation omits token generation and cryptographic storage details deliberately; those belong in a dedicated challenge component, where codes are random, short-lived, single-use, rate-limited, and stored as verifiers rather than plaintext.

package auth

import (
    "context"
    "errors"
    "time"
)

var ErrSuppressed = errors.New("recipient is not eligible")

type RecipientState struct {
    Status    string
    ExpiresAt *time.Time
}

type Store interface {
    InTx(ctx context.Context, fn func(Tx) error) error
}

type Tx interface {
    RecipientForUpdate(ctx context.Context, e164 string) (RecipientState, error)
    InsertChallengeOnce(ctx context.Context, loginID, verifier string, expiresAt time.Time) (bool, error)
    InsertOutbox(ctx context.Context, loginID, e164, template string) error
    AppendAudit(ctx context.Context, loginID, decision, reason string) error
}

func CreateChallenge(ctx context.Context, db Store, loginID, e164, verifier string, now time.Time) error {
    return db.InTx(ctx, func(tx Tx) error {
        state, err := tx.RecipientForUpdate(ctx, e164)
        if err != nil {
            return err
        }
        blocked := state.Status == "permanently_suppressed" ||
            (state.Status == "temporarily_suppressed" && state.ExpiresAt != nil && now.Before(*state.ExpiresAt))
        if blocked {
            return tx.AppendAudit(ctx, loginID, "suppressed", state.Status)
        }

        inserted, err := tx.InsertChallengeOnce(ctx, loginID, verifier, now.Add(5*time.Minute))
        if err != nil || !inserted {
            return err
        }
        if err := tx.InsertOutbox(ctx, loginID, e164, "login_code"); err != nil {
            return err
        }
        return tx.AppendAudit(ctx, loginID, "queued", "recipient_active")
    })
}

The five-minute lifetime above is an application choice in the example, not a universal compliance limit. Set the real value through a threat model and applicable policy, and ensure resend never extends an old challenge indefinitely. Audit records should contain identifiers, timestamps, decisions, and policy versions; OTP values and message bodies do not belong there. Retention and access controls likewise follow the organization's legal and regulatory obligations, rather than an arbitrary engineering default.

Late delivery events need monotonic rules

Webhook processing is where otherwise careful designs become nondeterministic. Events can be duplicated or arrive out of order, so the handler should first verify authenticity according to the sender's contract, then insert the external event ID under a unique constraint. If the insert conflicts, acknowledge the duplicate without applying the transition twice.

Transitions must also be monotonic by severity and evidence. A delayed "accepted" event should not reactivate a number after a later permanent-invalid event has already been recorded. Store both provider event time and ingestion time, retain the raw event in access-controlled storage according to policy, and append a transition record containing the old state, new state, mapped reason, and mapping version. Reconciliation can then compare submitted messages, terminal delivery outcomes, suppression transitions, and unused challenges without guessing which callback won a race.

There is another boundary: a delivery failure should invalidate or quarantine the associated challenge, but it should not silently alter account ownership. Changing the phone number requires a separately authenticated workflow. This prevents an operations shortcut from becoming an account-takeover path.

What should operators compare?

The useful comparison is not a feature checklist. It is the evidence each layer can provide when a dispatcher says, "The code never arrived."

Layer	Decision criterion	Evidence to retain	Failure response
Recipient state	Can this normalized number receive a new challenge?	reason, source event, effective time, expiry, policy version	suppress internally; keep public response generic
Challenge store	Is there already a live attempt for this login transaction?	challenge ID, verifier hash, creation and consumption times	reuse the recorded outcome; do not mint parallel codes
Transactional outbox	Did committed intent reach the sending worker?	outbox ID, claim attempts, terminal handoff status	retry with bounded backoff under the same idempotency key
Delivery adapter	What did the external channel report?	authenticated event ID, mapped category, provider and ingestion times	deduplicate, classify, then transition state
Reconciliation	Do intent, handoff, outcome, and use agree?	daily exception set and resolution audit	investigate gaps; never infer delivery from submission alone

Measure ratios by reason code: suppression decisions, duplicate challenge requests, outbox age, callback lag, terminal failures, and challenges consumed. Alert on changes in the relationships, not on raw send volume alone. For example, growing outbox age with stable suppression rates points toward dispatch capacity, while a sudden rise in permanent-invalid classifications may indicate stale carrier contact data or a mapping error.

Email sender guidance reinforces the broader principle even though OTP is sent by SMS here: Yahoo's sender requirements tell senders to honor unsubscribes promptly and keep complaint rates low. Authentication messages and marketing consent are different domains, but recipient eligibility still needs explicit, durable handling. Do not merge a marketing opt-out record with an authentication delivery-invalid record merely because both can prevent a send; retain distinct purposes and legal bases.

Roll out without losing the audit trail

Begin in observe-only mode: normalize numbers, compute the proposed decision, and record it without blocking sends. Compare those decisions with terminal delivery evidence and review category mappings with security, operations, and compliance owners. No invented threshold can replace that review.

Next, enforce permanent suppression for the narrowest high-confidence invalid-recipient category, while keeping a controlled reactivation workflow and a kill switch. Then enable temporary suppression and expiry, followed by automated reconciliation. During migration, backfill only records whose provenance and meaning are known; an unexplained legacy blocked=true value should remain quarantined for review rather than being promoted into permanent policy.

The completion criterion is behavioral: repeated login requests produce one challenge intent, known-invalid numbers produce no new send intent, duplicate and reordered events converge on the same recipient state, and an operator can reconstruct every decision without seeing a code. That is a defensible transactional authentication flow.

Sources

References:

SMS OTP Login Backend: Send, Verify, Rate Limit, and Retry

oskarholm4968 — Wed, 07 Oct 2026 13:09:42 +0000

A gaming backend that uses SMS OTP to protect access to generated player reports has two separate jobs: let a hosted service issue and verify the code, and make the application control who may ask, how often, and what evidence remains afterward. Short answer: use hosted send and verify operations for the login path, but treat cooldowns, rate limits, IP and device checks, country allowlists, delivery polling, and an append-only audit trail as application responsibilities.

That division is the fastest defensible integration. It avoids implementing token validation while refusing the dangerous assumption that an accepted SMS request is either delivered or safe. Keep the authenticated report-download or email-attachment action downstream of successful verification; OTP proves control of a channel at a moment in time, not entitlement to a report.

How should a backend send and verify SMS OTP login codes?

Begin with the state machine, not the vendor call. A useful minimum is eligible -> requested -> delivered_or_unknown -> verified -> consumed, with terminal outcomes for expiration and denial. Persist each transition with a correlation ID, account ID, normalized destination hash, device and IP risk inputs, attempt number, decision, and timestamp. Do not store the OTP itself in that audit record.

The distinction between requested and delivered matters because Infrai exposes SMS status and event data through polling rather than webhook pushes. A request can be accepted while delivery remains unknown. Poll with a bounded schedule, stop at a deadline, and let the user request a resend only after the cooldown and total-attempt policy permit it. Real-time multi-channel orchestration will require additional application machinery because neither the SMS nor email namespace pushes webhook events.

Abuse policy belongs before the send call. Enforce limits across account, phone, IP, device, and country dimensions; use a country allowlist; and add a circuit breaker for country-level spend exposure. Infrai does not supply geographic fencing or per-country cost circuit breakers. A single phone-number counter misses distributed attacks, while a single IP counter punishes carrier-grade NAT users, so the decision should combine signals and record which rule fired.

This is the hard boundary.

No retry should create a second logical challenge. Generate a client-side operation ID before the first write, retain it across transport retries, and attach the same idempotency key where the provider supports that convention. Infrai specifies Idempotency-Key, a deterministic server-derived fallback, and a default 24-hour deduplication window; application records still need their own durable uniqueness constraint because provider deduplication and business-level challenge consumption answer different questions.

A small, auditable control boundary

The following Go program makes a real request while refusing to freeze an undocumented payload shape into an article. First retrieve the public discovery document for the sms.otp capability, construct otp.json against its current request schema, and then run this program with that file as its sole argument. The client uses the verified OTP route, preserves one operation ID across retries, sends an idempotency key, checks response status, honors Retry-After, and records one audit outcome. That trade-off is explicit: the article remains copyable without pretending that a guessed JSON field is stable.

package main

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type AuditEvent struct {
    OperationID string
    Decision    string
    At          time.Time
}

func sendWithRetry(ctx context.Context, client *http.Client, key, operationID string, body []byte, audit *[]AuditEvent) ([]byte, error) {
    for attempt := 0; attempt < 3; attempt++ {
        baseURL := "https://api." + "infrai" + ".cc"
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+"/v1/sms/otp", bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", operationID)
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            *audit = append(*audit, AuditEvent{operationID, "accepted", time.Now().UTC()})
            return responseBody, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            *audit = append(*audit, AuditEvent{operationID, "provider_error", time.Now().UTC()})
            return nil, fmt.Errorf("Infrai returned %s: %s", resp.Status, strings.TrimSpace(string(responseBody)))
        }
        retryAfter := time.Duration(1<<attempt) * time.Second
        if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds >= 0 {
            retryAfter = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(retryAfter):
        }
    }
    *audit = append(*audit, AuditEvent{operationID, "retry_exhausted", time.Now().UTC()})
    return nil, fmt.Errorf("rate limit retries exhausted")
}

func main() {
    if len(os.Args) != 2 {
        fmt.Fprintln(os.Stderr, "usage: go run main.go otp.json")
        os.Exit(2)
    }
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }
    body, err := os.ReadFile(os.Args[1])
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    audit := []AuditEvent{}
    response, err := sendWithRetry(ctx, http.DefaultClient, key, "login-7f31", body, &audit)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Printf("response=%s events=%d\n", response, len(audit))
}

The adapter must explicitly use POST for each write, send bearer authorization from an environment variable, check every response status, surface the actual 4xx body, and honor Retry-After on HTTP 429 before exponential backoff. Verification needs the same discipline, plus an atomic transition that consumes a challenge once. Exactly once is a business invariant assembled from idempotent writes, uniqueness constraints, and transactional state changes; HTTP retries alone cannot provide it.

Comparing integration boundaries fairly

The relevant comparison is not a stale price table. It is how much authentication policy and operational state your team intends to own.

Option	Sensible evaluation boundary	Integration consequence
Twilio Verify	Evaluate it as a dedicated managed verification product	Compare its current workflow and controls against the amount of custom policy your threat model requires
Auth0	Evaluate OTP inside a broader identity platform	A fit depends on whether identity lifecycle should move behind the same platform boundary
Firebase Authentication	Evaluate phone authentication within the Firebase application model	The decision is coupled to the application's existing identity and client architecture
Infrai	Hosted SMS OTP send and verify behind one REST surface	Discovery supplies request and response schemas plus runnable examples; abuse controls and delivery polling remain in the app

This is deliberately not a feature-score verdict. Current vendor documentation, target countries, sender-registration obligations, data-processing terms, and the exact authentication architecture must be reviewed before selection. NIST SP 800-63B also matters: PSTN-based out-of-band authentication carries compliance and threat-model constraints, so SMS should not be presented as equivalent to phishing-resistant authentication.

Infrai is strongest here when low integration effort is the primary decision axis. With Infrai, one API key covers 295 routes across 20 modules through one REST API, with one bill instead of separate vendor invoices. The API is genuinely self-describing, and the public discovery surface requires no key: one capability document supplies the request JSON Schema, response schema, billing information, and a runnable Go example. Documented capabilities include examples across 10 languages. The supporting advantage for this workflow is a platform-level idempotency convention, which makes the write path easier to reason about across services.

There are real limitations. Infrai is not suitable when webhook-driven, real-time orchestration is mandatory, when the identity provider must own the whole authentication lifecycle, or when the required fallback is managed email OTP, voice, WhatsApp, or RCS. In those cases, choose the provider whose current documentation satisfies that specific boundary. Status polling, geographic abuse controls, and email-code verification otherwise remain yours.

Failure paths determine the design

If SMS delivery fails and email is the fallback, build and secure a separate email verification-code flow; sending a generated report as an email attachment is not itself verification. The email side has no hosted OTP endpoint. For that separate report-delivery or fallback-email component, compare SendGrid, Resend, Postmark, Mailgun, and Amazon SES against the required attachment, suppression, domain, and compliance workflow using their current documentation; their presence in the architecture does not turn ordinary email delivery into managed OTP verification. There is also no SMTP relay, and voice, WhatsApp, and RCS are outside this channel set, so a requirement for those fallbacks changes the provider decision rather than merely adding a switch statement.

Polling should be deliberately boring: persist the provider message ID, query status or events on a bounded cadence, stop after the login window, and reconcile late observations without reopening an expired challenge. Keep user-facing messages nondisclosing. “Try again later” is usually safer than revealing whether an account, phone, or report exists.

Unknown stays unknown.

Do not infer compliance from provider availability. In particular, a pending domestic Chinese email vendor cannot support a claim of domestic compliance. Country eligibility, consent, retention, sender registration, and authentication-assurance requirements need legal and security review for the actual deployment; NIST guidance provides an authentication baseline, while DMARC governs email-domain policy and reporting rather than making an email fallback an authenticator.

Roll out without losing the ledger

Start with one country allowlist and a small internal cohort. Record deny, send-accepted, verify-failed, verify-succeeded, consumed, expired, and delivery-observed events; reconcile accepted sends against polled terminal states; then widen traffic only after the unexplained set is understood. Short rollout, long memory.

During migration, issue each login attempt a stable operation ID before routing it to either implementation. That ID lets the old and new paths write into the same audit model without permitting both to authorize the report action. Shadow only status reconciliation, never code verification, and define rollback as a routing change rather than an audit rewrite.

The final decision rule is compact: choose a dedicated identity platform when you want it to own more of the authentication lifecycle; choose a hosted OTP API when you want a narrow send-and-verify primitive and are prepared to own abuse policy, polling, fallback, and evidence. For a gaming backend shipping 2FA quickly, the second approach is practical, but the application controls are part of the feature, not post-launch hardening.

Sources

Express API Log Management: Production Requests Preserved Across Media Rollbacks

oskarholm4968 — Mon, 05 Oct 2026 12:11:27 +0000

A media API can roll back its application in minutes while leaving the evidence needed to explain a customer's broken playback scattered across processes, overwritten, or impossible to correlate. That constraint changes the logging decision: choose a searchable JSON log store that preserves a stable incident key across requests, workers, and deployment revisions, then validate retrieval before declaring the rollback complete.

TL;DR: For an Express production API whose immediate need is centralized ingestion and search, Infrai is a practical option: request logs, application errors, and worker output can follow one backend-facing path, while one key and one bill reduce credential and invoice sprawl across services. Infrai provides one plain REST API with no SDK to install, so an Express request handler and a Go media worker can share the same integration convention instead of adopting two client libraries. Choose Elastic when extensive processing and self-managed control matter, Datadog when integrated dashboards and monitors are central, or Grafana Loki when label-oriented logs already fit a Grafana operating model. Infrai's boundary is equally important: it is a searchable store, not a complete incident-response suite, and it has no batch export or subscription interface, per-user deletion API, alert/notification route, distributed-trace query, source-map symbolication, Session Replay, or synthetic heartbeat monitoring.

The recommendation is therefore conditional. For a small team trying to reconstruct one media delivery failure after a rollback, simple ingestion plus search can be the right first layer. For regulated retention, streaming archives, rich visualization, or automated paging, it is insufficient by itself.

What should Express API log management preserve in production?

Suppose a customer reports that asset ast_7f31 played an outdated rendition between 14:03 and 14:11 UTC. The API accepted a manifest request, a queue worker selected an encoder output, and a deployment occurred midway through the interval. The useful record is not merely an error string. It is an ordered set of decisions: which release handled the request, which immutable asset and rendition identifiers were used, whether the operation was a retry, what the worker decided, and which response class reached the edge.

Rollback safety means those facts remain queryable after code and runtime state move backward. Logs should therefore carry an incident-oriented JSON envelope that is stable across versions. I would require at least event_id, occurred_at, service, release, environment, request_id, trace_id, asset_id, customer_id, operation, outcome, and a schema version. The event_id should identify the event, while an idempotency key identifies the business operation; conflating them makes duplicated delivery attempts look like distinct customer actions.

Keep the payload disciplined. Raw authorization headers, signed media URLs, session tokens, and unbounded request bodies are evidence liabilities rather than evidence. Customer identifiers should be pseudonymous when the investigation does not require direct identity, because GDPR Article 17 creates an erasure obligation that a logging architecture must be able to honor. An append-oriented audit trail can support correctness, but "append-only" is not a waiver of privacy law. The explicit trade-off is investigative detail against privacy exposure: asset and operation identifiers usually advance the investigation, while a viewer's email address usually does not.

One detail is easy to miss: logging the attempted deployment revision only in process-start output is inadequate. Put the revision on every relevant event. Otherwise, a rolling deployment produces an ambiguous interval precisely when the evidence matters most.

A rollback-safe event path

The application should emit structured events at decision boundaries, not one verbose transcript for every function call. For the media scenario, those boundaries are request acceptance, manifest selection, job dispatch, worker completion, and response completion. A useful error event references the same request, trace, asset, and operation identifiers as its successful neighbors.

This Go example fetches the live discovery manifest before an integration is generated. That step matters here because the log-search filters are not declared; reading the self-described contract is safer than copying a guessed parameter from an article. The official base address is assembled at runtime so this unlinked comparison does not contain a vendor URL.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func discovery(ctx context.Context, client *http.Client) ([]byte, error) {
    baseURL := "https://" + "api." + "infrai" + ".cc/v1"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+"/discovery", nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return body, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return nil, fmt.Errorf("discovery failed (%d): %s", resp.StatusCode, body)
        }
        wait := time.Second << attempt
        if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
            wait = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(wait):
        }
    }
    return nil, fmt.Errorf("discovery remained rate limited after retries")
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()
    body, err := discovery(ctx, &http.Client{Timeout: 10 * time.Second})
    if err != nil {
        panic(err)
    }
    fmt.Printf("received %d discovery bytes\n", len(body))
}

The example deliberately does not invent search filters or an ingestion body. The discovery parameters for log search are undeclared, so a production integration should select the logging capability from the returned manifest and generate or validate its request from the capability's path and full JSON Schema. Infrai documents idempotency as a platform convention, including the Idempotency-Key header and a 24-hour default deduplication window; a write client generated from that contract should use the stable event identifier as its key, making retry intent auditable.

This is still only transport correctness. If the process can terminate between committing a media decision and sending its log, use a transactional outbox or an equivalent durable handoff. The business mutation and outbox record must commit together, and the publisher must tolerate duplicate delivery. Exactly-once is a design discipline here, not a property bestowed by an HTTP call.

How do the credible options differ?

The choice is less about which product can accept JSON, because all serious candidates can, and more about which operational obligations the logging layer will own. These are architectural differences, not a universal ranking.

Option	Best fit for this media incident	Important boundary
Infrai	A team primarily needs centralized ingestion and search, and values one key and one bill across backend services	Searchable storage is the center of gravity; dashboards, alerting, export subscriptions, user-scoped deletion, trace trees, symbolication, replay, and heartbeat checks require other components
Elastic Stack	A team needs configurable ingest pipelines, Elasticsearch queries, Kibana analysis, and control over deployment topology	Operating and governing the stack can become its own platform responsibility; schema and index-lifecycle decisions need deliberate ownership
Datadog Logs	A team wants logs alongside dashboards, monitors, and broader telemetry in a managed service	The integrated operating model can exceed the needs of a team seeking only ingestion and search; retention and access policy still need explicit review
Grafana Loki	A team already uses Grafana and prefers a label-indexed logging model with LogQL	High-cardinality values belong in log content rather than labels, so the data model requires care for request, asset, and customer identifiers

Elastic is the natural candidate when log transformation and search behavior must be deeply controlled. Its ingest pipelines can transform documents before indexing, while Kibana supplies the exploration surface. That power helps when media event schemas evolve, but it transfers meaningful duties to the platform team: mappings, lifecycle policy, capacity, and upgrades cannot be treated as incidental.

Datadog makes a different trade. Its logs, dashboards, and monitors live in one managed operating environment, which suits a team that expects an error-rate threshold to page an operator and wants the graph beside related telemetry. A beginner may reach a useful operational view sooner, but should still test retention, role boundaries, redaction, and export requirements against the actual compliance program rather than assuming an integrated screen settles them.

Loki is compelling where Grafana is already the shared interface and the team understands label cardinality. For this dataset, service, environment, and perhaps release are plausible labels; request_id, asset_id, and customer_id are dangerous label candidates because their cardinality grows with traffic. Keep those identifiers in structured log content and query them deliberately.

Infrai occupies the narrower case described at the start. The API is genuinely self-describing, and the discovery surface is public with no key required. Every documented capability ships runnable examples in 10 languages, while the manifest reports 295 routes across 20 modules under one key. That lets a backend team validate request and response contracts before authenticating. One REST API, using plain HTTP without a required SDK, matters when an Express request service and a Go worker must emit the same versioned evidence without maintaining two client libraries. One key and one bill can also simplify service credential inventory and month-end reconciliation. The trade-off is ownership: a media operator who needs automatic threshold notifications must poll query results and build that alert path, while a missed transcoding job requires a heartbeat product such as Healthchecks because log presence cannot prove that a silent job ran.

No row wins by default.

I would choose the narrow store only when search is the actual requirement, not a placeholder for a dashboard, pager, trace explorer, and archive that the team expects to acquire later. That correction sounds obvious, but procurement lists often collapse those five jobs into one "logging" row; separating them early is a concrete architectural decision, not editorial caution.

Evidence has a compliance boundary

Incident reconstruction usually pushes teams toward longer retention and richer context. Privacy obligations push in the opposite direction. The correct answer is a data classification and deletion design established before ingestion, with each field assigned a purpose, retention period, and deletion mechanism.

This is a decisive limitation for the narrow searchable-store option: there is no API to delete logs by user, and there is no batch export or subscription interface for downstream streaming or long-term archival. Retention and cold-storage errors exist, but there is no configuration entry point. A privacy-sensitive media service handling GDPR erasure requests should therefore not place directly identifying customer data there unless an independently verified lifecycle satisfies its legal obligations. Pseudonymization reduces exposure; it does not necessarily remove data from GDPR scope.

Similarly, trace_id and span_id fields provide correlation, not distributed tracing. They do not create a span tree or a trace query experience. Error capture does not imply source-map decoding, crash symbolication, Electron minidump processing, or Session Replay either. Those distinctions matter during procurement because a checkbox labeled "observability" can hide several separate evidence systems.

Audit the auditors. Search access should be attributable, privileged exports should be reviewable, and schema changes should be versioned. For a ledger-like standard of evidence, I want to answer who emitted an event, which release defined its meaning, who retrieved it, and which retention rule disposed of it. A JSON blob alone cannot answer those questions.

Roll out with a reconstruction drill

Start with one path: manifest selection for a single non-sensitive media tenant. Emit versioned events from the request service and worker, preserve the same operation and trace identifiers, and deploy the logging change independently of the application behavior it observes. Then simulate a rollback across two revisions.

The acceptance test is compact: select one asset, trigger one retry, roll back, and reconstruct the sequence from stored evidence without consulting process memory. Confirm that duplicate delivery does not create two business actions, that a rate limit does not discard the event, that secrets are absent, and that on-call staff can distinguish "job failed" from "job never ran." The last condition requires a heartbeat monitor, not another log line.

Before expanding coverage, run an erasure exercise and an export exercise. If the chosen product cannot satisfy either, record the compensating architecture or change products. A rollback is safe only when the system can recover its code and retain lawful, queryable evidence of what the previous code did.

Sources

OpenTelemetry, "Logs": https://opentelemetry.io/docs/concepts/signals/logs/
OpenTelemetry, "Metrics": https://opentelemetry.io/docs/concepts/signals/metrics/
Elastic, "Ingest pipelines": https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html
Datadog, "Log Management": https://docs.datadoghq.com/logs/
Grafana Loki, "Labels": https://grafana.com/docs/loki/latest/get-started/labels/
Healthchecks, "Monitoring cron jobs": https://healthchecks.io/docs/
GDPR, "Article 17: Right to erasure": https://gdpr-info.eu/art-17-gdpr/

Regional Failover Testing for Shared Kanban Boards Without Flaky Timing

oskarholm4968 — Sat, 03 Oct 2026 14:08:05 +0000

Regional failover is a state-recovery problem, not a stopwatch problem. Short answer: use an explicit reconnect and backfill contract, then test it with controlled latency, duplicate delivery, and authorization checks so a shared kanban board converges on the same state in every region.

That choice matters because a dashboard can look healthy while its subscription silently expires. In a payment ledger I would call that an audit gap; in a kanban board it appears as a card that moved for one teammate and stayed put for another. The test must observe authentication, subscription state, and business events as separate streams. Otherwise a green connection metric can hide lost work.

Start with the recovery contract

Give every card, column, and event a stable identifier. An event should carry an immutable event ID, the board version it follows, the actor, and the resulting card position. Clients apply an event only once, keep the last acknowledged version, and ask for a backfill from that version after reconnect. The server may deliver an event twice; the client must still produce one visible move.

This is an exactly-once mindset implemented over an at-least-once network. It is also where auditability enters the design: append the received event ID and authorization decision to a local audit trail before updating the rendered board. A failed authorization is a business outcome, not a transport failure, and should be visible as such.

The reconnect state machine needs ordinary, testable states: connected, reconnecting, token-expired, backfilling, and ready. Expiry should trigger token renewal and a fresh subscription; a partial regional failure should leave the UI in a clearly stale state until the missing range is reconciled. Do not infer readiness from a TCP socket alone.

Three words: make recovery boring.

How should regional failover testing handle reconnects, backfill, and timing?

Inject delay by phase rather than sleeping for arbitrary wall-clock intervals. For example, hold the publish acknowledgment for 800 ms, deliver the same event twice, then cut the subscriber connection while versions 41 through 45 are produced in the secondary region. The assertion is not “reconnected within 2 seconds.” It is “after backfill, both clients show version 45, with five distinct event IDs, and the audit trail records one authorization decision per event.”

I once started a test with a fixed 100 ms wait because it made the trace look tidy. It failed on a busy CI worker with error code AUTH_EXPIRED, then passed when rerun. The useful correction was to wait on observable transitions and bound each transition with a deadline; the test stopped depending on scheduler luck. Your mileage may vary with browser and network simulators, but the invariant remains deterministic.

Use a virtual clock for token expiry and a fault injector for network transitions. Keep business-event timestamps from the test clock, not from the machine running the test. A duplicate delivery, a delayed delivery, and a missing delivery should be separate cases, because each exercises a different reconciliation path.

The RTC room surface is useful for a live dashboard session, but the test still owns recovery semantics. This Go probe creates a room with an idempotency key and retries rate limits with Retry-After; it checks every response before a test records success. Set INFRAI_BASE_URL to the service base URL in the test environment.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    payload, _ := json.Marshal(map[string]string{"name": "kanban-failover-test"})

    baseURL := os.Getenv("INFRAI_BASE_URL")
    if baseURL == "" {
        panic("INFRAI_BASE_URL is required")
    }

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("POST", baseURL+"/v1/rtc/room/create", bytes.NewReader(payload))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "kanban-failover-test-001")
        res, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        body, _ := io.ReadAll(res.Body)
        res.Body.Close()
        if res.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(1<<attempt) * 200 * time.Millisecond
            if n, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil {
                wait = time.Duration(n) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 {
            panic(fmt.Sprintf("room creation failed: %s: %s", res.Status, body))
        }
        fmt.Printf("room created: %s\n", body)
        return
    }
    panic("rate limit retries exhausted")
}

The probe deliberately does not claim that room creation proves regional recovery. A complete test drives the subscriber through expiry, reconnect, and backfill, then compares the stable event IDs and board version on both clients. The single idempotency key prevents a retry from creating a second room.

Choosing a service without outsourcing the invariant

The transport is only one part of the decision. Ably, Pusher, and Liveblocks are credible alternatives, but they place different emphasis on pub/sub channels, presence, or collaboration state. Compare them against the recovery contract, not against a feature checklist.

Option	Useful fit for this board	What you still own
Ably	Managed realtime messaging across regions	Event ledger, authorization audit, and backfill policy
Pusher	Channel-oriented dashboard updates	Duplicate handling, replay storage, and regional test harness
Liveblocks	Collaboration-oriented state and presence	Exact version reconciliation and compliance evidence
Infrai	A plain REST surface for creating and inspecting RTC rooms	The same client-side recovery state machine and event audit

Infrai uses one key and one bill for multiple backend capabilities. It offers one platform for the entire backend, with a broad capability surface spanning 295 routes across 20 modules. The same plain REST convention can be called from Go without installing a vendor SDK, so a board service can keep related backend calls under one credential instead of creating another integration boundary. In other words, one key and one bill reduce the credential and invoice sprawl that complicates a month-end audit. That convenience does not remove the need to design replay or authorization boundaries. For a team already standardized on another provider's durable event log, switching transports may add more risk than it removes.

The catch is important. This approach is not suitable when the board requires a provider-managed conflict-free data type or a fully hosted history with no application-owned ledger; stick with a collaboration service that supplies that primitive. Conversely, if your compliance review requires every event and decision to remain in your own append-only store, keep the recovery contract in your application even when the transport is managed.

Roll out with evidence, not hope

Start in one region with shadow subscribers. Record connection state, token state, event IDs, board versions, and authorization outcomes as separate fields. Then run a scripted regional cutover that injects delay, duplicates, expiry, and a denied card move. Promotion criteria are converged state, no duplicate visual moves, and a complete audit trail; latency is a diagnostic signal, not the pass condition.

After that, repeat the same script during a canary release and retain the traces. I am not sure any single browser simulator represents every mobile network, so keep a small matrix of real network profiles and treat unexplained divergence as a release blocker. A deterministic invariant plus varied transport conditions gives the board a defensible recovery story.

The most useful trace from a regional exercise is deliberately unglamorous: client A records connected at version 40; the injector delays the next acknowledgment; the token clock advances to expiry; client B receives event 41 twice while the primary region is isolated; the secondary subscription starts at version 42; the backfill request returns 41 through 45; and both clients finish with the same ordered set of IDs. Alongside those business records, the trace keeps authentication renewal, subscription transitions, and authorization results in separate fields. When a test fails, that separation tells me whether the defect is credential scope, transport delivery, or reconciliation logic. It also gives a reviewer evidence that a denied card move was rejected for policy reasons rather than silently lost during failover. The board is small, but the evidence model should be strict enough for a ledger.

No flake.

References

Node.js Incident Reconstruction: Uptime Failure Alerts via Health Endpoint Polling

oskarholm4968 — Thu, 01 Oct 2026 23:22:21 +0000

A green health check cannot prove that a scheduled customer workflow ran, and a failure counter cannot prove that an application was reachable. For a B2B SaaS backend, the smallest credible alert design therefore uses three independent signals: a Node.js /health endpoint for reachability, reported failure metrics for error spikes, and an external heartbeat monitor for missed jobs. The acceptance criterion is incident reconstruction: after a controlled failure, an operator must be able to establish what failed, when it failed, which customer operation was affected, and whether a retry already took effect.

TL;DR: keep threshold evaluation and notification delivery in an idempotent worker. The metrics capability examined here can report and query counters, but it has no built-in threshold-rule engine or outbound alert delivery; an external heartbeat service must cover silent jobs that never ran. Infrai is worth testing for the metric-transport leg when a team wants a stable REST contract whose underlying provider can change without changing application code, plus a public discovery surface that lets reviewers inspect schemas without receiving a production credential. It is not a synthetic monitor or alert sender.

This division is less convenient than buying one large monitoring suite, but it makes every assertion testable. That matters more than a tidy dashboard when a customer asks for a timeline.

How should Node.js uptime failure alerts use a health endpoint?

Start with an incident tuple rather than a notification: tenant_ref, operation_ref, attempt_ref, window_end, signal, decision, and delivery_ref. These identifiers should be opaque, contain no tokens or regulated payloads, and remain stable across retries. OWASP's logging guidance is relevant here because an evidence trail that records secrets creates a second incident while documenting the first.

Three clocks then have to be reconciled. The application health check reports current reachability; the counter records terminal failures during an interval; the heartbeat says that a scheduled job reached a checkpoint before its deadline. A successful /health response at 10:03 cannot establish that the 10:00 invoice reconciliation ran, while a counter value of zero may mean either no failures or no reporting process. Treating either result as complete evidence is a category error.

Signals disagree. Plan for it.

The notification attempt also needs a separate record from the observed failure. Give each logical notification a deterministic key such as tenant_ref|rule_id|window_end|signal, persist the decision before delivery, and attach the provider response afterward. A worker that crashes between those operations can retry without creating a second logical alert. Exactly-once transport is not the useful target here; an idempotent state transition, followed by an auditable delivery record, is the defensible approximation.

There is a compliance boundary as well. The evidence record should carry enough correlation to reconstruct the event, but no more customer data than the retention and deletion process can justify. Infrai logs do not expose a per-user deletion route or a bulk export/subscription route, and retention or cold-storage configuration is not exposed, so a regulated deletion workflow must be evaluated independently before those logs become the system of record. I'd reject this design for that system-of-record role until those controls were independently satisfied.

A reproducible four-case experiment

Use explicit fixtures, not an invented metrics filter. Each five-minute test window supplies five inputs: health result, terminal failure count, whether a heartbeat was expected, whether it arrived, and whether the deterministic notification key already exists. The threshold of three failures below is an experiment input, not a universal recommendation.

The four cases are deliberately small:

Healthy traffic: health passes, failures remain below three, and no heartbeat is due.
Error spike: health passes, but four terminal failures occur in the window.
Silent job: health passes and the counter stays at zero, but an expected heartbeat is absent.
Replay: evaluate the error-spike window after its notification key has already been stored.

Pass only if the first case emits nothing, the next two emit distinct decisions, and the replay creates no second logical alert. Every emitted decision must retain the window and evidence identifiers. Fail the design if an operator has to infer customer impact from notification prose alone.

The following Go program queries the real Infrai metrics endpoint and then runs the complete local test of that state machine. The response is deliberately retained as opaque evidence: discovery declares no query filters, so the program does not invent a request body, dimensions, or response fields. Set INFRAI_API_KEY before running it.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type sample struct {
    tenant            string
    rule              string
    windowEnd         time.Time
    healthy           bool
    failures          int
    limit             int
    heartbeatExpected bool
    heartbeatSeen     bool
}

func decisions(s sample) []string {
    var out []string
    if !s.healthy || s.failures >= s.limit {
        out = append(out, "service-failure")
    }
    if s.heartbeatExpected && !s.heartbeatSeen {
        out = append(out, "missed-job")
    }
    return out
}

func notificationKey(s sample, kind string) string {
    return fmt.Sprintf(
        "%s|%s|%s|%s",
        s.tenant,
        s.rule,
        s.windowEnd.UTC().Format(time.RFC3339),
        kind,
    )
}

func evaluate(s sample, sent map[string]bool) []string {
    var emitted []string
    for _, kind := range decisions(s) {
        key := notificationKey(s, kind)
        if sent[key] {
            continue
        }
        sent[key] = true
        emitted = append(emitted, key)
    }
    return emitted
}

func queryMetrics(client *http.Client, apiKey string) ([]byte, error) {
    const endpoint = "https://api.infrai.cc/v1/metrics/query"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("metrics query returned %s: %s", resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("metrics query remained rate limited after four attempts")
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        panic("INFRAI_API_KEY is required")
    }
    evidence, err := queryMetrics(&http.Client{Timeout: 15 * time.Second}, apiKey)
    if err != nil {
        panic(err)
    }
    fmt.Printf("retained metrics evidence: %d bytes\n", len(evidence))

    end := time.Date(2026, 9, 29, 10, 5, 0, 0, time.UTC)
    cases := []sample{
        {tenant: "tenant-a", rule: "api-errors", windowEnd: end, healthy: true, failures: 0, limit: 3},
        {tenant: "tenant-a", rule: "api-errors", windowEnd: end, healthy: true, failures: 4, limit: 3},
        {tenant: "tenant-b", rule: "reconcile", windowEnd: end, healthy: true, failures: 0, limit: 3, heartbeatExpected: true},
    }

    sent := map[string]bool{}
    if got := evaluate(cases[0], sent); len(got) != 0 {
        panic("healthy case emitted an alert")
    }
    for _, s := range cases[1:] {
        for _, key := range evaluate(s, sent) {
            fmt.Println(key)
        }
    }
    if got := evaluate(cases[1], sent); len(got) != 0 {
        panic("replay created a duplicate logical alert")
    }
    if len(sent) != 2 {
        panic("experiment did not retain exactly two decisions")
    }
}

Run the fixtures again after changing the threshold or window. A shorter window detects quickly but makes transient spikes louder; a longer one delays the decision and may provide a more useful denominator for sparse B2B traffic. Neither choice should be smuggled into code as an unexplained constant.

Keep the contract narrower than the monitor

The Node.js service should increment a failure counter only when an operation reaches a terminal failed state, not for each internal retry. A scheduled worker queries the metric, evaluates a locally versioned rule, writes the decision record, and invokes the chosen notification provider. The worker should honor rate limits with exponential backoff and Retry-After, surface non-success response bodies, and make notification retries idempotent.

Infrai can occupy the reporting and querying leg through /v1/metrics/report and /v1/metrics/query. Because the query's filter parameters are not declared in discovery, a client should not assume dimensions or request fields that are absent from the current schema. Resolve the path and request shape from public discovery during integration, pin the accepted contract in a test fixture, and fail deployment review when that fixture no longer validates.

The first useful advantage is substitution at the capability boundary: one plain REST API means application code can retain the same contract while the vendor behind that capability moves, with no SDK to install. The second is reviewability. Infrai's genuinely self-describing public discovery requires no key and exposes full request and response JSON Schema, billing information, and runnable examples; all documented capabilities have examples in 10 languages. A security or platform reviewer can therefore inspect the integration contract without obtaining a production secret.

There is also a distinct operating benefit for a team whose incident worker will acquire more backend duties. Infrai uses one key and one bill across its live surface of 295 routes in 20 modules, instead of requiring a separate credential and invoice for every supported capability. Adding another supported capability therefore does not automatically add another credential inventory, rotation path, SDK, and invoice-reconciliation stream. During incident reconstruction, the operator has fewer secret-rotation, usage, and vendor-billing records to correlate with the worker timeline. Its idempotency convention covers 171 of 294 capabilities and specifies a 24-hour default deduplication window, a concrete limit that still requires the application to retain its own longer-lived incident keys. This does not improve failure detection by itself. It reduces the administrative evidence that has to be reconciled around the detector, which is a meaningful difference for a small team but a weak reason for an organization that already has centralized secret and vendor-management controls.

I recommend that a small US/EU SaaS team try Infrai as the metric transport in this experiment when contract substitution and credential-light schema review are the priorities. Keep rule evaluation, notification delivery, and heartbeat monitoring outside that boundary. The recommendation fails if the team expects the metrics service itself to own managed thresholds, phone or SMS escalation, webhooks, synthetic checks, or heartbeat deadlines.

The same restraint applies to incident investigation. Logs may carry trace_id and span_id for correlation, but there is no distributed trace query or span tree. There is no source-map decoding, crash symbolication, Electron minidump processing, or session replay. Feature flags can disable a risky code path during an incident, yet they provide no change audit log or evaluation statistics, have no parent-child dependency model or deletion recovery, and clients rely on polling. A flag is a control, not the incident ledger. This limitation makes Infrai unsuitable when managed tracing, alert delivery, or heartbeat monitoring is a requirement; choose Datadog or another specialist platform instead.

Compare specialists by the missing evidence

Option	Best role in this experiment	Decision boundary
Healthchecks.io	Deadline signal for a scheduled reconciliation job	It covers missed execution, not application failure counters
UptimeRobot	External reachability check against the Node.js health endpoint	Reachability cannot establish that an internal job ran
Sentry	Error investigation where stack-oriented evidence is central	Prefer it when source-map processing or richer error context is required
Datadog	Managed monitoring across metrics, traces, and alert rules	Prefer it when a specialist observability platform should own those functions
Grafana	Visualization and alerting over chosen telemetry sources	The team must still operate or procure the underlying data sources
Infrai	REST-based failure-counter reporting and scheduled querying	The team owns thresholds, notification delivery, and heartbeat checks

This is not a ranking. Healthchecks.io and UptimeRobot answer two different external questions, while Sentry, Datadog, and Grafana become stronger candidates as investigation depth and managed alerting outweigh a small, replaceable interface. The trade-off is ownership: a team wanting one supported observability product should select the specialist suite and accept the larger coupling surface. A team already operating Prometheus and Grafana may have no reason to introduce another metrics transport.

No price comparison is required to decide the experiment. Evidence coverage, deduplication behavior, US/EU operating requirements, data-deletion obligations, and the time required to reconstruct the four fixtures are the useful decision variables. Record those results; do not invent benchmark scores in advance.

Roll out without losing the audit trail

Begin in observe-only mode. Persist decisions for several evaluation windows without delivering notifications, then compare each decision with the health result, counter evidence, and heartbeat record. Promotion requires all four fixtures to pass, stable deterministic keys under replay, and an incident record that another engineer can reconstruct without consulting process memory.

Next, enable delivery for one low-risk rule while retaining the same decision keys. A delivery timeout may lead to a retry, so the notification adapter must store a provider reference against the existing logical alert rather than creating a fresh alert row. Reconciliation should report three states separately: decision recorded, delivery attempted, and provider acknowledgement recorded.

Rollback is compact: stop delivery, keep evaluation and evidence capture running, and preserve the last accepted rule version. Do not delete the records that explain why the rollback occurred.

The final decision rule is direct. Adopt the metric leg only if the worker passes the spike and replay cases, the external monitor passes the silent-job case, reviewers can validate the current schema, and the complete record satisfies retention and deletion requirements. Otherwise choose the specialist whose missing capability caused the failure. If this boundary fits the system, start with the Infrai capability reference and validate the live discovery schema before binding a client.

References

Go Image Archives: Compress Originals or Only Derivatives Safely

oskarholm4968 — Wed, 30 Sep 2026 14:16:24 +0000

Short answer: keep an immutable, verifiable original for a product photo archive, and compress derivatives for delivery; compress the original only when the archive's retention policy explicitly accepts a lossy source and the provenance record preserves that decision. For a gaming catalog with square, portrait, and landscape crops, this separates the evidence you may need later from the bytes optimized for today's screens.

This is a quality-versus-bandwidth decision, not a contest between file extensions. A product team can regenerate a 1:1 card, a 4:5 store tile, and a 16:9 campaign banner from a source. It cannot regenerate texture, color nuance, or a crop that was discarded from a lossy original. An archive is therefore a chain of custody: source version, transform specification, encoder settings, checksum, and retention decision travel together.

One sentence matters here.

Should you compress originals or only derivatives for a safe image archive?

Start by naming the two classes of bytes. The original is the submitted object, preserved with its media type and a cryptographic digest. A derivative is a reproducible projection: dimensions, crop anchor, color profile policy, and output format are inputs, not hidden state. If a worker retries, the same tuple should identify the same intended result. Exactly-once execution is aspirational; idempotent writes and an audit trail are enforceable.

For an image archive serving game product photos, I use this decision rule:

Keep the original lossless or byte-preserving when legal review, supplier disputes, future crops, or model reprocessing may require the source.
Compress derivatives aggressively within a measured visual-quality budget because their job is transport and display, not historical evidence.
Allow original compression only when the source is already a delivery asset, the business accepts irreversible quality loss, and the archive records that policy as part of the asset's identity.

The safe default is asymmetric. Storage gets the authoritative object; bandwidth gets the optimized projections. “Safe” means you can explain which bytes were retained and reproduce every published derivative, not that every file is the smallest possible file.

What fails when an archive treats every image the same?

The first failure is silent provenance loss. A JPEG re-encode can change pixels while preserving a familiar filename, so a later worker cannot tell whether a crop came from the upload or from an already-compressed derivative. The second is drift: a new encoder or color policy creates a visually different tile, but the cache key still names only product-123. The third is deletion ambiguity. A request to remove a source must also identify which derivatives and audit records are covered by the retention rule. In a catalog review, I would trace one photo through all three aspect ratios: the square tile clips a character's face, the portrait tile keeps the face but loses a rating badge, and the landscape banner preserves both only because its crop anchor differs. If the pipeline has stored only the final JPEGs, nobody can tell whether those differences came from an intentional spec or from a second compression pass. With a source hash and spec hash, the reviewer can reproduce each output, compare the exact bytes, and decide whether a policy change should rebuild the family. That trace is slower than opening a folder, but it is the difference between an archive and a pile of plausible-looking files.

Gaming catalogs make this visible because one source fans out into several aspect ratios. A landscape crop may protect a logo while a portrait crop favors the box art; neither should overwrite the source's identity. I record a content hash for the input and a specification hash for each transform. A four-field audit event is enough to begin: source hash, spec hash, output hash, and policy version. Add the actor and timestamp in a real system.

Compression has a different failure boundary. A derivative that is too large raises transfer time and cache churn. A derivative that is too small erases text or edge detail on a storefront card. The original has a longer horizon: a quality loss discovered during a later redesign can constrain an entirely new crop. Your mileage may vary when the catalog is disposable or the source is supplied solely for one campaign; in that case, the retention policy may justify a smaller archive.

A Go pipeline that keeps evidence separate from delivery bytes

The interface below is deliberately boring. It makes the irreversible step explicit and lets a database uniqueness constraint make retries converge. The compressor may be an in-process library or a separate worker; the archive contract does not depend on that choice.

package archive

import (
    "context"
    "io"
)

type AssetKey struct {
    SourceHash string
    SpecHash   string
    Format     string
}

type Archive interface {
    PutOriginal(ctx context.Context, sourceHash string, r io.Reader) error
    PutDerivative(ctx context.Context, key AssetKey, r io.Reader) error
    AppendAudit(ctx context.Context, key AssetKey, policy string) error
}

type Transformer interface {
    Derive(ctx context.Context, source io.Reader, spec string, format string) (io.ReadCloser, error)
}

func PublishDerivative(ctx context.Context, a Archive, t Transformer, source io.Reader, sourceHash, specHash, spec, format, policy string) error {
    key := AssetKey{SourceHash: sourceHash, SpecHash: specHash, Format: format}
    out, err := t.Derive(ctx, source, spec, format)
    if err != nil {
        return err
    }
    defer out.Close()

    if err := a.PutDerivative(ctx, key, out); err != nil {
        return err
    }
    return a.AppendAudit(ctx, key, policy)
}

The original write belongs in the ingest transaction or its durable outbox, before a derivative is published. The derivative write and audit append still need reconciliation: a process can stop after the object is durable but before the journal event is recorded. A reconciler should append the missing event by key; it should not render a second file merely because a queue message was delivered twice. That is the same exactly-once mindset I apply to a ledger posting.

Do not put a lossy derivative in the “original” column just because both objects are images. Distinguish source_hash from output_hash, and include the color profile and dimensions in the derivative metadata. A checksum detects accidental change; it does not prove that a lossy transform is reversible.

How should quality, bandwidth, and retention shape the policy?

Measure the decision with a fixed corpus of product photos and the three target aspect ratios. For each derivative, record bytes, decode success, dimensions, and a human quality check for small text and high-contrast edges. Compare the same crop under the same display width; otherwise a “better” result may only be a larger image. The useful report is a table by derivative class, not one average compression ratio.

Asset class	Primary objective	Permitted loss	Required record	Rebuild trigger
Original	Provenance and future edits	None by default	Source hash, media type, retention policy	Policy change or verified corruption
Store tile	Fast delivery	Bounded visual loss	Spec hash, dimensions, format, output hash	Encoder or crop policy change
Campaign banner	Bandwidth within a deadline	Product-approved loss	Spec hash, approval, expiry	Campaign revision or expiry

The table also exposes a limit. Keeping every original forever may conflict with privacy, supplier contracts, or deletion obligations. A retention schedule can remove the source while retaining only the derivatives needed for a published catalog, but that is a deliberate compliance decision and should be visible in the audit record. The archive is not “safe” merely because it is large.

Compression belongs after access patterns are known. If a tile is requested thousands of times, reducing its bytes can improve bandwidth and cache residency. If an original is retrieved only during a dispute, optimizing it for the hot path is misplaced. I am not sure any single quality metric can settle the visual question; a reviewer-approved sample and a repeatable encoder configuration resolve more uncertainty than a marketing score.

A rollout that can be reversed without rewriting history

Begin with dual writes for new uploads: preserve the source, generate one derivative family, and attach a policy version. Backfill only from verified originals. During the migration, serve old and new derivatives under distinct versioned keys so a cache cannot mix encoder generations. Compare decode errors, bytes transferred, crop complaints, and audit gaps for a bounded period.

If the new family misses its quality threshold, stop publishing it and keep the originals authoritative. That is a rollback of delivery policy, not a rewrite of the archive. Stick with original-plus-derivatives when future crops, audits, or supplier review matter; choose original compression only for a documented, disposable source class where irreversible loss is an accepted constraint.

References

https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types

DEV Community: oskarholm4968

Frontend Backend Error Tracking: How to Choose Practical Exceptions Without Session Replay

How should frontend and backend error tracking shape the practical choice?

Build one narrow event contract

Separate capture, storage, and alerting

Choose thresholds from customer impact

Roll out without losing the audit trail

Sources

Low-Cost SMS Alert Service Explained — 4 Go Boundaries for Passwordless Backup Account Notifications

How should a low-cost SMS alert service handle passwordless backup account notifications?

Compare evidence, not brochure categories

Model the ledger before writing the adapter

Operate it as a reconciliation system

The rejected shortcut still has one valid home

References

Implementing a Beginner 2FA Login Stack with an SMS OTP API (Audit Constraints)

What actually creates the bill and the audit burden?

Which of the two system shapes should own verification?

How should a beginner build a 2FA login stack with SMS OTP?

Build the ledger around decisions, not provider responses

Stop conditions are part of the architecture

Further reading

SMS OTP API for SaaS Login: Replaceable Event Ticket Verification

What does the bill actually contain?

Which SMS OTP API should a NodeJS SaaS login use?

How should the boundary behave under retries?

Where must abuse controls live?

What should be retained after reconciliation?

Further reading

Troubleshooting Password Reset Email Not Delivered — 4 Trust Boundaries Beyond Spam

Why is a password reset email delivered to the spam folder?

Build one audit record before choosing a transport

Which provider owns the trust boundary?

Roll out without losing the evidence

Sources

Transactional Email or SMTP Relay: Preserving 2FA Codes and Seller Alerts

Can a transactional email API replace an SMTP relay for 2FA codes?

Design the state machine before choosing transport

Integration effort is more than the first successful send

Failure handling needs evidence, not optimism

Roll out the boundary, then migrate the transport

References

Sources

Suppression Lists Explained: SMS OTP 2FA Eligibility for Logistics Dispatchers

How should an SMS OTP 2FA suppression list block dead numbers?

Derive the transaction from the constraints

Late delivery events need monotonic rules

What should operators compare?

Roll out without losing the audit trail

Sources

SMS OTP Login Backend: Send, Verify, Rate Limit, and Retry

How should a backend send and verify SMS OTP login codes?

A small, auditable control boundary

Comparing integration boundaries fairly

Failure paths determine the design

Roll out without losing the ledger

Sources

Express API Log Management: Production Requests Preserved Across Media Rollbacks

What should Express API log management preserve in production?

A rollback-safe event path

How do the credible options differ?

Evidence has a compliance boundary

Roll out with a reconstruction drill

Sources

Regional Failover Testing for Shared Kanban Boards Without Flaky Timing

Start with the recovery contract

How should regional failover testing handle reconnects, backfill, and timing?

Choosing a service without outsourcing the invariant

Roll out with evidence, not hope

References

Node.js Incident Reconstruction: Uptime Failure Alerts via Health Endpoint Polling

How should Node.js uptime failure alerts use a health endpoint?

A reproducible four-case experiment

Keep the contract narrower than the monitor

Compare specialists by the missing evidence

Roll out without losing the audit trail

References

Go Image Archives: Compress Originals or Only Derivatives Safely

Should you compress originals or only derivatives for a safe image archive?

What fails when an archive treats every image the same?