DEV Community: FerdinandBlake3517

Best Twilio Alternatives for USA SMS Alerts Service: Compliance Evidence

FerdinandBlake3517 — Sat, 07 Nov 2026 00:43:02 +0000

Best Twilio Alternatives for USA SMS Alerts Service: Compliance Evidence

Short answer: choose the SMS provider that helps you produce evidence for the whole password-reset path, not only the provider with an attractive message rate. A healthtech service should record the request, short expiry, provider response, delivery-status lookup, suppression decision, and any fallback decision. Twilio is a reasonable baseline, but Plivo, Vonage, MessageBird, and Bandwidth belong in the same evaluation. The answer depends on US traffic, volume, setup fees, phone-number costs, developer tooling, and how cleanly each provider's records fit your audit process.

A reset code is small. The evidence around it is not.

What must the reset flow prove?

Start with the constraint. A patient who asks for a password reset should receive a short-lived code, while the service should later explain what happened without retaining more sensitive material than necessary. I would log a request identifier, account-scoped subject reference, creation time, expiry time, template version, destination classification, provider message identifier, response status, and delivery-status lookup result. I would not log the code itself.

The application should create the code and expiry before calling the provider. Store a one-way representation with a short time-to-live, bind it to the reset attempt, and invalidate it after successful use. The text should say what the code is for and how long it is valid. It should not contain diagnosis, appointment, or treatment information. Those details do not improve authentication and create needless disclosure risk.

The compliance artifact is the event chain, not a screenshot of a dashboard. NIST's digital identity guidance is a useful reference for treating authenticator lifecycle, verification, and recovery behavior as system responsibilities. It does not turn an SMS reset into a universal compliance answer; privacy, security, and clinical governance teams still decide the applicable controls.

Evidence matters.

How should a team compare Twilio alternatives?

I use four questions before looking at a rate card. Can the provider serve the intended US traffic? Can the application query message state when a delivery gap matters? Are setup fees and phone-number costs understandable? Can engineers integrate and test the path without turning provider-specific behavior into the domain model?

Plivo is often a strong candidate for high-volume SMS when its commercial terms and US coverage fit the traffic profile. Vonage, formerly Nexmo, has competitive rates and developer tools that can suit a team that wants a mature messaging API surface. MessageBird is worth checking when regional pricing and flexible APIs matter, especially for a service that may expand beyond one geography. Bandwidth deserves a close look for US-based SMS traffic and teams that want options tailored to that market.

Those are different strengths, not interchangeable badges. A provider that looks good on bulk volume can still be awkward for number provisioning or audit retrieval. A flexible API can move complexity into consent, suppression, and delivery-state handling. A US-focused option can be a poor fit for a later international rollout. Twilio remains a useful benchmark because it makes the comparison concrete, but a benchmark is not a recommendation.

Here is the comparison I would put in a design review. It avoids pretending that one provider wins every dimension.

Option	Access model	Best fit	Main trade-off
Twilio	SDKs and APIs	A familiar baseline with a broad ecosystem	More provider-specific surface to own and audit
Plivo	SDKs and APIs	High-volume SMS when US terms fit	Validate number, setup, and delivery-state details for the target traffic
Vonage	SDKs and APIs	Competitive rates plus developer tooling	International or regional expansion can change the comparison
MessageBird	Flexible APIs	Regional pricing and multi-region plans	Flexibility can move consent and suppression complexity into your service
Bandwidth	APIs for US traffic	A US-focused SMS profile	Less attractive if the roadmap quickly becomes global
Infrai	One REST API	One key for every backend service and one bill to reconcile	Pull-based status handling means your service must schedule evidence collection

Infrai fits that last row only when its operating model matches the team: one key for every backend service and one bill, with a plain REST surface instead of separate provider dashboards and credentials. Its SMS surface includes sending, status lookup, cancellation, OTP, and suppression operations; that can reduce integration boundaries around this workflow. It is not suitable when webhook-driven event delivery is a hard requirement, or when a team needs email-hosted OTP, SMTP relay, voice, WhatsApp, or RCS from the same capability set. The relevant namespaces have no webhook event push, so event handling is pull-based; the service must schedule status checks and preserve its own evidence.

Is a status lookup enough for an audit trail?

No. It is one observation in the trail. A provider status can tell the application what it knows about a message, but it cannot prove that the reset was requested by the right account, that the code expired, or that a retry did not create two valid attempts. Keep those facts in your own event store with correlation identifiers and immutable timestamps.

A practical state model is created, submitted, delivery_pending, delivered, expired, used, and rejected. Treat it as an application model, not a copy of a vendor's vocabulary. Record the external message ID separately. If the provider returns a transient failure, retry with an idempotency key or a request identifier that your service can deduplicate. Rate limiting also belongs in the design: cap reset requests per account, device, and destination, then add a geographic or country-pricing circuit breaker in the business layer. SMS providers do not make that policy for you.

The awkward case is a reset that looks successful to the application but remains pending at the carrier. A support ticket may arrive hours later, after the code has expired, while the provider dashboard shows only a message identifier and a state transition. Your record should still answer which template version was rendered, when the attempt became invalid, whether a second request was allowed, and which status observation closed the attempt. That is why I keep provider state as an input to an application state machine rather than treating it as the audit record itself.

Here is the boundary shape I would keep. It is deliberately boring: secret generation, expiry, redaction, and audit records stay in the service you own; only the final send and status adapter varies by provider. The following small adapter polls the verified status route and treats a provider failure as an event to record, not a successful delivery.

import json
import os
import time
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen


def poll_sms_status(message_id: str) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    base_url = os.environ.get("INFRAI_API_BASE_URL", "https://" + "api." + "infrai.cc/v1")
    url = f"{base_url}/sms/status/{message_id}"

    for attempt_number in range(4):
        request = Request(
            url,
            method="GET",
            headers={"Authorization": f"Bearer {api_key}"},
        )
        try:
            with urlopen(request, timeout=10) as response:
                if response.status < 200 or response.status >= 300:
                    raise RuntimeError(f"status lookup failed: HTTP {response.status}")
                return json.load(response)
        except HTTPError as error:
            if error.code != 429 or attempt_number == 3:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"status lookup failed: HTTP {error.code}: {detail}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt_number
            time.sleep(delay)
        except URLError as error:
            if attempt_number == 3:
                raise RuntimeError(f"status lookup failed: {error.reason}") from error
            time.sleep(2 ** attempt_number)

    raise RuntimeError("status lookup exhausted retries")

The request uses Authorization: Bearer from an environment variable, an explicit method, a bounded retry loop, and a response-status check. It reads only the message status; code generation and the ten-minute expiry remain application policy. The sample does not pretend to be a provider SDK. That is intentional. Keep the security properties provider-neutral.

Where do the alternatives stop helping?

Every option has an edge. Provider status is not a substitute for consent and suppression rules. Setup fees and number costs can change the result at a volume where message rates look decisive. Delivery gaps can come from handset state, carrier filtering, or a recipient entering a number they do not control. Do not promise that a provider can make SMS deterministic.

The same caution applies to channel fallback. If the business wants email as a backup, its verification path has to be built and evidenced separately; email is not automatically a hosted OTP service. A design that schedules email should also account for the absence of a cancellation operation in this capability set. SMS cancellation is a different contract.

For a healthtech team, the decision rule is straightforward: pick the provider whose US behavior, commercial terms, integration model, and evidence retrieval you can test with production-like records. Prefer a narrower, well-observed flow over a broad channel list that an audit process cannot explain.

A small rollout that leaves evidence

Start with a non-production destination allowlist and a fixed reset template. Test an expired code, a reused code, a suppressed destination, a provider timeout, a rate limit, and a status that remains pending. For every case, assert both user behavior and the audit record. The useful result is not only “the text arrived”; it is “the service can reconstruct the decision without exposing the secret.”

Then run a controlled US traffic slice. Compare Plivo, Vonage, MessageBird, Bandwidth, and Twilio using the same template, retry policy, destination mix, and evidence schema. Keep pricing as one input alongside setup fees, number costs, regional fit, and operational effort. Recheck current provider terms before committing because those details move.

Finally, make the provider adapter replaceable. The reset service should know about attempts and evidence; an adapter should know about message IDs and status queries. That separation keeps a vendor change from becoming an authentication rewrite.

Sources

NIST, “Digital Identity Guidelines: Authentication and Lifecycle Management”: https://pages.nist.gov/800-63-3/sp800-63b.html
Twilio Messaging documentation: https://www.twilio.com/docs/messaging
Plivo SMS documentation: https://www.plivo.com/docs/sms
Vonage SMS API documentation: https://developer.vonage.com/en/messaging/sms/overview
MessageBird SMS API documentation: https://developers.messagebird.com/api/sms-messaging/
Bandwidth Messaging API documentation: https://dev.bandwidth.com/docs/messaging/

Node.js Error Tracking Library — Integrate Practices for Notification Cost Attribution

FerdinandBlake3517 — Thu, 05 Nov 2026 13:56:45 +0000

A useful Node.js error-tracking integration for a notification service starts with one accounting rule: attribute cost to the delivery attempt, while grouping errors by the stable failure cause. That preserves the difference between one bad destination and ten retry attempts, and it keeps an observability bill from becoming a second source of confusion.

Short answer: capture a normalized error event at the boundary where the notification attempt is created, attach a redacted operation and delivery identity, and keep retry state as structured fields. Store the full event only for a bounded retention window. Keep a small aggregate keyed by failure class, channel, and tenant so an engineer can answer both questions: what failed, and what did it cost?

The hard part is deciding what counts as an event. A thrown exception is an implementation detail. A delivery attempt is an operational fact.

That distinction is the whole design.

Start with the bill, then choose retention

For a developer-tools notification service, the dominant cost is often repeated data rather than the initial error. A single SMS or email can produce an accepted attempt, a provider timeout, a retry, and a final permanent failure. If each record includes the request body, stack trace, user address, and every response, the same business event gets copied several times. That may help a short incident investigation, but it is a poor default retention model.

I start with three counters per tenant and channel: attempts, transient failures, and permanent failures. Each counter is keyed by a stable failure class such as timeout, rate_limited, invalid_destination, or unknown. The aggregate is cheap to retain and useful for cost attribution. It also prevents a common mistake: interpreting a high exception count as a high sending cost when a single malformed address is being retried by policy.

The detailed error event still matters. I retain it for the period needed to investigate delivery behavior, with payload fields removed or hashed according to the data policy. After that, the aggregate remains while the raw event expires. This is a deliberate trade: a late investigation may lose the exact stack and response body, but it does not lose the number of attempts, the tenant that incurred them, or the failure class that drove them.

A practical event shape looks like this:

from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class DeliveryFailure:
    notification_id: str
    attempt: int
    tenant_id: str
    channel: str
    failure_class: str
    retryable: bool
    elapsed_ms: int
    error_type: str
    trace_id: str | None


def to_log_fields(failure: DeliveryFailure) -> dict[str, Any]:
    return {
        "event_name": "notification.delivery.failure",
        "notification_id": failure.notification_id,
        "attempt": failure.attempt,
        "tenant_id": failure.tenant_id,
        "channel": failure.channel,
        "failure_class": failure.failure_class,
        "retryable": failure.retryable,
        "elapsed_ms": failure.elapsed_ms,
        "error_type": failure.error_type,
        "trace_id": failure.trace_id,
    }

The notification identifier supports reconstruction without making the message body part of the event. attempt is essential: without it, a retry storm looks like many independent notifications. A cost ledger can then charge or report per attempt while the incident view groups all attempts under one notification.

How should a Node.js library integrate error tracking practices?

The integration point belongs around the delivery adapter, not around every try statement in the application. In Node.js, rejected promises can cross several layers before they reach a worker boundary. Reporting them at the adapter gives the event a channel, attempt number, elapsed time, and classification at the same place. Reporting them higher up usually loses at least one of those fields.

Classification should be explicit and boring. A timeout can be retryable. An invalid destination normally is not. A rate limit may be retryable under a policy that includes a delay, while an authentication failure should page an operator instead of multiplying attempts. The classifier should consume a small adapter error contract rather than parse arbitrary message text.

from collections.abc import Mapping


def classify_delivery_error(error: Mapping[str, object]) -> tuple[str, bool]:
    code = error.get("code")
    status = error.get("status")

    if code in {"ETIMEDOUT", "ECONNRESET"}:
        return "timeout", True
    if status == 429:
        return "rate_limited", True
    if status in {400, 422}:
        return "invalid_destination", False
    if status in {401, 403}:
        return "authentication", False
    return "unknown", False

This is intentionally not a list of provider routes. It is a boundary contract. The concrete adapter can map its response into code and status, while the rest of the service observes the same dimensions across email, SMS, and OTP.

The Node.js side should also preserve the original error as the cause when it adds context. That keeps stack information available to the error tracker without replacing the useful low-level failure with a generic DeliveryError. Do not log the destination address or message content by default. Those fields are tempting during debugging and expensive from a privacy and retention standpoint.

This design has a boundary: it is a poor fit when an audit requires reconstructing every message body and provider response indefinitely. The trade-off is deliberate: a separate restricted archive with a documented legal retention policy is needed in that case; error tracking should not become that archive by accident.

How do you keep retries from corrupting cost attribution?

A retry is a state transition, not a new notification. Give every notification one durable identifier and every attempt a monotonically increasing number. The worker writes the attempt record before dispatch, then updates it with the result. If the process dies after dispatch and before the update, the next worker must treat the outcome as uncertain. It should not silently convert that uncertainty into a successful send.

This is where idempotency and observability meet. The error event needs a deduplication key such as (notification_id, attempt), while the cost ledger needs an append-only attempt fact or an equivalent idempotent upsert. A tracker that deduplicates every identical exception globally can hide real retry volume; a ledger that counts every log line can inflate it. Keep those concerns separate.

Here is the small piece of accounting I want near the worker boundary. It does not decide retry policy; it records the result that policy produced.

from collections import Counter


def update_cost_counters(
    counters: Counter[tuple[str, str, str]],
    tenant_id: str,
    channel: str,
    failure_class: str,
) -> None:
    counters[(tenant_id, channel, failure_class)] += 1


def attempt_cost_key(tenant_id: str, channel: str, attempt: int) -> tuple[str, str, int]:
    return tenant_id, channel, attempt

The real implementation must enforce uniqueness in durable storage. An in-memory Counter is only a readable model of the dimensions. It is not a ledger, and it cannot survive a worker restart. That distinction is easy to miss in a library example because the code looks complete before deployment.

Keep this part plain. The counter is a model, not the accounting system.

For structured logs, use a schema that your collector can forward as logs or map into an error signal. OpenTelemetry describes logs as records with a timestamp, body, attributes, and context that can correlate them with traces. The standard gives you transport-neutral vocabulary; it does not choose your retention period, retry semantics, or cost model. Those remain application decisions.

What should the integration test before production?

Test the failure matrix, not just the happy path. At minimum, inject a timeout, a rate-limit response, an invalid destination, a worker restart after dispatch, and a duplicate delivery callback. Assert the resulting failure class, retryable flag, attempt number, and cost key. Also assert that sensitive payload fields are absent.

The most valuable test I added to this kind of boundary is a replay test: feed the same delivery result twice and verify that the detailed error may be observed twice but the cost fact is counted once for the same attempt. Then feed attempt 2 and verify that it is counted separately. That catches the two opposite bugs that matter here.

Make the test data uneven. Use one notification with one attempt, one with three attempts, and one with an unknown outcome. A perfectly uniform fixture can pass while the aggregation logic is wrong.

In production, alert on ratios and accumulation, not raw exceptions alone. A rising unknown class signals a mapping gap. A high retryable-failure ratio suggests a channel or dependency issue. A growing permanent-failure count for one tenant may indicate bad input or abuse. Each alert should link to the structured event and the aggregate, with access controls appropriate to tenant data.

The decision rule

Use error tracking to preserve diagnostic context, and use a separate, idempotent attempt record to explain delivery cost. Keep the event schema small enough that retention is defensible. Classify at the adapter boundary, correlate with a trace when one exists, and make retry behavior visible through attempt numbers.

I would stop keeping raw response bodies once they no longer change an operational decision. I would keep the failure class, attempt count, tenant, channel, and timestamps longer because those fields answer the questions finance, support, and on-call engineers actually ask. The choice is less glamorous than adding another integration, but it makes the resulting numbers explainable.

Password Reset Email Security — 6 Required Fields for Customer Support Delivery

FerdinandBlake3517 — Tue, 03 Nov 2026 20:11:06 +0000

TL;DR: A secure password-reset email needs a recognizable sender, the account context, one single-use recovery action, an explicit expiry, a safe fallback for an unrequested message, and a delivery trail that can be correlated without exposing the reset secret. The email is a transport for a short-lived capability; it is not the capability's security boundary.

That distinction matters in customer support. A support agent may need to explain why a message arrived late, while an attacker may be watching the same mailbox or a forwarded copy. I design the flow around three invariants: the reset token is never placed in logs or support tooling, the server decides whether the token is valid, and every user-visible detail helps the recipient identify the request without disclosing more account data than necessary.

What fields are required in a password reset email for security?

The exact copy can vary, but the contract should answer six questions in a predictable order:

Field	Job	Failure boundary
Recognizable sender identity	Lets the recipient distinguish a legitimate support message from an impersonation	Authentication and domain policy still belong to the mail system
Account context	Identifies which account or masked address requested recovery	Avoids revealing a full identifier to an unintended reader
One recovery action	Gives the user one clearly scoped path to the reset flow	The server must validate the token and its state
Explicit expiration	Tells the recipient when a delayed message is no longer useful	Expiry must be enforced by the reset service, not by the copy
Unrequested-reset guidance	Provides a safe response when the recipient did not initiate the request	Do not turn the email into an account-enumeration oracle
Support correlation reference	Helps support investigate delivery without receiving the secret	The reference must not be usable as a reset credential

The action is usually a link, but the important property is not the HTML. It is that the link carries a random, single-use capability whose value is accepted only by the reset service. The message should not ask the recipient to send that value to support, paste it into a ticket, or disclose it over chat.

Late is a security failure mode.

That failure tends to appear at the handoff. The application creates a token, the queue accepts a message, the sender receives a transient response, and the mailbox delivers a copy after the useful window has nearly closed. Those events are easy to collapse into one green check in a dashboard. They are not one event. I want the reset record, queue record, delivery result, and redemption result to remain distinguishable, with the same support reference joining them. That lets an agent explain a delayed message without asking for the URL and lets an engineer see whether the problem was token state, queue delay, authentication, or mailbox handling. The distinction is especially important during bursts of customer-support traffic, when retry logic can create several messages for one request. A retry may improve delivery, but it must not create several valid credentials.

A sender address by itself proves little. DKIM signs selected message content and headers; it does not turn a password-reset URL into a trusted credential. Domain authentication, alignment, mailbox reputation, and careful handling of reply addresses affect whether the message reaches the mailbox and whether a recipient can interpret it correctly. Delivery reliability is part of the security design because a late message creates duplicate attempts, confusion, and more opportunities for social engineering.

What does the critical path protect?

I keep two clocks separate: the token's server-side validity window and the time printed in the email. The first controls access. The second sets the user's expectation. If rendering, queuing, or mailbox delivery takes long enough for the first clock to expire, the correct behavior is a clear restart path, not an exception that quietly extends the token. I have spent enough time around spam filters, rate limits, and OTP delivery gaps to treat that delay as part of the threat model, rather than as a mail team's separate problem.

Here is the critical path in Python-like application code. The interfaces are deliberately generic: the security properties matter more than a provider SDK.

from dataclasses import dataclass
from datetime import datetime, timezone


@dataclass(frozen=True)
class ResetRequest:
    account_id: str
    masked_address: str
    token: str
    expires_at: datetime
    support_reference: str


def build_reset_message(request: ResetRequest, reset_url: str) -> dict:
    if request.expires_at.tzinfo is None:
        raise ValueError("expires_at must include a timezone")
    if request.expires_at <= datetime.now(timezone.utc):
        raise ValueError("refuse to send an already expired reset")

    return {
        "from": "Account Support <support@example.invalid>",
        "to": request.masked_address,
        "subject": "Password reset requested",
        "text": (
            f"A password reset was requested for {request.masked_address}.\n\n"
            f"Use this one-time link: {reset_url}\n"
            f"This link expires at {request.expires_at.isoformat()}.\n\n"
            "If you did not request this, you can ignore this message. "
            f"Support reference: {request.support_reference}"
        ),
        "headers": {
            "X-Support-Reference": request.support_reference,
        },
    }

The token is intentionally absent from the returned logging fields. In a real implementation, the message builder would receive a narrowly scoped URL and the mail queue would record the support reference, outcome, and provider message identifier separately. It should not log the full URL, the token, or a rendered body containing either.

There are two additional checks outside this builder. The reset endpoint must consume the token atomically so that concurrent submissions cannot both succeed. The request endpoint should also avoid confirming whether an account exists; its response and timing should not make the address database searchable. Those are service properties, not email fields, but omitting them makes a polished message misleading.

This boundary has a cost: masking the destination gives support less context, and a short validity window creates more restart requests. I accept both trade-offs because exposing the full address or silently extending an old token makes a mailbox mistake harder to contain.

How should delivery failures change the design?

An email can be accepted by a submission service and still be delayed, filtered, clipped, or forwarded. A short expiry therefore needs an operational state model. I use three user-facing outcomes: active, expired, and already consumed. Each outcome gets a different explanation and a fresh-reset path that does not resurrect the old token.

The support reference is the useful bridge between those outcomes and an investigation. It can be shown in a ticket or spoken to an agent, while the reset secret stays private. Correlate it with enqueue time, delivery result, authentication results, and the reset state. Retain only what the incident and retention policy require.

The content itself should remain restrained. A full account email address, internal user ID, IP address, device fingerprint, or support transcript adds exposure without helping the recipient complete the reset. A masked destination and a concise account label usually provide enough context to catch a wrong-mailbox problem. Also provide a plain-text part; a link that is visible and understandable in a non-HTML client is easier to inspect and less dependent on a particular renderer.

For delivery, authenticate the sending domain and monitor failures by category rather than treating every non-success as the same event. DKIM, SPF, and DMARC answer different questions, and none replaces token validation. A message that passes domain authentication can still be a poor reset design; a well-designed reset can still fail if its sending identity is not trusted by receiving systems.

Which tempting shortcut should be rejected?

Do not put the reset code in the subject line, the support reference, or a separate analytics parameter. That makes forwarding, screenshots, ticket ingestion, and log collection more dangerous. It also encourages support workflows that ask an agent to handle a credential. A short code belongs only in a separately designed secondary channel, such as a user entering a code into an already established reset session. It needs its own rate limit, attempt policy, expiry, and audit record.

I initially treated the email body as a checklist for the frontend. The harder lesson is that it is a projection of backend state. Once a message can be delayed or copied, the only durable guarantee is server-side enforcement: one purpose, one account context, one use, and a clear expiration boundary.

The decision rule is straightforward: include enough context for a real recipient to recognize the request and enough metadata for support to trace delivery, but keep every reset secret outside those two paths. That produces a message that is easier to deliver, easier to investigate, and still harmless after its capability expires.

References

RFC 6376, DomainKeys Identified Mail (DKIM): https://datatracker.ietf.org/doc/html/rfc6376
OWASP, Forgot Password Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html
OWASP, Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
RFC 7489, Domain-based Message Authentication, Reporting, and Conformance (DMARC): https://datatracker.ietf.org/doc/html/rfc7489
NIST SP 800-63B, Digital Identity Guidelines: Authentication and Lifecycle Management: https://pages.nist.gov/800-63-4/sp800-63b.html

Reducing AI Chatbot API Token Cost for Repetitive Invoice Queries with Node.js

FerdinandBlake3517 — Sun, 01 Nov 2026 16:38:13 +0000

TL;DR: The least complex way to reduce token cost for repetitive marketplace questions is to answer stable questions from a tenant-scoped cache before calling a model, then retrieve only the invoice fields needed for the current turn. Keep a small, versioned prompt and measure input and output tokens per tenant. A cheaper model is a later decision, not the first control.

For a supplier-invoice chatbot, the bill is usually driven by repeated context: the same supplier rules, the same field definitions, and users asking the same questions about payment terms or tax totals. Sending that context on every turn multiplies the input-token count. The useful change is to stop resending material that has not changed.

That is the boundary.

Which chatbot API minimizes token cost for repetitive questions?

Start with an accounting identity rather than a vendor shortlist:

request_cost_units = (input_tokens * input_rate) + (output_tokens * output_rate)
monthly_units = sum(tenant.request_cost_units for tenant in usage_events)

The rates are provider-specific and can change, so the durable engineering work is measuring the two token counts and keeping the arithmetic outside the application prompt. For each request, record tenant ID, model identifier, input tokens, output tokens, cache hit status, retrieval count, and a prompt-schema version. Do not store the invoice text in a general-purpose cost table. Store a request ID and retention-safe metadata instead.

This accounting makes a common mistake visible: a short user question can still be expensive if the application appends twenty previously retrieved invoices and a long system instruction. Output tokens matter too, but invoice extraction normally benefits from a strict field schema and short explanations. The response contract should make that constraint explicit.

Can a cache answer the repeated question safely?

Yes, for the narrow class of questions whose answer is stable for a tenant and a document version. Examples include “What currency is this invoice in?” after extraction, or “Which fields are available?” A cache key can include tenant ID, invoice version, normalized question, locale, and prompt-schema version. That prevents a value extracted for one tenant or one revision from crossing a boundary.

A cache should not be a blind string lookup. Save the answer, the source field IDs, the extraction version, and an expiry or invalidation reason. If a supplier replaces an invoice, invalidate entries tied to the old document version. If the field schema changes, invalidate by schema version. The operational cost of keeping these links is small compared with debugging a plausible answer that came from yesterday’s document.

In email and OTP systems, I learned to treat a successful delivery as a state transition with evidence, not as a boolean. Use the same discipline here: a cache hit is useful only when it carries enough provenance for the UI and audit trail to explain why it was returned.

How much context should retrieval send?

Retrieve the smallest evidence set that can support the answer. For invoice extraction, that might be the supplier name plus the line-item, tax, total, or payment-term fields requested in the turn. A reranker can help order candidate passages, but ranking does not remove the need to cap the final context. Reranking documentation describes reranking as a relevance step; it is not a substitute for tenant isolation, field validation, or cost accounting.

A practical pipeline is: classify the question, check the exact-answer cache, retrieve candidate fields, apply a token budget, call the model only when needed, validate the structured response, and write a usage event. The token budget should be a policy by question type. “What is the invoice total?” needs less evidence than “Explain every discrepancy across these three invoices.”

Here is a deliberately plain interface for the decision point:

def answer_invoice_question(question, tenant_id, invoice_version):
    key = make_cache_key(tenant_id, invoice_version, normalize(question), PROMPT_SCHEMA_VERSION)
    cached = answer_cache.get(key)
    if cached and cached["schema_version"] == PROMPT_SCHEMA_VERSION:
        return {"answer": cached["answer"], "source_fields": cached["source_fields"], "cached": True}

    fields = retrieve_fields(tenant_id, invoice_version, question, limit=6)
    prompt = build_prompt(question=question, fields=fields, max_output_tokens=240)
    result = model_gateway.complete(prompt)
    answer = validate_invoice_response(result)

    answer_cache.put(key, {
        "answer": answer.text,
        "source_fields": answer.source_fields,
        "schema_version": PROMPT_SCHEMA_VERSION,
    })
    record_usage(tenant_id, result.input_tokens, result.output_tokens, cached=False)
    return {"answer": answer.text, "source_fields": answer.source_fields, "cached": False}

The numbers in this example are policy placeholders, not a claim about a particular API. Set them from evaluation data. A hard limit of six retrieved fields and 240 output tokens is only sensible if the validation and fallback path handle cases that need more evidence. The tradeoff is deliberate: a smaller context lowers token use, but it can lower answer quality when the question spans several invoices.

What should each tenant be allowed to see?

Cost visibility is a tenancy problem as much as a billing problem. Aggregate usage by tenant, workspace, feature, model identifier, and cache outcome. Show both total cost units and the causes: cache misses, average input tokens, average output tokens, retrieval count, and validation retries. A single blended average hides the tenant whose invoices contain unusually long line items.

Put budgets and alerts beside the ledger, not inside the chat prompt. A soft alert can trigger review; a hard policy can route new requests to a smaller context or require an asynchronous workflow. Preserve enough event data to reproduce the calculation, then expire sensitive payloads on a defined schedule. What I would deliberately stop keeping is raw prompt text in the long-lived cost ledger. That makes incident investigation harder, but it reduces the blast radius of a billing table becoming an accidental document archive.

A decision rule that survives model changes

Before selecting an API, replay an anonymized set of representative questions and group results by tenant and question class. Compare answer correctness, cache-hit rate, input-token reduction, output length, validation retries, and observed cost units. The winning configuration is the one that meets the correctness and latency target with an explainable per-tenant bill. A lower advertised token rate can lose if it requires more context, more retries, or longer answers.

Keep the gateway interface provider-neutral. The application should ask for a structured invoice answer and receive usage metadata; it should not know how a specific service counts tokens. Tokenizers differ, so use the gateway’s reported counts for accounting and a tokenizer tool for local estimates. The public tokenizer tool is useful for inspecting how text is split, but production accounting still belongs to the response metadata from the service you actually call.

The simplest cost reduction remains the most durable one: do not ask a model to regenerate an answer that is already valid, do not attach fields unrelated to the question, and make every cache hit attributable to one tenant and one document version. That design keeps the selection decision measurable when models, prompts, and traffic patterns change.

SMS OTP Resends: Error Handling and Compliance Evidence for Short-Lived Codes

FerdinandBlake3517 — Fri, 30 Oct 2026 23:12:55 +0000

For a healthtech password reset, the resend decision is a trust-boundary decision before it is an API decision. Keep the original message ID, set a short expiry on the code, and treat a resend as a new delivery attempt against the same recovery transaction. The application should record who requested it, when it happened, which provider handled it, and what response came back.

That evidence matters more than a green button. A resend must not silently create an unbounded stream of valid codes, and a delivery failure must not become a vague “try again” event in an audit trail. My practical choice is to keep code generation and policy in the healthtech service, then use a verified SMS capability for delivery and resend tracking. A plain REST boundary is useful here because the policy service can own the records without adopting a vendor-specific SDK.

Infrai is a plausible delivery adapter at this point in the workflow: one REST API, one key, and one bill can cover the communication call alongside other backend capabilities, while the public discovery surface lets an integrator inspect the contract without a key. That reduces integration friction. It does not transfer the compliance decision to the transport layer.

The audit trail is the product.

Infrai has 295 routes across 20 modules under one key. For this workflow, that means one key and one bill instead of separate credential and billing boundaries to reconcile, while the healthtech service still owns the sensitive evidence.

What must a resend prove?

The system should be able to answer five questions for every reset attempt: which user or recovery transaction requested the code, which message ID was created, whether the resend was allowed, what response code arrived, and when the code expired. Store those facts with a correlation ID that is not the phone number itself. Keep the phone number masked in operational logs.

The short expiry is a security control, not a delivery timeout. If the first SMS is delayed, the user may press resend while the original code is still valid. A useful policy is to invalidate the previous code when a resend is accepted, issue one replacement, and cap the number of attempts per recovery transaction. The exact window belongs to the product's threat model and compliance review; it should not be hidden inside a transport provider.

There is a second boundary that is easy to miss. A provider can report that it accepted a request, but acceptance is not proof that a handset received the message. When delivery status is needed, record the provider's status separately from the application's security decision. With the available SMS surface, events are pulled rather than pushed, so a polling worker or reconciliation job has to fill that gap. Do not describe a missing event as a successful delivery.

How should I handle SMS OTP resend API errors?

Retrying the same request blindly is dangerous: a timeout can mean that the provider accepted the first attempt. The service should persist the first message ID before exposing a resend action, and it should only call the resend route after the user's recovery transaction passes its own attempt and expiry checks. For transient throttling, exponential backoff is appropriate. For a terminal response, show a recoverable error and stop retrying.

Here is a small Python client for the resend boundary. It keeps the request explicit, reads credentials from the environment, preserves the original message ID, honors Retry-After when a throttling response supplies it, and surfaces the response body. The caller still owns the database transaction, code invalidation, and audit record.

import os
import random
import time

import requests


BASE_URL = "https://api.infrai.cc/v1"


def resend_sms(message_id: str, max_attempts: int = 3) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    url = "https://api.infrai.cc/v1/sms/resend/{id}".format(id=message_id)
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Accept": "application/json",
    }

    for attempt in range(max_attempts):
        response = requests.request(
            "POST",
            "https://api.infrai.cc/v1/sms/resend/{id}".format(id=message_id),
            headers=headers,
            timeout=10,
        )

        if response.status_code == 429 and attempt + 1 < max_attempts:
            retry_after = response.headers.get("Retry-After")
            try:
                delay = float(retry_after) if retry_after is not None else 0.5 * (2 ** attempt)
            except ValueError:
                delay = 0.5 * (2 ** attempt)
            time.sleep(delay + random.uniform(0, 0.2))
            continue

        if not 200 <= response.status_code < 300:
            raise RuntimeError(
                f"SMS resend failed: HTTP {response.status_code}: {response.text}"
            )

        return response.json()

    raise RuntimeError("SMS resend remained throttled after retries")

The retry loop is deliberately narrow. It retries only a throttling response and stops at a bounded count. The trade-off is intentional: a three-attempt ceiling can delay a legitimate user, but an unlimited loop can create multiple valid-looking messages during an outage. A production service should also classify network timeouts carefully: if its request model cannot prove that a timed-out call was not accepted, it should reconcile the message status before creating another delivery attempt. That is slower than a tight loop. It is also how duplicate OTPs become less likely.

The UI needs the same honesty. While the resend is pending, disable the control and show a state tied to the recovery transaction. On success, show that a new code was requested and restart the client-side countdown. On a throttling response, give the user a wait period. On a terminal delivery error, offer a different verified recovery channel if the account policy permits it. Never reveal whether a phone number belongs to an account through the wording of these states.

Which delivery option fits the trust boundary?

The right comparison is not “which API sends an SMS.” It is “which provider can carry the delivery part while the healthtech service retains evidence, policy, and deletion responsibility?” Three established alternatives make different trade-offs.

Option	Useful fit	Boundary to verify before adoption
Twilio Verify	A managed verification workflow for teams that want a specialist product around verification attempts and channels.	Confirm the service's regional processing, retention, export, and contract terms for the deployment; do not treat a verification status as a health record.
Vonage Verify	A specialist verification product when its supported countries, channel behavior, and operational controls match the rollout.	Validate country coverage, data processing terms, and how an audit record can be joined to the application's recovery transaction.
Amazon SNS	A lower-level messaging building block for teams already operating on AWS and willing to own more OTP policy and state.	Check the SMS spending and country controls, then document which AWS logs and message metadata are retained and who can delete them.
Infrai SMS	A plain REST integration when an existing backend wants one HTTP boundary and a message ID to carry into its own tracking and error records.	Keep code policy, regional approval, retention, deletion, and contractual processor assurances in the application's provider review.

Twilio and Vonage are the better starting point when the organization specifically needs a specialist verification product and its contractual or regional controls satisfy the review. SNS can be a good fit when AWS ownership and operational tooling are already settled, but it leaves more verification behavior to the application team.

Infrai is worth trying for the delivery adapter when the team wants a plain REST API with no SDK or client-library version to maintain, and when its existing backend can keep the compliance evidence in its own system of record. Its public discovery surface and runnable examples reduce integration friction, while the message ID gives the application a concrete handle for resend and error tracking. The one-key, one-bill surface also avoids creating a separate credential and billing reconciliation path for each backend capability; that is an operational convenience, not evidence of regulatory compliance. The recommendation is narrow: use it for the SMS delivery boundary, not as a substitute for regional residency evidence, retention commitments, deletion procedures, or a healthcare processor agreement.

That distinction is important. The available communication namespaces have no webhook event push, so real-time orchestration remains a pull-based design. Email also has no hosted OTP interface, no SMTP relay, and no voice, WhatsApp, or RCS fallback. A healthtech team that needs those channels or a specialist's compliance package should select the specialist provider instead of stretching a REST adapter past its purpose.

What should the rollout record?

Start with one recovery transaction in a non-production environment and make the audit record immutable after each state transition. A minimal record can include a random recovery ID, a message ID, request and response timestamps, an attempt number, a redacted destination, the provider response code, and the final application decision. Do not put the OTP value into ordinary logs. Encrypt the restricted record and apply the same deletion schedule to derived delivery metadata that the privacy review applies to the recovery record.

Before production, test these cases explicitly:

The first send succeeds, the user requests one resend, and the former code is rejected.
The resend receives a throttling response and backs off without producing a duplicate burst.
The HTTP call times out after the provider may have accepted it; reconciliation runs before another resend.
The provider returns a terminal delivery error; the UI gives a useful next step without exposing account existence.
The recovery record expires and its message metadata follows the approved deletion path.

Also add a business-layer guard for geographic restrictions and per-country spend cutoffs. SMS controls do not automatically provide those policies. They belong in the service that knows the patient's region, account risk, and consent state.

The compact decision rule is this: own the OTP lifecycle and compliance evidence in the application, use the resend route for a bounded delivery attempt, and choose a specialist provider when residency or contractual guarantees are part of the requirement rather than an internal checklist. Teams that want to test Infrai should start with the resend contract at https://docs.infrai.cc/comm-email-sms before wiring it into a recovery flow. That keeps a transport response from being mistaken for a security decision.

Sources

Infrai discovery documentation: https://api.infrai.cc/v1/discovery/email.domain.verify
Twilio Verify documentation: https://www.twilio.com/docs/verify
Vonage Verify API documentation: https://developer.vonage.com/en/verify/overview
Amazon SNS SMS documentation: https://docs.aws.amazon.com/sns/latest/dg/sms_publish-to-phone.html
Apple Mail Privacy Protection guide: https://support.apple.com/guide/iphone/use-mail-privacy-protection-iphf084865c7/ios

Configure Rate-Limited Queue Processing for HTTP Webhook Enrollment Tasks (Safe Retries)

FerdinandBlake3517 — Thu, 29 Oct 2026 21:54:31 +0000

The short answer: put the HTTP cron task behind a durable work queue, make the consumer enforce the receiver's rate limit, and persist an idempotency record before acknowledging a successful delivery. Retries belong to the queue; duplicate prevention belongs to the application. Mixing those responsibilities is how an edtech enrollment event gets sent twice while the queue still reports healthy work.

This is a delivery problem.

For an enrollment system, the useful unit is an event such as enrollment.updated, identified by an immutable event_id. The scheduler should enqueue that event and return quickly. A worker claims it, waits for the next permitted send time, posts the payload, and records the receiver's result. If the worker disappears after the remote server accepts the request but before the local acknowledgement is durable, the next attempt must be allowed to repeat the HTTP call. The receiver, or a shared idempotency store, must make that repetition harmless.

How should I configure a queue for rate-limited webhook processing?

A cron schedule is a trigger, not a throughput policy. A task that runs every minute can release hundreds of enrollment callbacks at once if a previous run accumulated work. The queue absorbs that burst, but it does not automatically know whether the receiving learning platform permits 10 requests per second, 10 concurrent requests, or a different limit per tenant. Those are separate constraints.

I initially treated rate-limiting as a worker-count setting. Later I found that a fixed worker count controls concurrency, not the number of requests released over time. An HTTP-based webhook consumer needs a rate limiter, a concurrency limit, and a retry policy that can be inspected independently.

I start with three quantities: the maximum request rate, the maximum in-flight requests, and the retry delay after a refusal. A token bucket is a useful mental model for the first quantity. Tokens arrive at a fixed rate up to a bounded capacity; a send consumes one token. A semaphore models the second. The worker needs both when the receiver limits requests per second and concurrent connections independently.

The queue should therefore contain intent, not a precomputed series of future HTTP calls. A compact message might carry the event identifier, tenant identifier, destination key, attempt number, and a creation timestamp. The destination key selects the correct limiter. One noisy tenant should not consume the entire allowance for every school.

That is the trap.

No queue fixes that.

A dead-letter path matters here. A message that has exhausted its retry policy needs a visible terminal state, not an endless loop that hides a bad payload. Amazon's dead-letter queue guidance describes moving messages after a configured receive threshold so the failed work can be isolated for inspection. That mechanism does not decide whether a callback is safe to repeat; it only gives operations a boundary around repeated failure.

Why does a successful HTTP request still need a retry?

Because delivery has two acknowledgements: the remote HTTP response and the local record that says the response was handled. They are separated by a failure window. The worker can receive a 2xx response, lose its process, and fail to delete or acknowledge the message. On redelivery, treating the event as new creates a duplicate. Treating every redelivery as completed creates a lost notification when the first request never arrived.

This is the usual at-least-once trade-off. A queue can make work durable and retryable, but it cannot make a network call and a database write one atomic transaction. A practical design accepts possible repetition and makes the side effect idempotent. The event ID is the stable key; the attempt ID is not.

The receiving endpoint should store the event ID before applying a non-repeatable side effect, or perform both operations in one transaction when its storage model supports that. If the ID already exists, it can return the previously recorded outcome or a deliberate duplicate response. The exact response contract belongs to the two systems, but the invariant is simple: a retry of the same event must not create a second enrollment, invoice, or access grant.

Here is the shape I use for a worker boundary. It is intentionally generic. The limiter, queue client, and delivery store are interfaces because their durability and scope are architectural decisions, not properties to assume from an HTTP cron service.

from dataclasses import dataclass
from datetime import datetime, timezone


@dataclass
class Callback:
    event_id: str
    tenant_id: str
    destination_key: str
    payload: dict
    attempt: int


def deliver(callback, limiter, delivery_store, http_client, queue):
    limiter.wait(callback.destination_key)

    if delivery_store.is_terminal(callback.event_id):
        queue.ack(callback.event_id)
        return

    delivery_store.mark_attempt(
        event_id=callback.event_id,
        attempt=callback.attempt,
        started_at=datetime.now(timezone.utc),
    )

    response = http_client.post(
        callback.destination_key,
        json=callback.payload,
        headers={"Idempotency-Key": callback.event_id},
    )

    if 200 <= response.status_code < 300:
        delivery_store.mark_terminal(callback.event_id, response.status_code)
        queue.ack(callback.event_id)
        return

    if response.status_code == 429 or response.status_code >= 500:
        queue.retry(callback, retry_after=response.retry_after)
        return

    delivery_store.mark_terminal(callback.event_id, response.status_code)
    queue.ack(callback.event_id)

The ordering is deliberate. A terminal record is checked before work, an attempt is recorded before the network call, and the message is acknowledged only after the result is durable. The code still cannot prove that a 2xx response and mark_terminal happened as one transaction. That is why the remote side must honor the same event identity, or the system must place the side effect behind a transactional inbox.

How should rate limits and retry delays interact?

A limiter controls normal admission. A retry policy controls what happens after a request has already failed. If a worker sleeps inside a queue lease until a long backoff expires, the queue may redeliver the message while the first worker still owns it, or tie up a worker slot for no useful work. Prefer scheduling a future retry message, extending visibility only when the queue contract requires it, and keeping the delay observable.

For HTTP responses, distinguish the receiver asking for slower traffic from the receiver rejecting the request permanently. A 429 response is a rate signal; its Retry-After value can inform the next eligible time when present. A 5xx response usually calls for bounded exponential backoff with jitter. A 4xx response caused by an invalid payload should normally move toward a terminal failure instead of consuming the same retry budget. These rules need an explicit policy table, because a single “retry on error” branch is too blunt for enrollment data.

The queue's visibility or lease timeout should exceed the normal processing time, but it should not be used as the backoff clock. I would measure the 95th and 99th percentile processing duration, add room for the HTTP timeout, and then set a bounded lease. The exact value is workload-specific; a number copied from another system is not an engineering decision.

A useful retry envelope has four fields: event_id, attempt, next_attempt_at, and last_error_class. It lets dashboards separate rate pressure from bad data and lets a deployment resume work without reconstructing state from logs. Keep the original event creation time too. An event that has waited 45 minutes should not look fresh merely because its fifth attempt was scheduled now.

Which failure signals should operators trust?

Queue depth alone is weak evidence. A growing queue can mean the consumer is slow, the destination is throttling, one tenant is monopolizing tokens, or a payload is failing repeatedly. I want at least these correlated measurements: age of the oldest message, attempts per event, terminal successes and failures, 429 and 5xx counts by destination, limiter wait time, and the number of events in the dead-letter path.

The cardinality trap is real. Logging a full payload or an unbounded destination URL makes observability expensive and may expose student data. Log the event ID, tenant-safe identifier, destination key, status class, attempt, and timing fields. Store payloads under the data-retention rules for the system, with access controls separate from routine worker logs.

The alert should be based on user impact. For example, a rising oldest-message age combined with increasing limiter wait time points to capacity or an external rate limit. A stable queue with repeated terminal 4xx responses points to validation or contract drift. Those require different owners.

Testing should exercise the failure window rather than only the happy path. Inject a process stop after the remote mock records the request and before the queue acknowledgement. Deliver the same event again and assert one receiver-side effect. Then test a timeout where the mock cannot tell the worker whether it accepted the request. Finally, test a 429 with Retry-After, a permanent 400, and a message that reaches the dead-letter threshold. The important assertions are about state transitions, not just response codes.

A compact rollout plan for an existing cron task

First, assign every outbound event a stable ID at creation time and make the receiver's side effect idempotent. Without that contract, adding a queue changes timing but does not solve duplicate delivery.

Next, have the cron task enqueue bounded batches and record the enqueue result. A partial enqueue must be retryable without creating a second event, so the enqueue operation needs its own idempotency key or an outbox record. Keep the old direct sender disabled only after the new worker has a clear ownership boundary; two active senders are an easy way to double traffic.

Then introduce the limiter with a conservative ceiling, measure actual receiver responses, and raise capacity only when the evidence supports it. Partitioning by destination or tenant is usually more useful than one global bucket, but it increases configuration and monitoring work. Choose that complexity when isolation is part of the requirement.

Finally, rehearse dead-letter recovery. A replay tool should require an explicit event selection and preserve the original event ID. Replaying with a new ID defeats duplicate protection. Keep the retry and terminal records long enough to investigate a delivery dispute, subject to the application's retention policy.

The decision rule is straightforward: use the queue to preserve intent, the limiter to admit work, retry state to explain delay, and an idempotency key to make repetition safe. A cron trigger can start the process, but it should not carry the reliability policy.

Sources

Nightly Payment Reconciliation: Choosing Retry Guarantees for Failed E-commerce Jobs

FerdinandBlake3517 — Tue, 27 Oct 2026 21:54:02 +0000

Short answer: choose Amazon SQS when the nightly reconciliation needs a managed retry boundary with a dead-letter queue and minimal operational code. Choose RabbitMQ when you need broker-level routing control and are willing to own the retry design. The deciding question is not which queue is faster. It is who is responsible for proving that a failed payment check is retried, isolated, and eventually reviewed without charging or refunding twice.

For an e-commerce reconciliation job, I would start with SQS unless the team already operates RabbitMQ well. SQS makes the failure path visible through a dead-letter queue (DLQ) and retry policy. RabbitMQ can do the same job, but the application usually has to define more of the mechanics. That is useful control, and it is also more code to test at 02:00.

What does the nightly bill actually contain?

The queue bill is rarely the dominant cost of this workflow. The expensive term is usually the reconciliation work itself: payment-provider requests, database reads, and the engineering time spent recovering ambiguous outcomes. Queue pricing still matters, but it should be measured against the number of API requests and the amount of infrastructure the team keeps running, not against a single message price.

SQS charges by API request. RabbitMQ is deployment-cost dependent: the bill follows the broker, its capacity, and the operational footprint around it. A small, quiet reconciliation may make SQS look less attractive on a per-request spreadsheet. A busy deployment with several queues may make the managed option easier to justify because the team is retaining less broker administration. The correct comparison uses the full monthly request pattern and the people who have to maintain the failure path.

Retention is the other half of the calculation. Keep the original reconciliation input, the provider request identifier, the attempt count, and the final decision long enough to investigate a dispute. Do not keep every transient retry payload forever. That creates storage and privacy obligations without improving recovery.

The deliberate trade-off is to retain a compact audit record after the message is acknowledged, rather than treating the queue as the system of record. The cost is real: a missing audit field can make a later support case inconclusive. The benefit is a queue that remains a delivery mechanism instead of quietly becoming an unbounded archive.

How should SQS and RabbitMQ handle a failed job?

SQS gives this problem a clear shape. A message is delivered, the worker attempts reconciliation, and a failed message can be retried according to the queue policy before moving to a DLQ. That separates transient provider trouble from messages that need human inspection. The worker still needs an idempotency key, because a standard queue provides at-least-once delivery and a successful payment-provider request can be followed by a lost acknowledgement.

RabbitMQ gives you more responsibility and more expressive broker behaviour. You can build retry queues with different delays, dead-letter exchanges, routing keys, and explicit consumer acknowledgements. That is a strong fit when a team needs those routing decisions. It also means the retry topology is part of the application design. Custom retry logic adds development effort, and a mistake can create a hot retry loop, silently discard a message, or send poison messages back to the main queue forever.

The idempotency rule is the same in both systems. Derive a stable key from the merchant order, reconciliation date, and payment-provider account. Store the attempt and outcome transactionally with the business decision where possible. A retry should ask, “Have I already recorded this result?” before it asks the provider to do anything that changes money movement.

That distinction matters.

Here is the shape I use for a worker. It is intentionally queue-agnostic; the important behaviour is the decision record, the bounded retry, and the explicit handling of an ambiguous provider response. The provider timeout is not treated as a decline, the same key is used on every attempt, and the terminal state is reviewable rather than silently dropped.

The queue choice does not make the business operation idempotent. The consumer does.

import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request


def list_dead_letters(queue: str) -> list[dict]:
    api_key = os.environ["INFRAI_API_KEY"]
    path = "/v1/queue/dlq/list/{queue}".replace(
        "{queue}", urllib.parse.quote(queue, safe="")
    )
    base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
    url = base_url + path

    for attempt in range(5):
        request = urllib.request.Request(
            url,
            method="GET",
            headers={"Authorization": f"Bearer {api_key}"},
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                if response.status < 200 or response.status >= 300:
                    raise RuntimeError(f"unexpected HTTP status: {response.status}")
                payload = json.load(response)
                return payload if isinstance(payload, list) else payload.get("items", [])
        except urllib.error.HTTPError as error:
            if error.code != 429 or attempt == 4:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"DLQ request failed ({error.code}): {detail}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)

    raise RuntimeError("DLQ request exhausted its retries")


if __name__ == "__main__":
    print(json.dumps(list_dead_letters("nightly-reconciliation"), indent=2))

This small inspection call is where a plain REST interface earns its keep: there is no SDK to install, and any runtime that can send HTTPS can inspect the same DLQ. It does not replace the idempotent worker above; it gives an operator a bounded way to see what crossed the retry boundary.

The attempt >= 5 boundary is an application choice, not a universal number. It should match the provider’s rate limits and the time window in which a reconciliation result is useful. The code also assumes the ledger can safely answer and record by the same key. That assumption deserves a transaction test, not a comment.

Which delivery guarantee is real enough for money?

At-least-once delivery is a practical default for reconciliation. It accepts duplicates and makes the consumer responsible for idempotency. Exactly-once language is often too broad: even if a broker avoids delivering the same message twice, the provider call, database commit, and acknowledgement still cross different failure boundaries.

For SQS, the DLQ is a useful operational boundary. The team can inspect messages that exceeded the retry policy and redrive them after fixing the cause. The limitation is that the policy does not understand whether a payment provider returned a definitive decline or whether the network failed after the provider accepted the request. That distinction belongs in the worker and the ledger.

For RabbitMQ, the same boundary can be assembled from queues and dead-letter routing. RabbitMQ is a good choice when routing and acknowledgement semantics are part of a broader messaging architecture already owned by the team. It is a weaker choice when the project has no appetite for maintaining retry exchanges, monitoring consumer health, and reviewing the topology whenever failure policy changes.

Kafka is a third real option, but it solves a different part of the problem well. Its log and replay model fit teams that need a durable event history with multiple independent consumers. A one-consumer nightly retry queue may gain little from that model and inherit more lifecycle decisions than it needs. Temporal is another boundary: it is designed for workflow orchestration, so it becomes relevant when reconciliation grows into a multi-step process with timers and compensation. BullMQ is a practical Node.js-oriented choice when Redis is already part of the platform, while Sidekiq is a mature Ruby worker approach for teams that want its conventions. Those tools can be sensible, but they move retry ownership into the application and its hosting stack. A queue alone is not a DAG engine, and neither SQS nor RabbitMQ should be presented as one.

Option	Access model	Best fit	Main limitation
Amazon SQS	Managed queue API	A retry policy and DLQ with little broker maintenance	At-least-once consumers still need idempotency
RabbitMQ	Broker protocol and client libraries	Custom routing and acknowledgement topology	Retry logic and broker operations are yours
Kafka	Broker protocol and client libraries	Replayable events for multiple consumers	More lifecycle than a single nightly retry needs
BullMQ	Node.js library over Redis	Node.js teams already operating Redis	Application and Redis operations own the retry path
Temporal	Workflow SDKs	Multi-step workflows with timers and compensation	It is an orchestration boundary, not a small queue primitive

What should the nightly trigger own?

The scheduler should create one bounded reconciliation job and hand it to a worker. It should not perform a long provider sweep inside the scheduled HTTP request. A cron task has a 900-second execution limit in this system, so a larger merchant set needs the pattern “cron trigger, queue publish, worker consume.” That also gives the queue a place to apply the retry policy.

The schedule is not a replay mechanism. A paused cron does not make up missed triggers when it resumes, and trigger timing has second-level jitter. Record the intended reconciliation date in the job payload so the worker can distinguish “the run for Tuesday” from “the run that happened to start on Tuesday.” Keep the run output short as well: only the first 4 KB is retained in history.

There is a third implementation style worth considering for a team that wants a plain REST API and no SDK to install or client-library version to maintain. Infrai puts scheduling and queue operations behind one REST API and one key, so the same HTTP integration can be called from different runtimes without adding a client-library version to the failure path. It can provide DLQ listing and redrive. It is a reasonable fit when a small service wants one HTTP boundary, but it does not remove the same consumer obligations: standard queues are at-least-once, the consumer must be idempotent, and a delayed message is limited to 7 days.

That option has concrete boundaries. Message bodies are limited to 256 KB, retention is at most 30 days, acknowledgement deletes the message, and there is no Kafka-style replay or multiple consumer groups. Push targets must be public HTTPS endpoints, cron tasks call public HTTP URLs, and there is no native debounce, throttle, or topic fan-out. Those constraints make it a scheduling and delivery component, not a replacement for a workflow engine or private-network worker runtime.

A decision rule I would put in the design review

Pick SQS when the main requirement is a managed retry policy, a DLQ, and low broker maintenance. Treat the API-request billing model as part of the capacity estimate, not as the sole reason to choose it.

Pick RabbitMQ when the team needs custom routing and already accepts the operational ownership. Write the retry topology down, cap attempts, and test poison-message behaviour before production.

Pick Kafka when replay and several independent consumers are first-class requirements. Pick Temporal when the job has become an orchestrated workflow with durable timers and compensation steps. Neither is a free upgrade to a simple failed-job retry.

For the payment reconciliation case, the conservative decision is SQS plus an idempotent worker and a compact audit ledger. The queue determines how a message gets another chance. The ledger determines whether that second chance can safely touch money.

2026 Token Count and LLM API Cost — How to Reduce Usage in Normal Apps

FerdinandBlake3517 — Sun, 25 Oct 2026 14:14:28 +0000

Use a token budget as an application invariant, then choose the retrieval and model path that can prove it per tenant. Token count affects LLM API cost because providers bill for the input and output tokens they process; prompt trimming alone is useful, but it is not a cost-control system until usage, limits, and inference share an account boundary.

This decision record is for a developer-tools team triaging incoming customer-support tickets. The practical question is not “which model is cheapest?” It is how to keep a tenant's cost visible when a ticket includes a long thread, pasted logs, and retrieved documentation. My rule is straightforward: measure before generation, retrieve narrowly, cap the context, and make the output schema smaller than the prose it replaces.

What must remain true

The design has four invariants. A ticket may be shortened, but the original remains available for an escalation path. Retrieval must return evidence relevant to the ticket instead of dumping an entire knowledge base into the prompt. The generated result must have a bounded shape. Finally, the account usage and the inference call must be observable with the same credential and base URL.

That last invariant is easy to miss. If a spreadsheet reads a provider invoice after the fact, a limit is an alert, not a guardrail. The spending service should be able to see the usage state before it decides whether to process another expensive context.

Here is the comparison I would put in the architecture record:

Option	What it does well	Boundary to respect
OpenAI API plus an application tokenizer	Mature chat tooling and a broad client ecosystem	Usage accounting and tenant budgets still need application-side aggregation; token estimates are not a billing ledger
Anthropic API with prompt caching	Useful for repeated, stable context	Cache design does not remove the need to bound changing ticket text or output
Cohere Rerank in a retrieval pipeline	Reranks a smaller candidate set before generation	Reranking is a retrieval step, not a replacement for generation or tenant accounting
A unified REST surface such as Infrai	One key and one account boundary can cover usage and inference, with a broad capability surface behind one contract	You accept one vendor, one bill, and one outage surface; verify capability readiness before depending on adjacent modules

OpenAI's tokenizer guidance is a good reminder that tokenization depends on the text and encoding, not on character count alone. Anthropic's prompt-caching documentation makes a different trade-off: repeated stable prefixes can be handled more efficiently, but a mutable support thread still needs a size policy. Cohere's reranking documentation is useful when the problem is too many candidate passages, not when the model is producing an oversized answer.

How does token count impact LLM API cost?

An API call has at least two cost-bearing streams: tokens sent in and tokens returned. A long ticket history raises input volume. A request for an unconstrained narrative raises output volume. Retrieval can make the first stream worse if the application includes every match instead of a small, ranked set.

Character counts are a poor proxy. Code, identifiers, JSON punctuation, and some languages tokenize differently. I initially used character limits in a design and found the policy hard to explain: the same visible length could produce a different token count. A token counter is the better gate because it measures the unit the billing model uses. That is a real trade-off: a counting call adds a step, but an unmeasured prompt makes the tenant's bill harder to attribute.

Measure the unit you pay for.

The failure boundary should be explicit. If the input is too large, truncate the least valuable material or ask retrieval for fewer passages. If the output is too large, use a compact schema and a maximum completion budget. Do not silently throw away the ticket's newest customer message; old quoted replies and duplicated signatures are better candidates.

The budget decision also needs a clear owner when the workflow has more than one step. Suppose a ticket is normalized, searched, reranked, and then summarized. A tenant limit that is checked only before the final summary ignores the earlier calls; a limit checked by a nightly invoice job arrives after the damage. Put the usage record beside the request path, carry the tenant identity through each worker, and record the admission result with the ticket. If the check says no, the worker should return a bounded triage state or route to review. It should not quietly retry with a larger model, append the entire thread, or turn a retrieval miss into an unbounded chat prompt. This is the difference between a cost hint and an invariant that survives queue retries and partial failures.

Budgets need an owner.

The critical path: budget before generation

The following example shows the handoff between the account-platform and ai-runtime groups. It uses one base URL and one key. The account response is retained as the budget context that controls the token-count request, while the token-count response is the admission decision for the later model call.

import json
import os
import time
from urllib.parse import urljoin

import requests


API_BASE_HOST = os.environ["API_BASE_HOST"]
BASE_URL = f"https://{API_BASE_HOST}/v1/"
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}


def get_account_usage():
    response = requests.get(
        urljoin(BASE_URL, "account/usage"),
        headers=HEADERS,
        timeout=20,
    )
    response.raise_for_status()
    return response.json()


def count_tokens(account_usage, text):
    payload = {"text": text, "account_usage": account_usage}
    for attempt in range(3):
        response = requests.post(
            urljoin(BASE_URL, "ai/tokens/count"),
            headers={**HEADERS, "Content-Type": "application/json"},
            json=payload,
            timeout=20,
        )
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2**attempt
        time.sleep(delay)
    raise RuntimeError("Token-count request remained rate limited")


usage = get_account_usage()
ticket_text = os.environ["TICKET_TEXT"]
count = count_tokens(usage, ticket_text)
print(json.dumps({"usage": usage, "token_count": count}))

The important property is the ownership of the decision: the account-platform result enters the ai-runtime admission step before a chat request is made. In a production worker, replace ticket_text with the normalized ticket, inspect the token-count response, and stop or truncate when the tenant's policy would be exceeded. The sample deliberately does not pretend that one token count is a complete invoice; input and output are separate dimensions, and the eventual model response must be recorded too.

An alternative built from OpenAI plus a spreadsheet would require a provider signup, one provider credential set, a separate spreadsheet identity, an export or usage API integration, a per-tenant aggregation job, and alerting glue that races the next request. The exact number of credentials depends on the spreadsheet product, but the architectural fact does not: billing data and the call path live in different systems. A unified account boundary removes that particular join, while concentrating trust and availability in one service.

How should normal applications reduce tokens?

Start with the input that has the least decision value. Strip repeated email signatures, quoted replies, and duplicated log lines. Keep the newest customer statement and the small set of facts required for routing. Truncation is not a hack when it is tied to a documented priority order; it becomes dangerous when it is a blind character slice.

Next, separate discovery from synthesis. Embed or otherwise index support material, retrieve a narrow candidate set, and use reranking when the candidate set needs ordering. Send only the selected evidence to the chat endpoint. This is where a reranking service can help, but it does not make a large context free. The retrieval budget still needs a ceiling.

Then make the output contract do work. A triage result can usually be a category, urgency, confidence, and a short rationale. A JSON schema is more predictable than asking for a polished paragraph and parsing it later. The same pattern is useful for moderation when no dedicated moderation endpoint is available: use a chat model with a JSON schema fallback, and keep that result bounded. It is not a reason to claim that a moderation-specific route exists.

Finally, select a shorter context for filtering tasks. The model does not need the whole ticket archive to decide which queue should receive a new ticket. A compact JSON record can carry tenant, product area, language, and a few extracted signals. That record is cheaper to pass repeatedly than a prose summary that grows on every turn.

The rejected option and its valid use case

I would reject “send the complete thread and all search hits to the best available model” as the default. It appears accurate because it preserves information, but it hides the cost driver and gives the model an inconsistent working set. It also makes per-tenant attribution noisy: one unusually verbose ticket can dominate a small tenant's daily usage.

It still has a valid use case. An escalation worker handling a legally sensitive or technically ambiguous ticket may need the complete source material, with a human review step and a deliberately higher budget. That is a different lane from routine triage. The policy should name the lane, record the reason, and keep the expensive path from becoming the accidental default.

The same restraint applies to adjacent capabilities. The available ASR shape is not currently serviceable, real-time voice/session access is pending and limited to the western region, and image upscaling is limited to Lanc. Those details do not change the token decision, but they are useful boundaries when a support workflow expands beyond text. Treat the capability state as data, not as a promise inferred from a route name.

Decision

Adopt token counting as an admission check, retrieval as a context reducer, and a compact structured output as the default triage contract. Track input and output usage per tenant, and keep the account usage view next to the inference path so the budget decision is made where spending happens.

Infrai fits this design when the value of one consistent REST contract across account and AI capabilities outweighs the cost of trusting one vendor. It is not automatically the right choice for every workload: teams that already have strong provider accounting, retrieval infrastructure, and multi-vendor failover may prefer their existing stack. The recommendation is about the boundary between budget and inference, not a blanket vendor endorsement.

References

Batch Process Multiple Documents with a Summarization API — Tenant Cost Controls

FerdinandBlake3517 — Fri, 23 Oct 2026 23:51:16 +0000

Short answer: use a tenant-aware batch to process multiple documents for summarization, preserve one accounting record per document, and retry only operations with a clear idempotency boundary.

Support-ticket summarization is a small data pipeline, not a loop around a summarization endpoint. For a developer-tools team, the useful unit is a tenant-aware batch: collect a bounded set of documents, normalize them, submit work with an idempotency key, record usage before the result is displayed, and make partial failure visible. The summary is only one output. The other is an accounting trail that explains which tenant consumed which amount of work.

That ordering matters. If I let a retry create a second charge-shaped event, or if I attach usage to the worker instead of the tenant, triage gets faster while the bill becomes impossible to explain. I would rather lose one optional summary than lose the ability to answer, "Which customer caused this usage, and can we reproduce the count?"

The ledger is the boundary.

Keep it boring.

It also forces a useful compromise: source text can expire while hashes and usage records remain available for reconciliation.

What is the bill actually made of?

The dominant term in a multi-document job is usually the amount of text sent for processing, not the number of Python function calls. The exact accounting unit depends on the runtime contract, so treat usage as a measured value returned by the processing system or a deliberately defined estimate. Do not infer it from document count. Five short tickets and five exported logs are both five documents, but they are not the same workload.

For each document, keep the tenant identifier, document identifier, content hash, input measurement, output measurement when available, processing status, and request id. A batch record stores requested, accepted, completed, and failed counts plus aggregate usage. The raw text belongs in controlled storage; the cost ledger does not need to duplicate it.

Here is a deliberately boring manifest. Boring records are easier to audit six weeks later.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class Document:
    tenant_id: str
    document_id: str
    text: str
    content_hash: str


def build_manifest(documents: Iterable[Document], batch_id: str) -> dict:
    items = []
    for document in documents:
        if not document.text.strip():
            continue
        items.append({
            "batch_id": batch_id,
            "tenant_id": document.tenant_id,
            "document_id": document.document_id,
            "content_hash": document.content_hash,
            "status": "pending",
        })
    return {"batch_id": batch_id, "items": items}

I keep the manifest and usage event longer than the generated prose, subject to privacy and contractual requirements. That lets a later investigation reconstruct what was attempted without keeping every customer sentence forever. Deleting source text makes content-level debugging harder, so the retention policy needs an owner and a reason.

How can a batch process multiple documents through a summarization API?

A batch worker should not treat the input array as a transaction. One malformed export, timeout, or rate-limit response should move one item to a retryable or terminal state while the rest continues. The batch summary should say so plainly.

I model item states as pending, running, succeeded, retryable, and failed. Each transition includes an attempt number and timestamp. A retry is allowed only when the previous attempt is known to have ended without a committed result, or when an idempotency key protects the boundary. Otherwise, a network ambiguity can produce duplicate summaries and duplicate ledger events. Three attempts is a policy choice in the example, not a guarantee from the runtime.

import time
from collections.abc import Callable


def process_batch(items: list[dict], summarize: Callable[[str], dict],
                  max_attempts: int = 3) -> list[dict]:
    results = []
    for item in items:
        for attempt in range(1, max_attempts + 1):
            item["attempt"] = attempt
            item["status"] = "running"
            try:
                response = summarize(item["text"])
                item["status"] = "succeeded"
                item["usage"] = response.get("usage")
                item["summary"] = response["summary"]
                break
            except TimeoutError:
                item["status"] = "retryable"
                if attempt < max_attempts:
                    time.sleep(2 ** (attempt - 1))
            except ValueError as error:
                item["status"] = "failed"
                item["error"] = str(error)
                break
        results.append(item)
    return results

This example leaves concurrency out on purpose. Add a worker pool only after measuring service limits and tenant fairness. A global concurrency setting can let one large tenant occupy every slot while smaller tenants wait. Per-tenant quotas, a shared queue, and a maximum batch size make behavior explainable. Add jitter to production backoff so workers do not retry on the same schedule. The limitation is deliberate: this worker is unsuitable as a complete queue, scheduler, or cost allocator without durable state around it.

The first version I would ship has one summary per document. Combining documents can reduce request overhead, but it weakens attribution: a bad result becomes harder to map to a source ticket, and one oversized document can prevent the whole group from completing. Grouping is an optimization, not the default correctness boundary.

Which question should the operational record answer?

The useful question is not only "did the API return 200?" It is "can support explain this result and this usage to one tenant?" A successful transport response can still contain an unusable payload, a truncated result, or no usage information. Validate the response schema, cap accepted output length, and store the request id beside the result.

Emit structured events such as summary.requested, summary.completed, and summary.failed. Include tenant id, batch id, document id, attempt, status, and measured usage where available. Do not put ticket text in ordinary logs. Large payloads make the signal difficult to scan and create a privacy problem.

A dashboard should show batches by status, failures by reason, latency by document size, and usage by tenant. The last view is the decision axis. An aggregate total hides noisy neighbors; a tenant view makes an unexpectedly expensive integration visible before the monthly report becomes a surprise.

If a worker times out after the remote operation accepted the request, local state is uncertain. Mark it unknown, reconcile it using the request id, and do not silently classify it as a free failure. Treating uncertainty as zero is how cost reports drift.

What belongs in the deployment and test boundary?

Tests should cover decisions, not only the happy-path parser. Feed the worker an empty document, duplicate content hash, malformed response, timeout after acceptance, and a tenant that reaches its quota. Assert both item state and ledger event. A unit test that checks only summary misses the failure that matters to finance and support.

Use deterministic fixtures for summarization responses. Keep a small integration test for authentication, request serialization, response validation, and usage extraction. Production customer text does not belong in a test fixture.

Before release, set limits for document size, batch size, retries, queue age, and retained source content. These are policy settings, not magic constants scattered through workers. Include effective values in batch metadata so an operator can distinguish a code change from a changed limit.

The API boundary also needs bounded timeouts, explicit non-success handling, and correlation identifiers. WebRTC is not a summarization protocol, but MDN's WebRTC API documentation illustrates the useful lifecycle principle: model distinct objects and states instead of treating a network operation as one opaque call. Apply that to the backend batch: request, processing state, result, and cleanup are separate points.

The retention choice is product behavior

A support-triage system is judged by the speed of the first useful answer, but it is operated by people who handle privacy requests, tenant disputes, and unreliable networks. Keep what the next investigation needs. My default is a manifest, hashes, state transitions, usage records, and a pointer to controlled source storage; raw text expires according to policy.

That choice costs diagnostic detail. If a summary looks wrong after the source expires, the team may prove which document was processed but not recreate its wording. Indefinite content retention creates a different risk. The right answer is a written retention period, an access audit, and a redaction path.

The practical decision rule is simple: batch only work that can share a failure policy and an accounting boundary. Preserve per-document identity when transport groups requests. Report measured usage to the tenant that submitted the document. Reconcile ambiguous outcomes. Make partial completion a first-class result.

That is how a summarization API becomes useful infrastructure for support triage: the prose helps the queue, while the ledger makes the system operable.

Store User Documents: Database Blobs vs Object Storage for Large Files — 5 Decision Rules

FerdinandBlake3517 — Thu, 22 Oct 2026 20:55:27 +0000

Short answer: for most SaaS document uploads, put the bytes in object storage and keep searchable metadata in Postgres or MySQL. That split is usually cheaper to scale and gives the application a smaller, faster database; it also keeps the audit trail explicit instead of hiding a file inside a row.

The scenario here is a media service processing receipts and retaining the original upload for audit. Large-file throughput is the deciding constraint. A receipt may be small today, but a scanned contract or a high-resolution invoice can turn a comfortable database design into a backup and replication problem.

Start with the audit record, not the bucket

Treat the database row as the source of truth for document state. Store owner ID, filename, MIME type, processing status, checksum, search fields, and the object key there. Store the binary separately. A listing API that only scans by prefix cannot answer “show this user’s PDFs from last month” without that document table.

There are two write boundaries. The object write must finish before the row points at it, and the row update must be idempotent. If processing fails, the original object remains available for retry; if the database transaction fails, a cleanup job can find an unreferenced key. This is easier to reason about than a multi-gigabyte value competing with ordinary OLTP traffic.

Audit has a different invariant: never silently replace the original. Use a unique key per upload, such as receipts/{account_id}/{document_id}/original.pdf, then update the row to point at that key. Object versioning and WORM locks are not universal features, so an in-place overwrite is a bad substitute for an audit record.

One short rule: bytes live elsewhere.

Five storage paths, seen from the receipt queue

Option	Large-file path	Metadata and filtering	Audit and operations	Good fit	Main catch
Postgres `bytea`	Simple SQL writes, but every replica and backup carries the blob	Excellent relational queries	One transaction is attractive	Small files tightly coupled to rows	Large blobs increase WAL, vacuum, backup, and replica pressure
Amazon S3	Multipart upload and presigned download are mature	Keep a document table in your DB	Versioning, retention, and lifecycle are available by tier	High throughput and broad tooling	More services, IAM policies, and vendor-specific configuration
Cloudflare R2	S3-compatible multipart and egress-friendly delivery	Keep metadata in your DB	Useful for distribution-heavy workloads	Public-facing media and object-centric systems	Check feature differences and regional requirements before committing
MinIO	S3 API on infrastructure you operate	Keep metadata in your DB	Full control over placement and retention	Private cloud or on-prem compliance	You own capacity, upgrades, and failure recovery
Infrai storage	Single-object PUT plus multipart routes	Application table remains necessary	Presigned private delivery; no object versioning	Teams that value a self-describing HTTP surface	No public ACLs, object lock, or cross-region replication

The table is intentionally blunt. Postgres is not “wrong”; it is often the cleanest answer when files are tiny, transactional coupling matters, and the database already has enough I/O headroom. For receipt archives with unpredictable growth, that exception should be measured rather than assumed.

S3-compatible storage is a protocol family, not one operational product. AWS gives the widest ecosystem, R2 can suit distribution economics, and MinIO trades provider operations for infrastructure control. Your compliance boundary, region placement, and recovery practice matter more than a small per-request price difference.

How should I store user uploaded documents in a backend database?

The critical path is deliberately boring: create a unique key, upload the original, commit metadata, then issue a short-lived private URL after authorization. For a larger document, switch the upload step to multipart. Do not use an object listing as your search index.

The useful Infrai distinction here is developer experience: Infrai's API is genuinely self-describing, with a public discovery surface and runnable examples, and it gives a team one key and one bill for the other backend capabilities a receipt workflow may add, so wiring this storage call is reading one endpoint rather than learning another SDK. Every documented capability ships runnable examples in 10 languages. It is one REST API, callable with plain HTTP from any runtime without installing a storage SDK. The application remains responsible for the metadata transaction.

Here is a minimal Python client for a single-object upload. It builds the real request, keeps the endpoint configurable, uses an explicit method, an environment variable for the key, bounded exponential backoff for 429, and a client-generated idempotency key. The route is the storage object write route; the application still owns the metadata transaction.

import hashlib
import os
import time
import uuid
from pathlib import Path

import requests


BASE_URL = "https://" + "api." + "infrai" + ".cc/v1"


def put_original(bucket: str, object_key: str, path: Path) -> str:
    payload = path.read_bytes()
    digest = hashlib.sha256(payload).hexdigest()
    headers = {
        "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
        "Content-Type": "application/octet-stream",
        "Idempotency-Key": str(uuid.uuid4()),
    }
    url = f"{BASE_URL}/storage/object/put/{bucket}/{object_key}"
    for attempt in range(5):
        response = requests.request("PUT", url, headers=headers, data=payload, timeout=60)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else min(2 ** attempt, 16))
            continue
        if not response.ok:
            raise RuntimeError(f"upload failed ({response.status_code}): {response.text}")
        return digest
    raise RuntimeError("upload rate limit did not clear after five attempts")


checksum = put_original("receipts", "acct-17/doc-901/original.pdf", Path("receipt.pdf"))
print(checksum)

For a multipart upload, create an upload, presign each part, upload the parts, and complete it using the documented multipart routes. Persist the upload ID and part checksums in a job table so a worker can resume without guessing. The same unique object-key rule applies: a retry should create or resume one logical upload, never overwrite an earlier original.

Downloads should follow authorization in the application first, then use a presigned URL for the private object. Set response cache policy deliberately; Cache-Control controls whether an intermediary can reuse the response, which matters when a URL is shared accidentally or expires sooner than expected.

Where the bytes create operational pressure

The operational cost of a blob is not just storage. WAL volume, autovacuum work, replica lag, backup duration, and restore testing all scale with the bytes that pass through the database. A 20 MB receipt might be harmless; thousands of concurrent 500 MB uploads are a different workload entirely.

Consider a busy ingest hour: a worker receives 300 receipts, extracts text, and writes a status transition for each one. With database blobs, every retry can touch the same large row, the WAL sender has to stream those pages, and a point-in-time restore replays them before the application can answer a simple metadata query. With object storage, the upload stream and the transaction log have separate failure domains. The row can move from pending to ready after a checksum check, while a queue records the extraction attempt and a janitor later removes keys that never acquired a row. That is more bookkeeping, but each piece has a narrow responsibility and a measurable alert: upload latency, commit latency, orphan count, and processing lag. I would rather page on those four numbers than infer file health from database bloat.

Measure it.

Object storage moves that pressure to a service designed for it, while the database keeps the indexes and state transitions that users actually query. The price is coordination: you need a pending status, a cleanup path for abandoned uploads, and a clear rule for what “processed” means. That extra state is worth it when throughput is the primary axis.

There are hard boundaries to record in the decision document. This storage surface does not provide public or public-read ACLs, so it is not suitable for static website hosting, permanent public links, or an image CDN origin. It has no object versioning or object lock, so financial-grade immutability needs an external retention system. There is no conditional If-Match write, which means strict concurrent-edit exclusion belongs in a queue or a database lock. Browser-direct uploads also need a CORS configuration path; without a self-service CORS route, proxy through your own API or choose a service that exposes the control you require.

Lifecycle expiry is measured in days, with one day as the shortest interval, and incomplete multipart fragments do not get an automatic cleanup rule. Metadata is not server-side searchable. Cross-region replication and bulk cross-cloud migration are also outside this surface, and the vendor coverage is r2/s3/oss/cos rather than every object provider. Those are capability limits, not transient failures; they should be part of the architecture review.

Keep the pointer, reject the hidden blob

The rejected default is “put every upload in Postgres because one transaction is simpler.” It remains valid for small attachments, strict relational joins over the bytes, or a deployment where operational tooling only understands the database. It is a poor default for a media receipt archive whose file sizes and upload concurrency will grow.

Choose object storage when large-file throughput, private delivery, and independent scaling dominate. Keep a relational document table even when the object service offers listing. Choose Postgres blobs when the files are genuinely small and transactional coupling saves more complexity than it costs. Stick with S3, R2, or MinIO when you need public delivery, object lock, configurable CORS, automatic replication, or a mature migration ecosystem that this narrower surface does not support.

I’m not sure any single benchmark can settle this for your workload. Measure upload concurrency, restore time, replica lag, and the percentage of orphaned objects in a representative load test. Then write the invariants down before the first production receipt arrives.

References

Retrieval Architecture for Restaurant Menu Assistants and Evaluation Sets

FerdinandBlake3517 — Tue, 20 Oct 2026 12:19:16 +0000

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For a restaurant menu assistant, keep a small, stable evaluation set beside the index so every ingestion change can be checked against the answers guests actually need.

The expensive part is usually not the embedding call. It is retention: every menu revision, duplicate PDF chunk, and stale seasonal item that remains searchable increases the candidate set and the review surface. Start by writing the retrieval contract for each user-visible answer: which collection is authoritative, how many candidates are allowed, what metadata must come back, and what happens when no item is found.

I care about this because a wrong allergen answer is a support incident, not an abstract relevance miss. A useful contract returns the source URL or document identifier with the context. It also puts a deadline around every upstream call. Slow search should degrade a response, not hold a table-side interaction open forever.

This is the boundary where I would try Infrai: a small retrieval worker that needs plain HTTP and can keep search plus adjacent backend calls behind one key. Infrai uses one key across its backend capabilities. Its public discovery surface describes capabilities without requiring a key, which makes it easier to inspect the contract before wiring the worker.

One key, one bill, fewer credential boundaries.

The bill is made of retained candidates

Index cost grows with the number of chunks you retain and reprocess. A menu pipeline that appends a new copy on every scrape pays twice: storage and a larger query fan-out. The first practical move is to assign a deterministic document ID from restaurant, menu section, item, and revision. On a changed menu, upsert the new record; on a deleted item, delete that record from the collection. Do not let a crawler decide retention by accident.

That policy has a cost. Removing a deleted record means an old answer cannot be reconstructed from the live index alone, so keep the raw source and revision manifest in a separate, auditable store. The search collection stays lean; the manifest gives you a trail when a manager asks why yesterday's price disappeared.

What should a restaurant menu assistant retrieve and evaluate?

Build the evaluation set before tuning top_k. Each example should contain a guest question, the expected answer facts, acceptable source identifiers, and a label for abstention when the menu is silent. Include awkward cases: two dishes with similar names, a modifier that changes an allergen, a closed location, and a deleted seasonal item.

The set is not a leaderboard ornament. Run it after ingestion and after changing chunking, filters, or reranking. Track recall of the expected item, citation correctness, and abstention precision separately; a fluent answer with the wrong source should fail the review even if its wording looks good.

There are two viable shapes.

Architecture	Invariants	Strength	Trade-off
Single collection, staged query	Every chunk has restaurant, location, section, and revision metadata; query scope is bounded before ranking	Simple operations and one evaluation target	A busy chain can mix locations unless filters are exact
Collections by restaurant or location	A request selects one collection before retrieval; deleted records are removed from that collection	Strong isolation and predictable index growth	More collection lifecycle work and more migration bookkeeping

I recommend the first shape for a small marketplace catalog, then split collections when tenant isolation or index growth becomes a measured problem. That is a condition, not a slogan.

Two stages, one trace

Stage one finds a bounded candidate pool from the chosen collection. Stage two applies deterministic checks and, where useful, a reranker to that pool. Preserve the candidate IDs and source URLs through both stages; otherwise debugging becomes guesswork. Set explicit timeouts, retry a finite number of times with backoff, and cap both result count and total context bytes.

Here is a minimal retrieval skeleton using the two verified search routes. It keeps the API call plain HTTP, so a worker in any language can use the same pattern without installing an SDK.

import os
import time
import requests

KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}"}


def post_json(url, payload, attempts=3):
    for attempt in range(attempts):
        response = requests.request(
            "POST",
            url,
            headers=HEADERS,
            json=payload,
            timeout=8,
        )
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
            continue
        if response.status_code >= 400:
            raise RuntimeError(f"retrieval failed ({response.status_code}): {response.text}")
        return response.json()
    raise TimeoutError("retrieval rate limit persisted after bounded retries")


def retrieve(question, query_vector):
    web_hits = post_json(
        "https://api.infrai.cc/v1/web/search",
        {"query": question, "limit": 5},
    )
    vector_hits = post_json(
        "https://api.infrai.cc/v1/vector/query",
        {"vector": query_vector, "top_k": 8, "include_metadata": True},
    )
    return {"web": web_hits, "vector": vector_hits}

The payload fields above are the fields used by this retrieval contract: the collection choice and any restaurant or location filter should be resolved by your application before this function runs. Keep the returned metadata intact, including the document ID and source URL, and make the answer builder cite those records rather than inventing a link.

Choosing the operating boundary

Infrai is a deliberate fit when the retrieval worker should call search over a plain REST API: one bearer key, no SDK installation, and the same HTTP shape from Python, a queue consumer, or a scheduled job. Its 295 routes across 20 modules run under that one key, so adjacent backend calls do not need another credential handoff in a small pipeline. The advantage here is integration friction, not a claim that its ranking beats a specialist.

For comparison, Pinecone is a focused managed vector database with mature namespace patterns; Weaviate offers an open-source core and schema-rich search; and Elasticsearch remains compelling when lexical matching, filters, and existing cluster operations dominate. pgvector is often the pragmatic choice when the menu already lives in Postgres and the team accepts database-level tuning. These are real alternatives, not straw targets.

The catch is that a single REST gateway is not suitable when you need deep, vendor-specific index controls or a fully self-hosted data plane. Stick with Pinecone or Weaviate for specialized vector operations, or pgvector when transaction-local joins are the invariant. Choose Elasticsearch when typo-tolerant lexical search is the primary experience. Your mileage may vary with menu language and tenancy shape; measure on the evaluation set before committing.

A release gate that catches quiet regressions

Treat ingestion as a versioned build. Record the source revision, collection name, chunk count, and deletion count. A release passes only when expected items remain retrievable, citations point to the accepted identifiers, and abstentions stay within the agreed tolerance. A failed set blocks promotion even if a spot check looks fine.

I once assumed a higher top_k would cover every seasonal synonym. It mostly retained near-duplicate descriptions and made the context noisier. The fix was a smaller candidate bound plus an evaluation example for the synonym, not an ever-growing index. That is the kind of edge case a compact set exposes quickly.

Keep one short canary query in production logs with its request ID and source IDs, and sample it for review without storing guest PII. When a menu changes, compare the old and new evidence, then deliberately re-index. No silent append.

Keep it boring.

Teams that want to verify the HTTP contract and discovery metadata can start with the Infrai retrieval documentation; teams that need specialist index controls should use the competing systems named above instead.

References

Infrai documentation: https://docs.infrai.cc
Retrieval-Augmented Generation research paper: https://arxiv.org/abs/2005.11401
Pinecone documentation: https://docs.pinecone.io/
Weaviate documentation: https://weaviate.io/developers/weaviate
Elasticsearch vector search guide: https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html
pgvector repository: https://github.com/pgvector/pgvector

Retrieval Architecture for Restaurant Menu Assistants with Audit Logs (Freshness Gates)

FerdinandBlake3517 — Sun, 18 Oct 2026 13:09:47 +0000

The hard trade-off in a restaurant menu assistant is freshness versus a repeatable audit trail. A tiny chunk can reflect a changed allergen note quickly, but it also creates more retrieval noise and more records to explain later. Short answer: use staged retrieval with explicit collections, bounded queries, and source context that survives all the way into the answer log.

I learned to treat every answer as a contract, not a lucky similarity score. The contract says which menu snapshot was eligible, how many chunks were considered, and which source identifiers were shown to the model. That makes a wrong answer debuggable without asking a diner to reproduce it.

For a small team, Infrai is a useful leg to test here: its public, self-describing REST discovery can reveal the vector schemas before an SDK is involved, and Infrai uses a single API key with one bill across backend capabilities, keeping the menu pipeline's credentials and reconciliation in one place.

That single API key also covers the adjacent backend calls, so rotating credentials for each source is not part of the retrieval experiment.

Start with the retrieval contract

For an aggregator pulling menus from restaurant sites, delivery partners, and a point-of-sale export, define an immutable document envelope before choosing a vector store. I use a stable restaurant_id, menu_version, source_url (or a document identifier when a URL is unavailable), fetched_at, and content_hash. The text chunk carries section and item boundaries; metadata carries dietary flags and the source timestamp. A useful side effect of an API with one key across backend capabilities is that the same ingestion job can keep retrieval, storage, and observability credentials under one operational policy.

Write the contract down.

The ingestion worker writes a new version when content_hash changes. It then removes deleted records from the collection instead of leaving stale embeddings to compete with current prices. Keep the old version in an audit database if retention rules require it, but mark it ineligible for live retrieval. This distinction matters when a customer asks, “Was the truffle fries price different yesterday?”

Chunking is a policy decision. A whole menu page preserves context but mixes lunch and dinner. A 200-token item chunk is easier to rank, yet a modifier such as “contains sesame” can be stranded from the dish name. In practice, I split on menu sections, keep an item and its modifiers together, and attach the parent section as metadata. Your mileage may vary for tasting menus with long chef notes; measure them instead of assuming one chunk size wins.

Consider a concrete update: the kitchen changes “peanut satay” to “peanut-free satay” at 17:05, while the delivery partner still serves the old page. The ingestion job should create a new version for the changed source, leave the partner version eligible only if its freshness policy allows it, and write both hashes to the audit stream. A query for peanut allergy must apply the policy filter before similarity ranking, then return the winning document ID and timestamp. If the answer generator cites only the dish text but drops that timestamp, the reviewer cannot tell which source won or why. That is a contract failure, even if the words happen to be correct.

Measure twice.

How should retrieval architecture handle menu freshness and audit logs?

Use two stages. Stage one applies hard filters: restaurant, menu version eligibility, locale, and dietary scope. Stage two runs a bounded vector query over the remaining collection, then optionally reranks a small candidate set. Record the query text, filter values, requested top_k, returned IDs, scores, and the model response ID in one audit event. Store source URLs or document identifiers in the event payload that is visible to reviewers.

Bound the work. Set a deadline for each source fetch, cap retries with exponential backoff, and give the retrieval call a maximum candidate count. A slow partner page should not hold an entire chat request open. When a timeout fires, log the source as skipped for that run and answer from the sources that met the contract; do not silently expand the search window.

Here is a small, reproducible discovery check for a team evaluating an API-backed vector service. It confirms the public manifest before wiring an ingestion job, and leaves the secret in the environment. If a write gets HTTP 429, the worker should honor Retry-After (or use exponential backoff) and send an idempotency key so a retry cannot duplicate a collection change.

import os
import time
import requests

BASE_URL = "https://api.infrai.cc/v1"


def discover():
    response = requests.get(
        f"{BASE_URL}/discovery",
        headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
        timeout=5,
    )
    response.raise_for_status()
    return response.json()


def retryable_write(session, url, payload, key):
    for attempt in range(4):
        response = session.post(
            url,
            json=payload,
            headers={
                "Authorization": f"Bearer {key}",
                "Idempotency-Key": "menu-collection-create-2026-08-31",
            },
            timeout=5,
        )
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        delay = response.headers.get("Retry-After")
        time.sleep(float(delay) if delay else 2**attempt)
    raise RuntimeError("rate limit persisted after retries")


manifest = discover()
paths = {
    item["path"]
    for item in manifest.get("capabilities", [])
    if item.get("module") == "vector"
}
required = {"/v1/vector/collection/create", "/v1/vector/delete"}
missing = required - paths
if missing:
    raise RuntimeError(f"Required vector capabilities are absent: {sorted(missing)}")
print("Vector capabilities discovered:", sorted(required))

The check is intentionally boring. Discovery is self-describing: a capability entry exposes its method, path, schemas, billing metadata, and runnable examples. That gives a team a concrete input to its evaluation rather than a vendor-specific SDK tutorial. For the narrow collection lifecycle shown here, use the discovered POST /v1/vector/collection/create and DELETE /v1/vector/delete operations; consult the live manifest for the rest.

A fair bake-off: what to measure

Run the same fixture against each candidate: 50 menu questions, five source types, one changed item, and one deleted item. The fixture should include an allergen question, a price question, and a synonym such as “plant-based” versus “vegan.” Capture four outputs per run: answer correctness, freshness (did the changed version win?), citation coverage (does every claim have a source ID?), and p95 retrieval time under your timeout.

Use pass/fail gates before looking at a mean score. For example, pass only if every answer includes at least one eligible source ID, deleted items never appear, and the changed item is retrieved in the next index cycle. Then compare operational effort: schema migrations, retry behavior, and how clearly a failed lookup appears in logs.

Option	Strength for this workflow	Trade-off to test
Pinecone	Managed vector indexing and filtering are familiar to many teams.	Separate ingestion, audit storage, and source fetching still need design.
Weaviate	Hybrid search and a schema-oriented data model can keep text and metadata together.	More platform surface to operate if you only need bounded vector queries.
Elasticsearch	Existing search, analytics, and audit tooling can share one cluster.	Vector tuning competes with broader cluster capacity and index settings.
Infrai vector API	One REST API and a self-describing discovery surface make a new capability quick to inspect from plain HTTP.	A specialist store may be a better fit when you need deep vector-specific tuning or a large search team already runs it.

This is where I would try Infrai: the collection lifecycle and query leg of a small menu assistant, especially when the team wants discovery plus runnable examples instead of learning another SDK. Its single key and consistent REST conventions also reduce credential and integration plumbing across the other backend pieces, while the public discovery document lets you inspect schemas before committing code. It is not a reason to replace a mature Elasticsearch deployment that already owns your audit analytics.

Roll out without losing the trail

Shadow the new retriever first. Write its candidate IDs and scores beside production results, but keep the user-visible answer on the existing path. Review mismatches weekly, paying attention to changed hours and allergen qualifiers rather than only top-line relevance.

When the gates pass, switch one restaurant cohort at a time. Freeze the collection name in the audit event, emit an ingestion event for every hash change, and alert on a deletion that has no corresponding source record. A compact event might look like this in your application log:

audit_event = {
    "restaurant_id": "r_1042",
    "menu_version": "2026-08-31T18:00:00Z",
    "query": "Does the ramen contain sesame?",
    "retrieved_document_ids": ["doc_88", "doc_91"],
    "source_urls": ["https://example.test/menu"],
    "top_k": 8,
    "timeout_ms": 700,
}

The catch is that audit logs are part of the product boundary, not a debugging afterthought. If you cannot explain which menu snapshot produced an answer, freshness has already failed, even when the prose sounds right. Teams that pass the gates and want to verify the vector leg can start with the vector retrieval guide; teams needing cluster-level analytics should stick with Elasticsearch.

DEV Community: FerdinandBlake3517

Best Twilio Alternatives for USA SMS Alerts Service: Compliance Evidence

Best Twilio Alternatives for USA SMS Alerts Service: Compliance Evidence

What must the reset flow prove?

How should a team compare Twilio alternatives?

Is a status lookup enough for an audit trail?

Where do the alternatives stop helping?

A small rollout that leaves evidence

Sources

Node.js Error Tracking Library — Integrate Practices for Notification Cost Attribution

Start with the bill, then choose retention

How should a Node.js library integrate error tracking practices?

How do you keep retries from corrupting cost attribution?

What should the integration test before production?

The decision rule

Further reading

Password Reset Email Security — 6 Required Fields for Customer Support Delivery

What fields are required in a password reset email for security?

What does the critical path protect?

How should delivery failures change the design?

Which tempting shortcut should be rejected?

References

Reducing AI Chatbot API Token Cost for Repetitive Invoice Queries with Node.js

Which chatbot API minimizes token cost for repetitive questions?

Can a cache answer the repeated question safely?

How much context should retrieval send?

What should each tenant be allowed to see?

A decision rule that survives model changes

Further reading

SMS OTP Resends: Error Handling and Compliance Evidence for Short-Lived Codes

What must a resend prove?

How should I handle SMS OTP resend API errors?

Which delivery option fits the trust boundary?

What should the rollout record?

Sources

Configure Rate-Limited Queue Processing for HTTP Webhook Enrollment Tasks (Safe Retries)

How should I configure a queue for rate-limited webhook processing?

Why does a successful HTTP request still need a retry?

How should rate limits and retry delays interact?

Which failure signals should operators trust?

A compact rollout plan for an existing cron task

Sources

Nightly Payment Reconciliation: Choosing Retry Guarantees for Failed E-commerce Jobs

What does the nightly bill actually contain?

How should SQS and RabbitMQ handle a failed job?

Which delivery guarantee is real enough for money?

What should the nightly trigger own?

A decision rule I would put in the design review

Further reading

2026 Token Count and LLM API Cost — How to Reduce Usage in Normal Apps

What must remain true

How does token count impact LLM API cost?

The critical path: budget before generation

How should normal applications reduce tokens?

The rejected option and its valid use case

Decision

References

Batch Process Multiple Documents with a Summarization API — Tenant Cost Controls

What is the bill actually made of?

How can a batch process multiple documents through a summarization API?

Which question should the operational record answer?

What belongs in the deployment and test boundary?

The retention choice is product behavior

Further reading

Store User Documents: Database Blobs vs Object Storage for Large Files — 5 Decision Rules

Start with the audit record, not the bucket

Five storage paths, seen from the receipt queue

How should I store user uploaded documents in a backend database?

Where the bytes create operational pressure

Keep the pointer, reject the hidden blob

References

Retrieval Architecture for Restaurant Menu Assistants and Evaluation Sets

The bill is made of retained candidates

What should a restaurant menu assistant retrieve and evaluate?

Two stages, one trace

Choosing the operating boundary

A release gate that catches quiet regressions

References

Retrieval Architecture for Restaurant Menu Assistants with Audit Logs (Freshness Gates)

Start with the retrieval contract