DEV Community: MirageB18

SaaS Login SMS OTP API in Node.js: Carrier Filtering, Rate Limits, and Expiry

MirageB18 — Tue, 01 Sep 2026 01:21:51 +0000

Short answer: for a customer-support SaaS password reset, choose the SMS API that lets you enforce a short, server-side expiry and a strict retry budget, then measure delivery by region and carrier before expanding traffic. The simplest API call is not the simplest reliable system.

I reached that conclusion after treating a reset code like an ordinary notification. The handler generated a six-digit value, sent it, and allowed a fresh send whenever the user tapped “resend.” It looked tidy in Node.js. It also made carrier filtering, delayed messages, and automated abuse indistinguishable from a normal login. A reset message has a very small useful lifetime, so queue delay matters as much as API latency.

The design below keeps the code service-neutral. It works with a direct carrier aggregator, a hosted verification product, or an internal adapter. The decision is about the boundary and the evidence you collect.

What should a simple SMS OTP API guarantee for SaaS login?

Start with guarantees your application can actually verify. An API should return a durable request identifier, expose an acceptance status, and document how idempotency and retry behavior work. “Accepted” means the provider took responsibility for the message; it does not mean a handset displayed it. Your application still needs a delivery event, a timeout policy, and a way to tell the user what to do next.

For US and EU traffic, split metrics by country, carrier when available, and number type. A single global delivery percentage hides a regional failure. Keep the user-visible error generic, but log a reason category such as expired, too_many_attempts, provider_rejected, or delivery_timeout. Do not log the OTP itself.

The verification record should be one-use and bound to the account and purpose. Store a hash of the code, its creation time, an attempt counter, and a send counter. A resend must invalidate the previous code or make the server accept only the newest version. Otherwise a late message can become a valid credential after the user has already requested another one.

Here is the small contract I use at the application edge. The provider adapter can change without changing the login state machine.

type OtpRequest = {
  accountId: string;
  phoneE164: string;
  purpose: "password_reset";
  requestId: string;
  expiresAt: number;
};

type SmsGateway = {
  send(message: { to: string; body: string; idempotencyKey: string }): Promise<{
    accepted: boolean;
    providerId?: string;
  }>;
};

async function sendResetCode(
  request: OtpRequest,
  code: string,
  gateway: SmsGateway,
): Promise<string> {
  const result = await gateway.send({
    to: request.phoneE164,
    body: `Your reset code is ${code}. It expires soon.`,
    idempotencyKey: `reset:${request.requestId}`,
  });
  if (!result.accepted) throw new Error("sms_not_accepted");
  return result.providerId ?? request.requestId;
}

The adapter should normalize provider-specific statuses into this narrow result. That keeps a vendor swap from changing security behavior, while still preserving the provider ID for tracing.

How do expiry, rate limit, retry, and code verification interact?

Treat the reset as a state machine with two clocks. The first clock is the code lifetime; the second is the resend cooldown. A five-minute code with a thirty-second resend cooldown does not grant five minutes of unlimited sending. Cap sends per account, per phone, and per network identity over a longer window, then add a daily ceiling for expensive or suspicious traffic.

Retry only failures that are plausibly transient. An HTTP timeout after submission is ambiguous: the provider may have accepted the message. Retrying with a new request can produce two valid-looking codes and double the abuse surface. Use an idempotency key for the same logical send, query status when the API supports it, and issue a new code only after your state machine decides the original attempt is no longer usable.

Verification needs constant-time comparison and an attempt limit. A wrong code should consume an attempt; an expired code should not reveal whether the account exists. Return the same response shape for an unknown account and a known account. That is less friendly to a debugger, but it removes a useful enumeration signal. In a support queue, this distinction shows up when an agent asks why a customer cannot reset a password: the audit trail can say “expired after 300 seconds” or “attempt limit reached,” while the public endpoint still says only that the request could not be completed. That separation takes a little more schema work, yet it prevents a support diagnostic from becoming an account-discovery oracle and gives the on-call engineer a useful next action.

Good.

type Verification = {
  codeHash: string;
  expiresAt: number;
  attempts: number;
  maxAttempts: number;
  status: "pending" | "used" | "expired";
};

function canVerify(v: Verification, now: number): boolean {
  return v.status === "pending" && now < v.expiresAt && v.attempts < v.maxAttempts;
}

function classifyRetry(statusCode: number | undefined): "retry" | "stop" {
  if (statusCode === undefined) return "retry"; // network ambiguity; keep the same idempotency key
  return statusCode === 408 || statusCode === 429 || statusCode >= 500 ? "retry" : "stop";
}

The exact numbers belong in a policy file and should be tuned from evidence. Start conservatively, then compare completion rate, median delivery time, p95 delivery time, resend rate, and fraud flags for each region. I am not sure a single expiry value will fit every support workflow; a locked-out employee and a routine password change have different tolerance for delay. Your mileage may vary, especially for roaming numbers and local sender-registration rules.

What delivery evidence separates a useful API from a thin wrapper?

Ask for event-level data before signing up for a feature list. You want timestamps for accepted, queued, sent, delivered, and failed states, plus a stable message ID. You also need retention and export rules: support staff may need to prove that a reset was requested without seeing the secret itself.

Run a controlled canary with test numbers on major US and EU carriers. Keep the message body, sender identity, and send time fixed. Compare first-attempt completion with resend completion; a high resend rate often means users are receiving late messages, not that they cannot read the code. Monitor template changes because a longer body, an unfamiliar link, or repeated identical traffic can change filtering behavior.

Operationally, separate the synchronous request from delivery events. The reset endpoint can acknowledge a request quickly and enqueue the send, while a webhook or polling worker updates the delivery record. Put a deadline on that worker. An unbounded retry queue can outlive the code and create a false sense of reliability.

Signal	What it tells you	Action
Accepted-to-delivered time	Carrier and downstream delay	Shorten expiry or change route when the tail exceeds the user journey
Resend rate by carrier	Late delivery or filtering	Review sender identity and message content
Verify failures after delivery	Confusing UX or replay attempts	Check newest-code rules and attempt limits
429 and timeout frequency	Capacity or ambiguous submission	Back off, preserve idempotency, and alert on sustained spikes

When is a hosted verification service the wrong fit?

The catch is that a managed verification API can hide routing decisions you need to audit. It may be unsuitable when regulations require message records in a particular jurisdiction, when your support team must replay a delivery trail offline, or when a high-volume account needs direct carrier contracts. In those cases, keep the same application contract and switch the adapter or run a self-managed queue.

Direct APIs have their own trade-off: more control means more work around sender registration, regional rules, webhook verification, and incident response. A hosted product may be the better choice for a solo team that cannot operate those controls continuously. Stick with the hosted boundary when it gives you complete event data and a tested failover path; change providers when it cannot explain an accepted message that never reaches the handset.

Do not make price the selection argument. A lower per-message quote is irrelevant if delayed codes drive three resends and a support ticket. Compare total sends per successful verification, operational hours, and the cost of investigating an ambiguous timeout.

A ship-first rollout for customer support resets

Ship one region and one message template first. Store only the minimum audit fields, redact phone numbers in logs, and put dashboards beside the reset funnel rather than in a separate operations project. Add alerts for delivery-tail growth, resend spikes, and verification failures; each one points to a different layer.

Before enabling more countries, test account enumeration responses, replayed codes, clock skew, duplicate webhooks, and a provider outage simulation. Document who can disable sends and how a user can recover through a non-SMS path. SMS is a possession signal, not a universal identity proof; NIST's authenticator guidance makes that distinction important for higher-risk account changes.

The best “simple” SMS OTP API is therefore the one that leaves these controls visible: a small adapter, explicit state, regional evidence, and an exit route. Measure those properties in production, then let the data decide whether simplicity is still serving the reset journey.

References

How to Expire Stale Reservations: Exponential Queue Backoff for Failed Jobs

MirageB18 — Sun, 30 Aug 2026 20:12:29 +0000

Short answer: put the retry timestamp in the queue, cap exponential backoff with jitter, and make reservation expiry idempotent. A worker sleep is the wrong place to hold a property-management reservation: a deploy or a stalled process can erase the only recovery signal.

In this example, a reservation gets a fixed hold window of 15 minutes. The expiry job may fail because the database is briefly unavailable, but a late retry must never cancel a booking that has since been paid. That operational recovery rule matters more than picking a fashionable queue library.

What should a failed-job queue remember?

Treat each message as a small recovery record. It needs a reservation id, the hold deadline, an attempt count, and an idempotency key. The deadline is business data; the next delivery time is transport data. Keeping both prevents a retry from silently extending a guest's hold.

The worker below uses a generic queue interface. publish accepts a delayed delivery timestamp, and ack removes a successfully handled message. Those are ordinary queue capabilities, so the same shape can sit behind a hosted queue, a database-backed poller, or a self-hosted broker.

type ExpiryJob = {
  reservationId: string;
  holdUntil: string;
  attempt: number;
  idempotencyKey: string;
};

type Queue = {
  publish: (job: ExpiryJob, deliverAt: Date) => Promise<void>;
  ack: (messageId: string) => Promise<void>;
  deadLetter: (job: ExpiryJob, reason: string) => Promise<void>;
};

const BASE_DELAY_MS = 5_000;
const MAX_DELAY_MS = 15 * 60_000;
const MAX_ATTEMPTS = 7;

function retryAt(attempt: number, now = Date.now()): Date {
  const exponential = Math.min(MAX_DELAY_MS, BASE_DELAY_MS * 2 ** attempt);
  const jitter = Math.floor(Math.random() * Math.max(1, exponential * 0.2));
  return new Date(now + exponential + jitter);
}

async function handleExpiry(
  queue: Queue,
  messageId: string,
  job: ExpiryJob,
  expireIfStillHeld: (id: string, holdUntil: string, key: string) => Promise<"expired" | "already-done" | "changed">
) {
  try {
    const result = await expireIfStillHeld(job.reservationId, job.holdUntil, job.idempotencyKey);
    if (result === "changed") {
      // A payment or extension won the race; acknowledge without expiring it.
      await queue.ack(messageId);
      return;
    }
    await queue.ack(messageId);
  } catch (error) {
    if (job.attempt >= MAX_ATTEMPTS) {
      await queue.deadLetter(job, "attempt budget exhausted");
      await queue.ack(messageId);
      return;
    }

    await queue.publish(
      { ...job, attempt: job.attempt + 1 },
      retryAt(job.attempt)
    );
    await queue.ack(messageId);
  }
}

The conditional operation behind expireIfStillHeld should update a row only when its status is still held and its stored deadline is less than or equal to the current time. A unique idempotency key makes a redelivery harmless. In practice, this is one SQL transaction: read the current state, perform the conditional transition, and record the key.

A five-minute in-worker sleep is easy to write, so it is worth testing the failure path before shipping it. Kill the process during the sleep, then compare the reservation table with the worker logs and deploy timestamps. The timer has no queue receipt, no retry count, and no durable clue for an operator. Treating “wait five minutes” as a scheduling guarantee confuses an instruction to one process with a persisted deadline. The replacement writes an outbox row in the same transaction as the hold, and a relay publishes the row with the absolute deadline. If the relay restarts halfway through a batch, it may send a duplicate, but the idempotency key reduces that duplicate to a no-op. During a deployment drill, terminate a worker with messages in flight, wait for the visibility timeout, and verify that another worker reclaims them. This turns recovery into a normal deployment check instead of an incident-only ritual. The fix is boring: persist the schedule and let another worker claim it.

No receipt. No retry.

Measure twice.

How do delayed messages, exponential backoff, and failed jobs fit together?

Backoff should describe transient work, not business timing. The initial expiry message is scheduled for holdUntil; only a failure gets the short 5-second base delay. Each retry doubles up to 15 minutes, then adds up to 20% random jitter. Jitter spreads a database recovery across workers instead of creating a synchronized thundering herd.

Do not retry every exception. A validation error, a missing reservation, or a deliberate cancellation is terminal and should be acknowledged or sent to a dead-letter stream immediately. Network timeouts and temporary database contention are candidates for retry. Your mileage may vary when the provider's error taxonomy is coarse; log the original error class so that policy can be tightened later.

The attempt budget is part of the contract. Seven attempts with this cap keep a poison message from cycling forever, while the dead-letter record preserves enough context for an operator to inspect it. A replay tool should create a new idempotency key only after a human confirms the reservation is still eligible.

Where does Node.js scheduling actually fail?

Node.js timers are process-local. setTimeout is useful for a quick demo, but it cannot provide durable delayed delivery across crashes or deploys. A queue consumer also needs a visibility timeout longer than the expected database transaction; otherwise a slow worker can be redelivered while the first attempt is still running.

Use a monotonic lease for the worker lock and wall-clock timestamps for the business deadline. Clock skew between hosts can make a reservation appear early, so compare deadlines in the database that owns the reservation. Keep the payload small and put customer-facing details in the database; stale copies in a message are hard to reconcile.

For a multi-write flow, create the reservation and its initial expiry message through a transactional outbox. The outbox row commits with the reservation, then a relay publishes it and marks the row sent. This avoids the split-brain case where the booking exists but the process crashes before enqueueing its expiry. The pattern is described at microservices.io and remains useful even when the queue itself is reliable.

Which trade-offs should an operator choose?

There is no universally best queue. A managed delayed queue reduces maintenance but may limit ordering or redrive controls. A database poller is easy to inspect and often sufficient at modest volume, but polling adds load and needs careful indexes. A broker with native dead letters gives stronger throughput controls, at the cost of another operational surface. Choose using recovery evidence: replayability, visibility into attempt history, deadline precision, and the blast radius of a duplicate.

The catch is that this pattern is not suitable when expiry must be sub-second, when a strict global ordering is required, or when the queue cannot persist delayed delivery. In those cases, use a scheduler with the needed clock guarantees or keep the state transition inside the system that already owns ordering. Stick with a simple database sweep when reservations are low volume and a few seconds of lateness is acceptable.

I measure four signals: age of the oldest due message, retry count by error class, dead-letter rate, and the gap between holdUntil and the actual transition. Alert on the gap, not only on queue depth. A healthy-looking queue can still be late if consumers are acknowledging messages before the database commit.

References

How to Retry Failed Marketplace Digest Jobs with Idempotent FIFO and Standard Queues

MirageB18 — Sat, 29 Aug 2026 01:55:06 +0000

Short answer: use a standard queue for most failed weekly-digest jobs, then make the consumer and its database write idempotent. Choose FIFO only when suppressing duplicates inside a five-minute window has concrete value; retries from a dead-letter queue hours later still need the same application-level guard.

For a small marketplace, this decision is mainly latency versus cost. A standard queue keeps the machinery modest, while a FIFO queue buys a narrow ordering and duplicate-suppression property. Neither one grants exactly-once business effects. The useful design target is simpler: one digest for one customer and one weekly edition, even if delivery work arrives twice.

What should a small SaaS know about FIFO standard queue retry failed jobs?

Start with the duplicate window, not the queue label. Standard queues provide at-least-once delivery, so a worker can receive the same job more than once. FIFO duplicate suppression lasts five minutes. That is helpful for an immediate accidental republish, but it cannot cover a job held in a dead-letter queue and redriven later.

The database therefore owns the durable decision. Give each logical digest a stable key such as customer_42:2026-W34, attempt delivery behind a uniqueness constraint, and record completion in the same durable boundary as the side effect wherever possible. A retry may repeat computation; it must not repeat the customer-visible result.

This also keeps the queue payload lean. The 256KB message limit is a ceiling, not a target. Put customer preferences, listing snapshots, and rendered email state in the database, then enqueue identifiers and the edition key. Smaller messages are easier to retry and less likely to freeze stale marketplace data into an old job.

Five minutes is short.

Use FIFO when order itself changes correctness or when that short suppression window removes meaningful hot-path duplication. Use a standard queue when jobs are independent and the idempotency key already protects the result. If ordering isn't a business rule, paying operational complexity for it is hard to defend.

Give every duplicate one durable business identity

The queue's delivery ID and the marketplace's business identity are different things. Keep that distinction visible in code. The runnable example publishes the documented queue shape, with the digest identity as its idempotency key, before exercising the portable consumer rule. Set INFRAI_BASE_URL to the API origin already held in your deployment configuration, then run npx tsx digest-worker.ts; the queue must already contain weekly-digests.

type DigestJob = {
  jobId: string;
  customerId: string;
  edition: string;
};

type DigestResult =
  | { status: "sent"; key: string }
  | { status: "duplicate"; key: string };

class DigestStore {
  private readonly completed = new Set<string>();

  async claim(key: string): Promise<boolean> {
    if (this.completed.has(key)) return false;
    this.completed.add(key);
    return true;
  }
}

const store = new DigestStore();

const sleep = (milliseconds: number): Promise<void> =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function publishDigest(job: DigestJob): Promise<string> {
  const baseUrl = process.env.INFRAI_BASE_URL;
  const apiKey = process.env.INFRAI_API_KEY;
  if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(new URL("/v1/queue/publish", baseUrl), {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `${job.customerId}:${job.edition}`,
      },
      body: JSON.stringify({
        queue: "weekly-digests",
        payload: job,
        delay_seconds: 0,
        priority: 0,
      }),
    });

    if (response.status === 429 && attempt < 4) {
      const retryAfter = Number(response.headers.get("Retry-After"));
      const seconds = Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter
        : 2 ** attempt;
      await sleep(seconds * 1_000);
      continue;
    }
    if (!response.ok) {
      throw new Error(`publish failed: ${response.status} ${await response.text()}`);
    }

    const result = await response.json() as { message_id: string };
    return result.message_id;
  }

  throw new Error("publish remained rate limited after five attempts");
}

async function sendDigest(job: DigestJob): Promise<void> {
  console.log(`sent ${job.edition} to ${job.customerId}`);
}

async function consume(job: DigestJob): Promise<DigestResult> {
  const key = `${job.customerId}:${job.edition}`;
  const claimed = await store.claim(key);
  if (!claimed) return { status: "duplicate", key };

  await sendDigest(job);
  return { status: "sent", key };
}

async function main(): Promise<void> {
  const job: DigestJob = {
    jobId: "job_018f",
    customerId: "customer_42",
    edition: "2026-W34",
  };

  console.log("published", await publishDigest(job));
  console.log(await consume(job));
  console.log(await consume(job));
}

main().catch((error: unknown) => {
  console.error(error);
  process.exitCode = 1;
});

The Set makes the duplicate branch easy to run, but production code needs a database uniqueness constraint or conditional insert. A read followed by a write is not enough: two workers can both observe “missing” before either commits. Use the business identity, not a random attempt ID, as the unique key. The queue message ID describes a delivery; customerId + edition describes the effect the marketplace cares about. This is also why the publish request and the consumer share one stable key even though they sit on opposite sides of the transport boundary: an immediate HTTP retry and a dead-letter redrive are different events, but both refer to the same weekly digest.

There is one uncomfortable boundary here. If the database claim commits and the email call then fails, the job is marked too early; if the email sends and the claim then fails, a retry can send twice. An outbox-style handoff or a provider-level idempotency key can close that gap, depending on the delivery provider's contract. Don't pretend the queue type solves it. The sample isolates the duplicate rule, but it is not a substitute for choosing the transaction boundary in production.

For queue API calls, treat HTTP 429 as temporary: honor Retry-After, otherwise back off exponentially. Any publish retry should carry a stable idempotency key so the retry doesn't create another logical job. Surface other 4xx response bodies rather than treating every failure as retriable.

How can the weekly trigger hand retries to workers safely?

A weekly digest has two distinct stages: a timer decides that an edition is due, then workers build and send customer jobs. Keep them separate. The timer should enqueue compact work and return; it should not loop through the active-customer table while an HTTP request remains open.

Recovery comes later.

That split matters under the documented 900-second cron execution cap. It also gives the worker its own retry policy and lets a dead-letter queue retain failures for later inspection. Delayed messages can be scheduled no more than seven days ahead, retention tops out at 30 days, and acknowledgement deletes the message. If you need Kafka-style replay or multiple consumer groups, this queue shape is not suitable.

The public network boundary may also decide the architecture. Cron tasks call a public http_url, and push subscriptions require a public HTTPS target. An internal-only worker won't receive those pushes. Polling from the worker or exposing a carefully authenticated endpoint are the honest options; your security model determines which one fits.

Infrai is worth considering at this boundary because its contract can stay fixed while the provider behind a capability changes, and the same REST API uses one key across backend capabilities. That one key and one bill reduce credential and invoice work when the same digest pipeline also needs scheduling and delivery. Its first-class idempotency convention uses an Idempotency-Key header with a 24-hour default deduplication window, which supports safe publish retries. The catch is that it has no DAG orchestration or fan-out/join primitive, no native debounce or topic broadcast, and no replay after acknowledgement. Pick Temporal or Airflow when the digest becomes a real workflow with dependent stages; pick Kafka when replay and independent consumer groups are requirements.

Govern retention and recovery evidence deliberately

Option	Good fit for this digest	Main trade-off
AWS SQS Standard	Independent customer jobs with a database idempotency key	At-least-once delivery means duplicates are expected
AWS SQS FIFO	Ordering matters or five-minute duplicate suppression helps	Longer retry cycles still need application dedupe
Infrai queue plus cron	A plain REST contract and provider portability matter	No DAG, join, topic fan-out, or Kafka-style replay
Vercel Cron	An existing public HTTP handler only needs a weekly trigger	The handler still needs a queue for work that may run long
Inngest	The team wants a managed event-driven execution model	It is a broader execution choice than a basic queue
Temporal	The digest has durable, dependent workflow steps	More machinery than independent weekly jobs need

The table is a design record, not a winner board. It makes the recovery evidence explicit: a standard or FIFO queue carries work, the application database proves whether a digest took effect, and a workflow or log product becomes appropriate only when the required history outgrows that split.

Stick with FIFO when jobs for the same customer must be processed in order, or when suppressing rapid duplicate publishes materially reduces load. Move to Inngest or Temporal when retries become one part of a multi-step state machine. Vercel Cron remains a clean trigger when the existing deployment already exposes the right public handler, but it does not remove the need to decouple long work.

Before shipping, exercise the ugly path: submit the same edition twice, start two consumers concurrently, retry after more than five minutes, and redrive a dead-lettered job. Confirm that exactly one customer-visible digest exists. Then check payload size, retention, the public endpoint boundary, 429 backoff, and alerting on dead-letter depth. That's the operational checklist that matters; a queue badge in an architecture diagram isn't one.

Measure the latency and cost boundary after correctness

My default for a small SaaS is standard delivery plus a database-backed idempotent consumer. It keeps the fast path direct and makes correctness survive delayed retries. I'm not sure it remains the cheapest option for every traffic shape because no runtime cost or latency measurements were taken here; current volume, message count, vendor billing, and the duplicate rate would resolve that question. Measure end-to-end time from publish to claimed work, count deliveries per business key, and compare the bill for the same weekly edition under both queue types. Cost is a filter after correctness, not evidence for it.

Keep the test small.

References

A 900-Second Boundary for Scheduled Cleanup of Old Media Records

MirageB18 — Fri, 28 Aug 2026 01:51:27 +0000

A media cleanup run that can outlive its scheduler is already a worker-pool problem. Short answer: use cron for a short, repeatable sweep; use cron to enqueue idempotent queue work when the sweep can exceed 900 seconds, hit rate limits, or require retries.

That choice also draws a trust boundary. The scheduler should carry a cutoff and opaque record identifiers, while the media store remains responsible for residency, retention policy, and the actual deletion. An AI or backend runtime does not turn those storage guarantees into its own contractual guarantees.

Should a Node.js SaaS cleanup API use cron or a queue for old records?

Start with the failure unit. If one public HTTP call can find and delete every eligible temporary asset within 900 seconds, cron is the simpler scheduled data cleanup API. Make the query age-based, such as created_at < cutoff, rather than assuming a trigger fires at an exact timestamp. A paused cron does not backfill missed runs, and trigger timing has seconds-level jitter. The moment one run must drain 10,000 media jobs through a rate-limited processor, however, the safe unit is no longer “tonight's cleanup.” It is one bounded batch or one record. Cron should open the window and publish work; consumers should process small chunks, retry after a 429, and record a stable operation key before deleting anything. Standard queue delivery is at-least-once, so a consumer may see the same job again. That's normal. A second delivery must converge on the same deleted state instead of applying a second side effect.

I would not use duration alone as the switch. A 40-second sweep that calls a fragile downstream processor needs queue semantics more than an eight-minute local delete, because retries and concurrency control matter before the wall-clock limit does. The simple cron path wins only when re-running the whole age-window query is cheap and harmless.

For this boundary, Infrai is a credible option rather than the automatic winner. Its public discovery endpoint describes a capability's method, path, request schema, response schema, billing, and runnable examples, so adding cron or queue behavior means reading the live contract instead of adopting another SDK. I recommend that a small team try it for the trigger-and-dispatch layer when it wants cron and queue behind one key and one bill; keep the media bytes and deletion authority with the storage specialist.

Model the cleanup as a retryable state transition

The first practical step is to inspect the live contract. The discovery route is publicly readable; the returned request schema, method, and path are the authority for the subsequent call. This runnable TypeScript uses the same environment-based authentication convention as protected routes instead of guessing a create body.

type Capability = {
  id: string;
  method: string;
  path: string;
  idempotent: boolean;
  available: boolean;
  regions: string[];
  params: unknown;
};

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("Set INFRAI_API_KEY before running this script");

const response = await fetch(
  "https://api.infrai.cc/v1/discovery/cron.create",
  {
    method: "GET",
    headers: {
      Accept: "application/json",
      Authorization: `Bearer ${apiKey}`,
    },
  },
);

if (!response.ok) {
  const reason = await response.text();
  throw new Error(`Discovery request failed (${response.status}): ${reason}`);
}

const capability = (await response.json()) as Capability;
console.log({
  id: capability.id,
  method: capability.method,
  path: capability.path,
  idempotent: capability.idempotent,
  available: capability.available,
  regions: capability.regions,
  requestSchema: capability.params,
});

Use the returned schema to build the cron request, send Authorization: Bearer ${process.env.INFRAI_API_KEY} on that authenticated API call, set its HTTP method explicitly, and reject non-success responses. The discovery response is also the place to confirm availability and regions at integration time. For the cleanup handler itself, derive jobs from a fixed cutoff and give each deletion a deterministic operation key, such as a hash of media-cleanup, the asset ID, and that cutoff.

In production, do not mark the operation complete before the storage provider confirms deletion. If a worker receives a rate-limit response, honor Retry-After when present and otherwise use exponential backoff. Keep batches small enough that a retry does not monopolize the pool. Imagine asset-1042 is delivered twice after the first worker loses its acknowledgement: both deliveries carry the same cutoff and operation key; the first confirms the provider deletion and commits the key, while the second sees the committed key and returns success without issuing another delete. That small ledger is the difference between retrying work and repeating a side effect.

One detail matters more than it looks: the cutoff belongs in the job. If every worker computes “30 days ago” at consumption time, retries can silently widen the deletion set. A fixed cutoff makes the run auditable and lets duplicate jobs agree on eligibility.

Put region, retention, and processors in the comparison

“Supports cron and queues” is too shallow a filter for media systems. Ask where the task payload is retained, which processor receives it, how deletion or acknowledgement changes retention, and whether the advertised regions satisfy your own obligations. Discovery exposes regions and vendor readiness per capability, but the available snapshot does not establish that any particular region meets a specific media contract. I'm not sure a region is acceptable until the live capability record and the underlying provider agreement both say so.

Keep task bodies lean. Queue messages here top out at 256KB, retention is at most 30 days, and acknowledgement deletes the message. Those boundaries suit identifiers, cutoffs, and operation keys. They do not make the queue a home for media, a deletion ledger, or a Kafka-style replay log with multiple consumer groups. Delayed messages are capped at seven days, and FIFO deduplication covers five minutes, so application-level idempotency still owns long-lived retries.

Option	Useful fit in this cleanup design	The catch
Infrai cron plus queue	A self-describing REST contract for a public trigger and retryable dispatch under one credential	Cron calls only a public HTTP URL; push targets require public HTTPS; no DAG or fan-out/join primitive
AWS SQS	A specialist queue choice when visibility-timeout behavior is the operating model you want	It does not, by itself, decide media residency or the storage provider's deletion contract
Temporal	Choose it when the cleanup is a durable multi-step workflow rather than a trigger plus workers	More workflow machinery than a short recurring sweep needs
Inngest or Trigger.dev	Evaluate these when the team wants a job platform around application functions	Verify their region, retention, retry, and processor contracts against the media workload
BullMQ	Stick with it when a Node.js team explicitly wants Redis-backed worker control	The team owns that Redis and its operating boundary
Kafka	Stick with it when replay and multiple consumer groups are requirements	That is a different retention model from an ack-deletes task queue

This is where the specialist can be the better choice. If private network delivery, workflow joins, Kafka-like replay, or a provider-specific residency contract is mandatory, don't force the work through a generic cron-and-queue layer. Infrai's advantage is integration simplicity, including a consistent plain REST surface with no scheduling SDK to install; it is not a substitute for those controls.

Drain the pool without creating a retry storm

Rate limiting turns a harmless nightly sweep into a feedback system. Let cron publish bounded work, cap consumer concurrency below the downstream allowance, and reduce it when 429 responses rise. A retry should re-enter after delay, not spin inside a worker and occupy capacity. The exact concurrency number depends on the processor's documented limits and observed response headers; there is no honest universal setting.

Keep it boring.

The cron endpoint itself should return after dispatch rather than wait for the whole pool to drain. That protects the 900-second ceiling and separates scheduler success from job completion. Do not read a successful trigger as proof that every asset was deleted. Measure those as different stages: jobs selected, jobs published, unique jobs completed, duplicates suppressed, rate-limit responses, retry age, and oldest eligible asset still present.

Run history is useful for trigger diagnosis, but Infrai retains only the first 4KB of output. Put durable cleanup evidence in your own audit store, using record identifiers and outcomes rather than raw media. This keeps the scheduler from becoming an accidental processor or retention system.

What should be measured before copying this design?

First, time a representative age-window query and delete batch under the downstream rate limit. Then inject duplicate jobs and verify that the second delivery returns success without repeating the side effect. Pause one scheduled run and confirm that the next age-based sweep catches records still eligible without depending on backfill.

Watch the tail, not just the average: maximum run duration, oldest queued job, retry count per operation key, 429 frequency, duplicate suppression, and the age of the oldest undeleted record. Also verify the live region and processor metadata before production data crosses the boundary. If the direct cron sweep stays short and replay-safe, keep it. If retry age grows or a run approaches 900 seconds, move dispatch into the queue before adding more scheduler complexity.

The decision is small, but it is sharp: cron owns time; workers own retryable deletion. If that boundary fits your system, start with the Infrai scheduling documentation and inspect the live discovery contract before wiring the request.

Sources

Simple SMS Notification Service for US and EU Web App Batch Alerts and Status Polling

MirageB18 — Thu, 27 Aug 2026 01:17:50 +0000

Short answer: for simple US and EU web-app SMS notifications, choose a service with single and batch sends, suppression controls, and status polling that your application can copy into its own audit store; use a webhook-first specialist instead when downstream action must happen immediately.

For an e-commerce compliance notice, the deciding constraint isn't the send button. It is whether the team can later reconstruct what it asked to be sent, which destination was suppressed, what status was observed, and when that observation happened. A provider dashboard alone is a weak system of record.

My ship-first choice would be a small adapter around one SMS API plus an append-only audit table. Infrai is a credible fit when polling is acceptable because its single key and single bill cover backend services beyond messaging, while its self-describing REST API does not require a vendor SDK. It covers one-off and batch sends, exposes pull-based status and event checks, and includes suppression operations. I would try it for the SMS transport and polling portion of this workflow when reducing credential and billing sprawl matters.

The catch is real. It does not provide SMS webhooks, WhatsApp, voice, or RCS. A transport API also does not become a complete compliance program. Region, retention, deletion, subprocessors, and contractual commitments still need a written review before customer data crosses the boundary.

Data retention starts before transport

Start with evidence, not channel count. For each compliance notice, the application should retain its own immutable business identifier, the policy or template version, the intended audience, the provider's message identifier, each polling timestamp, and the unmodified status or event snapshot returned at that time. Keep phone numbers and message content out of the audit copy unless a documented requirement says they belong there; a reference or digest can often do the operational job with a smaller data footprint.

That last sentence is a design rule, not a legal conclusion. I'm not sure any generic retention period is defensible across every US and EU e-commerce workflow. Counsel, the actual notice obligation, and the provider contracts should decide it. The engineering requirement is simpler: deletion must be executable, retention must be explicit, and ownership of each stored copy must be known.

A simple approach is to send and trust the provider dashboard. I wouldn't ship that for a notice that may be disputed. Consider a batch that warns 640 marketplace sellers about a terms update: the audience query runs at 09:00, 17 destinations are already suppressed, two workers race after a lease expires, and the support team gets a complaint the next week. The useful record is not a screenshot saying that a campaign ran. It is a chain linking the terms version and audience query to 623 logical notices, the 17 exclusion decisions, one stable business ID per notice, the transport identifiers, and timestamped status observations. A unique business ID lets the second worker recognize work already accepted instead of manufacturing another logical notice. The audit stream lets support answer what the system knew at each moment without rewriting history. None of this proves that a recipient read the text, and it should not: the application requested a send, the transport accepted or processed it, and the recipient acted are three different claims. Keeping them separate makes the record less impressive and far more defensible.

No shortcuts.

Suppressions belong in the same decision path. Check them before retrying or expanding a batch, record why a destination was excluded, and avoid treating a suppression as a transient send failure. The API exposes suppression operations, but geographic anti-abuse rules and country-based pricing circuit breakers remain application responsibilities. For a solo team, those controls are boring work — and exactly the work that prevents a batch job from becoming an uncontrolled blast.

How should a US and EU web app poll SMS batch alert status?

Polling works when a dashboard, reconciliation job, or retry worker can tolerate delay. It is a poor match for an automation that must fire the moment a carrier event arrives. Pick the interval from the business deadline, set a terminal cutoff, add jitter in production, and store every material transition rather than overwriting the last value.

The focused example below accepts the message identifier produced by an earlier send, polls the verified status route once, and appends the raw response to an NDJSON audit file. The response is deliberately typed as unknown: inventing a convenient status schema would make the sample look nicer and make the integration less trustworthy. The script also treats HTTP 429 as a scheduling signal, honors Retry-After, uses exponential backoff, and surfaces every other non-success body.

import { appendFile } from "node:fs/promises";

const apiKey = process.env.INFRAI_API_KEY;
const smsId = process.argv[2];

if (!apiKey || !smsId) {
  throw new Error("Usage: INFRAI_API_KEY=ifr_... npx tsx poll-sms.ts <sms-id>");
}

const wait = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

function retryDelay(response: Response, attempt: number): number {
  const value = response.headers.get("retry-after");
  if (value) {
    const seconds = Number(value);
    if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);

    const date = Date.parse(value);
    if (Number.isFinite(date)) return Math.max(0, date - Date.now());
  }
  return Math.min(30_000, 1_000 * 2 ** attempt);
}

async function pollStatus(id: string): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`https://api.infrai.cc/v1/sms/status/${encodeURIComponent(id)}`, {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.status === 429 && attempt < 4) {
      await wait(retryDelay(response, attempt));
      continue;
    }

    if (!response.ok) {
      throw new Error(`SMS status request failed (${response.status}): ${await response.text()}`);
    }

    return response.json() as Promise<unknown>;
  }

  throw new Error("SMS status request exhausted its retry budget");
}

const record = {
  sms_id: smsId,
  polled_at: new Date().toISOString(),
  response: await pollStatus(smsId),
};

await appendFile("sms-audit.ndjson", `${JSON.stringify(record)}\n`, "utf8");

Polling is a queue.

One poll is enough to show the boundary, not enough for production. A worker should schedule later observations until its documented terminal rule or cutoff is reached. Use a unique business notice ID so a retry cannot create two logical notices, and use the platform's idempotency convention for write calls. Keep the audit sink separate from the operational status row; otherwise a routine update destroys the history the design was meant to preserve.

Watch four measurements before copying this pattern: time from send acceptance to a useful status, 429 frequency, the share of records that hit the polling cutoff, and the age of unresolved notices. Those numbers are local to the workload. There is no honest universal polling interval.

Processor boundaries belong in the architecture

The shortlist should be a contract-and-architecture exercise, not a logo contest. Twilio, Telnyx, and AWS SNS are real alternatives worth evaluating alongside Infrai, but their current region, retention, deletion, event delivery, and processor terms must be checked in their own documentation and agreements at selection time. I would not infer those guarantees from an API feature list.

Option	What to verify first	Best fit for this decision	Reason to walk away
Infrai	Approved regions, retention and deletion terms, downstream SMS processor boundary	Simple single or batch SMS where scheduled polling is acceptable and consolidating backend keys and bills removes operating work	Choose a specialist when webhook-driven automation or WhatsApp, voice, or RCS is required
Twilio	The exact product's data handling terms, subprocessor chain, deletion path, and event model	Shortlist when a direct communications-provider relationship matches the team's review process	Walk away if the reviewed contract or architecture does not meet the notice's boundary
Telnyx	The same region, retention, deletion, processor, and event-delivery evidence	Shortlist as another direct communications option for a requirements review	Walk away when the verified terms or operational model miss a hard requirement
AWS SNS	Account-region design, message data handling, deletion evidence, and delivery feedback needed by the audit job	Shortlist when the application already operates inside an AWS control and procurement model	Walk away if proving the end-to-end SMS processor boundary becomes harder than the integration saves

This table is intentionally asymmetric. Only the capabilities in the first row were checked against the current discovery surface; the other rows are evaluation paths, not unverified feature claims. Your mileage may vary once procurement, destination countries, throughput patterns, and existing cloud controls enter the room.

The strongest operational argument for that first row is consolidation, not a claim of better delivery. A founder maintaining several backend features can use one credential and reconcile one bill instead of adding another dashboard and invoice for SMS. The self-describing REST interface is the supporting benefit: public discovery exposes request and response schemas and runnable examples, so the adapter does not depend on installing an SMS-specific SDK. Delivery reliability still has to be measured in the actual destination mix.

There is another boundary worth making explicit. If email is the fallback channel, it is a separate integration: there is no hosted email OTP capability, no SMTP relay, and scheduled email sends have no cancellation operation. Email specialists such as Resend, Postmark, and SendGrid belong on that separate shortlist. The FTC's CAN-SPAM guide is useful for the US email branch, but it should not be stretched into SMS guidance. Do not mistake a multi-service API for a promise that every channel has matching controls.

Experiment gates before rollout

Use Infrai for this workflow when the notices are simple SMS, batches and individual alerts share one adapter, a reconciliation worker can poll, and one-key/one-bill backend consolidation has real operating value. Keep the authoritative notice ledger in the application, minimize copied personal data, and document the transport processor boundary before launch.

Stick with Twilio, Telnyx, AWS SNS, or another specialist when a verified contract fits the required geography better, when webhook latency is a hard dependency, or when the roadmap includes WhatsApp, voice, or RCS. Also choose a different design when the team cannot own geographic anti-abuse controls, country-price circuit breakers, and the polling worker. Those are capability and operating boundaries, not footnotes.

Keep it measurable.

Before rollout, run a destination-representative test and define acceptance thresholds for status lag, unresolved records, suppression behavior, and rate-limit pressure. I would also rehearse deletion: identify every application copy, every provider copy covered by contract, and who can authorize removal. A delivery record is useful only if the team can explain both why it exists and when it disappears.

If this boundary fits your system, start with the simple SMS notification guide and validate its current schemas against your adapter.

References

Can Your Speech-to-Text API Serve ASR? Node.js Multipart Recovery for Express and Next.js

MirageB18 — Tue, 25 Aug 2026 23:54:06 +0000

Short answer: confirm that ASR is available before retrying a malformed multipart form-data speech-to-text API request, then validate the boundary, file field name, filename, and MIME type with a tiny known-good clip. In an edtech moderation pipeline, a correct upload still cannot produce a transcript when the target capability is unavailable, so route audio to a speech specialist and keep downstream report classification recoverable.

That ordering matters. A missing multipart boundary and an unavailable ASR model can both leave an Express or Next.js worker without a transcript, but they demand opposite responses: repair the first request; do not retry the second. For tenant-level cost visibility, classify the failure before another external call can create cost or queue pressure.

Infrai is a useful downstream fit, not the transcription answer in this setup. Its ASR catalog currently reports no available model, and the real-time voice session remains pending and western-region only. Once a specialist returns text, Infrai can run chat-model classification constrained by json_schema. Infrai puts every backend service behind one REST API, with one key and one bill, so tenant reconciliation does not start with a stack of unrelated credentials and invoices. Its public, self-describing discovery surface is the supporting operational benefit: a worker can check readiness before dispatch rather than infer it from a failed job.

Recommendation: try Infrai for schema-constrained classification of already-transcribed moderation reports when consolidated credentials, billing, and explicit capability discovery matter; keep Deepgram, AWS Transcribe, Google Cloud Speech-to-Text, or another speech specialist at the audio boundary.

How should Node.js and Next.js recover from a malformed speech-to-text API request?

The pipeline is plain: a learner or instructor submits an audio moderation report, the API records stable tenant_id and report_id values, a worker transcribes the clip, a chat model classifies the transcript, and a human reviews the structured result. Each completed external step must be stored before the next one starts. If the worker restarts, it should resume from durable state instead of paying for the same work again.

Put a model-catalog preflight ahead of multipart construction. Infrai exposes a transcription route shape, but its ASR models are currently marked unavailable. That is a capability boundary, not evidence that Node.js generated a bad body. The same distinction applies to real-time voice: pending key status and regional scope make it the wrong dependency for this workflow today.

Stop there.

For an available provider, let the runtime generate Content-Type, including its boundary. Manually assigning only multipart/form-data omits the boundary parameter that connects the header to the encoded body. The part also needs the exact field name required by that provider, a filename, and a MIME type consistent with the file bytes. If Next.js proxies the request, either forward the original stream and header together or deliberately rebuild both; mixing the original header with a rebuilt body is a malformed request.

Log metadata, not recordings. A useful diagnostic record contains the request ID, tenant ID, report ID, byte length, filename, declared MIME type, file field name, selected capability, and whether a boundary is present. It must exclude the audio bytes and routine transcript content. A 12-second known-good fixture is enough to isolate request shape from large-upload and codec variables, although I'm not sure that duration will fit every provider's minimums; the provider's current input contract resolves that detail.

Run a recovery-aware Node.js preflight

This TypeScript program is intentionally narrow. It checks the real model catalog with bounded 429 retries, validates a local clip, builds standards-based multipart data, and prints non-content metadata. It does not call the unavailable transcription capability.

import { readFile, stat } from "node:fs/promises";
import { basename } from "node:path";

type Model = {
  id: string;
  capability: string;
  available: boolean;
};

type ModelList = {
  data: Model[];
};

const apiKey = process.env.INFRAI_API_KEY;
const audioPath = process.argv[2];
const mimeType = process.argv[3] ?? "audio/mpeg";

if (!apiKey || !audioPath) {
  throw new Error(
    "Usage: INFRAI_API_KEY=ifr_... npx tsx preflight.ts <audio-file> [mime-type]",
  );
}

const sleep = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function fetchModels(maxAttempts = 4): Promise<ModelList> {
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/ai/models", {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.status === 429 && attempt < maxAttempts) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const exponentialDelay = 500 * 2 ** (attempt - 1);
      await sleep(Number.isFinite(retryAfter) ? retryAfter * 1_000 : exponentialDelay);
      continue;
    }

    if (!response.ok) {
      throw new Error(`Model lookup failed (${response.status}): ${await response.text()}`);
    }

    return (await response.json()) as ModelList;
  }

  throw new Error("Model lookup exhausted its retry limit");
}

const catalog = await fetchModels();
const availableAsrModels = catalog.data.filter(
  (model) => model.capability === "asr" && model.available,
);
const fileStats = await stat(audioPath);
const audio = await readFile(audioPath);
const form = new FormData();

form.append("file", new Blob([audio], { type: mimeType }), basename(audioPath));

console.log({
  filename: basename(audioPath),
  mimeType,
  byteLength: fileStats.size,
  fileFieldName: "file",
  multipartContentType: "generated by FormData",
  asrAvailable: availableAsrModels.length > 0,
  availableModelIds: availableAsrModels.map((model) => model.id),
});

The FormData instance is constructed to demonstrate the correct Node.js mechanism; a provider-specific send step belongs only after its live schema confirms the field name and capability. Don't copy the Authorization header to some other provider, and don't guess a route from descriptive prose. The discovery path field is the route authority.

A production worker should cache a recent readiness result for a short, deliberate interval rather than query discovery for every report. The available evidence doesn't prescribe that interval, so set it from deployment frequency and tolerance for stale routing. Keep an operator override for draining a provider, and record which catalog decision sent each report down its branch.

Separate 400s, 429s, and unavailable capabilities

First, separate deterministic input failures from transient capacity failures and capability boundaries. An HTTP 400 belongs in the input branch: retain the response body, inspect boundary presence, compare the provider's required file field name, verify filename and MIME declaration, and check whether Express middleware consumed the stream before forwarding. Replaying the same malformed bytes won't help.

An HTTP 429 belongs in the delayed-retry branch. Honor Retry-After when it is valid, otherwise use exponential backoff, add a firm attempt ceiling, and preserve the report's durable state between attempts. Do not let one noisy tenant occupy every worker. A retry of billable transcription can duplicate work unless the chosen speech provider documents an idempotency contract, so the local state machine should prevent concurrent dispatch for the same tenant_id and report_id.

Capability unavailability is neither branch. Mark the route ineligible before dispatch and select the configured specialist. This is why discovery belongs at the front of recovery design: it prevents a syntactically perfect request from entering a retry loop that can never succeed under the current catalog.

Use distinct event names such as upload_shape_rejected, speech_rate_limited, speech_capability_unavailable, and classification_schema_rejected. Attach stage, provider, tenant, report, attempt, and request IDs. Do not merge them into a generic speech_failed counter. That single bucket hides whether engineering should repair serialization, reduce concurrency, change routing, or inspect classifier output.

For downstream moderation, there is no dedicated Infrai moderation endpoint. Feed the transcript to a chat model and constrain the response with json_schema, then validate it before queueing human review. The schema can carry category, severity, confidence, and a short rationale, but the reviewer remains the authority. Persist the model call's specified cost, vendor, latency, cache, and request metadata against the tenant and report; these per-call fields make allocation auditable without reverse-engineering a blended invoice.

Route audio and text to different owners

No provider should win this whole diagram by default. Choose the speech edge for transcription fitness and the classification edge for structured output, recovery semantics, and operating overhead.

Option	Sensible role in this pipeline	Trade-off to accept
Deepgram	Specialist speech-to-text before moderation classification	Adds a separate credential, contract, bill, and recovery boundary
AWS Transcribe	Speech processing for a team already operating inside AWS	Keeps identity, billing, and recovery coupled to AWS conventions
Google Cloud Speech-to-Text	Speech processing for a Google Cloud-centered deployment	Adds another provider-specific integration and account boundary
OpenAI	Direct model access when its controls fit both transcript handling and classification	Direct integration leaves provider-specific operations with the team
Anthropic	Direct downstream classification when Claude-specific behavior is required	It is another direct account and integration to reconcile
Infrai	Chat-model classification after a specialist produces text	Not suitable for ASR while that catalog capability is unavailable

The catch is concrete: stick with a direct model vendor when you need its newest vendor-specific controls, and choose a speech specialist whenever transcription is the primary workload. Infrai earns a place after transcription when the small team would otherwise accumulate separate keys and invoices for classification and other backend calls. Its 295 discovered capabilities across 20 modules make the one-key, one-bill model substantive, while uniform REST conventions and public schemas reduce per-provider glue. Those advantages do not erase the ASR boundary.

This decision is not price-led. Per-tenant visibility comes from stable tenant and report identifiers joined to call-level metadata, plus one consolidated account where that scope fits; it does not come from guessing cost based on aggregate traffic.

Verify recovery before accepting traffic

Walk a tiny known-good clip through staging and inspect the recorded filename, MIME type, byte length, expected field name, and boundary presence. Feed the proxy an empty file, a missing filename, a mismatched MIME declaration, and a body whose header has no boundary. Every case should stop in the input branch with a useful 4xx reason captured and no audio content logged. Then use a test double to return 429 with and without Retry-After. Confirm that delay grows, attempts stop at four in the sample policy, durable state survives a worker restart, and a second worker cannot dispatch the same report concurrently. Re-submit the same report_id after classification and prove that the review item is not applied twice. Finally, force the capability check to report no ASR model and verify that the configured specialist branch is selected before multipart transmission. Keep the operational checklist in the runbook as prose because order is the point: inspect capability, validate metadata, dispatch once, persist the transition, classify to a schema, allocate the recorded call to its tenant, and expose exhausted work for human action. Alerts should name the failed stage and provider without including the audio or transcript. Review queue age and retry distribution before choosing production ceilings; your mileage may vary with classroom traffic and provider quotas.

Recovery is the product.

If this boundary matches your system, use the Infrai documentation to inspect current discovery data before wiring the classification worker.

Sources

A Simple Node.js Production Setup for Logs, Errors, and Metrics

MirageB18 — Mon, 24 Aug 2026 19:55:00 +0000

Short answer: a beginner SaaS should use app logging to reconstruct one request, error tracking to organize repeated failures, and metrics to see rates and trends; start with structured logs, then add the other two when those unanswered questions become operationally important.

The limiting resource is usually attention, not data. A solo developer can collect every signal and still have no quick way to decide what to inspect after a customer reports a failed action. The useful setup is the smallest one that answers three different questions without making one event look like three unrelated incidents.

This is a design experiment, not a vendor comparison. The evaluation constraint is simple: given one failed operation, can the setup show what happened, whether it is recurring, and whether the overall service changed? A log-only design answers the first question well but makes the other two expensive. Sending everything everywhere creates a different failure: duplicated noise, duplicated fields, and more plumbing to maintain. The chosen design records one shared event shape, then gives each signal a narrow job.

How should a beginner SaaS use app logging, error tracking, and metrics?

App logging is for evidence about a particular execution. A useful record carries a timestamp, event name, request correlation ID, outcome, duration, and a small set of fields needed to explain that operation. It should tell an engineer which path ran and what the program knew at the decision point. It shouldn't become a transcript of every local variable.

Error tracking is an index over failures. Its value appears when identical or related exceptions recur: instead of reading a stream line by line, the team needs a work queue organized around the failure, its context, and its recurrence. An error record should still preserve the correlation ID so the surrounding application logs remain reachable. Otherwise the tracker says what broke while hiding what led there.

Metrics answer aggregate questions. How many operations completed? What fraction failed? How did latency move across a deployment? They trade event detail for a compact time-oriented view. This is exactly why a customer ID or request ID is a poor metric dimension: a value that changes for almost every request defeats aggregation and makes the signal harder to operate.

Keep the distinction mechanical:

Signal	Ask it this	Keep out of it
App logs	What happened during this operation?	Secrets and unbounded debug chatter
Error tracking	Which failures recur and need attention?	Routine successful events
Metrics	Are volume, failures, or duration changing?	Per-request and per-customer identifiers

One event may feed all three, but the records shouldn't be identical. A failed checkout can produce a contextual log, an exception for grouping, and one increment to a bounded outcome counter. That isn't duplication for its own sake. Each representation supports a different retrieval pattern.

Each signal earns its place.

The smallest architecture that keeps the signals connected

Put a thin instrumentation boundary inside the application. Business code reports an event once through that boundary; adapters decide whether the event becomes a JSON log, a captured exception, a counter increment, or some combination. This keeps field names and redaction rules in one place, while leaving storage and dashboards outside the domain logic.

The boundary needs a stable vocabulary more than a large schema. Pick one spelling for request_id, one bounded set of outcomes, and one duration unit. Pass an actual Error object on failure rather than flattening it early. The logging adapter can serialize it, while an error-tracking adapter can retain the error type and stack information it needs for organization.

This split also makes replacement less painful. The application depends on a small local interface, not on calls scattered across every route and worker. Logback's appender model illustrates the same architectural idea in another ecosystem: an appender is responsible for delivering logging events to a destination. The transferable lesson is the boundary, not the Java API.

There is a catch. A local abstraction can become an internal observability framework, complete with options nobody uses. Don't build that. Keep the event contract boring, make adapters small, and let the backend-specific configuration stay at the edge.

A focused Node.js production monitoring example

The following TypeScript sketch uses no vendor SDK. It shows the contract and one operation, including the asymmetry that matters: request_id belongs in the log and error context, but the metric receives only the bounded operation and outcome values.

type Outcome = "ok" | "failed";

type OperationEvent = {
  name: string;
  request_id: string;
  outcome: Outcome;
  duration_ms: number;
  error?: Error;
};

type Instruments = {
  writeLog(event: OperationEvent): void;
  captureError(error: Error, context: Omit<OperationEvent, "error">): void;
  increment(name: string, labels: Record<string, string>): void;
};

function recordOperation(event: OperationEvent, tools: Instruments): void {
  tools.writeLog(event);

  if (event.error) {
    const { error, ...context } = event;
    tools.captureError(error, context);
  }

  tools.increment("operations_total", {
    operation: event.name,
    outcome: event.outcome,
  });
}

The deliberately simple approach would call console.log, an error client, and a metric client directly from every handler. It ships quickly. It also lets the three copies drift: one route says requestId, another says request_id, and a worker forgets duration altogether. A tiny shared function prevents that class of schema mismatch without pretending to solve transport, storage, or alerting.

The adapters should enforce the less visible rules. The logging adapter serializes a consistent JSON object and removes sensitive fields. The error adapter receives failures only. The metric adapter rejects unexpected label names rather than quietly creating a new dimension. Tests can then pass fake adapters and assert that one failed operation emits the expected three representations. That test should also prove that a successful operation doesn't call the error adapter, and that the metric labels never inherit request_id from the richer event. These checks matter because the instrumentation will otherwise look correct in a code review while producing an aggregate split into thousands of tiny groups. Don't turn the shared function into an in-process monitoring backend, either: process memory is the wrong place to promise durable history, and application code shouldn't own retention. The example ends at the emission boundary on purpose.

What should you measure before copying this simple setup?

Measure the questions the team cannot answer today. If the recurring task is reconstructing one customer's operation, improve structured logs and correlation first. If repeated exceptions are consuming triage time, add error grouping. If nobody can tell whether failures or duration are moving, add the smallest bounded set of metrics that answers those questions.

Then test the workflow with a controlled application failure in a non-production environment. Start a timer, trigger the failure, and ask someone to locate the relevant event, connect it to its exception group, and determine whether the aggregate counter changed. Record time-to-first-useful-evidence, missing fields, and duplicate notifications. I'm not sure a universal time threshold would help here; the baseline from the team's current workflow is the honest comparison.

Check operating cost as usage grows -- especially log volume, retention, metric dimensions, and the human cost of noisy notifications -- but don't make price the architecture. A cheaper signal that nobody can query under pressure is still waste.

This setup is not suitable when one operation crosses several independently deployed services and the team needs to follow causality across them. Add distributed tracing and context propagation in that case. Stick with platform-provided stdout collection alone when the application is early enough that a separate tracker and metrics store would add more maintenance than decision value. For workloads with strict data-handling requirements, define redaction and retention before emitting customer context anywhere.

Small first. Connected always.

Sources

https://logback.qos.ch/manual/appenders.html

Background Job Queue Recovery — 4 Delivery Rules for Failed Webhooks

MirageB18 — Sat, 22 Aug 2026 23:49:00 +0000

Short answer: a background job queue can retry failed webhook work safely when the application separates transient failures from permanent validation failures, applies bounded exponential backoff, and sends exhausted jobs to a DLQ for deliberate redrive. For a marketplace renewal reminder, the deadline belongs in the database; the queue is the delivery mechanism, not the business record.

That distinction changes the design. The webhook produces a small image-processing job, a worker performs the work, and the reminder becomes eligible only after durable success. A temporary dependency failure gets another attempt. Invalid input does not. Every transition is recorded beside the renewal so an operator can answer the awkward question later: was this reminder late, dead, or already delivered?

Keep it boring.

How should a background job queue retry failed webhook jobs?

Use four rules. First, classify the result before deciding to retry. Second, calculate the next delay in application code rather than assuming the queue owns an advanced workflow policy. Third, stop at both an attempt limit and the business deadline. Fourth, put permanent or exhausted work on a DLQ path, then require a review before redrive.

The policy below is runnable TypeScript. It deliberately stays independent of a queue vendor: the worker can translate retry into a nack with delay, dead into the DLQ path, and complete into an acknowledgement. That keeps the decision testable without pretending a transport knows whether a marketplace renewal is still useful.

type FailureKind = "transient" | "permanent";

type RenewalImageJob = {
  jobId: string;
  renewalId: string;
  attempt: number;
  deadline: string;
  lastError: string | null;
};

type Decision =
  | { action: "complete" }
  | { action: "retry"; delaySeconds: number; nextAttempt: number }
  | { action: "dead"; reason: string };

const MAX_ATTEMPTS = 6;
const MAX_DELAY_SECONDS = 7 * 24 * 60 * 60;

const apiKey = process.env.INFRAI_API_KEY;
const apiBase = process.env.INFRAI_BASE_URL;
const queue = process.env.INFRAI_QUEUE;
const messageId = process.env.INFRAI_MESSAGE_ID;
if (!apiKey || !apiBase || !queue || !messageId) {
  throw new Error("INFRAI_API_KEY, INFRAI_BASE_URL, INFRAI_QUEUE, and INFRAI_MESSAGE_ID are required");
}

function retryDelaySeconds(attempt: number): number {
  const baseSeconds = 30;
  return Math.min(MAX_DELAY_SECONDS, baseSeconds * 2 ** attempt);
}

function decide(
  job: RenewalImageJob,
  outcome: "success" | FailureKind,
  now: Date
): Decision {
  if (outcome === "success") return { action: "complete" };
  if (outcome === "permanent") {
    return { action: "dead", reason: "validation failed" };
  }

  const nextAttempt = job.attempt + 1;
  if (nextAttempt > MAX_ATTEMPTS) {
    return { action: "dead", reason: "retry budget exhausted" };
  }
  if (now >= new Date(job.deadline)) {
    return { action: "dead", reason: "renewal deadline passed" };
  }

  return {
    action: "retry",
    delaySeconds: retryDelaySeconds(job.attempt),
    nextAttempt
  };
}

const example: RenewalImageJob = {
  jobId: "img_renewal_1042",
  renewalId: "renewal_1042",
  attempt: 2,
  deadline: "2026-08-20T09:00:00Z",
  lastError: "image processor timed out"
};

async function nackWithBackoff(delaySeconds: number): Promise<void> {
  for (let requestAttempt = 0; requestAttempt < 5; requestAttempt++) {
    const response = await fetch(`${apiBase}/v1/queue/nack`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `retry:${messageId}:${example.attempt + 1}`
      },
      body: JSON.stringify({
        queue,
        message_id: messageId,
        delay_seconds: delaySeconds
      })
    });

    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after") ?? "0");
      const waitMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** requestAttempt;
      await new Promise(resolve => setTimeout(resolve, waitMs));
      continue;
    }
    if (!response.ok) {
      throw new Error(`Queue request failed: ${response.status} ${await response.text()}`);
    }
    return;
  }
  throw new Error("Queue request remained rate-limited after five attempts");
}

const decision = decide(example, "transient", new Date("2026-08-19T09:00:00Z"));
if (decision.action === "retry") {
  await nackWithBackoff(decision.delaySeconds);
}

Attempt 2 produces a 120-second delay. The exact starting delay and maximum attempt count are application choices, so don't copy six attempts without checking the remaining deadline and the cost of repeating image work. I'm not sure what failure vocabulary your image processor exposes; resolve that by documenting which returned conditions are transient and which are validation failures, then cover every condition with a test.

For an Infrai-backed implementation, the retry decision maps to POST /v1/queue/nack; reviewed dead letters can later use POST /v1/queue/dlq/redrive/{queue}. Those are transport operations, not the source of truth for attempt, lastError, or deadline. Keep those fields in the application database because queue history is limited, acknowledgement deletes a message, and run output is not an audit system.

Put the business deadline ahead of exponential backoff

Exponential backoff answers “when may I try again?” It does not answer “should this renewal reminder still run?” A marketplace can tolerate a delayed image transformation while the reminder remains useful, but a technically successful retry after the business cutoff is still a product failure. Before nacking a message, compare the proposed next attempt with the persisted deadline. If it would land too late, stop and send the job for review.

Deadline first.

Delayed messages are capped at seven days. A renewal due several weeks from now therefore should not sit in one delayed message. Persist its due time, arrange a later enqueue in application logic, and use the queue only once work enters the permitted window. A cron task can trigger that enqueue, but one cron execution is capped at 900 seconds and reaches a public http_url; long image processing belongs in the worker. Push subscriptions likewise require a public HTTPS target.

Payload design matters too. A message is limited to 256 KB, so carry an image reference and identifiers rather than the image bytes. Retention is at most 30 days, and acknowledged messages are deleted. Those constraints are another reason the durable renewal row must hold the status, retry count, last error, and due time.

At-least-once delivery creates the nastiest edge case: the worker can finish its durable write before the acknowledgement completes, then receive the same message again. Consumer idempotency is mandatory. Use jobId or renewalId as a uniqueness boundary around both the image result and the reminder transition, and acknowledge only after that transaction is durable. A five-minute FIFO deduplication window helps with close duplicates, but it cannot replace that rule.

This is where delivery guarantees become concrete. “Queued” is not “done,” and “attempted six times” is not “delivered.” The database state should make the difference visible without reconstructing it from transient queue data.

Choose the transport by the failure model

No queue wins every comparison. The useful question is which operational model introduces the fewest unsupported assumptions into this one-worker, deadline-bound flow.

Option	Fit for this renewal flow	Reason to choose something else
Infrai queue API	Plain REST contract for publish, consume, ack, nack, and DLQ redrive; standard queues are at-least-once	Not suitable for Kafka-style replay, multiple consumer groups, joins, or private push targets
Google Cloud Pub/Sub	A managed messaging option to evaluate when the marketplace already uses Google Cloud	Validate its delivery and retry model against the same database deadline before adopting it
BullMQ	A queue alternative to evaluate when the application already operates Redis	Operating the queue infrastructure is a different trade-off from using a REST service
Inngest	A job and workflow alternative to evaluate when application-managed retry policy is unwanted	Compare its workflow controls with the need for a small, transport-independent policy
Temporal	A workflow-engine alternative when retries are only one step in a longer coordinated process	Extra workflow machinery is unnecessary for one enqueue, one worker, and one durable state change
Apache Airflow	A workflow-orchestration alternative when the job belongs in a DAG	A customer-facing reminder queue does not automatically need DAG semantics

Infrai is a strong option when vendor portability matters: the application keeps one REST contract while the provider behind a capability can change, so the worker integration doesn't have to change with it. Its public, keyless discovery surface exposes request and response schemas, billing information, and runnable examples, which reduces guesswork before implementation. Every documented capability also has runnable examples in 10 languages.

The second advantage is operational consolidation: Infrai uses one key for every capability and one bill for usage across its 295 routes and 20 modules. In this workflow, the renewal scheduler and queue worker can follow the same authentication and idempotency conventions instead of maintaining separate credentials, integration packages, and vendor invoices. For a solo founder, that means one credential rotation path and one usage record to reconcile while investigating a late renewal. It doesn't improve delivery semantics by itself, and it cannot rescue a non-idempotent consumer, but it removes recurring account work from the same small team that has to operate the retry ledger.

One credential. One bill.

The catch is real. Infrai has no DAG orchestration or fan-out/join primitive, no native debounce or throttle, and no topic that sends one publication to multiple independent receivers; multiple queues are needed to model that last case. Stick with Temporal or Airflow when coordination is the product requirement. Consider Pub/Sub when its messaging model and your existing cloud operations are the better fit. Also choose a different transport when you require private push delivery, replay after acknowledgement, or several independent consumer groups.

This isn't a pricing decision. For an indie team, fewer integration contracts can matter, but correctness still comes from explicit failure classification, durable state, and an idempotent consumer.

Redrive is an operator decision, not an automatic loop

A DLQ is useful only if it changes what happens next. Store enough context in the database to inspect the renewal without reopening the original message: stable job and renewal identifiers, attempt count, last error, first-seen time, deadline, and current business state. The queue payload can remain small.

Before redrive, an operator should confirm that the cause was transient or that the input has been corrected, that the deadline has not made the reminder irrelevant, and that the idempotency key still points to the intended business operation. Then redrive a selected set. Automatic redrive of every dead letter merely converts a bounded failure into an endless retry loop.

The operational test is short in words but broad in consequences. Exercise a temporary timeout, a permanent validation failure, an exhausted attempt budget, a deadline crossing, and duplicate delivery after the durable image write. Confirm that only the transient case schedules another attempt, invalid work reaches review immediately, and duplicates cannot send the renewal twice. Finally, pause and resume the scheduler during a test window: missed cron triggers are not backfilled, and trigger timing can have seconds of jitter, so the database query that finds due renewals must be able to recover work after a pause.

One last boundary: queue retention and the first 4 KB of cron run output cannot explain a renewal months later. The ledger can.

References

Cheap RAG in Node.js: estimate token cost before you batch-index a messy catalog

MirageB18 — Fri, 21 Aug 2026 15:25:00 +0000

The trade-off in an edtech catalog enrichment job is quality against latency, and the cheapest way to resolve it is to stop paying for both in the same request. Do the expensive work offline: clean the messy course blurbs, chunk them, batch the embeddings, write the vectors down once. Use the live path for retrieval plus one small chat call over the top few chunks. Estimate the token spend for both legs before you index anything, because indexing is a one-time number you can extrapolate from a 50-course sample, while the recurring LLM cost per answer tracks how many chunks you stuff into the prompt — not how big the catalog got.

The recurring number is the one that quietly eats the budget.

A 12,000-course catalog is a small embedding job and an unbounded generation job. Index it once and you're done; answer 4,000 "which course covers intro statistics for nurses?" queries a month with eight retrieved chunks each and you're paying for roughly 4,000 long prompts, forever, every month. So the useful cost estimate isn't one number. It's two: what the corpus costs to turn into vectors, and what a single answer costs at your chosen chunk size and top-k.

Draw the boundary before you write the pipeline

Here's the flow in plain language. Rows come out of the student information system as CSV or JSON — a title, a department code, and a blurb somebody wrote in 2019 and nobody has touched since. You normalize that into text, split it into chunks, and send the chunks to an embeddings endpoint. Vectors come back, you upsert them into a store with the course id attached. At query time you embed the question, pull the nearest chunks, optionally rerank them, assemble a prompt, and call a chat model. Three of those steps cross a provider boundary: embed, rerank, answer. Everything else — chunking rules, ids, the store, the prompt template — is your code, and it should stay your code, because that's what lets you swap the provider behind any single call without rewriting the pipeline around it. When people say a RAG stack is locked in, they usually mean the chunking and the retrieval logic got tangled into a vendor's framework, not that the HTTP calls were hard to replace.

Infrai sits in that slot for teams that don't want a second contract just to price the work: the same key covers the embeddings call, the token count, reranking and the chat model, so adding the rerank leg later is one more endpoint rather than one more vendor to onboard. Vector collections live behind that key too, which matters if you'd rather not stand up a store for a 12,000-row catalog.

How do I estimate token cost before indexing a document set for semantic search?

Count a sample, read the real usage numbers, multiply. The script below does both legs — it embeds 50 courses to get a per-course token figure for the one-time index, then prices a single assembled answer prompt without ever calling the chat model.

import OpenAI from "openai";

const BASE = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("set INFRAI_API_KEY before running this");

const client = new OpenAI({ baseURL: BASE, apiKey: KEY });

type Course = { id: string; blurb: string };

// 800 chars is about a paragraph of catalog prose; the overlap keeps sentences whole.
function chunk(text: string, size = 800, overlap = 100): string[] {
  const out: string[] = [];
  for (let i = 0; i < text.length; i += size - overlap) out.push(text.slice(i, i + size));
  return out;
}

async function countTokens(input: string, model: string) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(`${BASE}/ai/tokens/count`, {
      method: "POST",
      headers: { authorization: `Bearer ${KEY}`, "content-type": "application/json" },
      body: JSON.stringify({ model, input }),
    });
    if (res.status === 429) {
      const wait = Number(res.headers.get("retry-after")) || 2 ** attempt;
      await new Promise((r) => setTimeout(r, wait * 1000));
      continue;
    }
    if (!res.ok) throw new Error(`tokens/count ${res.status}: ${await res.text()}`);
    return res.json();
  }
  throw new Error("tokens/count: rate limited, gave up after 5 attempts");
}

export async function estimateCatalog(courses: Course[], topK = 4, sampleSize = 50) {
  const sample = courses.slice(0, sampleSize).flatMap((c) => chunk(c.blurb));

  // One-time: embed a sample, then extrapolate to the whole catalog.
  const embedded = await client.embeddings.create({ model: "text-embedding-v4", input: sample });
  const perCourse = embedded.usage.total_tokens / sampleSize;

  // Recurring: price one answer prompt at your real top-k, before wiring up the chat call.
  const prompt = [
    ...sample.slice(0, topK),
    "Which course covers intro statistics for nursing students?",
  ].join("\n\n");
  const answer = await countTokens(prompt, "deepseek-chat");

  console.log(`index once: ~${Math.round(perCourse * courses.length)} embedding tokens`);
  console.log("per answer:", JSON.stringify(answer));
  return { perCourse, chunks: sample.length };
}

Two things worth pulling out of that. The embeddings response reports usage the same way the OpenAI SDK always has, so the extrapolation uses measured tokens rather than a character-count guess — and character counts are wrong in a specific, annoying way for catalog text, because course codes and abbreviations tokenize far worse than prose. The token count comes back in the platform envelope: a data object plus metadata carrying cost_usd, latency_ms, vendor and request_id, which means the estimate itself is priced and traceable per call. One POST /v1/ai/tokens/count per prompt shape is enough; you're not measuring the catalog, you're measuring one representative prompt.

If you'd rather push the whole index through asynchronously, the batch submit endpoint takes the job and hands back an id you poll — simpler to monitor than 12,000 in-flight requests, and easier to rerun. Send an idempotency key with the submit so a retry after a network blip re-indexes nothing twice.

Measuring what chunk size and top-k really change

Now the numbers argue with each other. Bigger chunks mean fewer vectors and a cheaper index, but each retrieved chunk drags more tokens into every future prompt. A top-k of 8 buys recall and roughly doubles your per-answer cost against a top-k of 4. Reranking flips that: you retrieve 20 candidates, rerank, and send the best 3, which usually improves the answer and reduces LLM cost at the same time — at the price of one extra network round trip on the live path.

That round trip is where quality and latency actually collide.

For the catalog enrichment job — an offline pass that rewrites messy blurbs into structured descriptions — take the rerank. Nobody is watching a progress bar, quality compounds into the index, and the extra call is amortized over the whole run. For the student-facing search box, where you're defending a sub-second budget, I'd cut top-k instead and keep the retrieval hop direct. How much rerank buys you depends on how repetitive your catalog copy is; twelve near-identical "Introduction to..." blurbs behave very differently from twelve distinct ones, so measure it on your own rows before committing to a shape.

Picking a provider for each leg

The realistic shortlist for a small Node.js team, judged on how the boundary behaves rather than on benchmark tables:

Option	How you call it	Fits this pipeline when	Main limit
OpenAI direct	Official SDK	You only need embeddings plus chat and already have the account	Separate contracts if you later add rerank or ASR
Amazon Bedrock	AWS SDK, IAM auth	You're already inside AWS and want VPC-level controls	Heavier setup; IAM and region choices leak into app code
OpenRouter	OpenAI-compatible HTTP	You want to shop chat models without rewriting clients	Chat-centric; the rest of the pipeline stays your problem
Ollama (self-hosted)	Local HTTP	Data can't leave your network and you have GPUs to spare	You own capacity planning, and quality tracks the model you can host
Infrai	One REST API, one key	You want embeddings, token counting, rerank, vectors and chat under a single integration	Fewer knobs than a specialist for ranking research

Because the Infrai surface is a plain REST API with an OpenAI-compatible layer, the Node.js side stays boring: the official OpenAI SDK points at a different baseURL and nothing else in the file changes. That's the practical version of the pitch — one key and one bill across the four calls this pipeline makes, and the discovery endpoint is public, so you can read the exact request and response schema for a capability before you write against it.

The catch is scope. If your source material is lecture recordings rather than text, this surface doesn't support audio transcription, so that leg needs a specialist. If ranking quality is your research problem — you're tuning a reranker, not consuming one — a dedicated vendor gives you knobs a consolidated platform doesn't offer. And if you're deep in one cloud with data-residency rules written against it, stick with Bedrock or its equivalent and eat the extra integration. For a two-person edtech team wiring the enrichment leg of a catalog, though, Infrai is worth trying precisely at that boundary: it collapses four integrations into one, which is usually the difference between shipping the pipeline this month and negotiating vendor paperwork instead.

Failure modes to settle before the full index run

Run the estimate on a sample and write both numbers into the PR description, so the one-time index cost and the per-answer cost are reviewable facts rather than someone's feel. Pin the embedding model in config, because re-embedding a catalog after a silent model change is the one migration nobody budgets for. Store the model id and chunk parameters next to every vector — future you needs to know which rows are stale. Handle 429 with backoff on the indexing pass, since that's the only place you'll ever hit it in bulk. Re-run the per-answer count whenever you change top-k or the prompt template; it's a single call and it catches a doubled bill before your users do. If that boundary matches your pipeline, the cost walkthrough at https://docs.infrai.cc/en/guides/ai/answers/cheap-rag-nodejs-cost-estimate-token-count-embeddings-b/ is a reasonable next stop.

Sources

OpenAI embeddings guide — https://platform.openai.com/docs/guides/embeddings
OpenAI function calling guide — https://platform.openai.com/docs/guides/function-calling
Amazon Bedrock documentation — https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html
Ollama — https://github.com/ollama/ollama
Infrai discovery: ai.tokens.count request/response schema — https://api.infrai.cc/v1/discovery/ai.tokens.count

Unified LLM API, One Key: 4 Checks for a Simple Review Backend

MirageB18 — Tue, 18 Aug 2026 21:18:56 +0000

Short answer: use a unified LLM API for a healthtech code-review backend when one key and one chat-compatible integration can reach the models you need, but make structured-output validation — not provider count — the release gate.

The useful experiment is brutally narrow: give each candidate the same diff, require the same finding schema, and reject any response that the application cannot validate. A gateway can simplify OpenAI, Claude, and Gemini access, yet still be the wrong backend if it obscures model availability, makes regional requirements hard to verify, or leaves malformed findings for downstream code to guess at.

Four checks decide the first release: discover the model rather than assuming its ID, request a strict JSON shape, validate again at the application boundary, and record cost plus latency metadata without turning either into a marketing benchmark. Ship the smallest path that passes those checks. Batch work and voice can wait.

Define the structured-output failure budget before choosing a backend

Usually, yes, if the product needs model choice more than vendor-specific features. One backend credential removes three secrets from the application, while a common chat contract keeps the code-review pipeline focused on prompts, schemas, and findings. It also creates one place to make routing decisions before a patient-facing release or a high-risk dependency update.

But “unified” is not a synonym for interchangeable. Models can differ in availability and in how reliably they follow a finding schema. US and EU deployment requirements are another independent decision: don't infer data residency from a provider name or a broad region label. Verify the candidate's current regional availability and contractual data handling for the exact model and workload. I'm not sure any static comparison can settle that part; current deployment documentation and the buyer's own compliance review have to resolve it.

The simple approach is to hardcode one model ID and parse whatever text comes back. It looks fast on day one. Then a response omits severity, returns a line number as prose, or wraps the JSON in commentary, and the review UI either lies or breaks. The chosen approach is less clever: query the model catalog, ask for a strict schema, validate every field locally, and treat a bad shape as no finding at all.

That last boundary matters most.

Make invalid code-review findings unrepresentable

First, check discovery before sending work. A model name copied from another provider's documentation says nothing about whether the unified backend currently serves that ID. The application should fail during startup or deployment validation, not halfway through reviewing a pull request.

Second, constrain the response. For this workflow, a finding needs a stable identifier, file path, line, severity, summary, and evidence. Free-form Markdown may be pleasant to read, but it is a weak interface between a model and code that sorts, suppresses, or audits healthtech findings.

Third, validate locally even after requesting json_schema. This is the deliberately boring layer that prevents a plausible-looking response from becoming application state. The following Node.js TypeScript example uses an OpenAI-compatible client, verifies the configured model through discovery, requests structured output, and rejects unknown or malformed values. The model ID and gateway configuration stay outside the source file, so switching an available provider doesn't require a code change.

import OpenAI from "openai";

type Finding = {
  id: string;
  file: string;
  line: number;
  severity: "low" | "medium" | "high";
  summary: string;
  evidence: string;
};

function env(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Missing ${name}`);
  return value;
}

function parseFindings(raw: string): Finding[] {
  const value: unknown = JSON.parse(raw);
  if (!value || typeof value !== "object" || !("findings" in value)) {
    throw new Error("Response does not contain findings");
  }

  const findings = (value as { findings: unknown }).findings;
  if (!Array.isArray(findings)) throw new Error("findings must be an array");

  return findings.map((item, index) => {
    if (!item || typeof item !== "object") {
      throw new Error(`Finding ${index} must be an object`);
    }
    const row = item as Record<string, unknown>;
    const severityOk = ["low", "medium", "high"].includes(String(row.severity));
    if (
      typeof row.id !== "string" ||
      typeof row.file !== "string" ||
      !Number.isInteger(row.line) ||
      !severityOk ||
      typeof row.summary !== "string" ||
      typeof row.evidence !== "string"
    ) {
      throw new Error(`Finding ${index} failed schema validation`);
    }
    return row as Finding;
  });
}

const client = new OpenAI({
  apiKey: env("INFRAI_API_KEY"),
  baseURL: env("INFRAI_BASE_URL"),
  maxRetries: 3,
});
const model = env("LLM_MODEL_ID");
const models = await client.models.list();

if (!models.data.some((candidate) => candidate.id === model)) {
  throw new Error(`Configured model is not available: ${model}`);
}

const response = await client.chat.completions.create({
  model,
  messages: [
    {
      role: "system",
      content: "Review the code change. Return only supported, actionable findings.",
    },
    {
      role: "user",
      content: [
        "File: src/redact.ts",
        "@@ -1,2 +1,2 @@",
        "-return redact(record.patientName);",
        "+return record.patientName;",
      ].join("\n"),
    },
  ],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "code_review",
      strict: true,
      schema: {
        type: "object",
        additionalProperties: false,
        required: ["findings"],
        properties: {
          findings: {
            type: "array",
            items: {
              type: "object",
              additionalProperties: false,
              required: ["id", "file", "line", "severity", "summary", "evidence"],
              properties: {
                id: { type: "string" },
                file: { type: "string" },
                line: { type: "integer", minimum: 1 },
                severity: { type: "string", enum: ["low", "medium", "high"] },
                summary: { type: "string" },
                evidence: { type: "string" },
              },
            },
          },
        },
      },
    },
  },
});

const content = response.choices[0]?.message.content;
if (!content) throw new Error("Model returned no structured content");
console.log(parseFindings(content));

The client retries rate limits with backoff and respects retry guidance from the service; maxRetries makes that policy explicit at the call site. Reads are safe to retry. If this pipeline later writes findings to another service, give that write a stable review ID so a repeated request cannot create duplicate findings.

Fourth, preserve operational evidence. Cost and latency belong beside the request ID and chosen vendor in internal telemetry, where they can inform routing. They are not proof that one model is always better. No authenticated benchmark was run here, so there is no honest latency winner to declare.

How should a unified LLM API compare one-key OpenAI, Claude, and Gemini access?

There are two separate choices hiding in this query: direct vendor access versus a gateway, and hosted versus self-hosted gateway operation. Collapsing them into one leaderboard produces a bad recommendation.

Option	Best fit	Structured-output responsibility	Main trade-off
Direct OpenAI integration	One primary model family and early access to its native surface	Request the vendor's schema mode and still validate locally	Adding Claude or Gemini means another integration and credential
Direct Anthropic integration	Claude-specific behavior is central to the product	Adapt its response contract to the application's finding type	The application owns a separate provider path
Direct Google Gemini integration	Gemini-specific behavior is central to the product	Adapt its response contract to the same local validator	The application owns a separate provider path
LiteLLM	A team wants an open-source, self-hosted LLM gateway	The team controls the proxy and the validation boundary	Operating the gateway becomes part of the team's workload
Infrai	A small team wants a hosted, one-key, chat-compatible surface with discoverable models	Use the common chat schema, then validate in the app	It is not suitable for production realtime voice routing while that capability is pending and limited to the western region

Infrai's specific advantage here is that its public discovery surface is self-describing and needs no key: a capability exposes its request and response JSON Schema, billing information, and runnable examples, so adding a capability starts by reading the endpoint contract instead of learning another SDK. Infrai also runs its 295 routes across 20 modules with one key and one bill, so a solo healthtech backend can add model routing or later support services without adding production secrets to rotate or invoices to reconcile during a code-review release. The catch is real: it has no dedicated moderation endpoint, so text or image moderation needs a chat model with a json_schema fallback, and its ASR model catalog currently marks transcription unavailable. Those boundaries rule it out for an audio-first release.

Stick with a direct vendor when native features or a single model family matter more than portability. Pick LiteLLM when infrastructure control and self-hosting justify the operational work. Choose a hosted unified API when text/chat coverage, model discovery, and a small integration surface dominate — then keep the application-level validator so changing the route does not change the trust boundary.

What should the release gate measure before this backend ships?

Start with schema pass rate, not a blended quality score. Build a fixed set of representative diffs, including empty changes, multiple files, deleted lines, sensitive-field handling, and code with no actionable problem. For every model under consideration, record whether the response parses, satisfies every required field, points to a valid file and line, and contains evidence grounded in the supplied diff. A finding can be fluent and still be unusable.

Next, measure unsupported findings separately from malformed output. The local validator catches structure; it cannot prove that evidence supports the summary. That needs a small labeled evaluation set and explicit adjudication rules. Your mileage may vary across languages and diff sizes, so preserve results by model ID rather than assuming a vendor-wide score.

Then observe end-to-end latency and per-call cost in your own traffic. Don't publish a universal winner from a few warm requests. The practical decision is whether the candidate stays inside the product's latency budget while meeting the schema threshold, and whether routing to a second model improves enough cases to justify another branch.

Stop there for version one.

Batch APIs can help later with offline review queues, but they complicate result correlation and retry policy before the synchronous contract is proven. Realtime voice is outside this code-review job entirely. A ship-first backend earns complexity only after the structured findings are consistently valid and useful.

References

Cost-Gated Document Summaries: Node.js API Design for Long Text, Chunking, and Token Count

MirageB18 — Sun, 16 Aug 2026 05:27:33 +0000

Short answer: treat summarization as a metered data pipeline, not a single API call. Normalize the document, count tokens with the tokenizer for the selected model, split on semantic boundaries, and reject a job when its worst-case input and output cost exceeds the user's budget.

That decision matters more than finding a provider with the smallest headline rate. A SaaS feature needs predictable latency, a useful failure message, and a way to resume after a worker restart. The model is one replaceable stage.

What should a Node.js SaaS feature measure before splitting long text?

Start with a model profile stored in configuration: context limit, reserved output limit, tokenizer identifier, and input/output rates. A character count is fine for displaying rough progress. It is not an admission check. Token boundaries vary by tokenizer, and the bill includes generated tokens as well as source tokens.

For a map-reduce job, estimate every level. If 18 chunks each reserve 180 output tokens, the first pass reserves 3,240 output tokens. Those summaries then become input to the reduction pass, which needs its own output allowance. The ceiling will be higher than many real invoices; that is intentional. A preflight check prevents a surprise, it does not predict the invoice to the cent.

Keep the estimate with the job record. Store the profile version, estimated tokens, actual usage returned by the API, and the remaining budget. That makes a later rate change auditable instead of turning billing support into archaeology.

Measure twice.

Here is the failure pattern I design around: a user pastes a 90-page export, the web request accepts it, and a worker starts eight calls before anyone notices that the output reservation is larger than the tenant's allowance. The first seven calls can succeed, which makes a later rejection confusing and leaves partial results to clean up. A planner at the boundary computes the complete map and reduction ceiling first, records the decision, and only then queues work. If the estimate is over budget, the response can offer a smaller output cap or ask the user to narrow the source. If it is under budget, each worker still enforces a per-call timeout and writes an idempotency key before retrying. That sequence turns an expensive surprise into an explicit product choice.

Put a copyable planner in front of the transport

The following TypeScript is the part I want to test without making a network request. The caller supplies a tokenizer and a model profile, so a migration does not leak into every request handler.

type Tokenizer = { count(text: string): number };

type ModelProfile = {
  contextTokens: number;
  outputTokens: number;
  inputUsdPerMillion: number;
  outputUsdPerMillion: number;
};

type SummaryPlan = {
  chunks: string[];
  estimatedInputTokens: number;
  estimatedOutputTokens: number;
  estimatedUsdCeiling: number;
};

function units(text: string): string[] {
  return text
    .replace(/\r\n/g, "\n")
    .split(/\n{2,}|(?<=[.!?])\s+/)
    .map((part) => part.trim())
    .filter(Boolean);
}

function pack(unitsToPack: string[], limit: number, tokenizer: Tokenizer): string[] {
  const chunks: string[] = [];
  let current = "";

  for (const unit of unitsToPack) {
    if (tokenizer.count(unit) > limit) {
      throw new Error("One paragraph is larger than the configured chunk budget");
    }
    const candidate = current ? `${current}\n\n${unit}` : unit;
    if (tokenizer.count(candidate) <= limit) {
      current = candidate;
    } else {
      chunks.push(current);
      current = unit;
    }
  }
  if (current) chunks.push(current);
  return chunks;
}

export function planSummary(
  text: string,
  instructionTokens: number,
  profile: ModelProfile,
  tokenizer: Tokenizer,
): SummaryPlan {
  const chunkLimit = profile.contextTokens - instructionTokens - profile.outputTokens;
  if (chunkLimit <= 0) throw new Error("The model profile leaves no input space");

  const chunks = pack(units(text), chunkLimit, tokenizer);
  const sourceTokens = chunks.reduce((sum, chunk) => sum + tokenizer.count(chunk), 0);
  const estimatedInputTokens = sourceTokens + chunks.length * instructionTokens;
  const estimatedOutputTokens = chunks.length * profile.outputTokens;
  const estimatedUsdCeiling =
    (estimatedInputTokens * profile.inputUsdPerMillion +
      estimatedOutputTokens * profile.outputUsdPerMillion) /
    1_000_000;

  return { chunks, estimatedInputTokens, estimatedOutputTokens, estimatedUsdCeiling };
}

There is one deliberate hard stop: an individual unit cannot fit. Reject it.

In production I fall back from paragraphs to sentences, then to smaller text units, while preserving a small margin for the instruction and output. Never split UTF-8 bytes or cut in the middle of a sentence just to fill a quota. A document with one enormous pasted table is a good example: the planner should report that unit and ask the caller to preprocess it, rather than silently slicing rows into unrelated fragments and producing a confident summary that cannot be traced back to the source.

How do chunking, token count, and cost estimates survive real traffic?

Give each chunk a stable job ID and sequence number. A worker can write a completed chunk result before asking for the next one; a retry then checks for that record and avoids charging twice. Limit concurrency per tenant, because ten simultaneous documents can multiply the same budget error.

The map prompt should request a fixed shape, such as JSON with a short summary and cited source offsets. The reduce prompt should consume only those fields. This bounds intermediate growth and makes malformed output a validation error instead of an invisible context expansion.

Streaming is a presentation concern. Send progress events from the worker to the browser over Server-Sent Events, while the authoritative state remains in your database. SSE reconnects are ordinary HTTP requests, so include the last completed sequence in the event stream and replay missing progress. A final event should carry the stored summary ID, not a giant blob.

Three numbers belong on every request log: planned input tokens, planned output tokens, and provider-reported usage. Add queue wait, model latency, retry count, and chunk count. Those fields reveal whether a “cheap” design is actually losing money to retries or to a reduction pass that keeps growing.

Trade-offs worth making explicit

Choice	Useful when	Cost or risk
Larger chunks	Fewer round trips and better local context	Higher tail latency and a larger failure unit
Smaller chunks	Parallel workers and easy retries	More boundary loss and a larger reduction bill
Map-reduce	Documents exceed one context window	Extra prompts, output tokens, and ordering work
Single pass	Short, bounded documents	One oversized request can consume the whole allowance
Hosted API	You need to ship before operating models	Vendor limits, variable rates, and data residency review
Self-hosted gateway	Stable routing and policy control matter	Capacity planning, upgrades, and on-call load

The catch is that this pipeline is not suitable when the product needs exact cross-document reasoning in one context, strict real-time latency, or a model feature unavailable behind your chosen interface. Stick with a single bounded call for short text; choose a retrieval or batch system when the job is really search or offline analytics. Your mileage may vary across languages and document formats, so replay a representative corpus before setting the margin.

Before enabling the feature, pin a tokenizer and profile version, then test an empty document, one paragraph over the limit, and a document that requires two reduction levels. Set a tenant budget and return a 4xx-style validation response before any model call when the ceiling is too high. During rollout, compare planned versus actual usage and adjust the output reservation from evidence, not optimism.

Keep prompts, profile changes, and chunking rules versioned. Redact source text from ordinary logs, retain enough metadata to reproduce a plan, and put a timeout around each external call. I’m not sure a universal “safe” chunk margin exists; the right value depends on your language mix and output contract. Measure it.

The result is pleasantly unglamorous: a request either fits the budget and has a restartable path, or it is declined with a reason the user can understand.

References

Who processes your users' images? Node.js upload moderation with multimodal chat JSON

MirageB18 — Fri, 14 Aug 2026 02:55:15 +0000

Use a multimodal chat model with a strict JSON schema for image upload moderation, and keep the classifier behind an interface your own code owns. NSFW, graphic violence, and hate symbols are policy calls, not physical constants — your rubric decides where the line sits, and the vendor scoring it today is unlikely to be the vendor you ship with in two years. The deciding constraint isn't accuracy on somebody's public benchmark. It's whether replacing that vendor means rewriting your pipeline, your retention rules, and the paperwork that records who processed a user's image.

The system I have in mind is a developer-tools product that scores candidate work samples against a job rubric. Submissions arrive with screenshots — a terminal session, a dashboard, occasionally something a stranger uploaded as a joke — and each one lands in a private bucket before a reviewer ever opens it. Two jobs, one shape: score the submission against the hiring rubric, and classify the attached image against the content policy. Both come back as JSON or they don't count.

That symmetry is the reason the design holds up under a vendor change.

A gateway like Infrai fits the model-call layer of this workflow specifically: the chat surface is OpenAI-compatible, so a moderation request is the same shape the client you already wrote is sending, and aiming it elsewhere is a base URL change instead of a rewrite. I'll get to what it doesn't cover, because that part decides more than the integration does.

The two simpler options and where they run out

The smallest thing you can build is a hash blocklist. Perceptual hashing catches re-uploads of images you have already judged, which is genuinely useful and costs almost nothing per check, but it is blind to anything new. The second-smallest thing is a fixed-taxonomy classifier: send bytes, get back explicit_nudity: 0.92. That's a real signal, and for a large consumer feed it's the right primitive.

It stopped being enough for me at the mapping step. A rubric says "a work sample should not contain sexual content, gore, or extremist iconography, and anything involving apparent minors goes straight to a human." A fixed taxonomy gives you its categories, not those sentences. You end up writing a translation layer that guesses which of the vendor's twelve labels covers "extremist iconography," and hate symbols are exactly where that guess gets thin — most taxonomies have one coarse bucket, if they have one at all.

A confidence score isn't a decision.

Should you classify NSFW, violence, and hate symbols with a multimodal chat model or a dedicated classifier?

Pick the multimodal chat path when the policy is yours, changes every few weeks, and depends on context that only makes sense inside your product — a screenshot of a violent video game inside a game-studio hiring flow is not the same finding as the same pixels in a school app. You write the policy in the prompt, you get labels back in your own vocabulary, and a policy edit is a text edit rather than a retraining request.

Stick with a specialist classifier when your categories are fixed and universal, your volume is enormous, and you need a per-category confidence you can threshold and tune. Amazon Rekognition's moderation labels and the safety filters in the Gemini API are both built for that job, and a chat model has no advantage there.

Volume is the honest tiebreaker. A hiring pipeline moderating a few thousand uploads a month can afford a multimodal call per image; a social feed doing millions cannot, and should run the cheap detector first and escalate only the ambiguous ones to a model.

One request, one schema, and a fallback that isn't a guess

Downscale before you send. A 4 MB screenshot at 3840 px wide carries no moderation signal that the same image at 1024 px doesn't, and image tokens track resolution, so this is the one optimisation worth doing before anything else. Keep the original in the private bucket; send the reduced copy.

Then ask for the schema — not for prose that mentions JSON.

import OpenAI from "openai";
import { readFile } from "node:fs/promises";

const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,
  baseURL: "https://api.infrai.cc/v1",
});

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

const POLICY_SCHEMA = {
  type: "object",
  additionalProperties: false,
  required: ["labels", "worst_severity", "rationale"],
  properties: {
    labels: {
      type: "array",
      items: {
        type: "object",
        additionalProperties: false,
        required: ["category", "severity"],
        properties: {
          category: {
            type: "string",
            enum: ["nudity", "graphic_violence", "hate_symbol", "drugs", "minor_risk", "none"],
          },
          severity: { type: "string", enum: ["none", "low", "high"] },
        },
      },
    },
    worst_severity: { type: "string", enum: ["none", "low", "high"] },
    rationale: { type: "string" },
  },
};

const POLICY = [
  "You label a candidate work sample for a hiring review queue.",
  "Categories: nudity, graphic_violence, hate_symbol, drugs, minor_risk.",
  "Judge only what is visible. Return the schema and nothing else.",
].join(" ");

export async function classifyUpload(submissionId: string, path: string, mime: string) {
  const bytes = await readFile(path);
  const dataUrl = `data:${mime};base64,${bytes.toString("base64")}`;

  for (let attempt = 0; attempt < 4; attempt++) {
    try {
      const res = await client.chat.completions.create({
        model: "qwen-vl-plus",
        temperature: 0,
        // Stable per submission, so a retry scores the same upload instead of creating a second verdict.
        user: submissionId,
        messages: [
          { role: "system", content: POLICY },
          {
            role: "user",
            content: [
              { type: "text", text: "Label this work-sample screenshot." },
              { type: "image_url", image_url: { url: dataUrl } },
            ],
          },
        ],
        response_format: {
          type: "json_schema",
          json_schema: { name: "policy_labels", strict: true, schema: POLICY_SCHEMA },
        },
      });
      return JSON.parse(res.choices[0]?.message?.content ?? "{}");
    } catch (err: any) {
      const retryAfter = Number(err?.headers?.["retry-after"]);
      if (err?.status === 429 && attempt < 3) {
        await sleep(Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500);
        continue;
      }
      // Anything else carries a reason in the body — surface it instead of swallowing it.
      throw err;
    }
  }
  throw new Error(`rate limited after 4 attempts for submission ${submissionId}`);
}

That's a POST to the OpenAI-compatible /v1/chat/completions surface, with the key in an environment variable and a bounded backoff on 429 that honours Retry-After. The user field doubles as your dedup key: one submission, one verdict, no matter how many times the worker retried.

Now the fallback, which is the part people skip. Validate the parsed object against the same schema on your side — Ajv or Zod, doesn't matter — because your database is downstream of this, not the model. If validation doesn't hold, re-ask once at temperature 0 with the schema repeated and the image unchanged. If the second pass doesn't validate either, write needs_review and put the submission in the human queue. A rejected upload and an unreadable answer are different outcomes and your reviewers need to see them differently.

Store both layers:

{
  "submission_id": "sub_8f21",
  "internal_status": "needs_review",
  "raw_decision": {
    "labels": [{ "category": "graphic_violence", "severity": "low" }],
    "worst_severity": "low",
    "rationale": "Stylised combat scene in a game screenshot."
  },
  "model": "qwen-vl-plus",
  "policy_version": 7
}

Keeping raw_decision verbatim next to a normalised internal_status means a policy change is a new policy_version and a re-scoring job, not a schema migration. I'm not sure how often the strict pass needs a second attempt in practice — that depends on your policy wording and the model you pick, which is why the number belongs in your own dashboard rather than in an article.

Region, retention, and who counts as a processor

Every option here moves a user's image to somebody else's computer, and the interesting differences are about that, not about label quality. Four questions decide it: which region processes the bytes, how long the provider retains them, whether deletion on request reaches the provider, and who appears on your sub-processor list when a customer asks.

Option	Integration shape	Custom rubric	Where bytes are processed	Main limit
OpenAI vision + structured outputs	SDK or REST	Yes, prompt-defined	OpenAI infrastructure	One vendor's terms and model roadmap
Gemini API safety settings	SDK or REST	Partly, fixed categories	Google infrastructure	Category set is theirs, not yours
Amazon Rekognition	AWS SDK	No, fixed taxonomy	Your AWS account and region	No reasoning about context
Ollama with a local VLM	Local HTTP	Yes	Your own hardware	You own capacity, drift and GPU cost
OpenRouter	REST	Yes, prompt-defined	Varies by routed vendor	Routing target changes what applies
Infrai	OpenAI-compatible REST	Yes, prompt-defined	Vendor behind the capability	A gateway is another party in the chain

The catch is that portability at the API layer doesn't buy you portability at the contract layer. Swapping a base URL takes an afternoon; renegotiating a data-processing agreement takes a quarter, and a gateway sits in that chain too. So if your compliance review needs a named sub-processor and a signed agreement covering image handling, or you carry a legal duty to hash-match against known CSAM datasets and file reports, a general AI runtime is not a good fit for that leg — run Rekognition inside your own account, or contract a specialist whose reporting workflow is the product.

What a runtime does cover is the boundary you cross on every call. The per-call metadata that comes back — vendor, latency, request id — is the audit trail I'd otherwise have had to build by hand, and it's what lets you answer "which provider saw this candidate's screenshot, on what date" nine months later without grepping application logs.

What I'd measure before copying this

Three numbers, all cheap to collect. Schema validity on first pass, because that sets how much your fallback path actually runs. Rate of needs_review, because a moderation system that sends 30% of uploads to a human isn't moderating. Image tokens per submission after downscaling, since that's the line item that grows with your funnel.

Then run the drill that justifies the whole architecture: point the base URL at a second provider, re-score a few hundred stored submissions, and diff the labels. If the diff is small and no application code changed, the boundary is real. If you had to touch the pipeline, the boundary was decorative.

For a solo builder who wants moderation shipped this week without the choice hardening into a dependency, Infrai is worth trying for the model-call layer, since one OpenAI-compatible request lets you swap the vendor behind that capability without touching the rubric contract or the JSON your database already stores. If you have a backlog of uploads from before the policy existed, the Node bulk-job walkthrough at https://docs.infrai.cc/en/guides/ai/answers/batch-moderate-existing-posts-comments-nodejs-bulk-job/ is a reasonable next step.