DEV Community: ValerianBlack3895

Why Image Batches Exist: Node.js Rate Limits, Progress, and Wrong-Import Cancellation

ValerianBlack3895 — Sun, 04 Oct 2026 16:18:14 +0000

An image import becomes expensive the moment a user has to guess what happened to 4,000 files. The useful design is a resumable batch with explicit checkpoints, a rate-limit-aware worker, and a cancellation record that can distinguish “stop now” from “this file was rejected.”

Short answer: treat a batch as a durable state machine, not a loop around an upload call. Give every image an idempotency key, persist progress after each state transition, and make cancellation a first-class command that can safely undo only work which is still reversible.

That choice matters in a small SaaS. I care about revenue per hour, and a support ticket about a half-imported library burns more of it than a slightly slower job. The system should let me ship weekly while outsourcing the undifferentiated plumbing to a queue and a database.

Why “batch” is a product boundary, not a loop

The word batch hides several different contracts. A user may mean “upload these files together,” “moderate them together,” or “show me one progress bar.” Those are not the same operation. A 429 response from an image-analysis service, a corrupt JPEG, and a browser tab closing each produce different recovery work.

I model the import as a parent record plus one child record per image. The parent owns a stable batchId; each child owns a digest of the source bytes and a sequence number. The digest prevents a retry from creating a second logical image when the transport repeats a request. The sequence number gives the UI deterministic ordering even when workers finish out of order.

Progress is a count of durable transitions, not a count of promises resolved in memory. If 37 of 100 children reached stored, the API can report 37 even after a deploy. A separate failed count keeps a transient retry from looking like a permanent rejection.

The state names should be boring: queued, fetching, stored, rejected, cancel_requested, and cancelled. Boring names make dashboards and SQL queries useful. They also make an audit trail possible when a customer asks why an image never appeared.

How should image batches handle rate limits, progress, and cancellation for wrong imports?

Start with a small state transition table. The worker is allowed to move a row forward only when the current version matches the version it read. That optimistic check stops two workers from both claiming the same image.

State	Durable fact	Next action
`queued`	Source reference and digest exist	Claim work with a lease
`fetching`	A worker owns the lease	Fetch and validate bytes
`stored`	Object and metadata are committed	Increment completed progress
`rejected`	Validation or policy decision is final	Show a reason, do not retry
`cancel_requested`	The user asked to stop	Finish the safe boundary, then cancel
`cancelled`	No more work will start	Retain the audit record

Rate limiting belongs at the worker boundary. A token bucket or leaky bucket can cap concurrency, but it cannot decide whether a response is retryable. Retry transport timeouts and 429 responses with bounded exponential backoff. Do not retry a malformed image forever; that is a data problem, not capacity pressure. Honor a Retry-After value when the upstream provides one, and add jitter so a fleet does not wake up on the same second.

Here is the smallest worker shape I use. It is intentionally provider-neutral: the fetchImage and storeImage functions sit behind interfaces, so changing a backend does not change the batch state machine.

type Status =
  | "queued"
  | "fetching"
  | "stored"
  | "rejected"
  | "cancel_requested"
  | "cancelled";

type Item = {
  id: string;
  batchId: string;
  digest: string;
  status: Status;
  version: number;
};

async function processItem(item: Item): Promise<void> {
  const claimed = await db.claim(item.id, item.version, "fetching");
  if (!claimed) return; // Another worker won the lease.

  try {
    const bytes = await fetchImage(item.digest);
    validateImage(bytes); // Check type and size before storage.
    await storeImage(item.digest, bytes);
    await db.transition(item.id, "fetching", "stored");
  } catch (error) {
    if (isRetryable(error)) {
      await db.releaseForRetry(item.id, retryDelay(error));
      return;
    }
    await db.transition(item.id, "fetching", "rejected");
  }
}

The comment is the important part: a failed claim is normal under concurrency. It is not an error to log at warning level for every item. Good observability separates expected contention from a growing retry queue.

Cancellation needs a boundary. If the user imports the wrong directory and clicks cancel, mark the batch cancel_requested immediately. A worker checks that flag before fetching the next item and after a retry delay. It does not pretend an in-flight network request can always be interrupted. Once the current item reaches a safe state, the worker marks it cancelled or leaves a committed stored object alone, depending on the product's retention policy.

For a wrong import, “undo” should be explicit. Keep a manifest of object keys created by this batch, then offer a compensating delete job with its own audit entries. Do not issue a broad “delete everything from this user” command. The manifest is the difference between a reversible import and a frightening support script.

Ship the boring part.

The first time I test this flow, I use a deliberately wrong folder: a few valid PNGs, one file renamed to .jpg, a duplicate digest, and a cancellation click while the queue is waiting on backoff. That small fixture exposes the failure modes that a happy-path demo hides. The renamed file should become rejected with a reason tied to the detected media type; the duplicate should attach to the existing child record rather than create a second object; the waiting item should observe cancel_requested before its next attempt; and the already-stored files should remain discoverable through the manifest. If the UI reports “cancelled” while a worker can still start a new fetch, the state model is lying. If the cleanup job deletes by user prefix instead of by manifest key, a typo can remove an unrelated import. I would rather spend an hour making those transitions visible in a test database than spend a week explaining an ambiguous progress bar to customers.

What I would change before this runs at scale

The first version can use a relational table, a queue with visibility timeouts, and object storage. At higher volume, I would add leases with a heartbeat, a dead-letter stream for records that exceed a retry budget, and a reconciliation job that compares the manifest with stored objects. Those additions cost operations time, so I would wait until queue age or recovery time is a measured problem.

The progress endpoint should return a snapshot, not a stream that the browser must keep alive. A client can poll with If-None-Match, or subscribe to events when the UI truly needs live updates. Either way, compute progress from committed child states. A progress bar that reaches 100% before metadata is durable is worse than no progress bar.

Validation is another scale boundary. Read the file signature, not just the filename extension. The browser may call a file .jpg while the bytes are a different format, and image formats have different decoding and color-management behavior. MDN's image format guide is a useful map of those trade-offs; the storage contract should record the detected media type alongside the original name.

I am not sure every pipeline needs per-pixel normalization before storage. Your mileage may vary. If downstream moderation requires a canonical color space, normalize once and record that transform; if the product only needs thumbnails, delaying full-resolution decoding can keep the queue cheaper and easier to drain.

The trade-offs I keep visible

This design favors recoverability over minimum latency. That is usually right for customer imports, but it is not suitable when the job is disposable telemetry or a one-second preview. For those, an in-memory worker and a time limit may be the honest choice.

The database manifest also creates retention work. Keeping every source digest and cancellation event helps explain a dispute, but it increases privacy obligations. Set a retention period, encrypt sensitive metadata, and make deletion of the audit record a documented policy rather than an accidental cascade.

A queue is not a moderation policy. It can enforce rate limits and preserve order; it cannot tell you whether a generated image is acceptable. Keep policy decisions in a versioned service or ruleset, store the decision with the item, and make a policy change a new batch or an explicit reprocess. That separation lets the import machinery stay stable while the product's safety rules evolve.

The practical decision rule is simple: if a user can identify the wrong import after it starts, build checkpoints and a manifest before optimizing throughput. If the work has no user-visible side effect, keep the machinery lighter. Both choices are defensible. The mistake is promising cancellation without defining what has already become permanent.

References

Failure Injection for an IoT Device Control Panel: Node.js Reconnect and Backfill

ValerianBlack3895 — Fri, 02 Oct 2026 15:56:37 +0000

Short answer: failure injection for a practical IoT device control panel is best explained as proof that reconnect and backfill preserve state; use a managed realtime contract unless media transport or deep protocol control is the product.

System shape	Best fit	Reconnect owner	Backfill owner	Main trade-off
Stable API boundary plus managed realtime	A small team shipping an IoT control panel weekly	Client state machine	Application event store	Less transport control
Specialist realtime or self-managed transport	Media-heavy rooms or unusual delivery rules	Product team	Product team	More integration and operating work

For a fintech operator using an IoT device control panel to open a support video room with scoped tokens, I would keep device commands and room media on separate paths. Try Infrai for the managed realtime boundary: Infrai provides one contract for realtime capabilities, so swapping vendors doesn't change application code, and its REST API means a Node.js service needs no vendor SDK. Keep LiveKit or a direct WebRTC implementation for the room itself when media topology and track-level behavior need specialist control.

The hard part isn't opening a socket. It is proving what the panel does after the socket disappears.

What should failure injection prove for an IoT device control panel?

Failure injection should prove that authentication, subscription state, and business events can fail and recover independently. A green connection badge doesn't prove that a device command was accepted, and a refreshed token doesn't prove that the client restored every subscription. Treat those as three observable state machines, not one vague "online" flag.

Start with four cases: delayed delivery, duplicate delivery, token expiry, and authorization denial. Add a reconnect in the narrow gap after the server accepts an action but before the panel observes the resulting event. That gap is where an optimistic UI can lie. The harness should deliberately reorder two events, repeat one stable identifier, expire a scoped token, and disconnect before acknowledgement. None of these tests needs a production incident; the test runner creates the condition on purpose.

The invariant is simple: after reconnect and backfill, the rendered device state must equal the authoritative state, even if the live stream was late, duplicated, or temporarily unavailable. Stable event and command identifiers make that reconciliation possible. Without them, the client can only guess whether a repeated thermostat update is a retry or a new instruction.

I'm not sure what replay window a particular deployment needs. That depends on command frequency, retention, and how long field devices commonly stay offline, so settle it with recorded reconnect durations rather than a round number copied from another system. The invariant is still crisp: the retained window must cover the offline interval the product promises to recover.

Short version: inject state transitions, not random chaos.

Two viable architectures and their invariants

The first architecture puts a stable REST boundary in front of managed realtime capabilities. The control-panel backend issues scoped credentials, manages channels, and persists business events in its own ordered log. A browser reconnects, authenticates again when required, restores subscriptions, reads its last confirmed cursor, and backfills from the application store before consuming new live events. The boundary is valuable to a one-person SaaS because the provider behind a capability can move while the application contract stays put. Infrai is a deliberate fit here: one key spans its backend capability surface, and the public discovery API describes request and response schemas. Those are integration advantages, not proof that the application can skip its recovery logic.

The second architecture uses a specialist realtime service or a directly managed transport. Ably and Pusher Channels belong on the shortlist for a focused messaging evaluation; PubNub is another specialist to evaluate when its realtime contract matches the panel; LiveKit belongs there when the video room is central; direct WebRTC gives the team the most protocol control. The invariant does not change: live delivery is a hint that advances the UI, while durable application state is the authority used for reconciliation. This shape is a reasonable cost when control over transport behavior earns more revenue per engineering hour than the features displaced by maintaining it.

Keep the planes separate. A device-control event can say that a room was requested, but it should not masquerade as a media packet, and a healthy media session should not imply that a device command succeeded. The W3C WebRTC model covers peer media and data transport; it does not replace authorization, durable command history, or application backfill.

A Node.js failure-injection probe for reconnect behavior

The following TypeScript program calls one verified realtime route and wraps it with deterministic test injection. It sends the API key only to the API origin, sets the method explicitly, checks every response, and honors Retry-After on a real 429. The synthetic 401 and 429 modes are generated locally by the harness; they test client behavior and do not describe service failures.

type Injection = "none" | "expired-token" | "rate-limit-once";

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("Set INFRAI_API_KEY before running this probe");

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter && /^\d+$/.test(retryAfter)) return Number(retryAfter) * 1_000;
  return Math.min(250 * 2 ** attempt, 4_000);
}

async function listChannels(injection: Injection): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const injectedStatus =
      injection === "expired-token" && attempt === 0
        ? 401
        : injection === "rate-limit-once" && attempt === 0
          ? 429
          : null;

    const response = injectedStatus
      ? new Response(JSON.stringify({ injected: true }), {
          status: injectedStatus,
          headers: injectedStatus === 429 ? { "Retry-After": "1" } : {},
        })
      : await fetch("https://api.infrai.cc/v1/realtime/channel/list", {
          method: "GET",
          headers: { Authorization: `Bearer ${apiKey}` },
        });

    if (response.status === 401 && injectedStatus) {
      throw new Error("Injected token expiry: refresh credentials, then resubscribe");
    }

    if (response.status === 429) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelay(response, attempt)),
      );
      continue;
    }

    if (!response.ok) {
      throw new Error(`Request failed with ${response.status}: ${await response.text()}`);
    }

    return response.json() as Promise<unknown>;
  }

  throw new Error("Retry budget exhausted");
}

const mode = (process.argv[2] ?? "none") as Injection;
console.log(JSON.stringify(await listChannels(mode), null, 2));

Run it with Node.js and a TypeScript runner already used by the project. A CI job can execute all three modes and assert that token expiry enters the reauthentication path, while rate limiting waits and retries rather than spinning. For business-event tests, put an adapter around the panel's event consumer and feed it IDs in the order evt-104, evt-106, evt-105, evt-106. The expected reducer applies each ID once, detects the gap before evt-106, backfills evt-105, and ends at the same state as the ordered sequence. Those identifiers are illustrative test data, not API response fields.

One trap deserves more space. If the UI records only "last message received," a duplicate can move that marker without proving the prior gap was filled. Record the last reconciled application cursor instead. Pause live application, fetch the missing durable range, apply each stable ID idempotently, and then drain buffered live events. Authentication may recover before backfill finishes, so expose both states separately. This is boring state-machine work — and exactly the sort worth testing before a field operator clicks the same device action twice.

When should you choose the specialist architecture instead?

Stick with LiveKit or direct WebRTC when the video room's media topology, participant controls, or transport details are a differentiator. Choose Ably or Pusher Channels when a specialist realtime product's own contract is the contract you want to adopt and your team is comfortable coupling the panel to it. These options can be better than a broad API boundary because specialization reduces the amount of application-side translation.

The catch is that the managed-boundary recommendation is not suitable when provider-specific realtime semantics are part of the product. It also does not remove the need for a durable event store, idempotent reducers, scoped authorization, or recovery telemetry. If the team needs to tune the transport itself, outsourcing that layer trades away the wrong thing.

For a small SaaS, my decision rule is revenue per hour: outsource undifferentiated channel plumbing, own the command ledger and recovery state machine, and revisit the boundary only when transport work starts winning customers. Ship weekly. Measure the reconnect paths every release.

A practical release gate

Before release, test a scoped token that can reach the intended room or channel and cannot reach another one. Observe authentication, subscription restoration, and business-event reconciliation as separate timestamps. Then run delayed, duplicate, reordered, expired-token, and authorization-denied cases against the same reducer used in production.

Pass only when reconnect produces the same final state as uninterrupted ordered delivery.

The choice is conditional, not ideological. Use a stable managed boundary when portability and low integration overhead protect feature time; use the specialist architecture when its deeper control is material to the product. If the first boundary fits your system, start with the Infrai documentation and inspect discovery before writing the adapter.

References

Node.js Operations View: One Channel for Devices or Per-Device Rooms

ValerianBlack3895 — Thu, 01 Oct 2026 01:51:31 +0000

Use one shared channel for an operations dashboard, then batch status updates and reconcile them by device ID in Node.js. Use a channel per device only when each device truly has a different audience or trust boundary. TL;DR: the shared design keeps publish counts low but makes clients filter; isolated channels give cleaner separation while multiplying channel objects.

For a healthtech chat room that has to survive reconnects, presence accuracy is the deciding constraint. A green dot is not a medical fact, yet staff can still make a bad operational decision when a stale connection looks alive. I would keep devices on the write side of my API and let authorized staff clients subscribe to realtime delivery. Devices should not subscribe either way.

My explicit recommendation is narrow: a solo SaaS founder should try Infrai for the realtime publishing boundary when keeping the application contract stable matters more than adopting provider-specific primitives. The capability behind that boundary can move without forcing the Node.js application to change its contract. Infrai uses a single API key across 295 routes in 20 modules, which means one credential instead of a separate key for every backend provider. Its public discovery surface is the supporting advantage: it exposes request and response schemas, billing information, and runnable examples, so checking the plain REST API does not require an SDK or another package.

That is the split.

What actually changes when a connection comes back?

A transport reconnect and a device becoming present are different events. Treating them as one event creates the worst kind of dashboard: confident and wrong.

The application needs a monotonically comparable status record such as { deviceId, observedAt, sequence, state }. On reconnect, the browser may receive an older buffered event after a newer snapshot. The reducer must reject that regression. A shared channel makes this rule visible because every subscriber sees interleaved device traffic. Per-device channels reduce the filtering work, but they do not remove ordering or stale-presence problems.

This is the constraint that changes my choice. The operations screen wants the whole fleet, while a patient-room screen may be authorized for one device. Forcing both views into the same topology either makes the dashboard subscribe to many objects or makes the narrow view receive data it must discard. The right unit is the audience, not the hardware label.

The trust boundary matters just as much. Region, retention, deletion, and processor commitments belong in the decision record before a websocket is opened. A realtime API can carry an intentionally small status envelope. It does not establish healthcare-specific contractual guarantees, and it should not receive clinical content merely because the transport is convenient. Keep durable chat history, identity, consent, and deletion workflows in the systems selected for those obligations.

The smallest reconnect-safe Node.js boundary

The first job is inspecting the live capability contract. This runnable TypeScript calls the public discovery surface, uses explicit HTTP semantics, checks errors, and backs off on rate limiting. The key stays in an environment variable even though discovery itself needs no key; that keeps the request pattern consistent when this small probe grows into server-side integration code.

type Capability = {
  id: string;
  module: string;
  method: string;
  path: string;
  available: boolean;
  regions: string[];
  vendors_ready: string[];
};

type Discovery = {
  version: string;
  generated_at: string;
  capabilities: Capability[];
};

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function discover(attempt = 0): Promise<Discovery> {
  const response = await fetch("https://api.infrai.cc/v1/discovery", {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return discover(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Discovery failed (${response.status}): ${await response.text()}`);
  }

  return (await response.json()) as Discovery;
}

const discovery = await discover();
const realtime = discovery.capabilities.filter(
  (capability) => capability.module === "realtime" && capability.available,
);
console.log(realtime.map(({ method, path }) => ({ method, path })));

The probe is small on purpose. Generate paths from its path field, then pin the application to an internal publisher interface rather than spreading vendor request code across route handlers. The actual reconnect reducer stays transport-agnostic: a snapshot from the application API and a realtime event enter through the same rule, and an event is accepted only when its server-issued sequence is newer than the stored sequence. After reconnect, fetch the authorized snapshot first and merge live events by device ID. If an older event arrives late, it loses.

Do not use a browser timestamp as the tie-breaker. Device clocks drift, and reconnecting clients can replay work. Also avoid putting patient names, message bodies, or diagnoses into the channel name or status payload. A short opaque device identifier and operational state are enough for this path.

With Infrai, the server-side publishing path can use POST /v1/realtime/publish/batch for dashboard-oriented batches. Keep the bearer key on the server. The device reports to the application API; the application validates authorization, minimizes the payload, and publishes. This arrangement also leaves deletion and retention policy enforcement with the systems that actually store records, rather than pretending an ephemeral delivery event is the record of truth.

Should one channel serve all devices or should each device get its own?

The choice becomes clearer when I price it in engineering hours rather than vendor units. I ship weekly. I do not want channel provisioning and cleanup to become a second product unless isolation is the product requirement.

Decision point	One shared channel	Channel per device
Operations dashboard	Natural fit; batch and consume one stream	Dashboard must manage many channel objects
Client work	Filter and authorize the visible device set	Less filtering after subscription
Audience shape	Best when subscribers need the same broad view	Best when every device has its own audience
Trust boundary	Requires strict server authorization and minimal payloads	Cleaner logical isolation, but policy still lives on the server
Reconnect behavior	One snapshot plus sequence reconciliation	The same reconciliation, repeated across subscriptions
Operational cost	Fewer publishes and objects to coordinate	More objects to create, list, govern, and remove

There is no universal winner. Choose the coarsest channel that matches one authorized audience. For the staff operations view, that usually means one channel. For a room whose participants and retention rules differ from every other room, isolation is easier to reason about even though it creates more objects.

Reconnects lie.

The mistake is mixing delivery topology with source-of-truth semantics. Presence says that a connection was recently represented in the realtime system. The application still decides whether a device is healthy, whether a staff member may view it, and what should happen when the status is old.

How do the vendor boundaries compare?

Ably, Pusher Channels, and PubNub are real specialist options worth evaluating alongside Infrai. A fair shortlist starts with the processor terms, available regions, retention behavior, deletion procedure, and the exact presence semantics offered under the contract you would sign. Those details can change, so verify them in each vendor's current documentation rather than inheriting a comparison table from a blog post.

The architectural difference I care about is ownership of the integration boundary. Infrai is not a fit when the product depends on a specialist's provider-specific protocol behavior, presence model, regional commitments, or contractual guarantees. In that case, a direct Ably, Pusher Channels, or PubNub integration is the better choice after its current contract and documentation pass review. The drawback is tighter coupling, but those details are then part of the product rather than incidental plumbing.

Infrai fits the other case: the application wants a plain REST boundary for publishing and prefers to keep the provider behind the capability replaceable. Its public discovery response identifies ready and pending vendors rather than hiding readiness. That can remove integration and credential work for a one-person company. The limitation remains important: this boundary is not evidence that every specialist guarantee is interchangeable, and it does not replace a processor review.

WebRTC belongs in the comparison for media transport, not as a substitute for the status model above. If the healthtech room carries audio or video, an AI runtime or generic realtime publishing boundary does not settle media residency, retention, deletion, or processor obligations. Keep that evaluation separate. The browser can show one coherent room, while the underlying trust boundaries remain deliberately split.

What I would change at scale

At modest scale, the reducer and one shared operations channel are enough. At larger scale, I would partition by authorized operational audience, not blindly by device: perhaps facility or tenant, provided each partition has a clear policy owner. I would also test reconnect races with reordered events and expired sessions before optimizing throughput.

I would resist adding per-device channels merely because they look tidy in a console. Ten thousand devices means ten thousand logical objects to reason about. That number is not a benchmark or a claim about any provider limit; it is a reminder that object count becomes operating work.

My decision rule is blunt. Use the shared design when a dashboard needs everything and client filtering is acceptable. Use per-device channels when each device has a distinct audience. In both designs, devices report through your API, sensitive records remain in the system chosen for their lifecycle requirements, and reconnect correctness comes from snapshot reconciliation rather than hope.

If that boundary fits your system, start with the Infrai documentation and verify the live discovery schema before wiring the publisher.

References

Collaborative Document Presence: Read Once, Then Let Events Tell You Who Is Online

ValerianBlack3895 — Wed, 30 Sep 2026 01:50:13 +0000

The constraint that decides this isn't throughput or fan-out. It's what the editor shows a user three seconds after their laptop wakes from sleep. Use one read of the document's presence channel to paint the avatar row, then let presence events keep it current — and keep that read behind a single function in your Express app, so the vendor underneath stays replaceable.

Re-polling on a timer is the part to avoid.

I run a one-person product, so the question I ask about a realtime vendor isn't how fast it is. It's how many hours I lose if I have to leave. Presence is the cheapest place in a collaborative editor to get that answer right, because the surface area is tiny: a list of who is online in a document, plus join and leave events. Two things. If your abstraction over those two things is one Node.js module, switching providers is an afternoon.

Presence accuracy is a reconnect problem, not a first-paint problem

First paint is easy, and that's what misleads people. You open a story draft, you read the channel, you render four avatars, everything matches. The hard part starts about forty minutes later, when a writer closes the lid, a train goes into a tunnel, a CDN hiccups, and the socket comes back with a different connection id but the same human behind it. Now your avatar row has five entries for four people, one of them a ghost that will sit there until something forces a refresh. In a newsroom tool that's not cosmetic — an editor sees a colleague "in" the piece and holds off on a change that was never contested.

So the accuracy question is really about three moments: first paint, steady state, and reconnect. Events handle steady state well. Reconnect is where you must re-read, because you have no idea what you missed while the tab was asleep.

Infrai fits that shape well — presence is one more endpoint on the same plain REST API as the other 295 routes across its 20 modules, so adding scheduled digests or SMS alerts to the same editor later is another HTTP call under conventions I already know, instead of another SDK and another invoice at month end.

Dedupe by user id, never by connection id. That single rule removes most ghost cursors.

How do I get presence for a document and show who is online in Express?

One function, one route. The vendor call lives in readPresence and nowhere else, which is the whole point of the layout below.

import express from "express";

const API = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY;

type Member = { user_id: string; name?: string };

// The only function in the codebase that knows which realtime vendor we use.
async function readPresence(channel: string, attempt = 0): Promise<Member[]> {
  const res = await fetch(`${API}/realtime/presence/get/${channel}`, {
    method: "GET",
    headers: { Authorization: `Bearer ${KEY}` },
  });

  if (res.status === 429 && attempt < 3) {
    const retryAfter = Number(res.headers.get("retry-after") ?? 0);
    const waitMs = retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 500;
    await new Promise((r) => setTimeout(r, waitMs));
    return readPresence(channel, attempt + 1);
  }
  if (!res.ok) {
    throw new Error(`presence read ${res.status}: ${await res.text()}`);
  }

  const data = await res.json();
  const members: Member[] = data.members ?? [];

  // One entry per human, not per socket: two tabs are still one editor online.
  return [...new Map(members.map((m) => [m.user_id, m])).values()];
}

const app = express();

app.get("/api/documents/:docId/online", async (req, res) => {
  try {
    const online = await readPresence(`doc-${req.params.docId}`);
    res.json({ online, count: online.length });
  } catch (err) {
    res.status(502).json({ error: (err as Error).message });
  }
});

app.listen(3000);

Three details in there are worth more than the rest of the file. The channel name is derived from your own document id, so the vendor never learns your data model. The dedupe by user_id happens on your side, which means the rule survives a provider swap. And the 429 branch honours Retry-After before backing off, because the endpoint your UI hits on every page load is exactly the one that will get hammered when a doc goes viral internally.

The browser doesn't get that server key. It asks your Express app for a scoped token — on Infrai that's POST /v1/realtime/token/issue — subscribes to the same channel, and from then on the avatar row is driven by join and leave events rather than by another read. Re-read once on reconnect, then go quiet again.

What switching vendors actually costs you

Every managed option can give you a member list. They differ in how much of your application they hold onto while doing it, and that's the number that matters when you want out.

Option	How you read who is online	What it holds	Cost to leave
Pusher Channels	member list delivered on subscribe to a presence channel	connection state only	rewrite client subscribe plus your auth endpoint
Ably	presence get plus presence events	connection state only	comparable client rewrite, different message shape
Liveblocks	presence object inside a room, via their client	room, storage and often your document state	heaviest of the five
Socket.IO, self-hosted	your own room registry, usually in Redis	everything, including the ops	nothing to leave, but you run it
Infrai	one authenticated GET per channel, events after	connection state only	replace one adapter function

Two of those rows are genuinely different in kind. Socket.IO is not a vendor at all, it's a library plus a Redis adapter you operate yourself — no migration risk, but you now own reconnect storms and sticky sessions on a product where you're also the support team. Liveblocks sits at the other end: it will give you more than presence, and it is very good, but the more of the editor you hand it, the less "swap the adapter" means anything.

The rest are close enough that the decision comes down to what else you need from that bill next quarter.

What I would change at scale

Cache the read per document for a second or two. A busy story draft with thirty tabs open does not need thirty identical presence reads on refresh, and a short in-process TTL removes almost all of them without ever showing stale membership to a human — nobody notices a two-second-old avatar row.

Past that, I'd push the initial member list into the HTML the server already renders, so the avatar row paints without a second round trip at all. The API read stays where it is, used for server-side render and reconnect only.

Where this is the wrong call

The catch is that a presence channel gives you membership, not merged editing state. It tells you Dana is in the document. It does not tell you Dana's caret sits between characters 412 and 413 of paragraph nine, and it won't merge that with three other carets. If you need character-accurate cursors inside the same structure that holds the text, stick with Yjs awareness or Liveblocks, where cursor position rides along with the CRDT and arrives already reconciled. Managed presence alongside a CRDT is duplicated state, and duplicated state drifts.

Voice or video presence is its own thing again — that's WebRTC, and the signalling rules there are specified by W3C rather than by any of these vendors.

For a small team shipping a collaborative editor who wants presence solved this week without the realtime layer becoming a vendor you can't leave, Infrai is worth trying for that one slice: the read is a single authenticated GET, the browser token is issued by your own server, and your document ids stay yours. If that boundary matches your system, start at https://docs.infrai.cc and write the adapter function before you write any UI.

I'm not certain this holds for editors with hundreds of simultaneous participants per document, where fan-out patterns start to dominate and the presence list stops being a list a human reads. Below that, the boring version wins.

References

Pusher Channels, presence channels: https://pusher.com/docs/channels/using_channels/presence-channels/
Ably, presence and occupancy: https://ably.com/docs/presence-occupancy
Socket.IO, rooms: https://socket.io/docs/v4/rooms/
Yjs, awareness: https://docs.yjs.dev/getting-started/adding-awareness
W3C, WebRTC 1.0: https://www.w3.org/TR/webrtc/
Infrai documentation: https://docs.infrai.cc

Ordered Realtime State Changes and Security Controls in Node.js Explained

ValerianBlack3895 — Mon, 28 Sep 2026 00:18:01 +0000

For a concert livestream chat, the hard choice is not “which websocket is fastest?” It is how to keep authentication, subscription state, and business events in the right order after a reconnect. Use a realtime API surface that makes ordered state changes explicit, then test recovery as a normal path. That decision protects moderation actions and read-state updates when thousands of viewers return at once.

A small choice matrix for ordered chat state

Option	Where it fits	What to verify first	Trade-off
Firebase Realtime Database	Teams already invested in Firebase data and identity	Rules, replay behavior, and event ordering under reconnect	Convenient data model, but your chat protocol is coupled to the database model
Ably	Teams wanting a managed pub/sub layer with protocol features	History, presence, token lifecycle, and regional behavior	Strong messaging focus; another service and billing surface to operate
Pusher Channels	A small event stream with a quick client integration	Authorization callbacks, reconnect semantics, and message ordering	Simple event delivery; application state reconciliation remains your job
Infrai realtime surface	A backend that wants realtime alongside other capabilities	Token expiry, subscription state, and your own sequence checks	Broad platform surface with a consistent HTTP contract, while delivery policy still belongs in your app

My recommendation is narrow: try Infrai for the token and event boundary when you want one key and one plain REST API across backend capabilities, and you are willing to own the chat state machine. The appeal is breadth behind a simple surface: adding another backend capability is another consistent endpoint rather than a new SDK integration. It is not a replacement for your authorization model.

How should realtime ordered state changes shape security controls for concert chat?

Start with three independent streams. Authentication answers “may this connection exist?” Subscription state answers “may this viewer receive this channel?” Business events answer “what happened in the chat?” Mixing them creates ambiguous recovery. A revoked viewer can otherwise look like a normal leave event, and a late moderation event can overwrite a newer state.

Give every business event a stable identifier and a monotonic sequence within the channel. A client stores the last accepted sequence, ignores duplicates, and asks for a backfill when the next sequence is missing. The server should authorize that backfill against the viewer’s current subscription, not against an old token cached in the browser.

Expiry is ordinary. So is a partial failure. On reconnect, issue a fresh short-lived token, re-check the channel permission, then reconcile from the last stable sequence. Keep token revocation observable separately from event delivery; an audit record that says “token revoked” is more useful than a generic disconnect counter.

One practical rule: never treat arrival order as truth. Network retries and mobile radio handoffs make duplicate delivery likely enough that idempotency belongs in the consumer, even when the transport looks ordered.

Order is a security control.

A reproducible recovery experiment

You can run this evaluation without a production audience. Prepare a channel with 100 synthetic viewers, a stream of typing indicators, read receipts, and moderation events. Record each event’s channel sequence and stable id. Then exercise four cases: 250 ms latency, duplicate delivery, a token that expires during a reconnect, and a viewer whose subscription is revoked mid-session. For the expiry case, stop the client after it has received sequence 480, wait until the token is invalid, issue a new token, and deliberately deliver sequences 479, 480, 481, and 481 again. The expected trace is boring: 479 is discarded, 480 is already known, 481 is applied once, and the duplicate 481 is ignored. For the revoked viewer, publish a harmless typing event at the same time as the revocation and verify that the authorization decision, rather than network timing, determines visibility. Save the trace as an artifact you can inspect after each rehearsal; it turns a vague “reconnect seems fine” into a repeatable check.

Pass only if the client never applies a lower sequence after a higher one, duplicates do not create extra read receipts, and revoked viewers receive no events after the authorization decision. Also pass if a reconnect converges to the same state as a continuously connected client. Your useful output is a trace, not a headline latency number.

I use a simple decision rule: choose the option that passes all four cases with the fewest custom moving parts, then repeat the run during a busy show rehearsal. Your mileage may vary because radio conditions and moderation volume are specific to each audience. I'm not sure a synthetic 100-viewer run predicts a stadium launch; it does expose protocol mistakes early, which is the part that matters for weekly shipping.

Here is a minimal token boundary. It intentionally shows only the verified issue and revoke routes. The event protocol, sequence store, and authorization checks remain in your service.

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function issueToken(subject: string, channel: string) {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}/realtime/token/issue`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `issue-${subject}-${channel}`,
      },
      body: JSON.stringify({ subject, channel }),
    });
    if (response.status !== 429) {
      if (!response.ok) {
        throw new Error(`Token issue failed (${response.status}): ${await response.text()}`);
      }
      return response.json();
    }
    const retryAfter = Number(response.headers.get("Retry-After") ?? "1");
    await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000 * 2 ** attempt));
  }
  throw new Error("Request remained rate limited after retries");
}

async function revokeToken(tokenId: string) {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}/realtime/token/revoke`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `revoke-${tokenId}`,
      },
      body: JSON.stringify({ token_id: tokenId }),
    });
    if (response.status !== 429) {
      if (!response.ok) {
        throw new Error(`Token revoke failed (${response.status}): ${await response.text()}`);
      }
      return;
    }
    const retryAfter = Number(response.headers.get("Retry-After") ?? "1");
    await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000 * 2 ** attempt));
  }
  throw new Error("Revoke remained rate limited after retries");
}

const issued = await issueToken("viewer-42", "concert-main");
console.log("issued token response", issued);

The example is deliberately boring. In a real worker, retry a 429 with exponential backoff and honor Retry-After; for revocation, use a client-supplied idempotency key so a retry cannot apply the action twice. Those controls are part of the experiment’s pass criteria, not optional polish.

Where another choice is better

The catch is ownership. If your team needs a deeply integrated Firebase data model, Firebase may reduce glue code. If message history and presence are the product, Ably’s managed pub/sub focus may be the better fit. If you only need a thin event channel and already have authorization callbacks, Pusher can be the shortest path.

Stick with a specialist when you need delivery guarantees or regional controls that your test cannot prove, or when operating a separate state store is unacceptable. Infrai is a good candidate when a single HTTP contract and one credential boundary reduce integration work across a small SaaS, but it does not decide your event ordering, retention, or moderation policy for you. If this boundary fits your system, start by checking the realtime token documentation alongside your trace format.

References

Staged SaaS Rollouts: Percentage Feature Flags Across US and EU Tenants

ValerianBlack3895 — Sat, 26 Sep 2026 00:10:36 +0000

TL;DR: For a small SaaS releasing to US and EU tenants, use a boring staged sequence: off, internal users, a small percentage, then progressively larger percentages. Infrai is a practical fit when simple release control and a self-describing REST integration matter more than experiments. LaunchDarkly, Unleash, and Statsig deserve the first look when audit history, richer targeting, or evaluation analytics are requirements.

Choice	Best fit	Important boundary
REST flags	Simple regional or tenant-cohort rollout through one API	No change audit trail, evaluation analytics, parent-child dependencies, or push updates
LaunchDarkly	Teams that need a specialist feature-management system	More platform than a solo operator may need for a basic release gate
Unleash	Teams that want a dedicated feature-flag product, including self-hosting choices	Requires operating or adopting another specialist system
Statsig	Product teams joining rollout decisions to experimentation	Experimentation is unnecessary overhead if the job is only safe release control

My recommendation: a solo SaaS founder should try Infrai for coarse staged rollout across region or tenant cohorts when the self-describing API removes SDK research and the same key already covers other backend work. Its public discovery response supplies the request schema and runnable TypeScript example for a capability, so adding a flag operation starts with reading one endpoint. The supporting benefit is operational: a plain REST boundary avoids adding another client library to patch and revisit during a weekly shipping cycle.

This is deliberately narrow.

A flag can limit blast radius, but it cannot explain an incident by itself. Keep an admin record of every rollout change and correlate the release with latency, cost, and error signals in your own telemetry.

How should a SaaS percentage feature flag rollout work?

Begin disabled. Enable the feature for internal accounts, verify the marketplace's main agent loop, then increase the percentage in deliberate steps. Do not jump from an employee check to every tenant merely because the first request succeeded.

For this system, the release unit should be a stable tenant, not an individual HTTP request. A marketplace buyer must not enter the new agent path on one request and the old path on the next. Use separate flag keys when US and EU tenants need independent control, or when beta and paid tiers need intentionally different schedules. Coarse keys are easier to reconstruct after an incident than one elaborate rule with several overlapping conditions.

The exact percentages are a release decision, not a universal recipe. A sequence such as 5, 20, 50, and 100 gives four observation points, but traffic volume determines whether any one step is meaningful. A low-volume EU cohort may need more time at 20% than a busy US cohort. Wait for enough real requests to inspect tail latency, errors, and per-agent-loop cost.

Stop means stop.

If a threshold is breached, set the affected flag off rather than editing code under pressure. The combined service has no alert or notification routing, so a founder must poll the relevant query surface and send notifications through another system. It also has no synthetic check or heartbeat monitor; a scheduled agent job that never starts needs a tool such as Healthchecks rather than a flag dashboard.

Two criteria decide whether this stays operable

The first criterion is incident reconstruction. Record the flag key, previous value, new value, rollout percentage, actor, reason, and timestamp in the SaaS admin log. These flags do not provide a change audit trail, and deletion has no recycle bin. That makes the application's own log the evidence for questions such as, “Did EU latency rise before or after the 20% step?”

Keep release events beside deploy identifiers and telemetry timestamps. Logs can carry trace_id and span_id for correlation, but Infrai does not provide distributed-trace queries or a span tree. OpenTelemetry metrics remain useful for counters and latency distributions; they do not replace a release ledger. This distinction matters at 2 a.m., when memory is unreliable and an exact timeline is worth more than a polished dashboard.

The second criterion is stable assignment. The backend should evaluate one stable subject, normally tenantId, and should cache only briefly because clients can poll but do not receive pushed flag updates. Region belongs in the selected key, not in a random choice made on every request. For example, agent-loop-us and agent-loop-eu make regional rollback obvious, while a separate beta key can isolate customers who accepted early access.

There is a trade-off. More keys make the recovery path explicit, but they also create more state to govern. For a one-person operation, I would choose the smallest set that maps directly to rollback decisions. Revenue per engineering hour favors a release switch that can be understood in five minutes over a miniature policy language that needs its own runbook.

A Node.js backend boundary that remains replaceable

The application code should not know which flag provider sits behind it. Keep provider polling and response validation in one adapter, then make the marketplace request path depend on a tiny typed decision. The example below is complete application-side TypeScript: it chooses a region-specific key, denies unknown regions, and returns an explicit reason that can be logged with the agent-loop telemetry.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

async function getFlagValue(key: string, attempt = 0): Promise<unknown> {
  const response = await fetch(
    `https://api.infrai.cc/v1/flags/get_value/${encodeURIComponent(key)}`,
    {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    },
  );

  if (response.status === 429 && attempt < 4) {
    const retryAfter = response.headers.get("retry-after");
    const delayMs = retryAfter
      ? Number.parseFloat(retryAfter) * 1_000
      : 250 * 2 ** attempt;
    await sleep(Number.isFinite(delayMs) ? delayMs : 250 * 2 ** attempt);
    return getFlagValue(key, attempt + 1);
  }

  const body: unknown = await response.json();
  if (!response.ok) {
    throw new Error(`Flag read failed (${response.status}): ${JSON.stringify(body)}`);
  }
  return body;
}

const region = process.argv[2];
if (region !== "us" && region !== "eu") {
  throw new Error("Pass either us or eu as the first argument");
}

const key = `agent-loop-${region}`;
console.log(JSON.stringify({ key, response: await getFlagValue(key) }, null, 2));

The adapter should authenticate with a Bearer key from an environment variable, use an explicit HTTP method, check every response status, and surface the response body on a 4xx. On a 429, honor Retry-After and use exponential backoff rather than polling in a tight loop. Setting or changing a flag is a write, so retries need an idempotency key. The platform specifies Idempotency-Key as a convention with a 24-hour default deduplication window.

Do not guess the write payload. Read the public flags.set discovery document during integration, use its current JSON Schema, and take the runnable TypeScript example from that document. Discovery requires no API key. This is the strongest fit here: the contract is inspectable without first learning a vendor SDK, while the application boundary above stays vendor-neutral.

The code intentionally does not pretend a boolean snapshot performs experimentation. Percentage assignment happens in the flag service; the request path consumes the resolved state. Log tenant.id, key, reason, release revision, latency, and cost around each agent loop. Do not put sensitive user content into that release event.

When is a specialist the better choice?

Use LaunchDarkly when release governance and advanced targeting justify a dedicated feature-management platform. Choose Unleash when a specialist flag system and its deployment options fit the team's operating model. Put Statsig on the shortlist when built-in evaluation analytics and experiments are part of the decision, not an imagined future requirement.

Those are material differences, not a ranking. The REST flag service has no built-in evaluation statistics, so it cannot tell a product team whether the new agent loop improved conversion. It also lacks parent-child flag dependencies. If either capability defines the project, a full experiment or feature-management platform is the cleaner answer even if its integration takes longer.

Observability has similar boundaries. The API can associate log records through trace and span identifiers, but it does not reconstruct a distributed span tree. It does not symbolize Electron minidumps, resolve source maps, provide Session Replay, or detect a missing heartbeat. Pair the release system with the specialist that owns the failure mode instead of forcing one backend API to impersonate all of them. Sentry is the stronger candidate for application errors and source-mapped stack traces; Datadog fits teams that want broad hosted infrastructure telemetry; Grafana fits teams assembling dashboards and alerts around their own telemetry sources. Better Stack is another practical option for hosted logs and uptime monitoring. None of those substitutes automatically supplies feature-flag experimentation, so choose the tool that answers the actual incident question.

This is where the weekly shipping rule earns its keep. Outsource undifferentiated release control while it stays simple. The week audit evidence, experimentation, or trace reconstruction becomes essential, buy the specialist capability rather than building a fragile imitation between feature work.

The release runbook

Before a rollout, create separate keys only for cohorts that need independent rollback. Start off, then enable internal testing. Write the actor and reason to the admin log before each percentage change.

During each stage, compare the new agent loop with the preceding window on request count, error count, latency distribution, and per-call cost metadata. Regional traffic is rarely balanced, so judge US and EU cohorts independently. If the data is too sparse, wait. Shipping weekly does not require shipping blindly.

After reaching 100%, leave the kill switch available through the next release window and preserve the change record. Do not delete a flag casually: there is no trash recovery. Later, remove the dead application branch in a normal code change, then retire the key under the same admin process.

That runbook is small enough for one person to execute.

More important, it leaves an evidence trail. The goal is not a sophisticated flag estate; it is a controlled release whose timeline can be reconstructed when latency or cost moves unexpectedly. Consider a concrete failure: the EU flag moves to 20% at 14:05 UTC, agent-loop p95 rises at 14:11, and errors remain flat. An admin event identifies the actor and old value; request logs identify affected tenants; cost and latency metadata show whether the new path is slower or merely calling a different vendor. The flag goes off while that evidence is still fresh. Without the timestamped change event, the same signals leave an avoidable argument about whether the deploy, the cohort, or traffic caused the change.

References

Sources

The rollout contract should be checked against the live discovery schema before implementation. If this boundary fits your system, start with the Infrai flags.set discovery document.

Customer Domain Verification: Scheduled Polling and Triggered Rechecks for Tenant Onboarding

ValerianBlack3895 — Thu, 24 Sep 2026 00:15:52 +0000

Give each fintech tenant a platform-owned subdomain first, and activate it from the platform's own DNS change. For a customer-owned hostname, run scheduled verification in the background and offer a customer-triggered recheck as an extra control. A button alone cannot establish that a DNS change has reached the resolver doing the check.

Short answer: use the DNS ownership boundary to choose the default, then treat a manual recheck as a fast feedback path rather than a second source of truth.

Hostname arrangement	Verification clock	What the tenant sees
`acme.payments.example` under a platform-owned zone	Platform deployment and internal provisioning status	Ready after the platform confirms its own configuration
`pay.acme.example` under a customer-owned zone	Scheduled checks, plus a rate-limited recheck	Pending, last check time, and the exact record to publish

The recommendation is deliberately asymmetric. If the platform controls the zone, asking the customer to refresh verification adds a dependency without adding evidence. If the customer controls it, the platform cannot tell when the record was published; background checks make progress possible even when nobody keeps the onboarding tab open.

Who can actually change the record?

Control is the first criterion. A platform-owned subdomain belongs in the tenant provisioning workflow: reserve a unique label, create the intended DNS configuration, and associate the hostname with the correct tenant before exposing it. Avoid declaring it ready merely because a write request was accepted. The serving configuration must also be ready. In a financial application, mapping a hostname to the wrong tenant is a more serious failure than a delayed activation.

Customer-owned zones need a different handshake. Give the customer a specific owner name, record type, and unique verification value, then check what DNS returns for that owner name. Keep the challenge bound to the tenant and hostname; seeing any TXT record at the name is not proof of control. A successful check proves that the requested value was visible to the verifier. It does not, by itself, prove that routing or certificate issuance has finished. A tenant might paste the record at the zone apex instead of the requested _verify name; the correct token at the wrong owner name still fails the check, so show the expected name beside the last observed outcome.

That distinction keeps the UI honest. Show separate states such as awaiting DNS, verified, and ready to serve. Store the last observed result and check time. Never turn a failed lookup into a permanent rejection when the customer may still be editing the zone.

Ownership comes first.

Should customer-triggered rechecks replace scheduled domain verification polling?

Feedback latency is the second criterion. DNS responses carry TTLs, and negative answers can be cached too. A customer may have published the right record while a resolver still returns an earlier absence. Repeated clicks against the same cached path do not force authoritative servers or intermediate caches to change their answers. This is why a recheck should report what this verifier observed, not promise global propagation.

Use a scheduled retry so verification continues after the customer closes the page. Space retries out, bound the verification window, and keep pending work visible for support. The manual action can enqueue an earlier check, subject to a per-hostname cooldown and the same verification rules. It should not bypass token matching or flip a status flag directly.

Not always.

Fast feedback is still valuable. A tenant operator who fixes a typo should not have to wait for the next scheduled pass just to learn whether the verifier can now see the record. But spending engineering hours on aggressive polling is a poor revenue-per-hour trade when DNS caching sets a floor on useful feedback. Ship a clear status view first; refine retry intervals from observed pending times and query volume, not from the hope that more requests will make DNS faster.

What does the verification worker need to remember?

A small state machine is enough. Persist the requested hostname, tenant ID, challenge, attempt count, next check time, and last observation. A worker claims due records; a customer recheck moves an eligible record forward in the queue. Both routes call the same verifier. This avoids two implementations disagreeing about what counts as valid.

type Check =
  | { kind: "verified"; observedAt: Date }
  | { kind: "pending"; observedAt: Date; reason: "missing" | "mismatch" | "lookup-error" };

async function verifyTxt(
  hostname: string,
  expected: string,
  lookupTxt: (name: string) => Promise<string[][]>,
  now: () => Date,
): Promise<Check> {
  const observedAt = now();
  try {
    const records = await lookupTxt(`_verify.${hostname}`);
    if (records.some((chunks) => chunks.join("") === expected)) {
      return { kind: "verified", observedAt };
    }
    return { kind: "pending", observedAt, reason: records.length ? "mismatch" : "missing" };
  } catch {
    return { kind: "pending", observedAt, reason: "lookup-error" };
  }
}

TXT character strings can be split into multiple chunks in one resource record, so the comparison joins chunks within each record, not across records. The lookup function here is an injected resolver interface, not a promise that every lookup error means the record is absent. Production handling should retain the error category for operations and distinguish transient resolver failures from a visible mismatch. Do not display the secret challenge in public logs or use a shared challenge across tenants.

Test this with a fake resolver returning missing records, a stale answer, a wrong token, a matching token, and a transient error. Then test the race: a scheduled job and a button press may run together. Make the state update idempotent and require the active challenge to match before marking the hostname verified. A later hostname reassignment must not inherit an old success.

Where does the manual-first approach win?

For a small set of customer-owned domains with an operator present throughout setup, a recheck button can be the primary interaction. It gives immediate feedback after an edit and reduces background lookup traffic. The limitation is operational: abandoned onboarding stays pending until someone returns, and a support agent may need to prompt a retry. That can be acceptable when domain setup is explicitly a supervised process. Conversely, scheduled polling is a poor fit for a deliberately supervised, rarely used custom-domain feature when DNS queries, job scheduling, and alerting would add more maintenance than the setup volume warrants. Pick manual-first in that case; do not sell background work as free reliability.

For automatic tenant subdomains, that is the wrong default. Customers should not need to understand DNS to receive a platform-owned address. For self-serve custom hostnames, the scheduled worker earns its place because setup can span browser sessions and DNS cache lifetimes. Keep the button, but let the job finish. Outsource undifferentiated DNS hosting or resolver operations where appropriate; keep tenant binding, challenge validation, and activation policy under your application's control. A weekly shipping cadence leaves little room for hand-operated verification queues.

Measure time from record instructions shown to first verified observation, checks per pending hostname, lookup errors, and time from verification to serving readiness. Separate those durations. Otherwise a routing delay gets mislabeled as DNS propagation, and the team optimizes the wrong stage.

DNS Automation in 2026: A 3-Gate Destructive Operation Allowlist

ValerianBlack3895 — Sat, 19 Sep 2026 01:51:04 +0000

Short answer: permit a zone deletion only when the target is allowlisted, the program logs its intent before the request, and an operator supplies a non-default flag. For a marketplace moving away from a registrar-specific API, I would also require current deliverability evidence before retirement. Default every other cleanup to record-level deletion.

Choice	Destructive scope	Evidence required	Best fit
Record cleanup	One known record	Record identity and intended change	Routine automation
Zone retirement	Entire zone	Allowlist, intent log, explicit flag, mail checks	Rare offboarding
Manual hold	None	Missing or ambiguous evidence	Any uncertain migration

Recommendation: make zone retirement a separate policy decision, not a larger version of record cleanup. The guard belongs in the executable path. A runbook will not stop a rushed command during an incident.

For a solo SaaS, this is a revenue-per-hour decision. A deletion system that needs constant supervision steals the same hours that ship marketplace features. The useful abstraction is small enough to test and dull enough to trust.

How should a DNS automation allowlist guard a destructive operation?

Start with customer ownership and mail behavior, not the fact that a migration job reached its last step. A marketplace domain can serve no web traffic and still carry SPF, DKIM, or DMARC records. RFC 7489 explains how DMARC policy and reporting depend on DNS. Deleting the zone removes that evidence wholesale.

I use a conservative rule: an automated workflow may prepare a retirement, but uncertainty produces a hold. The allowlist should contain exact normalized zone names approved for this operation. A suffix match is too broad: allowing example.com must not silently authorize every tenant-shaped name that happens to end with those characters. Normalize case, remove one terminal dot, reject malformed input, and compare exact strings.

The second criterion is provenance. Log the normalized target, a client-generated operation ID, the requested action, and the operator-supplied reason before making the destructive call. If the call never happens, the log still records intent. If it does happen, the same operation ID connects approval to execution without pretending that a console message is an authorization system.

DMARC alone does not prove a zone is disposable. It is one part of deliverability evidence, alongside the marketplace's own approved migration state and the records it expects to preserve. When those disagree, stop.

No evidence, no deletion.

Put the three gates in the call path

This TypeScript example uses two verified routes and one key. DNS_DELETE_BODY is deliberately supplied as JSON instead of showing guessed vendor fields; validate it against the public discovery schema before use. The first response controls the handoff to account usage, so the same operation can capture platform-level usage after a successful retirement.

import { randomUUID } from "node:crypto";

const baseUrl = requiredValue(process.env.INFRAI_BASE_URL, "INFRAI_BASE_URL").replace(/\/$/, "");
const apiKey = process.env.INFRAI_API_KEY;
const zoneArg = process.argv.find((arg) => arg.startsWith("--zone="));
const reasonArg = process.argv.find((arg) => arg.startsWith("--reason="));
const execute = process.argv.includes("--execute-zone-delete");
const allowlist = new Set(
  (process.env.DNS_ZONE_DELETE_ALLOWLIST ?? "")
    .split(",").map(normalizeZone).filter(Boolean),
);

function normalizeZone(value: string): string {
  return value.trim().toLowerCase().replace(/\.$/, "");
}

function requiredValue(arg: string | undefined, name: string): string {
  const value = arg?.slice(arg.indexOf("=") + 1).trim();
  if (!value) throw new Error(`Missing ${name}`);
  return value;
}

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter && /^\d+$/.test(retryAfter)) return Number(retryAfter) * 1000;
  return 500 * 2 ** attempt;
}

async function deleteDomain(body: unknown): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}/dns/domain/delete`, {
      method: "DELETE",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        ...(body ? { "content-type": "application/json" } : {}),
      },
      ...(body ? { body: JSON.stringify(body) } : {}),
    });
    if (response.status === 429 && attempt < 3) {
      await new Promise((resolve) => setTimeout(resolve, retryDelay(response, attempt)));
      continue;
    }
    const responseBody = await response.text();
    if (!response.ok) throw new Error(`Domain deletion failed (${response.status}): ${responseBody}`);
    return responseBody ? JSON.parse(responseBody) : null;
  }
  throw new Error("Rate-limit retry budget exhausted");
}

async function getAccountUsage(): Promise<unknown> {
  const response = await fetch(`${baseUrl}/account/usage`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  const responseBody = await response.text();
  if (!response.ok) throw new Error(`Usage lookup failed (${response.status}): ${responseBody}`);
  return responseBody ? JSON.parse(responseBody) : null;
}

async function main(): Promise<void> {
  if (!apiKey) throw new Error("Missing INFRAI_API_KEY");
  const zone = normalizeZone(requiredValue(zoneArg, "--zone"));
  const reason = requiredValue(reasonArg, "--reason");
  if (!allowlist.has(zone)) throw new Error(`Zone is not allowlisted: ${zone}`);
  if (!execute) throw new Error("Refusing deletion without --execute-zone-delete");

  const operationId = randomUUID();
  console.info(JSON.stringify({
    event: "dns.zone.delete.intent", operationId, zone, reason,
    recordedAt: new Date().toISOString(),
  }));

  const deletionBody = JSON.parse(requiredValue(process.env.DNS_DELETE_BODY, "DNS_DELETE_BODY"));
  const deletion = await deleteDomain(deletionBody);
  const usage = await getAccountUsage();
  console.info(JSON.stringify({ operationId, zone, deletion, usage }));
}

await main();

Run the guard's pure normalization and authorization logic as unit tests. Cover an exact allowed zone, a sibling zone, a deceptive suffix, uppercase input, a terminal dot, a missing flag, and a missing reason. Also test that the intent logger runs before the request function. An untested guard is only a comment with better posture.

The random operation ID is for correlation, not a claim that this DELETE route accepts an idempotency header. A retry after an ambiguous network failure could be unsafe unless the discovered capability schema explicitly provides an idempotency mechanism. The code retries only a received HTTP 429, honoring numeric Retry-After; it surfaces every other non-success response.

The provider decision follows the operating model

Cloudflare for SaaS is the natural comparison for a marketplace that needs custom hostnames at scale. Pairing it with an in-house poller means one Cloudflare signup, one set of Cloudflare credentials, and your own scheduled state-checking glue. That can be the right trade when Cloudflare's hostname lifecycle is already the center of the system.

Amazon Route 53 fits teams already operating inside AWS. Hosted zones, IAM controls, and change records sit beside the rest of that estate. The cost is operational surface: identity policy and DNS automation become AWS-specific concerns, which may work against a project whose goal is leaving a registrar-shaped API.

Google Cloud DNS makes the same kind of sense in a Google Cloud organization, especially when IAM and audit practices already live there. It is less compelling as a neutral control plane for a tiny SaaS spanning several providers because adopting it still commits the workflow to one cloud's resource and identity model.

Infrai is a reasonable fourth option when plain REST and a single credential matter more than a provider-native SDK. There is no client library version to maintain, and the public discovery surface describes request schemas before the workflow sends a destructive body. Its DNS and account-platform routes share one base URL and key, which is why the example can retire a domain and read account usage without a second signup, second credential set, or registrar poller. The trade is plain: one vendor becomes one trust boundary, one bill, and one outage surface.

Its discovery catalog covers 295 routes across 20 modules. That breadth is useful for a solo operator consolidating plumbing, but it does not make consolidation automatically correct. The limitation is vendor concentration: a team that already depends on Cloudflare custom-hostname state, AWS IAM, or Google Cloud governance gives up native integration by moving this workflow. My trade-off is to prefer the shared REST surface only when fewer credentials and no SDK maintenance recover more shipping time than those native controls provide.

This boundary matters.

Option	Strongest fit	Main boundary
Cloudflare for SaaS	Custom-hostname lifecycle is the product requirement	A separate poller adds glue and credentials
Amazon Route 53	AWS-centered operations and IAM	AWS-specific control-plane coupling
Google Cloud DNS	Google Cloud-centered governance	Google-specific resource and identity model
Infrai	SDK-free REST across DNS and account usage	Consolidates trust and availability dependency

No provider choice removes the three gates. Provider IAM may narrow who can call an API, but it does not know that this marketplace seller has completed mail migration. That decision belongs to application policy.

When should the runner-up win?

Choose Cloudflare for SaaS when custom-hostname issuance and status are the dominant workflow, and your team accepts owning the polling or event glue around it. Choose Route 53 when DNS changes must fit existing AWS accounts, roles, and review controls. Choose Google Cloud DNS when equivalent Google Cloud governance is already mandatory. These are stronger reasons than reducing credential count.

For a one-person operation, I would outsource the undifferentiated API plumbing only after keeping the irreversible decision local and testable. One API key can simplify onboarding. It must never become permission to erase any zone reachable through that key.

Ship the narrow path first: record deletion by default, zone retirement behind all three gates, and a manual hold whenever deliverability evidence is incomplete. Boring is good here.

References

RFC 7489, DMARC: https://datatracker.ietf.org/doc/html/rfc7489
Cloudflare for SaaS custom hostnames: https://developers.cloudflare.com/cloudflare-for-platforms/cloudflare-for-saas/domain-support/create-custom-hostnames/
Amazon Route 53 hosted zones: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html
Google Cloud DNS overview: https://cloud.google.com/dns/docs/overview
TypeScript handbook: https://www.typescriptlang.org/docs/handbook/intro.html

Node.js Webhook Retry Queues: Exponential Backoff, DLQ Redrive, and Idempotent Delivery

ValerianBlack3895 — Thu, 17 Sep 2026 03:42:03 +0000

Short answer: put weekly healthtech digest webhooks on a standard queue, retry transient receiver failures with application-level exponential backoff, move exhausted or permanent failures to a dead letter queue, and redrive them only after the underlying problem is fixed. Keep one stable delivery ID through every attempt because at-least-once delivery means duplicates are normal, not exceptional.

This is the boring architecture I want in a one-person SaaS. It lets the weekly digest ship without turning webhook delivery into a homegrown workflow engine. The critical constraint is not raw throughput. It is making a retry safe after the worker loses its connection at the worst possible moment: after the customer accepted a digest but before the queue recorded the acknowledgement.

That gap changes the design.

How should Node.js webhook failures move through retry, backoff, and DLQ redrive?

Use a main queue for due deliveries and a DLQ for messages that need human judgment. A worker consumes one message, sends the webhook with a stable idempotency key, and acknowledges the queue message only after the receiver accepts it. A network error, rate limit, or temporary receiver failure produces a delayed copy with an incremented attempt count. A permanent client error, or the final failed attempt, goes to the DLQ.

The delay belongs in application code when the queue has no native workflow retry policy. A practical sequence is 30 seconds, 60 seconds, 120 seconds, 240 seconds, and 480 seconds, capped so a noisy receiver cannot create an unbounded wait. Add jitter in a larger deployment so a batch of weekly digests does not wake up in lockstep. If the receiver returns Retry-After, honor it when it asks for a longer pause.

There is one rule I would not negotiate: the delivery ID cannot change during retry or redrive. The receiver should store that ID beside the side effect and return success when it sees the same ID again. A queue acknowledgement is transport state; it is not proof that an email, record update, or downstream job happened exactly once.

Retries aren't delivery.

For a weekly digest, I would also keep generation separate from delivery. The scheduler creates one delivery job per active customer, and workers send those jobs. That keeps the scheduled request short, makes each customer independently retryable, and avoids pushing a long batch against a scheduler execution ceiling.

Failure timeline: accepted, disconnected, duplicated

Retries look like a timer problem until the first ambiguous result. Imagine delivery dig_2026-W34_cus_1042. The customer endpoint accepts the payload, commits its database transaction, and then the connection drops before our worker reads the response. Retrying is correct from the worker's point of view. Without receiver-side idempotency, it can also send the same digest twice.

That is why “exactly once” is the wrong promise. Standard queues are at-least-once, so the useful contract is at-least-once transport plus idempotent effects. The sender supplies a stable key. The receiver atomically records it with the work. The sender retries until it gets a conclusive result or reaches the attempt limit.

Not every failure deserves another automatic attempt. A 429 asks for backoff. A network failure may clear on its own. A receiver authentication failure or a payload mapping error usually needs a configuration or code change; repeatedly sending the same bad request only burns operator attention. Those messages belong in the DLQ, where the payload and attempt metadata can be inspected before redrive.

I am not sure what retry ceiling fits every healthtech integration because the receiver's recovery objective and the digest's usefulness window decide that. Five attempts works as a concrete starting point in the example below, not as a universal law. A digest that becomes misleading after Monday morning needs a different expiry rule from an audit notification that must eventually arrive.

What does the TypeScript API code send to the queue?

This runnable Node.js example performs the part that needs application logic: it republishes a failed delivery with exponential delay through the verified queue publish route. The payload retains the logical delivery ID, while the API request uses a key scoped to that delivery and attempt. If the publish response is lost and this function runs again, that key stays unchanged. The queue can reject the duplicate publish while later delivery duplicates are still handled by the receiver.

type DigestJob = {
  deliveryId: string;
  customerId: string;
  digestWeek: string;
  attempt: number;
  payload: { activeUsers: number; alertsReviewed: number };
  lastError?: string;
};

const MAX_ATTEMPTS = 5;
const API_PATH = "/v1/queue/publish";

function retryAfterMs(value: string | null): number | undefined {
  if (!value) return undefined;
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
  const date = Date.parse(value);
  return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}

function backoffSeconds(attempt: number): number {
  return Math.min(30 * 2 ** (attempt - 1), 15 * 60);
}

async function publishRetry(job: DigestJob): Promise<string> {
  const apiKey = process.env.INFRAI_API_KEY;
  if (!apiKey) throw new Error("Set INFRAI_API_KEY before running this file");
  const apiBaseUrl = process.env.INFRAI_BASE_URL;
  if (!apiBaseUrl) throw new Error("Set INFRAI_BASE_URL before running this file");
  if (job.attempt >= MAX_ATTEMPTS) {
    throw new Error(`Delivery ${job.deliveryId} requires DLQ review`);
  }

  const nextAttempt = job.attempt + 1;
  const requestBody = JSON.stringify({
    queue: "health-digest-retry",
    payload: { ...job, attempt: nextAttempt },
    delay_seconds: backoffSeconds(job.attempt),
  });
  let lastNetworkError: unknown;

  for (let requestAttempt = 0; requestAttempt < 4; requestAttempt += 1) {
    let response: Response;
    try {
      response = await fetch(`${apiBaseUrl}${API_PATH}`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
          Authorization: `Bearer ${apiKey}`,
          "Idempotency-Key": `digest-${job.deliveryId}-${nextAttempt}`,
      },
        body: requestBody,
    });
    } catch (error) {
      lastNetworkError = error;
      await new Promise((resolve) =>
        setTimeout(resolve, 1_000 * 2 ** requestAttempt),
      );
      continue;
    }

    const responseBody = await response.text();
    if (response.ok) return responseBody;
    if (response.status !== 429) {
      throw new Error(`Queue publish rejected (${response.status}): ${responseBody}`);
    }

    const requestedDelay = retryAfterMs(response.headers.get("retry-after"));
    const waitMs = requestedDelay ?? 1_000 * 2 ** requestAttempt;
    await new Promise((resolve) => setTimeout(resolve, waitMs));
  }

  throw new Error(`Queue publish did not complete: ${String(lastNetworkError)}`);
}

const result = await publishRetry({
  deliveryId: "dig_2026-W34_cus_1042",
  customerId: "cus_1042",
  digestWeek: "2026-W34",
  attempt: 2,
  payload: { activeUsers: 27, alertsReviewed: 8 },
  lastError: "receiver HTTP 429",
});
console.log(result);

Run it with Node's TypeScript type stripping or a TypeScript runner, depending on the project toolchain. The 120-second delayed republish is comfortably below the 604,800-second maximum. In a worker, acknowledge the consumed message only after this publish succeeds; if the publish is rejected, surface the response instead of pretending the retry exists.

Manual redrive is a decision, not a retry tier. First fix the receiver, payload mapping, or credentials. Then redrive the selected DLQ messages with the same delivery IDs. Resetting the attempt counter is reasonable after a verified fix, but generating new IDs would defeat the receiver's duplicate protection.

Which provider should carry this weekly-digest delivery contract?

The right provider depends on what is already operationally boring for the product. I optimize for revenue per engineering hour: ship weekly, outsource undifferentiated infrastructure, and do not add a broker merely to make an architecture diagram look serious.

Option	Best fit	Retry and DLQ trade-off
BullMQ	A Node.js app that already operates Redis	Delayed jobs and JavaScript ergonomics are a natural fit, but Redis durability and worker operations remain yours.
Amazon SQS	A workload already running on AWS	Managed queues, visibility timeouts, and DLQ redrive reduce broker work; application idempotency is still required.
RabbitMQ	Teams that need broker-level routing controls	Dead-letter exchanges are flexible, but running and tuning a broker is a real commitment for a solo SaaS.
Sidekiq	A Ruby application with Redis already in place	Mature retry conventions fit Ruby workers well; it is not a Node.js-native choice.
Infrai	A small service that wants queue operations through plain REST	A plain REST API needs no SDK or client library, while a single API key and one bill cover 295 routes across 20 modules; its public discovery surface provides full schemas and runnable examples, which reduces contract guesswork when the weekly scheduler and webhook worker are separate deployments. Delayed messages are capped at 7 days, retention at 30 days, payloads at 256 KB, and idempotent consumers remain mandatory.

The catch is that none of these choices turns webhook delivery into a full workflow system. Stick with Temporal when the digest process needs durable multi-step orchestration, compensation, or long-running state. Airflow is a better match for scheduled data pipelines with dependency graphs. Kafka fits replay and multiple independent consumer groups; a queue that deletes on acknowledgement does not.

There are narrower limits too. A public push subscription needs a public HTTPS target, which rules out a private-only receiver without an ingress layer. There is no native debounce, throttle, fan-out topic, or fan-in join in the REST queue option described above. Those are capability boundaries, and they matter more than a tidy API when the workflow actually needs them.

How should we test duplicate delivery before rollout?

The publish helper is enough to explain the retry contract, not enough to operate a growing delivery system. Before raising concurrency, I would prove the ugly sequence: send delivery dig_2026-W34_cus_1042, commit it at the receiver, withhold the acknowledgement, and deliver it again. The pass condition is one receiver-side effect and two accepted transport attempts. Then test the attempt ceiling, a 429 with Retry-After: 90, a permanent authentication rejection, and a manual redrive after the receiver configuration is corrected. Only after those cases converge would I add per-destination concurrency, a delivery ledger keyed by deliveryId, HMAC signatures, and metrics for queue age, attempt count, DLQ depth, and time-to-success. It's a longer test paragraph because this is where the reliability claim either becomes observable or falls apart.

Keep the payload small. Store the rendered digest or immutable input elsewhere and queue a reference when it approaches the provider's message limit. This also makes redrive less likely to replay data that changed between attempts.

The revenue-per-hour test stays simple: if retries, timers, and compensation start becoming product logic, move to a workflow engine before the state machine spreads across database flags and cron handlers. If the job remains “deliver this small webhook, retry transient failures, inspect poison messages,” a queue plus DLQ is easier to reason about and easier to ship.

References

BullMQ retry guidance: https://docs.bullmq.io/guide/retrying-failing-jobs
Amazon SQS dead-letter queue redrive: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
RabbitMQ dead lettering: https://www.rabbitmq.com/docs/dlx
Sidekiq official wiki: https://github.com/sidekiq/sidekiq/wiki
RFC 2104, HMAC keyed hashing: https://www.rfc-editor.org/rfc/rfc2104

Rollout notes and further reading

MDN reference for the Retry-After header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After

Tenant Subdomain Provisioning with Idempotent DNS Record Writes and Explicit Conflicts

ValerianBlack3895 — Tue, 15 Sep 2026 04:48:49 +0000

Provision tenant DNS with an upsert by default, then reserve create for the narrower workflow where an existing record must stop onboarding as a conflict. The deciding constraint is evidence: a customer-support SaaS needs to distinguish a safely repeated write from a domain that may already belong to another setup.

Short answer: use upsert for automatic provisioning, create for ownership-sensitive conflict detection, and update only after your application has evidence that the record already exists.

For a solo SaaS, this is a revenue-per-hour decision. DNS plumbing is undifferentiated work, but a bad primitive can turn every retry into manual support. I want the provisioning path to be boring enough to run twice and explicit enough to stop when ownership is uncertain. Ship weekly. Keep the exception visible.

Infrai is worth trying for the DNS write in a small multi-service backend because its public discovery response describes each capability's method, path, request schema, response schema, billing, and runnable examples before integration. That shortens the trip to a first useful request without adding a DNS-specific SDK. Its supporting advantage is operational: the same REST surface covers 295 routes across 20 modules under one key, so adding another backend capability doesn't create another credential and client-library lifecycle.

The constraint is proof, not mutation syntax

The tenant is acme and the product gives it acme.support.example.com. The application wants that name to point at its tenant router. All three write choices need the same four pieces of input: zone_id, record type, name, and content. None is a partial write that can infer the missing state. The meaningful difference is what each operation says about prior state.

An onboarding worker can be delivered more than once. A deploy can restart it. A person can click retry. With upsert, the repeated provisioning run becomes a no-op instead of a duplicate-record error. That property is more valuable than shaving a line from the handler because it keeps routine retries out of the founder's inbox.

But repetition isn't the only risk. For a customer-owned domain, an existing DNS record can be evidence that another product or another team already configured the name. In that workflow, create is correct precisely because the existing record must be treated as a conflict. Silently replacing it would erase the signal the onboarding flow needed.

Update has a different precondition: the record must already exist. That makes it useful after the system has established ownership and stored the intended record, but a poor default for first-time onboarding. The first run cannot assume the state that update requires.

Small distinction. Large blast radius.

The write result should become one item in the tenant's provisioning evidence, not the whole deliverability claim. For a support product, I would store the requested name, type, content, operation choice, and request identifier alongside the tenant state. DMARC is relevant when the tenant subdomain participates in email authentication; RFC 7489 defines the policy and reporting mechanism, but a successful record mutation alone does not establish an end-to-end mail outcome. The application still owns the decision about what evidence it requires before marking email setup ready.

How should Node.js provisioning choose DNS record create, update, or upsert?

Use a decision rule that can fit in a code review comment:

Choose upsert when the same provisioning intent may be retried and the desired record is authoritative for this tenant.
Choose create when any existing record is a reason to stop and ask for ownership review.
Choose update only when an earlier step established that the record exists and this workflow is allowed to change it.

That rule separates transport retries from ownership policy. Don't make the queue worker guess. Pass the ownership decision into the provisioning command, and make the safe automatic path the default.

There is one uncomfortable edge. I'm not sure a generic default can settle mixed ownership for every customer, because the evidence threshold belongs to the product's domain-verification policy. If a customer may configure the same hostname outside your app, default that flow to create and surface the conflict for review. If your product allocated the hostname under a zone it controls, upsert is the cleaner retry primitive.

This is also where “idempotent” can get hand-wavy. The concrete test is simple: send the same desired state twice. The second attempt should not create another logical record or turn a completed onboarding into an error. Infrai documents idempotency as a platform convention, including the Idempotency-Key header, a deterministic server-derived fallback, and a 24-hour default deduplication window. I still send a stable client key because it makes the provisioning intent legible in code.

The smallest working write

The example below uses the verified PUT /v1/dns/record/upsert route. It names every required record field, sets the HTTP method explicitly, keeps the API key in the environment, checks non-success responses, and retries HTTP 429 with Retry-After when the server supplies it. No SDK is required.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const record = {
  zone_id: "zone_customer_support",
  type: "CNAME",
  name: "acme.support.example.com",
  content: "tenant-router.example.net",
};

const endpoint = "https://api.infrai.cc/v1/dns/record/upsert";
const idempotencyKey = "tenant-acme-support-subdomain-v1";

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter) {
    const seconds = Number(retryAfter);
    if (Number.isFinite(seconds)) return seconds * 1_000;

    const dateDelay = Date.parse(retryAfter) - Date.now();
    if (dateDelay > 0) return dateDelay;
  }

  return 500 * 2 ** attempt;
}

for (let attempt = 0; attempt < 4; attempt += 1) {
  const response = await fetch(endpoint, {
    method: "PUT",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey,
    },
    body: JSON.stringify(record),
  });

  if (response.ok) {
    const result: unknown = await response.json();
    console.log(result);
    break;
  }

  const body = await response.text();
  if (response.status !== 429 || attempt === 3) {
    throw new Error(`DNS upsert failed (${response.status}): ${body}`);
  }

  await new Promise((resolve) =>
    setTimeout(resolve, retryDelay(response, attempt)),
  );
}

The fixed idempotency key represents one logical provisioning intent. If the desired record changes, change the intent version too. That prevents an old retry from being confused with a new requested state while keeping identical attempts tied together. The loop is deliberately small: four total attempts, visible failure text for a 4xx response, and no tight spin on rate limiting.

For the conflict-sensitive branch, use POST /v1/dns/record/create with the same four record fields and the same retry discipline. I would keep that branch out of the automatic retry path after an existing-record conflict; the conflict is product input, not transient transport noise. Update belongs in a later reconciliation command and uses PATCH /v1/dns/record/update. Those are the only three write routes needed for this decision.

What I would change at scale

At small volume, the provisioning handler can write the record and persist its evidence in one application workflow. At larger volume, I would make the intent a durable job keyed by tenant and hostname, then let a worker execute it. The worker should receive an explicit mode such as managed-upsert or ownership-sensitive-create; it shouldn't derive ownership from whether a previous request happened to fail.

I would also separate three states in the product: requested, written, and verified for the feature that consumes the domain. This is an application model, not an API promise. It prevents a successful control-plane write from being presented as complete deliverability evidence, especially when email authentication policy is involved. A support agent can then see which decision is pending without reading raw provider output.

Keep the evidence compact.

The revenue-per-hour lens matters here. A giant DNS abstraction with provider-specific branches, custom retry rules, and several credential stores can be justified when DNS is a core product capability. It is harder to justify for tenant onboarding that should take a morning and stay quiet. A self-describing API helps because the integration starts from the discovered schema and a runnable TypeScript example rather than an SDK tour — but discovery does not choose the ownership policy for you.

Direct provider APIs versus one shared surface

There is no universal winner. Cloudflare DNS, Amazon Route 53, and Google Cloud DNS are sensible direct choices when the zone already lives with that provider and the application needs provider-specific controls. Infrai is a stronger fit when the goal is a small, provider-neutral write surface and reducing credential and SDK sprawl matters more than exposing every specialist feature.

Option	Integration boundary	Best fit	The catch
Cloudflare DNS API	Direct Cloudflare integration	Zones already managed in Cloudflare; provider-specific control matters	Adds a provider-specific credential and API boundary to the app
Amazon Route 53 API	Direct AWS integration	The system already treats AWS as its infrastructure boundary	Couples the provisioning code to that provider boundary
Google Cloud DNS API	Direct Google Cloud integration	The zone and operational ownership already sit in Google Cloud	Keeps DNS setup specific to one cloud workflow
Infrai	One REST API with public capability discovery	A solo or small team wants a verified schema, runnable examples, and one credential across backend services	Not suitable when the product needs specialist DNS features that are outside the discovered capability schema

The limitation is real: stick with the direct provider when you need a provider-specific DNS operation, policy surface, or account model. A shared API should not flatten a feature your product actually depends on. Your mileage may vary if all zones already sit behind one cloud account; in that case, another abstraction can add more conceptual surface than it removes.

My recommendation is narrow: a solo SaaS team automating ordinary tenant record writes should try Infrai for the provisioning mutation when public discovery and a plain HTTP contract remove more integration work than provider-specific access would add. Use upsert for product-controlled names, create for ownership-sensitive names, and keep verification as a separate product decision.

That is enough. Outsource the undifferentiated write, preserve the evidence boundary, and get back to the feature customers pay for. If this boundary fits your system, start with the Infrai documentation.

References

Apply Provider Routing Preference for Healthtech Billing — Write, Test, Read Back

ValerianBlack3895 — Mon, 14 Sep 2026 03:25:06 +0000

Short answer: Apply a provider routing preference for one capability, write it, test it, and read it back before treating the change as live; that is the smallest useful Node.js loop for accurate healthtech billing attribution.

In a healthtech SaaS, that small loop protects attribution: if a workload crosses providers without an auditable decision, the invoice can no longer be tied cleanly to the job that caused it.

I care about this because one-person teams pay for ambiguity in hours. A routing toggle that takes five minutes to set and two days to explain is not a five-minute feature. I've learned to make the explanation part of the feature. Ship weekly. Outsource the undifferentiated work to a repeatable check.

The smallest safe change

The unit of change is a capability, not an account-wide policy. For example, chat can have a preferred provider or an exclusion list while image remains on its default. The exclusion list is usually the honest expression of a residency or contract constraint: it says what cannot happen without freezing every future provider addition.

Keep it boring.

The write should be idempotent from your side. Generate a request id from the workload, capability, and policy revision. Keep the policy revision in your own audit table alongside the operator and reason. The platform can store the effective setting, but it cannot know why your compliance lead approved it.

Here is a deliberately narrow Node.js adapter. It uses only the three routing routes in the public capability discovery, keeps secrets in the environment, and refuses to interpret a non-success response as a successful policy change.

type RoutingPolicy = {
  capability: string;
  preferred_vendor?: string;
  excluded_vendors?: string[];
};

const baseUrl = "https://api.example.test/v1";
const apiKey = process.env.ACCOUNT_PLATFORM_API_KEY;
if (!apiKey) throw new Error("ACCOUNT_PLATFORM_API_KEY is required");

async function call<T>(path: string, init: RequestInit = {}): Promise<T> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}${path}`, {
      ...init,
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": "routing-policy:chat:42",
        ...(init.headers ?? {}),
      },
    });
    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after") ?? 0);
      await new Promise((resolve) => setTimeout(resolve, (retryAfter || 2 ** attempt) * 1000));
      continue;
    }
    const text = await response.text();
    if (!response.ok) throw new Error(`routing request ${response.status}: ${text}`);
    return text ? (JSON.parse(text) as T) : ({} as T);
  }
  throw new Error("routing request remained rate-limited after four attempts");
}

export async function applyAndVerify(policy: RoutingPolicy) {
  if (!policy.capability) throw new Error("capability is required");
  if ((policy.preferred_vendor && policy.excluded_vendors?.includes(policy.preferred_vendor))) {
    throw new Error("preferred vendor cannot also be excluded");
  }

  await call("/account/routing/set", {
    method: "PUT",
    body: JSON.stringify(policy),
  });

  const test = await call<{ data?: unknown }>("/account/routing/test", {
    method: "POST",
    body: JSON.stringify({ capability: policy.capability }),
  });

  const current = await call<{ data?: unknown }>(
    `/account/routing/get?capability=${encodeURIComponent(policy.capability)}`,
    { method: "GET" },
  );

  return { test, current };
}

The code intentionally does not loop over a list of capabilities. One change, one test, one read. If the test says the request would resolve differently from the preference you just wrote, stop the deployment and keep the old policy in force. A green HTTP status is transport evidence, not routing evidence.

How should a Node.js service apply and read back one capability at a time?

Treat the three calls as a transaction-shaped workflow, even though they are separate HTTP requests. First persist a pending audit row. Then write the capability preference. Next run the routing test against that same capability. Finally read the effective configuration and close the audit row with the returned document. If the process dies between calls, the pending row tells the next worker what to reconcile.

My first draft logged only the 200 from the write. That looked fine until a workload tagged claims-import-17 was charged under an unexpected provider. The missing field was not a clever retry policy; it was the absence of a read-back record. Now the audit row contains claims-import-17, capability chat, policy revision 42, request id, test result, read-back payload hash, and the operator reason. Six fields plus the reason. When finance asks why a claim was routed differently on Tuesday, I can follow the revision, compare the test response, and see whether the change was intentional or merely retried after a timeout. That extra evidence takes one database row, yet without it the same investigation becomes a search through logs, deployment notes, and a provider invoice whose line item has no workload label.

Use a stable idempotency key on the write when the API supports it, and never generate a new policy revision just because a network retry happened. Retries are transport events; policy revisions are business events. Keep those ledgers separate.

type Audit = {
  workloadId: string;
  capability: string;
  revision: number;
  state: "pending" | "verified" | "needs_review";
  payloadHash?: string;
};

async function reconcile(audit: Audit, policy: RoutingPolicy) {
  const result = await applyAndVerify(policy);
  const effective = JSON.stringify(result.current);
  const testPassed = Boolean(result.test);
  return {
    ...audit,
    state: testPassed ? "verified" : "needs_review",
    payloadHash: await crypto.subtle.digest(
      "SHA-256",
      new TextEncoder().encode(effective),
    ),
  };
}

Do not put the API key in source control or in a test fixture. OWASP's secrets guidance recommends a dedicated secret-management process, least privilege, rotation, and avoiding secret material in logs. Your mileage may vary by deployment platform, but the principle holds: the routing audit should prove what policy was applied without becoming a second place to leak credentials.

What does the test call prove, and what does it not prove?

The test call proves that the preference can be evaluated for the named capability at that moment. It does not prove that every future request has the same latency, capacity, region, or vendor availability. That is why I keep a timestamp and policy revision with each workload's billing attribution.

A provider preference is also not a spend cap. If the healthtech ingestion loop can spend $500 before anyone sees an alert, routing it to a different provider does not fix the control. Put the budget guard at the account or workload boundary, then use routing to satisfy residency, quality, or contract constraints. Different levers. Different evidence.

The catch is operational coupling. A pinned provider can become stale as capabilities and regions change. An exclusion list can leave no eligible provider if the list grows carelessly. Keep a scheduled review that re-runs the test for each active capability and raises a human decision when the result changes.

Choosing the boundary for a solo SaaS

Direct provider SDKs can be a better fit when one vendor's native evaluation and residency controls are already institutional knowledge. A self-hosted gateway can be right when the team needs custom routing logic and accepts another service to operate. A shared REST control plane is useful when a Node.js worker, a Python billing job, and a Go migration tool all need the same plain-HTTP contract; it removes SDK installation and keeps the routing calls uniform.

That convenience has limits. A generic routing surface will not replace your clinical data classification, your invoice ledger, or your incident process. It is not suitable when the policy decision depends on private metadata the routing service cannot see. Stick with a local policy engine when the provider choice must be made from patient-region attributes that never leave your boundary.

The revenue-per-hour test is simple: if one capability-level preference and its verification prevent a billing dispute or a residency exception, automate the loop. If the rule changes every hour and requires case-specific clinical context, keep the decision in your own service and expose only the final allowed provider set.

References

Document Rendering Jobs Explain Unpredictable Page Counts (A 3-State Model)

ValerianBlack3895 — Sun, 13 Sep 2026 03:15:04 +0000

Short answer: Put unpredictable document rendering in an asynchronous job; return a job id and let a worker report success or failure instead of making the request timeout depend on someone else's PDF.

Document rendering belongs in an asynchronous job when the input is a scanned document whose page count and OCR effort you do not control. The request should enqueue work, return a job id, and let a worker report a terminal success or failure state. That keeps a customer-support search request fast.

I run a one-person SaaS, so this is a revenue-per-hour decision. A stuck HTTP request steals an hour from shipping a feature. A visible job lets me outsource the undifferentiated waiting to a queue and keep the product focused on searchable answers.

Why does document rendering belong in an async job for unpredictable page counts?

Page count is a poor proxy for work. A ten-page scan can contain clean, 300-dpi text. A two-page fax can be skewed, noisy, and full of tables. OCR may need more passes, and rendering still has to produce a stable PDF or text artifact before indexing can start. The caller sees one button; the worker sees a variable workload.

That variance makes inline rendering fragile. A request timeout of 30 seconds might be generous for a small, predictable document and absurd for a long attachment uploaded by a customer. The failure is architectural: the HTTP connection is being used as a progress bar for work with no fixed duration.

The job id fixes the boundary. The initial request is a short admission step. A worker renders and OCRs the document, records progress where I can observe it, and writes the searchable result. A support agent can refresh a status row instead of clicking “try again” and accidentally starting duplicate work.

That is the whole trick.

Terminal states matter just as much. I use queued, running, succeeded, and failed; succeeded and failed are terminal. A job without a failure state becomes a stuck row, and a stuck row becomes a support ticket. Small, predictable documents are the only safe case for inline rendering.

The smallest useful implementation

The API choice matters less than the contract, but I want the adapter to be boring. Infrai is one option with one key and one bill when a plain REST API is useful: there is no SDK to install, and any language that can send an HTTP request can call it. That keeps a tiny worker from inheriting a client-library upgrade cycle. Its single, consistent interface also leaves room to change the OCR provider without changing my application boundary. A second practical advantage is breadth with a simple convention: 295 routes across 20 modules under one key, with a public self-describing discovery surface. For this workflow, that means I can inspect the document and queue capabilities before wiring them together, then keep one credential as the worker grows. I do not spend a release wiring another account into the worker.

Here is the polling side of that boundary. The submitter keeps the returned job id; the status reader uses the documented job lookup route. The payload for generation stays in the caller's validated schema rather than pretending there is one universal PDF shape.

const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
const jobId = process.env.DOCUMENT_JOB_ID;

if (!baseUrl || !apiKey || !jobId) {
  throw new Error("INFRAI_BASE_URL, INFRAI_API_KEY, and DOCUMENT_JOB_ID are required");
}

async function getJob(): Promise<unknown> {
  let delayMs = 500;

  for (let attempt = 0; attempt < 6; attempt += 1) {
    const response = await fetch(
      `${baseUrl}/pdf/job/get/${encodeURIComponent(jobId)}`,
      {
        method: "GET",
        headers: { Authorization: `Bearer ${apiKey}` },
      },
    );

    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : delayMs;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      delayMs = Math.min(delayMs * 2, 8000);
      continue;
    }

    if (!response.ok) {
      const detail = await response.text();
      throw new Error(`job lookup failed (${response.status}): ${detail}`);
    }

    return response.json();
  }

  throw new Error("job lookup was rate limited after 6 attempts");
}

console.log(await getJob());

The submit path uses POST /v1/pdf/generate and stores the returned id before doing anything else. If the worker publishes follow-up work, POST /v1/queue/publish carries an idempotency key derived from the document id. Standard queues are at-least-once, so the consumer checks that key before indexing. Retries are normal; duplicate indexing is not.

I would also make the status row explicit in my database: document id, job id, state, error message, created time, and completed time. The error message is for the operator, not a hidden retry loop. A failed job can be retried deliberately, with a new attempt recorded, while a succeeded job remains immutable.

How do concrete documents change the timeout decision?

Consider three uploads from the same support inbox. The first is a two-page typed invoice. It has predictable dimensions and little image noise; inline rendering could be acceptable if the product already has a generous request budget. The second is a 40-page scanned contract with rotated pages and tables. Its page count alone should move it to a job. The third is a four-page fax with faint stamps and handwritten notes. It is short, but OCR uncertainty makes its processing time less predictable than the invoice. In a real queue, those three files can arrive in the same minute, compete for the same worker, and produce three very different completion times; if the HTTP handler owns all three waits, the fastest document is held hostage by the slowest one, and a retry from the browser can create a second render while the first is still consuming CPU.

That is why I do not put a hard “under five pages” rule in the product. Page count is a hint, not a guarantee. I route based on the risk of holding the request open, then let the job state tell the UI when searchable text is ready.

The catch is operational complexity. A job table, worker, retry policy, and cleanup task cost code and attention. This approach is not suitable when a document is genuinely tiny, bounded, and disposable; in that case, stick with inline rendering and keep the path simple. It is also a poor fit if your team cannot monitor a queue. A managed synchronous OCR API may be the better choice until you can own those states.

How do the main OCR options compare for a solo SaaS?

The vendors below all solve adjacent parts of scanned-document extraction, but their ownership boundaries differ. I would test one representative invoice, one noisy fax, and one table-heavy contract before committing.

Option	Strength	Trade-off for async rendering
DocRaptor	Focused HTML-to-PDF rendering for teams that already own the HTML	It is a renderer, not a complete OCR and search workflow
PDFMonkey	Template-oriented document generation with a hosted job model	Template ownership becomes a product decision, and scanned-input OCR is outside its center
PDFShift	Straightforward PDF conversion over HTTP	You still need a separate OCR stage and durable job state
A plain REST gateway such as Infrai	One HTTP boundary, so a worker can call it without an SDK	You still own the job state, indexing, and provider-fit tests

This is not a price contest. Your real cost is the number of integration edges you maintain and the hours spent diagnosing a stuck document. I am not sure which provider wins for your language mix or scan quality; your mileage will vary, and a small labeled test set will answer that faster than a spreadsheet.

What I would change when volume grows

At low volume, polling is easy to understand. At higher volume, I would add a bounded poll interval, a dead-letter view for repeated failures, and a retention policy for source files. I would keep the public contract unchanged: submit, receive a job id, read a terminal state.

I would also separate rendering from indexing. Rendering produces an artifact; OCR produces text; indexing makes it searchable. Splitting those steps means a failed index does not force a second render, and it gives me a clearer revenue-per-hour view of where work is accumulating.

The decision rule is short: if content size or OCR effort can surprise the caller, make it a job. Inline rendering is reserved for the small predictable edge. Everything else gets an id, observable states, and a deliberate failure path.

DEV Community: ValerianBlack3895

Why Image Batches Exist: Node.js Rate Limits, Progress, and Wrong-Import Cancellation

Why “batch” is a product boundary, not a loop

How should image batches handle rate limits, progress, and cancellation for wrong imports?

What I would change before this runs at scale

The trade-offs I keep visible

References

Failure Injection for an IoT Device Control Panel: Node.js Reconnect and Backfill

What should failure injection prove for an IoT device control panel?

Two viable architectures and their invariants

A Node.js failure-injection probe for reconnect behavior

When should you choose the specialist architecture instead?

A practical release gate

References

Node.js Operations View: One Channel for Devices or Per-Device Rooms

What actually changes when a connection comes back?

The smallest reconnect-safe Node.js boundary

Should one channel serve all devices or should each device get its own?

How do the vendor boundaries compare?

What I would change at scale

References

Collaborative Document Presence: Read Once, Then Let Events Tell You Who Is Online

Presence accuracy is a reconnect problem, not a first-paint problem

How do I get presence for a document and show who is online in Express?

What switching vendors actually costs you

What I would change at scale

Where this is the wrong call

References

Ordered Realtime State Changes and Security Controls in Node.js Explained

A small choice matrix for ordered chat state

How should realtime ordered state changes shape security controls for concert chat?

A reproducible recovery experiment

Where another choice is better

References

Staged SaaS Rollouts: Percentage Feature Flags Across US and EU Tenants

How should a SaaS percentage feature flag rollout work?

Two criteria decide whether this stays operable

A Node.js backend boundary that remains replaceable

When is a specialist the better choice?

The release runbook

References

Sources

Customer Domain Verification: Scheduled Polling and Triggered Rechecks for Tenant Onboarding

Who can actually change the record?

Should customer-triggered rechecks replace scheduled domain verification polling?

What does the verification worker need to remember?

Where does the manual-first approach win?

Further reading

DNS Automation in 2026: A 3-Gate Destructive Operation Allowlist

How should a DNS automation allowlist guard a destructive operation?

Put the three gates in the call path

The provider decision follows the operating model

When should the runner-up win?

References

Node.js Webhook Retry Queues: Exponential Backoff, DLQ Redrive, and Idempotent Delivery

How should Node.js webhook failures move through retry, backoff, and DLQ redrive?

Failure timeline: accepted, disconnected, duplicated

What does the TypeScript API code send to the queue?

Which provider should carry this weekly-digest delivery contract?

How should we test duplicate delivery before rollout?

References

Rollout notes and further reading

Tenant Subdomain Provisioning with Idempotent DNS Record Writes and Explicit Conflicts

The constraint is proof, not mutation syntax

How should Node.js provisioning choose DNS record create, update, or upsert?

The smallest working write

What I would change at scale

Direct provider APIs versus one shared surface

References

Apply Provider Routing Preference for Healthtech Billing — Write, Test, Read Back

The smallest safe change

How should a Node.js service apply and read back one capability at a time?

What does the test call prove, and what does it not prove?

Choosing the boundary for a solo SaaS

References

Document Rendering Jobs Explain Unpredictable Page Counts (A 3-State Model)

Why does document rendering belong in an async job for unpredictable page counts?

The smallest useful implementation

How do concrete documents change the timeout decision?

How do the main OCR options compare for a solo SaaS?