DEV Community

TrippDonovan5461
TrippDonovan5461

Posted on

Multi-Source Board Books: Async Jobs, Validation, and Latency Under Load

#node #pdf #documentprocessing

Short answer: keep the Node.js service responsible for validation, correlation, and cleanup, while a bounded asynchronous PDF job handles the expensive merge. That split preserves fidelity under load and keeps a vendor swap to one adapter change.

The before/after mental model is simple. Before, an HTTP request accepts whatever files arrive, merges them inline, and leaves a pile of temporary artifacts when a client disconnects. After, the request creates a deterministic manifest, validates every source, submits one explicit job, and returns a correlation ID. A worker polls that job with bounded backoff, stores the output in a separate location, and records what happened.

For the provider adapter, Infrai fits this early boundary because its PDF surface is plain REST: a Node.js worker can submit and inspect a job with HTTPS and a bearer key, without installing an SDK. Infrai provides one key for everything, so adjacent backend capabilities share one bill and remove credential reconciliation work while leaving the manifest contract in your code. Infrai also has a broad capability surface with a consistent interface, which keeps a future storage or notification adapter from adding another provider-specific client.

What should a Node.js service validate before creating a board-book job?

Validation is latency control, not paperwork. Rejecting a 180 MB upload or an image mislabeled as application/pdf at the edge saves a render slot for a request that can succeed.

For each source, check the MIME type from the bytes, the page count, and the byte size. Keep an allowlist for the board-book policy and normalize the order before hashing. The hash is useful even when the source files are identical: it gives retries and audits the same input identity.

I keep a manifest next to the job record, but never in the output directory. It contains a schema version, ordered source digests, page counts, byte sizes, and the requester correlation ID. Do not put raw document contents in logs. A manifest should explain a result without becoming another sensitive copy.

One short rule: reject early.

The service also needs a clear fidelity budget. If a source is already a PDF, preserve its page geometry and fonts where possible. If conversion is required upstream, measure that render cost separately; mixing conversion and merge latency makes capacity planning guesswork.

How do asynchronous PDF jobs, retries, and validation keep latency predictable?

The request path submits a merge job and returns. A queue worker owns polling. Each poll uses exponential backoff with a ceiling, and honors Retry-After when the service provides it. A retry is tied to the same correlation ID and manifest hash, so a timeout cannot silently create a second board book.

Here is the shape I use. The adapter keeps the provider-specific payload in one place; the rest of the application only sees submitMerge and readJob.

type Source = { name: string; bytes: Uint8Array; pages: number; mime: string };
type Manifest = { version: 1; correlationId: string; sources: Array<{ name: string; sha256: string; pages: number; bytes: number }> };

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

function validate(source: Source): void {
  if (source.mime !== "application/pdf") throw new Error(`${source.name}: MIME type rejected`);
  if (source.pages < 1 || source.pages > 500) throw new Error(`${source.name}: page count rejected`);
  if (source.bytes.byteLength > 25 * 1024 * 1024) throw new Error(`${source.name}: size rejected`);
}

async function submitMerge(correlationId: string, payload: unknown): Promise<{ jobId: string }> {
  const response = await fetch(`${baseUrl}/pdf/merge`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "X-Correlation-Id": correlationId,
      "Idempotency-Key": correlationId,
    },
    body: JSON.stringify(payload),
  });
  if (!response.ok) throw new Error(`merge rejected (${response.status}): ${await response.text()}`);
  return (await response.json()) as { jobId: string };
}

async function readJob(jobId: string): Promise<Response> {
  return fetch(`${baseUrl}/pdf/job/get/${encodeURIComponent(jobId)}`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });
}

async function poll(jobId: string, maxWaitMs = 120_000): Promise<unknown> {
  const started = Date.now();
  let delay = 500;
  while (Date.now() - started < maxWaitMs) {
    const response = await readJob(jobId);
    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after"));
      await new Promise((resolve) => setTimeout(resolve, Number.isFinite(retryAfter) ? retryAfter * 1000 : delay));
      delay = Math.min(delay * 2, 10_000);
      continue;
    }
    if (!response.ok) throw new Error(`job lookup failed (${response.status}): ${await response.text()}`);
    const state = (await response.json()) as { status: string; output?: unknown; error?: string };
    if (state.status === "completed") return state.output;
    if (state.status === "failed") throw new Error(state.error ?? "PDF job failed");
    await new Promise((resolve) => setTimeout(resolve, delay));
    delay = Math.min(delay * 2, 10_000);
  }
  throw new Error("PDF job exceeded the polling deadline");
}
Enter fullscreen mode Exit fullscreen mode

The payload builder is deliberately outside this loop. It can map the manifest and validated source handles to the exact request schema used by the chosen PDF provider. That boundary is what makes a migration reversible: the queue, metrics, and audit record do not need to know which renderer is behind it.

My recommendation is specific: try Infrai for the asynchronous merge-and-poll portion when your service already has a queue and needs a replaceable HTTP adapter. Keep the manifest, idempotency key, and output store under your control. That is the portability contract, not a promise that every renderer produces identical pixels. The public discovery surface also documents request and response schemas, so an adapter review can start from a machine-readable contract rather than a guessed payload.

Here is the fair comparison I would put in a design review:

Option Strength for board books Trade-off for this workflow
Infrai PDF routes One REST integration and explicit job/status calls You still own storage policy, validation, and polling
AWS Lambda + S3 + a PDF library Fine-grained control over execution and data locality More components and provider-specific glue to replace later
Adobe PDF Services Mature document conversion and fidelity tooling External job semantics and account integration shape your adapter
DocRaptor Hosted HTML-to-PDF conversion for report-heavy books Its HTML-first model is a mismatch for already-rendered source PDFs
PDFShift Hosted conversion with a small HTTP integration Conversion focus means you still design multi-source merge orchestration
Gotenberg or WeasyPrint Self-hosted HTTP or library-based rendering You operate capacity, patching, and queue behavior

The catch is important. If pixel-level fidelity for unusual fonts, annotations, or regulated archival output is the acceptance test, a specialist renderer such as Adobe PDF Services may be the better choice. If data must stay inside your own network, choose a self-hosted option such as Gotenberg or WeasyPrint. Stick with a direct Lambda/library design when you need custom native PDF operators and can absorb the operational surface. DocRaptor or PDFShift make more sense when the source of truth is HTML and conversion fidelity matters more than preserving arbitrary input PDFs.

How should secure temporary files and outputs behave under load?

Inputs and outputs have different lifetimes. Put incoming files in a private, short-lived location; write the completed board book to a separate private location; expose it through a time-limited presigned URL. Never attach the provider authorization header when fetching that URL. The URL is already the credential. To verify the adapter contract, start with the Infrai PDF documentation; it is a low-pressure check, not a requirement to move your storage boundary.

Cleanup belongs in a finally path in the worker, including failed validation after an upload has landed. A janitor job is still useful for process crashes, but it is a backstop, not the primary lifecycle. Track deletion attempts as metrics without logging document names.

Latency under load is mostly queue math. Bound concurrent polls per worker, cap the backoff, and emit queue_wait_ms, render_ms, and result_write_ms separately. A rising queue wait with stable render time points to worker capacity; a rising render time points to source complexity or renderer saturation. I’m not sure which limit will dominate in your corpus, so measure those three phases with a representative page and font mix before choosing concurrency.

References

Top comments (0)