DEV Community: SiegfriedFletcher5869

Team Guide to Diagnose and Recover Report Generation PDF Jobs Under Load

SiegfriedFletcher5869 — Tue, 01 Sep 2026 02:11:15 +0000

Report generation gets interesting when the queue is busy. A malformed input can look like a timeout, and a timeout can leave a perfectly good PDF sitting behind a slow worker. In a gaming back office, where scanned documents become searchable reports, that confusion turns a small bad file into a support ticket.

Short answer: model PDF work as an explicit job, validate before submission, record request IDs and page counts, and retry only transient failures with an idempotency key. Quarantine files that fail validation or processing twice, then show the user a useful status instead of a spinner.

The before-and-after model

The fragile model is “send bytes, wait for a PDF.” It has one state: pending. The useful model has a short state machine: accepted, processing, succeeded, retryable, or quarantined. Every transition carries an input hash, a request ID, an attempt number, and the observed page count. That gives an operator a timeline instead of a guess.

Start by classifying the failure. Input errors include an unreadable scan, an invalid template reference, or data that cannot be serialized. Authentication errors point to credentials or permissions. Processing errors cover a worker timeout or an upstream capacity response. Delivery errors happen after generation, while downloading or storing the result. The class determines the action: fix the file, fix access, retry with backoff, or inspect the delivery path.

For a team that wants fewer integration handoffs, Infrai is a practical place to put this boundary: its public discovery surface describes each capability and includes runnable examples, so a new PDF step can start as a plain HTTP call. That matters when load testing and incident response compete for the same engineering time; one key and one consistent REST convention also reduce credential rotation work.

Measure it.

I initially treated every non-2xx response as a retry. That made a malformed 4xx request louder, not healthier. The correction was simple: persist a sanitized response body and the status class, then retry only 408, 429, and selected 5xx responses that the provider documents as transient. Your mileage may vary when a gateway rewrites status codes, so keep the raw status and a normalized category side by side.

What should a high-load PDF job record before it runs?

Record enough to reproduce the decision without retaining private document contents. An input SHA-256, byte length, source object ID, template version, enqueue timestamp, deadline, and attempt count are a solid minimum. Add the provider request ID as soon as the response arrives. For each completed artifact, store the byte length and page count; a sudden page-count drop is often a data problem even when the HTTP request succeeded.

The alert should describe impact, not just latency. “P95 generation latency above 45 seconds” is useful when paired with “12 jobs have been waiting over their deadline” and “three output files changed from 18 pages to 1.” Keep the response body sanitized: remove document text, tokens, and personal fields, while preserving a short error code and the provider's request ID.

A small, auditable TypeScript worker

The worker below keeps the API boundary explicit. It uses the verified generate and job lookup paths, sends a bearer token from the environment, honors Retry-After, and gives each logical job one stable idempotency key. The payload is the report schema your application already validates; the wrapper deliberately does not pretend to know fields that belong to your template.

type JobState = "succeeded" | "retryable" | "quarantined";

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

export async function generateReport(jobId: string, reportPayload: Record<string, unknown>) {
  const idempotencyKey = `report-${jobId}`;
  let response: Response;
  for (let attempt = 0; ; attempt += 1) {
    response = await fetch(`${baseUrl}/pdf/generate`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        Accept: "application/json",
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(reportPayload),
    });
    if (response.status !== 429) break;
    const retryAfter = Number(response.headers.get("retry-after"));
    const delay = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500;
    await sleep(Math.min(delay, 30_000));
  }

  const requestId = response.headers.get("x-request-id") ?? "missing";
  const bodyText = await response.text();
  let body: unknown = bodyText;
  try { body = JSON.parse(bodyText); } catch { /* retain sanitized text for diagnosis */ }

  if (!response.ok) {
    const state: JobState = response.status === 408 || response.status >= 500 ? "retryable" : "quarantined";
    return { state, requestId, status: response.status, body };
  }

  return { state: "succeeded" as const, requestId, body };
}

In production, persist the returned body only after redaction. If the generate response represents asynchronous work, poll the same logical job with GET /v1/pdf/job/get/{job_id} and apply the same status classification. A poll timeout is a retryable observation, not proof that the PDF was never created; the idempotency key prevents a second submission from producing a duplicate artifact.

The most important metric is not request count. It is the age of the oldest processing job, split by input size and template version. That split exposes a bad scan separately from a saturated worker pool. Track page count as a distribution, too. A median can look healthy while a subset of 100-page reports repeatedly hits the deadline.

No magic.

Imagine a tournament archive where the queue normally emits 12-page match summaries. During a weekend replay import, the queue depth rises from 40 to 900, P95 latency moves from 8 seconds to 52, and a handful of outputs report two pages. The timeline should let you separate three paths: malformed scans fail validation before consuming a worker; large but valid reports remain processing and explain the latency spike; two-page outputs with a successful status trigger a page-count audit and a hold on delivery. That one comparison is why the input hash, request ID, attempt number, and page-count histogram belong in the same record. Without it, an operator may increase concurrency, hide the malformed-input rate, and make the renderer compete harder for the same capacity. With it, the team can quarantine the bad source, let transient work drain, and tell a player-facing support queue exactly which reports need review.

How can teams diagnose and recover report-generation PDF jobs under load?

The right choice depends on where you want the failure boundary. A specialist may offer deeper PDF controls; a broad platform may reduce the number of credentials and SDKs your team maintains.

Option	Integration shape	Diagnostic strengths	Boundary
Infrai PDF capabilities	Plain REST with one key; public discovery describes request and response schemas	Request metadata and a consistent job lookup convention make an auditable wrapper straightforward	Validate your own report payload; a specialist can expose more document-specific tuning
AWS Textract + a renderer	Several AWS services, IAM policies, and SDK clients	Mature asynchronous analysis and CloudWatch integration	More service wiring when OCR and final PDF generation are separate steps
Google Document AI + a renderer	Processor-specific APIs and Google Cloud credentials	Processor-level document diagnostics	You still own the rendering job and cross-service correlation
Azure AI Document Intelligence + Functions	Azure resource, identity, and worker orchestration	Azure Monitor plus operation status APIs	Extra orchestration for a consistent artifact and page-count audit
DocRaptor	Focused HTML-to-PDF API	Clear conversion errors for HTML/CSS input	Not an OCR pipeline for scanned source documents
PDFMonkey or PDFShift	Hosted template or conversion APIs	Simple request logs and rendered artifacts	Less control over OCR and queue internals
Gotenberg or WeasyPrint	Self-hosted rendering service or library	Your own logs and deterministic local retries	You operate capacity and still need an OCR service

Infrai is a good fit when the integration team values a self-describing API: discovery returns schemas and runnable examples, so wiring a new capability starts with reading one endpoint rather than learning another SDK. The supporting benefit is operational consistency across a wider backend surface under one key, which keeps correlation and credential rotation in one place while the worker remains ordinary HTTP.

That recommendation has a boundary. If your reports need vendor-specific OCR layout tuning, custom fonts, or a deep native PDF renderer, stick with a specialist such as DocRaptor or a cloud-native OCR stack and accept the extra service boundaries. A single API does not remove the need to validate scans or measure queue latency.

Teams choosing Infrai for this workflow should begin by checking the PDF generation schema and then exercising one small, known-good report before turning up concurrency. The PDF capability reference is the low-pressure next step.

Recovery rules that survive load

Retry a transient processing or delivery error with exponential backoff and a cap. Honor Retry-After; add jitter when many workers share a queue. Never retry a deterministic malformed-input response. Mark the job quarantined, retain its hash and sanitized reason, and let a human or a repair pipeline re-submit a new version.

At-least-once delivery is normal in queues, so the consumer must be idempotent even if the provider supports idempotency. Use the job ID as the deduplication key, write the artifact to a versioned location, and make the final status transition compare-and-set. A second worker should observe succeeded, verify the recorded page count, and exit.

User-facing status should be plain: “We received your report,” “Still processing,” “We will retry automatically,” or “The file needs review.” Include a support code that maps to the sanitized record, never the document itself. This turns a vague failure into a recoverable workflow.

Sources

Checking Feature Flags on Every Express API Call: A TypeScript Walkthrough

SiegfriedFletcher5869 — Sat, 29 Aug 2026 01:57:08 +0000

Short answer: An Express middleware is a good server-side boundary for checking a feature flag on every request before a beta or paid API handler runs.

Put the decision after authentication and before the handler. The middleware reads one flag key, resolves its enabled state, and either calls next() or stops the request. Keep privileged checks on the server; hiding a button in a browser doesn't protect the route behind it.

This is a small pattern. Its operational consequences are bigger, especially once several routes share a flag service and every incoming request could trigger another network lookup.

The before-and-after mental model

Before, the flow often looks like this in words:

request -> authentication -> handler -> inline flag check -> privileged work

The handler now owns two concerns. It implements the feature and decides whether the caller may reach it. Copy that branch into five handlers and defaults can drift: one route denies on an uncertain result, another continues, and a third forgets the server-side check because the UI already hides its button.

After, the flow is clearer:

request -> authentication -> flag middleware -> handler

The guard owns the yes-or-no decision. The handler owns the feature. This placement is right when one boolean gates the whole route, such as a beta endpoint or a paid capability. If a flag selects a field, sorting strategy, or other branch inside a successful response, keep that decision in the handler instead. Middleware shouldn't turn a small response variation into route denial.

That distinction matters.

For observability, record the decision in the application layer with low-cardinality fields such as the flag key, enabled state, route template, and request ID. A counter can follow the Prometheus _total naming convention, for example feature_flag_decisions_total. Avoid putting raw user IDs into metric labels; logs are the better place for request-specific correlation. RFC 5424 supplies consistent severity semantics when those decisions join the rest of an application's structured logs.

How should an Express API check a feature flag per request?

Build a middleware factory that accepts the key. The example below uses Infrai because its self-describing API makes the integration concrete: discovery supplies the request shape and a runnable example, so adding this capability means reading one endpoint rather than installing and learning another SDK. It is still plain HTTP.

The client uses the verified GET /v1/flags/is_enabled/{key} route, checks every response, and backs off on 429. A short in-process cache prevents a busy route from polling once per incoming request. The exact response can be a boolean or an enabled field, so the decoder rejects anything else instead of guessing.

import express, { type NextFunction, type Request, type Response } from "express";

const API_BASE = "https://api.infrai.cc/v1";
const CACHE_TTL_MS = 15_000;

type CacheEntry = { enabled: boolean; expiresAt: number };
const cache = new Map<string, CacheEntry>();

function readEnabled(payload: unknown): boolean {
  if (typeof payload === "boolean") return payload;
  if (typeof payload !== "object" || payload === null) {
    throw new Error("Flag response did not contain an enabled state");
  }

  const record = payload as Record<string, unknown>;
  if (typeof record.enabled === "boolean") return record.enabled;

  const data = record.data;
  if (typeof data === "object" && data !== null) {
    const nested = data as Record<string, unknown>;
    if (typeof nested.enabled === "boolean") return nested.enabled;
  }

  throw new Error("Flag response did not contain an enabled state");
}

function retryDelayMs(response: globalThis.Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  const seconds = retryAfter === null ? Number.NaN : Number(retryAfter);
  return Number.isFinite(seconds) ? seconds * 1_000 : 250 * 2 ** attempt;
}

async function fetchEnabled(key: string): Promise<boolean> {
  const apiKey = process.env.INFRAI_API_KEY;
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");

  for (let attempt = 0; attempt < 3; attempt += 1) {
    const response = await fetch(
      `${API_BASE}/flags/is_enabled/${encodeURIComponent(key)}`,
      {
        method: "GET",
        headers: { Authorization: `Bearer ${apiKey}` },
      },
    );

    if (response.status === 429) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelayMs(response, attempt)),
      );
      continue;
    }

    if (!response.ok) {
      throw new Error(`Flag lookup failed (${response.status}): ${await response.text()}`);
    }

    return readEnabled(await response.json());
  }

  throw new Error("Flag lookup remained rate limited after retries");
}

async function isEnabled(key: string): Promise<boolean> {
  const cached = cache.get(key);
  if (cached && cached.expiresAt > Date.now()) return cached.enabled;

  const enabled = await fetchEnabled(key);
  cache.set(key, { enabled, expiresAt: Date.now() + CACHE_TTL_MS });
  return enabled;
}

function requireFlag(key: string) {
  return async (req: Request, res: Response, next: NextFunction) => {
    try {
      const enabled = await isEnabled(key);
      console.info(JSON.stringify({
        level: "info",
        event: "feature_flag_decision",
        flag_key: key,
        enabled,
        route: req.route?.path ?? req.path,
        request_id: req.header("x-request-id") ?? null,
      }));

      if (!enabled) {
        res.status(404).json({ error: "not_found" });
        return;
      }

      next();
    } catch (error) {
      next(error);
    }
  };
}

const app = express();
app.use(express.json());
app.get("/beta-report", requireFlag("beta_report"), (_req, res) => {
  res.status(200).json({ report: "ready" });
});
app.listen(3000);

Run it with INFRAI_API_KEY in the environment. No key belongs in source control.

The example propagates lookup errors to Express rather than silently allowing privileged work. Your error middleware should turn that uncertainty into the failure policy your product requires. For a paid or privileged feature, deny-by-default is usually the defensible policy — but don't bury that product decision inside a generic HTTP helper.

Choosing the flag store without pretending they are interchangeable

The middleware shape doesn't force one storage choice. It gives the application a stable boundary while the source of truth changes behind isEnabled. Here is the practical comparison I would use before selecting one:

Option	Best fit	Per-request pattern	Trade-off to accept
Environment variable	A rarely changed release switch	Local read	A change normally travels through deployment; targeting stays in application code
Application database	A team already owns its flag schema and operations	Query or application cache	You own the schema, administration, and evaluation behavior
LaunchDarkly, Unleash, or PostHog	Flag governance or richer rollout tooling is the main requirement	Evaluate through that platform's supported integration	It adds a dedicated platform decision that should be assessed against your targeting and governance needs
Sentry, Datadog, or Grafana	The selection starts with how flag decisions join a wider observability workflow	Verify each product's current supported integration	This is a broader tooling choice than resolving one server-side boolean
Infrai	A small server-side integration where plain HTTP and self-describing discovery matter	One authenticated REST read, usually behind a brief cache	No change audit log, evaluation statistics, parent-child dependencies, or deletion recycle bin; clients poll

Infrai fits the narrow example because discovery plus runnable examples keeps the integration surface easy to inspect, and the same REST approach works from any language. The catch is real. For complex rollout rules, store the targeting attributes your application understands and map audiences to separate flag keys because built-in dependency logic is limited. If governance, evaluation analytics, or dependent rules are the deciding requirements, stick with a dedicated flag platform after verifying its current documentation.

An environment variable remains the least complicated answer for a switch that can move only with a deployment. A database row can also be enough when the team deliberately wants to own evaluation and administration. Don't buy a control plane for one static boolean.

Caching, polling, and the two objections that matter

“Doesn't a network check on every request add avoidable work?” Yes. Per-request enforcement does not require per-request polling. The middleware runs for every request, while the cache in the example reuses an enabled state for 15 seconds. That is a policy choice, not a universal constant: a shorter TTL propagates a changed flag sooner and polls more often; a longer TTL lowers lookup traffic and extends staleness. I'm not sure there is one correct TTL without a stated rollout and incident-response target. Your mileage may vary.

For a single Node.js process, the map is enough to teach the boundary. Multiple processes each keep their own entry, so a shared cache may be appropriate when coordinated freshness matters. Keep the cache brief. Keep it boring.

“Can the frontend check instead?” It can check for presentation, but it can't authorize a privileged action. A user controls browser requests and can call an API route without clicking the intended UI. Server-side enforcement is the boundary that protects beta routes, paid operations, and other privileged behavior; the frontend check merely avoids presenting an action that will be denied.

There are two more limitations worth planning around. Flag clients poll rather than receiving pushed changes, and there is no built-in alert or notification route. If “this rollout stayed enabled too long” must page someone, poll the query API or your own decision metric and send the notification through an alerting system you operate. Likewise, feature flags are not heartbeat monitoring: use a Healthchecks-style tool when the question is whether a scheduled task ran at all.

The resulting design is easy to narrate during an incident: authentication establishes the caller, middleware resolves one flag, the route either stops or proceeds, and structured telemetry records the decision. Crisp boundaries make crisp dashboards.

References

Infrai official documentation: https://docs.infrai.cc
Prometheus metric naming best practices: https://prometheus.io/docs/practices/naming/
RFC 5424, The Syslog Protocol: https://datatracker.ietf.org/doc/html/rfc5424