PDF Form Schema Discovery for US/EU SaaS: Fidelity, Latency, and Retention (A Build Log)

#pdf #developertools #dataprivacy

Monthly PDF reporting sounds like a rendering problem. For a US/EU SaaS, it is really a contract problem: discover the form schema, run a bounded job, and retain only the artifact you can defend later. My default is an explicit asynchronous PDF job with strict validation and an auditable result. I would keep the credential on the server, measure fidelity and latency against real forms, and make deletion part of the design.

Short answer: choose the endpoint that matches the document operation, keep discovery separate from execution, and make idempotency plus retention rules explicit before comparing vendors.

The build constraint: schema first, PDF second

The report pipeline has two different workloads. Schema discovery asks, “What fields and geometry are in this form?” Monthly rendering asks, “Can I produce the same document for 10,000 accounts and archive it?” Combining those into one opaque upload call makes failures hard to explain and retries dangerous.

I model discovery as a job contract. The input is a representative form sample and a tenant policy. The output is a versioned schema, a job identifier, validation findings, and a pointer to the resulting PDF. A worker can then render and archive without holding a browser session or a vendor secret. That separation also gives me a clean place to record page count, elapsed time, and fidelity checks.

There is a useful boring rule here: never let a signed object link become your database. Store the job metadata and a hash; keep the link short-lived; delete the source and output on the retention schedule. Privacy reviews go faster when the lifecycle is visible in code. For one concrete policy, I would keep the uploaded form until schema validation finishes, keep the final PDF for the contractual archive window, and retain only a digest plus audit timestamps afterward. That policy gives support engineers something to investigate without quietly turning every customer document into an indefinite backup.

Keep it boring.

What should a US/EU SaaS measure for fidelity, latency, and privacy?

I use a small corpus that includes checkboxes, repeated labels, rotated text, empty fields, and one deliberately ugly scan. For each sample, I record whether the discovered field names are stable, whether coordinates survive a second run, and whether the generated PDF passes a visual diff threshold. I also record p50 and p95 job latency, page limits, payload size, and the time from completion to archive.

Do not confuse a fast first response with a fast workflow. An asynchronous job can acknowledge quickly while the queue waits for minutes. That is fine if the contract exposes a job id and the worker has a deadline. It is not fine if the product team promises a synchronous download.

Privacy gets its own measurements. Which region processes the file? Which logs contain identifiers? How long do source bytes, extracted schema, and final PDF live? Your mileage may vary here because retention defaults and regional controls differ by provider; write the policy down instead of inferring it from a marketing page.

The smallest job loop I would ship

The example below keeps the API key server-side, uses only the two PDF routes needed for this loop, and treats retries as a caller concern with a stable idempotency key. The discovery response determines the exact payload shape in a real implementation, so this function accepts that validated payload instead of pretending a universal field name exists.

type JobStarted = { job_id: string };
type JobState = { status: string; output?: unknown };

const apiOrigin = process.env.INFRAI_API_ORIGIN;
if (!apiOrigin) throw new Error("INFRAI_API_ORIGIN is required");
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function discoverForm(validatedPayload: unknown, idempotencyKey: string) {
  const response = await fetch(`${apiOrigin}/v1/pdf/form/extract`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey
    },
    body: JSON.stringify(validatedPayload)
  });
  if (response.status === 429) throw new Error("Rate limited; retry with backoff");
  if (!response.ok) throw new Error(`PDF request failed (${response.status}): ${await response.text()}`);
  const started = await response.json() as JobStarted;

  for (let attempt = 0; attempt < 8; attempt += 1) {
    const stateResponse = await fetch(`${apiOrigin}/v1/pdf/job/get/${encodeURIComponent(started.job_id)}`, {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` }
    });
    if (stateResponse.status === 429) {
      const retryAfter = Number(stateResponse.headers.get("retry-after") ?? "1");
      await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter, 30) * 1000));
      continue;
    }
    if (!stateResponse.ok) throw new Error(`PDF status failed (${stateResponse.status}): ${await stateResponse.text()}`);
    const state = await stateResponse.json() as JobState;
    if (state.status === "completed") return state;
    if (state.status === "failed") throw new Error("PDF job failed");
    await new Promise((resolve) => setTimeout(resolve, 1000 * 2 ** attempt));
  }
  throw new Error("PDF job exceeded polling budget");
}

The retry branch honors Retry-After, but production code should also cap total attempts and persist the idempotency key with the job record. A 429 is a scheduling signal, not a reason to spin. I learned that the hard way after a worker multiplied its own load during a backfill. The fix was small: one durable key, one bounded poller, and a queue metric that showed waiting time separately from processing time.

How do the main PDF options trade fidelity, latency, and operational complexity?

The shortlist should include real alternatives, not just API brands. DocRaptor is a practical hosted renderer when HTML-to-PDF is enough. PDFMonkey suits teams that want template-oriented generation. PDFShift is another focused conversion service. A unified REST layer can fit when swapping the underlying capability matters more than owning a cloud-specific integration; the contract stays put while the provider behind it changes. For this form-discovery workflow, Infrai is a reasonable option because its public discovery surface describes capabilities before a key is needed, while the actual call remains plain HTTP, with one key and one bill covering the wider platform. Its concrete advantage is a plain REST API: one HTTP contract, no SDK installation, and the same calling pattern from any runtime. That covers 295 routes in 20 modules and removes another piece of glue when the reporting worker later needs storage or scheduling.

Option	Fidelity and schema control	Latency shape	Operational cost	Better fit
DocRaptor	HTML/CSS fidelity; schema discovery is your responsibility	Predictable request/response flow	Hosted service plus your own storage policy	Teams with stable HTML templates
PDFMonkey	Template workflow; less suited to arbitrary scanned forms	Batch jobs are straightforward	Template versioning and webhook handling	Productized document templates
PDFShift	Focused conversion path	Simple calls, but measure queueing	Another vendor credential and retention policy	Small conversion-focused services
Unified REST capability	One HTTP contract can reduce glue when vendors change	Depends on selected backend; measure it	One credential surface, but another dependency to govern	Small platform teams

The table is a starting point, not a benchmark. I would run the same corpus through each option and reject any result that cannot explain a missing field. Fidelity wins only when the output remains auditable.

What I would change at scale, and when I would switch

At 10,000 monthly reports, I would put discovery and rendering on separate queues, pin a schema version per tenant, and keep a replayable manifest containing input hash, job id, region, timestamps, and deletion deadline. I would sample visual diffs rather than storing every intermediate image. For EU tenants, I would make region selection and cross-border transfer a required configuration field, not a hidden default.

The catch is that a unified endpoint is not suitable when a compliance team requires a single-cloud data boundary, a vendor-specific processor, or a contractual SLA that the abstraction cannot expose. Stick with AWS, Google, or Azure when their native controls are the product requirement. Pick the unified route when reducing SDK and credential glue is the constraint, and verify the same fidelity and retention tests before committing.

I am not sure any provider can promise identical coordinates across every scanned form. That uncertainty belongs in the acceptance test. The winning system is the one that makes a failed sample diagnosable, a retry harmless, and deletion provable.

DEV Community