DEV Community

XenonCross2718
XenonCross2718

Posted on

Flattened PDF order bundles: background merge jobs, status polling, and retention

#pdf #node #queues #storage

If you want an order-documents route that answers in milliseconds, use the least complex thing that gets you there: accept the request, write a job row, return a job id, and let the caller poll that id for status. No websocket. No holding an Express socket open for nine pages of PDF rendering while the load balancer counts down to a 60-second cut.

That part is easy. The bill is the interesting part, and almost nobody looks at it until the storage line stops being a rounding error.

The system I'm describing is an e-commerce returns portal. A shopper starts a return, and the backend has to fill a carrier's returns form — a real AcroForm PDF with field names nobody chose sensibly — flatten it so the values become page content instead of editable widgets, then merge it with the invoice and a packing slip into one downloadable bundle. Fill, flatten, merge, hand back a link. Four steps, one job.

What the order-bundle bill is actually made of

Take a store doing 40,000 returns a month. A three-document bundle lands somewhere around 2.4 MB once you've flattened the form and kept the invoice's embedded fonts. That's about 96 GB of new objects a month, and if you never delete anything, month twelve is carrying a little over a terabyte of PDFs that almost nobody opens twice.

Now compare the two cost terms. Rendering a bundle is CPU-seconds — call it a couple of seconds of a worker you already pay for. Keeping the bundle is GB-months, forever, compounding.

Retention is the dominant term. Not render.

That inverts the usual instinct. Most teams cache the finished bundle aggressively because rendering feels expensive, and rendering is the thing they can see in a flame graph. Storage is invisible until finance asks why the bucket grew 12x in a year. The change that actually moves the dominant term is boring: keep bundles for 30 days, keep the inputs to the job for much longer, and re-render on the rare late request. You're trading a few seconds of CPU on a cold hit for an order of magnitude less storage.

Which only works if a re-render produces the same bytes, and that is a fidelity question.

Fidelity versus render cost when you fill and flatten the form

Filling an AcroForm is cheap and deterministic — you're writing field values into an existing object graph. Flattening is where libraries diverge, because flattening means drawing the widget appearance streams into the page and then dropping the form. Fonts that were referenced by the widget now have to be embedded. Get that wrong and a French address renders with missing glyphs, which a carrier will reject and a customer will screenshot.

Here's the honest comparison of the options I'd actually shortlist:

Approach Integration Fidelity on flatten Best fit Main limitation
pdf-lib (Node, in-process) npm library Good for simple fields; appearance streams need care with non-Latin fonts Low volume, full control, no new infra You own font embedding and memory pressure
PyMuPDF / MuPDF Python binding High; mature widget rendering Python workers doing fill, flatten and merge together AGPL unless you buy a licence
Puppeteer or Gotenberg (HTML to PDF) Self-hosted container Excellent for documents you design, wrong tool for an existing AcroForm Invoices and packing slips generated from templates Headless Chrome is the heaviest thing in your cluster
DocRaptor Hosted HTML to PDF Excellent typography, Prince under the hood Teams who want print-grade output from HTML Not a form-filling API
Apryse SDK or hosted Very high, including signatures and redaction Regulated document workflows Commercial licensing, heaviest SDK footprint
Infrai POST /v1/pdf/merge Plain REST, no SDK to install Server-side, consistent across languages Teams who want the merge step to be one HTTP call from any runtime Hosted service, so bundles leave your network

For the returns portal, the split that held up was: render the invoice and packing slip from HTML with Gotenberg (they're our templates, so HTML is the right source), fill and flatten the carrier form with a PDF library that understands AcroForms, and treat the merge as a job.

A note on the last row, since the storefront is Express and the document worker is Python. The reason a hosted merge step is tolerable in a polyglot shop is that it's an HTTP call with a JSON body — no SDK to install, no second dependency tree, no version skew between the Node service and the Python worker. Infrai leans into that: its discovery endpoint is public and self-describing, so before wiring the call you read the capability's request schema, response schema and a runnable example, instead of learning another client library. That's the difference between an afternoon and a sprint when you add the fourth document type.

How should a Node.js service poll merge job status without blocking the Express request?

Three rules, and they're the whole production checklist for the job lifecycle.

Bound the poll. Back off. Have a give-up state that is a real state, not an exception that vanishes into your log aggregator.

The Express handler does almost nothing: validate the order, enqueue, respond 202 with the job id and a poll URL. The worker calls the merge, then polls. A client-supplied idempotency key on the enqueue means a retried request never produces two bundles for the same order revision — worth caring about, because standard queues are at-least-once and your worker will see the same message twice eventually.

import os
import time
import requests

API = os.environ["INFRAI_BASE_URL"].rstrip("/")   # base URL from the provider's docs
KEY = os.environ["INFRAI_API_KEY"]                # never hardcode; ifr_... lives in the env

SESSION = requests.Session()
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}


def start_bundle(order_id, revision, file_urls):
    """Enqueue the merge. Same order revision -> same key -> one bundle, however many retries."""
    idem = f"bundle-{order_id}-r{revision}"
    for attempt in range(6):
        r = SESSION.post(
            f"{API}/pdf/merge",
            headers={**HEADERS, "Idempotency-Key": idem},
            json={"files": file_urls},
            timeout=30,
        )
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"merge rejected {r.status_code}: {r.text[:300]}")
        return r.json()["job_id"]
    raise RuntimeError("merge enqueue rate limited after 6 attempts")


def wait_for_bundle(job_id, budget_seconds=180):
    """Poll with exponential backoff and a hard give-up, so no worker spins forever."""
    delay, spent = 1.0, 0.0
    while spent < budget_seconds:
        r = SESSION.get(f"{API}/pdf/job/get/{job_id}", headers=HEADERS, timeout=15)
        if r.status_code >= 400:
            raise RuntimeError(f"poll rejected {r.status_code}: {r.text[:300]}")
        state = r.json()
        if state["status"] in ("succeeded", "failed"):
            return state
        time.sleep(delay)
        spent += delay
        delay = min(delay * 1.6, 15.0)
    return {"status": "gave_up", "job_id": job_id}
Enter fullscreen mode Exit fullscreen mode

Field names in the response come from the capability's own schema — read it once when you wire the call, and pin what you depend on in a test, the same way you'd pin an API version.

Two details people skip. Store the input list with the job row, not just the output URL: when a bundle looks wrong in March you want to replay exactly what went in, and "the three files we merged" is not something you can reconstruct from the order later. And deliver the result through a short-lived signed URL against a private bucket — a returns bundle carries a name, an address and an order total, and a guessable public object key is how that ends up indexed.

I'd also put a dead-letter queue behind the give-up state rather than a retry loop. A merge that hasn't finished in three minutes is not going to finish in six, and the operator needs the job row, not another attempt.

What you stop keeping, and what it costs you at 2 a.m.

So: 30 days of bundles, 7 years of input references and job metadata, private bucket, signed URLs only. The metadata row is tiny — an order id, a revision, three input keys, timings, the idempotency key — so keeping it for the full accounting-records window costs essentially nothing.

Here's the catch, and it's a real one. On day 31 the bundle is gone, and when a customer disputes a return you're re-rendering from inputs that have themselves moved on. If the invoice template changed in February, the March re-render is not byte-identical to what the customer saw. That's not an academic worry — it's the exact thing a chargeback argument turns on.

Two ways out, and you pick by how much you're regulated. Either pin the template version in the job row so a re-render reproduces the original, or keep a hash of the delivered bundle and be willing to say in writing that the reissued copy is equivalent rather than identical. The first costs a little engineering discipline. The second costs you an argument you might lose.

And if your documents are legally required to be immutable — signed originals, tax invoices in jurisdictions with strict archival rules — this whole retention strategy is not a good fit. Stick with write-once storage and a longer bundle lifetime, and accept the storage bill as a compliance cost. A hosted merge API doesn't support that decision for you either; it merges, and where the archive lives stays your problem.

One last thing, less certain than the rest. I'd expect a well-tuned Python worker doing fill, flatten and merge in-process to beat any network round trip on latency for small bundles, and the hosted call to win the moment you need the same behaviour from three different runtimes without maintaining three toolchains. Where that crossover sits depends on your document sizes and your team, so measure it on your own bundles before committing. Infrai's pitch is the one-key, one-bill version of that trade — one credential and one integration across the backend capabilities a store needs, instead of a separate account, SDK and invoice per service — and that matters more to a four-person platform team than to a shop with a dedicated documents squad.

Queue the merge. Poll the id. Delete the bundle, keep the recipe.

Further reading

Top comments (0)