DEV Community

DorianReed2186
DorianReed2186

Posted on

Original PDFs Over Compressed Archive Storage for Node.js Batch Form Flattening

#pdf #node #storage #archive

Store the filled document exactly as the pipeline produced it, and use a lossless structural rewrite as the only step allowed to compress an archive copy. Anything that resamples the images inside the PDF is trading fidelity — and batch throughput — for a storage line item that is usually the cheapest part of the whole system.

That's the decision. The rest is the arithmetic behind it, plus the places a logistics document pipeline breaks when you get it backwards.

The system I'll work through is a freight back office. A nightly job fills and flattens shipping paperwork — bills of lading, commercial invoices, delivery receipts — one document per shipment leg, say 40,000 of them, all of which have to land before the 06:00 cutoff when the first drivers start scanning. Every one of those files can be pulled back years later by a customs broker, an insurer arguing over a damaged pallet, or a court. So the archive isn't a backup. It's the record.

The flatten step that decides everything downstream

The flow is short enough to hold in your head: a template with an AcroForm sits in object storage, the job pulls a shipment row, fills the fields, flattens the form, hashes the result, writes the bytes to a hot bucket keyed by that hash, and records the key, hash, byte length, page count and template version in Postgres. A lifecycle rule moves the object to a colder tier after 30 days. Nothing else ever touches the bytes.

Flattening is the moment fidelity gets decided, because it's the last point at which the document is still structured data rather than ink.

import { createHash } from "node:crypto";
import { PDFDocument } from "pdf-lib";

interface ShipmentLeg {
  bolNumber: string;
  consignee: string;
  pieces: number;
  weightKg: number;
}

interface ArchiveRecord {
  bytes: Uint8Array;
  sha256: string;
  pages: number;
  scanHeavy: boolean;
}

// One filled, flattened document. What comes back here is the archival
// original; nothing downstream is allowed to re-render it.
export async function fillAndFlatten(
  template: Uint8Array,
  leg: ShipmentLeg,
): Promise<ArchiveRecord> {
  const doc = await PDFDocument.load(template);
  const form = doc.getForm();

  form.getTextField("bol_number").setText(leg.bolNumber);
  form.getTextField("consignee").setText(leg.consignee);
  form.getTextField("pieces").setText(String(leg.pieces));
  form.getTextField("weight_kg").setText(leg.weightKg.toFixed(1));

  // Flattening burns the values into page content and drops the AcroForm,
  // so no later reader can re-render a field with different fonts.
  form.flatten();

  // Object streams pack the many small indirect objects a form leaves
  // behind and Flate-compress them. Lossless: decoded page content is
  // byte-identical to what the renderer emitted.
  const bytes = await doc.save({ useObjectStreams: true });

  const pages = doc.getPageCount();
  return {
    bytes,
    pages,
    sha256: createHash("sha256").update(bytes).digest("hex"),
    // A generated form is text and vectors, so it stays small per page.
    // A page carrying a scanned JPEG lands an order of magnitude higher,
    // and that single bit decides which archive policy applies.
    scanHeavy: bytes.length / Math.max(1, pages) > 250_000,
  };
}
Enter fullscreen mode Exit fullscreen mode

If you want an extra pass for the text-heavy documents, do it out of process and prove it was lossless in the same breath:

qpdf --object-streams=generate --compress-streams=y --recompress-flate \
  --compression-level=9 filled.pdf archive.pdf

qpdf --check archive.pdf

pdftotext -layout filled.pdf - | sha256sum
pdftotext -layout archive.pdf - | sha256sum
Enter fullscreen mode Exit fullscreen mode

Those last two hashes have to match. That comparison is the whole fidelity assertion in one line, it costs almost nothing, and running it on a 1% sample of each batch is the difference between believing your archive is intact and knowing it.

Should you compress a PDF archive or store the originals in production?

Store the originals. Then allow exactly one class of compression on top: the kind that rearranges how objects are packed without changing what they decode to. Every other option on the table pays for bytes with something you can't buy back.

Policy What you keep Fidelity risk Cost during the batch window
Flattened original, as produced baseline bytes none none
Lossless object-stream rewrite meaningful cut on text-heavy forms, near zero on scans none; content decodes identically one CPU-bound pass
Image downsampling (Ghostscript /ebook and friends) large cut on scans permanent; 300 dpi drops to 150 dpi and OCR accuracy follows heavy
Full-page rasterization predictable size destroys the text layer and any signature heaviest
Keep shipment rows, re-render on demand almost nothing the re-render stops matching the document you sent deferred, and it lands mid-audit

That last row deserves more than a table cell, because it's the one a cost-obsessed engineer reaches for first. I did. Storing 40,000 JSON rows instead of 40,000 documents is obviously cheaper, and regenerating on request sounds like a caching problem.

It isn't. The template drifts — legal changes a disclaimer, someone swaps a logo, a font gets substituted on a rebuilt image — and eighteen months later your regenerated bill of lading is a different document from the one the consignee signed. You can version templates and pin the renderer to defend against that, and if your templates live in the same repo as the pipeline and get tagged with every release, the argument gets a lot stronger. Even then you're choosing to rebuild evidence under deadline, with a toolchain that has to still exist.

I'm not going to pretend that's never worth it. For internal picking sheets nobody will ever dispute, stick with re-rendering and take the savings. For anything a third party countersigned, storing the bytes is the cheap option disguised as the expensive one.

Where recompression actually costs you throughput

The cores that compress are the cores that fill and flatten. There's no separate budget.

Deflate at level 9 pays for its extra ratio with search effort, and on the wrong input it buys nothing at all: images inside a PDF are typically already DCTDecode streams, and re-running Flate over compressed JPEG data is close to pure waste. This is why the classifier in the code above matters more than the compression settings. Gate the extra pass on bytes-per-page, run it only on the text-heavy generated forms where it pays, and the scan-heavy documents skip straight to storage instead of eating the window.

Then there's the part of storage cost that isn't measured in bytes at all. Forty thousand objects a night is roughly fourteen and a half million objects a year, and at that count per-object overheads, listing costs and retrieval charges start to dominate the bill that compression was supposed to fix. Cold tiers add their own terms: minimum storage durations commonly run 30 to 180 days, so an object deleted early is still billed, and restore latency ranges from minutes to hours depending on tier. Packing a day's output into a single container object makes the per-object math disappear and makes retrieving one bill of lading a nightmare. That trade-off is yours to make deliberately, not to discover during an audit.

Storage is cheap. Retrieval under deadline is not.

Fidelity that customs brokers and auditors actually check

A digitally signed document is signed over a byte range, so any rewrite invalidates the signature — including a lossless one. Order of operations is therefore not negotiable: fill, flatten, compress, then sign. Compress after signing and you've turned a valid signature into a broken one, which is worse than no signature because now it looks like tampering.

Brokers search these files. They type a B/L number into a viewer and expect a hit, which works because flattened text stays real text in the content stream. Rasterize the page to save space and that search becomes an OCR project, run years later, on a document whose accuracy nobody can vouch for. Archival profiles like PDF/A-3 exist to lock this down: embedded fonts, device-independent colour, no external dependencies. Lossy normalization steps are exactly where those guarantees quietly get dropped.

The catch is inbound photographs, and it's a real limitation on everything above. Proof-of-delivery images from driver phones arrive as 4000-pixel JPEGs with no text layer worth preserving, already lossy, and keeping them at capture resolution for seven years is spending real money on camera noise. Downsample those once at ingest — before you hash, before you sign, and recorded in the ledger as the normalization step it is — and treat the result as the original. Same reasoning applies if your retention is 90 days rather than seven years: the archival argument mostly evaporates, and the right answer becomes whatever your batch window can produce fastest.

The operational rules that keep this honest are short. Write the hash, page count, template version and pipeline version to the ledger before the object reaches cold storage, because an archive you can't attribute to a build is an archive you can't defend. Sample every batch through the text-extraction comparison rather than trusting the flags you set months ago. Alert on the bytes-per-page distribution, since a shift there means a template changed or someone adjusted a scanner, and both show up as fidelity problems long before anyone complains. And restore something from the cold tier on a schedule — quarterly is enough — because an archive nobody has read back is a hypothesis, not a record.

Compression is a decision about what you're willing to lose. In freight paperwork, the answer is usually nothing.

Further reading

Top comments (0)