DEV Community

hao jia
hao jia

Posted on

Your PDF is already compressed: unpacking every stream made a text-only file 2.25 larger

A lossless "optimize PDF" step looks free to add to a document pipeline, so the real question is how much is left for it to find. One way to get a number is to run the process backwards first. qpdf 12.4.0 with --compress-streams=n --decode-level=generalized writes every Flate stream back out uncompressed. A ten-page, text-only file of 137,782 bytes (X1 below) came out at 309,649 bytes, 2.25 times the original. A constructed file storing one image and one font subset five times each (D1, 756,741 bytes) came out at 2,363,800, or 3.12 times. So these files had their big lossless win long before any optimizer saw them. What a structural rewrite can still catch is the part the producer compressed badly, plus the part it stored twice.

Five files went through this on one Apple-silicon Mac, each tool run once, so every figure means "on these samples". Besides X1 and D1, T1 is two A4 pages of text with one photo (2,312,093 bytes), T2 is the same pages as a 300 DPI scan (2,357,639 bytes), and X2 is an eight-page fact sheet with images and no embedded fonts (1,497,423 bytes). Every qpdf output kept its page count and text, and rendered pixel-identical at 50 dpi. Image-and-text sample: NASA/JPL Mars Exploration Rover NASA Facts (2004), Wikimedia Commons, public domain. The other samples are self-made with fictional content. The photo in T1 comes from Wikimedia Commons (CC0).

Unpacking the streams shows what was already done

On the other three files the same control is milder: T1 grew 14.46%, X2 3.07%, and the scan T2 went to 2,357,711 bytes, a rounding error. Sorting bytes by object type, with offsets from qpdf --show-xref, explains why. In X1, fonts are 121,442 bytes (88.14%), content streams 9.16%, page and structure objects 1.95%, and the xref table plus trailer 0.71%. Across X1, T1, X2 and T2, the structural part (page objects, xref, header) is 0.04% to 2.70% of the file. Deleting all of it would still leave you within three percent of where you started.

Level 9 Flate found about 1 percent

Running --recompress-flate --compression-level=9 on its own took X1 to 136,255 bytes (−1.11%) and X2 to −0.11%, left T2 where it was, and made T1 and D1 slightly larger (+0.07% and +0.99%). The larger results come from the rewrite itself: a plain qpdf in out with no options already put T1 at +0.07% and D1 at +1.03%, so recompression at level 9 won back almost nothing on top of what the rewrite cost.

Object streams found about 4 percent

--object-streams=generate alone was the biggest structural switch on every file: X1 132,065 bytes (−4.15%), T1 −0.74%, X2 −0.25%, T2 −0.02%, and D1 −16.26%. With level-9 recompression stacked on, X1 reached 130,933 (−4.97%). PyMuPDF 1.26.5 with garbage=4, deflate and use_objstms gave 132,169 (−4.07%). Where files may not leave the network, the useful browser-side comparison is a local one. ImgIng https://imging.ai/ , which I use for PDFs, processes files locally in a desktop browser, and in its lossless-structure mode X1 went to 131,156 bytes (−4.81%) and the scan T2 moved 0.03%, again the same order as qpdf.

Duplicates are the other target

PyMuPDF's save() merges duplicate non-stream objects at garbage=3, which took D1 down 13.69%. At garbage=4 it also merges duplicate streams, and D1 went from 756,741 to 152,455 bytes (−79.85%), a figure from a file built to have duplicates, not an average. On the other four files the garbage levels changed almost nothing. None of the qpdf options used here merged duplicate streams: with all of them stacked D1 stayed at 633,342 bytes and the five copies of the image were still there. Other qpdf options were not tried.

What was measured and what is inferred

Measured: the sizes, page counts, text and render comparisons above, plus one byte split of X1's original: its non-stream objects come to 6,197 bytes (4.50%), 3,512 of them font-related. Also measured, and unexplained: the macOS Reduce File Size filter turned D1 into 1,275,546 bytes, 68.6% larger than the input. Nobody has looked into why yet.

Inferred, not verified. The small level-9 gain suggests the producers already used a reasonable level, but nobody read the original level out of the files. The object stream gain fits the usual explanation: a classic PDF writes dictionaries and arrays such as a font's /W widths as plain text outside any filter, and packing them into an object stream finally puts them under Flate. On D1 the font-related bytes fell from 266,381 to 134,332 with object streams on, which points that way. So does X1: non-stream objects plus xref, trailer and header add up to 5.25% there, just above the 4.15% that object streams saved. T2 not growing when decompressed probably means its page images use a filter that the generalized decode level leaves alone, and that was not checked either.

For a pipeline, the cheap check is to decompress a few representative files and sort their bytes by type before turning anything on. If fonts and images are nearly the whole file, a lossless rewrite tops out at a few percent, mostly from object streams. If the same logo or font subset repeats across pages merged from separate documents, a duplicate-stream pass is the switch that matters. Anything beyond that means re-encoding images, a lossy decision with its own review.

Top comments (0)