The first test suite I wrote for a PDF pipeline compared generated output against a saved fixture file, byte for byte. It failed constantly, not because the pipeline was broken, but because the underlying library embedded a timestamp, a generator string, or a slightly different compression pass on every run, none of which had anything to do with whether the output was actually correct.
Why byte-diffing is the wrong instinct here
A PDF file isn't a deterministic serialization of its content the way JSON or a plain text file is. Two PDFs can be visually and functionally identical, same pages, same text, same layout, and still differ at the byte level because of metadata, internal object ordering, or compression choices made by whatever tool produced them. Testing for byte equality tests the wrong thing: it asserts on the encoding, not on the content, and it breaks on every harmless change to how the encoding happens.
What actually matters about a generated PDF
The useful question for a test isn't "is this file identical to a reference file," it's "does this file have the properties I actually care about." For most PDF pipeline tests, that decomposes into a short, specific list: the right number of pages, the right text content somewhere in it, a file size that's plausible for what was requested, and a success response from whatever produced it.
from pypdf import PdfReader
def test_merge_produces_expected_page_count():
result = pdf_api.run({"action": "merge", "files": [FILE_A, FILE_B, FILE_C]})
assert result.status == "success"
reader = PdfReader(result.local_path)
assert len(reader.pages) == EXPECTED_TOTAL_PAGES
def test_merge_preserves_text_content():
result = pdf_api.run({"action": "merge", "files": [FILE_A, FILE_B]})
reader = PdfReader(result.local_path)
full_text = "".join(page.extract_text() for page in reader.pages)
assert "Invoice #4471" in full_text
assert "Invoice #4472" in full_text
Neither test cares what the PDF's internal structure looks like. Both care about the thing a human checking the output by hand would actually check.
Testing compression without a brittle size assertion
Compression is the operation most tempting to test with an exact number, and the operation where an exact number is most likely to break for reasons that have nothing to do with a real regression. A slightly different compression library version can shift output size by a few percent without any actual quality change. The useful assertion is a range, not a point:
def test_compress_reduces_size_meaningfully():
original_size = FILE_A.stat().st_size
result = pdf_api.run({"action": "compress", "file": FILE_A})
compressed_size = result.local_path.stat().st_size
assert compressed_size < original_size * 0.7 # meaningfully smaller
assert compressed_size > original_size * 0.05 # not suspiciously tiny, likely corrupted
The lower bound matters as much as the upper one. A compression bug that produces a near-empty, corrupted file will pass a test that only checks "smaller than before," which is exactly the kind of bug you actually want a test to catch.
Testing failure paths, not just success paths
It's easy to write tests that only exercise the happy path, valid files, reasonable batch sizes, no surprises, because those are the cases you have on hand while writing the tests. The failures that actually show up in production are almost always the ones that weren't in the happy-path fixture set: a password-protected file, a zero-byte upload, a file with a .pdf extension that isn't actually a PDF.
def test_merge_rejects_non_pdf_gracefully():
result = pdf_api.run({"action": "merge", "files": [FILE_A, FAKE_PDF_FILE]})
assert result.status == "invalid_input"
assert "FAKE_PDF_FILE" in result.error_detail
def test_merge_handles_encrypted_file():
result = pdf_api.run({"action": "merge", "files": [FILE_A, PASSWORD_PROTECTED_FILE]})
assert result.status == "encrypted_input"
Asserting on the specific failure reason, not just "it didn't crash," is what makes these tests useful for catching regressions in error handling specifically, which is usually the part of a pipeline that degrades quietly over time as new code gets added around it.
Testing visual properties without a full rendering diff
Some regressions genuinely are visual, a watermark shifted position, a page rotated the wrong direction, and a text-only assertion won't catch them. The instinct here is often to render both PDFs to images and do a pixel diff, which reintroduces the same fragility as byte-diffing, since anti-aliasing and rendering library versions shift pixels in ways that have nothing to do with correctness. A narrower, more targeted check tends to hold up better: render just the page in question, and assert on a specific, bounded region rather than the whole image.
def test_watermark_appears_in_expected_region():
result = pdf_api.run({"action": "watermark", "file": FILE_A, "text": "DRAFT"})
image = render_page_to_image(result.local_path, page=0)
bottom_right_region = crop(image, x=0.7, y=0.85, w=0.3, h=0.15)
assert region_contains_text(bottom_right_region, "DRAFT")
That's a meaningfully smaller assertion than a full-page pixel comparison, and it only fails when the thing it's actually checking, the watermark's presence and rough position, is actually wrong.
Keeping fixture files honest
A small, deliberately curated set of fixture PDFs, one clean multi-page file, one password-protected file, one corrupted file, one with unusual page sizes, covers most of what a pipeline needs to be tested against, and it's worth checking those fixtures into the repo rather than generating them fresh in every test run, so a test failure means the pipeline changed, not that the fixture generation did.
Why this is worth the setup
None of this testing approach requires understanding PDF internals, which is exactly the point. The pipeline calls a testable PDF API covering merge, split, compress, rotate, watermark, and convert, and the tests assert on the API's own structured response plus a few content-level checks on the output, never on the file's internal encoding. That's a test suite that survives library upgrades, compression setting changes, and format version bumps, because it was never coupled to any of those things in the first place.
Top comments (0)