DEV Community

BrodyVance2149
BrodyVance2149

Posted on

Compress PDF Archives Explained: Store Originals for Fidelity and Control Storage Cost

#pdf #edtech #sre

Short answer: compress a verified copy for routine retrieval, but store the original PDF as the audit record until its retention period ends. For a monthly student report archive, that ordering protects fidelity without forcing every download to carry the largest possible file.

The decision is operational: if the copy cannot be regenerated and compared against the original, it is not an archive replacement.

This matters because a report is more than a page count. It can contain accommodation notes, attendance totals, charts, and a final grade whose visual relationship to its label matters. At 3am, I do not want a dashboard telling me that storage is healthy while the only surviving copy has lost a font, changed a page boundary, or made a table unreadable. I want to know which page fired, which artifact was promoted, and whether the source is still intact.

Should I compress PDF archives or store originals?

Treat the rendered PDF as an immutable artifact. Store its digest, report period, student or cohort identifier, generator version, and the inputs used to produce it. A compressed derivative belongs beside that record, with its own digest and a clear relationship to the original. Do not overwrite the original merely because the derivative is smaller.

I first thought the byte count was the useful success metric. It was not. A smaller file that loses the relationship between a grade and its label has saved storage while damaging the record.

The PDF specification defines the format's structure, but it does not turn visual equivalence into a single universal test. That is why the archive contract needs explicit acceptance criteria. Text extraction should retain required names, dates, totals, and headings. Page count should remain stable unless a documented policy permits a change. Important visual regions, such as a grade table or an approval block, need rendered-image comparison or human review for the cases that automation cannot judge reliably.

Compression is a transformation, not a storage setting. Lossless stream compression may leave visual content unchanged while reducing repeated structure; image recompression can change small text and chart edges. The correct choice depends on what the report contains and what a later reviewer must be able to inspect.

That tradeoff has a hard boundary: this pattern is a poor fit when policy requires one exact byte sequence and prohibits derivatives. In that case, store the original only and accept the storage burden; fidelity is the requirement.

Why does fidelity fail after a successful render?

The first render can be correct and the archive can still fail. A pipeline may select the wrong output, attach a derivative under the original name, or validate only that the object exists. A PDF that opens is not necessarily the PDF that the registrar approved.

There is a quiet failure mode in monthly jobs: the report template changes between generation and verification. If the worker records only a report ID and not the generator revision, a later re-render can look plausible while disagreeing with the archived artifact. Keep the original bytes and metadata together; reproducibility is useful evidence, but it is not a substitute for preserving the delivered file.

The page can be healthy. The artifact can still be wrong.

Check twice.

I use a promotion boundary with three states: rendered, verified, and archived. A failed verification leaves the original render available for diagnosis and prevents the derivative from becoming the canonical object. The alert should identify the report batch and the failed check. “Archive job failed” is a poor page; “12 of 2,400 derivatives changed page count” tells the person carrying the pager where to start.

A small promotion worker in Go

The worker below models the important boundary without prescribing a storage vendor. It writes the original first, derives a candidate, and promotes the candidate only after checks pass. The comparison function is deliberately an interface: production code should implement it with the text, page, and visual checks appropriate to the report class.

package archive

import (
    "context"
    "crypto/sha256"
    "fmt"
)

type ObjectStore interface { Put(context.Context, string, []byte) error }
type Verifier interface { Equivalent(context.Context, []byte, []byte) error }
type Candidate struct { Bytes []byte }

func Archive(ctx context.Context, store ObjectStore, verifier Verifier, key string, original []byte, candidate Candidate) error {
    if err := store.Put(ctx, key+"/original.pdf", original); err != nil { return fmt.Errorf("store original: %w", err) }
    if err := verifier.Equivalent(ctx, original, candidate.Bytes); err != nil { return fmt.Errorf("reject candidate: %w", err) }
    if err := store.Put(ctx, key+"/derivative.pdf", candidate.Bytes); err != nil { return fmt.Errorf("store derivative: %w", err) }
    manifest := fmt.Sprintf("%x  original.pdf\n%x  derivative.pdf\n", sha256.Sum256(original), sha256.Sum256(candidate.Bytes))
    return store.Put(ctx, key+"/manifest.sha256", []byte(manifest))
}
Enter fullscreen mode Exit fullscreen mode

The manifest is not a proof that two PDFs look identical. It is a durable statement of which bytes were checked. The verifier still needs a policy. For a text-heavy progress report, I would require exact required-field extraction and stable page count, then inspect rendered differences above a chosen threshold. For a chart-heavy report, I would add representative image comparisons because text-only checks can miss shifted axes or clipped legends.

How should the team verify and roll back?

Verification belongs in the same release path as the transformation, with a sample set that includes the awkward reports: long student names, empty sections, non-Latin text, page breaks near totals, and the largest chart. A small happy-path fixture is not enough. Record the original and derivative sizes, verification outcome, processing duration, and failure category. Storage cost is a useful signal, but it is not the acceptance criterion.

For a report set with unusually large images, the derivative may remain close to the original size. That is a normal result, not a reason to weaken the fidelity check.

Run the checks before changing the read path. During a rollout, readers should continue to use originals while a shadow process measures derivatives. Promote derivatives only after the sample set passes and the team has a retention policy for both forms. If a later audit finds a mismatch, switch reads back to originals, freeze derivative promotion, and retain the rejected bytes and manifest for investigation. Rollback should be a metadata decision, not a scramble to reconstruct deleted files.

The practical rule is compress copies, preserve provenance. Estimate storage using the actual report mix and retention window, but make fidelity failures expensive enough to detect before deletion. A monthly archive that is slightly larger and explainable is a better operational result than a smaller archive whose visual history cannot be defended.

References

Top comments (0)