DEV Community

Haelion14
Haelion14

Posted on

Cutting PDF storage cost in a signed-contract archive — the S3 and Go approach

#pdf #storage #go #architecture

The cheapest contract archive I can defend at a budget review keeps every executed PDF exactly once, byte for byte, and treats everything else in the bucket as a cache. Use a compress-PDF API on the derived copies — the preview, the enrollment packet a parent asks for, the term-end bundle — and never on the executed original. That one rule reduces storage cost more than any encoder setting, and it's the approach that leaves the audit trail standing.

The page that fires after a well-meaning cleanup job

Picture the 02:40 page on an edtech platform that signs enrollment and tutoring contracts server-side. The alert is signature_verification_failed, firing on documents pulled for a district audit, and the count is climbing rather than flat. On-call opens the runbook, downloads one contract, and it looks completely normal: the text renders, the signature widget is in the footer, the signer name and timestamp are right. The verifier still refuses it, because the digest no longer covers the file it is bound to. Upstream, a nightly job walked the archive, pushed every PDF through a re-encoder to claw back space, and wrote the smaller output back under the same key.

That's the page you can't fix at 02:40.

A PDF signature covers bytes, not appearance. The signer's digest runs over a byte range of the file with a gap cut out for the signature container itself, which is how ISO 32000-2 defines it in clause 12.8. Re-encode an image, subset a font, rebuild the cross-reference table, and every offset after the change moves; the digest computed at verification time no longer matches the one inside the signature. The only sanctioned way to change a signed PDF is an incremental update that appends to the end of the file — later signatures can be added, earlier ones stay verifiable, and nothing that was already digested is rewritten.

What should you compress in a PDF archive, and what document bytes stay untouched?

Classify every object in the archive before any job is allowed to read it. Authoritative objects are the executed contracts, the ones a district lawyer will ask for in three years. Derived objects are previews, thumbnails, merged packets, and any rendition a service can produce again from inputs you still hold. Authoritative bytes are immutable by construction — put them under an object-lock retention policy so a cleanup job physically cannot overwrite them, rather than trusting a code review to catch the next well-intentioned optimization.

Derived objects are where a compression API earns its keep, and honestly where most archives are fat.

The "no Puppeteer" constraint in the question is the easy part. Headless-Chrome rendering is a generation concern that happens before the file exists, while compression is a file-level transform over an object you already have, so the two decisions are independent. What actually drives the size of a document archive is rarely the vector text: it is scanned raster pages, full font files embedded once per rendition instead of subset, and the same layout stored four times because four services each wrote their own copy. For capacity planning I work in bands rather than pretending to a measurement — a text-native contract generated from a template sits in the low hundreds of kilobytes, and a 300 dpi scan of the same pages sits in the low megabytes, roughly an order of magnitude apart. Fixing the scan path pays for itself; squeezing the template output does not.

There is a floor worth knowing before anyone writes a re-encoding job. Infrequent-access and instant-retrieval storage classes bill a minimum of 128 KB per object regardless of the real size, and cold classes add a per-object metadata overhead on top. Compress a 140 KB contract down to 60 KB and the line item does not move. The trade-off only pays on large derived objects, and the archive job should refuse to touch anything under that floor instead of burning CPU to produce a rounding error.

Template ownership decides what you are allowed to delete

This is the axis that actually settles the design, and it is a buy-versus-build question rather than an encoding one. If you can reproduce a document, you don't have to store it. If you cannot, every byte is permanent, and no compression ratio saves you from a retention schedule measured in years.

Template ownership What the archive must keep permanently Dominant cost driver Regeneration story
Third-party owns the template and the rendering every rendered byte, plus an exported copy you control duplicate renditions living in two systems none; layout cannot be reproduced
Third-party owns the template, you store the output every rendered byte renditions per contract none, unless the template is archived too
You own the template, you render server-side signed artifact, template version, field values executed contract count deterministic re-render for anything derived
You own template and data, artifact under object lock signed artifact only contract count, not page count everything derived is disposable

The catch is that deterministic regeneration is harder than the table makes it look. Fonts get upgraded, hyphenation dictionaries change, layout engines round differently between releases, and a preview regenerated next year will not be byte-identical to the one you deleted. For derived copies nobody cares. For anything a court might look at, it matters enormously, which is why the signed artifact stays authoritative and the template pipeline is only ever allowed to rebuild the disposable layer.

Self-hosting the renderer buys you that determinism at the price of owning a font stack and a patch cadence. A hosted document API removes the on-call load and gives you no leverage over the archive exit path. Neither answer is universal; the honest version is that if legal will not accept a regenerated document, template ownership stops being a cost decision and becomes a records-retention decision.

The signal that should have fired months before the page

The verification alert was the last possible signal, not the first. The one that should have fired is boring: bytes retained per executed contract, sampled weekly, with the derived-to-authoritative ratio next to it. A contract archive that grows faster than enrollment has a rendition problem, and you can see that months before a bill or an audit forces the conversation. Pair it with a hard invariant rather than an SLO with an error budget — recompute the digest of a sample of authoritative objects on a schedule, and treat a single mismatch as a page, because there is no acceptable rate of silently altered contracts.

// Objects come in exactly two classes, and the class decides what a storage
// job may do with the bytes. Anything not provably derived is off limits.
type Class string

const (
    Authoritative Class = "authoritative" // executed, signed, immutable
    Derived       Class = "derived"       // previews, packets, term bundles
)

type Object struct {
    Key      string
    Class    Class
    Digest   string // sha256 recorded at write time
    Bytes    int64
    Template string // template id and version; empty for scanned input
}

// Cold and infrequent-access tiers bill a per-object floor, so re-encoding
// anything at or below it spends CPU to save nothing.
const billedFloorBytes int64 = 128 << 10

func compressible(o Object) (bool, string) {
    switch {
    case o.Class != Derived:
        return false, "signed bytes are covered by the signature digest"
    case o.Bytes <= billedFloorBytes:
        return false, "already at or under the billed per-object floor"
    case o.Template != "":
        return false, "regenerate from the template instead of storing a copy"
    }
    return true, ""
}
Enter fullscreen mode Exit fullscreen mode

Emit the capacity signal per contract rather than per object. The bill follows contracts, so the model has to as well, and a per-object histogram hides the case that actually hurts: one contract quietly carrying nine renditions.

type ArchiveSample struct {
    ContractID     string
    AuthoritativeB int64
    DerivedB       int64
    Renditions     int
    DigestMatches  bool
}

func (s ArchiveSample) Report(m Metrics) {
    m.Observe("archive.bytes_per_contract", float64(s.AuthoritativeB+s.DerivedB))
    m.Observe("archive.derived_ratio", ratio(s.DerivedB, s.AuthoritativeB))
    m.Observe("archive.renditions", float64(s.Renditions))
    if !s.DigestMatches {
        m.Inc("archive.digest_mismatch", 1) // page on the first one
    }
}
Enter fullscreen mode Exit fullscreen mode

Two metrics and one invariant. That is the whole instrumentation change.

Getting the threshold wrong costs more than the storage

Lossy compression on scanned documents has a failure mode that no dashboard will catch for you. JBIG2 pattern-matching encoders reuse a symbol dictionary across a page, and in 2013 David Kriesel documented Xerox scanners substituting digits in scanned documents because two glyphs were matched as the same symbol. On a tuition agreement, a silently altered figure is not a rendering artifact, it is a contract dispute you will lose. Bitonal symbol matching is not a good fit for anything with numbers that matter, which rules it out for most of an edtech archive even on the derived side.

The other threshold is the lifecycle rule. Cold storage classes carry minimum storage durations of 90 or 180 days depending on the class, so a policy that demotes documents too early pays early-deletion charges on every contract a support agent pulls back. Set the transition on real access patterns, and expect a seasonal spike — a rule tuned on July traffic will misfire during enrollment week, and paging on that costs more attention than the storage it saves.

I'm not certain where the line sits for any particular archive, and your mileage may vary with retention policy and audit frequency. If the whole archive is under a couple of terabytes, stick with a lifecycle policy and a deduplication pass, skip the compression pipeline entirely, and spend the engineering time on the scan intake path instead. The bill is not the expensive part at that size. The 02:40 page is.

References

Top comments (0)