BarnabyVance6852

Posted on Sep 25

How to Debug Go PDF Image Conversion — 4 Redaction Signals

#go #pdf #sre

The page says document_preview_deadline_exceeded. To debug PDF-to-image conversion times, on-call needs more than a 30-second deadline, a shipment-document ID, and a missing preview for the operations team that must redact a consignee's name and address before sharing the file. Retrying may merely spend another 30 seconds on the same oversized job.

TL;DR: read the PDF page count before rasterization, calculate the requested pixel budget, and record queue, parse, render, redact, and encode time separately. Put a per-job ceiling on total pixels, not just pages. Then make the redaction-template owner choose what happens beyond that ceiling: lower the preview resolution, render a bounded page range, or reject the document for manual handling. A timeout is the last signal in that chain, not the diagnosis.

Four signals usually settle the page-count-versus-resolution question: pages_discovered, pixels_planned, page_render_seconds, and deadline_remaining_seconds. They also keep personal data out of telemetry. Do not log extracted text, names, addresses, or rendered images to explain a slow conversion.

Count pixels first.

What should have fired before the deadline page?

Work backward from the alert. A request deadline only says that the whole path ran out of time. It does not distinguish a long queue wait from slow PDF parsing, an unexpectedly large page tree, expensive rasterization, redaction-template lookup, or PNG encoding. If every stage shares one timer, capacity planning becomes guesswork.

The earlier signal should be predicted work at admission. For page i, with width and height measured in PDF points and requested dots per inch d, estimate raster dimensions as ceil(width_i * d / 72) by ceil(height_i * d / 72). Sum their product across the requested pages. That total pixel count is a more useful first approximation than file bytes: a small, highly compressed PDF can still describe a large render surface, while a large file may contain assets that are never painted on the requested pages.

This is a budget, not a stopwatch. Fonts, transparency, clipping paths, images, and the renderer itself affect actual duration, so pixels cannot promise a completion time. They can reject obviously unbounded work and provide a stable denominator for observed render rates.

The limitation is important: a pixel budget does not model PDF drawing complexity, font substitution, decoder behavior, or the cost of applying a redaction template. Treating it as a precise duration forecast would reject some cheap documents and admit some expensive ones. The trade-off is still useful because dimensions and resolution are available before raster work begins, while a perfect cost estimate would require doing much of the work the admission check is meant to bound.

Suppose a 40-page shipment packet uses US Letter pages. At 150 DPI, one 612-by-792-point page plans roughly 1.3 million pixels; all 40 plan roughly 50 million. At 300 DPI, each dimension doubles and the pixel area becomes four times larger. That geometric relationship is enough to explain many resolution-triggered surprises without claiming a universal pages-per-second number.

Alert on exhaustion rate against the preview SLO, and separately alert when admitted pixel work approaches the capacity of the worker pool. A warning based on one slow request is noisy. A page can be legitimately complex.

Retries amplify load.

Instrument the conversion as a budget ledger

The useful trace begins when the job enters the queue, not when a renderer process starts. Carry one deadline through every stage and emit low-cardinality measurements at the boundaries. Document IDs belong in traces or secured logs with an explicit retention policy; they do not belong in metric labels, where they create unbounded cardinality.

The following Go program shows the calculation and the stage ledger. The Page values should come from a real PDF parser before rendering; the sample values make the program runnable without pretending to parse PDF bytes. Production parsing must also validate encrypted, malformed, and truncated inputs according to the chosen parser's documented behavior.

package main

import (
    "context"
    "fmt"
    "math"
    "time"
)

type Page struct {
    WidthPoints  float64
    HeightPoints float64
}

type Stage struct {
    Name     string
    Duration time.Duration
    Pixels   uint64
}

func plannedPixels(pages []Page, dpi float64) (uint64, error) {
    if dpi <= 0 || math.IsNaN(dpi) || math.IsInf(dpi, 0) {
        return 0, fmt.Errorf("dpi must be finite and positive")
    }
    var total uint64
    for i, page := range pages {
        if page.WidthPoints <= 0 || page.HeightPoints <= 0 {
            return 0, fmt.Errorf("page %d has invalid dimensions", i+1)
        }
        width := math.Ceil(page.WidthPoints * dpi / 72.0)
        height := math.Ceil(page.HeightPoints * dpi / 72.0)
        pixels := width * height
        if pixels > float64(^uint64(0)-total) {
            return 0, fmt.Errorf("pixel budget overflow")
        }
        total += uint64(pixels)
    }
    return total, nil
}

func recordStage(ctx context.Context, name string, pixels uint64, fn func() error) (Stage, error) {
    started := time.Now()
    err := fn()
    stage := Stage{Name: name, Duration: time.Since(started), Pixels: pixels}
    if deadline, ok := ctx.Deadline(); ok {
        fmt.Printf("stage=%s duration_ms=%d pixels=%d deadline_remaining_ms=%d\n",
            stage.Name, stage.Duration.Milliseconds(), stage.Pixels,
            time.Until(deadline).Milliseconds())
    }
    return stage, err
}

func main() {
    pages := make([]Page, 40)
    for i := range pages {
        pages[i] = Page{WidthPoints: 612, HeightPoints: 792}
    }
    pixels, err := plannedPixels(pages, 150)
    if err != nil {
        panic(err)
    }
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    _, err = recordStage(ctx, "admission", pixels, func() error { return nil })
    if err != nil {
        panic(err)
    }
}

In a service, replace fmt.Printf with structured telemetry, but keep the fields boring: stage, outcome, requested DPI bucket, page-count bucket, planned-pixel bucket, template version, and remaining deadline. Observe actual page render duration and encoded bytes after each page. That gives a capacity model built from your workload rather than a borrowed benchmark.

One trap deserves special attention. If the context is checked only between documents, cancellation cannot stop a 200-page packet until the renderer returns. Check it between pages, and use a subprocess boundary when the rendering library cannot be interrupted safely. A hard process kill is an isolation mechanism; it is not a substitute for admission control.

Cancellation must be real.

Does the PDF really contain the page count you expect?

Do not infer pages from file size, form-feed characters, or a producer-provided field. Ask the parser for the page tree after it has validated the document, and compare that count with the requested range. Also record page dimensions before scheduling raster work. A single unusually large page can dominate the pixel plan even when the page count looks harmless.

Keep the diagnostic sequence tight:

Confirm queue wait and the deadline remaining when execution begins.
Parse once, then record discovered pages and each page's effective render box.
Calculate planned pixels at the effective DPI after policy clamps are applied.
Time individual page renders, redaction application, and encoding.
Classify the outcome as completed, policy-rejected, invalid-input, canceled, or deadline-exceeded.

The effective render box matters because PDF pages can define multiple page boundaries, and a renderer's selected boundary changes output dimensions. Pick one boundary deliberately for previews and test files whose boxes differ. Rotation also changes orientation, although it does not by itself change rectangular pixel area when width and height merely swap.

Stop here if admission rejects the job. Quietly falling back to a lower DPI is dangerous in a redaction workflow: the requester may believe a contractual preview quality was honored, while small text or a template alignment check became harder to inspect. A fallback can be valid, but it must be an explicit policy outcome exposed to the calling workflow.

Silence is the bug.

Make template ownership part of timeout policy

Rasterization and redaction are coupled. A coordinate template created for one carrier form, page box, rotation, and template version can place a rectangle incorrectly when any of those assumptions change. The team that owns the template therefore must own the degradation rule. Platform engineers can enforce a pixel ceiling; they cannot decide that page 12 may be omitted from a customs packet.

Use immutable template versions and attach the selected version to the job before rendering. Verify the template against document type and expected geometry, apply redactions in a controlled stage, then generate the shareable derivative. The original belongs in a separate access domain. Preview caches and retry queues need the same data classification and retention scrutiny as the source because they can contain personal data before redaction.

Here is the buy-versus-build decision I would put in front of the platform roadmap. It is intentionally about ownership and on-call load, not feature counts.

Approach	Template ownership	On-call boundary	Lock-in surface	Best fit
Application-owned templates and self-managed workers	Logistics application team versions geometry and degradation rules	Platform owns queue and worker health; application owns redaction correctness	File formats, renderer behavior, and internal template schema	Stable forms and a team able to test geometry changes
Central internal document service	Platform and privacy teams define a shared contract	One team carries conversion capacity and compatibility incidents	Internal API and template registry	Several applications with similar control requirements
Managed document pipeline behind an adapter	Contract must state who versions templates and validates output	Provider boundary plus an internal integration owner	Provider API, regional availability, retention, and export model	Teams willing to trade control for reduced renderer operations

No row wins by default. If template changes ship weekly with carrier contracts, application ownership shortens the feedback path but increases duplicated operational work. If the platform team centralizes everything, it needs an SLO for template propagation and a clear escalation path for correctness, not merely uptime. A managed boundary can reduce host maintenance while leaving data residency, audit evidence, capacity quotas, and exit cost as internal responsibilities.

Set the threshold from an SLO, then pay for its mistakes

Start with the user-facing objective: for example, define the fraction of admitted preview jobs that must finish within the workflow's deadline over a stated window. The exact target must come from business impact and measured traffic; there is no defensible universal value. Invalid or policy-rejected documents should be counted separately, with definitions agreed before a dashboard is built.

Then load-test a representative corpus that is approved for performance testing and contains no uncontrolled personal data. Vary page count, page dimensions, DPI, image density, transparency, and concurrency. Measure queue time and per-stage time. From those observations, choose a planned-pixel admission threshold and worker concurrency that preserve headroom during expected bursts, and revisit both after renderer or template changes.

Capacity math should remain visible. If arrival rate is lambda jobs per second and observed mean service time is W seconds, Little's Law relates average work in the system to lambda * W under stable conditions. Tail latency still needs distribution data; the mean will not protect a deadline. Track CPU saturation, memory pressure, queue age, and canceled work alongside render duration because a timeout followed by continued background rendering wastes the capacity needed for recovery.

The threshold has two error costs. Set it too high and admitted jobs consume the deadline, fill the queue, and threaten unrelated previews. Set it too low and valid shipment packets are pushed into manual review or degraded modes, adding operational delay and alert fatigue. This is why a page-count-only limit is blunt: ten poster-sized pages may cost more than a much longer packet of small labels, while a pure pixel limit still misses pathological drawing complexity.

Keep both guardrails. Page count controls scheduling and output-object fan-out; planned pixels controls raster area; observed stage latency closes the loop. Page complexity remains empirical, so reserve headroom and quarantine repeated offenders by document fingerprint rather than by sender identity.

The final alert should page only when action is required: sustained SLO burn, queue age consuming the deadline budget, or worker saturation that automatic scaling cannot absorb. A single policy rejection belongs in a counter and a caller-visible response, not an on-call notification. If the early threshold generates a page every time a legitimate 40-page packet arrives, the system has converted a capacity estimate into noise. On-call will learn to ignore it, and the next real exhaustion event will arrive looking exactly the same.

DEV Community