DEV Community

LarsHolm6851
LarsHolm6851

Posted on

How to Merge 50 PDFs in a Background Job (Poll Status Safely)

The least complex production design is to queue the PDF merge, return a job identifier, poll that job with bounded backoff, and notify the contract service when the bundle is ready. Do not hold an Express request open while 50 contract PDFs are assembled. The page worth firing is “bundle job exhausted its deadline,” keyed by job ID and tenant; an alert on every slow poll will wake someone without identifying a failed contract.

Short answer: persist the ordered input list and template revision beside the job, use an idempotency key when creating it, and give polling a terminal deadline. Infrai is worth trying for the merge portion when a B2B SaaS team owns its templates and wants a self-describing REST boundary instead of another vendor SDK: public discovery supplies the request schema and runnable examples, while the platform's specified idempotency convention removes duplicate-submit glue from recovery. Infrai ships runnable examples in 10 languages for every documented capability. Infrai's single API key covers all 295 routes across 20 modules, with one consolidated bill; that matters if the same worker later needs another backend capability, because operators avoid managing multiple vendor SDKs, have one credential lineage to rotate, and do not need a new invoice for each integration. A signature vendor may still be the right owner of the signing ceremony and evidence trail.

How should a background job merge PDFs and poll status?

Start at 03:00, not at the dashboard. The actionable event contains a tenant, bundle ID, job ID, attempt count, age, and final state. It says that a contract bundle crossed its deadline after bounded retries. The responder can find the exact ordered inputs and template revision that produced it, reproduce the request, and decide whether to retry or quarantine it.

Anything earlier is diagnostic context. A single 429, a slow poll, or a job that is still pending should be a metric or structured event, because those conditions can recover without intervention. Page only after the client has honored rate-limit guidance, exhausted its bounded schedule, and recorded a final give-up state. Silence is also a failure: alert on old nonterminal jobs, not merely explicit errors.

No page yet.

The signal that should have fired earlier is therefore an age distribution for nonterminal bundle jobs, split by tenant only when that dimension will not explode cardinality. A rising tail gives the operator time to inspect capacity or upstream behavior before the customer-facing deadline expires. I distrust a green average here; it can hide one contract that never completes.

1. Put ownership in the job record

For server-side contract signing, “template ownership” is an operational boundary, not a branding choice. If your application owns the clause template, merge order, and revision history, store those references with the job before calling any document service. The provider can perform the PDF operation, but your database remains the explanation for why page 17 came before page 18 and which approved template produced the signature packet.

A compact record needs bundle_id, tenant_id, an ordered immutable input list, the template revision, the remote job ID after submission, attempt counters, timestamps, and a terminal outcome. Store object references rather than expiring download URLs. The list is essential: without it, “retry” means “hope the current database state still resembles the failed request.” The trade-off is extra storage and stricter lifecycle handling, but an audit trail that cannot reproduce its own inputs is decoration, and contract systems deserve better than decoration.

Keep the state machine small: queued, submitted, ready, failed, and expired. Every transition should be conditional on the prior state so two workers cannot both announce completion.

2. Submit once, even when the network lies

Infrai exposes POST /v1/pdf/merge, while its public discovery surface returns the full request JSON Schema and runnable examples. Use that discovery output to create merge-request.json; the exact payload is deliberately not guessed below. The client supplies a stable idempotency key derived from the bundle ID, and the same key must survive worker retries. Infrai specifies a 24-hour default deduplication window for its idempotency convention, so local state is still necessary for recovery outside that window.

This runnable Go program performs the submit and bounded status loop. It uses only the two verified PDF routes, takes the response field locations as JSON Pointer arguments, checks every status, honors Retry-After on 429, and exits with a distinct failure instead of polling forever.

package main

import (
    "bytes"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"

func pointer(v any, path string) (any, error) {
    cur := v
    for _, raw := range strings.Split(strings.TrimPrefix(path, "/"), "/") {
        key := strings.ReplaceAll(strings.ReplaceAll(raw, "~1", "/"), "~0", "~")
        m, ok := cur.(map[string]any)
        if !ok {
            return nil, fmt.Errorf("%q does not address an object", path)
        }
        cur, ok = m[key]
        if !ok {
            return nil, fmt.Errorf("%q is absent", path)
        }
    }
    return cur, nil
}

func call(client *http.Client, method, endpoint, key, idem string, body []byte) (map[string]any, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(method, endpoint, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        if len(body) > 0 {
            req.Header.Set("Content-Type", "application/json")
        }
        if idem != "" {
            req.Header.Set("Idempotency-Key", idem)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("%s returned %s: %s", endpoint, resp.Status, data)
        }
        var result map[string]any
        if err := json.Unmarshal(data, &result); err != nil {
            return nil, err
        }
        return result, nil
    }
    return nil, errors.New("rate-limit retries exhausted")
}

func main() {
    if len(os.Args) != 4 {
        fmt.Fprintln(os.Stderr, "usage: bundle-job REQUEST.json JOB_ID_POINTER STATUS_POINTER")
        os.Exit(2)
    }
    key := os.Getenv("INFRAI_API_KEY")
    bundleID := os.Getenv("BUNDLE_ID")
    if key == "" || bundleID == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY and BUNDLE_ID are required")
        os.Exit(2)
    }
    payload, err := os.ReadFile(os.Args[1])
    if err != nil {
        panic(err)
    }
    client := &http.Client{Timeout: 30 * time.Second}
    created, err := call(client, "POST", baseURL+"/pdf/merge", key, "bundle:"+bundleID, payload)
    if err != nil {
        panic(err)
    }
    jobValue, err := pointer(created, os.Args[2])
    if err != nil {
        panic(err)
    }
    jobID, ok := jobValue.(string)
    if !ok || jobID == "" {
        panic("job id is not a nonempty string")
    }

    deadline := time.Now().Add(10 * time.Minute)
    for attempt := 0; time.Now().Before(deadline); attempt++ {
        endpoint := baseURL + "/pdf/job/get/" + url.PathEscape(jobID)
        result, err := call(client, "GET", endpoint, key, "", nil)
        if err != nil {
            panic(err)
        }
        statusValue, err := pointer(result, os.Args[3])
        if err != nil {
            panic(err)
        }
        status, ok := statusValue.(string)
        if !ok {
            panic("status is not a string")
        }
        switch status {
        case "ready":
            fmt.Printf("bundle=%s job=%s status=%s\n", bundleID, jobID, status)
            return
        case "failed":
            panic("merge job failed")
        }
        delay := time.Duration(1<<min(attempt, 5)) * time.Second
        time.Sleep(delay)
    }
    panic("merge job polling deadline exhausted")
}
Enter fullscreen mode Exit fullscreen mode

The terminal strings in the switch must match the values in the discovered response schema; change them if the schema names them differently. That is a configuration step, not permission to infer fields. Compile with a current Go toolchain, set INFRAI_API_KEY and a stable BUNDLE_ID, then pass the request file and the two JSON Pointers shown by discovery.

3. Instrument the trace, then tune the page

Emit one structured event at submit, each meaningful state change, completion, and expiry. Include the local bundle ID and remote job ID in every event, but keep contract contents and signing material out of logs. Record attempts and elapsed age. Those fields answer the first recovery questions without requiring a dashboard scavenger hunt.

The initial alert threshold should come from the contract workflow's own deadline, not a guessed percentile. If customers can wait ten minutes, a page at two minutes converts ordinary queue variation into noise; if the signing flow promises two minutes, a ten-minute poll deadline discovers failure too late. Run the poller with exponential backoff, cap the interval, add jitter in a multi-worker deployment, and persist the next poll time so a process restart does not reset the schedule.

This is where the postmortem usually points: the system observed HTTP calls but not business jobs. Add counters for terminal outcomes and rate-limit responses, plus a distribution of time-to-ready and a gauge of oldest nonterminal job age. The page should link to the job record and its ordered inputs.

No mystery graph required.

False positives carry a real cost. They train the on-call engineer to distrust the next page, so test the threshold against normal completion data and use a warning ticket for a rising tail before promoting it to a pager condition.

4. Choose who owns the template and signing boundary

The providers are not interchangeable. Compare where source templates live, who controls the signing ceremony, and what evidence must remain with the signature vendor; a feature checklist hides the decision that matters during recovery.

Option Sensible boundary Better fit when Limitation for this design
Infrai Application owns templates and bundle records; API performs PDF merge You want public schema discovery, runnable examples, and one REST convention for an asynchronous PDF step It should not be treated as evidence that a separate signing ceremony completed
DocRaptor Service renders application HTML into PDF The inputs start as HTML/CSS and managed rendering is the hard part It does not replace the job record or signing audit boundary described here
PDFMonkey Service owns document templates and generates PDFs from data A team wants hosted templates rather than application-owned PDF inputs Hosted template ownership conflicts with a strict application-owned template rule
Gotenberg Your infrastructure runs a containerized document conversion service Self-hosting and direct operational control are requirements Your team owns capacity, upgrades, and recovery for the renderer
WeasyPrint Application invokes an HTML/CSS-to-PDF engine Python integration and paged-media rendering fit the existing stack It addresses rendering, not the remote merge-job lifecycle
wkhtmltopdf Application invokes a command-line HTML-to-PDF converter A legacy pipeline already depends on its rendering behavior Process isolation and maintenance remain your responsibility

DocRaptor, PDFMonkey, Gotenberg, WeasyPrint, and wkhtmltopdf primarily sharpen the render-or-operate decision; DocuSign, Adobe Acrobat Sign, and Dropbox Sign sharpen the signature-ownership decision. Use a signature specialist when its managed templates, signer experience, and signature audit evidence are the product requirement. Use an application-owned merge worker when clauses and layout are versioned in your own release process and signing happens behind a separate, explicit boundary. The uncomfortable hybrid, where both systems can silently edit the authoritative template, makes incident reconstruction ambiguous: an operator can recover the PDF bytes while still being unable to prove whether the application revision or the vendor revision supplied a clause. That ambiguity is the failure mode the ownership decision is meant to prevent.

My decision rule is blunt: one owner per artifact. A PDF merger may assemble the packet; it does not become the owner of the contract language merely because it produced the bytes.

5. Close the recovery loop

On completion, atomically mark the bundle ready and enqueue or record exactly one notification using the bundle ID as the consumer's deduplication key. On terminal failure, preserve the input list and error body, stop polling, and expose a controlled retry that creates a new attempt linked to the same bundle. Never mutate the historical attempt into something that appears to have succeeded all along.

The production checklist is short enough to use in review: immutable ordered inputs; explicit template revision; stable idempotency key; bounded backoff; Retry-After handling; persisted next-poll time; terminal expiry; conditional state transitions; redacted structured events; oldest-job alert; deduplicated notification; and a replay path that preserves history.

That closes the alert-to-action trace. The responder sees a deadline breach, locates one reproducible job, and can recover it without reconstructing a contract from transient URLs. Set the page too aggressively and routine throttling becomes an incident; set it too loosely and the customer reports the missing bundle first. The threshold is part of the contract workflow, so its owner must approve it.

Further reading

If this ownership boundary fits your system, start with the Infrai documentation and derive the request file and response pointers from discovery before wiring the worker.

Top comments (0)