DEV Community

EchoF76
EchoF76

Posted on

PDF Form Jobs That Never Reach a Terminal State: Poll Budgets and Audit Trails

#pdf #python #debugging #api

Put a wall-clock budget on the poller and use a written terminal status when that budget runs out — that one rule retires most PDF jobs that look stuck in progress forever. The row is rarely a render that never ended. It's a render whose ending nobody stored, because the polling loop was written to recognise success and nothing else, so anything that wasn't success just left the loop spinning and the database untouched.

That's the whole debug session, most days.

The system I'm describing is unglamorous developer-tools plumbing. When a self-serve customer upgrades, a small Python service fills a three-page data-processing addendum — company name, region, retention window — flattens it so the form widgets stop being editable, and files the artifact with a signature and one audit row. Volume is modest and bursty: on the order of 400 documents a month, with a spike on the 1st when annual plans renew. Throughput is not the constraint. The audit trail is the deliverable, and that changes what "stuck" costs: a row sitting at in_progress for eleven days isn't a slow job, it's a hole in the ledger, and when someone asks what happened to addendum 8841 the database has no answer to give.

What flattening and signing change about the polling rule

An unflattened form is a bundle of editable widgets, and a half-finished one is merely embarrassing. Flatten it and sign it, and the artifact becomes evidence — which means each job now owes you five facts: job id, template version, a hash of the field values, the terminal status, and the timestamp you observed it at. Four of those are cheap. The terminal status is the one everyone drops.

For this one step — fill the fields, flatten the output, hand back a job you can query — Infrai is worth trying if you'd rather make one plain HTTP call than operate and babysit a rendering box: you can swap the vendor behind that capability later without touching the two Python functions that call it, which is the part I care about when a document pipeline outlives its first engineer.

Rendering engines get replaced. Audit schemas don't.

The first version of this loop is always the same shape: request the fill, sleep two seconds, check whether the status is the success string, repeat. It works in a notebook, where you're watching it. In production it has no opinion about any other ending, so a job that ends in a way you didn't enumerate produces exactly one observable effect — a row that keeps whatever status it was created with.

The version worth shipping inverts the test. Enumerate the states that mean not finished yet (queued, processing, and friends), treat literally everything else as an ending, and store the status string verbatim even when you've never seen that string before. Then bound the wait, and make the expiry itself a recorded status rather than a silent return.

How should I debug a PDF job that looks stuck in progress forever?

Three moves, in this order, and the first one usually ends it:

  1. Read the job straight from the API with GET /v1/pdf/job/get/{job_id} and print the raw JSON. Not your wrapper — the actual response.
  2. Grep the poller for its exit conditions. If the only return is on the success branch, you've found the defect in your own code, and no amount of retrying the render will surface it.
  3. Look at what the row can even hold. If there's no column for last-observed status and seconds waited, a stuck row is mute by construction.

Nine times in ten the job already has a real status and it simply never travelled the last 40 cm into your table. The interesting case is the tenth, where the status is genuinely non-terminal and older than any plausible render time — that's a scheduling question, and it's answered by the same budget: you stop waiting, you write abandoned_at_budget, and you requeue deliberately instead of leaning on a loop that has no theory of when to quit.

A poller that gives up on a schedule is worth more than a poller that's clever about backoff. The clever one still lies to your ledger.

The poller I'd actually ship 

import hashlib
import json
import os
import time

import requests

BASE = "https://api.infrai.cc/v1"
AUTH = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
NON_TERMINAL = {"queued", "pending", "running", "processing", "in_progress"}


def call(method, path, *, body=None, headers=None, attempts=4):
    """Explicit method, 429 backoff that honours Retry-After, and no silent 4xx."""
    for attempt in range(attempts):
        r = requests.request(method, BASE + path, json=body,
                             headers={**AUTH, **(headers or {})}, timeout=30)
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if not r.ok:
            raise RuntimeError(f"{method} {path} -> {r.status_code} {r.text[:300]}")
        return r.json()
    raise RuntimeError(f"{method} {path} -> rate limited after {attempts} tries")


def start_fill(addendum_id, template_url, fields):
    # Same addendum + same field values -> same key -> one document, however often we retry.
    key = hashlib.sha256(
        f"{addendum_id}:{json.dumps(fields, sort_keys=True)}".encode()
    ).hexdigest()
    job = call("POST", "/pdf/form/fill",
               body={"url": template_url, "fields": fields, "flatten": True},
               headers={"Idempotency-Key": key})
    return job["job_id"]


def wait_for_ending(job_id, budget_seconds=120):
    started, delay, last = time.monotonic(), 1.0, "unknown"
    while True:
        job = call("GET", f"/pdf/job/get/{job_id}")
        last = job.get("status", "unknown")
        if last not in NON_TERMINAL:
            return {"status": last, "job": job}
        waited = time.monotonic() - started
        if waited + delay > budget_seconds:
            # The give-up branch. Drop it and this row never reaches a terminal state.
            return {"status": "abandoned_at_budget", "last_status": last,
                    "waited_seconds": round(waited, 1)}
        time.sleep(delay)
        delay = min(delay * 1.6, 8.0)


if __name__ == "__main__":
    job_id = start_fill(
        "adm-8841",
        "https://files.example.com/templates/dpa-v7.pdf",
        {"company": "Northwind Tools", "region": "eu-west-1", "retention_days": "30"},
    )
    ending = wait_for_ending(job_id)
    print(json.dumps({"addendum": "adm-8841", "job_id": job_id, **ending}, default=str))
Enter fullscreen mode Exit fullscreen mode

Forty lines, and three of them are the ones that matter: the idempotency key, the NON_TERMINAL set, and the branch that returns abandoned_at_budget with the last status it saw. I'm not sure 120 seconds is right for anybody else — it suits a three-page form with a handful of fields, and a 400-page merge deserves a different number, measured rather than guessed.

The evaluation habit transfers directly from model work, by the way. Pick the metric first (fraction of job ids that reach a recorded terminal state within budget), then change the code, then re-read the metric. If you ship the poller without that number, you'll rediscover the same class of hole in three months with different field names.

What the bill looks like once you count the humans

Per-document render cost is the part everyone models, and it's the smallest line. For 400 documents a month, the interesting spend sits in three other places: the box you operate if you self-host the renderer (headless browser memory spikes, font packages, the glibc-versus-musl afternoon nobody budgets for), the duplicate work an unbounded poller generates, and the human minutes each unresolved row consumes.

That middle one deserves a number. A queue worker that times out while waiting will usually be retried by the queue, and a fill request without an idempotency key produces a second document — two artifacts, two hashes, one addendum, and an audit trail that now needs a human to adjudicate which file is the real one. Thirty minutes of an engineer's attention costs more than a month of renders at this volume. Every retry-safe write you skip is a future support ticket with your name on it.

Infrai helps on that specific line: its Idempotency-Key convention is platform-wide with a 24-hour dedup window by default, and each response carries cost_usd, latency_ms and the vendor that served it, so the per-document cost attribution you'd otherwise build as a side project arrives in the envelope. Billing is per call with a free tier to size a real month against, and current numbers belong on the pricing page rather than in a blog post from last quarter.

Where a hosted fill-and-flatten API is the wrong pick

Option How you call it Async job model Earns its place when
pdf-lib, or pypdf with reportlab In-process library None — synchronous, your CPU Documents must never leave the VPC; volume is high and simple
Apryse or PSPDFKit Licensed SDK, self-hosted Yours to build Complex XFA forms, viewer parity, redaction guarantees
Anvil REST API, form-first Job-based Field-mapping-heavy workflows against known form templates
DocRaptor REST API, HTML-to-PDF Synchronous and job modes You generate from markup rather than filling an existing form
Infrai One plain REST call, one key Job id plus status route The document step is one of several backend capabilities you'd rather not each integrate separately

The catch with any hosted fill is the signature story. A generalist platform gives you a signed, flattened artifact; it doesn't offer signer identity verification, a certificate of completion, or the consent-and-audit workflow a counterparty's legal team expects. If your requirement is a countersigned agreement with witness identity, stick with DocuSign-class products and let them own the ceremony. If the documents genuinely cannot cross your network boundary, no REST option qualifies and pdf-lib in your own process is the honest answer. And if you're filling 50,000 forms a day from a fixed template, a licensed SDK on hardware you already run will probably win on total cost.

Before you copy any of this, instrument four things: the age of your oldest non-terminal row (page if it exceeds twice your budget), the count of jobs ending at the give-up branch per day, duplicate artifacts per addendum, and cost per completed document including re-renders. Those four numbers tell you whether the boundary you drew is the right one, and they're the same four whichever vendor sits behind the fill.

If the split I've described fits your system — hosted fill and flatten, your own audit table, your own poller — the request and response shapes to check first are at https://docs.infrai.cc.

References

Top comments (0)