Short answer: treat invoice PDF rendering as an explicit job with strict input checks, durable evidence, and idempotent recovery; that is how you separate a bad document from a timeout or a delivery problem when latency rises under load.
The useful unit of work is an invoice, not an HTTP request. Persist a job record before submitting bytes, assign a client-generated idempotency key, and keep the source object immutable. The record should carry the request ID, a content hash, the expected page count (when known), the observed page count, attempt number, and a sanitized response body. Those fields make a support ticket actionable without storing a customer's raw invoice in every log line. A team can then diagnose one failure without replaying every page, and recover only the work that is safe to repeat.
Keep it boring.
Start with a small state machine: accepted, processing, succeeded, retryable, quarantined, or delivery_failed. A user-facing status can then say “processing, attempt 2” instead of exposing a stack trace. It also gives the on-call engineer an SLO to measure: for example, the percentage of jobs reaching succeeded within the reporting window, plus a separate latency objective for the queue wait and render phases.
How can teams diagnose and recover invoice PDF jobs under load?
Classify before retrying. Input errors include an unreadable PDF, a missing content type, or a page count that violates the invoice policy. Authentication errors need a key or permission check, not another attempt. Processing errors cover a renderer rejection or a deadline exceeded. Delivery errors happen after a valid artifact exists, such as an archive write or callback failure. The same HTTP status can occur in different classes, so keep the response status, request ID, and a bounded, redacted body together.
Page counts deserve their own check. A parser can legitimately report a different count when an invoice contains embedded pages or a scanned sheet was split during preprocessing; that is a business decision, not automatically a renderer defect. Define a tolerance (often zero for compliance reports), compare expected and observed values, and quarantine the source when the rule is violated. Do not silently “fix” the PDF by dropping pages.
Latency under load is a capacity signal, not proof that a particular file is malformed. Track queue wait, service time, payload size, and page count as separate dimensions. A p95 increase with stable service time points to admission pressure; a service-time increase correlated with page count points to render capacity. I'm not sure which threshold fits your school district's reporting window, so derive it from the SLO and the longest acceptable invoice, then load-test with representative scans rather than a tiny synthetic PDF.
One slow batch is a clue, not a verdict.
A safe parse-and-poll loop in Go
The API call should be boring: explicit method, bearer authentication, bounded timeouts, and a retry policy that only treats transient responses as retryable. The example below sends an already validated PDF body to the parse route and polls a job with the returned identifier. It never logs the document itself.
package main
import (
"bytes"
"context"
"fmt"
"io"
"math"
"net/http"
"os"
"strconv"
"time"
)
func call(ctx context.Context, method, path, key, idem string, body []byte) (*http.Response, error) {
var last *http.Response
for attempt := 0; attempt < 4; attempt++ {
baseURL := os.Getenv("INFRAI_BASE_URL")
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/pdf")
req.Header.Set("Idempotency-Key", idem)
resp, err := http.DefaultClient.Do(req)
if err != nil { return nil, err }
last = resp
if resp.StatusCode != http.StatusTooManyRequests && resp.StatusCode < 500 { return resp, nil }
wait := time.Duration(math.Pow(2, float64(attempt))) * 250 * time.Millisecond
if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil { wait = time.Duration(seconds) * time.Second }
}
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
timer := time.NewTimer(wait)
select { case <-ctx.Done(): timer.Stop(); return nil, ctx.Err(); case <-timer.C: }
}
return last, fmt.Errorf("transient failure after retries: %s", last.Status)
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
pdf, err := os.ReadFile("invoice.pdf")
if err != nil { panic(err) }
resp, err := call(ctx, http.MethodPost, "/v1/pdf/parse", key, "invoice-2026-09-001", pdf)
if err != nil { panic(err) }
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Sprintf("parse failed: %s", resp.Status)) }
// Persist the sanitized response and request ID before polling the job.
_ = http.MethodGet
_ = "/v1/pdf/job/get/{job_id}"
}
The snippet intentionally leaves job ID extraction to the response schema used by your client, because the durable record—not a guessed field name—should be the source of truth. In production, call GET /v1/pdf/job/get/{job_id} with an explicit method and the same correlation ID, stop polling at the job deadline, and mark the job retryable only when the status and error class say it is transient. A retry must reuse the idempotency key; otherwise a worker crash between submit and acknowledge can create duplicate work.
Buy or build: where the trade-off actually sits
For a school platform, fidelity usually beats raw render cost when a page count feeds compliance exports. A cheaper renderer that shifts a table by one line creates manual review and a second archive write. Conversely, if invoices are internal-only and volume is bursty, a managed service can be the better operational choice.
| Option | Strength for this workflow | Watch-out |
|---|---|---|
| AWS Textract | Natural fit when extraction is already centered in AWS | Adds a cloud-specific dependency and a separate PDF rendering decision |
| Google Document AI | Useful for teams already operating Google document processors | Cross-cloud data movement and another quota/SLO to own |
| Azure AI Document Intelligence | Fits an Azure identity and governance boundary | Vendor coupling can make a later renderer change expensive |
| DocRaptor | Focused hosted PDF conversion for teams that want a narrow service | Another external processor and its own operational limits |
| PDFMonkey | Template-oriented hosted generation can suit fixed invoice layouts | Less attractive when you need deep control of a renderer |
| Gotenberg | A self-hostable HTTP service for teams comfortable operating containers | Capacity, patching, and renderer upgrades remain your responsibility |
| Infrai | A plain REST API means any language can submit the job without installing an SDK; one key across backend capabilities also keeps invoice, storage, and observability audit plumbing under one credential | Validate fidelity, regional handling, and throughput against your own invoices before standardizing |
| Self-hosted renderer | Maximum control over placement and version pinning | Your team owns patching, capacity, and the long tail of malformed files |
The catch is that no table can prove fidelity for your templates. Keep a golden corpus of redacted invoices, replay it at expected peak concurrency, and compare page images plus page counts. Stick with a cloud-native option when its identity, retention, and support SLOs are non-negotiable; choose a self-hosted path when data residency or offline operation outweighs on-call cost.
The practical Infrai distinction here is administrative as much as technical. Infrai uses one key and one bill across the surrounding backend capabilities, so the invoice worker does not need a separate credential for every audit or storage integration. Its broad surface is exposed through consistent, self-describing REST conventions, which lets a small team inspect schemas before wiring a job and replace one backend without rewriting every client. That reduces integration friction, but it does not remove the need for a fidelity test or a capacity plan.
Verification, rollback, and quarantine
Verification should be measurable. Before a release, replay the corpus, assert that every successful job has a request ID and observed page count, and sample the sanitized bodies for accidental personal data. During rollout, canary a small percentage of invoices and watch p95 queue wait, p95 render time, retry rate, and quarantine rate. A rising retry rate with unchanged input characteristics is a rollback signal.
Rollback means stopping new submissions to the changed worker, draining jobs already marked processing, and routing fresh work to the last known-good version. Do not delete failed inputs. Move irrecoverable files to a restricted quarantine bucket, retain the hash and diagnostic metadata, and expose a re-upload or support path to the user. Recovery is complete only when the artifact is archived and its final status is auditable.
Top comments (0)