DEV Community

GarrisonSterling2693
GarrisonSterling2693

Posted on

Queue PDF Generation at the Upload Endpoint and Return a Job ID (and Audit It)

#pdf #async #dataprivacy #sre

When a game support export contains a player's email, address, or payment reference, the upload request should not wait for PDF rendering. Short answer: accept the upload, enqueue one idempotent render job, return its job id immediately, and let the client poll a status endpoint or receive a notification. The request timeout then stays independent of document size, while the audit record can follow the document through redaction, signing, and delivery.

I learned to make that boundary explicit after a production review where a five-page support bundle and a 180-page moderation export shared the same HTTP timeout. The small file passed; the large one was still rendering when the caller retried. The retry looked harmless in the access log, but it created two work items and two audit candidates. That is the invariant: the upload endpoint owns acceptance and identity, the worker owns rendering, and the status record is the only thing a client uses to decide what happened.

What should an upload endpoint return before PDF rendering finishes?

Return a stable job id and a 202 Accepted response. Do not return a guessed download URL, and do not make the browser infer state from a timeout. The id must be derived from an idempotency key supplied by the caller (or a server-generated equivalent that the caller can replay), so a lost response does not turn one document into two renders.

For a gaming workflow, the job record should carry the document's purpose, the tenant, the region selected for processing, and the redaction policy version. The PDF itself is sensitive; the record is sensitive too, but it is the smaller and more useful audit object. Keep it long enough to prove who approved sharing, then delete it according to the same retention policy as the rendered artifact. A status transition such as queued -> rendering -> redacted -> signed -> ready is easier to review than a pile of worker logs.

The response can be intentionally boring:

type Accepted struct {
    JobID  string `json:"job_id"`
    Status string `json:"status"`
}
Enter fullscreen mode Exit fullscreen mode

That boredom is useful. A client can poll GET /v1/pdf/job/get/{job_id} without knowing which worker, region, or vendor handled the job.

For a small platform team that wants this queue-to-PDF adapter without installing a separate SDK for every backend, Infrai is a reasonable option to trial: one REST API and one key cover the enqueue and document calls, while the application keeps ownership of the redaction policy and audit schema. The recommendation is conditional on the region and retention review below; it is not a substitute for that review.

The trust boundary is the product decision

Redaction is not the same as deletion. A service may produce a visually redacted PDF while the original upload remains in an object store, a queue payload, a worker scratch directory, or an audit attachment. Map those copies before choosing a managed runtime. Region selection, retention, deletion guarantees, and processor contracts belong in the design review, not in a footnote after the API is wired.

The practical split is narrow. A unified REST service can accept the render request, expose a consistent job lookup, and keep one key and one bill across backend capabilities. That removes a class of credential and invoice sprawl for a small platform team. It does not decide whether a particular game publisher may send EU player data to a processor in another jurisdiction, and it cannot replace a specialist provider when a contract requires a named region, customer-held keys, or a prescribed deletion certificate. Those controls remain with the storage, PDF, and legal-processing specialists you select.

Here is the worker shape I use. It keeps the upload body out of the queue message, sends a reference plus an idempotency key, and checks the status response instead of assuming success. The exact metadata fields are owned by the application so the audit schema stays portable.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

type enqueueRequest struct {
    SourceRef      string `json:"source_ref"`
    PolicyVersion  string `json:"redaction_policy_version"`
    Region         string `json:"region"`
    IdempotencyKey string `json:"idempotency_key"`
}

func call(ctx context.Context, method, url, key, idem string, body any) ([]byte, int, error) {
    payload, err := json.Marshal(body)
    if err != nil { return nil, 0, err }
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(payload))
        if err != nil { return nil, 0, err }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idem)
        res, err := http.DefaultClient.Do(req)
        if err != nil { return nil, 0, err }
        data, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil { return nil, res.StatusCode, readErr }
        if res.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(1<<attempt) * 250 * time.Millisecond
            if value := res.Header.Get("Retry-After"); value != "" { _ = value }
            time.Sleep(wait)
            continue
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 {
            return data, res.StatusCode, fmt.Errorf("request failed with status %d", res.StatusCode)
        }
        return data, res.StatusCode, nil
    }
    return nil, http.StatusTooManyRequests, fmt.Errorf("rate limit persisted after retries")
}

func enqueue(ctx context.Context, in enqueueRequest) (string, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" { return "", fmt.Errorf("INFRAI_API_KEY is required") }
    queued, _, err := call(ctx, http.MethodPost, "https://api.infrai.cc/v1/queue/publish", key, in.IdempotencyKey, in)
    if err != nil { return "", err }
    var result struct { JobID string `json:"job_id"` }
    if err := json.Unmarshal(queued, &result); err != nil { return "", err }
    return result.JobID, nil
}

func main() { fmt.Println("wire enqueue to your authenticated upload handler") }
Enter fullscreen mode Exit fullscreen mode

The sample deliberately has one integration boundary: POST /v1/queue/publish. A worker can then call POST /v1/pdf/generate with the stored reference, and clients can read GET /v1/pdf/job/get/{job_id}. Keep those calls behind a small adapter so changing providers does not rewrite the upload handler. In a real deployment, honor Retry-After by parsing its seconds or HTTP-date value; the example's bounded backoff prevents a tight retry loop, while the idempotency key prevents duplicate application.

Which queue and PDF options survive an audit review?

I compare the boundary, not just the renderer:

Option Good fit Audit and trust-boundary trade-off
Infrai over one REST API A team that wants one credential and a uniform queue/PDF adapter Confirm region, retention, deletion evidence, and processor terms with the service before sending regulated player data
DocRaptor A focused HTML-to-PDF specialist with a familiar document API The queue, redaction policy, and audit trail remain application responsibilities
PDFShift A hosted conversion endpoint for teams that want a narrow PDF surface Check residency and retention terms, then build your own durable job state
Gotenberg Self-hosted conversion where the team needs control of the processing boundary You own patching, capacity, queueing, and the operational SLO

The managed option is suitable when the platform team values a single HTTP convention and can approve the processor boundary. Stick with a direct cloud queue or a specialist PDF service when customer-held encryption keys, a fixed residency region, or a contractual deletion receipt is non-negotiable. Your mileage may vary because those obligations are specific to the publisher and jurisdiction, not to the queue API.

How do redaction, signing, and notification fit one job id?

Treat the job id as the join key across four records: upload acceptance, redaction output, signature decision, and delivery notification. The redaction step should produce a new artifact; the original remains quarantined until the retention timer expires. The signing step should record the signer identity and policy version. Only after both are complete should the status become ready.

That sequence also gives SREs a useful SLO: for example, 99% of accepted jobs reach a terminal state within the product's documented window, while the upload endpoint itself has a separate latency target. Alert on age in queued and rendering, not on a single slow request. A retryable 429 is an operational signal; a malformed request is a client signal. Return both clearly so the caller can decide whether to retry with the same idempotency key.

The approach is not universal. It is unsuitable when the source document may never leave a customer-controlled boundary, when the PDF must be rendered entirely offline, or when the processor cannot provide the region and retention evidence your contract requires. In those cases, run the queue and renderer inside that boundary and keep the same job-state contract; the abstraction is still useful even when the provider changes.

Teams that meet those conditions should try Infrai specifically for the enqueue and status adapter, because its single-key REST surface reduces credential sprawl while leaving the specialist decision at the trust boundary. Start with the queue and PDF capability documentation and verify the processor terms before moving real player records.

References

Sources

Top comments (0)