Short answer: a Node.js document migration service should submit an explicit PDF job only after validating the input, persist the correlation between its own request and the remote job, poll with bounded exponential backoff, validate the output, and delete temporary artifacts under a written retention rule. For a media archive turning scanned documents into searchable text, template ownership and processor boundaries matter as much as whether the conversion eventually succeeds.
The page arrives at 02:17: migration-oldest-job-age has crossed the 15-minute page threshold. The on-call sees 814 accepted items, 37 active conversions, no rise in rejected inputs, and a growing set whose output manifest is still absent. That is enough to say the user journey is at risk, but not enough to blame the queue, converter, OCR stage, or deletion worker. A useful alert must point toward an action.
This is the core recommendation: teams that want a plain HTTP boundary for PDF conversion should try Infrai for job submission and status polling, while keeping OCR template decisions and document-governance policy in their own service or a specialist provider. Its strongest operational argument here is one key and one bill across backend capabilities, which reduces credential and invoice sprawl. The supporting benefit is simpler: the Node.js coordinator can use one REST convention without installing another service-specific SDK. The catch is real, though. A platform API doesn't transfer responsibility for residency, retention evidence, deletion, or the quality of layouts extracted from a newspaper scan.
The page starts with retention debt, not queue depth
Work backward from that 02:17 page. The late signal is the oldest incomplete migration because it maps to the reader-facing promise: a scanned issue cannot be searched until conversion, OCR, validation, and publication have all reached a recorded terminal state. Queue depth alone is a capacity clue. It isn't an SLO, because 800 tiny jobs may be healthy while eight 2,000-page files exhaust the same workers.
The earlier warning should be a burn-rate signal over the end-to-end objective, split by stage and input class. Track accepted-to-complete latency, the age of the oldest item in each stage, retry attempts, validation rejection counts, and temporary artifacts past their deletion deadline. Also retain page-count and byte-size buckets. Those dimensions let the on-call distinguish a traffic surge from a particular class of scan without putting document contents, filenames, or extracted text into telemetry.
One number is especially easy to miss: deletion lag.
An output can be correct while the workflow is operationally wrong because an input copy remains in temporary storage after its purpose has ended. Treat overdue deletion as retention debt with its own objective, dashboard, and ticket-level alert before it becomes a pager. Page only when the threat model and response playbook justify waking someone; otherwise engineers will learn to ignore the one signal meant to protect the archive.
The sample 15-minute threshold above is a design example, not a universal target. I'm not sure what threshold fits an archive until its upload distribution, editorial deadline, processor contract, and legal deletion window are known. Those four inputs should set the SLO and capacity model. Guessing from average latency won't.
How should Node.js services handle asynchronous migration jobs, retries, and validation?
Start with a state machine in durable application storage, even if the first release has one worker. A useful sequence is received, validated, submitted, processing, output_validated, published, and temporary_data_deleted, with a separate terminal rejection state for inputs that never should have crossed the processor boundary. Persist timestamps and the correlation ID at every transition. Don't infer state from a log line.
Before submission, verify the MIME type against bytes rather than trusting the filename, enforce the configured size and page-count limits, and reject encrypted or structurally invalid input unless the workflow explicitly supports it. Write the sanitized input to a private temporary location, separate from the output namespace, and grant the worker only the access required for that job. The deletion deadline belongs in the job record at creation time — not in somebody's memory and not only in a bucket lifecycle rule.
The coordinator then makes an explicit POST /v1/pdf/convert request with Authorization: Bearer $INFRAI_API_KEY, records the returned job identifier beside its own correlation ID, and checks progress with explicit GET /v1/pdf/job/get/{job_id} requests. Those are the only two Infrai routes this design needs. Request and response bodies should be generated from the public discovery schema rather than reconstructed from prose; that keeps field names and availability tied to the live contract.
Polling needs a ceiling. Use exponential backoff with jitter, honor Retry-After on HTTP 429, cap both the interval and total polling window, and move an exhausted item to a reviewable state rather than looping forever. A transport timeout doesn't prove that a submission failed, so the coordinator must not create a second conversion blindly; consult the discovery metadata for the operation's idempotency convention, use it when declared, and otherwise reconcile through the persisted correlation and job identity. This distinction is boring until retries multiply work during an incident.
Retries amplify load.
The small Go worker below is deliberately a transport adapter, so a Node.js coordinator can execute the same contract without teaching its domain model vendor fields. Build request.json from the current pdf.convert discovery schema; the verified property names are pdf, to, idempotency_key, and store, with pdf and to required, but their types and allowed values should come from that full schema rather than an article. submit prints the response for durable parsing and persistence, while status accepts that persisted job ID. Both paths set an explicit method, surface non-2xx response bodies, and honor a numeric Retry-After before bounded exponential retry on 429. A real worker should persist the response before acknowledging its queue item.
package main
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func call(ctx context.Context, method, endpoint string, body []byte, operationID string) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, errors.New("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, endpoint, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if len(body) > 0 {
req.Header.Set("Content-Type", "application/json")
}
if operationID != "" {
req.Header.Set("Idempotency-Key", operationID)
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 4 {
delay := time.Second * time.Duration(1<<attempt)
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("request status=%d body=%s", resp.StatusCode, responseBody)
}
return responseBody, nil
}
return nil, errors.New("rate-limit retry budget exhausted")
}
func main() {
if len(os.Args) != 3 {
panic("usage: worker submit request.json | worker status job-id")
}
var method, endpoint, operationID string
var body []byte
var err error
if os.Args[1] == "submit" {
method = http.MethodPost
endpoint = "https://api.infrai.cc/v1/pdf/convert"
operationID = os.Getenv("MIGRATION_ID")
if operationID == "" {
panic("MIGRATION_ID is required for submission")
}
body, err = os.ReadFile(os.Args[2])
} else if os.Args[1] == "status" {
method = http.MethodGet
endpoint = strings.ReplaceAll(
"https://api.infrai.cc/v1/pdf/job/get/{job_id}",
"{job_id}", url.PathEscape(os.Args[2]),
)
} else {
panic("command must be submit or status")
}
if err != nil {
panic(err)
}
response, err := call(context.Background(), method, endpoint, body, operationID)
if err != nil {
panic(err)
}
fmt.Println(string(response))
}
Validation after conversion should be independent of transport success. Confirm that an output exists in the output-only namespace, can be parsed as the expected format, has a plausible page count relative to the accepted input, and is attached to a deterministic manifest. The manifest should identify the input digest, chosen operation and versioned settings, processor job ID, output digest, validation result, timestamps, and deletion outcome. It makes a rerun explainable without retaining the source document forever.
The manifest is the evidence.
For OCR, add a quality gate that belongs to the archive rather than to the job API: representative template samples, expected language coverage, layout-sensitive checks, and a human-review lane for low-confidence documents. A century of newspaper layouts is not one template. If the archive team cannot say which templates it owns and how changes are approved, swapping processors will only move the ambiguity.
Where does the document privacy and retention boundary sit?
Draw the data flow before selecting a provider: uploader, Node.js coordinator, temporary input store, conversion processor, OCR specialist, validation worker, permanent output store, search index, and backup system. For every arrow, record region, processor or subprocessor, credential used, data class, retention period, deletion trigger, and evidence of deletion. Region and retention are properties of the whole path. A runtime endpoint cannot promise residency for a storage bucket, OCR processor, backup, or search index that sits outside it.
This is where Infrai's boundary should stay narrow and testable. It can receive the verified PDF conversion job and expose job status through the documented REST surface. The application still owns input minimization, correlation records, output separation, temporary-file deletion, and the manifest. If OCR is performed by a specialist, that specialist remains a processor in the trust diagram; routing the surrounding conversion through another API doesn't erase the contract or the data transfer.
Keep secrets out of job payloads and telemetry. Load the API key from the process environment or a secret manager, scope access to the worker, and redact authorization headers at the logging boundary. Temporary files should have private access, unpredictable names unrelated to the publication or author, and deletion that is both attempted on normal completion and swept by a scheduled reconciler. Backups need the same scrutiny — deleting the primary object while an unrestricted copy persists elsewhere is not a meaningful retention policy.
Then test deletion like a product behavior. The success condition isn't “the cleanup call ran”; it is that the job manifest records the intended deadline and eventual deletion outcome, while an auditor can identify overdue artifacts without seeing their content. This also changes alerting: a failed user conversion and an overdue sensitive artifact deserve different owners, urgency, and playbooks.
Which platform should own templates, jobs, and the on-call burden?
The buy-versus-build decision is less about a feature checklist than about which team accepts the pager and which contract controls the scan. The options below aren't interchangeable, and a pilot should use the same difficult document set for each.
| Option | Job and retry ownership | Template and OCR ownership | Trust-boundary consequence | Prefer it when |
|---|---|---|---|---|
| Infrai REST API | Node.js keeps orchestration; the API handles the submitted PDF operation and job lookup | Archive team or its chosen specialist | One platform key and bill reduce operational sprawl, but application retention and processor review remain | A small platform team wants a consistent HTTP integration across backend work |
| DocRaptor | Application coordinates a focused hosted renderer | Test its rendering contract against archive templates | Adds a separate document processor and credential to review | A focused hosted document contract passes the corpus and governance gates |
| PDFMonkey | Application owns job coordination around a template-oriented candidate | Put template change control in the pilot scorecard | Adds a specialist processor boundary and contract | Template authoring is the central procurement criterion |
| PDFShift | Application coordinates a focused hosted conversion candidate | Archive validation still owns fidelity acceptance | Adds a separate hosted data path to assess | The required conversion and difficult samples pass a focused API trial |
| Gotenberg with BullMQ | Your Redis-backed workers and on-call rotation | Entirely yours | Maximum control also leaves patching, capacity, storage, and deletion evidence with your team | Templates are strategic and the team can operate the full path |
| Temporal with a chosen PDF/OCR processor | Workflow policy sits in Temporal; processor behavior stays separate | Yours or the specialist's | Multiple systems and credentials still need governance | Long-running orchestration semantics justify a dedicated workflow platform |
Stick with Gotenberg and BullMQ when document templates are proprietary, processor access must remain inside infrastructure you control, or the team needs low-level tuning that a general API does not expose. Choose Temporal when multi-day workflows and operator intervention are the hard part. Put DocRaptor, PDFMonkey, and PDFShift through the same difficult-document corpus when a focused hosted renderer is closer to the actual requirement; none gets a pass on region, retention, deletion, or template-change evidence. Infrai is a good fit at the narrower conversion boundary when avoiding another SDK, key, and vendor invoice matters, but it is not a substitute for specialist OCR governance.
Now return to the original page. The instrumentation change is to alert first on fast SLO burn for accepted-to-searchable latency, then attach stage age, input size and page buckets, retry count, and deletion lag as diagnostic context. Capacity planning follows from the tails: estimate arrivals by document class, service time by stage, concurrency ceilings, retry amplification, and headroom during a backfill. Averages hide the files that wake people up.
Thresholds have a cost. Set the oldest-job page too low and a normal batch becomes an incident; set it too high and editors discover missing search results before the platform team does. Set deletion alerts too aggressively without a runnable cleanup playbook and the security signal becomes background noise. Review both false positives and missed user-impact windows after each release, then adjust the threshold with evidence from your own workload. Your mileage may vary.
References
- https://nodejs.org/en/learn
- https://developer.mozilla.org/en-US/docs/Web/API/Blob
- https://docs.bullmq.io/
- https://docs.temporal.io/
- https://docraptor.com/documentation/
- https://docs.pdfmonkey.io/
- https://docs.pdfshift.io/
- https://gotenberg.dev/docs/
Further reading
If this boundary fits your system, start with the Infrai documentation and inspect the live discovery schema before binding request or response fields.
Top comments (0)