Short answer: validate the file and password before enqueueing, give every PDF job an idempotency key, and record request ID, page count, latency, and a sanitized response for each state change. Retry only a transient processing or delivery failure. Quarantine malformed or unauthenticated files with a status a support agent can explain.
For a property-management team signing contracts server-side, batch throughput is the decision axis. A single lease may be fine; a move-in batch of 2,000 customer files is where hidden coupling appears. The useful mental model is a conveyor belt: intake, authentication, transformation, and delivery each have their own queue and evidence. When one belt slows, the evidence tells you which belt to repair.
| Option | Pick this when | Trade-off |
|---|---|---|
| Synchronous validation, asynchronous PDF job | You can reject bad input quickly and process valid files in the background | The caller must poll or receive a callback |
| One shared worker pool | Volume is modest and files have similar sizes | A large encrypted file can consume capacity needed by small leases |
| Separate lanes by size or tenant | You need predictable latency during a batch | More queues, limits, and dashboards to operate |
| Quarantine plus manual review | A file cannot be safely repaired automatically | Recovery is slower, but the original evidence is preserved |
1. What should a team measure before changing PDF jobs?
Start with a job envelope, not a log line. Store a generated job ID, an idempotency key supplied by the contract workflow, tenant and document identifiers, input byte length, expected page count (when the source provides one), and a creation timestamp. Never store the password in logs. A hash of the encrypted bytes is enough to correlate duplicate uploads without exposing customer content.
The state machine can stay small: received, validated, queued, running, succeeded, retryable, quarantined, and delivered. Every transition emits an event with the request ID and elapsed milliseconds. This makes a timeout different from a slow queue, and a wrong password different from a parser failure.
I like one dashboard row per state transition. It is less glamorous than a giant trace, but it answers the support question: “Did we ever start this file?” Add counters for malformed input, authentication rejection, timeout, inconsistent page count, and delivery failure. Track p50, p95, and p99 latency separately for validation, processing, and delivery. A healthy average can hide a p99 that misses a signing deadline.
Three seconds. That was the alert threshold I initially reached for. It was the wrong abstraction: under a batch, queue wait dominated parser time. The useful alert is a pair of signals, queue age and worker duration, split by file-size bucket and tenant. Your mileage may vary with the PDF mix.
Measure twice.
2. How can password-protected customer files, malformed input, timeouts, and page counts be diagnosed?
Classify first; recover second. The same HTTP status can wrap different causes, so keep a machine-readable reason alongside the human message.
- Input: The byte stream is truncated, not a PDF, or has a broken cross-reference table. Reject it before a worker spends CPU. Keep a short sanitized parser reason and the byte hash.
- Authentication: The password is missing or does not unlock the file. Do not guess repeatedly. Return an “action required” status and retain no secret beyond the request boundary.
- Processing: The file opens, but rendering, encryption, or page counting exceeds the budget. Capture worker duration, memory pressure, and the page count observed at each stage.
- Delivery: The output exists, but object storage, callback delivery, or the signing service did not accept it. The PDF can be retried without re-running expensive parsing if the output hash is present.
For inconsistent page counts, compare three values: the source metadata, the parser's count after unlock, and the count in the final artifact. A mismatch is a quarantine signal when the contract workflow requires every page. It is not automatically corruption; incremental updates and attachments can make “page count” mean different things. Record which counter you used.
A compact event shape keeps this data queryable:
type PdfEvent = {
jobId: string;
requestId: string;
state: 'received' | 'validated' | 'queued' | 'running' | 'succeeded' | 'retryable' | 'quarantined' | 'delivered';
reason?: 'malformed_input' | 'bad_password' | 'timeout' | 'page_count_mismatch' | 'delivery_failure';
inputBytes: number;
observedPages?: number;
elapsedMs: number;
outputSha256?: string;
};
Sanitize response bodies before they reach logs or tickets. Keep status code, request ID, reason, and a bounded message; strip file names that contain personal data, passwords, and raw PDF bytes. That gives an operator something actionable without turning observability into a second customer-data store.
3. How should retries and recovery work when latency rises under load?
Make the queue the shock absorber. The API that accepts a contract should perform cheap checks, persist the envelope, and return a job ID. Workers pull jobs with a visibility timeout. A lease on the job prevents two workers from processing the same input at once; an idempotency key prevents a client retry from creating a second contract.
Retry a timeout, a temporary dependency refusal, or a lost callback. Do not retry malformed input or a wrong password. Use exponential backoff with jitter and a small attempt cap, then move the job to quarantine. A retry record should reference the same job ID and input hash, so an audit trail reads as one story.
type RetryDecision = { retry: boolean; delayMs: number; reason: string };
function decideRetry(reason: string, attempt: number): RetryDecision {
const transient = new Set(['timeout', 'dependency_unavailable', 'delivery_failure']);
if (!transient.has(reason) || attempt >= 4) {
return { retry: false, delayMs: 0, reason: 'quarantine_or_manual_review' };
}
const base = Math.min(30_000, 1_000 * 2 ** attempt);
const jitter = Math.floor(Math.random() * 250);
return { retry: true, delayMs: base + jitter, reason: 'transient_failure' };
}
Under load, bound concurrency per lane. A worker processing a 400-page scan should not starve a lane of two-page leases. Apply backpressure when queue age crosses a threshold, and expose a retry-after hint to callers that poll too aggressively. Keep output writes atomic: upload to a temporary key, verify the hash and page count, then publish the final reference. That ordering prevents a consumer from seeing a half-written artifact. During a contract-signing batch, I would graph queue age beside worker duration for each lane, then inspect a single slow job end to end: intake timestamp, unlock duration, parser duration, output upload, callback attempt, and the exact attempt number. If only queue age rises, add capacity or lower admission. If worker duration rises with page count, tune the parser budget or split the lane. If delivery alone rises, protect the completed artifact and replay delivery without parsing again. This sequence turns “latency under load” into a bounded choice with evidence attached.
One practical recovery path is replay from the original encrypted bytes. The replay command should require a job ID, load the immutable input hash, and create a new attempt under the same audit record. Operators should never have to paste a customer password into a shell command.
4. Which implementation details make batch throughput predictable?
Measure bytes and pages, not just jobs. A batch of 2,000 tiny files can have lower CPU cost than 20 huge scans, yet both count as 2,000 jobs in a basic chart. Use weighted capacity: reserve worker tokens for input bytes or estimated pages, and release them when the artifact is published.
Keep validation close to intake. Checking the MIME header, maximum byte size, and password presence there is cheap. Full parsing belongs in workers where it can be timed and cancelled. Cancellation matters: a client timeout should not leave an orphaned parser consuming a worker indefinitely.
For signing, preserve the exact bytes that were counted and signed. Re-encoding a PDF between page counting and signing can change incremental updates and invalidate an audit comparison. Store the final hash, page count, signer request ID, and publish timestamp together.
Use synthetic fixtures in load tests: a valid encrypted file, a truncated file, a wrong password, a file with a deliberate page-count mismatch, and a large multi-page file. Assert state transitions and audit events, not only HTTP responses. Inject queue delay and dependency timeouts so the retry cap is exercised before production.
5. What are the limits of this recovery design?
The catch is that no queue can repair a file whose bytes are genuinely incomplete, and no service can infer a password that the customer did not provide. Quarantine needs a retention policy and access controls; otherwise it becomes an unmanaged archive of sensitive contracts. A separate large-file lane costs capacity when the workload is quiet. Strong page-count checks can also reject legitimate PDFs whose metadata uses a different definition of “page.”
Stick with a simpler synchronous path when files are small, volume is low, and the caller can tolerate the worst-case processing time. Choose a staged queue when batch throughput, tenant isolation, or an audit trail matters more than a single-request response. I am not sure which latency budget fits your signing SLA until you measure the p99 by size bucket; that measurement should decide the worker limits, not a generic timeout copied from another system.
The durable outcome is boring and useful: each customer file has one identity, one evidence trail, and a clear next action. That is what lets a support engineer recover a contract job without rerunning good work or exposing a password.
References
- https://developer.mozilla.org/en-US/docs/Web/API/Blob
- https://www.adobe.com/devnet/pdf/pdf_reference.html
- https://www.rfc-editor.org/rfc/rfc9110
Top comments (0)