The first version of this pipeline was a single long-running process handling PDF jobs off a queue directly inside the app server. It worked until it didn't: a deploy would restart the app mid-job, an in-flight PDF job would just vanish, and nobody would notice until a customer asked where their file was. Moving the actual PDF work into its own containerized worker pool fixed that, but only once a few specific things were done deliberately rather than assumed to work by default.
Why a worker pool beats a single long-running process
A single process handling jobs sequentially has one obvious failure mode: whatever kills that process kills every job it was holding, whether that's an out-of-memory crash, a deploy restart, or a bad input that takes the process down with it. A pool of worker containers pulling from a shared queue doesn't have that problem in the same way. If one worker dies mid-job, the message it was processing simply becomes visible in the queue again, and any other running worker can pick it up. The unit of failure shrinks from "the whole pipeline" to "one job, briefly delayed."
Making the worker actually safe to kill at any moment
None of that resilience matters if the worker's PDF logic doesn't tolerate being interrupted and retried. A worker needs to be safe to kill at literally any point in its execution, which mostly comes down to not doing partial, unrecoverable writes before the job is confirmed done:
def process_job(message):
job = parse_job(message.body)
result = pdf_api.run({"action": job.action, "files": job.files})
if result.status != "success":
raise JobFailed(result.status)
write_output(job.output_location, result.output_url)
message.delete() # only acknowledge after output is durably written
The message only gets deleted from the queue after the output has actually landed somewhere durable. If the container gets killed anywhere before that last line runs, the message stays in the queue and another worker retries the whole job from scratch, which is safe precisely because nothing partial or misleading got written in the meantime.
Dockerizing the worker without dragging in the whole app
It's tempting to reuse the main app's Docker image for the worker, since most of the dependencies overlap. In practice a slimmer, purpose-built worker image is worth the extra Dockerfile: it starts faster, has a smaller attack surface, and doesn't rebuild every time an unrelated part of the app changes. A worker only needs the queue client, the PDF API client, and whatever minimal logic turns a queue message into an API call, nothing the rest of the app depends on for serving requests.
FROM python:3.12-slim
WORKDIR /app
COPY worker/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY worker/ .
CMD ["python", "worker.py"]
Scaling replicas without overrunning the rate limit
More worker replicas means more concurrent PDF API calls, which is exactly the throughput win the whole setup exists for, right up until the combined concurrency across all replicas exceeds the API's rate limit and starts generating 429s instead of finished jobs. The fix isn't to guess at a safe replica count and hope, it's to make each worker respect a shared concurrency budget, whether that's a per-worker semaphore sized against the replica count or a centralized rate limiter the whole pool checks against before firing a request.
Health checks that actually reflect whether the worker is stuck
A container orchestrator restarting an unhealthy worker is only useful if "unhealthy" is defined correctly. A worker that's alive but has silently stopped pulling from the queue, deadlocked on a bad message, say, will pass a naive liveness check indefinitely while doing nothing. A more useful health check tracks the timestamp of the worker's last successful job pull and fails if too much time has passed with the queue non-empty, which catches the stuck case a simple "is the process running" check misses entirely.
Logging that survives the container's death
A worker container that crashes mid-job takes its local logs with it unless those logs were already shipped somewhere durable. Centralized logging isn't optional infrastructure here, it's the only way to actually diagnose why a particular job failed after the container that was running it no longer exists. Structured logs tagged with the job ID, written as the job progresses rather than only at the end, mean a crash still leaves a trail even if the final "success" line never gets written.
Handling SIGTERM instead of just getting killed
Container orchestrators don't usually kill a container outright on a routine scale-down or deploy, they send SIGTERM first and give it a grace period before the harder kill signal follows. A worker that ignores this and gets killed mid-job the same way an OOM crash would is throwing away a resilience mechanism that's already available for free. Catching SIGTERM, finishing the in-flight job if it's close to done, and only then exiting cleanly turns a routine deploy into something that costs zero retried jobs instead of one per worker that happened to be mid-task:
import signal
shutting_down = False
def handle_sigterm(signum, frame):
global shutting_down
shutting_down = True
signal.signal(signal.SIGTERM, handle_sigterm)
while not shutting_down:
message = queue.receive(wait_seconds=5)
if message:
process_job(message)
The grace period the orchestrator gives before the hard kill needs to be long enough to cover the worst-case job duration, not just the typical one, or this pattern just moves the problem instead of solving it. A worker that occasionally handles a large file taking twenty seconds needs more headroom than the default grace period most orchestrators ship with.
What this pattern buys once it's running
The pipeline stopped losing jobs on deploys, which was the original problem, but the bigger change was that scaling stopped being a code change. Adding capacity for a busier day meant adding replicas, not rewriting the worker. None of the container orchestration logic needed to know anything about PDF operations specifically, that complexity stayed fully inside the containerized PDF processing API handling merge, split, compress, rotate, watermark, and convert, priced per successful result, with the worker pool responsible only for pulling messages, calling it, and acknowledging cleanly once the output actually exists.
Top comments (0)