Memory Blows Up at Scale: Lessons From Batch-Processing Thousands of PDFs

It started with a Slack message that every backend engineer dreads: "The batch job died again. Server's out of memory."

We were processing PDF uploads for a document pipeline that had grown from a few hundred files a day to tens of thousands. The code hadn't changed. The logic was the same as it had been for a year. But somewhere between 500 files and 5,000, the process started crashing with MemoryError, and no one could figure out why the "same code" suddenly couldn't handle the load.

The answer wasn't a bug. It was an assumption we never questioned: that loading a PDF fully into memory is cheap.

The assumption that quietly doesn't scale

Here's roughly what our original processing loop looked like:

def process_batch(file_paths):
   results = []
   for path in file_paths:
       with open(path, "rb") as f:
           data = f.read()  # entire file into memory
       pdf = PdfReader(io.BytesIO(data))
       text = extract_text(pdf)
       results.append(text)
   return results

For a handful of files, this is fine. A 2MB PDF loaded into RAM is nothing. But this pattern has a hidden cost that only shows up at scale: every file's full byte content sits in memory for the entire duration of its processing, and the results list keeps growing while you're still working through the batch.

Multiply that by thousands of files, some of which are scanned documents with embedded high-resolution images running 40-80MB each, and you're no longer dealing with a rounding error. You're dealing with gigabytes of transient data that the garbage collector can't clean up fast enough because references are still held by the loop and the results list.

The job wasn't inefficient. It was structurally incompatible with scale.

Why "just add more RAM" is the wrong fix

The first instinct on a lot of teams is to throw hardware at the problem. Bump the instance size, increase the container memory limit, move to a bigger box. It works, until it doesn't. We doubled memory allocation twice before realizing we were treating the symptom.

The real issue was architectural: our pipeline was written as if all files existed at once, needed to be held at once, and produced results that all needed to live in memory simultaneously. That model breaks the moment your batch size grows faster than your hardware budget. And in a batch-processing context, hardware budget is usually the thing finance asks you to justify first.

The shift: streaming instead of loading

The fix was to stop treating each file as a blob you load, transform, and hold, and start treating the batch as a stream you flow through the pipeline one unit at a time, releasing memory as soon as it's no longer needed.

Concretely, that meant three changes:

1. Stream file reads instead of loading full byte arrays.

Instead of reading the entire file into memory before parsing, we opened a stream and let the PDF library read pages incrementally where the library supported it.

def process_batch(file_paths):
   for path in file_paths:
       with open(path, "rb") as f:
           pdf = PdfReader(f)  # reads incrementally, not all at once
           text = extract_text(pdf)
       yield text  # generator, not a growing list

Two things matter here. First, we pass the file handle directly instead of reading it fully into a BytesIO buffer. Second, we use yield instead of appending to a results list, so each result is consumed and discarded before we even open the next file.

2. Process and write results immediately, don't accumulate them.

The original code held every extracted text result in a list until the whole batch finished, then wrote everything out at once. We changed this to write (or push to a queue, or insert into a database) as soon as each file finished processing.

def run_pipeline(file_paths, output_writer):
   for text in process_batch(file_paths):
       output_writer.write(text)
       # nothing lingers after this point

This one change alone cut peak memory usage dramatically, because the batch no longer needed to hold N results in memory at once. It only ever needed to hold one.

3. Chunk the batch itself.

Even with streaming, extremely large batches (tens of thousands of files) benefit from being processed in fixed-size chunks with explicit checkpoints. If a job dies at file 8,432 out of 20,000, you want to resume from there, not restart the whole batch. Chunking also gives you a natural place to force garbage collection and release any file handles or buffers that didn't get cleaned up automatically.

def run_in_chunks(file_paths, chunk_size=200):
   for i in range(0, len(file_paths), chunk_size):
       chunk = file_paths[i:i + chunk_size]
       run_pipeline(chunk, output_writer)
       gc.collect()
       checkpoint(i + chunk_size)

The results

After the rewrite, peak memory usage during batch runs dropped by roughly 80 percent, and the job stopped crashing regardless of batch size. We could process 500 files or 50,000 with the same memory footprint, because the pipeline never held more than a chunk's worth of data at a time. Throughput also improved slightly, since the process spent less time under memory pressure and less time in garbage collection pauses.

The deeper lesson wasn't really about PDFs specifically. It applies to any pipeline that processes a large number of files, records, or payloads: memory problems at scale are almost never about one file being too big. They're about the architecture assuming everything can live in memory at the same time.

Where this matters beyond backend pipelines

This same tension between "process everything at once" and "process incrementally" shows up constantly in document tooling, including on the consumer side. When we were testing edge cases for large scanned PDFs during this project, we ended up using PDF Conveter to quickly split and compress oversized test files without needing to install anything locally. It's a free, browser-based PDF toolkit with over 20 conversion and editing tools, and it's a handy way to prep or debug PDF samples when you're testing a pipeline like the one described here without spinning up your own tooling for a one-off task.

Takeaways for your own batch jobs

If you're building anything that processes files, records, or documents in bulk, ask yourself early:

• Does my pipeline hold the full dataset in memory at any point, even briefly?
• Am I accumulating results in a list, or streaming them out as they're produced?
• What happens if this job runs on 100x the current volume? Does memory usage grow linearly, or does it stay flat? If the answer to that last question is "it grows linearly," you don't have a bug yet. You have a countdown timer to one.

Streaming isn't just a performance optimization. At scale, it's often the difference between a pipeline that works and one that quietly waits for the day your input volume crosses a threshold nobody tested for.