DEV Community

Cover image for Why I Stopped Loading Entire Files Into Memory (And How It Saved My Web App)
Muhammad Waseem
Muhammad Waseem

Posted on

Why I Stopped Loading Entire Files Into Memory (And How It Saved My Web App)

When you build client-side tools, user privacy isn't just a feature. It's a strict architectural requirement.

No backend.
No uploads.
No server.

But nobody warns you about the hidden cost of running everything in the browser: Memory Limits.

I learned this the hard way while building ToolFlic, a suite of 100+ browser-based utilities.

When users started processing large PDFs and high-resolution images, the browser tabs started crashing.

Here is the story of why it happened, and how I fixed it.

The Naive Approach: Load Everything

When I first started building the file tools, my approach was simple.

I used the FileReader API to read the entire file, pass it to the processing library, and generate the output.

const reader = new FileReader();
reader.onload = (e) => {
  const buffer = e.target.result;
  // Process the entire buffer at once
};
reader.readAsArrayBuffer(file);
Enter fullscreen mode Exit fullscreen mode

For a 2MB image, this works perfectly.

But what happens when a user uploads a 50MB PDF?

The browser tries to allocate all of that data into RAM at the same time.

The browser's garbage collector cannot keep up.

The UI freezes, and eventually, the tab crashes with the "Aw, Snap!" error.

The Reality of Client-Side Processing

JavaScript does not give you direct control over memory management.

You cannot manually free up an ArrayBuffer after you are done with it.

You have to rely on the browser's garbage collector, and it only runs when it wants to.

If you want to process large files without crashing, you have to stop thinking like a backend developer.

You have to think like a systems engineer.

The Solution: Chunking and Web Workers

I had to completely rewrite the way files were read and processed.

Here is the architecture that fixed it.

1. Read Files in Chunks

Instead of reading a 100MB file all at once, I read it in smaller chunks using File.slice().

const chunkSize = 5 * 1024 * 1024; // 5MB
let offset = 0;

while (offset < file.size) {
  const chunk = file.slice(offset, offset + chunkSize);
  const buffer = await chunk.arrayBuffer();

  // Process the chunk
  offset += chunkSize;
}
Enter fullscreen mode Exit fullscreen mode

By reading in chunks, memory usage stays flat.

The browser only ever holds 5MB in memory at any given time, no matter how big the file is.

2. Offload to Web Workers

Even with chunking, heavy processing can block the main thread and freeze the UI.

I moved the processing logic into Web Workers.

Web Workers run on a separate thread.

The main thread stays free to handle user interactions, like dragging and dropping files.

3. The UI Feedback Loop

When you process large files client-side, the user needs to know the app hasn't frozen.

I used postMessage from the Web Worker to send progress updates back to the main thread.

// Inside the Web Worker
self.postMessage({ progress: 50, status: "Processing..." });
Enter fullscreen mode Exit fullscreen mode

This allowed me to build a progress bar that updates in real-time.

The Tradeoff

Building things to run entirely in the browser is harder than sending a request to a server.

You have to respect the constraints of the user's device.

You have to handle edge cases like low memory and slow CPUs.

But the tradeoff is worth it.

By doing the work client-side, you guarantee that the user's data is completely private.

What I Learned

If you are building browser-based tools, treat the browser like a resource-constrained environment.

Never load the whole file at once.
Use Web Workers for anything CPU-intensive.
Always assume the user is on a low-end device.

You can see these techniques in action on ToolFlic - All Free Online Tools.
It handles large files entirely on your device, without ever hitting a server.

I'd love to hear how other developers handle memory management in the browser.

What techniques do you use?

Top comments (5)

Collapse
 
komopdf profile image
KomoPDF •

Chunking via File.slice() and pushing work to Web Workers keeps the UI thread responsive, but there is an extra memory trap once you pass those buffers across threads.

If you transfer chunks with plain worker.postMessage({ chunk: buffer }), the engine performs a structured clone, briefly doubling the chunk's memory footprint in the V8 heap until garbage collection cleans it up. Passing the ArrayBuffer inside the transfer list parameter (worker.postMessage({ chunk: buffer }, [buffer])) detaches the buffer on the sender thread instantly, leaving zero copy overhead.

On random-access binary formats like PDFs or MP4s, linear chunking also runs into limits because parsers need to jump directly to trailing xref tables or atom headers. Pairing file.slice() with an on-demand sparse range reader backed by an LRU chunk cache keeps the memory ceiling flat without requiring you to stream the entire file from byte zero.

Collapse
 
toolflic profile image
Muhammad Waseem •

This is a brilliant insight, thank you for pointing this out!

You are completely right about the structured clone overhead. I was using the standard postMessage(), which does double the memory temporarily. I will definitely refactor my logic to use the transferList array (e.g., postMessage({ chunk }, [chunk])) to get that zero-copy performance.

Also, the point about PDF xref tables at the end of the file is a massive realization. Linear chunking is definitely a bottleneck for random-access formats like PDFs. Implementing a sparse range reader with an LRU cache is a much smarter approach to keep memory flat without streaming the whole file.

Thanks for sharing this, it gives me a clear path for the next optimization phase!

Collapse
 
mp3sss31 profile image
mp3sss •

That peak allocation is easy to miss on small fixtures: the raw bytes, the decoded representation, and the output buffer can all coexist, so memory climbs well above the file size before GC gets a chance. Chunked or streaming processing, plus offloading heavy work to a worker and releasing each buffer as soon as its slice is consumed, keeps the ceiling flat regardless of input size. On the desktop side the same rule applies — don't pull a large file into one Buffer in the main process; stream it through or spill to a temp file, and keep big objects out of the renderer's heap.

Collapse
 
toolflic profile image
Muhammad Waseem •

This is a fantastic addition,

You hit the nail on the head with "peak allocation." The raw bytes, the decoded representation, and the output buffer coexisting in memory is exactly the silent killer that causes browser tabs to crash. Releasing each buffer as soon as its slice is consumed is crucial.

The point about the desktop side is also incredibly valid. Whether it's an Electron app or a Node.js backend, pulling a massive file into a single Buffer in the main process will always choke the system. Streaming it through or spilling to a temp file is the only safe way to handle it.

Thanks for sharing this perspective, it's a great reminder that memory management principles apply across all environments!

Collapse
 
toolflic profile image
Muhammad Waseem •

Handling memory limits in the browser was definitely one of the hardest parts of building ToolFlic. I'd love to know how other developers handle large file uploads on the client side. What is your go-to approach?