Every compression library I tried ran out of RAM. So I built one that doesn't.

CrellinCreative — Fri, 02 Oct 2026 08:41:05 +0000

Compress a 1 GB file in the browser with most JavaScript libraries and you'll use 1.6 GB of RAM. Try 4 GB and the tab crashes.

zstd-stream does it in 35 MB, whatever the file size, and the file never leaves your device:

import { compressStream } from "zstd-stream";

const compressed = await compressStream(file.stream(), { level: 19 });

TL;DR

Any file size, ~35 MB of memory. Verified with 4 GB files in Chrome, Edge, Firefox, Safari and Node.js.

Beats gzip. 5.5× compression at 147 MB/s, vs 3.5× at 51 MB/s for the browser's built-in gzip.

One install, nothing else. The WebAssembly is embedded: no .wasm file to host, no CDN, zero dependencies.

NPM · Live Demo · GitHub

Why I built it

I wanted a simple way to compress and decompress files locally, in my browser, without uploading them anywhere.

I tried the usual libraries: pako, fflate, then the Zstandard ports zstd-codec and @bokuweb/zstd-wasm. Small files worked fine. Big files ran the tab out of memory, every time.

Switching libraries didn't help, because they all share the same design.

The 2 GB wall

Almost every JavaScript compression library looks like this:

const output = compress(input); // whole file in, whole file out

The entire input and the entire output sit in memory at once. A browser tab can't hold an ArrayBuffer much past ~2 GB, so anything beyond that: Finito.

No optimisation fixes that. The API shape is the problem.

The fix: never hold the whole file

zstd-stream takes Meta's open-source Zstandard C library, compiles it to WebAssembly, and puts a streaming layer on top. That streaming layer is the part I wrote.

A ReadableStream goes in and a ReadableStream comes out, the same type File.stream(), fetch and Node's Readable.toWeb() already give you.

128 KB at a time. Data flows through fixed buffers, so 4 MB and 4 GB use the same memory.
Backpressure. If the consumer is slow (a disk, a network), it stops reading the source instead of piling data up in RAM.
Checksummed. Corrupt or truncated input throws instead of quietly returning wrong bytes.
Standard .zst output. The zstd command-line tool opens it, and it opens files made by any other zstd tool.

And it's completely self-contained. The WebAssembly binary is embedded in the package, so there's no .wasm file to serve, no bundler plugin, no CDN path to get wrong, and zero dependencies. The whole thing is 163 kB installed.

Compress a local file, nothing uploaded

This is what I originally wanted. In the browser, pick a file and stream the result straight to disk with StreamSaver:

import { compressStream } from "zstd-stream";
import streamSaver from "streamsaver";

const compressed = await compressStream(file.stream(), { level: 19 });
await compressed.pipeTo(streamSaver.createWriteStream(`${file.name}.zst`));

The same API works in Node.js 18+:

import { createReadStream, createWriteStream } from "node:fs";
import { Readable, Writable } from "node:stream";
import { compressStream } from "zstd-stream";

const source = Readable.toWeb(createReadStream("big.log"));
const compressed = await compressStream(source, { level: 19 });
await compressed.pipeTo(Writable.toWeb(createWriteStream("big.log.zst")));

pipeTo handles backpressure for you: if the disk is slow, compression waits for it.

Doesn't the browser already do this?

Partly. Every browser ships CompressionStream with gzip. If gzip is enough, use it. But on 1 GB of log-style data, zstd wins comfortably:

	Compression	Speed	Memory
gzip (`CompressionStream`)	3.5×	51 MB/s	37 MB
zstd-stream (level 3)	5.5×	147 MB/s	35 MB

What about native zstd?

Firefox 138+ supports CompressionStream("zstd"), but nothing else does (yet):

Native zstd	Firefox 138+	Chrome / Edge	Safari
`CompressionStream` / `DecompressionStream`	✅	❌	❌

More importantly when this feature does become widely available, there are no options:

No compression level. For archiving, that's the one that matters. You compress once and store for years, so level 19 is worth the CPU.
No progress. Fine for 2 MB. Not fine for 4 GB.

zstd-stream gives you the same output in every engine, with one code path.

Bonus: compress before you upload

Most apps upload files as-is. That's a lot of wasted waiting:

Generally speaking, inbound traffic is free on AWS, GCP and Azure, so this isn't about your ingress bill. The wins are:

Users wait less. Upload is the slowest part of the trip, and compressing 1 GB takes ~7 seconds.
You store less, possibly forever.
Downloads cost less. Egress is billed, and the file leaves your bucket compressed.
No server-side compression job re-reading every file.

Uploading large files

In Chrome and Edge, the compressed stream can go straight into fetch (your server needs HTTP/2 or later):

await fetch("/upload", {
  method: "POST",
  headers: { "Content-Encoding": "zstd" },
  body: compressed,
  duplex: "half",
});

Firefox and Safari don't support streaming request bodies yet. There, upload the compressed stream in parts instead. Here's a minimal example (status checks and retries left out):

const reader = compressed.getReader();
const uploads = [];
let part = [], size = 0, n = 0;

const send = () => {
  uploads.push(fetch(`/upload/${id}/${++n}`, { method: "PUT", body: new Blob(part) }));
  part = []; size = 0;
};

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  part.push(value);
  size += value.length;
  if (size >= 8_000_000) {
    send();
    if (uploads.length >= 3) await uploads.shift(); // 3 in flight? wait for the oldest
  }
}
if (size) send();
await Promise.all(uploads);

It collects compressed output into 8 MB parts and keeps a few uploading while it compresses the next one. When enough parts are in flight, it simply stops reading, and compression pauses until the network catches up:

Part size and the number of parallel uploads are up to you; memory stays bounded either way. Parts also map neatly onto S3 multipart uploads, so a failed part can be retried on its own.

Downloads work too

Serve the .zst as-is and decompress it in the browser as it downloads. The bytes you pay egress on stay compressed, and the user's RAM stays flat:

const res = await fetch("/exports/logs.csv.zst");
const restored = await decompressStream(res.body);

await restored.pipeTo(streamSaver.createWriteStream("logs.csv"));

Want to process it instead of saving it? Add .pipeThrough(new TextDecoderStream()) and handle the text chunk by chunk as it arrives.

Before you ship it

⚠️ Use a Web Worker. Compression is CPU work; keep it off the main thread, otherwise your interface will become sluggish / freeze.
Skip already-compressed files. JPEG, MP4 and ZIP won't shrink much more.

Try it

The live demo is the tool I originally wanted: compress or decompress any file, entirely in your browser. Throw something big at it.

npm install zstd-stream

CrellinCreative / zstd-stream

High-performance Zstandard compression for Node.js and browsers with zero external dependencies

zstd-stream

Efficient Zstandard compression for Browsers and Node.js

zstd-stream compresses and decompresses Zstandard data through the Web Streams API. It streams files larger than available RAM in constant memory, propagates backpressure end to end, and ships the WebAssembly build embedded — no .wasm asset to host or configure.

🌊 Streaming first — process multi-GB data in fixed ~128 KB buffers
⚡ Backpressure built in — a slow writer automatically throttles the reader, so memory never runs away
📦 Zero external assets — WASM is bundled in; npm install and import
🛡️ Integrity checked — every frame carries a checksum; corrupt or truncated input is rejected
🌍 Universal — the same code runs in Node.js 18+ and every modern browser
🔧 ESM + TypeScript — full type definitions included
📊 Progress callbacks — observe bytes processed in real time

Unrivaled performance

Compressing 1 GB — zstd-stream vs. the browser-capable libraries…

View on GitHub

DEV Community: CrellinCreative