A batch image compressor CLI in Node with bounded concurrency

A batch image CLI in Node needs a bounded pool: a few workers that pull files from one shared list, so you never start hundreds of sharp pipelines at once. With sharp, the useful bound is the size of Node's libuv thread pool, which is 4 by default. On 24 large photos my CLI took 5.15 s one at a time, 1.33 s with 4 at a time, and 0.92 s with 8 at a time after raising UV_THREADPOOL_SIZE to 8.

Why not just Promise.all over every file?

Promise.all(files.map(compress)) starts every pipeline at the same moment. Each one opens its file and reserves memory for its decode. With 24 files that is survivable. With a folder of 5,000 phone photos it is how a CLI runs out of memory, and it gives you no single place to count failures or report progress. A pool keeps the number of files in flight fixed, whatever the folder holds.

The CLI

#!/usr/bin/env node
// squeeze: resize every JPEG in a folder to 1600 px wide and re-encode it,
// N files at a time.  Usage: node squeeze.mjs <in> <out> [--concurrency 4]
import { readdir, mkdir, stat } from 'node:fs/promises';
import { join } from 'node:path';
import { parseArgs } from 'node:util';
import sharp from 'sharp';

const { values, positionals } = parseArgs({
  allowPositionals: true,
  options: { concurrency: { type: 'string', default: '4' } },
});
const [inDir, outDir] = positionals;
const limit = Number(values.concurrency);
await mkdir(outDir, { recursive: true });
const files = (await readdir(inDir)).filter((f) => /\.jpe?g$/i.test(f));

async function squeeze(name) {
  const out = join(outDir, name);
  await sharp(join(inDir, name))
    .rotate()
    .resize({ width: 1600, withoutEnlargement: true })
    .jpeg({ quality: 75, mozjpeg: true })
    .toFile(out);
  return (await stat(out)).size;
}

// A bounded pool: `limit` workers pull from one shared index.
async function pool(items, limit, fn) {
  const results = new Array(items.length);
  let next = 0;
  async function worker() {
    while (next < items.length) {
      const i = next++;
      try { results[i] = { ok: true, value: await fn(items[i]) }; }
      catch (error) { results[i] = { ok: false, error }; }
    }
  }
  await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
  return results;
}

const t0 = performance.now();
let peakRss = 0;
const timer = setInterval(() => { peakRss = Math.max(peakRss, process.memoryUsage().rss); }, 20);
const results = await pool(files, limit, squeeze);
clearInterval(timer);
const ms = performance.now() - t0;
const failed = results.filter((r) => !r.ok).length;
const bytes = results.reduce((s, r) => s + (r.ok ? r.value : 0), 0);
console.log(JSON.stringify({ files: files.length, concurrency: limit, ms: Math.round(ms),
  failed, outKB: +(bytes / 1000).toFixed(1), peakRssMB: Math.round(peakRss / 1e6) }));

The pool has no library behind it. next++ is safe because JavaScript runs one worker's code at a time; each worker only yields at await. A failed file becomes a result with ok: false, so one corrupt JPEG never stops the batch. parseArgs is built into Node, so the CLI has one dependency: sharp.

How fast is each concurrency level?

I copied the site's 4964 × 2782 camera photo (9,804.3 KB) 24 times and ran the CLI 5 times per setting. One run prints:

{"files":24,"concurrency":4,"ms":1432,"failed":0,"outKB":2259.1,"peakRssMB":379}

The medians of 5 runs:

Setting	Median wall time	Runs (ms)	Peak RSS
concurrency 1	5.15 s	5142, 5145, 5152, 5176, 5562	245 MB
concurrency 2	2.62 s	2582, 2613, 2624, 2627, 2747	283 MB
concurrency 4	1.33 s	1312, 1317, 1328, 1332, 1340	343 MB
concurrency 8	1.33 s	1317, 1318, 1333, 1372, 1463	346 MB
concurrency 24	1.32 s	1312, 1313, 1322, 1325, 1326	347 MB
concurrency 8, UV_THREADPOOL_SIZE=8	0.92 s	913, 915, 919, 925, 953	479 MB
concurrency 24, UV_THREADPOOL_SIZE=8	0.92 s	874, 906, 920, 952, 1010	480 MB

Every run wrote the same 2,259.1 KB of output.

Why does concurrency stop helping at 4?

Because sharp runs each pipeline on a libuv worker thread, and Node's libuv pool has 4 threads by default (Node.js: UV_THREADPOOL_SIZE). The sharp docs say queued tasks wait "for libuv to provide a worker thread" (sharp: queue). So concurrency 8 and 24 only add files to a queue: the wall time stays at 1.33 s.

Inside each pipeline, libvips uses its own threads. sharp.concurrency() printed 6 on this machine; the docs say the default is the number of CPU cores, with an exception for glibc Linux without jemalloc, where it is 1 (sharp: concurrency).

Note: set UV_THREADPOOL_SIZE in the environment before Node starts, for example UV_THREADPOOL_SIZE=8 node squeeze.mjs in out --concurrency 8. The Node docs warn that setting process.env.UV_THREADPOOL_SIZE inside your script is not guaranteed to work, because the pool is created before your code runs.

Raising the pool to 8 cut the time from 1.33 s to 0.92 s and raised peak memory from 343 MB to 479 MB. That is the trade: more files in flight, more decoded pixels in memory at once.

Which concurrency should a CLI default to?

Match it to the thread pool, and let the user raise both. A default of 4 with the default pool got 3.9 times the speed of one at a time here, at 98 MB more peak memory. Past the pool size, extra concurrency did nothing but queue. If your files are 50 MP scans and not 13.8 MP photos, memory becomes the limit first, so measure your own folder with the peakRssMB field.

A CLI like this is fine for one machine. For thousands of files a day from many users, the work belongs in a queue worker; a hosted image compression API is the other route when you would rather not run the encoders yourself.

How I measured

• Machine: Apple M1 Pro (8 cores: 6 performance, 2 efficiency), macOS, 2026-10-17.
• Node v24.22.0, sharp 0.35.4, libvips 8.18.6, mozjpeg as bundled with sharp.
• Input: image-compressor-sample-camera-photo.jpeg from the site (4964 × 2782, 9,804,300 bytes), copied 24 times.
• Each setting ran 5 times in a row; the table gives the median and every run. Peak RSS is sampled every 20 ms, so it can miss a short spike.
• A first batch of runs happened while other jobs loaded the machine and scattered widely; I discarded it and reran when the load average had dropped. The numbers above are from that second batch.

What I did not test

Linux, where sharp's libvips thread default can differ. Mixed folders of PNG and HEIC. Concurrency above 24 or pools above 8.

---

I build Blinkimg, an image compressor and converter with a developer API. I wrote this article with help from an AI assistant (Claude), which ran the code on my machine; every number in it comes from those runs.