Testing a Batch Image Pipeline Before It Touches Your Files

Making crop work on one image is the easy part. Making it safe to run on two hundred of somebody else's files is the actual job, and what decided when it shipped was a test list.

The first two tests on that list failed. Not on the crop, which was the point of the list, but on the version of Convertim that had been out since July.

Crop shipped on Monday, in Convertim 2.1. This is how the pipeline is tested, what the crop had to pass before it shipped, and what those two failures taught me about the two hundred tests that had been passing all along.

What was already there

Convertim is a portable image converter for Windows: a desktop app and a CLI, both on top of one conversion engine built on sharp. The desktop app reaches the engine through Electron's main process; the CLI calls it directly. Same functions, same options.

That shared engine is where most of the tests point. There are a little over two hundred, on Node's built-in test runner, in a few layers:

• Engine and pipeline. Formats in and out, quality, lossless, both resize modes, rotation and flips, colour adjustments, text and image watermarks in each position, and the guards that refuse to vectorise a photo.
• The CLI as a real process. Flags, JSON output and exit codes. A batch with one corrupt file and one valid file has to convert the valid one and exit with 2, partial success.
• The packaged executable. A smoke test that runs the real binary and checks that what comes out decodes.
• The bridge. Every channel the preload script calls has a handler on the other side, and the renderer never gets require or process.

The presets live in one file, src/shared/presets.js, which the interface imports and the tests import too. A test of the Web preset reads the real preset, not a copy of its numbers, so changing a value there changes what gets tested.

Version 2.1 adds the command line to that list. The social presets behind its new --social flag come from the same file, and a saved profile goes through the same code in the app and in the CLI, with a test that checks both give the same image.

Fixtures built to make mistakes visible

None of the test images are photos. They're generated by code before the tests run, and each one exists to make one kind of mistake impossible to miss:

Fixture	What it is	What it catches
asym-h, asym-v	Half red, half blue	A flip that doesn't swap the halves
quadrants	A different colour in each quarter	Where a watermark, or now a crop, actually landed
gray120	Flat grey, 120 on every channel	Brightness, contrast and saturation as one measurable number
alpha	Semi-transparent green	Transparency lost where it should survive
corrupt	A sentence of text saved as .jpg	A broken file that takes the batch down with it
noise	Random pixels from a fixed seed	A "photo" that is identical on every run
high-freq	One-pixel black and white stripes	Only two colours, maximum detail: the case that got past the colour-only SVG guard

quadrants is the workhorse. Simplified, it's this:

// 80×60, a unique colour in each quarter: a test can tell exactly which
// corner or area an operation touched.
for (let y = 0; y < 60; y++) {
  for (let x = 0; x < 80; x++) {
    const i = (y * 80 + x) * 3
    const top = y < 30, left = x < 40
    const [r, g, b] =
      top && left ? [255, 0, 0]   // NW red
      : top       ? [0, 255, 0]   // NE green
      : left      ? [0, 0, 255]   // SW blue
      :             [255, 255, 0] // SE yellow
    buf[i] = r; buf[i + 1] = g; buf[i + 2] = b
  }
}
await sharp(buf, { raw: { width: 80, height: 60, channels: 3 } }).png().toFile(quadrantsPath)

Checking pixels without golden images

The tempting way to test an image pipeline is golden files: save a known-good output, then compare every new output against it byte for byte. It breaks every time the encoder changes how it rounds something, and when it breaks it tells you nothing about what is wrong.

So the suite asks narrower questions, each one with an answer that can't be in doubt:

• Dimensions and ratio, exactly.
• A pixel whose colour is certain. Flip asym-h horizontally and the pixel in the top-left corner must now be blue.
• A region that should, or shouldn't, change. A watermark in the south-east has to change that corner of quadrants and leave the opposite one alone, measured as a difference over the region against a threshold, not as an exact match.
• Exact equality only where it's promised. A lossless round trip has to give back identical pixels, because that is the entire promise of lossless.
• Order instead of value, where exact numbers would be brittle. A higher quality has to land closer to the original than a lower one, and the Min size preset has to spend fewer bytes per pixel than High quality.

Most of it comes down to a helper this small:

async function readPixel(filePath, x, y) {
  const { data, info } = await sharp(filePath).ensureAlpha().raw().toBuffer({ resolveWithObject: true })
  const cx = Math.min(Math.max(0, x), info.width - 1)
  const cy = Math.min(Math.max(0, y), info.height - 1)
  const i = (cy * info.width + cx) * 4
  return [data[i], data[i + 1], data[i + 2], data[i + 3]]
}

For crop, quadrants gets one more job. Crop it from 80×60 to a 60×60 square and the border between red and green tells you where the crop happened without anyone looking at the result. Anchored in the centre, the border lands at x = 30. Anchored left, at x = 40. Anchored right, at x = 20. Three pixel reads per anchor, and no image ever opened.

The crop test list

Before the crop was anywhere near done, I wrote down what it has to survive:

1. A phone photo stored sideways is upright before anything gets cropped.
2. The original survives. Always.
3. Every preset gives its exact ratio, for landscape, portrait and square inputs, and for an input that already matches.
4. The anchor lands where it says, read off quadrants.
5. Padding saved as JPG is the colour that was asked for, not black.
6. Transparency survives wherever the format supports it.
7. Rotation happens before the crop and the watermark after it, so a corner watermark is still there.
8. Small inputs are enlarged only when a final size asks for it, and the preview says so when they are.

The first two were supposed to be formalities. They're the two that failed.

Failure one: phone photos come out sideways

Many phone photos aren't stored the way you see them. The camera writes the pixels in one orientation and adds an EXIF tag saying how to turn them. To test that, I needed a photo like that, and none of the fixtures was one: every fixture is generated from raw pixels, and code doesn't hold the phone sideways.

So I made one. Stored as 200×100, red on the left and blue on the right, with orientation 6, which means "rotate 90° clockwise to display". Seen properly, it's a 100×200 portrait with red on top.

test('a photo stored sideways comes out upright', async () => {
  const red = await sharp({ create: { width: 100, height: 100, channels: 3, background: '#ff0000' } }).png().toBuffer()
  await sharp({ create: { width: 200, height: 100, channels: 3, background: '#0000ff' } })
    .composite([{ input: red, left: 0, top: 0 }])
    .withMetadata({ orientation: 6 })
    .jpeg()
    .toFile(phonePhoto)

  const result = await convertToFormat(phonePhoto, out, { outputFormat: 'webp' })
  const meta = await sharp(result.outputPath).metadata()

  assert.equal(meta.width, 100)   // fails: 200
  assert.equal(meta.height, 200)  // fails: 100
})

It failed with no crop involved at all. Convertim 2.0 writes it out at 200×100, without the tag. The engine never turns the image upright, and since it doesn't keep the metadata either, the tag that would have told a viewer how to turn it is gone too. A plain conversion, a resize: a photo stored sideways comes out sideways.

Crop only turns that from annoying into destructive. A sideways photo can be rotated back. A sideways photo cropped to 4:5 has already lost the wrong part of itself.

The fix turned out to be small. sharp 0.34 has autoOrient(), and in 2.1 it goes first in every path that reads an image (conversion, optimisation, vectorisation and the previews), always before the user's own rotation. The part that needed care was everything that measured the image: in 0.34, metadata() still reports the stored size, and the oriented size comes separately, in metadata().autoOrient. Every resize, every crop and every watermark is now computed from the second:

function getVisibleSize(metadata, transforms = {}) {
  const oriented = metadata.autoOrient || metadata
  const quarterTurn = transforms.rotation === 90 || transforms.rotation === 270
  return quarterTurn
    ? { width: oriented.height, height: oriented.width }
    : { width: oriented.width, height: oriented.height }
}

The test above is in the suite now, along with one for every other orientation value a camera can write.

Failure two: "never overwrite" had a gap

Convertim has one rule I considered unbreakable: it never writes over a file. If photo.webp already exists, the output becomes photo_1.webp. Crop makes that rule matter more, because a cropped image is missing pixels for good, and the original is the only copy of them.

The test for it had one thing none of the old tests had: two files with the same name, converted at the same time.

test('two inputs with the same name never share an output', async () => {
  const [a, b] = await Promise.all([
    convertToFormat(join(dir, 'photo.jpg'), dir, { outputFormat: 'webp' }),
    convertToFormat(join(dir, 'photo.png'), dir, { outputFormat: 'webp' }),
  ])
  assert.notEqual(a.outputPath, b.outputPath)   // fails: both are photo.webp
})

Both conversions chose photo.webp, and the second one wrote over the first. Every single time I ran it.

The name check and the write are far apart. The engine looks for a free name when a conversion starts, and the file only appears on disk when the encode finishes. In between, another conversion has time to look, see the same free name and take it too. The CLI converts one file at a time, so it never hits this. The desktop app runs several in parallel (half your CPU cores, by default), and its default output folder is the source folder, so photo.jpg and photo.png side by side are enough.

On its own, that costs one output. With Delete originals turned on, both originals are deleted after two "successful" conversions, and one of the two images is gone.

If you are still on 2.0, update. Until you do, set the concurrency to 1, or don't use Delete originals on a batch where names repeat.

The fix is in 2.1. The name is reserved in the same step that checks it: the output file is created empty and exclusively, with Node's 'wx' flag, which fails if the file already exists. If it does, the conversion moves on to the next suffix.

function reserveOutputPath(filePath) {
  const dir = path.dirname(filePath)
  const ext = path.extname(filePath)
  const base = path.basename(filePath, ext)
  for (let counter = 0; ; counter++) {
    const candidate = counter === 0 ? filePath : path.join(dir, `${base}_${counter}${ext}`)
    try {
      fs.closeSync(fs.openSync(candidate, 'wx'))
      return candidate
    } catch (e) {
      if (e.code !== 'EEXIST') throw e
    }
  }
}

The encoder then writes over its own reservation, and removes it if the encode fails. Behind that there's a second safety net: an original is only deleted if its output is on disk with exactly the size that conversion wrote.

Why two hundred tests never saw either

Neither bug was subtle once a test went looking for it. They survived because of what the fixtures were.

Every fixture is generated from raw pixels, so none of them carries metadata. Every test writes into its own fresh folder and converts one thing at a time. That makes the suite fast, reproducible and independent, which is exactly what a suite should be. It also means it lived in a world with no cameras and no two files with the same name.

Generated fixtures test the code you wrote. Real files bring the world with them: the tag a phone added, the folder where someone exported the same picture twice. Both bugs lived in the gap between the two.

So the list has a third kind of input now, next to the generated fixtures and the real files: generated ones that imitate what real files do, not only what their pixels look like. Orientation tags, repeated names, conversions that overlap.

What isn't automated

Some things are still checked by eye, on purpose. Before every release there's a manual checklist, around five hundred lines long, run against a folder of real files the generated fixtures can't imitate: a JPG over 5 MB, an animated GIF, a JPG truncated with a hex editor, a filename with spaces, an ñ and an emoji, and folders nested three deep.

For crop, that folder grows: real portraits, real landscapes, screenshots and logos on transparent backgrounds, run through every preset and looked at as a grid of thumbnails. It's the only test that can answer the one question the automated ones can't: does the default look right?

Done means it fails predictably

A batch feature isn't done when it works on one image, or even on two hundred clean ones. It's done when it fails predictably: when the two hundredth image behaves like the first, when a broken file fails on its own and the rest of the batch carries on, when two files with the same name can never become one, and when the worst a bad default can do is something you notice in a grid of thumbnails, not something you discover after posting.

Both fixes shipped in 2.1 on Monday, and both are tests now, so neither can come back unnoticed. They went out with the rest of the list, once running it had become boring. That was the whole release plan.