My compressor has a row of small buttons under it. Each one advertises a saving:
"about 3% smaller in our tests", "+0.6%", "about 2% larger", "5–10% smaller".
Those numbers came from six images and a PSNR score.
Six images is a spot check, not a test. And PSNR was the wrong metric — my own
encoder code says so, which is why it had already been replaced with SSIMULACRA2.
So the captions were quoting a number from a method I had stopped trusting, printed
next to a button four times a day.
I re-ran it on 320 images. Two captions didn't survive, and one of them wasn't an
error of degree — it was comparing the wrong two quantities.
How it was measured
320 images, at most six from each folder of a collection, so no single style
dominates. Each was scaled so its longest side is 768 px and saved as a PNG. That
PNG is the reference everything is scored against.
Then the same reference was encoded once per setting at quality 90, chroma sampling
held at 4:4:4. One variable: which button you press.
The settings aren't redefined for the test — they're imported from the same module
the app runs. That matters more than it sounds: if I'd copied the parameters into a
test script, I would have measured the script.
Scoring is SSIMULACRA2 against the reference. Higher is better.
What came back
- balanced (the default): 243.1 KB, mean score 86.12
- size (standard tables): 225.4 KB, −7.28% and score 86.74, smaller on 318 of 320
- quality (trellis): 244.2 KB, +0.44% and score 86.14, smaller on none
- compat (baseline): 250.3 KB, +2.94% and score 86.12, smaller on none
- jpegli: 244.0 KB, +0.36% and score 89.60, smaller on 166 of 320
Caption one said "about 3% smaller". It's 7.28% smaller, and the mean score is
slightly higher than the default rather than equal. On 306 of 320 images the score
went up.
I don't have a satisfying explanation for that. "Quality 90" doesn't mean the same
thing for the two quantisation tables — mozjpeg's tuned table is built for images
being squeezed hard, and at q90 it spends its budget differently. I haven't traced
which coefficients change. That's a guess and I'm labelling it as one.
Caption two said "5–10% smaller than the default at the same setting". At the
same quality number, jpegli comes out 0.36% larger — and scores 3.48 points
higher, winning on all 320. The 5–10% figure comes from a different comparison:
equal measured quality, not equal slider position. There's a calibration table for
exactly that, and at matched scores jpegli really is smaller by that much.
Both statements can be true. But the button sits directly under a quality slider, so
"at the same setting" reads as "at the same slider position". Under that reading the
caption was false.
There's a general lesson here that cost me less than it could have: when you write
a comparison, write down the thing you're holding constant. Both of these were
"true" measurements of something. Only one of them answered the question the UI
was asking.
What I changed
Three captions, in both languages. size now says 7%, compat says 3%, and jpegli
says what it actually buys: higher measured quality at roughly the same file size,
rather than a size saving at the same quality number.
The trellis caption held up better (+0.6% predicted, +0.44% measured) but its
"a little more detail" only held on 195 of 320 images. The wording says so now.
The number I did not change, and why
At q90 the standard-tables setting is smaller and scores higher than the tuned
default. That's a strong enough result that it looked like the default should move.
It moved — but only after a second run. I tested the rest of the quality curve,
because the tuned table's whole justification is that it degrades more gracefully
when you compress hard, and q90 says nothing about that:
- q40: tuned 73.0 KB / 60.44 · standard 75.1 KB / 62.89 → 2.88% bigger, 2.45 points higher
- q50: tuned 86.5 KB / 65.71 · standard 87.7 KB / 67.38 → 1.39% bigger, 1.67 higher
- q60: tuned 101.3 KB / 69.64 · standard 101.2 KB / 70.82 → dead even, 1.18 higher
- q70: tuned 122.9 KB / 74.23 · standard 121.2 KB / 75.05 → 1.38% smaller, 0.82 higher
- q90: tuned 243.1 KB / 86.12 · standard 225.4 KB / 86.74 → 7.28% smaller, 0.62 higher
The hypothesis was wrong in a specific way. The tuned table does produce smaller
files below q60. It just never scores higher at any quality I tested. So "designed
for heavy compression" means "smaller at the same nominal setting", not "better at
the same nominal setting". Those are different claims.
Above q60 it's bigger and scores lower, and my slider defaults to 80. So the
default became size, and balanced stays available for the case it's actually
good at.
Caveats
768 px images — most photos people compress are bigger, and encoders don't behave
identically across scales.
One collection, mostly already-compressed JPEGs, so this is the cost of a second
pass rather than a first one.
One build of mozjpeg and one of jpegli. Both ship changes.
Five quality points is a curve sketch, not a curve. The crossover is somewhere
between 50 and 60 by my data; I didn't refine it and I'm not going to quote a precise
number I didn't measure.
And a score is not your eyes. SSIMULACRA2 tracks human judgement well on average —
but "on average" is carrying weight in that sentence.
If you want to run it
The pipeline is short enough to rebuild in an afternoon. Resize each source so its
longest side is 768 px and save a PNG; that PNG is the reference. Encode that same
reference once per setting at quality 90, and score every output against the
reference with SSIMULACRA2. Pillow handles the resize, and ssimulacra2 is a single
binary you call once per file.
320 images took about eight minutes for the encoder comparison and eleven for the
quality curve. Most of that was the scorer rather than the encoders.
If you run it on your own library and get a materially different number, I'd like to
hear it. A contradiction with a concrete example is more useful than agreement, and
the address is on the about page.
Top comments (0)