Our upload service at work compresses every image with one JPEG setting. Product photos come out fine. Screenshots of order pages and admin tables come back with grey speckles around the text, and they are not even small. I want the service to pick a format per image: a palette PNG for screenshots, WebP for photos. The classifier is the part everyone asks about, but before proposing this I wanted numbers for the parts around it, because those are what I would have to defend in review. How much does the check cost per upload, what happens when the chosen format makes a file bigger, and how do we notice when someone changes a parameter six months from now?
The test set and the classifier
Two 1280×800 screenshots I rendered from made-up pages (Chinese meeting notes, and an order table with coloured status badges), two photos from Wikimedia Commons, downscaled to 1280 px and stored as lossless PNG (a lake by Jakub Fryš and a fruit stall by Josep M. Gracia, both CC BY-SA 4.0), the two screenshots re-saved as JPEG quality 80, and two 413×579 JPEG portraits cropped from CC BY-SA headshots on Commons. That is eight images, plus a 2996×4200 crop of one portrait used only for timing. Everything ran in Node with sharp 0.34.5 on an M4 Mac. I skipped the unique colour count as a classifier, since the order table had 21,500 colours after one JPEG save, close to a portrait's 26,827. I used two concentration measures instead: how much of the image the 16 most frequent colours cover, and how many pixels repeat the colour of their left neighbour in the same row.
Before writing anything I looked at how a shipped tool splits the two. I used ImgIng (https://imging.ai/) in Chromium 149 on the four originals and the two portraits at both sizes. Screenshots defaulted to PNG-8 with 256 colours and photos to WebP at quality 75. Two photos, the fruit stall and the 2996×4200 portrait, were labelled screenshot/UI, yet they only got quality 84 and stayed WebP. That behaviour is where my middle lane below came from.
Compute the features on a nearest-neighbour thumbnail
At full size the 12.6-megapixel portrait took 404 ms including decode. Shrinking to a 512-pixel long side first brought it down to 52 ms. The resize kernel mattered more than I expected. With nearest-neighbour, the screenshots lost at most 0.03 on either measure and the photos stayed at or below 0.25. With sharp's default lanczos3, interpolation invents in-between colours along every edge, and the screenshots' top-16 share fell to 0.697–0.812.
async function thumbFeatures(file) {
const { data, info } = await sharp(file)
.removeAlpha()
.resize(512, 512, { fit: 'inside', kernel: 'nearest', withoutEnlargement: true })
.raw()
.toBuffer({ resolveWithObject: true });
const tally = new Map();
let sameAsLeft = 0;
for (let y = 0; y < info.height; y++) {
let last = -1;
for (let x = 0; x < info.width; x++) {
const o = 3 * (y * info.width + x);
const rgb = data[o] * 65536 + data[o + 1] * 256 + data[o + 2];
tally.set(rgb, (tally.get(rgb) ?? 0) + 1);
if (rgb === last) sameAsLeft++;
last = rgb;
}
}
const n = info.width * info.height;
const top16 = [...tally.values()].sort((a, b) => b - a).slice(0, 16).reduce((s, c) => s + c, 0) / n;
return { top16, run: sameAsLeft / n };
}
Three lanes and a keep-original fallback
The rule, picked while writing and not tuned: if the smaller of the two values is at least 0.6, it is a screenshot and goes to a 256-colour palette PNG with dithering off. If the larger is at most 0.35, it is a photo and goes to WebP 75. Anything else goes to WebP 84. If the output is not smaller than the input, the service keeps the input. All eight images landed in the right lane and none in the middle, but the fallback still earned its place. The two screenshots that arrived as JPEG grew as palette PNGs, from 164,265 to 256,076 bytes and from 111,650 to 159,108, so the originals were kept. Forcing them through WebP 75 instead gave 105,738 and 54,482 bytes, and the text one scored a higher SSIM against the lossless original (0.9947) than the JPEG input itself did (0.9917). Measured against the JPEG input it looks worse, so the choice of reference can flip the conclusion. I haven't decided whether already-JPEG screenshots should get a lane of their own.
A wrong lane costs very different amounts depending on the direction. A lossless screenshot forced into WebP 75 came out about a tenth smaller than its palette PNG, but lossy (the text page: 103,054 bytes, PSNR 40.39). A photo forced into the palette blew up: the lake went from 56,414 to 417,298 bytes and the fruit stall from 127,602 to 461,340.
Test the route, not just the average
To find out whether a check would catch a bad change later, I simulated two: palette capped at 64 colours, and photo quality dropped to 50. The first check used fixed gates (screenshots PSNR ≥ 50, photos mean SSIM ≥ 0.93, output not bigger than input). It caught all four screenshots at 64 colours but only two of four photos at quality 50. The fruit stall (0.9543) and the blue-background portrait (0.9476) slipped through. The second check compared each image against a saved baseline run and flagged a lane change, size up by more than 10%, mean or worst-1% SSIM down by more than 0.01, or PSNR down by more than 3 dB. It flagged four of eight in both cases, which is every image the change touched.
Mean SSIM on its own would have waved the 64-colour change through: the four screenshots scored 0.9997, 0.9962, 0.9991 and 0.9913. The worst 1% of SSIM windows is far more sensitive on text. The order table at JPEG quality 80 averages 0.9874, while its worst 1% is 0.8386. One thing I could not measure is the coloured halo around status badges that is obvious when you zoom in. A chroma-only SSIM never dropped below the luma one on these images.
The thresholds (0.6 and 0.35) and tolerances (0.01, 10%, 3 dB) came from eight images, so I am not quoting an accuracy figure, and I haven't tried any of this with canvas in the browser. Before anyone changes the defaults at work, I plan to run the baseline over a sample of real uploads that our compliance team lets us copy out of production.
Top comments (0)