Counting colors stops telling screenshots from photos after one JPEG save

I wanted an upload pipeline to pick compression settings by content: palette PNG for screenshots, lossy JPEG or WebP for photos. The obvious classifier is a unique-color count, since a screenshot has a few hundred colors and a photo has hundreds of thousands. That held up right until I fed it a screenshot that had already been saved as JPEG once, which is how a lot of screenshots actually arrive.

The setup

Eight images, all measured on an M4 Mac with sharp 0.34.5 for encoding and NumPy for the math. Two screenshots I rendered myself at 1280×800: a fake Chinese meeting-notes page and a fake order table with colored status badges. Two photos from Wikimedia Commons, scaled to 1280 wide and kept lossless: a misty lake by Jakub Fryš and a fruit stall in Arequipa by Josep M. Gracia, both CC BY-SA 4.0. Two 413×579 portrait JPEGs, also from Commons, cropped to ID-photo proportions. Then each screenshot saved once more as JPEG quality 80. That is a small sample, and none of the numbers below are meant as accuracy figures.

Where the color count broke

Lossless, the count looks perfect: 174 and 1,402 colors for the screenshots, 119,846 and 634,115 for the photos. After one JPEG save at quality 80, the order table jumped to 21,500 colors. The blue-background portrait JPEG sits at 26,827. Same order of magnitude, so no threshold separates them.

The reason is how JPEG works. It transforms and quantizes 8×8 blocks and halves chroma resolution, so every sharp edge picks up ringing. A table full of lines and colored badges is nothing but sharp edges, and each one spawns a fringe of new in-between colors. Each new color covers only a few pixels, but there are a lot of them. The meeting notes only went from 174 to 2,360, because black text on white gives chroma little to smear.

What survived the JPEG save

So I stopped asking how many colors there are and asked how concentrated they are. A screenshot is mostly one background plus a handful of text colors, and JPEG noise does not move the big flat areas. Two numbers did the job:

def color_concentration(img_path, top_n=16):
    px = np.asarray(Image.open(img_path).convert("RGB"), dtype=np.uint8)
    packed = px.astype(np.uint32) @ np.array([65536, 256, 1], dtype=np.uint32)
    _, hist = np.unique(packed, return_counts=True)
    top_share = np.sort(hist)[-top_n:].sum() / packed.size
    repeat_right = np.mean(packed[:, 1:] == packed[:, :-1])
    return len(hist), top_share, repeat_right

The share of pixels covered by the 16 most common colors was 0.920 and 0.954 for the lossless screenshots and still 0.838 and 0.745 after JPEG. All four photos landed between 0.039 and 0.135. The fraction of pixels identical to their right neighbor told the same story: 0.740 and 0.786 for the JPEG screenshots, at most 0.260 for any photo. I also tried a Sobel strong-edge ratio. It barely moves under JPEG (0.1408 vs 0.1396 on the text page), but a busy photo and a sparse UI only differ by about 3x (0.0222 for the fruit stall, 0.0692 for the table), so I would only use it as a tiebreaker.

Why the direction of the mistake matters

Sending a photo to a 256-color palette costs real bytes: both photos came out 2.5 to 3.8 times larger than JPEG quality 80. On the lake's sky, turning dithering off produced flat steps of color, and turning it on traded the steps for grain. The fog in the same photo showed no bands at all, which I read as one hue only needing 256 steps along a single axis.

I used ImgIng (https://imging.ai/) in Chromium 149 as a reference point and just read its auto-detection label for eight images: the four originals plus both portraits at 413×579 and at full size. Both screenshots went to an 8-bit PNG preset. The lake, both small portraits and the full-size blue-background portrait were labeled photos at WebP quality 75. Two photos were labeled screenshot/UI: the fruit stall and a 2996×4200 portrait original. They stayed WebP and only got quality 84 instead of 75. No screenshot was labeled a photo.

The next part is my reasoning, not a measurement. A screenshot pushed through photo settings would get the gray speckles around text that I saw in the JPEG versions, and people notice that right away. A photo on the screenshot side only costs bytes, as long as that side never forces it into a palette, and the numbers above show how many bytes a palette can cost. So if I were tuning such a classifier, I would let uncertain images lean toward the screenshot label but give them a middle tier that stays lossy at a higher quality. From the outside, the quality-84 result looks roughly like that.

If you have a classifier like this, rerun it on screenshots that were re-saved as JPEG, and add a few detail-heavy photos to the test set before trusting the threshold.