DEV Community

yue xing
yue xing

Posted on

Limited range video has 220 grey levels, not 256: what one flag does to the same pixels

Limited range is usually summed up as "luma runs from 16 to 235". That line alone doesn't say what changes in a real picture, so this run used a file whose steps can actually be counted. The grey test pattern is homemade: 1280×720, a ramp across the top with every value from 0 to 255 at 5 px each, and nine flat patches below at 0, 8, 16, 64, 128, 192, 235, 247 and 255. It was encoded as 3-second 8-bit H.264 (CRF 10, BT.709) in four versions: real range full or limited, flagged full or limited. Each one was then decoded back to RGB with ffmpeg and the centre of every patch was read. Everything ran once on one Apple-silicon Mac, so read the numbers as "this file on this machine".

Limited range has 220 luma codes

Full range gives luma all 256 codes of an 8-bit byte. Limited range keeps 16 as black and 235 as white, which leaves 235 − 16 + 1 = 220 codes for the whole trip from black to white. The usual conversion squeezes each 0–255 value with roughly Y = 16 + 219 × v / 255 and rounds the result. In this file the stored codes matched that: mid grey 128 sat at 126, and white 255 at 235.

The ramp kept 220 of 256 steps

In the version that was really limited and correctly tagged limited, all nine patches decoded back to exactly 0, 8, 16, 64, 128, 192, 235, 247 and 255, the same as the full-range file. Judging by the patches alone, the two files would pass as identical. The ramp disagreed. Counting distinct steps across it, the full-range file had all 256 and the limited one had 220. That is 36 steps fewer. By the arithmetic (not a separate measurement), 36 neighbouring greys must have been rounded onto codes they share with a neighbour, and a decoder has no way to split them again. So in 8-bit, a correctly handled limited range costs fine gradation, not contrast. My guess is that this shows up as banding on slow gradients, but real footage wasn't tested this time.

One flag decides where black sits

Now the washed-out case. The correctly made limited file had only its range flag changed to full, without touching a single pixel. Decoded, the patches read 16, 23, 30, 71, 126, 181, 218, 228 and 235. Black came out as 16 and white as 235. The distance from black to white shrank from 255 code steps to 219, which works out to roughly 86%. That's the grey veil. Each reading lines up with the stored code from the formula above (these figures are my own arithmetic, not extra measurements): 8 becomes 22.87 and rounds to 23, 64 becomes 70.97 and rounds to 71, 192 becomes 180.9 and rounds to 181. The decoder was told the codes already covered 0–255, so it showed them as they were and skipped the stretch back.

The opposite mistake is not just "less grey". Full-range pixels tagged as limited read 0, 0, 0, 56, 130, 205, 255, 255 and 255: the 17 darkest levels (0–16) all became 0, the 21 brightest (235–255) all became 255, and the midtones were stretched apart. The shadows clip to pure black, the highlights clip to pure white, and the result is the reverse of washed out. So "my video looks grey" points at only one of the two possible mislabels.

An MP4 stores the flag twice

A second detail: an H.264 MP4 carries the range in the bitstream (the VUI) and again in the container (the colr box). Editing only the bitstream with the h264_metadata filter left the container saying the old thing, and ffprobe then gave two answers: -show_streams reported the container value, -show_frames the bitstream one. With two deliberately mismatched files, both ffmpeg's decoder and a desktop Chromium video element drawn onto a canvas followed the bitstream and ignored the container, and the two paths agreed within 1 on every patch. QuickTime, VLC, Safari, phone galleries and chat apps weren't tested, so whether players differ is still open.

A tool that converts on purpose

For a conversion done on purpose rather than by accident, the full-range file also went through the video compressor in ImgIng https://imging.ai/ on the same Mac in the same desktop Chromium, at the "visually near-lossless" quality level. The 3-second still pattern came out larger than it went in, so the tool didn't adopt the result by default; the candidate was saved manually. Both the H.264 and H.265 outputs came back as limited range and were tagged tv, the correct label. Decoded, black and white read 0 and 255 for H.265, and 2 and 254 for H.264. There was no 16-and-235 veil and no clipping, but full range wasn't preserved either: the pixels were converted, not passed through. One 3-second synthetic file says nothing about other footage, though.

What I check first now

When a clip looks grey, I now ask two things before re-encoding anything: which range the pixels are really in, and what the bitstream flag claims. If the codes sit between 16 and 235 and the flag says full, the picture is fine and only the label needs fixing. What I still can't tell you is what phones or screen recorders write by default. There was no original phone footage on hand, only re-processed files. HEVC, 10-bit and HDR are untested as well, so all of the above is 8-bit H.264 with BT.709.

Top comments (0)