DEV Community

AsyncMonk
AsyncMonk

Posted on

A skip rule for re-encode scripts, and why the bpp threshold moves with your CRF

#webdev #javascript

A re-encode job in a side project has one quiet failure mode: it runs over a file that is already small and hands you back a bigger one. I wanted a skip rule for that, so I measured where the line is. The line turned out not to belong to the file. It moves with the CRF you encode to.

Start from bits per pixel

The script cannot compare raw Mbps, since the same number is generous at 720p and stingy at 1080p. So it works in bits per pixel per frame: bpp = bitrate / (width * height * fps). Across my content types, the point where re-encoding stopped growing files sat at about 0.09 to 0.13 bpp.

The setup: three real clips (a crowd at 720p, a moose at 1080p, a street at 1080p) plus one synthetic testsrc2 clip, all with audio stripped. Each was encoded once at nine CRFs from 12 to 48 with x264 medium, which gives 36 "already compressed" files. Each of those was then re-encoded at CRF 23 and again at CRF 28. Numbers below are this set, one run, on an M4 Mac.

At CRF 23 the line sits near 0.10

Of the 36 files, 24 had a source bpp under 0.08, and every one of them came out bigger, from 1.34× to 6.67×. Four files between 0.08 and 0.13 saved 4% to 9%.

With a cutoff of 0.10, 26 files fall below it. 24 of those grow, and the other two save only 9% and 4%. Nothing that would have saved 10% or more gets skipped.

The same file gets two verdicts

Take the 1080p moose clip pre-encoded at CRF 28. It is 2,670,854 bytes with a bpp of 0.0440. Re-encoded at CRF 23 it becomes 3,949,234 bytes, 1.48×. Re-encoded at CRF 28 it becomes 2,402,497 bytes, which is 10.05% saved. Same input, opposite answer.

My guess is that the cutoff is roughly the bpp your target settings would produce on that content, and below it the encoder has to add bits. I did not test that mechanism on its own. What the table below shows is that the cutoff for CRF 28 comes out lower, around 0.06.

Cutoff Target Files below Of those, grew Files above that grew
0.06 CRF 23 23 23 1
0.06 CRF 28 23 20 0
0.10 CRF 23 26 24 0
0.10 CRF 28 26 20 0

Use 0.06 while encoding to CRF 23 and one file that grows 1.34× slips through. Use 0.10 while encoding to CRF 28 and three files that would have saved 10% or more get skipped.

Tie the threshold to the target

I did not fit a formula, because I only have two points. The script keeps a lookup and refuses anything else. The part that matters:

THRESHOLD = {23: 0.10, 28: 0.06}
crf = int(sys.argv[1])
if crf not in THRESHOLD:
    sys.exit(f"no threshold measured for CRF {crf}")
# ... bpp from ffprobe, then:
print(f"{'SKIP' if v < THRESHOLD[crf] else 'ENCODE':6s} crf={crf} bpp={v:.4f} {p}")
Enter fullscreen mode Exit fullscreen mode

Run against the moose clip above:

SKIP   crf=23 bpp=0.0440 clip-mid.mp4
SKIP   crf=28 bpp=0.0440 clip-mid.mp4
no threshold measured for CRF 35
Enter fullscreen mode Exit fullscreen mode

Notice the second line. At CRF 28 the rule skips a file that would have saved 10.05%. I accept that. The rule is meant to fail on the cautious side, and I would rather leave 10% on the table than have the job quietly hand back bigger files.

Where the rule stops

Everything here is H.264 to H.264 with preset medium. I did not test H.265, AV1, other presets, or target-bitrate mode, so treat the numbers as a starting point for your own CRF rather than a constant. Five real VP8 and VP9 clips from Wikimedia Commons agreed with the 0.10 rule at CRF 23, but five is a small set. WebM also reports bit_rate=N/A on the video stream, so the script falls back to file size ÷ duration, which runs a little high when audio is inside.

For the five Commons clips I did not need a script. I ran them through the video compressor at ImgIng https://imging.ai/, which takes several files at once and lets you pick the quality profile. It runs in a desktop browser only, since phones and tablets get image compression there. Its default smart profile kept the original for four of the five, so choose a profile if you want every file processed.

Clip credits

Sample videos are from Wikimedia Commons, re-encoded, trimmed and otherwise altered.

  • Moose by the road.webm, Forza1000, CC BY-SA 4.0
  • A Demonstration of Disabled People June 2018 1.webm, DGW, CC BY-SA 3.0
  • Worldwide Demonstration – March 2021 Amsterdam.webm, Karen Eliot, CC BY-SA 2.0
  • Demonstration of Ethiopian residents in Israel, July 2019.webm, Israel Police, CC BY-SA 3.0
  • Demonstration in Al-Zabadani, 23 March 2012.webm, Allfor Syria, CC BY 3.0

Top comments (0)