DEV Community

celticgemini001
celticgemini001

Posted on Edited on

Your upscaler's best PSNR can belong to the frame that looks broken

I spent an afternoon trying to answer a question that sounded simple: when a
video has black bars, should you crop them before upscaling or after?
Everyone crops first. I did too. But I wanted the number that proves it, and the
number turned out to be a bad choice of instrument.

This post is about that instrument. The crop-order result is real, but the more
useful finding is that PSNR — the metric you'd naturally reach for — is
structurally blind to this class of error, and there is one cheap check that
catches it.

The setup

A 960×720 picture inside a 1280×720 frame: 160 px of black bar on each side.
That's a 16:9 picture in a 4:3 frame, which is what happens the moment you
letterbox anything for an old display, a vertical platform, or a fixed-aspect
player.

Two paths:

  • Crop first: crop=960:720:160:0 on the source, then upscale the picture.
  • Crop after: upscale the whole 1280×720 frame, then cut the bars off the scaled output.

The picture has high-contrast detail deliberately placed at x=0, right where the
bar meets it — the worst case, not the average case.

import numpy as np
from PIL import Image

BAR, PW, PH = 160, 960, 720          # 160px bar each side, 960x720 picture

x = np.arange(PW, dtype=np.float32)
pic = np.clip((200.0 - 120.0 * (x / (PW - 1)))[None, :]
              * np.linspace(0.85, 1.15, PH, dtype=np.float32)[:, None], 0, 255)
pic = np.clip(pic + 18.0 * np.sin(2 * np.pi * x / 8.0)[None, :], 0, 255)
yy, xx = np.mgrid[0:PH, 0:PW]
pic[((xx - 480) ** 2 + (yy - 360) ** 2) < 200 ** 2] += 35.0
pic = np.clip(pic, 0, 255).astype(np.uint8)

frame = np.zeros((PH, PW + 2 * BAR), dtype=np.uint8)
frame[:, BAR:BAR + PW] = pic

def up(a, s, k):
    return np.asarray(Image.fromarray(a).resize((a.shape[1] * s, a.shape[0] * s), k),
                      dtype=np.float32)

def db(mse):
    return 10 * np.log10(255.0 ** 2 / mse) if mse > 0 else 99.0

for S in (2, 4, 8):
    truth = up(pic, S, Image.LANCZOS)                                  # crop first
    wrong = up(frame, S, Image.LANCZOS)[:, BAR * S:BAR * S + PW * S]   # crop after
    err = (wrong - truth) ** 2
    col = err.mean(axis=0)
    band = 0
    while band < len(col) and db(col[band]) < 50:
        band += 1
    print(f"S={S}x band={band}px ({band/S:.2f} src) "
          f"1px={db(col[0]):.2f}dB 8px={db(err[:, :8].mean()):.2f}dB "
          f"whole={db(err.mean()):.2f}dB")
Enter fullscreen mode Exit fullscreen mode

Actual output on this machine:

S=2x band=5px (2.50 src) 1px=15.73dB 8px=23.39dB whole=46.72dB
S=4x band=9px (2.25 src) 1px=11.31dB 8px=19.16dB whole=45.49dB
S=8x band=19px (2.38 src) 1px=9.62dB 8px=16.19dB whole=45.22dB
Enter fullscreen mode Exit fullscreen mode

Read the second number in each line. At 4×, the first column of the picture is
11.31 dB against ground truth. That is not "slightly soft". At 11 dB the
pixel is essentially unrelated to what it should be. Eight columns in, still
19.16 dB. And the whole-picture figure — the number you'd put in a test report
— is 45.49 dB, which sounds excellent.

The band is a property of the kernel, not the scale

I expected the damage to grow with the scale factor. It doesn't, in source
pixels:

Scale Kernel Band (output px) Band (source px) 1px PSNR 8px PSNR Whole picture
2× lanczos 5 2.50 15.73 23.39 46.72
2× bicubic 3 1.50 15.97 24.42 47.79
2× bilinear 1 0.50 14.16 23.19 46.54
4× lanczos 9 2.25 11.31 19.16 45.49
4× bicubic 6 1.50 11.33 19.80 46.13
4× bilinear 2 0.50 10.63 19.20 45.55
8× lanczos 19 2.38 9.62 16.19 45.22
8× bicubic 12 1.50 9.66 16.58 45.85
8× bilinear 4 0.50 9.28 15.97 45.33

Lanczos sits at 2.25–2.50 source px at 2×, 4×, and 8×. Bicubic is pinned at
exactly 1.50. Bilinear at 0.50. That's the filter support, which is fixed in
source coordinates — the number of output pixels moves, the number of source
pixels doesn't.

This is where the intuition "it only affects a pixel or two at the edge" goes
wrong. It affects two source pixels, which at 8× is nineteen output columns.
Scaling up doesn't spread the damage; it just renders the same damage in more
pixels.

Bilinear looking "cleanest" here (0.50 source px) is also a trap — it wins that
column by having the narrowest support, which is the same property that makes it
the softest kernel overall. It's not doing a better job; it's touching less.

Whole-frame PSNR prefers the broken frame

Here is the part that made me rewrite this post. Four candidates at 4×, all
scored against the same ground truth:

Candidate Whole PSNR Worst column Columns < 30 dB 8px left edge
Crop-first, lanczos (correct) — — 0 —
Upscale frame, then crop (bleed) 45.49 11.31 6 19.16
Crop-first, bilinear (uniformly soft) 49.25 42.11 0 45.14

The bleed path scores 45.49 dB. The uniformly soft path scores 49.25 dB.
If you rank by whole-frame PSNR, the soft frame wins by 3.76 dB — and the soft
frame is the one where every edge in the picture is mush.

The bleed frame has six columns below 30 dB. The soft frame has none. PSNR's
mean-squared aggregate is dominated by area, and the damaged area is 18 columns
out of 3840 — 0.47% of the picture. It cannot outvote 99.5% of correct pixels, no
matter how wrong those 18 columns are.

I also threw in a "pre-blur 1.5 px then lanczos" candidate expecting it to score
between the two. It scored 99.00 dB — bit-identical to ground truth. That was a
bug in my candidate list (the round-trip through a same-size resize was a no-op,
so I compared ground truth to itself). I'm leaving it in the post because it's
the exact failure mode this article is about: an instrument that reports a
perfect score and means nothing. If a number looks too good, check that you
actually changed the thing you think you changed.

How wide would the damage have to be to register?

Taking the measured first-column error at 4× lanczos (11.31 dB) and asking how
much of the picture would have to be that bad before the whole-frame number
moved:

Target whole-frame PSNR Band width needed % of picture width
45 dB 2 px 0.05%
40 dB 5 px 0.13%
35 dB 16 px 0.42%

So the whole-frame metric can see a 16-pixel band — but it needs the entire
band to be at first-column severity. The real band decays as you move inward
(11 dB, 23, 26, 21, 23, then clear), so the average over the band is far milder
than its first column, and the aggregate never fires. The math isn't wrong; it's
answering a different question than the one you asked.

The video path agrees

I didn't want to publish a stills-only result, so I ran the same comparison
through ffmpeg with lossless intermediates (-crf 0 -pix_fmt yuv444p) to avoid
letting the encoder hide or create the artifact:

# picture master, no bars
ffmpeg -loop 1 -i pic.png -t 2 -r 25 -c:v libx264 -crf 0 -pix_fmt yuv444p pic.mp4

# add bars, keep it lossless
ffmpeg -i pic.mp4 -vf "pad=1600:900:200:0:black" -c:v libx264 -crf 0 -pix_fmt yuv444p frame.mp4

# crop first, then scale
ffmpeg -i frame.mp4 -vf "crop=1200:900:200:0,scale=4800:3600:flags=lanczos" \
  -c:v libx264 -crf 0 -pix_fmt yuv444p A_cropfirst.mp4

# scale first, then crop
ffmpeg -i frame.mp4 -vf "scale=6400:3600:flags=lanczos,crop=4800:3600:800:0" \
  -c:v libx264 -crf 0 -pix_fmt yuv444p B_upscalefirst.mp4
Enter fullscreen mode Exit fullscreen mode

Per-frame PSNR against the lossless picture master, 5 frames:

Path Whole frame Left band Right band 1px 8px left
Crop first 99.00 0 px 0 px 99.00 99.00
Upscale first, then crop 45.97 9 px (2.25 src) 9 px 10.57 18.88

Same 9-pixel band, same 2.25 source pixels, arrived at through a completely
different pipeline. Two independent harnesses agreeing on the geometry is worth
more than either one alone.

Note the asymmetry in the stills test that doesn't appear here: the left edge
is always worse than the right (1px 11.31 dB vs right-edge 8px 28.59 dB). That's
because the test picture's brightness ramps left to right, so the left bar is a
much bigger step than the right one. Bar bleed scales with the contrast across
the seam, not with distance from it.

Cost

Upscaling the frame instead of the picture is 33.3% more pixels (921,600 vs
691,200 at this size). Measured, 4×:

Kernel Upscale-then-crop Crop-first Ratio
lanczos 123 ms 95 ms 1.30×
bicubic 95 ms 73 ms 1.31×
bilinear 72 ms 54 ms 1.34×

About 1.3× the compute for a worse result — you pay 30% more to break the edge.
On a 10-second 30 fps clip that's 300 frames, so on the order of 8 seconds per
clip at this size, and the ratio holds.

What I'd actually check

  1. Never compare whole frames when the defect is a thin band. Compute per-column MSE and look at the first and last 8 columns against ground truth. That's the metric that sees this.
  2. Crop, pad, or shift before the resample — not after. In ffmpeg, crop goes first in the filter chain. This is not a style preference; the filter order is the fix.
  3. If a metric reports a suspiciously perfect score, verify you changed the input. My pre-blur candidate scored 99.00 dB because it was a no-op. That's a more common failure than a genuinely perfect result.
  4. Sanity-check bar detection, don't trust the default. cropdetect's default limit of 24 uses max >= limit, so a bar with a luma of exactly 24 gets treated as picture. A noisy bar at luma 20 ± 4 also hits 24 and gets kept. Raising the limit past that starts eating genuinely dark picture instead — there is a range where no threshold is both sensitive and safe.

The clip-order question has a boring answer, which is why it's easy to get wrong
in a hurry: crop first. The interesting part is that the standard metric would
have let you ship the wrong one and reported 45 dB while doing it.


If you want to see these comparisons run on real footage rather than a synthetic
test pattern, upgolab.com has the same pipeline available
in the browser — it's the same crop-order logic, and it's free to try on your own
files.

Further reading, all measured the same way: crop the black bars before upscaling: the measured band that whole-frame PSNR cannot see; upgolab-ai-upscaler-guide; anamorphic SD and non-square pixels: what the 360 cycles/width ceiling costs, measured.

Top comments (0)