Real-CUGAN on Linux vs Windows: same flags, two NVIDIA GPUs, SSIM 0.9977

#ai #linux #ffmpeg #gpu

I upscale short AI-generated clips to 4K with Real-CUGAN, the ncnn-vulkan build. Two laptops with NVIDIA GPUs can do the work:

	Windows laptop	Linux laptop
CPU, RAM	i7-13700H, 32 GB	i5-9300HF, 23 GB
GPU	RTX 4060 Laptop, 8 GB	RTX 2060, 6 GB
OS	Windows 11 (build 22631)	Arch Linux, kernel 7.2.9
NVIDIA driver	610.88	615.78.08
ffmpeg	8.1.3	9.0.2
realcugan-ncnn-vulkan	20220728 release	20220728 release

Before I split one video's clips between them, I wanted to know whether they give the same picture. If they didn't, a cut between two clips upscaled on different machines could show it.

The test

One clip, 768x1344 at 24 fps, 104 frames, went through the same steps on both machines: crop to an exact 9:16, write PNG frames, Real-CUGAN at 3x with the models-se weights and denoise level 0, Lanczos down to 2160x3840, then lossless x264. The Linux side:

ffmpeg -i "$src" -vf "crop=$cw:$h" -fps_mode passthrough in/f%06d.png
realcugan-ncnn-vulkan -i in -o up -s 3 -n 0 -g 0 -m models-se -f png
ffmpeg -framerate 24 -i up/f%06d.png -i "$src" -map 0:v -map '1:a?' \
  -vf scale=2160:3840:flags=lanczos -c:v libx264 -qp 0 -preset veryfast \
  -pix_fmt yuv420p -c:a copy -frames:v 104 out_4k.mp4

Windows runs the same three commands from Python, with one difference: -g 1. On that laptop GPU 0 is the integrated Intel chip, and -g 0 would run the upscale there.

The result

Both files have 104 frames. I compared them with ffmpeg's own filters:

ffmpeg -i windows_4k.mp4 -i linux_4k.mp4 -lavfi \
  "[0:v]split[a0][a1];[1:v]split[b0][b1];[a0][b0]ssim=stats_file=ssim.log;[a1][b1]psnr=stats_file=psnr.log" \
  -f null -

SSIM came out at 0.9977 (Y 0.9976, U 0.9978, V 0.9980) and PSNR at 54.2 dB. Frame by frame the range was narrow: SSIM 0.99758 to 0.99779, PSNR 54.00 to 54.45 dB. Nothing drifts over the clip.

A single score hides how the differences are spread, so I also counted the absolute difference of every Y, U and V sample, 1.29 billion of them:

75.5 % of the samples are identical, and 99.97 % are within one level out of 255. The largest difference anywhere is 5 levels, and only 5 samples reach it. The mean absolute difference is 0.25 of a level. Side by side at 100 % zoom I couldn't tell the two apart.

The core of the counting script, in case you want to run it on your own pair:

p = lambda f: subprocess.Popen(["ffmpeg", "-v", "error", "-i", f, "-f", "rawvideo",
                                "-pix_fmt", "yuv420p", "-"], stdout=subprocess.PIPE)
a, b = p(win), p(lin)
size = 2160 * 3840 * 3 // 2          # one yuv420p frame
hist = np.zeros(256, np.int64)
while (x := a.stdout.read(size)) and (y := b.stdout.read(size)):
    d = np.abs(np.frombuffer(x, np.uint8).astype(np.int16) - np.frombuffer(y, np.uint8))
    hist += np.bincount(d, minlength=256)

Where the difference comes from: not separated

The two sides differ in GPU, driver, operating system and ffmpeg version, all at once. The candidates are the floating-point math in the Vulkan shaders, on two GPU generations and two drivers, and ffmpeg's crop, Lanczos scaling and RGB to YUV conversion in two versions. The x264 step at -qp 0 is lossless; the only loss in that step is the conversion to yuv420p, which belongs to the ffmpeg candidate.

To split it, compare the PNGs that Real-CUGAN writes before the encode step. My pipeline deletes them after each clip, so I don't have that answer yet.

Speed, from the job log

I run these jobs with stageload, a small open-source tool that runs heavy jobs one at a time and logs each one's run time. Its log gives whole-job numbers: copying the clip over the LAN, extracting frames, upscaling, encoding and copying the result back. Divided by frame count:

Job	Machine	Frames	Run time	Per frame
Real-CUGAN 3x, 4K lossless out	RTX 4060, Windows	506 (one job, 5 clips)	515.5 s	1.02 s
Real-CUGAN 3x, 4K lossless out	RTX 2060, Linux	104 (one clip)	232.7 s	2.24 s
Real-ESRGAN x4plus, 4K lossless out	RTX 4060, Windows	73 each, 11 jobs	383 to 422 s, mean 401.5 s	5.50 s
Real-ESRGAN x4plus, 1080p out	RTX 2060, Linux	73 each, 3 jobs	542 to 558 s, mean 548.8 s	7.52 s

Read these with care. The Real-CUGAN rows are one job each, and the Linux job also ran a 4K lossless x264 encode on a CPU with 8 threads, limited to 4 threads at low priority. The Real-ESRGAN rows come closer to a GPU comparison, though not all the way, because the Linux jobs encoded to 1080p, the lighter encode. There the RTX 2060 laptop took 1.37 times as long. One two-clip Windows job took 3,374 s, far above the rest, and I left it out because I don't know why.

On the same Windows laptop, Real-ESRGAN x4plus (4x) cost about 5.4 times as much per frame as Real-CUGAN at 3x. I still use it for photoreal footage: Real-CUGAN is trained on anime, and on realistic faces it turned skin into painted strokes.

What I do now

Before I mix machines in one video, I upscale one clip on both and run the two checks above. If the frame counts match and the differences look like this histogram, the clips go to whichever laptop is free. The SSIM and PSNR pass took 11 seconds on my Mac for 104 frames at 4K. It catches setting mistakes, such as the wrong -g or a different denoise level, before they reach an edit.

DEV Community