Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing

Originally published on my blog — read it there with the full context, code and updates.

The question I could not answer: which number do I ship?

My transcoding queue decides, per file, which encoder does the work: copy
when the source is already efficient, NVENC on the GPU, x265 on the CPU —
and now Intel Quick Sync. Each tier has two knobs that decide quality, and
every tier except QSV had numbers I had measured on my own library.

QSV had none. And the code refused to guess: a shipping default of
qsv_quality = 0 / qsv_preset = "" means nobody has calibrated this,
and the engine treats that as a hard exclusion — QSV can exist in the code
and never be selected on any machine, including the one that has the
hardware.

So the task was not "turn on QSV". It was "produce measured numbers for two
knobs, so the encoder becomes selectable honestly".

Problem one: the hardware is not in the machine where the app runs

The queue runs on Windows 11 in a VM. That VM has an RTX 3060 passed through
and no Intel graphics of any kind — which is exactly why QSV had never been
measured: the obvious move is to run one test locally and read the numbers
off.

The iGPU is on the host instead (unRaid, i9-13900K), idle and unclaimed:

00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770]
  driver: i915            refcount: 0
/dev/dri/card0  /dev/dri/renderD128        (present, world-accessible)
no running container declares a dri device
virsh dumpxml Win11: hostdevs = 01:00.0, 01:00.1 (the 3060 + its HDMI audio)

Two conclusions, both useful. The VM's Error creating a MFX session is
expected and not a bug to chase — the VM can never see that iGPU. And
the only way to measure QSV is on the Linux side, in a container, on the
host.

Measure with the engine's own gate, not with ffmpeg

This is the part that makes the numbers worth anything. I did not write
ffmpeg commands and compute SSIM myself — the calibration runs the engine
that will do the work, with its own verification and its own accept/reject
gate:

python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \
  --qsv-quality Q --qsv-preset P --path <real source> --keep-local --overwrite

--keep-local means encode, verify, run the gate, and then do not land
the file — the verdict is what I am collecting, not a new library. Sources
came in read-only over an existing SMB mount, and I hashed each source
before and after: unchanged.

The image is linuxserver/ffmpeg (ffmpeg 9.0.2, iHD 26.3.5, oneVPL 2.17)
plus python3 — plus numpy, which is not optional. Without it the gate
prints flat-frame scan skipped: No module named 'numpy' and quietly
changes how near-flat frames are judged, which changes verdicts.

-global_quality maps to RateControlMethod: ICQ on hevc_qsv, which is
the knob being calibrated.

The ladder: on a thin source, low ICQ makes the file bigger

First run, on a real 1080p25 source at 951 kbps (0.0184 bits per pixel — a
"thin" file, the kind that is already squeezed):

ICQ	engine verdict	mean SSIM	video = source
18	FAIL size — 176.8%	—	—
20	FAIL size — 141.8%	—	—
22	FAIL size — 107.6%	—	—
24	ok	0.9967	81.1%
26	ok	0.9959	61.0%
28	ok	0.9951	46.4%
30	FAIL quality	0.9942	—
32	FAIL quality	0.9942	—

Three things fall out of that table.

The counterintuitive one: low ICQ does not mean safe. ICQ 18–22 produce
files larger than the source, so the engine's size ceiling (100% of the
source's video bitrate) rejects them. QSV is not alone in this — NVENC at a
low cq does the same thing on thin inputs — but the boundary sits higher
here: the usable range starts at 24, not 18.

Second, there is a cliff between 28 and 30. 30 and 32 fail on quality,
and they fail on the first attempt. That phrase matters, because the
engine retries a failed quality attempt at global_quality - 2 — lower
ICQ, higher quality — and logs the retry. A line reading "ok at quality+2"
is a first-attempt failure, not a pass. I read that line backwards in my
own notes the first time; the rule is to read the first attempt, not the
recovery.

Third, the floor is where the choice gets made: with a 0.995 mean-SSIM
floor, q28 passes with 0.0001 of margin and q26 with 0.0010.

Full files confirm the ladder — and one shortcut survives

Excerpts (180-second lossless slices) are how you rank ten candidates
without ten full encodes. They are only legitimate if they agree with
full-file runs, so I ran the real files end to end:

source	ICQ	preset	video = source	file	SSIM min / mean	encode
254 MB, 2023 s, 872 kbps	26	medium	61.0%	168,435,266 B (66%)	0.9938 / 0.9960	360 s ≈ 5.6× realtime
the same file	28	medium	46.3%	54%	0.9927 / 0.9952	330 s
315 MB, 3280 s, 546 kbps	26	medium	50.4%	60%	0.9965 / 0.9977	600 s ≈ 5.5× realtime

The 180-second excerpt of the first file measured mean 0.9959; the full file
measured 0.9960. That is close enough to trust the shortcut for quality
ranking — and I now have the evidence to say so rather than hope so.

The preset that does nothing

The excerpt runs said preset only changes speed: veryfast 30 s / 61.6%, fast
30 s / 61.0%, medium 60 s / 61.0%, slow and veryslow 60 s. Clean story: pick
veryfast, save 30 seconds per file.

Then the full files disagreed with it. The same source at ICQ 26, one run
with fast and one with medium:

fast:    168,435,266 B   enc 360 s   mean SSIM 0.9960
medium:  168,435,266 B   enc 360 s   mean SSIM 0.9960

Byte-for-byte identical, same encode time, same quality. The 30-vs-60-second
difference in the excerpt layer was per-run fixed overhead, not encoding
speed — and it is the kind of number that would have gone into a config file
and a blog post if I had stopped one step early.

So: excerpts rank quality, never speed. Speed has to be measured on full
files, where the fixed overhead disappears.

What I shipped

qsv_quality = 26, qsv_preset = medium — 61.0% of the source's video
bitrate at mean SSIM 0.9960 (min 0.9938) at roughly 5.6× realtime, and 50.4%
at mean 0.9977 on the thinnest file in the library. q28's margin over the
floor is 0.0002, and the cost of a quality failure is two full encodes plus
a park, so the extra 15 percentage points of savings are not worth the tail
risk.

Two supporting measurements worth recording, because they answer "is it QSV
or is it the file?":

• Cross-encoder control. One 720p source with non-uniform cadence scored QSV q24 mean 0.9154, QSV q28 mean 0.9146, and x265 crf26 mean 0.9164 — and the engine's alignment search reported no alignment explains the mismatch. The rejection belongs to the source, not to Quick Sync.
• Concurrency. The engine refuses a second encoder on a shared work root: another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old). With separate work roots, two jobs run — and take 201 s where one takes 107 s. Twice the wall clock, zero throughput gain. One QSV job at a time.

Four traps, each of which cost a cycle

1. Never mount a shared library at /lib. I mounted the SMB share into the container at /lib — which is where the container's own dynamic loader lives. All eight jobs died in under a second with exec /bin/bash: no such file or directory, which reads like a broken image, not a bad mount. /library works.
2. Synthetic sources lie about ICQ. hevc_qsv's rate control is rejected by the driver for rgb24 input (Selected ratecontrol mode is unsupported) — so a testsrc/mandelbrot self-test concludes the opposite of the truth. Use a real file, or add format=nv12.
3. Already-HEVC sources cannot be calibrated at all. The gate answers SKIP already HEVC hvc1 and --overwrite does not lift it. The fat 1080p H.264 originals in this library had all been converted long ago; the remaining fat files are HEVC — so that tier has nothing left to measure on. That is a missing input, not a skipped task.
4. Read the first attempt, not the retry (see the cliff above).

What I would do differently

1. Measure speed on the artifact you will actually run, not on a slice of it. My one wrong number in this whole exercise came from trusting an excerpt's wall clock.
2. Make the machine prove it has the hardware before you plan the test. Six minutes of lspci and virsh dumpxml retired the "why does QSV fail in the VM" question that had been open for weeks.
3. Keep the gate in the loop. Because the engine's own verification judged every run, the tables above are decisions, not opinions — the same gate that will reject a bad file in production rejected my ICQ 30.
4. Write the honest scope line into the notes. These numbers are Linux iHD 26.3.5 + oneVPL 2.17 on a UHD 770; the production target is the Windows MSDK path on the same silicon, and I cannot claim they agree. What this establishes is that the command runs on real hardware, where the quality↔size curve sits, and how the lock behaves — leaving a re-measurement on Windows+Intel as one table, instead of another afternoon.

The result

QSV went from "present in the code and never selectable" to a calibrated
tier with published numbers, and the safety rule changed with it: the test
that used to pin shipping defaults must mean uncalibrated now pins
shipping defaults must be measured values, while keeping every assertion
that an empty value is a hard exclusion. The doctor output on a machine with
no Intel GPU moved from an informational "waiting for hardware" line to a
warning — which is its own small lesson about what a calibrated encoder
failing self-check should look like, and why hardware the machine does not
have belongs in a summary row rather than a warning.

---

Need Resilient Hardware Video Pipelines & Automated Quality Gates?

If you manage GPU/iGPU transcoding fleets (NVENC, Quick Sync, VAAPI) or need strict frame-level quality assertions (SSIM/VMAF) to keep defective encodes out of production libraries:

• WhatsApp: [phone]
• Email: [e-mail]
• Website: hoelee.com

---

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.