Two complaints arrived in the same week. The first was about time: upscaling a 2048px photo took nine seconds, even on the WebGPU path, and users on weaker hardware saw the progress bar crawl. The second was about taste: the results looked over-cooked — skin turned plastic, edges grew light halos, and if your source was a JPEG, the compression blocks came back louder than before. One complaint asked us to go faster; the other, if you squinted, asked us to add less. This is the story of how we chased the first one honestly, and why the fix for the second one ended up being a dial that gives some of that speed back.
We build ClearPix, the in-browser media toolbox from parts 1–9 of this series — no uploads, no server GPUs, Real-ESRGAN-class super-resolution running inside a browser tab. If you haven't followed along: our upscaler had already been through one generational swap (the 67MB RRDBNet flagship for a 4.9MB compact variant that ran 14–27× faster and won the quality shootout anyway) and one quantization gamble (fp16 weights, another 4.8× on top, told in part 3). What follows is the next chapter, where the easy wins were gone.
Act 1: Build the ruler before touching anything
We have a house rule by now, earned the expensive way: no optimization starts without a per-stage timing breakdown, and no quality change without a metric that can veto it. The upscale path had speed numbers but no quality numbers at all — "over-sharpened" was a vibe, not a measurement. So before touching a line of code we built the bench: 15 cases, five content types (people, structure, texture, sharp edges, flat gradients) crossed with three degradations (clean source, JPEG q70, JPEG q40), each scored for PSNR, SSIM, and a high-frequency ratio against the ground truth.
Only then did we learn what the second complaint actually was. Everyone, us included, had assumed "over-sharpened" meant edge halos — the classic ringing artifact. The bench said otherwise: on genuinely sharp content our output was actually too soft (hfRatio 0.29, meaning it had lost high frequency versus the original). The real villain was hiding in boring places: flat and soft regions, where the model hallucinates texture that was never there, and JPEG artifacts, which it amplifies faithfully. On flat gradients the hallucination metric climbed monotonically with source degradation — 3.47 on clean, 4.70 on JPEG q70, 5.55 on JPEG q40. Your compressed screenshot didn't just get upscaled; it got its blockiness upscaled too.
That number, 5.55, deserves a footnote because it nearly sent us chasing ghosts. High-frequency ratio divides by the original's high-frequency energy — and on a nearly flat original, that denominator is almost zero, so the ratio explodes regardless of absolute damage. We added an absolute metric (hfAbs) alongside the ratio and confirmed the flat-region hallucination is 0.12–0.16 intensity levels: real, but invisible. Half of "fixing" a problem is making sure you can see it accurately.
Act 2: The speed chase — three honest failures and one three-line win
With the bench as our safety net, we profiled. The per-stage breakdown held a surprise: model inference was 62% of wall time, sure — but pasting results back into the output canvas was 36%, and format conversion 1%. A third of our runtime was a CPU loop copying pixels, not AI at all.
That loop did per-channel Math.max(0, Math.min(255, value)) clamping while writing into a Uint8ClampedArray — which clamps on assignment by spec. Deleting the redundant math took three lines and bought 88ms → 68.4ms per image (−22%), with output that is bit-for-bit identical. It remains my favorite kind of optimization: zero risk, instant payback, embarrassing only in hindsight.
The natural next step was pipelining — hide the paste behind the next tile's inference, the classic overlap. We built it, verified bit-identical output, measured… nothing. Wall time didn't move. The reason was quietly funny: the pipeline was designed to hide a 36% bottleneck that the three-line fix had just reduced to 4%. We reverted the whole thing the same day and wrote the lesson into our runbook: re-profile after every fix; never parallelize against a profile that's already out of date.
Bigger tiles, then? 384px instead of 256 would mean fewer of them. On our Apple-silicon GPU it was a wash — 941/849ms against 888/825ms — the GPU was already saturated at 256. The nuclear option was skipping half the work: feed the model a half-resolution input and let ×2 cover the rest. It was glorious — 205ms down to 55ms per tile, a 3.8× speedup — and the bench killed it just as fast: photos lost 2.6–2.9dB, sharp content lost 6.5dB. The four-times cost is what quality-first upscaling is. We recorded the falsification prominently so nobody (especially future us) would re-propose it in six months.
A final systems check closed the case: every operator in the model runs natively on ORT's WebGPU backend with zero fallback warnings, and 115ms for a 256² tile is simply what this architecture costs on this hardware. Scheduling layer: squeezed dry. Model candidates: falsified. Runtime: current. The remaining 97% of inference time belongs to upstream — ORT's WebGPU evolution, or a hand-written native WebGPU super-res kernel, which is a project unto itself. We stopped, which is its own discipline.
Act 3: The honest dial
So we were as fast as honesty allowed, and still slightly over-cooked. The model's hunger for high frequency is baked into its training — SRVGGNet is a GAN-family network, and "adds crisp detail" is literally its job description. We couldn't retrain it. What we could do is scale what comes out.
The fix is almost comically small: a separable 2-pixel box filter applied to the model's output at final resolution, with a single gain factor k. k = 1 is the raw model. We swept k across the bench and the results settled a small argument in the team: k = 0.8 — giving back 20% of the output's high-frequency energy — improved PSNR on photos (+0.33dB on people, +0.44dB on texture, +0.27dB on edges) with SSIM flat-to-up. Read that again: the super-resolution model was adding high-frequency detail to photographs that the ground truth didn't have, and removing some of it made the output measurably more truthful. "Over-sharpened" wasn't user pickiness; it was a real, quantified bias. The default became k = 0.8, shipped with a 60–100% slider because taste is personal and JPEG-free line art genuinely likes 1.0.
Two adjacent ideas got proper burials. Rebuilding the output with matched film grain (to mask the flat-region hallucination) turned out to be unnecessary — the hallucination is k-immune but, per the absolute metric, invisible anyway. And Real-ESRGAN's official "denoise" model variants, which we hoped would be a gentler built-in preset, turned out to be detail-reconstruction strength knobs — the "denoise" setting adds more high frequency, not less. Both are archived in the bench directory with their falsification data, which is where good rejected ideas live.
One deliberate gap: the video upscaler shares this kernel but doesn't get the slider yet, because the sharpening helper costs 961ms at 4K — more than the inference itself. It needs a GPU-side implementation before it's worth anyone's frames.
Where the walls are
Today the pipeline is: clamp input, split into 256px tiles with 8px overlap, run the 2.4MB fp16 compact model on WebGPU, paste back only the centers so tile seams never show, then one honest pass of the sharpening dial. A 2048px photo costs about a second of GPU time on a modern laptop, and the output is — by our own bench, which exists precisely so we can say this — slightly closer to the truth than the raw model wanted to be.
The nine seconds became one, and the plastic skin became a preference slider. Both complaints, it turned out, were the same complaint: the model was working harder than the truth required. Speed and honesty weren't a trade-off — they were the same fix, applied at different layers.
If you want to see it yourself: clearpix.org/upscale-image runs the whole pipeline in your tab — the sharpen slider is under the result. The tile-and-overlap engine and memory walls behind the other half of this story are in part 5, the fp16 quantization adventure in part 3, and the delivery layer that gets the 2.4MB model into your browser in part 9.
Part 10 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.
Top comments (0)