DEV Community: Umasou

Automatic Watermark Detection: Cutting False Positives on Complex Backgrounds by Design

Umasou — Tue, 20 Oct 2026 15:16:17 +0000

Our automatic watermark detector returned 125 text boxes on 24 real user photos, and every one of them was pre-selected. Ten were on a baby's face, a film still, a rock, a towel — never text at all. The other 39 we now drop are real text, but it sits on backgrounds an in-browser inpainter cannot rebuild without hallucinating. So we changed the rule: auto-detect now reports only what the model is genuinely good at — 76 boxes, no subject false positives, tiled watermarks untouched — and hands the rest to the user.

The numbers up front

Measurement	Before	After
Text boxes on a 24-photo real-user set	125	76
Boxes dropped by the new gate	—	49 (10 by shape, 39 by background)
Subject false positives we could verify by eye	10	0
Tiled-watermark boxes kept (case-1 / case-2)	38 / 22	36 / 19
Site sample ("SAMPLE ONLY")	detected	still detected
Quality bench (13 synthetic cases)	recall 89.2 %, precision 65.3 %, psnrRegion 24.8 dB	bit-identical

Nothing in that table required a new model, a bigger model, or a server. It required deciding which detections we are willing to act on automatically.

What actually goes wrong: the detector fires on subjects

Our text detector is PP-OCRv5's DB head — the same 4.7 MB model described in part 6. It is a shape detector. It answers "where could text be", not "is this a watermark", and it fires on any structure that looks like strokes: high-contrast, roughly periodic, thin.

Natural photographs are full of such structures. Eyebrows, eyes and lips. Fingers and knuckles. The crumb structure of a dessert. Fabric folds and hair. Window mullions, balcony railings, a crowd. None of them are text, and none of them look like text to a shape detector.

The uncomfortable part is that these false positives are not weak. The boxes sitting on a baby's face and on a film still scored 0.81–0.94 mean detection probability — squarely inside the range of real watermarks on the same set (0.74–0.995). Every confidence gate we already had (mean score ≥ 0.6, minimum area, fill ratio, area cap) was blind to this, because the detector was not confused. It was confidently reporting a face.

That is the general lesson: "how confident" and "should we act" are different questions. No confidence threshold can separate "real watermark" from "real text-like texture", because the second one scores just as high.

The second half of the problem: the inpainter cannot rebuild texture

Detection is the ceiling on removal, but only one half of it. Our inpainting is MI-GAN by default and LaMa as an opt-in quality tier, both running at 512² tiles on the user's own GPU (or WASM on the CPU). That architecture reconstructs a hole from its surrounding context. On a homogeneous background it matches the statistics and the result is invisible. On a busy background it produces texture that is statistically plausible and spatially wrong.

We have measured both ends of this. On a city-texture case from our synthetic bench, leaving a faint watermark alone scored MAE 3.11 against the clean original; repainting the rectangle with the inpainter scored 24.79 — eight times worse than doing nothing. On real user photos the same effect shows up as a car-shaped hallucination on sand, a horizon-shaped smear band across a cloud layer, and a rock-like blob where a ridgeline used to be.

So the two failure modes compound. A false-positive box on a complex background is the worst possible combination: the model is asked to invent texture, and it invents it over the user's actual subject. A false-positive box on a plain wall is merely ugly.

Why we could not just make the model stronger

The usual fix for a noisy detector is a better detector. That was not available to us, and the constraints are worth stating plainly, because they are the reason this post is about product rules rather than about model work.

No server-side fallback. The product's promise is that your photo never leaves the tab. We cannot send a hard case to a bigger model and get an answer back; that would be an upload, and it is the one thing we refuse to do.
No elasticity. A hosted detector can be re-trained, re-scaled or replaced with a larger model when a bad case appears. A local model in the user's browser is whatever we shipped: a 4.7 MB OCR model, an 11 MB YOLO model, and a 102.8 MiB fp16 LaMa tier behind an opt-in. Making the detector meaningfully better means new weights, new hosting, and another download for every user.
The device is the budget. Phones and low-end laptops run the same 512² tile pipeline; the model's receptive field is fixed by its architecture, not by the request.
We have no training set for "watermark vs. scene text". Separating an overlay from a printed label on a product needs semantics and supervised data we do not have, and we cannot collect it from user photos without asking people to upload their images.

Given those constraints, the only lever left is deciding which detections we act on automatically. If the model cannot tell an overlay from a face, we can at least tell a simple background from a busy one, and we can tell a text line from a blob.

The decision: play to the model's strengths, hand the rest to the user

The rule we shipped is deliberately conservative: auto-detect only reports obvious text on a locally simple background. Everything else is the user's job — and the user's path for that is not worse than before. Drawing a box runs the same region-constrained detector inside the selection plus shape refinement, so a user who wants a watermark on a busy background can still get it, with the mask snapped to the actual strokes.

This is a change of philosophy, and we want to be explicit about it. Our detection pipeline had been running on "a miss is recoverable, a false positive is not" (the rule we stated in part 12): if we miss a faint watermark, the user brushes over it and the result is fine; if we eat their photo, there is no undo for trust. This change tightens that into: in auto mode, zero false positives first; a miss is handed to the user.

The gate, in two criteria plus one exemption

The gate runs after the existing box-level gates (score, area, fill) and before shapes are assembled. It applies to the image tool's auto-detect path only — the quality bench, the video pipeline and the manual refine path are untouched.

Criterion 1 — the box must look like a text line. A text box must be elongated: aspect ratio ≥ 1.5 (horizontal) or ≤ 0.4 (vertical). Small boxes (short side < 40 px) are exempt, because single characters in a tiled watermark are naturally squarish and tiny. Near-square large boxes are the classic OCR-on-a-subject shape, and this is what killed the baby face (111×85), the film-still boxes (66×54, 133×107, 197×109), the plate of food (46×100) and the cosplay portrait (277×344).

Criterion 2 — the background around the box must be simple. We measure a ring band around each box (pad = 35 % of the short side, clamped to 8–64 px) and compute two statistics over it: the standard deviation of luminance, and the mean absolute gradient. A box is accepted if the ring's std ≤ 35 and gradient ≤ 14.

Two details make that measurement honest rather than decorative. First, only ring pixels count — the pixels inside the box are the watermark's own strokes, and letting them into the statistic would make every watermark look like it sits on a busy background. Second, the gradient is only taken between ring pixels; a stroke edge is not background complexity.

Calibration came from the 24-photo set. Where a real watermark sat on a plain wall, water or sky, the ring measured std 7–30 and gradient 3–10. Where a false positive sat on a face, an arm, food or a crowd, it measured std 32–89 and gradient 4–27. The thresholds sit in the gap.

The exemption — repetition is evidence. A group of four or more text boxes with matching sizes (within ±30 % in width and height) is treated as a tiled watermark, and tiled watermarks are exempt from both criteria. Repetition is itself strong evidence that you are looking at an overlay rather than picture content, and tiled watermarks are small per instance, so a single instance is a local hole the inpainter can genuinely fill — on a mountainside if it has to. Our two tiled cases kept 36 of 38 and 19 of 22 boxes; what was dropped there is the low-confidence edge fragments plus, in one case, the single large central title that sits on the ridge.

One footnote on method, because we had been burned by it before. An earlier round concluded that "ring complexity has no discriminative power" — that result was about a different question (should this region use a rectangle mask or a glyph mask?). The label space matters: the same measurement separates nothing when it picks a mask strategy, and separates well when it picks whether to auto-select a detection. We have written that distinction into our tuning notes so the two results are not quoted as one.

What it costs and what it buys

Eliminated, verified by eye on the output overlays: a baby's face and a pair of legs in a family photo, three boxes on a film still, a rock formation in a mountain photo, a towel and a pillow in two selfies, a face-shaped box on a cosplay portrait, a plate of food.

Left to the user, deliberately: watermarks that sit on busy backgrounds — a large "Luma AI" title over a sunset ridge, a property agent's phone number across a fridge, "LIONSGATE" over clouds, a hand-written title over a cloud bank, a tattoo artist's name along a hand, a wordmark across a model's hair — plus in-screen text that is caption rather than overlay (meme captions, shop signage, a social-app username bar). Those detections are real, and we are choosing not to auto-erase them, because on that background the artifact is the inpainter's output, not the removal. A user who wants them can draw one box and get the same glyph-level mask.

The rest of the pipeline did not move. The synthetic bench — 13 cases, on average recall 89.2 %, precision 65.3 %, region PSNR 24.8 dB — is bit-identical, because the gate is off in the bench path. (Two bench gates were already failing before this change: one case at 12.3 dB region PSNR and the faint tiled case at 47.9 % recall. Those are pre-existing and unrelated.) The manual refine path is unchanged, and the gate is a single option on the detector call, so it can be turned on or off per surface.

We did turn it on for the video watermark remover, where a false positive is worse than in a still image: the mask is applied to the whole clip, so a hallucinated patch is replayed frame after frame. The subtitle remover keeps full auto-detection on purpose — a subtitle is the thing the user asked to delete, it can sit on a face or a street scene, and filtering by background would defeat the tool.

Try it

The detector described here runs in the free in-browser watermark remover, together with the glyph-level masking from part 6 and the mask-discipline fixes from part 12. Nothing is uploaded: detection, mask building and inpainting all happen in the tab. If you want the other half of the pipeline — upscale a photo 4× without uploading it — it runs on the same local models.

Part 15 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor

Umasou — Sun, 18 Oct 2026 04:23:18 +0000

Our fp16 quantization post-mortem was wrong in the expensive direction: it told anyone with a mask-gated model to give up on half-precision, and that verdict would have kept a 52%-smaller download off the internet. The real cause was sixteen BatchNorm variance tensors larger than fp16's maximum representable value, silently clamped by the converter, leaving sixteen layers normalizing their output by the wrong divisor. The fix is two lines of algebra. The reward is a model that matches fp32 and downloads 52% smaller.

This is a correction post, so it comes with receipts: the lab notes for this round are checked in next to the code, every figure below is reproducible from them on a CPU-only ONNX Runtime install, and the whole investigation fits in a day or two of work.

The claim we published

Part 3 of this series ends with a case study called "When fp16 quantization quality loss is real: the LaMa failure". The measurements in it are real, and the failure is reproducible: rebuilding that conversion from the same weights today, the unfixed fp16 model averages 15.8 dB against fp32's 29.7 dB across our 14 acceptance cases, and its worst case scores 11.1 dB — the watermark surviving, exactly as described. Part 5 repeats it as "its catastrophic mirror image when we tried it on LaMa". Part 1's series index summarises the lesson as "mask-gated architectures and precision sensitivity".

We offered a mechanism too, and it was a good story. LaMa is mask-gated: the mask doesn't just mark where to work, it gates the network's behavior. A mask-gated inpainter therefore has a decision boundary inside it — which regions are trustworthy context, which must be regenerated — and half-precision noise pushes the model across that boundary. The gating fails softly, the model starts copying the watermarked pixels through instead of repairing them, and the output looks plausible except the watermark is still there.

That story explained the symptom exactly. It was consistent with everything else we'd learned (browser inference is bandwidth-bound, so moving half the bytes is free money). And it produced a tidy rule that we then repeated to readers:

whether quantization is safe is a property of the model's architecture, not of your tooling

The rule is wrong. Not "wrong with exceptions" — wrong as a predictor, and expensive: it would have kept a quality-identical model that is 48% smaller out of the product forever, and it tells every reader with an attention head, a gating mask, or a long residual chain to stop investigating. Part 8 of this series was about a quality bug that turned out to be ours, not the model's. This is the same shape of mistake, one layer lower in the stack: our conversion tooling, not the model's architecture.

Step one: prove that precision is not even the variable

"FP16" bundles two independent changes: mantissa precision (how finely you can split a value) and numeric range (the largest magnitude you can hold at all). If the failure were precision sensitivity, then rounding the mantissa while keeping fp32's range would degrade quality in proportion. So we built models that keep the graph and all arithmetic in fp32, and only round the weights' mantissas:

Weights rounded to	Region PSNR (5 cases)	Output fidelity vs fp32
fp32 (baseline)	28.88	—
7-bit mantissa (bf16)	28.83	66.9–76.9 dB
10-bit mantissa (= fp16)	28.88	74.5–96.6 dB
13-bit mantissa	28.88	95.9–109.0 dB

fp16 has a 10-bit mantissa. The 10-bit row is 0.004 dB off the fp32 baseline, and even throwing away 16 of the 23 mantissa bits (7-bit, coarser than bf16) costs 0.054 dB. LaMa is not precision-sensitive — it is precision-indifferent. Whatever fp16 was doing to it, it wasn't the mantissa.

That left range, and range has a hard, checkable boundary: fp16's largest finite value is 65,504. So we dumped every fp32 initializer in the graph and sorted by magnitude.

Step two: find the numbers that don't fit

The graph is 17,480 nodes with 606 float32 initializer tensors, 51M parameters, 204 MB of weights — 197 MB of it in the FFT path (17,022 of those nodes are Fourier convolutions, 97%). Run the check below over the 606 tensors and sixteen come back above 65,504, and every single one is a BatchNorm running_var:

     484322.56  model.generator.model.5.conv1.bn_l.running_var
     440853.66  model.generator.model.25.running_var
     240497.69  model.generator.model.22.conv1.bn_l.running_var
     218544.31  model.generator.model.11.conv1.bn_l.running_var
     174809.92  model.generator.model.6.conv1.bn_l.running_var
      65564.05  model.generator.model.10.conv1.bn_l.running_var
      ...  16 of 606 tensors, every one of them a running_var

Note the last row: the marginal tensor clears the ceiling by 0.09%, and it gets clamped to exactly the same 65,504 as the one that clears it by 7×. That's the whole hazard — a clamped value sits exactly at fp16's ceiling, which is a value fp16 can barely hold, so the next fp16 pass over that tensor takes it to infinity.

This is a five-line check, and it is the first thing to run before believing any "this model can't be quantized" conclusion:

import numpy as np, onnx
from onnx import numpy_helper

m = onnx.load("lama_fp32.onnx", load_external_data=False)
worst = sorted(
    ((float(np.abs(numpy_helper.to_array(t)).max()), t.name)
     for t in m.graph.initializer if t.data_type == 1),   # 1 = float32
    reverse=True,
)[:16]
for value, name in worst:
    print(f"{value:>14.2f}  {name}")

Sixteen tensors. Not sixteen layers of masking logic — sixteen numbers.

Why a BatchNorm variance is the thing that overflows

A BatchNorm layer computes y = (x − μ) / sqrt(var + ε) · γ + β. running_var is a scale quantity measured over the layer's input during training. If a layer's inputs live at large magnitudes, its variance lives at the square of those magnitudes, and 484,322 is entirely legitimate in fp32: sqrt(484322) ≈ 696.

Clamp that variance to fp16's ceiling of 65,504 and the divisor becomes sqrt(65504) ≈ 256. The layer now normalizes by 256 instead of 696 — its output is 2.7× too large — and the convolution that consumes it was trained against the correct scale. Sixteen of those, in series, and the network's internal magnitudes are destroyed while the graph still runs and still produces plausible-looking images.

That last part is the trap. There is no NaN, no warning, no crash: just a model that quietly keeps the watermark.

The clamp that never said a word

The conversion never told us anything, and this is the part worth internalising. onnxruntime.transformers.float16.convert_float_to_float16 clamps to max_finite_val=65504; onnxconverter_common.float16 defaults to 10000 (48× more aggressive). Both clamp silently — that's what "clamp" means: the fp16 model has a finite, wrong number in it, and nothing anywhere prints a line.

You can watch the corruption happen tensor by tensor, which is how we finally believed it:

Stage	`model.5.conv1.bn_l.running_var`
fp32 source	484322.5625
after `keep_io_types` fp16 conversion	65504.0 (clamped to the fp16 ceiling)
after a further fp16 storage pass	inf
after our fix (see below)	1892.0 (fits comfortably in fp16)

Note the third row: the clamped value is already at the ceiling, so a second conversion overflows it to infinity. Whether your broken model lands on 65504 or inf depends on which converter pass touched it last — which is also why the failure looked so erratic in our old notes.

The fix is algebra

We tried the blunt version first: keep the sixteen offending BatchNorm nodes in fp32 and convert everything else. That works, costs 34 KB of fp32 parameters, and produced identical quality — but it leaves a graph with fp32 islands and sixteen permanent assertions about which layers are dangerous.

The better fix is to notice that a BatchNorm's variance and its scale are only meaningful relative to each other:

y = (x − μ) / sqrt(var + ε) · γ + β
  ≡ (x − μ) / sqrt(var/K² + ε/K²) · (γ/K) + β

So pick a power-of-two K per layer, divide the variance by K², divide the scale by K, and divide epsilon by K² — the layer is mathematically identical, and now var/K² fits in fp16. With K = 2^ceil(log2(max|var| / 40000)), that's a twenty-line script touching 19 of the 75 BatchNorm layers:

k = float(2 ** np.ceil(np.log2(max_abs_var / 40000.0)))   # K = 16 for the worst layer
var_init.CopyFrom(numpy_helper.from_array((var / (k * k)).astype(np.float32), var_init.name))
scale_init.CopyFrom(numpy_helper.from_array((scale / k).astype(np.float32), scale_init.name))
# and the epsilon attribute: a.f = a.f / (k * k)

The result is worth stating precisely, because it kills the "mixed precision is inherently fuzzy" intuition: the converted model has 606 fp16 initializers and zero fp32 residue. It is 100% fp16, every tensor, with no casts added around the BatchNorms. And the arithmetic is exact — powers of two, so nothing is lost to rounding.

What this is worth — and what it is not

Three independent measurements on the same converted weights, plus a control and a harness sanity check:

Check	fp32	fp16 (fixed)
14-case masked-region PSNR	29.74 dB	29.73 dB
Per-case ghost-score drift (mean / max)	—	0.0014 / 0.005
Browser runtime (ORT Web WASM), same tile, vs fp32 reference	—	66.5 dB
Browser runtime (ORT Web WebGPU), same tile, vs WASM reference	—	71.6 dB, at 0.86 s vs 4.06 s per tile
Unfixed fp16, same test (control)	—	22.6 dB
16 real photo regions, production-detected boxes	—	59.9 dB mean, 40.3 dB min

The browser row is the one that matters for shipping: that's the exact runtime our production HQ tier uses, and 66.5 dB means the two models disagree at the level of sub-grey-step rounding. The control row is what the old mechanism should have looked like if it were real precision damage — 22.6 dB, i.e. visibly different. (Sanity check on the harness: browser fp32 vs a CPU fp32 reference on the same tile is 156 dB, so the comparison chain isn't the limiting factor.)

The size win is the point of the exercise:

	Raw	brotli q11 (what users download)
fp32	198.4 MiB	179.2 MiB
fp16 (fixed)	102.8 MiB	86.3 MiB (−52%)

The brotli pipeline that makes a 198 MiB weight file a 179 MiB download in the first place is part 9's story — worth saying here because the win compounds: same compressor, half the bytes.

And now the honest part, which is the real lesson for anyone shipping fp16 in a browser: on the WASM execution provider, fp16 made our inpainting slower, not faster. 11.4 s → 17.4 s per tile, 2.7 s → 9.4 s on CPU. The WASM EP has no native fp16 convolution kernels, so it inserts casts and computes in fp32 anyway, paying the copy tax for nothing. Part 3's 3.5× speedup came from WebGPU, which has real fp16 ALUs; on a WASM-only path, half-precision buys bytes and nothing else.

Update (2026-10-18): the WASM-only path was never a permanent constraint, and the reason we thought it was had nothing to do with quantization. The "JSEP gap" keeping HQ off WebGPU was a one-element off-by-one in the default onnxruntime-web build's negative-step Slice. Fixed by importing the WebGPU build, the same fp16 weights now run at 0.86 s per tile against WASM's 4.06 s (4.7×), at 71.6 dB agreement — so on desktop fp16 collects the speed win too, and on WASM it stays a pure download-size trade. See part 14.

The int8 half, where the conclusion survives and the reason does not

The old post also reported int8 failing (24.5 → 15.6 dB on our dark-text case) and concluded that quantization is architecture-hostile in general. We re-ran both int8 recipes on the current five-case subset (our case definitions have drifted since 2026-10, so compare within a run, not across posts):

Variant	Region PSNR	vs fp32
fp32	28.88	—
int8 dynamic (weights only, per-channel)	17.82	−11.1 dB
int8 QDQ (activations too), all 75 BatchNorms excluded	19.11	−9.8 dB

So int8 is still dead for this model — the conclusion stands. But the mechanism is the opposite of the one we published: this isn't a weight-precision problem, and it isn't the BatchNorm overflow either. Explicitly excluding every BatchNorm from quantization doesn't rescue it, because the inputs to those 75 layers still arrive through DequantizeLinear. Counting the graph: 432 Einsum/MatMul nodes with 648 inputs wrapped in quantize/dequantize pairs. The FFT path is expanded into spectral matrix products during export, and that's the part that cannot survive 8-bit activations. Different failure, different size of blast radius, and it stays on the do-not-retry list.

The corrected decision order

Our old order was architecture → bit width → runtime. The new one starts one step earlier, because a conversion artifact can impersonate an architectural verdict:

Audit the values before you audit the architecture. Dump max |value| for every initializer, compare against the target dtype's ceiling, and check the subnormal floor while you're there. In our model 16 of 606 tensors overflowed fp16 (0.087% of parameters sat below the subnormal floor — harmless, but the same scan finds both). This costs minutes and can invalidate a conclusion you'd otherwise keep for months.
Separate precision from range. Round the mantissa, keep the range. If quality doesn't move, precision is not your variable and no amount of "the model is sensitive" theorising will help.
Then test the structure. Gating, long accumulation chains, adversarial spectral paths — all real failure modes, but they're hypotheses to test, not a diagnosis.
Verify in the runtime you ship. A Python CPU pass proves the weights; it does not prove that ORT Web's WASM EP, with its cast insertion and kernel gaps, produces the same thing.
Benchmarks decide admission. Unchanged, and it's the rule that caught this: the broken model looked plausible at 15.8 dB, the fixed one measures 66.5 dB against the fp32 reference, and only the benchmark could tell you which was which.

Two corrections to our own record, then: the LaMa fp16 entry comes off the do-not-retry list and onto the opt-in path (fp16 verified against fp32 on every check above, fp32 spec retained as the fallback), the int8 entry stays, and the sentence in parts 1 and 5 that makes quantization safety a property of architecture gets retired in favour of something less quotable and more true: a quantization failure is a measurement, and most measurements have a checkable cause.

Try it

Everything here runs client-side with no uploads. The fp32 model that this post improves powers the free watermark remover, and the inpainting engine is the same one behind the video watermark remover. The fp16 path that started all of this is live in our image upscaler, and the quality dashboard publishes the bench numbers these gates are built on.

Part 13 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor

Umasou — Sat, 17 Oct 2026 01:10:37 +0000

The claim we published

whether quantization is safe is a property of the model's architecture, not of your tooling

Step one: prove that precision is not even the variable

Weights rounded to	Region PSNR (5 cases)	Output fidelity vs fp32
fp32 (baseline)	28.88	—
7-bit mantissa (bf16)	28.83	66.9–76.9 dB
10-bit mantissa (= fp16)	28.88	74.5–96.6 dB
13-bit mantissa	28.88	95.9–109.0 dB

That left range, and range has a hard, checkable boundary: fp16's largest finite value is 65,504. So we dumped every fp32 initializer in the graph and sorted by magnitude.

Step two: find the numbers that don't fit

     484322.56  model.generator.model.5.conv1.bn_l.running_var
     440853.66  model.generator.model.25.running_var
     240497.69  model.generator.model.22.conv1.bn_l.running_var
     218544.31  model.generator.model.11.conv1.bn_l.running_var
     174809.92  model.generator.model.6.conv1.bn_l.running_var
      65564.05  model.generator.model.10.conv1.bn_l.running_var
      ...  16 of 606 tensors, every one of them a running_var

This is a five-line check, and it is the first thing to run before believing any "this model can't be quantized" conclusion:

import numpy as np, onnx
from onnx import numpy_helper

m = onnx.load("lama_fp32.onnx", load_external_data=False)
worst = sorted(
    ((float(np.abs(numpy_helper.to_array(t)).max()), t.name)
     for t in m.graph.initializer if t.data_type == 1),   # 1 = float32
    reverse=True,
)[:16]
for value, name in worst:
    print(f"{value:>14.2f}  {name}")

Sixteen tensors. Not sixteen layers of masking logic — sixteen numbers.

Why a BatchNorm variance is the thing that overflows

That last part is the trap. There is no NaN, no warning, no crash: just a model that quietly keeps the watermark.

The clamp that never said a word

You can watch the corruption happen tensor by tensor, which is how we finally believed it:

Stage	`model.5.conv1.bn_l.running_var`
fp32 source	484322.5625
after `keep_io_types` fp16 conversion	65504.0 (clamped to the fp16 ceiling)
after a further fp16 storage pass	inf
after our fix (see below)	1892.0 (fits comfortably in fp16)

The fix is algebra

The better fix is to notice that a BatchNorm's variance and its scale are only meaningful relative to each other:

y = (x − μ) / sqrt(var + ε) · γ + β
  ≡ (x − μ) / sqrt(var/K² + ε/K²) · (γ/K) + β

k = float(2 ** np.ceil(np.log2(max_abs_var / 40000.0)))   # K = 16 for the worst layer
var_init.CopyFrom(numpy_helper.from_array((var / (k * k)).astype(np.float32), var_init.name))
scale_init.CopyFrom(numpy_helper.from_array((scale / k).astype(np.float32), scale_init.name))
# and the epsilon attribute: a.f = a.f / (k * k)

What this is worth — and what it is not

Three independent measurements on the same converted weights, plus a control and a harness sanity check:

Check	fp32	fp16 (fixed)
14-case masked-region PSNR	29.74 dB	29.73 dB
Per-case ghost-score drift (mean / max)	—	0.0014 / 0.005
Browser runtime (ORT Web WASM), same tile, vs fp32 reference	—	66.5 dB
Unfixed fp16, same test (control)	—	22.6 dB
16 real photo regions, production-detected boxes	—	59.9 dB mean, 40.3 dB min

The size win is the point of the exercise:

	Raw	brotli q11 (what users download)
fp32	198.4 MiB	179.2 MiB
fp16 (fixed)	102.8 MiB	86.3 MiB (−52%)

The brotli pipeline that makes a 198 MiB weight file a 179 MiB download in the first place is part 9's story — worth saying here because the win compounds: same compressor, half the bytes.

And now the honest part, which is the real lesson for anyone shipping fp16 in a browser: fp16 made our inpainting slower, not faster. 11.4 s → 17.4 s per tile on the WASM execution provider, 2.7 s → 9.4 s on CPU. The WASM EP has no native fp16 convolution kernels, so it inserts casts and computes in fp32 anyway, paying the copy tax for nothing. Part 3's 3.5× speedup came from WebGPU, which has real fp16 ALUs; on the WebGPU-disabled HQ path, half-precision buys bytes and nothing else. If a future ORT Web release ships fp16 kernels on WASM — or the JSEP gap that keeps HQ off WebGPU gets closed upstream — this model collects the speed win too. The fix is the precondition for that, not the payoff.

The int8 half, where the conclusion survives and the reason does not

Variant	Region PSNR	vs fp32
fp32	28.88	—
int8 dynamic (weights only, per-channel)	17.82	−11.1 dB
int8 QDQ (activations too), all 75 BatchNorms excluded	19.11	−9.8 dB

The corrected decision order

Our old order was architecture → bit width → runtime. The new one starts one step earlier, because a conversion artifact can impersonate an architectural verdict:

Audit the values before you audit the architecture. Dump max |value| for every initializer, compare against the target dtype's ceiling, and check the subnormal floor while you're there. In our model 16 of 606 tensors overflowed fp16 (0.087% of parameters sat below the subnormal floor — harmless, but the same scan finds both). This costs minutes and can invalidate a conclusion you'd otherwise keep for months.
Separate precision from range. Round the mantissa, keep the range. If quality doesn't move, precision is not your variable and no amount of "the model is sensitive" theorising will help.
Then test the structure. Gating, long accumulation chains, adversarial spectral paths — all real failure modes, but they're hypotheses to test, not a diagnosis.
Verify in the runtime you ship. A Python CPU pass proves the weights; it does not prove that ORT Web's WASM EP, with its cast insertion and kernel gaps, produces the same thing.
Benchmarks decide admission. Unchanged, and it's the rule that caught this: the broken model looked plausible at 15.8 dB, the fixed one measures 66.5 dB against the fp32 reference, and only the benchmark could tell you which was which.

Try it

Part 13 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Upscale Video to 4K Without Uploading It: Up to 7 Faster, Sharper, and Nothing Leaves Your Laptop

Umasou — Fri, 16 Oct 2026 17:14:01 +0000

Two things were true about our video upscaler when this round started: it was honest about being slow, and we had no idea whether it was any good. The first half is fixable on paper. The second half is why we spent a week building a ruler before touching a single line of the optimizer. What came out the other side: static shots now upscale 4.7–7× faster, talking-head footage — a static background with a moving subject, the case that used to get zero benefit — goes from a flat 1.00× to about 1.8×, and the output is measurably sharper than what we shipped last month. All of it inside a browser tab, with your video never leaving your machine. Not "we delete the file after processing": there is no upload path in the code.

We build ClearPix, the in-browser media toolbox from parts 1–10 of this series — no uploads, no server GPUs, Real-ESRGAN-class super-resolution running on WebGPU inside your browser. The video pipeline itself (hardware decode, per-frame AI, hardware encode, audio passthrough, O(1) memory) was part 2's story. This round was about turning that pipeline up to 4K and answering the two questions that actually matter for a video tool: how fast, and how good?

Act 1: Build the ruler before you touch the dial

We have a house rule by now, and we earned it the expensive way: no optimization without a per-stage timing breakdown, and no quality change without a metric that can veto it.

The video path had the first half. bench-video-upscale.mjs reports per-frame milliseconds across every output tier, with the decode/inference/encode/paste split, on real clips. What it did not have was a quality number. Its outputPsnr measures the deviation between "frame reuse on" and "frame reuse off" — a regression guard for a specific optimization, not a statement about how the picture compares to the truth. So "is our video upscaling any good?" was, scientifically speaking, a vibe.

The fix was a ground-truth bench (bench-video-quality.mjs). We take our natural test photos at 800×500 as ground truth, synthesize a low-resolution source by downsampling each one by a factor (f = 2, 4, 8), run the exact production geometry on it, bring the result back to ground-truth size, and score PSNR, SSIM, and high-frequency variance against the original. Same seed, same model, same tiles — only the geometry changes between rows.

One reading caveat, stated up front, because it nearly fooled us: Real-ESRGAN is a perceptual GAN. PSNR actively under-rates it against a dumb bicubic resize (at f = 4 on a clean source: 23.57 dB versus bicubic's 25.13 dB), so these numbers must never be used to argue "AI versus bicubic." They are only valid for model-against-itself comparisons — and there, they are exactly what we needed. Two independent metrics (PSNR and SSIM) agreeing in direction is our bar for believing a result.

The ruler immediately found something embarrassing. Our geometry was resizing the source down to a quarter of the target — 960px wide for a 4K output — and handing that to the model. Any real detail living between that quarter-target resolution and the source's actual resolution was being thrown away before the AI ever saw it. We had been carefully building a pipeline that discarded the user's own pixels.

Act 2: The dial nobody had turned

Feeding the model a bigger input means it does its ×4 on a larger frame, and we then scale the result back down to the target. That sounds like extra work for nothing — the model was already being asked to upscale. It is not nothing: at the larger input, the source's genuine detail is still present, the ×4 invents detail around it, and the average of that dense 4× field down to target size quietly smooths out the model's own artifacts.

We swept the input scale (m = 1.0 is what we shipped before) across all four images and all three downsampling factors — twelve rows, averaged:

AI input scale	Δ PSNR	Δ SSIM	Cost (per frame)
m = 0.5	−1.88 dB	−0.0452	0.26×
m = 1.0 (what we had)	baseline	baseline	1.00×
m = 1.5 (shipped)	+0.93 dB	+0.0118	2.01×
m = 2	+1.64 dB	+0.0186	3.58×
m = 3	+1.95 dB	+0.0159	8.18×
native source width	+1.12 dB	+0.0168	1.48×

Two rows killed their own ideas, and one needs a warning label. Start with the last row: "just feed the model the source at its own width" looks like a better trade than m = 1.5 — more gain for less cost — but it isn't a setting, it's a relationship. The input scale that equals your source's own width changes with every source-to-target ratio (2 at f = 2, 1 at f = 4, 0.5 at f = 8), and at the small end it degenerates into the m = 0.5 we are about to bury. We tested the intuition behind it directly — "stop pre-upsampling past the source" — and at f = 8 the geometry we ship still beat it, by 0.6–0.8 dB on three of the four test images (and gave back 0.3 dB on the fourth). Pre-upsampling is not a bug we forgot to remove.

The top row buries a piece of folklore we had been carrying around for months: "feed the model less resolution and let it work harder" costs 1.88 dB on average. That direction is dead, and we wrote it into the runbook so nobody re-proposes it in six months.

m = 1.5 was the row that shipped, for a boring reason: it was non-negative on every single one of the twelve rows — +1.88 dB at f = 2, +0.77 dB at f = 4, +0.14 dB at f = 8. An optimization that can't go backwards is an optimization you can hand to users as a choice. Note the cost column: it follows the area of the AI input (1.5² = 2.25 nominal, 2² = 4, 3² = 9) rather than the tile count, and every number in it is measured, not assumed.

So the option became a three-position dial. Fast is the old geometry. Balanced is m = 1.5. Best is m = 2, where the curve starts bending — another +0.7 dB for nearly double the time. We also stopped asking users to guess what their machine can afford. The default position comes from device capability plus the clip in front of you. No WebGPU — where the fallback WASM path costs roughly 25× per frame — or a low-end device, and you land on Fast automatically. Otherwise we estimate each tier's finish time from the file you actually dropped: its duration and frame budget, with the real fps probed when the estimate is close enough to matter. The most expensive tier that still fits its time budget wins. A six-second 720p clip on a laptop with a real GPU defaults to Best. A four-minute 4K export defaults to Fast, without anyone having to be told why, and the number next to the buttons updates the moment you change the target. It is not a marketing figure; it is arithmetic, and it happens to be right.

Act 3: Teaching the upscaler to skip the boring parts

Here is the uncomfortable shape of the speed problem: we profiled everything, and inference is 97% of per-frame time. Decoding is nearly free (the OS media engine does it on silicon). Encoding is nearly free. Pasting tiles to a canvas is noise. There is exactly one lever, and it is not "make inference faster" — it is "don't run inference". Video is full of frames that did not change.

We already had frame-level reuse for this: compare each frame against the last one we actually ran through the model, and if it hasn't moved, reuse the previous output. The implementation detail that cost us real time is worth repeating because it is counterintuitive: the comparison cannot be byte equality. Decoded H.264 static frames are not bit-identical — we measured a max delta around 9 and a mean of 0.02 between consecutive "identical" frames, because the encoder makes slightly different decisions per frame. Byte comparison scores 0% hits on video that is visually frozen. The fix is an anchored, bounded comparison: each frame is compared to the last frame that really went through inference (not to its predecessor, so deviation can't chain), and a hit reuses that anchor's output without moving the anchor. Bounded tolerance, no drift. This is the same discipline we landed on for video watermark removal in part 7.

The numbers on real clips at τ = 1.0:

Content	Hit rate	Speedup	Deviation
Static 1080p clip	88%	4.72–7.02×	46.9–50.3 dB
Slow pan (1px/frame)	50%	~1.97×	64.2 dB
Camera moving (`clean.mp4`)	0%	1.00×	none

And then the obvious follow-up question, which took us embarrassingly long to ask: what does real footage actually look like? Talking heads. Screen recordings with a cursor. Product close-ups. All of them are subject-motion clips — a static background with something moving in the middle — and every one of them gets a 0% hit rate at frame level. The most common shape of video on the internet was the shape our optimization couldn't touch. One moving person flips the entire frame back to full inference, forever.

So we pushed the same idea down a level. The frame is already processed in 256px tiles internally, so each tile now keeps its own anchor and gets its own bounded comparison. A tile that hasn't changed copies its core region straight from the previous output canvas — no format conversion, no inference, no paste — while the tiles that did change go through the model as usual. Anchors still don't update on a hit, so the drift bound is the same as the frame-level version.

Measured in isolation on synthetic clips (24 frames, 1080p tier = 4 tiles), the results split cleanly:

Content	Frame-level	Tile-level
Fully static	19.4×	9.4×
Static background + moving subject	1.00×	1.79× (63/96 tiles)
Whole-frame pan	1.00×	0.99×

The last row is the honesty check: when nothing is reusable, tile-level costs about 1% (one comparison per tile) and correctly returns nothing. The middle row is the whole point — the 1.79× comes with a deviation of 49.77 dB against no-reuse output, which is a fraction of one 8-bit level. Invisible. And since frame-level is strictly better when the whole frame is static, we ship them stacked: frame first, tiles as the fallback layer. Users never choose; the picture decides.

One trap almost shipped, and it's a good argument for building the bench before the feature. Pasting a tile back has to use the core region — the 8-pixel overlap around each tile exists only to give the model context, and must not be written to the canvas. putImageData's dirty rectangle is relative to the source ImageData, not to the destination, so the destination offset has to be the tile origin; pass the core origin instead and every tile lands 32 pixels off. Our PSNR gate caught the shift: 54 dB became 16.9 dB. Visually, "32px of double-exposed edge" reads as slightly soft — exactly the kind of thing that ships and gets discovered by a user a month later.

Why the privacy part is structural, not a policy

Everything above runs in your tab: hardware decode via WebCodecs, inference via ONNX Runtime Web on WebGPU, muxing via MediaBunny, hardware encode back to H.264. There is no server component of the video path at all. The only bytes that travel are the model weights coming down — once, then cached in IndexedDB, after which the tool works offline — plus a handful of cookieless analytics events: an event name and coarse properties like backend and elapsed seconds, the full list published on our privacy page. No media, no filenames, no durations tied to a person.

That is a stronger claim than "we don't store your videos," and it is checkable from the outside: open devtools, drop a clip in, watch the network tab. Models come down; nothing goes up. This is the part that gets described as a marketing position by people who haven't tried the alternative, and it genuinely costs us — a server GPU would make the long-clip problem evaporate, and a queue would let us serve 4K to a five-year-old phone. It would also mean your footage crossing the wire to be upscaled, and that is the one trade we don't make. If you're upscaling something you wouldn't email to a stranger — a family video, an unreleased product shot, a client's footage under NDA — the tool's whole job is running on the laptop you already trust.

Where the walls are

Per-frame cost on our reference machine (Apple GPU, WebGPU, moving content): 720p ≈ 115 ms, 1080p ≈ 263 ms, 2K ≈ 464 ms, 4K ≈ 1.06 s. Inference is 97% of each of those numbers, which is why this post is about doing less inference rather than doing inference faster.

We also removed the hard duration cap this round. There's now a size guard (200 MB) and a frame budget, an honest per-clip estimate shown up front, and if the estimate exceeds eight minutes we ask instead of refusing. Slow is a conversation now, not an error message.

Dead ends, recorded so we don't walk them again: multi-worker parallelism is a no-op (inference is memory-bandwidth-bound; more workers only fight over the bus), bigger tiles are a wash, IOBinding buys 1–2% so we skipped it, feeding the model less resolution costs 1.88 dB, and detail injection at k = 0.7 helps clean sources but re-amplifies compression noise on real ones, so it stays on the bench with no perceptual metric to justify it. One more for the file: if you benchmark two variants back to back, the first one silently absorbs the video encoder's cold start and looks 2.17× faster for free. Warm up first or you will ship the wrong winner.

Try it

No account, no upload, no watermark on the output: upscale a video to 4K in your browser — the tier dial and the estimate are under the target selector, and the frame budget will tell you the truth before you commit. The in-browser image upscaler shares the model and the tile engine, the free video watermark remover shares this exact pipeline, and if the transport layer is the part you're here for, part 2 is the read.

The nine seconds and the plastic skin were the same complaint. This round's version of that: "fast" and "sharp" and "private" were never three separate problems — they were one pipeline, observed from three angles.

Part 11 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Browser Image Upscaling: How We Got 4 Super-Resolution 14 Faster — Then Made It Slower on Purpose

Umasou — Thu, 15 Oct 2026 14:55:12 +0000

Two complaints arrived in the same week. The first was about time: upscaling a 2048px photo took nine seconds, even on the WebGPU path, and users on weaker hardware saw the progress bar crawl. The second was about taste: the results looked over-cooked — skin turned plastic, edges grew light halos, and if your source was a JPEG, the compression blocks came back louder than before. One complaint asked us to go faster; the other, if you squinted, asked us to add less. This is the story of how we chased the first one honestly, and why the fix for the second one ended up being a dial that gives some of that speed back.

We build ClearPix, the in-browser media toolbox from parts 1–9 of this series — no uploads, no server GPUs, Real-ESRGAN-class super-resolution running inside a browser tab. If you haven't followed along: our upscaler had already been through one generational swap (the 67MB RRDBNet flagship for a 4.9MB compact variant that ran 14–27× faster and won the quality shootout anyway) and one quantization gamble (fp16 weights, another 4.8× on top, told in part 3). What follows is the next chapter, where the easy wins were gone.

Act 1: Build the ruler before touching anything

We have a house rule by now, earned the expensive way: no optimization starts without a per-stage timing breakdown, and no quality change without a metric that can veto it. The upscale path had speed numbers but no quality numbers at all — "over-sharpened" was a vibe, not a measurement. So before touching a line of code we built the bench: 15 cases, five content types (people, structure, texture, sharp edges, flat gradients) crossed with three degradations (clean source, JPEG q70, JPEG q40), each scored for PSNR, SSIM, and a high-frequency ratio against the ground truth.

Only then did we learn what the second complaint actually was. Everyone, us included, had assumed "over-sharpened" meant edge halos — the classic ringing artifact. The bench said otherwise: on genuinely sharp content our output was actually too soft (hfRatio 0.29, meaning it had lost high frequency versus the original). The real villain was hiding in boring places: flat and soft regions, where the model hallucinates texture that was never there, and JPEG artifacts, which it amplifies faithfully. On flat gradients the hallucination metric climbed monotonically with source degradation — 3.47 on clean, 4.70 on JPEG q70, 5.55 on JPEG q40. Your compressed screenshot didn't just get upscaled; it got its blockiness upscaled too.

That number, 5.55, deserves a footnote because it nearly sent us chasing ghosts. High-frequency ratio divides by the original's high-frequency energy — and on a nearly flat original, that denominator is almost zero, so the ratio explodes regardless of absolute damage. We added an absolute metric (hfAbs) alongside the ratio and confirmed the flat-region hallucination is 0.12–0.16 intensity levels: real, but invisible. Half of "fixing" a problem is making sure you can see it accurately.

Act 2: The speed chase — three honest failures and one three-line win

With the bench as our safety net, we profiled. The per-stage breakdown held a surprise: model inference was 62% of wall time, sure — but pasting results back into the output canvas was 36%, and format conversion 1%. A third of our runtime was a CPU loop copying pixels, not AI at all.

That loop did per-channel Math.max(0, Math.min(255, value)) clamping while writing into a Uint8ClampedArray — which clamps on assignment by spec. Deleting the redundant math took three lines and bought 88ms → 68.4ms per image (−22%), with output that is bit-for-bit identical. It remains my favorite kind of optimization: zero risk, instant payback, embarrassing only in hindsight.

The natural next step was pipelining — hide the paste behind the next tile's inference, the classic overlap. We built it, verified bit-identical output, measured… nothing. Wall time didn't move. The reason was quietly funny: the pipeline was designed to hide a 36% bottleneck that the three-line fix had just reduced to 4%. We reverted the whole thing the same day and wrote the lesson into our runbook: re-profile after every fix; never parallelize against a profile that's already out of date.

Bigger tiles, then? 384px instead of 256 would mean fewer of them. On our Apple-silicon GPU it was a wash — 941/849ms against 888/825ms — the GPU was already saturated at 256. The nuclear option was skipping half the work: feed the model a half-resolution input and let ×2 cover the rest. It was glorious — 205ms down to 55ms per tile, a 3.8× speedup — and the bench killed it just as fast: photos lost 2.6–2.9dB, sharp content lost 6.5dB. The four-times cost is what quality-first upscaling is. We recorded the falsification prominently so nobody (especially future us) would re-propose it in six months.

A final systems check closed the case: every operator in the model runs natively on ORT's WebGPU backend with zero fallback warnings, and 115ms for a 256² tile is simply what this architecture costs on this hardware. Scheduling layer: squeezed dry. Model candidates: falsified. Runtime: current. The remaining 97% of inference time belongs to upstream — ORT's WebGPU evolution, or a hand-written native WebGPU super-res kernel, which is a project unto itself. We stopped, which is its own discipline.

Act 3: The honest dial

So we were as fast as honesty allowed, and still slightly over-cooked. The model's hunger for high frequency is baked into its training — SRVGGNet is a GAN-family network, and "adds crisp detail" is literally its job description. We couldn't retrain it. What we could do is scale what comes out.

The fix is almost comically small: a separable 2-pixel box filter applied to the model's output at final resolution, with a single gain factor k. k = 1 is the raw model. We swept k across the bench and the results settled a small argument in the team: k = 0.8 — giving back 20% of the output's high-frequency energy — improved PSNR on photos (+0.33dB on people, +0.44dB on texture, +0.27dB on edges) with SSIM flat-to-up. Read that again: the super-resolution model was adding high-frequency detail to photographs that the ground truth didn't have, and removing some of it made the output measurably more truthful. "Over-sharpened" wasn't user pickiness; it was a real, quantified bias. The default became k = 0.8, shipped with a 60–100% slider because taste is personal and JPEG-free line art genuinely likes 1.0.

Two adjacent ideas got proper burials. Rebuilding the output with matched film grain (to mask the flat-region hallucination) turned out to be unnecessary — the hallucination is k-immune but, per the absolute metric, invisible anyway. And Real-ESRGAN's official "denoise" model variants, which we hoped would be a gentler built-in preset, turned out to be detail-reconstruction strength knobs — the "denoise" setting adds more high frequency, not less. Both are archived in the bench directory with their falsification data, which is where good rejected ideas live.

One deliberate gap: the video upscaler shares this kernel but doesn't get the slider yet, because the sharpening helper costs 961ms at 4K — more than the inference itself. It needs a GPU-side implementation before it's worth anyone's frames.

Where the walls are

Today the pipeline is: clamp input, split into 256px tiles with 8px overlap, run the 2.4MB fp16 compact model on WebGPU, paste back only the centers so tile seams never show, then one honest pass of the sharpening dial. A 2048px photo costs about a second of GPU time on a modern laptop, and the output is — by our own bench, which exists precisely so we can say this — slightly closer to the truth than the raw model wanted to be.

The nine seconds became one, and the plastic skin became a preference slider. Both complaints, it turned out, were the same complaint: the model was working harder than the truth required. Speed and honesty weren't a trade-off — they were the same fix, applied at different layers.

If you want to see it yourself: clearpix.org/upscale-image runs the whole pipeline in your tab — the sharpen slider is under the result. The tile-and-overlap engine and memory walls behind the other half of this story are in part 5, the fp16 quantization adventure in part 3, and the delivery layer that gets the 2.4MB model into your browser in part 9.

Part 10 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Brotli for WASM and ONNX Models: Cutting First-Run Downloads From 72MB to 13MB, and the Compatibility Call We Made on Purpose

Umasou — Wed, 14 Oct 2026 15:42:48 +0000

We build ClearPix, a set of media cleanup tools that run entirely in the browser. That means every engine we ship — the ONNX Runtime WebGPU backend (a 27MB jsep.wasm plus a 14MB core), the ffmpeg core for video (30.7MB), and the model zoo behind our inpainting, upscale, and detection tools (up to a 198MB LaMa) — is downloaded into a stranger's browser tab before anything useful happens. The download path is the product experience. This post is about the delivery layer we ended up with: pre-compressed brotli objects on Cloudflare R2, why we compress WASM hard and ONNX barely, the progress-bar bug that fell out of it, and the compatibility decision we made consciously instead of by default.

What actually constrains model delivery

Three hard facts shaped everything:

Cloudflare Pages has a 25MB per-file limit. Our biggest files (27–198MB) can't even live in the deploy bundle. The build strips them, so a "mirror chain" isn't an optimization — it's the only way the product works at all.
Public mirrors are a fallback, not a source. We learned this in production on 2026-09-26: huggingface.co was unreachable from some networks, and hf-mirror's CORS headers were inconsistent (sometimes Access-Control-Allow-Origin, sometimes nothing), so model downloads simply failed for real users. Everything large now lives in an R2 bucket we control — 10GB free, zero egress fees, one controllable origin.
The fallback path gets real traffic. For the ORT runtime binaries we probe jsDelivr first (measured roughly 7× the throughput of our R2 custom domain) and fall back to R2 when the CDN is blocked — which happens in entire network regions. So the R2 path is not a theoretical safety net; for a slice of users it is the path. We measured our own access too: the default pub-*.r2.dev URL crawls at 75–110 KB/s, while a same-zone custom domain (static.clearpix.org) sustains ~2.3 MB/s — a 20–30× difference that has nothing to do with compression.

Brotli: compress at upload, not at request time

R2 doesn't transcode on the fly. You store bytes; it serves bytes. So we compress at upload time in the setup script, using nothing but Node's built-in zlib:

const br = brotliCompressSync(readFileSync(p), {
  params: { [BROTLI_PARAM_QUALITY]: 11 },
});
// wrangler r2 object put ... --content-encoding br --content-type application/wasm

The object key doesn't change, Content-Type stays application/wasm, and the browser transparently decodes Content-Encoding: br before the bytes ever reach ONNX Runtime or our model loader. Zero runtime changes — the URLs are identical, SHA-256 pins validate the decoded bytes, and IndexedDB caches the decoded bytes.

Measured ratios at quality 11, on our real files:

File	Raw	Brotli	Ratio
`ort-wasm-simd-threaded.jsep.wasm`	27.0 MB	3.8 MB	14%
`ort-wasm-simd-threaded.wasm`	14.2 MB	2.2 MB	16%
`ffmpeg-core.wasm`	30.7 MB	6.9 MB	22%
`migan.onnx` (inpainting)	28.2 MB	23.1 MB	82%
`yolo11_watermark_det.onnx`	10.6 MB	8.8 MB	84%
`lama_fp32.onnx`	198.4 MB	179.2 MB	90%
`real_esrgan_x4.onnx`	63.9 MB	58.6 MB	92%
`ppocr_v5_mobile_det.onnx`	4.5 MB	4.2 MB	93%

The pattern is physics, not tuning: WASM is code — structured, repetitive, full of zeros — and crushes to 14–22%. ONNX models are mostly float32 weights, which are close to entropy-maximal noise; brotli shaves 8–18% and that's the ceiling. (Worth knowing before you promise "50% smaller models" somewhere.) The full archive, all ten files, drops from 384MB to 293MB — but the number users feel is the first-run set: ORT WASM + ffmpeg went from 71.9MB to 13.0MB, an 82% cut on the critical path. Upload-side cost is a one-time ~10 minutes of q11 compression; quality 11 matters for a file served a million times, not for one.

The bug this introduced: progress bars that lie

Compression broke one thing, and it's worth calling out because it's subtle. When a response carries Content-Encoding: br, the Content-Length header is the compressed size, but a streaming reader in JavaScript receives decoded bytes. Our model loader's progress bar counted decoded bytes against the compressed total — so a 198MB model served as 179MB of brotli would blow straight past "100%" and keep climbing.

The fix is three lines: if Content-Encoding is present, ignore Content-Length and use the spec's known original size (expectedSize) as the denominator. Decoded bytes over original size — progress is honest again. Integrity checking was never affected: we hash the decoded bytes against pinned SHA-256s, so a compressed mirror and an uncompressed mirror are interchangeable.

The decision: br-only, no negotiation, on purpose

Here's the call we get asked about. An object uploaded with content-encoding: br is served that way to every request — R2 stores the encoded bytes and doesn't do content negotiation. A client that can't decode brotli gets undecodable bytes. We could have stored two variants (identity + .br) and put a ~50-line Worker in front of the custom domain to switch on Accept-Encoding, with Vary headers and doubled storage.

We didn't. The reasoning, in one table:

	Brotli support	Our runtime baseline
Chrome	50 (2016)	WebGPU since 113 (2023)
Firefox	44 (2016)	WebGPU since 141 (2025)
Safari	11 (2017)	WebGPU since 26 (2025)
Edge	15 (2017)	113 (2023)

A browser that can't decode brotli is from 2016 or earlier. A browser that can run ONNX Runtime Web with WASM SIMD and WebGPU is from 2023 or later. The intersection of "can't decode br" with "can boot our app" is empty — the fallback we'd be building serves users who are stopped two checkpoints earlier anyway. Building dual-variant + negotiation to protect a theoretical client is solving a problem we don't have, at the cost of a Worker in the request path and an ops surface we have to maintain forever. If that ever changes — say, a genuinely non-br embedded client appears — the dual-variant design is a clean add-on, and we wrote down exactly how we'd do it.

The honest caveat, for your own tooling rather than users: anything that fetches from R2 with plain curl needs brotli support (curl --compressed), or it saves compressed bytes to disk. Fine for scripts, worth documenting.

One thing we still don't love: ORT WASM has no persistent cache

Our ONNX models and ffmpeg core have a two-level cache: immutable HTTP headers plus IndexedDB with self-healing SHA-256 verification — download once, keep forever, survive releases. The ORT runtime WASM is different: ONNX Runtime fetches it internally, so we can't feed it from IndexedDB, and its only cache layer is the browser's HTTP cache. Mobile browsers evict that under storage pressure, and our release cadence happens to match the eviction cadence — which is why "every release makes users re-download the WASM" is an illusion we'd like to kill properly with a Service Worker and Cache Storage. That's the next post in this layer of the stack.

Try it

Everything described here runs in production: the dual-engine MI-GAN/LaMa watermark remover at clearpix.org/remove-watermark (the 198MB LaMa is the opt-in HQ engine — never prefetched), the Real-ESRGAN upscaler at clearpix.org/upscale-image, and the WebCodecs/ffmpeg pipelines that motivated the ffmpeg core over at clearpix.org/remove-video-watermark. Open devtools on any of them: the only bytes on the wire are the engines and models themselves.

Part 9 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Watermark Removal Artifacts: The "Processed Look" Came From Our Own Blend, Not the AI Model

Umasou — Mon, 12 Oct 2026 23:23:27 +0000

This is part 8 of the ClearPix engineering series, and it is the story of a bug that lived in the last 40 lines of the pipeline: the step that pastes the inpainted result back into the image. Users kept telling us some photos "look processed" — a soft, smeared halo around where the watermark used to be. We spent a while suspecting the model's fill quality. The model was mostly innocent. Our blending band was taking the model's full-crop reconstruction — which is not the original image outside the hole — and smearing it into pixels that had been perfectly fine. And the metric we used to accept changes was structurally incapable of seeing it, because it averaged the hole and its surroundings into one number.

If you take one thing home, take this: before optimizing a generative pipeline, split your quality metric by region — inside the mask, outside the mask — and treat your feathering band as a first-class suspect. The fixes that fell out of that one measurement took our restored-region PSNR up, our video end-to-end score up 5dB, and dropped the damage to clean pixels to exactly zero — while a plausible-sounding optimization we tried along the way got fully refuted and reverted in a single bench run.

Where watermark removal artifacts actually come from

Our image inpainting pipeline (part 5 covers the model stack) ends with a paste step. The algebra of the old implementation looks harmless:

out = original + a · (inpaintedCrop − original)      a = max(binary mask, blur(mask, 6–22px))

The trap is inside inpaintedCrop. MI-GAN (and LaMa) reconstruct the entire crop, not just the hole. Outside the mask, the model's output is a reconstruction of the original — close, but never identical (codec-level drift, color shift, softened texture). So wherever a > 0, we are mixing model error into pixels that were clean to begin with. And because a includes a Gaussian-blurred feather band that extends 6–22px beyond the mask, every watermark we removed came with a rim of synthetic blur.

Here is the part that made this bug invisible: for the thin-stroke text watermarks that users complain about most, that rim is enormous relative to the watermark itself. We measured area(dilate(mask, 12px)) / area(mask) across our 13-case golden set:

Case family	Ring band area vs. mask area
Small/faint text watermarks	2.8 – 6.5×
Large text	1.8 – 2.9×
Dark badges	1.2 – 2.0×
Solid logos	0.4 – 0.8×

For a small text watermark, the majority of "the area we touched" was never watermark at all — it was the transition band. That is the processed look, quantified.

It also retroactively explained a contradiction in our own tuning log: dilating masks by 2–4px had measured −0.5dB on our golden set, which we had recorded as "expansion is bad." The real mechanism was the opposite of what we concluded — dilation itself is fine, but every pixel you add to the mask becomes a pixel the blend can pollute. The same measurement that punished dilation would reward removing the pollution.

Metrics that cannot see direction

Our acceptance metric, psnrRegion, computed PSNR inside dilate(groundTruth, 12) — the hole and its ring averaged together. A pipeline that damages the ring and a pipeline that heals the hole can post the same number. We added four split metrics before changing any pipeline code (all pure observation — the 12 shared cases re-ran bit-identical):

PSNR-in / PSNR-out — restoration error inside the mask vs. in the outer ring, measured separately;
ringDelta — mean |result − input| in the outer ring; 0 means clean pixels were untouched;
colorShift — Lab ΔE between the filled hole and the surrounding ring;
SSIM (region) — because PSNR and human eyes disagree in both directions.

Plus a new golden-set case, halo-flat-text: thin-stroke text on a perfectly flat gradient with a tight ground-truth mask. On a flat background there is no texture to hide in — any out-of-band contamination shows up in the numbers with nothing to blame.

And running the "harmless" check first paid for itself immediately: the current snapshot scored 23.384dB against the archived 23.824 baseline. That looked like a regression — but tracing it landed on a commit from two days earlier where we had fixed real watermark bleed-through (the blend used to put model output at ~50% alpha at thin-stroke centers, leaking half the watermark back). The fix legitimately trades metric for correctness. Knowing that before touching anything is the difference between chasing ghosts and measuring progress.

The fix: stop blending model pixels into clean ones

The change (A4 in our plan) is small enough to state in one formula. The alpha stays max(soft mask, hard threshold) — inside the mask we keep 100% model output, which is non-negotiable (the earlier fix that killed bleed-through). Outside the mask, the artificial feather band is deleted; the only blending left is the mask's own upscale ramp, a pixel or two wide:

Same image, same detection masks, same model — only the paste step differs. In the before panel, look at the ledge line and the background around the removed text: a soft band of reconstructed non-information. In the after panel the background is bit-identical to the input wherever the mask wasn't.

The split metrics moved exactly the way the diagnosis said they should:

ringDelta: 5.63 average → 0.00, on every case. Not "less visible" — zero. Nothing outside the mask is touched anymore.
PSNR-out: 30.8 → 84.3 dB (most cases hit the 99dB ceiling, meaning bit-identical to the input).
The halo case: 37.0 → 49.9 dB region PSNR.
Overall restored-region PSNR on the 12 shared cases: 23.4 → 23.9 dB — modest, exactly as our P0 analysis predicted for cases where false-positive repainting inside the mask dominates.
Video end-to-end (which shares the paste code): 33.9 → 38.9 dB masked-region PSNR, a +5dB jump from deleting a blending band.

The optimization that lied to us: when reference statistics are wrong

With the composite layer fixed, the next target was inner color blocks — cases where the fill's average color doesn't match its surroundings (up to ΔE 14.4 on our set). The textbook move is statistical color matching: shift the model output toward the mean of the surrounding ring before pasting. It sounds airtight. It failed, and the failure is the most useful thing we learned this round.

We built a scatter plot of the two references — the color difference against the ring (what the alignment sees) versus the color difference against the clean ground truth (what actually matters):

The ring lies in both directions:

tiled-med scored ΔE 14.4 against its ring but only 0.3 against ground truth — the fill's color was essentially perfect; the watermark band simply sits across content that genuinely differs from its surroundings. Aligning it to the ring dragged a near-perfect fill away from the truth: inner PSNR fell 24.9 → 21.5.
logo-br was the opposite: a real color bias of 10.3 hidden behind a ring score of 1.3 — the ring-based alignment could never even see it.

At runtime you cannot distinguish "biased fill" from "content that legitimately differs from its surroundings." Our bench run of the alignment confirmed the split — some cases improved (text-bl's ΔE 11.9 → 2.6 for nearly free), the tiled cases degraded past our per-case gate. We reverted it the same day, per the rule we set in part 7: decide the stop line first, and treat a tripped gate as a bug report, not an opinion.

What the attempt left behind is permanent: a new metric, ΔE-true (fill vs. clean ground truth — computable offline where ground truth exists), which is now the acceptance basis for any future color-alignment work. Baseline: average 3.6, worst case 10.3. The residual color-block problem is real; it will need a mechanism that respects content boundaries (gradient-domain blending aligns boundary continuity rather than forcing hole statistics), not a stronger version of the same lie.

Snap the mask to the real edge

The last piece targets the upstream cause of ragged edges: glyph masks are born at OCR probability-map resolution — roughly 1/4 to 1/16 of the image — so after upscaling, the boundary position is inherently off by 4–16 pixels. We added a guided filter (He et al., 2010) inside the glyph-mask builder: the detection image, resampled to the probability grid, acts as the guide, and the mask's 0.5 level set snaps to the visible watermark edge instead of the flood fill's incidental boundary.

Zero plumbing changes — the product path and the bench path both consume the refined bitmap as-is, the op contract and coordinate contracts untouched, and the filter runs on the quarter-scale grid so detection time doesn't move. The perceptual metric moved most where the complaint lived:

SSIM: 0.739 → 0.753 average, with the largest gains exactly on the perceptually worst cases: text-br-large 0.708 → 0.770, text-huge 0.471 → 0.524, dark-text-br 0.577 → 0.615.

One honest caveat for anyone copying this: precision against our ground-truth masks dropped slightly (47.4 → 47.0), and that is not a quality signal — the ground truth is looser than the real edge, and a mask that hugs the real edge loses points against a loose GT while covering more of what actually matters. Trust the region PSNR and SSIM pair, and always eyeball the worst case.

The scorecard

Across the 13-case suite (12 shared + the new halo case), one week of composite-and-mask work, two shipped fixes and one refuted experiment:

Metric	Before	After
ringDelta (damage to clean pixels)	5.63 avg, >0 everywhere	0.00 everywhere
PSNR-out (outer ring)	30.8 dB	84.3 dB (bit-identical in practice)
Restored-region PSNR (12 shared)	23.4 dB	24.2 dB
SSIM (region)	0.710	0.753
Video e2e, masked region	33.9 dB	38.9 dB
Halo case (flat background)	37.0 dB	49.9 dB

No case regressed more than 1dB; all acceptance gates held.

Three rules we kept, one we learned

Localize damage before you localize the fix. A single mixed PSNR number hid a real bug for weeks and misexplained a tuning result for months. The in/out split cost one bench run to build and paid for itself on the first look.
The blending band is a first-class suspect. Generative pipelines end with a paste, and the paste is where "the model's version of your image" meets "your image." Any alpha > 0 outside the hole is a place the model can be wrong.
Validate your reference, not just your method. Statistical alignment toward a ring assumes the ring represents what the hole should look like. Sometimes it doesn't — and the failure is invisible to the very metric that motivated the method. Measure against ground truth when you have it.
Revert cheap, keep the instrument. The refuted alignment cost one bench run and left behind ΔE-true, which will judge every future attempt. That is a good trade.

All numbers in this post come from our public bench snapshots (run-2026-10-01*.json), the quality dashboard at clearpix.org/quality, and the commits they were measured on — the same gates that block our releases.

Part 8 of the ClearPix engineering series. Earlier parts: in-browser architecture (1), WebCodecs vs ffmpeg.wasm (2), fp16 quantization (3), video frame skipping + EMA (4), LaMa and Real-ESRGAN in the browser (5), automatic watermark detection (6), the anchor-based frame-skip redesign (7).

Video Inpainting Optimization: We Deleted the Safety Net and Got Faster, Better Output

Umasou — Mon, 12 Oct 2026 00:01:20 +0000

This is a decision log, not a mechanism tutorial — part 4 of this series covers how our frame-skipping gate and temporal EMA work internally. Here we cover the other half of the story: why we replaced a shipped, working safety mechanism with a simpler one, how we validated the change without betting the product on it, where the first run regressed hard enough to trip our preset abort criterion, and the two small fixes that turned the redesign into a 1.5dB quality win over the old baseline. If you take one thing home, take the discipline: decide your stop line before you run the experiment, and make sure your test set includes the worst case you have ever shipped a bug on.

How to speed up video inpainting that already works

Some context. Our in-browser video watermark remover inpaints frames one by one, but most frames near a static watermark barely change, so a motion gate lets us skip inference and reuse an earlier result. The first version of that gate (v1) compared each frame to its immediate neighbor and reused the previous output. That design had already failed once, badly: on footage with a slow global gradient, per-frame differences stay tiny forever, the gate skips forever, and the repaired region freezes into a visible rectangle. That incident cost us 22dB on our end-to-end masked-region PSNR (38.7 down to 16.7) and earned itself a dedicated regression asset: a synthetic clip whose background drifts continuously, built precisely so no future change could reintroduce chain-freezing unnoticed.

The fix we shipped was a cumulative drift budget plus a brightness-delta compensation: accumulate the error of skipped frames, force re-inference when the budget is spent, and nudge reused pixels by the scene's measured exposure shift. It worked. The frozen-rectangle bug stayed dead, quality held at 38.44dB against a 38.74 no-skip reference, and the safety net went to production.

So the starting position of this story is not a broken system. It is a working one with a layer of indirection in it.

The anchor idea: keyframe reuse in video processing

In a design review, someone asked the question that started everything: why does each frame compare itself to its neighbor at all?

Think about what the skip error actually is. When we skip a frame, we reuse the output of the last frame that really ran inference — call it the anchor. The error we are committing is exactly "how different is the current frame from the anchor." The drift budget only ever approximated that quantity by summing per-step differences and hoping the steps add up. Comparing against the anchor directly measures the thing itself: the skip error is bounded by the threshold, by definition, with no accumulator, no proxy, no edge cases about whether slow drift can sneak past.

The theory also predicted we would lose nothing on speed. Summing noisy per-step differences can only overcount relative to a direct comparison (oscillation and noise accumulate in the sum but cancel in the direct diff), so the skip rate should stay flat or improve. Fewer lines of state, a stronger guarantee, equal-or-better performance. Elegant ideas like this are exactly the ones that blow up in your face, so we wrote the rules down first.

PSNR regression testing with a preset stop line

Our rule for pipeline changes is that they pass the end-to-end quality gates before they ship, full stop. For this experiment we made the gate explicit before running anything: the redesign would be aborted — not patched, not negotiated — if masked-region PSNR on any e2e asset dropped below 36dB. That number was chosen in advance, while we were still neutral about the outcome. Deciding the abort criterion after seeing the results is how regressions get rationalized into production.

The gates themselves deserve a mention, because they are the second time this story's real hero shows up. The e2e suite includes that synthetic gradient-drift clip born from the 16.7dB chain-freeze incident — deliberately the worst case for any reuse-based optimization, with a background that never holds still but changes too slowly to trip naive motion detection — plus a four-segment subtitle asset where the masked content changes constantly. One asset attacks stale reuse, the other attacks insufficient reuse. Whatever we changed, one of them would notice.

First run: pass, pass, pass, stop

The anchored rewrite went in, and the first full run was a split decision:

Subtitle asset: all four segments improved by +3–4dB (37.3 / 32.3 / 28.7 / 30.1dB). The direct comparison was doing exactly what theory promised on fast-changing content.
Gradient-drift asset: masked-region PSNR fell from 38.30 to 34.69dB — below the 36dB stop line.

This is the moment the discipline is for. The aggregate picture looked fine — better, even — and it would have been easy to average the results, declare victory on subtitles, and tune later. We stopped instead, exactly as agreed, and treated 34.69 as a bug report the experiment had written for us. No rollback either: the stop line meant "stop and explain," not "revert on sight."

Error accumulation in video processing, round two

The root cause analysis turned up something genuinely new — a failure mode the chained design had been hiding from us.

The gradient clip's background does not drift uniformly. Its exposure shift varies with vertical position (the asset synthesizes it roughly as sin(y/60 + t·4) and cos(y/80 + t·3) components), while our delta compensation is a single per-channel mean over the region. A mean cannot cancel a gradient. Under the old chained design, reused pixels were always exactly one frame old, so the uncompensated residual stayed small and roughly constant. Under anchoring, a reused output can be two or three frames away from its anchor, and the residual grows with anchor age. The redesign had not created the compensation error; it had amplified it by extending reuse lifetimes — the same mechanism that made everything else better.

We also found a second, dumber bug in the same territory. The temporal EMA blends the current output against the previous frame's output, so its history term needs compensation against the previous frame. In the rewrite we had fed it the to-anchor delta instead — a systematic overshoot on every blended pixel, because the compensation was answering a different question than the blend was asking.

Two small fixes, no rollback

Both fixes were small and stayed inside the new design:

Threshold ceiling 8 → 5. The skip threshold is self-calibrating (median of the first five frames' differences, times three, clamped to [2, 5]/255). Lowering the ceiling forces re-anchoring more often, which directly bounds anchor age — attacking the residual-accumulation mechanism rather than its symptom.
Split the deltas. Compositing reuses the anchor, so it uses the to-anchor delta. The EMA blends against the previous frame, so it gets the to-neighbor delta. One line of reasoning each, no shared state pretending two timebases are one.

The calibration logic itself did not change at all — only the clamp ceiling and the delta plumbing.

Results: beating the baseline

The final anchored version, same gates:

Gradient-drift asset: 39.84dB masked-region PSNR — 1.5dB above the chained baseline, not merely recovered. With the delta semantics fixed, direct comparison and exact to-anchor compensation turned out to be strictly better than the approximation we had been defending.
Subtitles: the +3–4dB gains held, with 67% of frames skipping inference.
Speed: 58% of frames skipped on the watermark asset, flat versus the budget version as predicted, with the entire drift-budget code path deleted.

The safety net we removed was never missed. The one we kept — the preset stop line and the worst-case test asset — caught the one regression that mattered.

Takeaways

Write the abort criterion before the experiment. "Masked PSNR below 36 means stop" was decided while we were neutral. After seeing +3–4dB wins, that line is much harder to draw honestly.
Your test set should contain your worst historical bug, weaponized. The gradient-drift asset was built to catch chain-freezing two years of refactors ago; it has now caught a second, completely different failure. Worst-case assets appreciate in value.
Aggregates hide regressions. Subtitles up 3–4dB and gradient down 3.6dB averages to "improved." We read per-asset results first, means second, and the stop line was per-asset too.
Simpler mechanisms fail differently, not gently. Deleting the accumulator removed a whole class of proxy-error bugs and introduced none — but it amplified a compensation error the old design had been silently capping. Budget analysis time for the new failure modes of elegant ideas.
Fix forward when the root cause is small. Both fixes were a few lines inside the new design. Reverting would have thrown away a real quality win to avoid understanding a small bug.

Try it

The anchored skip gate ships in our free video watermark remover, and the same pipeline drives our subtitle removal tool — both run entirely in your browser, no uploads. If you have footage that breaks our gates, we genuinely want to hear about it; our best test assets were born that way.

Part 7 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Automatic Watermark Detection in the Browser: YOLO + OCR, and the Glyph-Level Masks That Made It Work

Umasou — Sun, 11 Oct 2026 11:06:19 +0000

ClearPix removes watermarks entirely client-side — no upload, no server GPU. This post is the detection half of that story: how we get a browser to find watermarks on its own using a YOLO watermark detector in ONNX plus a PP-OCR text detector, why we show low-confidence boxes instead of silently dropping them, and how a seed-and-grow glyph mask pushed detection precision from 41.4% to 43.2% without losing a single text case. Every number below comes from our 12-case internal bench and twelve rounds of logged tuning, including the rounds that failed.

Detection is the ceiling on removal

The uncomfortable truth we learned early: inpainting quality is bounded by mask quality. A state-of-the-art inpainting model given a sloppy mask produces a confident, high-resolution smear. We have bench cases where a heuristic box over-selected a large clean region and the full-image PSNR collapsed to 8.3 dB — not because the inpainter was bad, but because it faithfully repainted a huge area that never needed repainting. In another acceptance test, a false-positive box over a clean star-trail photo made the inpainter hallucinate a ghost logo onto empty sky. Precision failures are user-visible in the worst possible way: they add artifacts.

Recall failures are just as bad, and quieter: a missed corner of a watermark survives removal and reads as "the tool doesn't work."

So we treat detection as its own product surface, with its own budget of engineering rounds. The stack that survived is a dual-engine design with arbitration, plus a glyph-level masking stage for text.

A YOLO watermark detector in ONNX, running in your tab

The first engine is a purpose-trained YOLO11 watermark detection model, exported to ONNX (~11 MB) and executed with onnxruntime-web. Its job is logos and graphic watermarks — the semi-transparent corner badges and channel marks that have no text in them.

The browser-side plumbing is ordinary YOLO fare, which is exactly why ONNX is nice here: letterbox the longest edge to 640, pad to 640×640 with neutral gray, normalize to 0–1, run one inference, decode the [cx, cy, w, h, score] rows, threshold at 0.35 confidence, NMS at 0.5 IoU, then map the boxes back through the letterbox into original-image coordinates. Everything runs on an OffscreenCanvas so detection never blocks painting.

One runtime detail mattered more than any model detail. On onnxruntime-web 1.20 the YOLO session failed with "No usable ONNX backend" and the inpainter silently fell back to WASM. Upgrading to 1.30 with providers ordered [webgpu, wasm] turned WebGPU on for real: inpaint time dropped from 2100–3500 ms to roughly 400 ms per tile, and single- vs multi-thread timings converged because the GPU doesn't care about your CPU thread count. For low-end devices without WebGPU we keep a load-shedding path that caps the detector input's longest side (our maxSide option downsamples before detection and scales the boxes back up).

YOLO is an opt-in "deep" tier because of the 11 MB download. When it fires, it also arbitrates: heuristic boxes are only kept if they overlap a YOLO detection by more than 15%. That single rule took the bench from recall 0.886 / precision 0.414 to recall 0.927 / precision 0.480 — both directions at once, which almost never happens. The tiled-light case jumped from 0.54 to 0.89 recall.

Detecting text watermarks automatically with PP-OCR

The second engine answers "where is text?" rather than "what does it say?" We use the PP-OCRv5 mobile DB text-detection model (~4.7 MB ONNX). It's script-agnostic — it emits a probability map of text-likeness, which we binarize at 0.3, morphologically close, and connected-component into boxes. Text watermarks, captions, and burned-in subtitles are its territory, and per our per-source instrumentation the OCR detector is genuinely accurate on its own.

Why two engines instead of one? Because each covers the other's blind spot. YOLO has no opinion about a faint diagonal text tile; OCR has no opinion about a solid logo badge with no glyphs. Alongside them we keep two cheap pixel heuristics (a light-overlay detector for semi-transparent white marks, a dark-badge detector for solid dark logo blocks) as the high-recall, high-noise safety net.

The tuning rule that survived twelve rounds is wide in, strict out: keep candidate generation loose, enforce quality at the box level with validators (fill ratio, internal edge density, mean saturation, an 8%-of-image area cap). Round 3 proved the reverse is fatal — tightening generation-side thresholds knocked recall from 0.912 to 0.852, and tiled/faint watermarks die first every time you do it.

We also learned which direction arbitration is allowed to flow. Gating noisy heuristics by the high-precision YOLO source worked. The mirror image — suppressing heuristic boxes that overlapped OCR boxes by more than 50% — was a disaster: one OCR false positive took a real neighboring watermark down with it (text-br-small recall 1.0 → 0.595). High-confidence sources may veto noisy ones. Noisy or merely-different sources may never veto high-recall ones. We reverted that round within a day and logged it as a do-not-retry.

Glyph-level masks: seed and grow, don't repaint the rectangle

Here is the problem with using detection boxes directly as inpainting masks for text: a bounding box around small glyphs is mostly background. Even a perfect text box has precision of only 0.5–0.9 in pixel terms, because rectangle pixels outnumber glyph pixels. Repainting the whole rectangle means asking the inpainter to hallucinate texture over a large area that was fine — and on complex backgrounds (city texture behind a faint centered watermark), that hallucinated texture is exactly the "it left a visible patch" complaint we got from real users. Our measurements: leaving the watermark alone scored MAE 3.11 against the clean original; full-rectangle inpainting scored 24.79. Eight times worse than doing nothing.

The naive fix — threshold the OCR probability map and use that as the mask — also failed, and it's worth understanding why. The DB detection head is trained against a Vatti shrink kernel, so its high-probability region is deliberately smaller than the true stroke. A pure threshold mask (prob > 0.15, fixed dilation) covered only about 50% of the glyphs; recall and PSNR both collapsed.

What shipped instead is a seed-and-grow glyph mask, built per text box from the cached probability map:

Seeds: pixels with prob > 0.5 — small but trustworthy.
Candidates: pixels inside the box whose color distance from the box's surrounding ring-band median exceeds 30 (the ring band, the box expanded by 3 px, is our local estimate of "what the background here looks like").
Growth: flood-fill from the seeds through candidate pixels only. Background noise without a seed never enters the mask.
Finish: dilate by 2 px to catch anti-aliased stroke edges.

And one escape hatch: if the grown mask covers less than 8% of its box, the glyph path has failed (this happens with very faint watermarks, whose color distance never clears 30) and that box falls back to a plain rectangle. That fallback alone keeps our tiled-light case above the recall gate.

Result: precision 41.4% → 43.2%, with recall held at 100% across every text case. Two smaller changes rode along: text-line clusters (union aspect ratio over 3, long side over 384 px) now get refined in native-resolution tiles of at most 340 px instead of being squeezed into one 512² crop — squeezing was a visible source of soft blur over text lines — and the mask feather radius adapts to region size, 6–22 px at long-side/40, so large removals blend their seams more gradually.

Low-confidence boxes are shown, not silently dropped

Detection will never be perfect, so the last design decision is about who absorbs the residual error. Our answer: the user, but with good defaults and full visibility.

In the image tool, boxes from the model sources (YOLO, OCR) are painted pre-selected. Heuristic boxes — the noisy ones — are shown as unselected suggestions and only when no model source fired at all, so a stray heuristic guess can't be swept into the mask by one click and repainted onto clean background. In the video tool, scrubbing the timeline shows the sampled re-detection as blue dashed preview boxes, so you can verify what the detector will do at that moment before committing. Nothing is discarded silently; the cost of a false positive is a click, not an artifact baked into your export.

Subtitle detection in video is a moving target

Still images get one detection pass. Burned-in subtitles are a different beast: the text changes every few seconds, so a static mask from frame one is useless by frame two hundred.

Our subtitle mode therefore reconfigures the whole pipeline. It drops YOLO and the heuristics entirely and runs text detection only — subtitles are text by definition, and every extra detector is pure false-positive risk. It caps the detection input at 640 px because per-segment re-detection has to be cheap. It filters boxes to the lower half of the frame (box center below 50% of frame height), which is where burned-in subtitles live, then expands each kept box with margin — 6% of width and 30% of height, floored at 4 px — because subtitle strokes carry outlines and shadows that extend past the tight box. And it re-detects on a sampling cadence: every 5 frames by default. (For static watermarks we instead detect once on the first frame and reuse the mask with frame-skipping — measured about 3× faster than per-frame re-detection — but sampling is the only correct strategy for a target that moves and changes content.)

Try it

Everything described here runs in your browser tab today, free and without sign-up:

Remove watermarks from images — the full dual-engine pipeline with glyph-level masks
Remove watermarks from videos — static-mark mode with first-frame detection
Remove burned-in subtitles from video — lower-half text detection with sampled re-detection

Part 6 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Running LaMa Inpainting and Real-ESRGAN in the Browser: Tiles, Memory Walls, and the Optimizations We Threw Away

Umasou — Fri, 09 Oct 2026 02:50:42 +0000

We build ClearPix, a set of media cleanup tools that run entirely in the browser — no uploads, no server GPUs. That means when a user removes a watermark, a 198MB LaMa inpainting model runs inside their tab, and when they upscale an image, Real-ESRGAN runs next to it in the same JavaScript heap. This post is about what it actually takes to ship that: why we pin a flagship inpainting model to WASM on purpose, how tiled inference keeps large images from blowing up the tab, which "obvious" optimizations we benchmarked and threw away, and the one rule that saved us repeatedly — profile before you touch anything.

The real constraint is memory, not FLOPS

Server-side model selection is mostly about quality-per-FLOP. In the browser that framing breaks down completely. Our constraints, in order of how often they bite:

Memory. Model weights, WASM linear memory, tensors, source image, and output canvas all live in one renderer process. A 4K image alone is ~33MB as RGBA. Hold a large ONNX session open at the wrong moment and Chrome kills the tab — no exception, the page just vanishes.
Driver stability. A WebGPU session that creates successfully says nothing about whether session.run() will return. We've seen big models deadlock mid-inference on specific GPU/driver combinations with no error at all.
FLOPS. Barely matters. Our profiling keeps showing browser inference is memory-bandwidth-bound, not compute-bound.

This is why our model choices look odd from a papers-first perspective: we run Real-ESRGAN's compact general-x4v3 variant instead of the classic RRDBNet x4plus — the "weaker" model won every acceptance case while running an order of magnitude faster, because "fast in the browser" is a property of the architecture, not the runtime. The full model-selection benchmark (and what happened when we later quantized the winner to fp16) is part 3 of this series; here we focus on what those models cost once they're loaded.

An honest inpainting model comparison: MI-GAN vs LaMa

Our watermark remover ships two inpainting engines, and we picked them on a bench suite rather than paper reputation:

	MI-GAN	LaMa (big-lama ONNX port)
Size	~29.5MB	198MB (fp32)
Input	fixed 512x512, single 4-channel float32 tensor	fixed 512x512, image + mask tensors
WASM speed per tile	~0.6s	~4.3s (about 7x slower)
Large-hole structural reconstruction	weak	strong — our big-hole text case gained +11.6dB over MI-GAN
Small isolated regions	slightly better	flat (-0.8dB on our small-logo case)

LaMa's fast Fourier convolutions give it a global receptive field, which is exactly what you need when a watermark sits on top of a face or a building edge and the fill has to reconstruct structure, not just smear texture. MI-GAN is the better tool for small, isolated regions — faster and marginally cleaner there. So the product defaults to MI-GAN and offers LaMa as an explicit "HQ" opt-in. 198MB is never prefetched; the user clicks HQ, sees the download size, and chooses.

If you want to run LaMa in a browser yourself, the tensor conventions that cost us a probe script to pin down: image is [1,3,512,512] float32 in 0–1, mask is [1,1,512,512] with 1 = inpaint, and the output is already on a 0–255 scale — the official demo casts straight to uint8, so don't normalize it again.

Why our LaMa runs on WASM on purpose

This is the decision people question most. Our MI-GAN path tries WebGPU first and falls back to WASM — and it's great when it works: after upgrading onnxruntime-web to 1.30, the WebGPU execution provider took inpainting from 2100–3500ms down to ~400ms. But LaMa at 198MB fp32 kept hitting a nastier failure class: session creation or inference would deadlock on certain GPU/driver combinations, with no error and no timeout that could rescue it. A fallback you can't trigger is not a fallback.

Meanwhile, LaMa on WASM does one tile in ~4.3s — slow, but fine for an opt-in quality mode. So we pinned it:

// initLama — reliability over peak speed: 198MB fp32 has repeatedly
// deadlocked under WebGPU (creation *and* inference), so we never try.
const s = await createSession(bytes, onProgress, ['wasm']);

For the smaller MI-GAN we still gamble on WebGPU, but with a runtime safety net, because creation succeeding proves nothing. Every WebGPU run() is wrapped in a watchdog — 120s for MI-GAN, 300s for LaMa-class workloads (WASM gets no timeout; it's the final path and legitimately slow). On timeout or error, we rebuild the session from the cached model bytes (IndexedDB hit, no re-download), retry once on WASM, and then stick to WASM for the rest of the page lifetime instead of re-rolling the dice every pass.

One more production landmine in the same file: warming up models with a dummy forward pass hides 1–5s of shader compilation, but the warmup must be awaited before the first real inference. ORT's WebGPU EP does not tolerate concurrent run() calls on one session — we reproduced Session already started and tensor detach races three times in a row before adding that await.

Where browser ONNX actually runs out of memory

The OOMs we hit were never where we expected. Two real ones:

Writing the 198MB model to IndexedDB in one chunk, while a session was being created on top of it, pushed the renderer over the edge and killed the tab. Chunking the IDB write into 32MB pieces fixed it.
Keeping inference sessions alive during video assembly. After per-frame processing finishes, we run the encoder — and that assembly step is the memory peak of the whole pipeline. If the ORT session is still holding hundreds of MB of WASM heap at that moment, low-end devices get their renderer OOM-killed: the "page silently crashes" bug report. So we explicitly release sessions before assembly; the model bytes stay in IndexedDB, so re-init later is cheap.

The lesson generalizes: in the browser, your memory budget is a timeline, not a number. What matters is what you're holding at the peak moment.

Tiles and overlap-crop: staying under the memory wall

Both engines take fixed 512x512 inputs (inpainting) or dynamic but memory-hungry inputs (super-resolution), so everything goes through tiles.

For Real-ESRGAN we feed overlapping 256x256 tiles and paste back only the center of each upscaled tile, discarding an 8px border on interior edges — classic overlap-crop. The border pixels of a super-resolved tile are where edge artifacts live, and discarding them makes tile seams invisible. Tiling also bounds peak memory: per-tile tensors are tiny and constant no matter how big the source is, so large inputs can't crash the tab. (We still cap input at 1400px on the long side in the MVP — product guardrail, not a technical one.) The light model even let us grow tiles: going from 128px tiles with 16px margins to 256px tiles with 8px margins cut the redundant overlap pixels from about 1.78x to 1.14x of the image area.

Inpainting tiles are planned from the mask, not a fixed grid. The pipeline: connected components of the mask → group nearby components into clusters (boxes expanded by 80px that intersect are one watermark field) → per-cluster crops with context margin. Fine-detail passes use native-resolution tiles of at most 340px core, which with a 1.5x context margin stays under the 512px model input — zero downscaling, so no blurry patches. LaMa's HQ path does the same with a 340px pixel grid and 1.3x context (crops stay ≤442px).

Two tiling lessons we learned the hard way:

Never shatter dense regions into independent tiles. Each tile is an independent hallucination; stitching them produces visible discontinuities. Triggering tiling by cluster size once tanked one bench case from 19.7 to 11.9dB. The fix: trigger on the largest single connected component, plus a special case for long thin clusters — text lines with a union aspect ratio over 3 — because squashing a whole text line into 512x512 is exactly what produced the "you can tell it was edited" smearing. Dense near-square fields still get one global pass: one consistent hallucination beats many stitched ones.
Repaint glyphs, not rectangles. For text watermarks we build glyph-level masks — probability map above 0.5 as seeds, candidate pixels by color distance, keep only what's connected to a seed, dilate 2px. If a box ends up under 8% covered we fall back to the plain rectangle (that guard is what keeps faint tiled watermarks from losing recall). This cut over-painting enough to move precision from 41.4% to 43.2% with recall untouched.

Optimizations we benchmarked and killed

The most valuable section of our internal docs is the "do not retry" list. Three model-engineering entries we measured and killed: a frame-level worker pool for parallel inference (no faster than the serial path), IOBinding on the WASM path (the transfer tax it saves is a rounding error next to run() itself), and swapping x4v3 for CUGAN (a lateral move dressed up as a breakthrough by benchmarking against the old baseline). The numbers behind all three are in part 3 of this series — along with the one quantization change that did work, fp16 super-resolution weights, and its catastrophic mirror image when we tried it on LaMa. The short version: browser inference is memory-bandwidth-bound, so you win by moving fewer bytes, and whether quantization is safe is a property of the model's architecture, not of your tooling.

Every entry in that kill list exists because we measured first. The rule we now enforce: no performance work starts without a per-stage timing breakdown and an expected share of the total. If you can't say what fraction of runtime you're attacking, you're guessing.

Try it

Everything described here runs in production, free, with no uploads — the models download to your browser and nothing leaves your machine. If you came here looking for LaMa inpainting online, clearpix.org/remove-watermark is the dual-engine MI-GAN/LaMa pipeline described above. For a Real-ESRGAN web demo with the tiled overlap-crop pipeline, try our image upscaler. Video versions of the same engines power our free video watermark remover.

Part 5 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

How We Remove Watermarks from Video Fast: Motion-Adaptive Frame Skipping + Temporal EMA

Umasou — Thu, 08 Oct 2026 12:29:00 +0000

(and the bug that froze our masks — a regression story about wiring performance switches to product options)

Running an inpainting model on every frame of a video is the obvious way to do video watermark removal — and unaffordable in a browser. This is the write-up of how we got 58–67% of frames to skip the model entirely: a motion-adaptive, anchor-based frame-skipping gate, a temporal EMA for flicker-free output, plus the failures behind the design — a chain-freeze bug (PSNR 38.7 → 16.7), an anchor-age regression (38.30 → 34.69), and a UX "simplification" that silently disabled the whole optimization and cost us 3x. All numbers are from our own benchmarks; all code is from the shipping pipeline.

Per-frame video inpainting is the cost center

ClearPix removes watermarks entirely client-side: ONNX-runtime inpainting on WebGPU or WASM, fed by WebCodecs decode, encoded back with MediaBunny. No server, no upload.

The profile was blunt. Inside one inpaint call, planning cost about 4.4ms while the model run() segment took 97% or more. Per frame, inpainting took 2100–3500ms on WASM; the WebGPU execution provider brought that to roughly 400ms — still the dominant line item. Pipeline overhead was already cut elsewhere (ffmpeg.wasm → WebCodecs + MediaBunny: 95s → 9s per job). The biggest lever left was not a faster model — it was not running the model at all.

Frame skipping for video processing: reuse, don't recompute

The observation that makes skipping viable: most watermarks sit on content that barely changes — a corner logo over a talking head, a username stamp over a screen recording. Under a static mask, the pixels inside the mask bounding box are nearly identical frame to frame. If nothing in the region moved since the last inference, that output is already the correct answer — composite it onto the current frame instead of inferring again.

The per-frame gate, trimmed from RemoveVideoWatermark.tsx:

const small = makeRegionGray(img, maskBbox);   // bbox region → ~64px-wide grayscale
// Compare against the anchor (last real inference frame); drift measured
// exactly against the anchor too
const mae = st.anchorSmall ? regionMAE(small, st.anchorSmall) : Infinity;
const deltaAnchor = st.anchorRaw
  ? estimateRegionDelta(img, st.anchorRaw, maskData, maskBbox)
  : [0, 0, 0];

if (mae < st.skipThreshold && st.anchorOut) {
  // Skip: composite the anchor's feathered mask region onto the current frame
  out = compositePrevRegion(img, st.anchorOut, maskData, maskBbox, deltaAnchor);
  st.skipped++;
} else {
  out = await runInpaint();
  st.anchorRaw = img;   // a real inference becomes the new anchor
  st.anchorOut = out;
  st.anchorSmall = small;
}

The cheap part matters: the diff runs on the mask bbox downsampled to ~64px grayscale — under 5ms per frame, against hundreds for inference. On a skip, compositePrevRegion pastes the anchor's mask region back with the inpainter's own feathering semantics (100% inside the mask, transition band on clean pixels outside), so a reused frame looks like an inferred one.

The current build skips 58% of frames on our watermark e2e clip (threshold 5.00) and 67% on the subtitle e2e (threshold 4.21); the earlier gate measured 94 of 144 (65%).

The threshold is not a magic constant. The first five frames always run full inference; we collect their MAEs and set skipThreshold = median × 3, clamped to [2, 5]/255 — dark and noisy footage get different baselines for free. (The ceiling of 5 is itself a measured fix — keep reading.)

The chain-freeze bug: PSNR 38.7 → 16.7

The first version (v1) compared each frame to its immediate neighbor and reused the previous output. It passed screen-recording tests, then failed catastrophically on a slow-gradient clip.

On gradient footage each frame drifts ~1/255, so per-frame MAE stays below threshold and the gate skips forever — the mask region freezes at the last inference while the rest of the frame changes around it: a visible, motionless rectangle. E2e masked-region PSNR collapsed from 38.7 to 16.7.

Patch one was a cumulative drift budget: accumulate skipped-frame MAE since the last inference and force re-inference when the total crosses the threshold. That recovered PSNR only to 22.9 — reused pixels still lagged the scene's exposure on gradients. Patch two added delta compensation (estimateRegionDelta): the per-channel mean difference between current and previous raw frames over the non-mask pixels in the bbox, added to historical pixels before blending. Budget + delta landed at 38.44 (vs a 38.74 no-skip reference), inside tolerance, with a clean three-clip visual check in real Chrome. That version shipped.

But the budget was always a proxy: the error we care about is "how far is this frame from the frame whose output we're reusing," and chained per-step MAEs only approximate that.

Anchored comparison: the fix that deleted the budget

So we refactored (v2): compare each frame directly against the anchor — the most recent frame that really ran inference — and reuse the anchor's output on skips. Skip error is now exactly what the threshold bounds; the drift budget was deleted outright. Delta compensation became exact too — measured against the anchor itself.

The first anchored run gave us a proper scare: subtitles gained +3–4dB on all four e2e segments, but the gradient watermark clip dropped from 38.30 to 34.69dB masked-region. Root cause: that clip's drift varies with y (a sinusoidal exposure gradient), so one mean delta can't fully compensate it, and the residual grows with anchor age. Two measured fixes:

Tighter threshold ceiling: 8 → 5. Anchors refresh more often, bounding how stale a reuse can get.
Split the deltas. Compositing reuses the anchor → to-anchor delta; the EMA blends against the previous frame's output → to-neighbor delta. One mean delta serving both timebases was the bug.

Final numbers: e2e-video masked region 39.84dB (+1.5dB over the chained baseline), 58% skipped (threshold 5.00); subtitles 37.3 / 32.3 / 28.7 / 30.1dB across four segments, 67% skipped (threshold 4.21).

Flicker-free video watermark removal with a temporal EMA

Skipping frames solves cost, but video inpainting temporal consistency needs one more layer. Even frames that do run inference can jitter: the model hallucinates slightly different textures each time, and at 30fps that reads as flicker inside the repaired region.

Our answer: a per-pixel exponential moving average, applied only inside the mask dilated by 4px, with a motion-adaptive blend factor (applyTemporalEMA, trimmed):

// Per-pixel motion from raw frames (0..1)
const m = (|dr| + |dg| + |db|) / (3 * 255);
const t = Math.min(1, m / 0.08);   // motion 0→8%
const s = t * t * (3 - 2 * t);     // smoothstep
const a = 0.4 + 0.6 * s;           // alpha 0.4→1.0
// History term: to-neighbor deltaPrev (composite uses to-anchor deltaAnchor)
out = curr * a + (prev + delta) * (1 - a);

Low motion → alpha 0.4 (lean on history, flicker dies); high motion → alpha 1.0 (pure current frame, no ghosting), with a smoothstep between. Three boundaries are deliberate:

It runs only inside the dilated mask. The rest of the frame is already bit-identical to the source; touching it would only blur real motion.
It runs on both inferred and skipped frames, so reused composites get the same treatment as fresh inferences.
Its history term is compensated against the previous frame, not the anchor — the split described above.

Moving watermark removal: sampled re-detection that still skips frames

Everything so far assumes a fixed mask — wrong for a watermark that moves. Our "Detect sampling" setting has three rates: First (detect once, reuse for the whole video), Med (every 5 frames), High (every frame). Moving watermarks need Med or High.

A rebuilt mask means all temporal state belongs to the old mask position — reusing it would smear content across the frame — so a refresh resets everything: the anchor group (anchorRaw/anchorOut/anchorSmall), the prev group (prevRaw/prevOut), and the dilation cache. No special-casing needed: anchorSmall = null makes the next MAE Infinity, forcing full inference on the re-detect frame (which becomes the new anchor), and prevOut = null makes the EMA skip itself. Between re-detects the watermark usually holds still, so skipping stays fully active on Med/High — the anchor gate handles the motion. (If detection fails or finds nothing, we keep the previous mask and leave temporal state untouched.)

The same machinery powers our subtitle-removal tool, which defaults to Med because burned-in subtitles change every few seconds.

The rollback: how a UX cleanup cost us 3x

Now the embarrassing one. In a September UX pass, the "First" sampling option looked like confusing jargon, so we removed it and forced per-frame re-detection for everyone.

What we didn't trace: skipping and the EMA are keyed on the mask's bounding box, and with per-frame re-detection the mask is rebuilt every frame, so maskBbox was never stable — the entire optimization path silently disabled itself, and video removal got ~3x slower. Users reported "streaming feels slower" — wrong; comparing the complaint timeline with the commit timeline pointed at the sampling default. We rolled back the next day.

The never-again rule: before deleting a "confusing" option, check whether a performance optimization is keyed to it. Performance switches must be explicitly bound to product options — the dependency belongs in a comment, a test, or the option's name, not in someone's memory. Our detectEvery state now carries exactly that comment.

Numbers, in one place

Change	Measurement	Result
Skip gate cost	mask-bbox frame diff	<5ms per frame
v1 chain-freeze bug (single-frame MAE)	e2e masked-region PSNR	38.7 → 16.7
v1 drift budget only	e2e masked-region PSNR	22.9
v1 drift budget + delta (first shipped version)	e2e masked-region PSNR / skip rate	38.74 → 38.44; 94/144 skipped (65%), pipeline 20s vs 170–370s WASM
v2 anchored, first run	gradient clip masked-region PSNR	38.30 → 34.69 (subtitles +3–4dB on all four segments)
v2 anchored, final (ceiling 5, split deltas)	e2e-video masked-region PSNR / skip	39.84dB (+1.5dB vs chained baseline), 58% skipped (thr 5.00)
v2 anchored, final	e2e-subtitles, four segments	37.3 / 32.3 / 28.7 / 30.1dB, 67% skipped (thr 4.21)
Removing the "First" option (UX pass)	video removal speed	~3x regression, rolled back next day

Try it

Everything here ships in our free video watermark remover — in-browser, no upload, the model runs on your GPU. If your real problem is burned-in text, the same pipeline drives our subtitle removal tool. Feedback on weird footage is how most of these fixes happened.

Part 4 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

FP16 Quantization Made Our In-Browser Real-ESRGAN 4.8x Faster — and Nearly Destroyed Our Inpainting Model

Umasou — Sun, 04 Oct 2026 09:19:50 +0000

We run super-resolution and watermark removal entirely in the browser at ClearPix, using ONNX Runtime Web on WebGPU. This post is the writeup of one optimization round that produced our single biggest inference speedup — 4.8x at page level from fp16 quantization — and, in the same week, two of our worst quality regressions when we applied the exact same trick to the wrong model.

If you're fighting slow browser GPU inference with onnxruntime-web, you'll get three things here: the performance model that explains why fp16 works and parallelism doesn't, the real numbers behind choosing SRVGGNetCompact over the "stronger" RRDBNet, and a concrete failure case study of fp16 quantization quality loss in mask-gated models (with PSNR before/after). At the end: the decision order we now follow before touching any model.

Why browser GPU inference is slow: bandwidth, not FLOPs

The mental model most engineers bring from CUDA land is wrong for the browser. On a desktop GPU with a fat model, you're often compute-bound and the answer to slowness is "more parallelism." In the browser, running through onnxruntime-web's WebGPU execution provider, our profiling kept saying the same thing: we are memory-bandwidth bound.

Two pieces of evidence forced this conclusion on us.

First, worker pools did nothing. We built a frame-level worker pool for parallel video inference — multiple workers, each with its own ONNX session, processing frames concurrently. The result: 396 seconds versus 393 seconds for the serial path. That's within noise. Adding workers only added contention for the same memory bus. If more parallelism can't help, FLOPs were never the bottleneck.

Second, IOBinding did nothing. We profiled the WASM path expecting the JS↔wasm tensor copies to be a meaningful tax, and planned to adopt IOBinding to shave it. The instrumented timings showed the transfer tax was 1–2% of the total; the run call itself was over 97%. Optimizing anything outside the run segment was rearranging deck chairs.

Once you accept "bandwidth-bound," the optimization list writes itself: move fewer bytes. Halve the weight precision and you halve the bytes read per inference. That's the entire theory behind what happened next.

Architecture first: SRVGGNetCompact (x4v3) vs RRDBNet x4plus

Before quantization, though, we made a bigger decision that fp16 later rode on top of: swapping the model architecture itself.

We started with Real-ESRGAN x4plus, the well-known RRDBNet — 1187 nodes, 67MB of weights. On paper it's the stronger model. In the browser it was a disaster in a specific, instructive way: on ORT's WebGPU EP it showed no speedup at all over CPU. Roughly 2.5 seconds per tile on both backends. A big RRDBNet pushes so many bytes through the graph that even the GPU just waits on memory. Runtime-side tuning space was, for practical purposes, exhausted.

So we switched to realesr-general-x4v3, which is SRVGGNetCompact — a compact architecture designed for general and video content. The numbers from our September benchmarks:

	RRDBNet x4plus	SRVGGNetCompact x4v3
Graph size	1187 nodes	71 nodes
Weight file	67MB	4.9MB
CPU inference speed	baseline	14–27x faster
Quality (3-case PSNR head-to-head)	baseline	wins every case, +0.1 to +1.5dB

The quality result deserves emphasis: the compact model didn't just hold even, it won all three benchmark cases, by up to 1.5dB. The heavier RRDBNet's texture hallucinations actually hurt fidelity on real content. Per-frame, the swap took us from about 75 seconds to about 7 seconds — 10x — before we touched precision at all.

Lesson one of model engineering for the browser: the first selection criterion is "designed for the deployment environment," not "SOTA in the paper." If you're shipping a Real-ESRGAN in browser code, picking the compact variant matters more than anything you can do to the runtime.

What fp16 actually bought us on onnxruntime-web WebGPU

With x4v3 in place, we quantized the weights to fp16 and measured carefully before shipping:

fp16 vs fp32 outputs compared against each other: PSNR 57dB+. For practical purposes the outputs are identical.
Single-tile inference on WebGPU: 403ms → 113ms. That's 3.5x — better than the naive 2x you'd predict from halving bandwidth, because Apple GPUs also get an fp16 ALU throughput bonus.
Page level on our video upscaler (24 frames of 720p): 31.4s → 6.6s. 4.8x end-to-end.
The fp16 file is 2.4MB versus 4.9MB for fp32, which also halves the download.

The implementation detail that kept this painless: we exported with keep_io_types, so the JavaScript side keeps its fp32 tensor contract and zero application code changed. The WASM EP also runs the fp16 model fine, so our Chromium e2e path needed no special-casing.

We still ship fp32 as a fallback spec. If every fp16 URL fails, the loader falls through to the original model; the feature keeps working, just at fp32 speed. This is a habit we'd recommend to anyone shipping models to browsers: every production incident we've had got closed by a fallback chain, not by the primary path.

export const UPSCALE_MODEL: ModelSpec = {
  key: 'realesr_general_x4v3_fp16_v1',
  urls: [
    `${STATIC_BASE}/models/realesr-general-x4v3-fp16.onnx`,
    `${BASE}models/realesr-general-x4v3-fp16.onnx`,
  ],
  sha256: '7e7b7f58...',
  expectedSize: 2444975,
  fallback: {
    key: 'realesr_general_x4v3_v1',
    urls: [/* R2, local, HF mirrors for the fp32 original */],
    expectedSize: 4866417,
  },
};

So fp16 was a free 4.8x. Naturally, emboldened, we reached for the same hammer on the other model in our stack. That's where the week went sideways.

When fp16 quantization quality loss is real: the LaMa failure

Our watermark removal pipeline uses LaMa, an inpainting model. Inpainting is a different beast from super-resolution in one structurally important way: LaMa is mask-gated. The mask doesn't just mark where to work — it gates the network's behavior. The model has to learn "trust these pixels, hallucinate those."

We quantized LaMa to fp16. The benchmark came back: masked-region PSNR 21.1 → 11.9dB. That's not a subtle regression — the watermark was left almost fully intact in the output. We tried int8 quantization too, both static and weight-only, because maybe it was an fp16-specific quirk. It wasn't: int8 gave 24.5 → 15.6dB on our dark-text case, with the text watermark surviving completely.

The mechanism, once we thought about it, is almost obvious. A super-resolution network has no gating structure; its job is a smooth, local, well-conditioned transform, and half-precision noise mostly washes out. A mask-gated inpainter has a decision boundary inside the network: which regions are trustworthy context and which must be regenerated from scratch. Numeric precision degradation pushes the model across that boundary — the gating fails softly, and the model starts "copying" the input through instead of repairing it. The watermarked pixels in the masked region get treated as content to preserve rather than a hole to fill. Output looks "fine" at a glance, except the watermark is still there.

Both failures were caught by the benchmark before shipping, and both directions are now on our internal do-not-retry list. The takeaway isn't "never quantize." It's that quantization feasibility is a property of the model's architecture: does it have precision-sensitive structure — mask gating, large-magnitude outputs, long error-accumulation chains? If yes, fp16/int8 will almost certainly kill it. If no, fp16 is free bandwidth.

The decision order we now follow

After this round, we wrote down the order of operations for any model change, and it's deliberately architecture-first:

Architecture. Pick a model designed for the deployment environment. Lightweight and purpose-built beats paper SOTA. Our x4plus → x4v3 swap (10x, quality up) was worth more than every other inference optimization combined.
Bit width. Only after the architecture is fixed, ask whether the model has precision-sensitive internals. No gating, no long accumulation chains — fp16 is on the table, and it's the cheapest 3–5x you'll ever get. Gating present — walk away.
Runtime. Last, and with measured expectations. Worker pools, IOBinding, EP tuning: in our profiling these were the 1–2% layer. Profile first, write down the expected share of the total before touching anything, and be ready to be told "0x."

And one meta-rule that saved us repeatedly: quality admission is always the benchmark, never the paper's numbers and never "it looks fine." The LaMa fp16 output looked plausible. The PSNR said 11.9dB. Benchmarks saw it; eyes didn't.

Try it

Everything above runs in production, client-side, with no uploads — your media never leaves the tab. You can feel the fp16 x4v3 path yourself in our image upscaler or the video upscaler (the 6.6-second page from this post). The LaMa side — kept safely at fp32 — powers our free video watermark remover and the image tools on clearpix.org.

Part 3 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.