DEV Community: Umasou

Automatic Watermark Detection: What 500 Real Cases Changed

Umasou — Sat, 07 Nov 2026 15:02:25 +0000

We tested automatic watermark detection on 500 real AI-platform frames: editable-selection coverage rose from 75.8% to 95.0%. A separate 36-image human-mask evaluation still measured only 47.1% precision and 11.0% recall. Those results are not contradictory: locating a useful corner box is much easier than selecting exactly the watermark pixels.

Check	Before	After	What it actually measures
Real platform frames with a selection	379 / 500 (75.8%)	475 / 500 (95.0%)	Whether Auto Detect leaves something editable
Frames with no selection	121	25	Coverage failures
Runtime errors	0	0	Batch stability
Human-labeled mask precision	—	47.1%	Selected pixels that match human masks
Human-labeled mask recall	—	11.0%	Human-mask pixels recovered by Auto Detect
Human-labeled IoU	—	9.8%	Pixel intersection over union

The last three rows are the warning label. This work made Auto Detect more useful; it did not make automatic masks exact.

Start with the job, not the model leaderboard

A person using a watermark remover wants the mark gone. They do not want to learn the difference between object detection, segmentation and inpainting before they can click Remove.

Our original pipeline already had three useful parts. A detector proposed boxes. Text heuristics produced glyph-shaped masks. LaMa or MI-GAN rebuilt the selected area. We added SlimSAM because a promptable segmenter can turn a rough click or box into an object-shaped selection. In manual use, that worked well: a person could click a mark, reject a bad candidate and add another prompt.

Auto Detect exposed a different problem. A detector box is not a segmentation prompt with known intent. It may contain a watermark, part of the scene, or both. Feeding every box to SAM simply gave us a more expensive way to select the wrong object.

The first rule became conservative: automatic selection should act only when several independent signals agree. If they do not, leave the target for a manual click, box or brush.

The first pipeline was too willing to paint

The initial combined route tried SAM candidates and glyph masks, then fell back to boxes. On a 26-image development set, an early full run reached 31.3% recall but only 6.5% precision. Large false-positive areas dominated the result. Eleven cases also failed in that early evaluator, so even its aggregate numbers were not a clean product baseline.

Filtering alone did not fix it. A more conservative run raised precision to 13.7%, with 34.1% recall. That trade was still unusable: too much scenery was being selected.

We then required the automatic mask to pass several geometric checks. SAM pixels had to intersect the detector box, remain mostly contained by it, overlap available glyph evidence, and stay below both box-relative and whole-image area limits. Text masks used the intersection of glyph and local-stroke evidence rather than their union. The 26-image run reached 57.7% precision and 19.9% recall, with 12 of 26 cases receiving an automatic mask.

That looked like progress until we evaluated a different set. On the 36 human-annotated images, Auto Detect measured 47.1% precision, 11.0% recall and 9.8% IoU. Only one case met the evaluator’s complete-case criterion. A threshold tuned on a small development set had not become a general solution.

This is why the original YOLO, OCR and glyph-mask design still matters, but cannot be the end of the pipeline. It supplies evidence about where text might be. It does not prove which pixels belong to an overlay.

Use SAM where its answer can be checked

SAM now runs automatically only for candidates with supporting structure: similar detected boxes or solid watermark-like regions. A lone weak text proposal does not automatically trigger segmentation.

For each accepted candidate, we check:

at least 5% of the detector box is covered;
at least 60% of the candidate stays inside that box;
text candidates have at least 35% overlap with glyph evidence;
the candidate is no larger than twice the box area;
the candidate covers no more than 20% of the whole image.

These are guards, not a measure of semantic correctness. They reject obvious spills and disconnected scene selections. They cannot prove that a white letter is a watermark rather than a road marking or clothing print.

The conservative text path is stricter still. It requires detection quality of at least 0.95, enough glyph pixels, sensible fill and containment, limited background complexity, and a text-like aspect ratio. When those conditions fail, it returns no bitmap rather than silently substituting a large rectangle.

That behavior follows the lesson from our earlier false-positive cleanup: a missed automatic selection can be repaired by the user, while an automatic mask over the main subject can ruin the image.

“Find Similar” needed evidence too

Repeated marks looked like an easy win. Select one watermark, match its gray-level and edge pattern in other detector boxes, then translate the accepted mask to matching positions.

The first attempts were not good enough. On a difficult five-case repeated-mark set, the evaluated variants selected large incorrect areas; precision stayed around 1–2%. Template similarity can be fooled by repeated texture, and a translated mask inherits the shape and alignment errors of the source.

We kept the capability but narrowed its job. Similar-box detection compares source type, watermark type, dimensions and masked edge appearance. The interface offers Find Similar only when at least three similarly sized candidates support the pattern. It is an optional way to reduce repeated clicking, not proof that every matching patch should be removed.

A second segmentation model did not rescue the metrics

We also tested a dedicated watermark-segmentation model on the human-labeled set. At a 0.3 threshold it reached 65.0% precision and 31.4% recall—better than the automatic hybrid on those metrics, but still short of the “selected means correct” behavior we wanted.

A two-model intersection did not reproduce the strong synthetic-set result that motivated the experiment. On the human set, its best listed operating points were around 65–66% precision with lower recall as thresholds rose. A lightweight student-model distillation route produced no usable Golden Set output and was not integrated.

Those experiments were worth doing because they closed off an attractive shortcut. Adding another model or taking an intersection is not automatically a precision strategy. The evaluation set decides whether the idea survives.

The 500-frame batch changed a different bottleneck

The human set is good for pixel truth. It is too small to describe the variety of platform marks people actually bring to the tool. We therefore built a separate batch runner for 500 frames from AI video platforms, with per-brand, position and style summaries, mask RLE output, resume support and screenshot capture.

This runner measures selection coverage, not pixel precision. It also checks that the red pixels used for measurement are new overlay pixels rather than red content already present in the source.

The largest remaining coverage problem was not SAM making bad masks. It was the pipeline returning nothing when no exact mask passed. For a single detector box, we already kept an editable box fallback. We extended that idea to multiple candidates, but only for a high-confidence, small edge region:

detection quality at least 0.95;
box center within 0.15 normalized units of an image edge;
box area no larger than 8% of the image;
candidates ranked by detection quality plus an edge-position bonus.

The result is deliberately a box, not a claim of exact segmentation. A person can move it, resize it or switch to Smart, Paint or Rect before removal.

Platform group	Frames	Selected after the change	No selection
Alice AI	133	123 (92.5%)	10
Dola AI	69	69 (100%)	0
Luma AI	46	42 (91.3%)	4
All platform and generic marks	500	475 (95.0%)	25

Offline candidate-ranking checks agreed with the expected watermark region for 437 of 451 selected candidates, or 96.9%. That number describes location metadata. It is not a substitute for the human-mask precision result above.

The remaining 25 no-selection frames mainly contained low-confidence proposals, oversized boxes, unusual positions or complete detection misses. We left them unselected rather than lowering every threshold to make the coverage chart reach 100%.

What this means for the product

Auto Detect is now a first pass with visible limits. It can produce a precise mask when the evidence agrees, or an editable box when a small high-confidence edge target needs review. It can also return nothing. Smart selection, a rectangle and a brush remain part of the same workflow rather than hidden fallbacks.

For video workflows built around stable platform overlays, we also published focused preset pages for Alice AI watermark removal, Dola AI watermark removal and Luma Dream Machine watermark removal. Their sample data shows why presets need review: all sampled Alice and Dola marks were bottom-right, while only 80.4% of Luma marks were upper-left.

The same sample now supports preset guides for DeeVid AI, Dreamina AI / Jimeng, POLLO AI, Kling AI, Hailuo AI / MiniMax, Vidu AI and DIGEN AI. Each page addresses its actual layout: Dreamina has two independently editable marks, POLLO includes the solid backing plate, and Hailuo accounts for a longer bottom-edge wordmark.

The next improvement is not a blanket lower threshold. We need more human masks across low-contrast text, solid badges and repeated marks, then per-type operating points with false-positive cost attached. Coverage tells us how often the tool offers a starting point. Human-labeled precision tells us whether that starting point can be trusted.

Try the current workflow in the image watermark remover, or use the video watermark remover for fixed overlays. Both run locally in the browser; review the mask before removing, and add missed areas manually.

Part 21 of the ClearPix engineering series. Evaluation code: apps/web/scripts/eval-auto-select-quality.mjs and apps/web/scripts/eval-watermark-batch.mjs. The 500-frame snapshot is a fixed internal QA sample; its coverage figures are not presented as public benchmark accuracy.

Remove Watermarks Without Uploading: How We Built an In-Browser AI Editor

Umasou — Sun, 01 Nov 2026 11:56:13 +0000

This post is cross-posted from the ClearPix engineering blog. ClearPix is free and open at clearpix.org; the watermark-removal core described here is MIT-licensed and on GitHub at SharksevenRo/browser-watermark-removal.

Remove Watermark Without Uploading: How We Built an In-Browser AI Editor

Most online watermark removers ask you to hand over your photo first and ask questions later. We built ClearPix the other way around: the AI models come to your device, your files never leave it, and inference runs entirely in your browser tab. This post is the architecture overview of how that actually works in production — the ONNX Runtime Web + WebGPU inference stack, the model distribution pipeline (R2 CDN, multi-mirror fallback, SHA-256 self-healing cache), the places it broke on us, and how we keep the privacy claim structural instead of rhetorical. No marketing math; every number below comes from our own engineering logs.

Does a watermark remover upload your photos? Usually, yes

The upload question is not paranoia — it is the default architecture of the category. Nearly every "free online watermark remover" is a thin frontend over a server-side inference API. Your image goes up, gets processed on someone else's GPU, and comes back down. Whether the privacy policy says the file is deleted afterward, you have no way to verify it, and neither do we. The structural fact is that your media crossed the wire.

We kept running into this as a genuine user anxiety, and we shared it. Photos with watermarks are often personal — screenshots, ID documents, family pictures, client work. "Trust us, we delete it" is a weak foundation for a tool people use on sensitive files. So we asked the harder question: can the inference just happen on the user's machine?

The answer, in 2026, is yes — if you are willing to solve the distribution and reliability problems that server-side tools offload to a data center. That trade is the whole story of ClearPix.

The architecture in one paragraph

ClearPix is a static site. There is no processing backend. When you open a tool page, your browser downloads ONNX model weights (once, then cached forever), loads them into ONNX Runtime Web, and runs inference on WebGPU where available with a WASM fallback. Your image or video is decoded locally, processed locally, re-encoded locally, and never serialized into a network request. The only bytes that travel are the models coming down to you, and a small set of aggregate analytics events that contractually cannot contain your media.

That single paragraph hides about 90% of the engineering. Let's unpack it.

In-browser AI image processing: the runtime layer

We run all neural inference through ONNX Runtime Web (ort 1.30.x), loaded via dynamic import so its ~1 MB parsed weight stays out of the initial page bundle — Core Web Vitals matter even for tools. Session creation prefers the WebGPU execution provider and falls back to WASM:

export async function createSession(
  modelBytes: Uint8Array,
  providers: readonly string[] = ['webgpu', 'wasm'],
) {
  const ort = await getOrt();
  // ... try each execution provider in order, return first that works
}

WebGPU is worth real money here: when we enabled the WebGPU EP (ort 1.30+), our inpainting step went from 2100–3500 ms to roughly 400 ms per region in our benchmarks, and in October 2026 the HQ LaMa path finally joined it at 0.86 s per 512×512 tile. But "session created" does not mean "inference works" — we have seen WebGPU runs fail in ways that looked environmental and were not. So the session runs behind a watchdog, and a dead WebGPU session falls back to WASM rather than hanging the tab. The fallback chain is not an afterthought; every production stall we have ever shipped was closed by one. (One of those "deadlocks" turned out to be a deterministic one-element Slice bug in ORT's default build — see part 14.)

Even the runtime binaries need their own distribution strategy. The WebGPU build we import now depends on ort-wasm-simd-threaded.asyncify.wasm (25.5 MB raw, 3.8 MiB after brotli) — if the runtime fails to load, ORT silently drops back to CPU inference and the user just experiences a mysteriously slow tool. We serve the ORT runtime from jsDelivr first (measured ~7× the throughput of our own R2 custom domain for these files) with a 5-second HEAD probe, falling back to our self-hosted R2 copy when the CDN is unreachable — which happens in some network environments.

A client-side watermark remover lives or dies by its model pipeline

Here is the uncomfortable truth of client-side AI: you are now a software distribution company. A server-side tool updates weights by deploying a container. We have to push tens of megabytes of model files to arbitrary browsers on arbitrary networks, keep them intact, and never strand a user.

Distribution. Models live in a Cloudflare R2 bucket behind a custom domain on our zone. This was a measured decision, not a default: direct r2.dev URLs gave us 75–110 KB/s, while the same-zone custom domain delivered 2.3 MB/s — a 20–30× difference that turns a model download from a coffee break into a pause. Models are also deliberately excluded from our deploy bundle; stripping them (plus internal bench assets) took our release zip from 27.4 MB down to 10 MB.

Permanent caching. Once downloaded, model bytes go into IndexedDB and are reused forever — repeat visits load from the device, and the tool works offline. Two details cost us real debugging time. First, big models are written in 32 MB chunks (above a 64 MB threshold), because a single structured clone of the 198 MB fp32 LaMa pushed the renderer process into OOM (the HQ weights are fp16 now, but the chunking rule predates that). Second, the cache is self-healing: we verify the SHA-256 of a cached model on every hit (~100 ms for a 67 MB file), because a download interrupted mid-write would otherwise poison a cache that never expires — permanently bricking that user's tool with no visible error.

Mirrors and fallback specs. Each model spec carries an ordered mirror list, a pinned SHA-256, and an optional fallback spec with a different weight format (fp16 → fp32) and an independent cache key. The first mirror whose bytes hash-match wins; a stalled connection (no bytes for 30 seconds) is aborted and handed to the next mirror; if every mirror fails, the fallback spec takes over:

for (const url of spec.urls) {
  try {
    const bytes = await fetchWithProgress(url, onProgress);
    if (hash(bytes) !== spec.sha256) throw new Error('integrity check failed');
    await idbSet(spec.key, bytes); // best-effort
    return bytes;
  } catch (e) { lastErr = e; } // next mirror
}
if (spec.fallback) return loadModel(spec.fallback, onProgress);

We learned this the hard way: on 2026-09-26 a mirror-chain failure took model loading down in production. Since then the rule is R2 first, public mirrors as backup, fallback spec chain underneath — designed before launch, not patched after an incident.

Making the privacy claim structural

"Private photo editor, no upload" is easy to print on a landing page. We wanted it to be checkable from inside the app.

The Privacy Shield. Every tool page renders a live status bar: a "0 bytes uploaded" chip, per-model download state with an offline-ready badge, and — once an inference session exists — the active backend translated into plain language. It is driven by a status bus inside the model loader, so it reflects what the engine is actually doing, not what the copywriter hoped. You can open devtools and verify the claim yourself: the network tab shows model bytes coming in and nothing going out.

An event contract we publish. We do collect a small set of aggregate events to know whether the tools work — things like process_complete (duration, backend, device tier) and result_feedback (thumbs up/down with reason tags). The contract, stated in our privacy policy, is that events never include images, file names, or file contents. Including them would defeat the point of the site.

Analytics that can't leak media. We use Microsoft Clarity for session replays to fix confusing UI. Because your files never reach a server this is already low-risk, but we additionally mark every canvas, image, and video element that can display user media with Clarity's data-clarity-mask attribute, so replays are structurally incapable of containing your content even if a selector goes wrong.

What this architecture costs us

Honesty section, because dev.to deserves one:

First-visit cost is real. Downloading models and compiling WebGPU shaders makes the first run slower than a server round-trip. We hide shader compilation and encoder init behind prefetch and warmup, and IndexedDB makes visit two effectively free — but visit one pays the toll.
We gave up control of the hardware. A data center GPU is a known quantity; a user's browser is not. WebGPU flakiness, missing cross-origin isolation (which caps WASM threading), and mobile memory limits are all our problem now. The fallback chains exist because each of these has bitten us.
Model choice is constrained by the browser. We pick architectures designed for the deployment environment over paper SOTA — swapping our upscaler to a compact architecture was a 10× per-frame win at equal-or-better quality, something no amount of tuning the heavyweight model would have produced. That is a full post on its own.

We think the trade is overwhelmingly worth it, but it is a trade.

Try it

Everything described above is live and free, no account:

Remove a watermark from a photo — detection, inpainting, and export all in-tab
Our free video watermark remover — streaming pipeline, up to two-minute clips
Remove hardcoded subtitles from a video
Upscale an image 4× — the compact SR model mentioned above (video upscaling lives on the same stack)

What's next in this series

This overview skipped the interesting fights. The rest of the series goes deep on each one:

The video pipeline rewrite — how replacing ffmpeg.wasm with WebCodecs + MediaBunny took full video processing from 95 s to 9 s, and how a streaming single-pass pipeline got us to O(1) memory.
Model selection and quantization in the browser — why fp16 was free bandwidth for our upscaler and, as we originally wrote it, destroyed our inpainting model (mask-gated architectures and precision sensitivity) — corrected: the real cause turned out to be a converter overflow bug, not the architecture, and the fp16 model now matches fp32 while downloading 52% smaller; see part 13.
Video inpainting performance — motion-adaptive frame skipping lets 65% of frames avoid inference entirely, plus the temporal EMA that keeps the result flicker-free (and the UX change that accidentally switched the whole optimization off).
Running a ~100–200MB model in a browser tab — tiles, memory walls, the watchdog/fallback structure, and the optimizations we benchmarked and killed.
Automatic watermark detection — a YOLO + OCR dual-engine detector and the glyph-level masks that raised precision without losing recall.

Part 1 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Can a General Real-ESRGAN Model Handle Anime? Our 45-Case Browser Baseline

Umasou — Wed, 28 Oct 2026 14:48:00 +0000

The answer is yes—with a qualification that matters more than the headline: ClearPix's current realesr-general-x4v3 pipeline produced usable anime enlargements, but it also softened faint lines, small accessories, and screentone. We are keeping that model for the first anime tool page. We are not calling it an anime-specific model, and we are not calling the output preservation-grade.

The suite contains 15 modern anime and illustration sources, each tested as clean, JPEG70, and JPEG40 input: 45 runs through the production ONNX Runtime Web pipeline. The average full-image PSNR was 23.581 dB and SSIM 0.867. Those numbers are useful for spotting regressions. They are not a quality verdict.

We had to fix the ruler first

Our first run produced impossible values, including NaN and extremely low PSNR on transparent JPEG cases. The model had not suddenly collapsed. The benchmark was wrong in two separate ways.

Some source dimensions were not divisible by four. Downsampling with rounded dimensions and upscaling by four produced a result one or two pixels larger than the reference, so pixels were compared at the wrong positions. The generator now makes every reference dimension a multiple of four before creating the input.

Transparent sources created a second problem. JPEG inputs were flattened on white, while the transparent reference still contained RGB values in invisible pixels. Comparing those RGB arrays directly measured a background-encoding difference rather than model quality. PSNR, SSIM, and high-frequency metrics now use images composited on white. Alpha is evaluated separately.

That alpha metric compares the output with a 4× resampling of the input alpha. Comparing a JPEG-flattened input with the original transparent file would be meaningless. After the corrections, all 24 cases with non-opaque source pixels report alpha MAE 0.

Alpha was a product bug, not a model choice

The shared upscaler wrote alpha 255 into every output pixel. This made transparent anime PNGs opaque regardless of which super-resolution model we chose.

We now reconstruct the source alpha at the output dimensions. Fully transparent pixels have their RGB normalized to zero, and the alpha-preservation path applies to both single-shot and tiled inference. This is a shared pipeline fix, so the existing 15-case general upscale suite and the media tools that use it still have to pass regression checks.

The important lesson is that “use an anime model” would have been a plausible but incomplete answer. File-format behavior can matter as much as model selection.

Where the general model worked

Flat colors, clean silhouettes, ordinary character illustrations, and moderately complex wallpapers remained usable across the three input variants. The fine freehand line art stayed coherent. The colored manga portrait retained its hair highlights and composition. The anime-expo wallpaper kept its mecha and background readable despite the expected softness.

The JPEG results also remind us not to read every higher number as “better.” Compression can remove some high-frequency noise before inference. A cleaner metric may coexist with a visible change in texture. The fixed crops remain the final check.

Where it did not meet a preservation claim

Five sources carry warnings in our review:

Screentone: the regular dot pattern visibly softens and changes. We cannot claim that dots are preserved.
Faint line art: pale lines and small details become weaker.
Detailed character with a camera: small accessories, hair, and camera details smear more than the large shapes.
Group wallpaper: faces and small costume details soften.
Pencil sketch: paper and pencil texture are amplified, which may be undesirable when a softer look is wanted.

The review was AI-assisted visual inspection of fixed crops, not owner-approved human QA. We record that distinction in the result files. Public review states should never be used to imply that a person has signed off when nobody has.

Why we are not adding a second model today

Real-ESRGAN distributes an anime-specific model. It may be a useful future experiment, but adding it would mean another download, another session lifecycle, another quantization check, and another browser memory budget. A model name is not enough to justify those costs.

The current model passed the “usable with disclosed boundaries” gate. The next anime page therefore uses the same general-x4v3 pipeline as the main image upscaler. It adds scene-specific guidance, evidence, and the alpha fix—not a fictional new engine.

If future cases show that the general model fails the tasks users actually care about, an anime-specific candidate will need to beat it on identical inputs, crops, sharpening, runtime, and download size. A win on one wallpaper would not be enough.

What we are publishing

The Quality dashboard now includes the anime suite, its methodology, real crops, metrics, review states, and per-case attribution. JSON and CSV downloads use the same published data.

The anime image upscaler uses that production path directly. It recommends ordinary anime wallpapers, illustrations, and avatars while warning against preservation expectations for screentone, faint lines, and tiny details. The companion test-set article explains the source licenses and coverage gaps.

That is the baseline: useful enough to ship honestly, not good enough to stop measuring.

Building a Licensed Anime Upscaling Test Set in Public

Umasou — Wed, 28 Oct 2026 00:02:40 +0000

Anime upscaling is often demonstrated with one dramatic wallpaper enlargement. That makes a good screenshot and a weak acceptance test. A model can sharpen a character outline beautifully while destroying the flat color beside it, turning a screentone pattern into mud, or painting a dark fringe around a transparent PNG. We wanted a test set that could catch those failures before we built an anime landing page.

So we started with sources and failure modes, not a new model. We collected 18 real, externally authored images with licenses that permit testing and publication. The first formal suite uses 15 modern anime and illustration sources across clean, JPEG70, and JPEG40 variants—45 deterministic runs. A cosplay photograph, a photo of a printed calendar, and a historic woodblock print stay out of the modern-anime aggregate. They are useful boundary material, but calling them modern anime would make the result easier to market and less honest.

What the set actually contains

The first two collection rounds produced a deliberately mixed set:

cel-shaded and flat-color characters, including low-resolution JPEGs;
fine and faint line art;
pencil and grayscale texture;
transparent PNGs;
detailed digital wallpapers;
painterly anime-style illustration;
a manga portrait in grayscale, screentone, and colored variants.

The three manga variants share one composition. That is intentional: they let us compare what the same pipeline does to ordinary grayscale shading, a regular dot pattern, and color highlights. They do not count as three independent topics in our coverage claims.

We also know what is still missing: licensed modern animation frames, more complex natural backgrounds, and explicit gradient banding cases. A test suite with a documented gap is more useful than a claim of universal coverage.

Licensing is part of the test fixture

Every source has an entry in our manifest with its author, source page, original file, downloaded version, dimensions, SHA-256, and license. The accepted licenses are public domain, CC0, CC BY, and CC BY-SA. Source review is not a legal warranty, but an image with no traceable origin does not belong in our public benchmark.

That metadata cannot collapse into one dataset-level license. Different cases carry different authors and terms. Share-alike derivatives need to remain under the same license when we publish them. Resized Wikimedia thumbnails are identified as thumbnails rather than described as original files, and the SVG manga variants are identified as raster renditions.

This is tedious work. It is also what allows the Quality page, downloadable JSON, CSV, and blog posts to show real inputs instead of mysteriously sourced anime art.

The degradation protocol

For the formal suite, each modern source is first resized to fit within 960 pixels on its longest side. That keeps the browser benchmark practical and gives us a consistent reference. We then create a 4× bicubic downsample, optionally encode it as JPEG at quality 70 or 40, decode it, and run the production upscaler back to 4×.

The clean and JPEG paths let us separate two different questions:

Can the model enlarge clean line art and flat color without inventing damage?
Does it treat compression artifacts as texture that should be amplified?

PSNR and SSIM compare the result with the resized source. They do not prove that an image is pleasant to look at. A model can score well and still break a line in a place a viewer notices immediately. That is why every case also has fixed crops and a human review state: pass, warn, fail, or unreviewed.

The 2× path is evaluated separately. ClearPix currently produces 2× by running 4× super-resolution and downsampling the result; a fast path that threw away that extra work has already failed our quality gate. We are not going to hide that trade-off in an anime-specific report.

The failure checklist

Our review is organized around the things anime viewers actually notice:

broken, thickened, or doubled line art;
bright or dark halos around clean edges;
synthetic texture in flat colors;
screentone that disappears, sticks together, or becomes a different pattern;
banding in gradients;
JPEG blocks and mosquito noise made more visible;
shifts in skin, hair, or highlight color;
transparent pixels turning opaque, or dark and light fringes around alpha edges.

During implementation we confirmed that the shared upscaler was writing alpha 255 for every output pixel. An anime model cannot fix that for us. Alpha now has to be reconstructed from the source at the output size and checked separately from RGB quality.

Why publish the set before claiming a model win

Our production model is realesr-general-x4v3, a compact general-purpose model. The Real-ESRGAN project also distributes an anime-specific model. The existence of that model is not evidence that we need it, and the word “anime” in a filename is not evidence that it will be better in a browser.

The general model gets the first run. If it handles line art, flat color, screentone, transparency, and compressed inputs well enough, we will keep the simpler pipeline. If it fails the core cases, any anime-specific alternative will have to beat it on the same inputs, crops, runtime, and download budget.

We are publishing the Quality data and real output crops so that decision can be inspected. The companion baseline report records what happened, including the cases that did not pass. The eventual anime image upscaler will use the same production path, not a separate demo pipeline.

That is the point of building the test in public: the page can describe the evidence, rather than asking the evidence to support the page.

Automatic Watermark Detection: Cutting False Positives on Complex Backgrounds by Design

Umasou — Tue, 20 Oct 2026 15:16:17 +0000

Our automatic watermark detector returned 125 text boxes on 24 real user photos, and every one of them was pre-selected. Ten were on a baby's face, a film still, a rock, a towel — never text at all. The other 39 we now drop are real text, but it sits on backgrounds an in-browser inpainter cannot rebuild without hallucinating. So we changed the rule: auto-detect now reports only what the model is genuinely good at — 76 boxes, no subject false positives, tiled watermarks untouched — and hands the rest to the user.

The numbers up front

Measurement	Before	After
Text boxes on a 24-photo real-user set	125	76
Boxes dropped by the new gate	—	49 (10 by shape, 39 by background)
Subject false positives we could verify by eye	10	0
Tiled-watermark boxes kept (case-1 / case-2)	38 / 22	36 / 19
Site sample ("SAMPLE ONLY")	detected	still detected
Quality bench (13 synthetic cases)	recall 89.2 %, precision 65.3 %, psnrRegion 24.8 dB	bit-identical

Nothing in that table required a new model, a bigger model, or a server. It required deciding which detections we are willing to act on automatically.

What actually goes wrong: the detector fires on subjects

Our text detector is PP-OCRv5's DB head — the same 4.7 MB model described in part 6. It is a shape detector. It answers "where could text be", not "is this a watermark", and it fires on any structure that looks like strokes: high-contrast, roughly periodic, thin.

Natural photographs are full of such structures. Eyebrows, eyes and lips. Fingers and knuckles. The crumb structure of a dessert. Fabric folds and hair. Window mullions, balcony railings, a crowd. None of them are text, and none of them look like text to a shape detector.

The uncomfortable part is that these false positives are not weak. The boxes sitting on a baby's face and on a film still scored 0.81–0.94 mean detection probability — squarely inside the range of real watermarks on the same set (0.74–0.995). Every confidence gate we already had (mean score ≥ 0.6, minimum area, fill ratio, area cap) was blind to this, because the detector was not confused. It was confidently reporting a face.

That is the general lesson: "how confident" and "should we act" are different questions. No confidence threshold can separate "real watermark" from "real text-like texture", because the second one scores just as high.

The second half of the problem: the inpainter cannot rebuild texture

Detection is the ceiling on removal, but only one half of it. Our inpainting is MI-GAN by default and LaMa as an opt-in quality tier, both running at 512² tiles on the user's own GPU (or WASM on the CPU). That architecture reconstructs a hole from its surrounding context. On a homogeneous background it matches the statistics and the result is invisible. On a busy background it produces texture that is statistically plausible and spatially wrong.

We have measured both ends of this. On a city-texture case from our synthetic bench, leaving a faint watermark alone scored MAE 3.11 against the clean original; repainting the rectangle with the inpainter scored 24.79 — eight times worse than doing nothing. On real user photos the same effect shows up as a car-shaped hallucination on sand, a horizon-shaped smear band across a cloud layer, and a rock-like blob where a ridgeline used to be.

So the two failure modes compound. A false-positive box on a complex background is the worst possible combination: the model is asked to invent texture, and it invents it over the user's actual subject. A false-positive box on a plain wall is merely ugly.

Why we could not just make the model stronger

The usual fix for a noisy detector is a better detector. That was not available to us, and the constraints are worth stating plainly, because they are the reason this post is about product rules rather than about model work.

No server-side fallback. The product's promise is that your photo never leaves the tab. We cannot send a hard case to a bigger model and get an answer back; that would be an upload, and it is the one thing we refuse to do.
No elasticity. A hosted detector can be re-trained, re-scaled or replaced with a larger model when a bad case appears. A local model in the user's browser is whatever we shipped: a 4.7 MB OCR model, an 11 MB YOLO model, and a 102.8 MiB fp16 LaMa tier behind an opt-in. Making the detector meaningfully better means new weights, new hosting, and another download for every user.
The device is the budget. Phones and low-end laptops run the same 512² tile pipeline; the model's receptive field is fixed by its architecture, not by the request.
We have no training set for "watermark vs. scene text". Separating an overlay from a printed label on a product needs semantics and supervised data we do not have, and we cannot collect it from user photos without asking people to upload their images.

Given those constraints, the only lever left is deciding which detections we act on automatically. If the model cannot tell an overlay from a face, we can at least tell a simple background from a busy one, and we can tell a text line from a blob.

The decision: play to the model's strengths, hand the rest to the user

The rule we shipped is deliberately conservative: auto-detect only reports obvious text on a locally simple background. Everything else is the user's job — and the user's path for that is not worse than before. Drawing a box runs the same region-constrained detector inside the selection plus shape refinement, so a user who wants a watermark on a busy background can still get it, with the mask snapped to the actual strokes.

This is a change of philosophy, and we want to be explicit about it. Our detection pipeline had been running on "a miss is recoverable, a false positive is not" (the rule we stated in part 12): if we miss a faint watermark, the user brushes over it and the result is fine; if we eat their photo, there is no undo for trust. This change tightens that into: in auto mode, zero false positives first; a miss is handed to the user.

The gate, in two criteria plus one exemption

The gate runs after the existing box-level gates (score, area, fill) and before shapes are assembled. It applies to the image tool's auto-detect path only — the quality bench, the video pipeline and the manual refine path are untouched.

Criterion 1 — the box must look like a text line. A text box must be elongated: aspect ratio ≥ 1.5 (horizontal) or ≤ 0.4 (vertical). Small boxes (short side < 40 px) are exempt, because single characters in a tiled watermark are naturally squarish and tiny. Near-square large boxes are the classic OCR-on-a-subject shape, and this is what killed the baby face (111×85), the film-still boxes (66×54, 133×107, 197×109), the plate of food (46×100) and the cosplay portrait (277×344).

Criterion 2 — the background around the box must be simple. We measure a ring band around each box (pad = 35 % of the short side, clamped to 8–64 px) and compute two statistics over it: the standard deviation of luminance, and the mean absolute gradient. A box is accepted if the ring's std ≤ 35 and gradient ≤ 14.

Two details make that measurement honest rather than decorative. First, only ring pixels count — the pixels inside the box are the watermark's own strokes, and letting them into the statistic would make every watermark look like it sits on a busy background. Second, the gradient is only taken between ring pixels; a stroke edge is not background complexity.

Calibration came from the 24-photo set. Where a real watermark sat on a plain wall, water or sky, the ring measured std 7–30 and gradient 3–10. Where a false positive sat on a face, an arm, food or a crowd, it measured std 32–89 and gradient 4–27. The thresholds sit in the gap.

The exemption — repetition is evidence. A group of four or more text boxes with matching sizes (within ±30 % in width and height) is treated as a tiled watermark, and tiled watermarks are exempt from both criteria. Repetition is itself strong evidence that you are looking at an overlay rather than picture content, and tiled watermarks are small per instance, so a single instance is a local hole the inpainter can genuinely fill — on a mountainside if it has to. Our two tiled cases kept 36 of 38 and 19 of 22 boxes; what was dropped there is the low-confidence edge fragments plus, in one case, the single large central title that sits on the ridge.

One footnote on method, because we had been burned by it before. An earlier round concluded that "ring complexity has no discriminative power" — that result was about a different question (should this region use a rectangle mask or a glyph mask?). The label space matters: the same measurement separates nothing when it picks a mask strategy, and separates well when it picks whether to auto-select a detection. We have written that distinction into our tuning notes so the two results are not quoted as one.

What it costs and what it buys

Eliminated, verified by eye on the output overlays: a baby's face and a pair of legs in a family photo, three boxes on a film still, a rock formation in a mountain photo, a towel and a pillow in two selfies, a face-shaped box on a cosplay portrait, a plate of food.

Left to the user, deliberately: watermarks that sit on busy backgrounds — a large "Luma AI" title over a sunset ridge, a property agent's phone number across a fridge, "LIONSGATE" over clouds, a hand-written title over a cloud bank, a tattoo artist's name along a hand, a wordmark across a model's hair — plus in-screen text that is caption rather than overlay (meme captions, shop signage, a social-app username bar). Those detections are real, and we are choosing not to auto-erase them, because on that background the artifact is the inpainter's output, not the removal. A user who wants them can draw one box and get the same glyph-level mask.

The rest of the pipeline did not move. The synthetic bench — 13 cases, on average recall 89.2 %, precision 65.3 %, region PSNR 24.8 dB — is bit-identical, because the gate is off in the bench path. (Two bench gates were already failing before this change: one case at 12.3 dB region PSNR and the faint tiled case at 47.9 % recall. Those are pre-existing and unrelated.) The manual refine path is unchanged, and the gate is a single option on the detector call, so it can be turned on or off per surface.

We did turn it on for the video watermark remover, where a false positive is worse than in a still image: the mask is applied to the whole clip, so a hallucinated patch is replayed frame after frame. The subtitle remover keeps full auto-detection on purpose — a subtitle is the thing the user asked to delete, it can sit on a face or a street scene, and filtering by background would defeat the tool.

Try it

The detector described here runs in the free in-browser watermark remover, together with the glyph-level masking from part 6 and the mask-discipline fixes from part 12. Nothing is uploaded: detection, mask building and inpainting all happen in the tab. If you want the other half of the pipeline — upscale a photo 4× without uploading it — it runs on the same local models.

Part 15 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor

Umasou — Sun, 18 Oct 2026 04:23:18 +0000

Our fp16 quantization post-mortem was wrong in the expensive direction: it told anyone with a mask-gated model to give up on half-precision, and that verdict would have kept a 52%-smaller download off the internet. The real cause was sixteen BatchNorm variance tensors larger than fp16's maximum representable value, silently clamped by the converter, leaving sixteen layers normalizing their output by the wrong divisor. The fix is two lines of algebra. The reward is a model that matches fp32 and downloads 52% smaller.

This is a correction post, so it comes with receipts: the lab notes for this round are checked in next to the code, every figure below is reproducible from them on a CPU-only ONNX Runtime install, and the whole investigation fits in a day or two of work.

The claim we published

Part 3 of this series ends with a case study called "When fp16 quantization quality loss is real: the LaMa failure". The measurements in it are real, and the failure is reproducible: rebuilding that conversion from the same weights today, the unfixed fp16 model averages 15.8 dB against fp32's 29.7 dB across our 14 acceptance cases, and its worst case scores 11.1 dB — the watermark surviving, exactly as described. Part 5 repeats it as "its catastrophic mirror image when we tried it on LaMa". Part 1's series index summarises the lesson as "mask-gated architectures and precision sensitivity".

We offered a mechanism too, and it was a good story. LaMa is mask-gated: the mask doesn't just mark where to work, it gates the network's behavior. A mask-gated inpainter therefore has a decision boundary inside it — which regions are trustworthy context, which must be regenerated — and half-precision noise pushes the model across that boundary. The gating fails softly, the model starts copying the watermarked pixels through instead of repairing them, and the output looks plausible except the watermark is still there.

That story explained the symptom exactly. It was consistent with everything else we'd learned (browser inference is bandwidth-bound, so moving half the bytes is free money). And it produced a tidy rule that we then repeated to readers:

whether quantization is safe is a property of the model's architecture, not of your tooling

The rule is wrong. Not "wrong with exceptions" — wrong as a predictor, and expensive: it would have kept a quality-identical model that is 48% smaller out of the product forever, and it tells every reader with an attention head, a gating mask, or a long residual chain to stop investigating. Part 8 of this series was about a quality bug that turned out to be ours, not the model's. This is the same shape of mistake, one layer lower in the stack: our conversion tooling, not the model's architecture.

Step one: prove that precision is not even the variable

"FP16" bundles two independent changes: mantissa precision (how finely you can split a value) and numeric range (the largest magnitude you can hold at all). If the failure were precision sensitivity, then rounding the mantissa while keeping fp32's range would degrade quality in proportion. So we built models that keep the graph and all arithmetic in fp32, and only round the weights' mantissas:

Weights rounded to	Region PSNR (5 cases)	Output fidelity vs fp32
fp32 (baseline)	28.88	—
7-bit mantissa (bf16)	28.83	66.9–76.9 dB
10-bit mantissa (= fp16)	28.88	74.5–96.6 dB
13-bit mantissa	28.88	95.9–109.0 dB

fp16 has a 10-bit mantissa. The 10-bit row is 0.004 dB off the fp32 baseline, and even throwing away 16 of the 23 mantissa bits (7-bit, coarser than bf16) costs 0.054 dB. LaMa is not precision-sensitive — it is precision-indifferent. Whatever fp16 was doing to it, it wasn't the mantissa.

That left range, and range has a hard, checkable boundary: fp16's largest finite value is 65,504. So we dumped every fp32 initializer in the graph and sorted by magnitude.

Step two: find the numbers that don't fit

The graph is 17,480 nodes with 606 float32 initializer tensors, 51M parameters, 204 MB of weights — 197 MB of it in the FFT path (17,022 of those nodes are Fourier convolutions, 97%). Run the check below over the 606 tensors and sixteen come back above 65,504, and every single one is a BatchNorm running_var:

     484322.56  model.generator.model.5.conv1.bn_l.running_var
     440853.66  model.generator.model.25.running_var
     240497.69  model.generator.model.22.conv1.bn_l.running_var
     218544.31  model.generator.model.11.conv1.bn_l.running_var
     174809.92  model.generator.model.6.conv1.bn_l.running_var
      65564.05  model.generator.model.10.conv1.bn_l.running_var
      ...  16 of 606 tensors, every one of them a running_var

Note the last row: the marginal tensor clears the ceiling by 0.09%, and it gets clamped to exactly the same 65,504 as the one that clears it by 7×. That's the whole hazard — a clamped value sits exactly at fp16's ceiling, which is a value fp16 can barely hold, so the next fp16 pass over that tensor takes it to infinity.

This is a five-line check, and it is the first thing to run before believing any "this model can't be quantized" conclusion:

import numpy as np, onnx
from onnx import numpy_helper

m = onnx.load("lama_fp32.onnx", load_external_data=False)
worst = sorted(
    ((float(np.abs(numpy_helper.to_array(t)).max()), t.name)
     for t in m.graph.initializer if t.data_type == 1),   # 1 = float32
    reverse=True,
)[:16]
for value, name in worst:
    print(f"{value:>14.2f}  {name}")

Sixteen tensors. Not sixteen layers of masking logic — sixteen numbers.

Why a BatchNorm variance is the thing that overflows

A BatchNorm layer computes y = (x − μ) / sqrt(var + ε) · γ + β. running_var is a scale quantity measured over the layer's input during training. If a layer's inputs live at large magnitudes, its variance lives at the square of those magnitudes, and 484,322 is entirely legitimate in fp32: sqrt(484322) ≈ 696.

Clamp that variance to fp16's ceiling of 65,504 and the divisor becomes sqrt(65504) ≈ 256. The layer now normalizes by 256 instead of 696 — its output is 2.7× too large — and the convolution that consumes it was trained against the correct scale. Sixteen of those, in series, and the network's internal magnitudes are destroyed while the graph still runs and still produces plausible-looking images.

That last part is the trap. There is no NaN, no warning, no crash: just a model that quietly keeps the watermark.

The clamp that never said a word

The conversion never told us anything, and this is the part worth internalising. onnxruntime.transformers.float16.convert_float_to_float16 clamps to max_finite_val=65504; onnxconverter_common.float16 defaults to 10000 (48× more aggressive). Both clamp silently — that's what "clamp" means: the fp16 model has a finite, wrong number in it, and nothing anywhere prints a line.

You can watch the corruption happen tensor by tensor, which is how we finally believed it:

Stage	`model.5.conv1.bn_l.running_var`
fp32 source	484322.5625
after `keep_io_types` fp16 conversion	65504.0 (clamped to the fp16 ceiling)
after a further fp16 storage pass	inf
after our fix (see below)	1892.0 (fits comfortably in fp16)

Note the third row: the clamped value is already at the ceiling, so a second conversion overflows it to infinity. Whether your broken model lands on 65504 or inf depends on which converter pass touched it last — which is also why the failure looked so erratic in our old notes.

The fix is algebra

We tried the blunt version first: keep the sixteen offending BatchNorm nodes in fp32 and convert everything else. That works, costs 34 KB of fp32 parameters, and produced identical quality — but it leaves a graph with fp32 islands and sixteen permanent assertions about which layers are dangerous.

The better fix is to notice that a BatchNorm's variance and its scale are only meaningful relative to each other:

y = (x − μ) / sqrt(var + ε) · γ + β
  ≡ (x − μ) / sqrt(var/K² + ε/K²) · (γ/K) + β

So pick a power-of-two K per layer, divide the variance by K², divide the scale by K, and divide epsilon by K² — the layer is mathematically identical, and now var/K² fits in fp16. With K = 2^ceil(log2(max|var| / 40000)), that's a twenty-line script touching 19 of the 75 BatchNorm layers:

k = float(2 ** np.ceil(np.log2(max_abs_var / 40000.0)))   # K = 16 for the worst layer
var_init.CopyFrom(numpy_helper.from_array((var / (k * k)).astype(np.float32), var_init.name))
scale_init.CopyFrom(numpy_helper.from_array((scale / k).astype(np.float32), scale_init.name))
# and the epsilon attribute: a.f = a.f / (k * k)

The result is worth stating precisely, because it kills the "mixed precision is inherently fuzzy" intuition: the converted model has 606 fp16 initializers and zero fp32 residue. It is 100% fp16, every tensor, with no casts added around the BatchNorms. And the arithmetic is exact — powers of two, so nothing is lost to rounding.

What this is worth — and what it is not

Three independent measurements on the same converted weights, plus a control and a harness sanity check:

Check	fp32	fp16 (fixed)
14-case masked-region PSNR	29.74 dB	29.73 dB
Per-case ghost-score drift (mean / max)	—	0.0014 / 0.005
Browser runtime (ORT Web WASM), same tile, vs fp32 reference	—	66.5 dB
Browser runtime (ORT Web WebGPU), same tile, vs WASM reference	—	71.6 dB, at 0.86 s vs 4.06 s per tile
Unfixed fp16, same test (control)	—	22.6 dB
16 real photo regions, production-detected boxes	—	59.9 dB mean, 40.3 dB min

The browser row is the one that matters for shipping: that's the exact runtime our production HQ tier uses, and 66.5 dB means the two models disagree at the level of sub-grey-step rounding. The control row is what the old mechanism should have looked like if it were real precision damage — 22.6 dB, i.e. visibly different. (Sanity check on the harness: browser fp32 vs a CPU fp32 reference on the same tile is 156 dB, so the comparison chain isn't the limiting factor.)

The size win is the point of the exercise:

	Raw	brotli q11 (what users download)
fp32	198.4 MiB	179.2 MiB
fp16 (fixed)	102.8 MiB	86.3 MiB (−52%)

The brotli pipeline that makes a 198 MiB weight file a 179 MiB download in the first place is part 9's story — worth saying here because the win compounds: same compressor, half the bytes.

And now the honest part, which is the real lesson for anyone shipping fp16 in a browser: on the WASM execution provider, fp16 made our inpainting slower, not faster. 11.4 s → 17.4 s per tile, 2.7 s → 9.4 s on CPU. The WASM EP has no native fp16 convolution kernels, so it inserts casts and computes in fp32 anyway, paying the copy tax for nothing. Part 3's 3.5× speedup came from WebGPU, which has real fp16 ALUs; on a WASM-only path, half-precision buys bytes and nothing else.

Update (2026-10-18): the WASM-only path was never a permanent constraint, and the reason we thought it was had nothing to do with quantization. The "JSEP gap" keeping HQ off WebGPU was a one-element off-by-one in the default onnxruntime-web build's negative-step Slice. Fixed by importing the WebGPU build, the same fp16 weights now run at 0.86 s per tile against WASM's 4.06 s (4.7×), at 71.6 dB agreement — so on desktop fp16 collects the speed win too, and on WASM it stays a pure download-size trade. See part 14.

The int8 half, where the conclusion survives and the reason does not

The old post also reported int8 failing (24.5 → 15.6 dB on our dark-text case) and concluded that quantization is architecture-hostile in general. We re-ran both int8 recipes on the current five-case subset (our case definitions have drifted since 2026-10, so compare within a run, not across posts):

Variant	Region PSNR	vs fp32
fp32	28.88	—
int8 dynamic (weights only, per-channel)	17.82	−11.1 dB
int8 QDQ (activations too), all 75 BatchNorms excluded	19.11	−9.8 dB

So int8 is still dead for this model — the conclusion stands. But the mechanism is the opposite of the one we published: this isn't a weight-precision problem, and it isn't the BatchNorm overflow either. Explicitly excluding every BatchNorm from quantization doesn't rescue it, because the inputs to those 75 layers still arrive through DequantizeLinear. Counting the graph: 432 Einsum/MatMul nodes with 648 inputs wrapped in quantize/dequantize pairs. The FFT path is expanded into spectral matrix products during export, and that's the part that cannot survive 8-bit activations. Different failure, different size of blast radius, and it stays on the do-not-retry list.

The corrected decision order

Our old order was architecture → bit width → runtime. The new one starts one step earlier, because a conversion artifact can impersonate an architectural verdict:

Audit the values before you audit the architecture. Dump max |value| for every initializer, compare against the target dtype's ceiling, and check the subnormal floor while you're there. In our model 16 of 606 tensors overflowed fp16 (0.087% of parameters sat below the subnormal floor — harmless, but the same scan finds both). This costs minutes and can invalidate a conclusion you'd otherwise keep for months.
Separate precision from range. Round the mantissa, keep the range. If quality doesn't move, precision is not your variable and no amount of "the model is sensitive" theorising will help.
Then test the structure. Gating, long accumulation chains, adversarial spectral paths — all real failure modes, but they're hypotheses to test, not a diagnosis.
Verify in the runtime you ship. A Python CPU pass proves the weights; it does not prove that ORT Web's WASM EP, with its cast insertion and kernel gaps, produces the same thing.
Benchmarks decide admission. Unchanged, and it's the rule that caught this: the broken model looked plausible at 15.8 dB, the fixed one measures 66.5 dB against the fp32 reference, and only the benchmark could tell you which was which.

Two corrections to our own record, then: the LaMa fp16 entry comes off the do-not-retry list and onto the opt-in path (fp16 verified against fp32 on every check above, fp32 spec retained as the fallback), the int8 entry stays, and the sentence in parts 1 and 5 that makes quantization safety a property of architecture gets retired in favour of something less quotable and more true: a quantization failure is a measurement, and most measurements have a checkable cause.

Try it

Everything here runs client-side with no uploads. The fp32 model that this post improves powers the free watermark remover, and the inpainting engine is the same one behind the video watermark remover. The fp16 path that started all of this is live in our image upscaler, and the quality dashboard publishes the bench numbers these gates are built on.

Part 13 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor

Umasou — Sat, 17 Oct 2026 01:10:37 +0000

The claim we published

whether quantization is safe is a property of the model's architecture, not of your tooling

Step one: prove that precision is not even the variable

Weights rounded to	Region PSNR (5 cases)	Output fidelity vs fp32
fp32 (baseline)	28.88	—
7-bit mantissa (bf16)	28.83	66.9–76.9 dB
10-bit mantissa (= fp16)	28.88	74.5–96.6 dB
13-bit mantissa	28.88	95.9–109.0 dB

That left range, and range has a hard, checkable boundary: fp16's largest finite value is 65,504. So we dumped every fp32 initializer in the graph and sorted by magnitude.

Step two: find the numbers that don't fit

     484322.56  model.generator.model.5.conv1.bn_l.running_var
     440853.66  model.generator.model.25.running_var
     240497.69  model.generator.model.22.conv1.bn_l.running_var
     218544.31  model.generator.model.11.conv1.bn_l.running_var
     174809.92  model.generator.model.6.conv1.bn_l.running_var
      65564.05  model.generator.model.10.conv1.bn_l.running_var
      ...  16 of 606 tensors, every one of them a running_var

This is a five-line check, and it is the first thing to run before believing any "this model can't be quantized" conclusion:

import numpy as np, onnx
from onnx import numpy_helper

m = onnx.load("lama_fp32.onnx", load_external_data=False)
worst = sorted(
    ((float(np.abs(numpy_helper.to_array(t)).max()), t.name)
     for t in m.graph.initializer if t.data_type == 1),   # 1 = float32
    reverse=True,
)[:16]
for value, name in worst:
    print(f"{value:>14.2f}  {name}")

Sixteen tensors. Not sixteen layers of masking logic — sixteen numbers.

Why a BatchNorm variance is the thing that overflows

That last part is the trap. There is no NaN, no warning, no crash: just a model that quietly keeps the watermark.

The clamp that never said a word

You can watch the corruption happen tensor by tensor, which is how we finally believed it:

Stage	`model.5.conv1.bn_l.running_var`
fp32 source	484322.5625
after `keep_io_types` fp16 conversion	65504.0 (clamped to the fp16 ceiling)
after a further fp16 storage pass	inf
after our fix (see below)	1892.0 (fits comfortably in fp16)

The fix is algebra

The better fix is to notice that a BatchNorm's variance and its scale are only meaningful relative to each other:

y = (x − μ) / sqrt(var + ε) · γ + β
  ≡ (x − μ) / sqrt(var/K² + ε/K²) · (γ/K) + β

k = float(2 ** np.ceil(np.log2(max_abs_var / 40000.0)))   # K = 16 for the worst layer
var_init.CopyFrom(numpy_helper.from_array((var / (k * k)).astype(np.float32), var_init.name))
scale_init.CopyFrom(numpy_helper.from_array((scale / k).astype(np.float32), scale_init.name))
# and the epsilon attribute: a.f = a.f / (k * k)

What this is worth — and what it is not

Three independent measurements on the same converted weights, plus a control and a harness sanity check:

Check	fp32	fp16 (fixed)
14-case masked-region PSNR	29.74 dB	29.73 dB
Per-case ghost-score drift (mean / max)	—	0.0014 / 0.005
Browser runtime (ORT Web WASM), same tile, vs fp32 reference	—	66.5 dB
Unfixed fp16, same test (control)	—	22.6 dB
16 real photo regions, production-detected boxes	—	59.9 dB mean, 40.3 dB min

The size win is the point of the exercise:

	Raw	brotli q11 (what users download)
fp32	198.4 MiB	179.2 MiB
fp16 (fixed)	102.8 MiB	86.3 MiB (−52%)

The brotli pipeline that makes a 198 MiB weight file a 179 MiB download in the first place is part 9's story — worth saying here because the win compounds: same compressor, half the bytes.

And now the honest part, which is the real lesson for anyone shipping fp16 in a browser: fp16 made our inpainting slower, not faster. 11.4 s → 17.4 s per tile on the WASM execution provider, 2.7 s → 9.4 s on CPU. The WASM EP has no native fp16 convolution kernels, so it inserts casts and computes in fp32 anyway, paying the copy tax for nothing. Part 3's 3.5× speedup came from WebGPU, which has real fp16 ALUs; on the WebGPU-disabled HQ path, half-precision buys bytes and nothing else. If a future ORT Web release ships fp16 kernels on WASM — or the JSEP gap that keeps HQ off WebGPU gets closed upstream — this model collects the speed win too. The fix is the precondition for that, not the payoff.

The int8 half, where the conclusion survives and the reason does not

Variant	Region PSNR	vs fp32
fp32	28.88	—
int8 dynamic (weights only, per-channel)	17.82	−11.1 dB
int8 QDQ (activations too), all 75 BatchNorms excluded	19.11	−9.8 dB

The corrected decision order

Our old order was architecture → bit width → runtime. The new one starts one step earlier, because a conversion artifact can impersonate an architectural verdict:

Audit the values before you audit the architecture. Dump max |value| for every initializer, compare against the target dtype's ceiling, and check the subnormal floor while you're there. In our model 16 of 606 tensors overflowed fp16 (0.087% of parameters sat below the subnormal floor — harmless, but the same scan finds both). This costs minutes and can invalidate a conclusion you'd otherwise keep for months.
Separate precision from range. Round the mantissa, keep the range. If quality doesn't move, precision is not your variable and no amount of "the model is sensitive" theorising will help.
Then test the structure. Gating, long accumulation chains, adversarial spectral paths — all real failure modes, but they're hypotheses to test, not a diagnosis.
Verify in the runtime you ship. A Python CPU pass proves the weights; it does not prove that ORT Web's WASM EP, with its cast insertion and kernel gaps, produces the same thing.
Benchmarks decide admission. Unchanged, and it's the rule that caught this: the broken model looked plausible at 15.8 dB, the fixed one measures 66.5 dB against the fp32 reference, and only the benchmark could tell you which was which.

Try it

Part 13 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Upscale Video to 4K Without Uploading It: Up to 7 Faster, Sharper, and Nothing Leaves Your Laptop

Umasou — Fri, 16 Oct 2026 17:14:01 +0000

Two things were true about our video upscaler when this round started: it was honest about being slow, and we had no idea whether it was any good. The first half is fixable on paper. The second half is why we spent a week building a ruler before touching a single line of the optimizer. What came out the other side: static shots now upscale 4.7–7× faster, talking-head footage — a static background with a moving subject, the case that used to get zero benefit — goes from a flat 1.00× to about 1.8×, and the output is measurably sharper than what we shipped last month. All of it inside a browser tab, with your video never leaving your machine. Not "we delete the file after processing": there is no upload path in the code.

We build ClearPix, the in-browser media toolbox from parts 1–10 of this series — no uploads, no server GPUs, Real-ESRGAN-class super-resolution running on WebGPU inside your browser. The video pipeline itself (hardware decode, per-frame AI, hardware encode, audio passthrough, O(1) memory) was part 2's story. This round was about turning that pipeline up to 4K and answering the two questions that actually matter for a video tool: how fast, and how good?

Act 1: Build the ruler before you touch the dial

We have a house rule by now, and we earned it the expensive way: no optimization without a per-stage timing breakdown, and no quality change without a metric that can veto it.

The video path had the first half. bench-video-upscale.mjs reports per-frame milliseconds across every output tier, with the decode/inference/encode/paste split, on real clips. What it did not have was a quality number. Its outputPsnr measures the deviation between "frame reuse on" and "frame reuse off" — a regression guard for a specific optimization, not a statement about how the picture compares to the truth. So "is our video upscaling any good?" was, scientifically speaking, a vibe.

The fix was a ground-truth bench (bench-video-quality.mjs). We take our natural test photos at 800×500 as ground truth, synthesize a low-resolution source by downsampling each one by a factor (f = 2, 4, 8), run the exact production geometry on it, bring the result back to ground-truth size, and score PSNR, SSIM, and high-frequency variance against the original. Same seed, same model, same tiles — only the geometry changes between rows.

One reading caveat, stated up front, because it nearly fooled us: Real-ESRGAN is a perceptual GAN. PSNR actively under-rates it against a dumb bicubic resize (at f = 4 on a clean source: 23.57 dB versus bicubic's 25.13 dB), so these numbers must never be used to argue "AI versus bicubic." They are only valid for model-against-itself comparisons — and there, they are exactly what we needed. Two independent metrics (PSNR and SSIM) agreeing in direction is our bar for believing a result.

The ruler immediately found something embarrassing. Our geometry was resizing the source down to a quarter of the target — 960px wide for a 4K output — and handing that to the model. Any real detail living between that quarter-target resolution and the source's actual resolution was being thrown away before the AI ever saw it. We had been carefully building a pipeline that discarded the user's own pixels.

Act 2: The dial nobody had turned

Feeding the model a bigger input means it does its ×4 on a larger frame, and we then scale the result back down to the target. That sounds like extra work for nothing — the model was already being asked to upscale. It is not nothing: at the larger input, the source's genuine detail is still present, the ×4 invents detail around it, and the average of that dense 4× field down to target size quietly smooths out the model's own artifacts.

We swept the input scale (m = 1.0 is what we shipped before) across all four images and all three downsampling factors — twelve rows, averaged:

AI input scale	Δ PSNR	Δ SSIM	Cost (per frame)
m = 0.5	−1.88 dB	−0.0452	0.26×
m = 1.0 (what we had)	baseline	baseline	1.00×
m = 1.5 (shipped)	+0.93 dB	+0.0118	2.01×
m = 2	+1.64 dB	+0.0186	3.58×
m = 3	+1.95 dB	+0.0159	8.18×
native source width	+1.12 dB	+0.0168	1.48×

Two rows killed their own ideas, and one needs a warning label. Start with the last row: "just feed the model the source at its own width" looks like a better trade than m = 1.5 — more gain for less cost — but it isn't a setting, it's a relationship. The input scale that equals your source's own width changes with every source-to-target ratio (2 at f = 2, 1 at f = 4, 0.5 at f = 8), and at the small end it degenerates into the m = 0.5 we are about to bury. We tested the intuition behind it directly — "stop pre-upsampling past the source" — and at f = 8 the geometry we ship still beat it, by 0.6–0.8 dB on three of the four test images (and gave back 0.3 dB on the fourth). Pre-upsampling is not a bug we forgot to remove.

The top row buries a piece of folklore we had been carrying around for months: "feed the model less resolution and let it work harder" costs 1.88 dB on average. That direction is dead, and we wrote it into the runbook so nobody re-proposes it in six months.

m = 1.5 was the row that shipped, for a boring reason: it was non-negative on every single one of the twelve rows — +1.88 dB at f = 2, +0.77 dB at f = 4, +0.14 dB at f = 8. An optimization that can't go backwards is an optimization you can hand to users as a choice. Note the cost column: it follows the area of the AI input (1.5² = 2.25 nominal, 2² = 4, 3² = 9) rather than the tile count, and every number in it is measured, not assumed.

So the option became a three-position dial. Fast is the old geometry. Balanced is m = 1.5. Best is m = 2, where the curve starts bending — another +0.7 dB for nearly double the time. We also stopped asking users to guess what their machine can afford. The default position comes from device capability plus the clip in front of you. No WebGPU — where the fallback WASM path costs roughly 25× per frame — or a low-end device, and you land on Fast automatically. Otherwise we estimate each tier's finish time from the file you actually dropped: its duration and frame budget, with the real fps probed when the estimate is close enough to matter. The most expensive tier that still fits its time budget wins. A six-second 720p clip on a laptop with a real GPU defaults to Best. A four-minute 4K export defaults to Fast, without anyone having to be told why, and the number next to the buttons updates the moment you change the target. It is not a marketing figure; it is arithmetic, and it happens to be right.

Act 3: Teaching the upscaler to skip the boring parts

Here is the uncomfortable shape of the speed problem: we profiled everything, and inference is 97% of per-frame time. Decoding is nearly free (the OS media engine does it on silicon). Encoding is nearly free. Pasting tiles to a canvas is noise. There is exactly one lever, and it is not "make inference faster" — it is "don't run inference". Video is full of frames that did not change.

We already had frame-level reuse for this: compare each frame against the last one we actually ran through the model, and if it hasn't moved, reuse the previous output. The implementation detail that cost us real time is worth repeating because it is counterintuitive: the comparison cannot be byte equality. Decoded H.264 static frames are not bit-identical — we measured a max delta around 9 and a mean of 0.02 between consecutive "identical" frames, because the encoder makes slightly different decisions per frame. Byte comparison scores 0% hits on video that is visually frozen. The fix is an anchored, bounded comparison: each frame is compared to the last frame that really went through inference (not to its predecessor, so deviation can't chain), and a hit reuses that anchor's output without moving the anchor. Bounded tolerance, no drift. This is the same discipline we landed on for video watermark removal in part 7.

The numbers on real clips at τ = 1.0:

Content	Hit rate	Speedup	Deviation
Static 1080p clip	88%	4.72–7.02×	46.9–50.3 dB
Slow pan (1px/frame)	50%	~1.97×	64.2 dB
Camera moving (`clean.mp4`)	0%	1.00×	none

And then the obvious follow-up question, which took us embarrassingly long to ask: what does real footage actually look like? Talking heads. Screen recordings with a cursor. Product close-ups. All of them are subject-motion clips — a static background with something moving in the middle — and every one of them gets a 0% hit rate at frame level. The most common shape of video on the internet was the shape our optimization couldn't touch. One moving person flips the entire frame back to full inference, forever.

So we pushed the same idea down a level. The frame is already processed in 256px tiles internally, so each tile now keeps its own anchor and gets its own bounded comparison. A tile that hasn't changed copies its core region straight from the previous output canvas — no format conversion, no inference, no paste — while the tiles that did change go through the model as usual. Anchors still don't update on a hit, so the drift bound is the same as the frame-level version.

Measured in isolation on synthetic clips (24 frames, 1080p tier = 4 tiles), the results split cleanly:

Content	Frame-level	Tile-level
Fully static	19.4×	9.4×
Static background + moving subject	1.00×	1.79× (63/96 tiles)
Whole-frame pan	1.00×	0.99×

The last row is the honesty check: when nothing is reusable, tile-level costs about 1% (one comparison per tile) and correctly returns nothing. The middle row is the whole point — the 1.79× comes with a deviation of 49.77 dB against no-reuse output, which is a fraction of one 8-bit level. Invisible. And since frame-level is strictly better when the whole frame is static, we ship them stacked: frame first, tiles as the fallback layer. Users never choose; the picture decides.

One trap almost shipped, and it's a good argument for building the bench before the feature. Pasting a tile back has to use the core region — the 8-pixel overlap around each tile exists only to give the model context, and must not be written to the canvas. putImageData's dirty rectangle is relative to the source ImageData, not to the destination, so the destination offset has to be the tile origin; pass the core origin instead and every tile lands 32 pixels off. Our PSNR gate caught the shift: 54 dB became 16.9 dB. Visually, "32px of double-exposed edge" reads as slightly soft — exactly the kind of thing that ships and gets discovered by a user a month later.

Why the privacy part is structural, not a policy

Everything above runs in your tab: hardware decode via WebCodecs, inference via ONNX Runtime Web on WebGPU, muxing via MediaBunny, hardware encode back to H.264. There is no server component of the video path at all. The only bytes that travel are the model weights coming down — once, then cached in IndexedDB, after which the tool works offline — plus a handful of cookieless analytics events: an event name and coarse properties like backend and elapsed seconds, the full list published on our privacy page. No media, no filenames, no durations tied to a person.

That is a stronger claim than "we don't store your videos," and it is checkable from the outside: open devtools, drop a clip in, watch the network tab. Models come down; nothing goes up. This is the part that gets described as a marketing position by people who haven't tried the alternative, and it genuinely costs us — a server GPU would make the long-clip problem evaporate, and a queue would let us serve 4K to a five-year-old phone. It would also mean your footage crossing the wire to be upscaled, and that is the one trade we don't make. If you're upscaling something you wouldn't email to a stranger — a family video, an unreleased product shot, a client's footage under NDA — the tool's whole job is running on the laptop you already trust.

Where the walls are

Per-frame cost on our reference machine (Apple GPU, WebGPU, moving content): 720p ≈ 115 ms, 1080p ≈ 263 ms, 2K ≈ 464 ms, 4K ≈ 1.06 s. Inference is 97% of each of those numbers, which is why this post is about doing less inference rather than doing inference faster.

We also removed the hard duration cap this round. There's now a size guard (200 MB) and a frame budget, an honest per-clip estimate shown up front, and if the estimate exceeds eight minutes we ask instead of refusing. Slow is a conversation now, not an error message.

Dead ends, recorded so we don't walk them again: multi-worker parallelism is a no-op (inference is memory-bandwidth-bound; more workers only fight over the bus), bigger tiles are a wash, IOBinding buys 1–2% so we skipped it, feeding the model less resolution costs 1.88 dB, and detail injection at k = 0.7 helps clean sources but re-amplifies compression noise on real ones, so it stays on the bench with no perceptual metric to justify it. One more for the file: if you benchmark two variants back to back, the first one silently absorbs the video encoder's cold start and looks 2.17× faster for free. Warm up first or you will ship the wrong winner.

Try it

No account, no upload, no watermark on the output: upscale a video to 4K in your browser — the tier dial and the estimate are under the target selector, and the frame budget will tell you the truth before you commit. The in-browser image upscaler shares the model and the tile engine, the free video watermark remover shares this exact pipeline, and if the transport layer is the part you're here for, part 2 is the read.

The nine seconds and the plastic skin were the same complaint. This round's version of that: "fast" and "sharp" and "private" were never three separate problems — they were one pipeline, observed from three angles.

Part 11 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Browser Image Upscaling: How We Got 4 Super-Resolution 14 Faster — Then Made It Slower on Purpose

Umasou — Thu, 15 Oct 2026 14:55:12 +0000

Two complaints arrived in the same week. The first was about time: upscaling a 2048px photo took nine seconds, even on the WebGPU path, and users on weaker hardware saw the progress bar crawl. The second was about taste: the results looked over-cooked — skin turned plastic, edges grew light halos, and if your source was a JPEG, the compression blocks came back louder than before. One complaint asked us to go faster; the other, if you squinted, asked us to add less. This is the story of how we chased the first one honestly, and why the fix for the second one ended up being a dial that gives some of that speed back.

We build ClearPix, the in-browser media toolbox from parts 1–9 of this series — no uploads, no server GPUs, Real-ESRGAN-class super-resolution running inside a browser tab. If you haven't followed along: our upscaler had already been through one generational swap (the 67MB RRDBNet flagship for a 4.9MB compact variant that ran 14–27× faster and won the quality shootout anyway) and one quantization gamble (fp16 weights, another 4.8× on top, told in part 3). What follows is the next chapter, where the easy wins were gone.

Act 1: Build the ruler before touching anything

We have a house rule by now, earned the expensive way: no optimization starts without a per-stage timing breakdown, and no quality change without a metric that can veto it. The upscale path had speed numbers but no quality numbers at all — "over-sharpened" was a vibe, not a measurement. So before touching a line of code we built the bench: 15 cases, five content types (people, structure, texture, sharp edges, flat gradients) crossed with three degradations (clean source, JPEG q70, JPEG q40), each scored for PSNR, SSIM, and a high-frequency ratio against the ground truth.

Only then did we learn what the second complaint actually was. Everyone, us included, had assumed "over-sharpened" meant edge halos — the classic ringing artifact. The bench said otherwise: on genuinely sharp content our output was actually too soft (hfRatio 0.29, meaning it had lost high frequency versus the original). The real villain was hiding in boring places: flat and soft regions, where the model hallucinates texture that was never there, and JPEG artifacts, which it amplifies faithfully. On flat gradients the hallucination metric climbed monotonically with source degradation — 3.47 on clean, 4.70 on JPEG q70, 5.55 on JPEG q40. Your compressed screenshot didn't just get upscaled; it got its blockiness upscaled too.

That number, 5.55, deserves a footnote because it nearly sent us chasing ghosts. High-frequency ratio divides by the original's high-frequency energy — and on a nearly flat original, that denominator is almost zero, so the ratio explodes regardless of absolute damage. We added an absolute metric (hfAbs) alongside the ratio and confirmed the flat-region hallucination is 0.12–0.16 intensity levels: real, but invisible. Half of "fixing" a problem is making sure you can see it accurately.

Act 2: The speed chase — three honest failures and one three-line win

With the bench as our safety net, we profiled. The per-stage breakdown held a surprise: model inference was 62% of wall time, sure — but pasting results back into the output canvas was 36%, and format conversion 1%. A third of our runtime was a CPU loop copying pixels, not AI at all.

That loop did per-channel Math.max(0, Math.min(255, value)) clamping while writing into a Uint8ClampedArray — which clamps on assignment by spec. Deleting the redundant math took three lines and bought 88ms → 68.4ms per image (−22%), with output that is bit-for-bit identical. It remains my favorite kind of optimization: zero risk, instant payback, embarrassing only in hindsight.

The natural next step was pipelining — hide the paste behind the next tile's inference, the classic overlap. We built it, verified bit-identical output, measured… nothing. Wall time didn't move. The reason was quietly funny: the pipeline was designed to hide a 36% bottleneck that the three-line fix had just reduced to 4%. We reverted the whole thing the same day and wrote the lesson into our runbook: re-profile after every fix; never parallelize against a profile that's already out of date.

Bigger tiles, then? 384px instead of 256 would mean fewer of them. On our Apple-silicon GPU it was a wash — 941/849ms against 888/825ms — the GPU was already saturated at 256. The nuclear option was skipping half the work: feed the model a half-resolution input and let ×2 cover the rest. It was glorious — 205ms down to 55ms per tile, a 3.8× speedup — and the bench killed it just as fast: photos lost 2.6–2.9dB, sharp content lost 6.5dB. The four-times cost is what quality-first upscaling is. We recorded the falsification prominently so nobody (especially future us) would re-propose it in six months.

A final systems check closed the case: every operator in the model runs natively on ORT's WebGPU backend with zero fallback warnings, and 115ms for a 256² tile is simply what this architecture costs on this hardware. Scheduling layer: squeezed dry. Model candidates: falsified. Runtime: current. The remaining 97% of inference time belongs to upstream — ORT's WebGPU evolution, or a hand-written native WebGPU super-res kernel, which is a project unto itself. We stopped, which is its own discipline.

Act 3: The honest dial

So we were as fast as honesty allowed, and still slightly over-cooked. The model's hunger for high frequency is baked into its training — SRVGGNet is a GAN-family network, and "adds crisp detail" is literally its job description. We couldn't retrain it. What we could do is scale what comes out.

The fix is almost comically small: a separable 2-pixel box filter applied to the model's output at final resolution, with a single gain factor k. k = 1 is the raw model. We swept k across the bench and the results settled a small argument in the team: k = 0.8 — giving back 20% of the output's high-frequency energy — improved PSNR on photos (+0.33dB on people, +0.44dB on texture, +0.27dB on edges) with SSIM flat-to-up. Read that again: the super-resolution model was adding high-frequency detail to photographs that the ground truth didn't have, and removing some of it made the output measurably more truthful. "Over-sharpened" wasn't user pickiness; it was a real, quantified bias. The default became k = 0.8, shipped with a 60–100% slider because taste is personal and JPEG-free line art genuinely likes 1.0.

Two adjacent ideas got proper burials. Rebuilding the output with matched film grain (to mask the flat-region hallucination) turned out to be unnecessary — the hallucination is k-immune but, per the absolute metric, invisible anyway. And Real-ESRGAN's official "denoise" model variants, which we hoped would be a gentler built-in preset, turned out to be detail-reconstruction strength knobs — the "denoise" setting adds more high frequency, not less. Both are archived in the bench directory with their falsification data, which is where good rejected ideas live.

One deliberate gap: the video upscaler shares this kernel but doesn't get the slider yet, because the sharpening helper costs 961ms at 4K — more than the inference itself. It needs a GPU-side implementation before it's worth anyone's frames.

Where the walls are

Today the pipeline is: clamp input, split into 256px tiles with 8px overlap, run the 2.4MB fp16 compact model on WebGPU, paste back only the centers so tile seams never show, then one honest pass of the sharpening dial. A 2048px photo costs about a second of GPU time on a modern laptop, and the output is — by our own bench, which exists precisely so we can say this — slightly closer to the truth than the raw model wanted to be.

The nine seconds became one, and the plastic skin became a preference slider. Both complaints, it turned out, were the same complaint: the model was working harder than the truth required. Speed and honesty weren't a trade-off — they were the same fix, applied at different layers.

If you want to see it yourself: clearpix.org/upscale-image runs the whole pipeline in your tab — the sharpen slider is under the result. The tile-and-overlap engine and memory walls behind the other half of this story are in part 5, the fp16 quantization adventure in part 3, and the delivery layer that gets the 2.4MB model into your browser in part 9.

Part 10 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Brotli for WASM and ONNX Models: Cutting First-Run Downloads From 72MB to 13MB, and the Compatibility Call We Made on Purpose

Umasou — Wed, 14 Oct 2026 15:42:48 +0000

We build ClearPix, a set of media cleanup tools that run entirely in the browser. That means every engine we ship — the ONNX Runtime WebGPU backend (a 27MB jsep.wasm plus a 14MB core), the ffmpeg core for video (30.7MB), and the model zoo behind our inpainting, upscale, and detection tools (up to a 198MB LaMa) — is downloaded into a stranger's browser tab before anything useful happens. The download path is the product experience. This post is about the delivery layer we ended up with: pre-compressed brotli objects on Cloudflare R2, why we compress WASM hard and ONNX barely, the progress-bar bug that fell out of it, and the compatibility decision we made consciously instead of by default.

What actually constrains model delivery

Three hard facts shaped everything:

Cloudflare Pages has a 25MB per-file limit. Our biggest files (27–198MB) can't even live in the deploy bundle. The build strips them, so a "mirror chain" isn't an optimization — it's the only way the product works at all.
Public mirrors are a fallback, not a source. We learned this in production on 2026-09-26: huggingface.co was unreachable from some networks, and hf-mirror's CORS headers were inconsistent (sometimes Access-Control-Allow-Origin, sometimes nothing), so model downloads simply failed for real users. Everything large now lives in an R2 bucket we control — 10GB free, zero egress fees, one controllable origin.
The fallback path gets real traffic. For the ORT runtime binaries we probe jsDelivr first (measured roughly 7× the throughput of our R2 custom domain) and fall back to R2 when the CDN is blocked — which happens in entire network regions. So the R2 path is not a theoretical safety net; for a slice of users it is the path. We measured our own access too: the default pub-*.r2.dev URL crawls at 75–110 KB/s, while a same-zone custom domain (static.clearpix.org) sustains ~2.3 MB/s — a 20–30× difference that has nothing to do with compression.

Brotli: compress at upload, not at request time

R2 doesn't transcode on the fly. You store bytes; it serves bytes. So we compress at upload time in the setup script, using nothing but Node's built-in zlib:

const br = brotliCompressSync(readFileSync(p), {
  params: { [BROTLI_PARAM_QUALITY]: 11 },
});
// wrangler r2 object put ... --content-encoding br --content-type application/wasm

The object key doesn't change, Content-Type stays application/wasm, and the browser transparently decodes Content-Encoding: br before the bytes ever reach ONNX Runtime or our model loader. Zero runtime changes — the URLs are identical, SHA-256 pins validate the decoded bytes, and IndexedDB caches the decoded bytes.

Measured ratios at quality 11, on our real files:

File	Raw	Brotli	Ratio
`ort-wasm-simd-threaded.jsep.wasm`	27.0 MB	3.8 MB	14%
`ort-wasm-simd-threaded.wasm`	14.2 MB	2.2 MB	16%
`ffmpeg-core.wasm`	30.7 MB	6.9 MB	22%
`migan.onnx` (inpainting)	28.2 MB	23.1 MB	82%
`yolo11_watermark_det.onnx`	10.6 MB	8.8 MB	84%
`lama_fp32.onnx`	198.4 MB	179.2 MB	90%
`real_esrgan_x4.onnx`	63.9 MB	58.6 MB	92%
`ppocr_v5_mobile_det.onnx`	4.5 MB	4.2 MB	93%

The pattern is physics, not tuning: WASM is code — structured, repetitive, full of zeros — and crushes to 14–22%. ONNX models are mostly float32 weights, which are close to entropy-maximal noise; brotli shaves 8–18% and that's the ceiling. (Worth knowing before you promise "50% smaller models" somewhere.) The full archive, all ten files, drops from 384MB to 293MB — but the number users feel is the first-run set: ORT WASM + ffmpeg went from 71.9MB to 13.0MB, an 82% cut on the critical path. Upload-side cost is a one-time ~10 minutes of q11 compression; quality 11 matters for a file served a million times, not for one.

The bug this introduced: progress bars that lie

Compression broke one thing, and it's worth calling out because it's subtle. When a response carries Content-Encoding: br, the Content-Length header is the compressed size, but a streaming reader in JavaScript receives decoded bytes. Our model loader's progress bar counted decoded bytes against the compressed total — so a 198MB model served as 179MB of brotli would blow straight past "100%" and keep climbing.

The fix is three lines: if Content-Encoding is present, ignore Content-Length and use the spec's known original size (expectedSize) as the denominator. Decoded bytes over original size — progress is honest again. Integrity checking was never affected: we hash the decoded bytes against pinned SHA-256s, so a compressed mirror and an uncompressed mirror are interchangeable.

The decision: br-only, no negotiation, on purpose

Here's the call we get asked about. An object uploaded with content-encoding: br is served that way to every request — R2 stores the encoded bytes and doesn't do content negotiation. A client that can't decode brotli gets undecodable bytes. We could have stored two variants (identity + .br) and put a ~50-line Worker in front of the custom domain to switch on Accept-Encoding, with Vary headers and doubled storage.

We didn't. The reasoning, in one table:

	Brotli support	Our runtime baseline
Chrome	50 (2016)	WebGPU since 113 (2023)
Firefox	44 (2016)	WebGPU since 141 (2025)
Safari	11 (2017)	WebGPU since 26 (2025)
Edge	15 (2017)	113 (2023)

A browser that can't decode brotli is from 2016 or earlier. A browser that can run ONNX Runtime Web with WASM SIMD and WebGPU is from 2023 or later. The intersection of "can't decode br" with "can boot our app" is empty — the fallback we'd be building serves users who are stopped two checkpoints earlier anyway. Building dual-variant + negotiation to protect a theoretical client is solving a problem we don't have, at the cost of a Worker in the request path and an ops surface we have to maintain forever. If that ever changes — say, a genuinely non-br embedded client appears — the dual-variant design is a clean add-on, and we wrote down exactly how we'd do it.

The honest caveat, for your own tooling rather than users: anything that fetches from R2 with plain curl needs brotli support (curl --compressed), or it saves compressed bytes to disk. Fine for scripts, worth documenting.

One thing we still don't love: ORT WASM has no persistent cache

Our ONNX models and ffmpeg core have a two-level cache: immutable HTTP headers plus IndexedDB with self-healing SHA-256 verification — download once, keep forever, survive releases. The ORT runtime WASM is different: ONNX Runtime fetches it internally, so we can't feed it from IndexedDB, and its only cache layer is the browser's HTTP cache. Mobile browsers evict that under storage pressure, and our release cadence happens to match the eviction cadence — which is why "every release makes users re-download the WASM" is an illusion we'd like to kill properly with a Service Worker and Cache Storage. That's the next post in this layer of the stack.

Try it

Everything described here runs in production: the dual-engine MI-GAN/LaMa watermark remover at clearpix.org/remove-watermark (the 198MB LaMa is the opt-in HQ engine — never prefetched), the Real-ESRGAN upscaler at clearpix.org/upscale-image, and the WebCodecs/ffmpeg pipelines that motivated the ffmpeg core over at clearpix.org/remove-video-watermark. Open devtools on any of them: the only bytes on the wire are the engines and models themselves.

Part 9 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.

Watermark Removal Artifacts: The "Processed Look" Came From Our Own Blend, Not the AI Model

Umasou — Mon, 12 Oct 2026 23:23:27 +0000

This is part 8 of the ClearPix engineering series, and it is the story of a bug that lived in the last 40 lines of the pipeline: the step that pastes the inpainted result back into the image. Users kept telling us some photos "look processed" — a soft, smeared halo around where the watermark used to be. We spent a while suspecting the model's fill quality. The model was mostly innocent. Our blending band was taking the model's full-crop reconstruction — which is not the original image outside the hole — and smearing it into pixels that had been perfectly fine. And the metric we used to accept changes was structurally incapable of seeing it, because it averaged the hole and its surroundings into one number.

If you take one thing home, take this: before optimizing a generative pipeline, split your quality metric by region — inside the mask, outside the mask — and treat your feathering band as a first-class suspect. The fixes that fell out of that one measurement took our restored-region PSNR up, our video end-to-end score up 5dB, and dropped the damage to clean pixels to exactly zero — while a plausible-sounding optimization we tried along the way got fully refuted and reverted in a single bench run.

Where watermark removal artifacts actually come from

Our image inpainting pipeline (part 5 covers the model stack) ends with a paste step. The algebra of the old implementation looks harmless:

out = original + a · (inpaintedCrop − original)      a = max(binary mask, blur(mask, 6–22px))

The trap is inside inpaintedCrop. MI-GAN (and LaMa) reconstruct the entire crop, not just the hole. Outside the mask, the model's output is a reconstruction of the original — close, but never identical (codec-level drift, color shift, softened texture). So wherever a > 0, we are mixing model error into pixels that were clean to begin with. And because a includes a Gaussian-blurred feather band that extends 6–22px beyond the mask, every watermark we removed came with a rim of synthetic blur.

Here is the part that made this bug invisible: for the thin-stroke text watermarks that users complain about most, that rim is enormous relative to the watermark itself. We measured area(dilate(mask, 12px)) / area(mask) across our 13-case golden set:

Case family	Ring band area vs. mask area
Small/faint text watermarks	2.8 – 6.5×
Large text	1.8 – 2.9×
Dark badges	1.2 – 2.0×
Solid logos	0.4 – 0.8×

For a small text watermark, the majority of "the area we touched" was never watermark at all — it was the transition band. That is the processed look, quantified.

It also retroactively explained a contradiction in our own tuning log: dilating masks by 2–4px had measured −0.5dB on our golden set, which we had recorded as "expansion is bad." The real mechanism was the opposite of what we concluded — dilation itself is fine, but every pixel you add to the mask becomes a pixel the blend can pollute. The same measurement that punished dilation would reward removing the pollution.

Metrics that cannot see direction

Our acceptance metric, psnrRegion, computed PSNR inside dilate(groundTruth, 12) — the hole and its ring averaged together. A pipeline that damages the ring and a pipeline that heals the hole can post the same number. We added four split metrics before changing any pipeline code (all pure observation — the 12 shared cases re-ran bit-identical):

PSNR-in / PSNR-out — restoration error inside the mask vs. in the outer ring, measured separately;
ringDelta — mean |result − input| in the outer ring; 0 means clean pixels were untouched;
colorShift — Lab ΔE between the filled hole and the surrounding ring;
SSIM (region) — because PSNR and human eyes disagree in both directions.

Plus a new golden-set case, halo-flat-text: thin-stroke text on a perfectly flat gradient with a tight ground-truth mask. On a flat background there is no texture to hide in — any out-of-band contamination shows up in the numbers with nothing to blame.

And running the "harmless" check first paid for itself immediately: the current snapshot scored 23.384dB against the archived 23.824 baseline. That looked like a regression — but tracing it landed on a commit from two days earlier where we had fixed real watermark bleed-through (the blend used to put model output at ~50% alpha at thin-stroke centers, leaking half the watermark back). The fix legitimately trades metric for correctness. Knowing that before touching anything is the difference between chasing ghosts and measuring progress.

The fix: stop blending model pixels into clean ones

The change (A4 in our plan) is small enough to state in one formula. The alpha stays max(soft mask, hard threshold) — inside the mask we keep 100% model output, which is non-negotiable (the earlier fix that killed bleed-through). Outside the mask, the artificial feather band is deleted; the only blending left is the mask's own upscale ramp, a pixel or two wide:

Same image, same detection masks, same model — only the paste step differs. In the before panel, look at the ledge line and the background around the removed text: a soft band of reconstructed non-information. In the after panel the background is bit-identical to the input wherever the mask wasn't.

The split metrics moved exactly the way the diagnosis said they should:

ringDelta: 5.63 average → 0.00, on every case. Not "less visible" — zero. Nothing outside the mask is touched anymore.
PSNR-out: 30.8 → 84.3 dB (most cases hit the 99dB ceiling, meaning bit-identical to the input).
The halo case: 37.0 → 49.9 dB region PSNR.
Overall restored-region PSNR on the 12 shared cases: 23.4 → 23.9 dB — modest, exactly as our P0 analysis predicted for cases where false-positive repainting inside the mask dominates.
Video end-to-end (which shares the paste code): 33.9 → 38.9 dB masked-region PSNR, a +5dB jump from deleting a blending band.

The optimization that lied to us: when reference statistics are wrong

With the composite layer fixed, the next target was inner color blocks — cases where the fill's average color doesn't match its surroundings (up to ΔE 14.4 on our set). The textbook move is statistical color matching: shift the model output toward the mean of the surrounding ring before pasting. It sounds airtight. It failed, and the failure is the most useful thing we learned this round.

We built a scatter plot of the two references — the color difference against the ring (what the alignment sees) versus the color difference against the clean ground truth (what actually matters):

The ring lies in both directions:

tiled-med scored ΔE 14.4 against its ring but only 0.3 against ground truth — the fill's color was essentially perfect; the watermark band simply sits across content that genuinely differs from its surroundings. Aligning it to the ring dragged a near-perfect fill away from the truth: inner PSNR fell 24.9 → 21.5.
logo-br was the opposite: a real color bias of 10.3 hidden behind a ring score of 1.3 — the ring-based alignment could never even see it.

At runtime you cannot distinguish "biased fill" from "content that legitimately differs from its surroundings." Our bench run of the alignment confirmed the split — some cases improved (text-bl's ΔE 11.9 → 2.6 for nearly free), the tiled cases degraded past our per-case gate. We reverted it the same day, per the rule we set in part 7: decide the stop line first, and treat a tripped gate as a bug report, not an opinion.

What the attempt left behind is permanent: a new metric, ΔE-true (fill vs. clean ground truth — computable offline where ground truth exists), which is now the acceptance basis for any future color-alignment work. Baseline: average 3.6, worst case 10.3. The residual color-block problem is real; it will need a mechanism that respects content boundaries (gradient-domain blending aligns boundary continuity rather than forcing hole statistics), not a stronger version of the same lie.

Snap the mask to the real edge

The last piece targets the upstream cause of ragged edges: glyph masks are born at OCR probability-map resolution — roughly 1/4 to 1/16 of the image — so after upscaling, the boundary position is inherently off by 4–16 pixels. We added a guided filter (He et al., 2010) inside the glyph-mask builder: the detection image, resampled to the probability grid, acts as the guide, and the mask's 0.5 level set snaps to the visible watermark edge instead of the flood fill's incidental boundary.

Zero plumbing changes — the product path and the bench path both consume the refined bitmap as-is, the op contract and coordinate contracts untouched, and the filter runs on the quarter-scale grid so detection time doesn't move. The perceptual metric moved most where the complaint lived:

SSIM: 0.739 → 0.753 average, with the largest gains exactly on the perceptually worst cases: text-br-large 0.708 → 0.770, text-huge 0.471 → 0.524, dark-text-br 0.577 → 0.615.

One honest caveat for anyone copying this: precision against our ground-truth masks dropped slightly (47.4 → 47.0), and that is not a quality signal — the ground truth is looser than the real edge, and a mask that hugs the real edge loses points against a loose GT while covering more of what actually matters. Trust the region PSNR and SSIM pair, and always eyeball the worst case.

The scorecard

Across the 13-case suite (12 shared + the new halo case), one week of composite-and-mask work, two shipped fixes and one refuted experiment:

Metric	Before	After
ringDelta (damage to clean pixels)	5.63 avg, >0 everywhere	0.00 everywhere
PSNR-out (outer ring)	30.8 dB	84.3 dB (bit-identical in practice)
Restored-region PSNR (12 shared)	23.4 dB	24.2 dB
SSIM (region)	0.710	0.753
Video e2e, masked region	33.9 dB	38.9 dB
Halo case (flat background)	37.0 dB	49.9 dB

No case regressed more than 1dB; all acceptance gates held.

Three rules we kept, one we learned

Localize damage before you localize the fix. A single mixed PSNR number hid a real bug for weeks and misexplained a tuning result for months. The in/out split cost one bench run to build and paid for itself on the first look.
The blending band is a first-class suspect. Generative pipelines end with a paste, and the paste is where "the model's version of your image" meets "your image." Any alpha > 0 outside the hole is a place the model can be wrong.
Validate your reference, not just your method. Statistical alignment toward a ring assumes the ring represents what the hole should look like. Sometimes it doesn't — and the failure is invisible to the very metric that motivated the method. Measure against ground truth when you have it.
Revert cheap, keep the instrument. The refuted alignment cost one bench run and left behind ΔE-true, which will judge every future attempt. That is a good trade.

All numbers in this post come from our public bench snapshots (run-2026-10-01*.json), the quality dashboard at clearpix.org/quality, and the commits they were measured on — the same gates that block our releases.

Part 8 of the ClearPix engineering series. Earlier parts: in-browser architecture (1), WebCodecs vs ffmpeg.wasm (2), fp16 quantization (3), video frame skipping + EMA (4), LaMa and Real-ESRGAN in the browser (5), automatic watermark detection (6), the anchor-based frame-skip redesign (7).

Video Inpainting Optimization: We Deleted the Safety Net and Got Faster, Better Output

Umasou — Mon, 12 Oct 2026 00:01:20 +0000

This is a decision log, not a mechanism tutorial — part 4 of this series covers how our frame-skipping gate and temporal EMA work internally. Here we cover the other half of the story: why we replaced a shipped, working safety mechanism with a simpler one, how we validated the change without betting the product on it, where the first run regressed hard enough to trip our preset abort criterion, and the two small fixes that turned the redesign into a 1.5dB quality win over the old baseline. If you take one thing home, take the discipline: decide your stop line before you run the experiment, and make sure your test set includes the worst case you have ever shipped a bug on.

How to speed up video inpainting that already works

Some context. Our in-browser video watermark remover inpaints frames one by one, but most frames near a static watermark barely change, so a motion gate lets us skip inference and reuse an earlier result. The first version of that gate (v1) compared each frame to its immediate neighbor and reused the previous output. That design had already failed once, badly: on footage with a slow global gradient, per-frame differences stay tiny forever, the gate skips forever, and the repaired region freezes into a visible rectangle. That incident cost us 22dB on our end-to-end masked-region PSNR (38.7 down to 16.7) and earned itself a dedicated regression asset: a synthetic clip whose background drifts continuously, built precisely so no future change could reintroduce chain-freezing unnoticed.

The fix we shipped was a cumulative drift budget plus a brightness-delta compensation: accumulate the error of skipped frames, force re-inference when the budget is spent, and nudge reused pixels by the scene's measured exposure shift. It worked. The frozen-rectangle bug stayed dead, quality held at 38.44dB against a 38.74 no-skip reference, and the safety net went to production.

So the starting position of this story is not a broken system. It is a working one with a layer of indirection in it.

The anchor idea: keyframe reuse in video processing

In a design review, someone asked the question that started everything: why does each frame compare itself to its neighbor at all?

Think about what the skip error actually is. When we skip a frame, we reuse the output of the last frame that really ran inference — call it the anchor. The error we are committing is exactly "how different is the current frame from the anchor." The drift budget only ever approximated that quantity by summing per-step differences and hoping the steps add up. Comparing against the anchor directly measures the thing itself: the skip error is bounded by the threshold, by definition, with no accumulator, no proxy, no edge cases about whether slow drift can sneak past.

The theory also predicted we would lose nothing on speed. Summing noisy per-step differences can only overcount relative to a direct comparison (oscillation and noise accumulate in the sum but cancel in the direct diff), so the skip rate should stay flat or improve. Fewer lines of state, a stronger guarantee, equal-or-better performance. Elegant ideas like this are exactly the ones that blow up in your face, so we wrote the rules down first.

PSNR regression testing with a preset stop line

Our rule for pipeline changes is that they pass the end-to-end quality gates before they ship, full stop. For this experiment we made the gate explicit before running anything: the redesign would be aborted — not patched, not negotiated — if masked-region PSNR on any e2e asset dropped below 36dB. That number was chosen in advance, while we were still neutral about the outcome. Deciding the abort criterion after seeing the results is how regressions get rationalized into production.

The gates themselves deserve a mention, because they are the second time this story's real hero shows up. The e2e suite includes that synthetic gradient-drift clip born from the 16.7dB chain-freeze incident — deliberately the worst case for any reuse-based optimization, with a background that never holds still but changes too slowly to trip naive motion detection — plus a four-segment subtitle asset where the masked content changes constantly. One asset attacks stale reuse, the other attacks insufficient reuse. Whatever we changed, one of them would notice.

First run: pass, pass, pass, stop

The anchored rewrite went in, and the first full run was a split decision:

Subtitle asset: all four segments improved by +3–4dB (37.3 / 32.3 / 28.7 / 30.1dB). The direct comparison was doing exactly what theory promised on fast-changing content.
Gradient-drift asset: masked-region PSNR fell from 38.30 to 34.69dB — below the 36dB stop line.

This is the moment the discipline is for. The aggregate picture looked fine — better, even — and it would have been easy to average the results, declare victory on subtitles, and tune later. We stopped instead, exactly as agreed, and treated 34.69 as a bug report the experiment had written for us. No rollback either: the stop line meant "stop and explain," not "revert on sight."

Error accumulation in video processing, round two

The root cause analysis turned up something genuinely new — a failure mode the chained design had been hiding from us.

The gradient clip's background does not drift uniformly. Its exposure shift varies with vertical position (the asset synthesizes it roughly as sin(y/60 + t·4) and cos(y/80 + t·3) components), while our delta compensation is a single per-channel mean over the region. A mean cannot cancel a gradient. Under the old chained design, reused pixels were always exactly one frame old, so the uncompensated residual stayed small and roughly constant. Under anchoring, a reused output can be two or three frames away from its anchor, and the residual grows with anchor age. The redesign had not created the compensation error; it had amplified it by extending reuse lifetimes — the same mechanism that made everything else better.

We also found a second, dumber bug in the same territory. The temporal EMA blends the current output against the previous frame's output, so its history term needs compensation against the previous frame. In the rewrite we had fed it the to-anchor delta instead — a systematic overshoot on every blended pixel, because the compensation was answering a different question than the blend was asking.

Two small fixes, no rollback

Both fixes were small and stayed inside the new design:

Threshold ceiling 8 → 5. The skip threshold is self-calibrating (median of the first five frames' differences, times three, clamped to [2, 5]/255). Lowering the ceiling forces re-anchoring more often, which directly bounds anchor age — attacking the residual-accumulation mechanism rather than its symptom.
Split the deltas. Compositing reuses the anchor, so it uses the to-anchor delta. The EMA blends against the previous frame, so it gets the to-neighbor delta. One line of reasoning each, no shared state pretending two timebases are one.

The calibration logic itself did not change at all — only the clamp ceiling and the delta plumbing.

Results: beating the baseline

The final anchored version, same gates:

Gradient-drift asset: 39.84dB masked-region PSNR — 1.5dB above the chained baseline, not merely recovered. With the delta semantics fixed, direct comparison and exact to-anchor compensation turned out to be strictly better than the approximation we had been defending.
Subtitles: the +3–4dB gains held, with 67% of frames skipping inference.
Speed: 58% of frames skipped on the watermark asset, flat versus the budget version as predicted, with the entire drift-budget code path deleted.

The safety net we removed was never missed. The one we kept — the preset stop line and the worst-case test asset — caught the one regression that mattered.

Takeaways

Write the abort criterion before the experiment. "Masked PSNR below 36 means stop" was decided while we were neutral. After seeing +3–4dB wins, that line is much harder to draw honestly.
Your test set should contain your worst historical bug, weaponized. The gradient-drift asset was built to catch chain-freezing two years of refactors ago; it has now caught a second, completely different failure. Worst-case assets appreciate in value.
Aggregates hide regressions. Subtitles up 3–4dB and gradient down 3.6dB averages to "improved." We read per-asset results first, means second, and the stop line was per-asset too.
Simpler mechanisms fail differently, not gently. Deleting the accumulator removed a whole class of proxy-error bugs and introduced none — but it amplified a compensation error the old design had been silently capping. Budget analysis time for the new failure modes of elegant ideas.
Fix forward when the root cause is small. Both fixes were a few lines inside the new design. Reverting would have thrown away a real quality win to avoid understanding a small bug.

Try it

The anchored skip gate ships in our free video watermark remover, and the same pipeline drives our subtitle removal tool — both run entirely in your browser, no uploads. If you have footage that breaks our gates, we genuinely want to hear about it; our best test assets were born that way.

Part 7 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.