I built a full image editor that never uploads your photo — here's how the on-device AI parts actually work

Most background-removal and object-removal tools online work the same way: you upload an image, a server runs a model on it, you download the result. Convenient, but your photo left your device and you have no idea what happened to that copy afterward.

I wanted to see how far I could push the "everything stays client-side" constraint for a full image editor — background removal, object removal, filters, format conversion, compression — using only what a browser can do natively: Canvas, WebAssembly, and an on-device ML model. This is APIC-Web, and here's what I learned building the two hardest parts.

1. Background removal: category masks vs. confidence masks

The obvious approach is MediaPipe's ImageSegmenter with outputCategoryMask: true — it hands you a hard 0/1 label per pixel (background or foreground). It works, but the output has a visible jagged, stair-stepped edge around hair and shoulders. There's no way to fix that after the fact with a blur, because the information needed to know how much a boundary pixel belongs to the foreground is already gone — it got rounded to a hard integer.

The fix is outputConfidenceMasks: true instead, which returns a float 0–1 probability per pixel before it gets thresholded. That single change unlocks a proper edge pipeline:

js
// 1. Smoothstep contrast — push clear foreground/background toward pure
// opaque/transparent while keeping the ~0.5 transition band soft
const s = v * v * (3 - 2 * v);

// 2. Erode ~1px inward — kills background-colour fringing around hair
// (a min-filter over a 3x3 neighbourhood)

// 3. Feather with a small separable box blur — turns the eroded edge
// into something smooth instead of jagged

Net effect: same underlying model, dramatically better-looking edges, because you're working with the mask's actual uncertainty instead of throwing it away too early.

One gotcha worth flagging for anyone doing the same thing: not every segmentation model outputs the same number of confidence channels. Some give you background, foreground, others give you a single foreground-probability channel. Don't hardcode confidenceMasks[1] — check the array length and handle both, and wrap model creation so a GPU delegate failure falls back to CPU instead of just dying.

1. Object removal without a generative model

Photo editors with "magic eraser" features (Photoshop's Generative Fill, Galaxy AI's object eraser) use trained diffusion models. That's not something you can ship for free in a static HTML page — no server, no GPU inference budget.

What is achievable client-side is exemplar-based texture synthesis — essentially the pre-AI generation of content-aware fill. The idea:

Take the painted (masked) region plus a padded area of real surrounding pixels
Break the hole into small blocks (8×8) and process them boundary-inward — the block with the most already-resolved neighbours goes first
For each block, search the surrounding valid texture for the patch that minimizes SSD against the parts of the target block that are already known
Copy those real pixels in — not an average, not a blur, an actual texture copy — which is what keeps the result sharp instead of smeared
js
for (let py = 0; py < bhei; py++) {
for (let px = 0; px < bwid; px++) {
const tIdx = (ty0 + py) * ww + (tx0 + px);
if (!validSrc[tIdx]) continue; // only compare known-good texture
const cIdx = (cy + py) * ww + (cx + px);
const dr = wd[tIdx*4] - wd[cIdx*4];
// ...accumulate squared error across candidate positions
}
}

The subtle bug I hit here: after background removal, transparent pixels still have leftover garbage RGB values sitting under alpha: 0 — they're just not painted, so a naive "is this pixel painted?" check treats them as valid source texture. The fix was splitting the concept into two separate arrays: isPainted (defines the hole) and validSrc (opaque and unpainted — the only pixels safe to copy from). Mixing those two concepts up is exactly how you get a random smear of background color pasted into your subject.

Performance-wise, brute-force patch search is O(hole_pixels × search_area), which gets ugly fast on a large image. Restricting the search to a padded bounding box around the mask (not the whole image) and adaptively adjusting the candidate sampling stride based on hole size keeps this bounded — typical fills run in well under a second.

Takeaways
If your segmentation output has visible jagged edges, you're probably throwing away confidence information too early — switch to soft masks and add erosion + feather.
You don't need a trained model to get decent object removal; exemplar-based patch matching gets you most of the way for everyday use cases (skin, sky, fabric, walls), it's just an older, well-understood technique.
When you're compositing filled/generated pixels back into a photo that's already been alpha-masked, be explicit about what counts as "safe to sample from" — transparency and "not edited" are not the same thing, and conflating them is a real bug, not a hypothetical one.

APIC-Web is live and free if you want to see the result:
https://akhouri-anmol-kumar.github.io/APIC-web/
Feedback — especially edge cases that break it — genuinely welcome.

APIC

"We Build What Others Forgot To Fix"

Comments

[Mustafa ERBAY]

Really nice work. What caught my attention most wasn’t that everything runs client-side, but how many small implementation details appear once you actually try to make that constraint work well.

The confidence-mask example is a great one. Keeping the model’s uncertainty until the final compositing stage instead of immediately collapsing it into a binary decision is such a small architectural choice, yet it completely changes what you can do with the edges afterward. The alpha: 0 / leftover RGB issue is another great example — exactly the kind of bug that looks obvious only after someone has spent hours finding it. 😄

I also like that you didn’t force a generative model into object removal just because that’s the fashionable solution. Exemplar-based filling has obvious limits, but for a browser-only tool, choosing an older deterministic technique that fits the constraints is good engineering.

One thing I’d be curious to test is how the patch matching behaves with repeating geometric patterns, text, or strong perspective lines. Those seem like the cases where local SSD similarity could produce convincing texture but break the structure of the image.

Very cool project — and keeping the whole image pipeline on-device makes the privacy claim meaningful rather than just a checkbox.