DEV Community: lxhsoluto-star

Erasing text from manga in the browser, without an inpainting model (and where that ran out)

lxhsoluto-star — Wed, 21 Oct 2026 13:55:00 +0000

imgtranslator.com translates the text in an image and redraws it in place: a menu, a product photo, a manga page. OCR and translation run on a server. Erasing the original text and typesetting the translation happen on a canvas in your browser.

That erasing step is the subject of this post. The obvious tool for it is an inpainting model, which I deliberately didn't use for the first two months. A generative fill can quietly change things that weren't text, like a price on a ticket or a face behind a speech bubble, and nothing in its output tells you it did. So I wrote the eraser by hand, as pixel rules.

What follows is how the second version of those rules worked, how I measured it when there's no ground truth for a real photo, and the point where more rules stopped helping, which I can show with numbers.

The input

For each line of text, OCR gives me a box. Sometimes it also gives per-character boxes, depending on the engine. The eraser gets the image and the boxes and has to make the text inside each box disappear, while leaving everything else alone.

"Everything else" is the hard part. In manga it includes the speech bubble's outline 2px from the text, the neighbouring column of text, screentone, and line art drawn right through the lettering.

Version 1: find the text colour, grow the strokes

The first eraser read the background from the strips above and below the box, decided the text colour was whatever contrasted most with that, seeded from pixels of that colour, and grew the strokes outward. Then it filled the strokes in from their surroundings.

On clean UI screenshots and product photos it was fine. Then I ran it over 166 real uploads (2,115 boxes) and sorted the failures:

68% were the growth flooding the window. Black text touches a bubble outline or line art, and growing from one floods into the other. A bubble's white gets read as the text colour.
45% of all boxes were vertical columns. In a vertical manga column, "above and below the box" is the artwork, so the text colour was wrong before anything started.

Neither was a tuning problem. The model of "text = the contrasting colour next to the background above and below" doesn't describe a manga page.

Version 2: what's inside the box that isn't around it?

The second eraser asks a different question. Text is whatever colour is common inside the box and rare just outside it. That's true for black on white, white with a black outline, coloured title lettering, and horizontal or vertical text. It needs no text colour up front.

Cluster the colours of the box plus a narrow ring around it (k-means, k=6, clusters closer than 22 RGB units merged). The ring goes all the way round, and other OCR boxes are excluded from it, so a neighbouring column doesn't count as "background".
A cluster is text when its share inside the box is at least 3× its share in the ring, and it covers at least 1.5% of the box. Anti-aliasing and outlines form their own clusters and are enriched too, so they come along automatically.
Mask the text-cluster pixels near the box. A connected run of text colour that's mostly outside the box and reaches the edge of the window is structure (a bubble outline, a panel border, line art) and is left alone.
Fill the mask by push-pull interpolation: build a pyramid that averages only known pixels on the way down, then fills holes from coarser levels on the way back up. It's smooth, fast, and has no opinions.
Judge the result the way a viewer would: compare edge energy inside the box after the erase with the edge energy of the background around it. When it fails, cover the box with a plain rounded patch and write the translation on that.

The second row in that figure is worth a look. On screentone, the dots are black and white too, so the letters' own colours are already common in the ring. Nothing reaches 3×, and the sound effect isn't erased as text. Keep that one in mind for the end of the post.

Two details that each took a failed version to learn:

The ring has to be narrow (0.3× the glyph size). A wide ring reached past a button into the page behind it. The button's own colour then looked far commoner inside the box than around it, and was erased as "text". On synthetic tests, panels came out 89–99% covered.
Black-and-white manga needs a second route. On a dense B/W page the anti-aliasing splits into four or five grey clusters, none enriched on its own, and the ring is 13–26% dark from other ink. So there's a separate "far enough from the background colour" route for dark ink, kept only as blobs that sit mostly inside the box. The two routes do connected components separately; merging them first threw lettering drawn on the art away as line art.

The result

Here are both versions on a panel I drew for this post, run with the production code and nothing changed. v1 leaves half the dialogue behind. Both of them report all five boxes as clean.

And on real uploads:

Real uploads, 166 images / 2,115 boxes	v1	v2
Boxes the eraser itself judged failed	358	96
Patches drawn	275	82
Images with any patch	89	48
Image area covered by patches	5.0%	2.1%
Time for the whole set	87 s	19 s

On 327 synthetic cases (real images as backgrounds, rendered text on top, so the answer is known), leftover original text went from 0.130 to 0.091, and damage to the surroundings from 0.137 to 0.067. Vertical text in speech bubbles went from 0.303 leftover to 0.

It lost in two places: horizontal CJK on busy backgrounds (0.079 → 0.082) and outlined horizontal text (0.201 → 0.241). I shipped it anyway, because the categories it won are most of the traffic.

The one-line bug that drew a dotted line in every bubble

51% of real boxes overlap another box, because in manga the gap between columns is narrower than the boxes OCR draws around them. Push-pull was using the neighbouring column's grey anti-aliasing as "known background", so every erased bubble had a faint dotted grey line where two columns met. The self-check called those boxes clean.

The fix was to stop treating ink inside other boxes as a fill source. The patch count didn't move (82 → 82), but 68 of 166 images changed. The synthetic tests couldn't see this bug at all. Only looking at real output did.

Measuring when there's no answer key

The table above uses the eraser's own verdict. That's a problem, because the verdict is part of what I was trying to improve.

For real images there's no ground truth. So I built a labeller. It crops each box before and after the erase and asks a headless claude -p run (Opus) to rate it as one of four labels:

clean
faint: visible only up close
trace: visible at normal viewing
damage: a smear, the wrong colour, or screentone wiped flat

It runs locally, on my machine, as an offline batch job.

The first thing it measured was v2 itself:

v2 on 166 real images	share of boxes
clean	58%
faint ghost	20%
visible leftover	9%
damaged	13%

About 400 boxes (19%) shipped with a problem while the eraser judged them clean. Of the 96 boxes it did patch, 36 had actually been erased fine. Its main self-check signal, post-erase edge energy, scored an AUC of 0.53 against the labels, which is a coin flip.

Then I measured the labeller. I labelled the same 296 crops twice:

The two runs agreed on the exact label 85% of the time, and on "bad vs. not bad" 93% of the time.
My original method compared versions on a weighted sample of boxes. With that much labeller noise, its margin of error was about ±0.017 in utility (clean = 1, faint = 0.6, everything else = 0). That was larger than most of the differences I'd been acting on.

The fix was to label only boxes whose output visibly changed between two versions, label both versions, and compare them as pairs. On around 400 changed boxes that brings the error down to about ±0.003. Several earlier "improvements" didn't survive the switch.

Four things that made it worse

With a ruler I trusted, I tried the obvious next steps. Baseline utility was 0.708.

Texture fill. After the smooth fill, copy high-frequency detail from a nearby clean region, so erased text on screentone isn't a smooth smudge. Result: 0.516. The "clean region" next to manga text usually has a bubble edge, another glyph or line art in it, and copying that added new strokes. A second version that also widened the mask on textured backgrounds scored 0.443.
Block matching (multi-scale PatchMatch) instead of push-pull. Of 375 changed boxes, 44 improved and 45 got worse, and it was 8× slower (0.13 s → 0.94 s per image). Where the mask had missed some ink, it found that ink and copied it in. So the fill wasn't the bottleneck.
A learned stroke mask. A small U-Net predicts the text pixels in a window, including outline and shadow. I trained it only on synthetic text over wallpapers and procedurally generated screentone and line art, never on user uploads. On synthetic validation, IoU went from 0.84 to 0.88. On real boxes, utility went down by 0.065 when it replaced v2's mask, and by 0.007 when the two masks were combined. It fixed about 60 of the 409 boxes v2 had ruined, and broke more clean ones than that. The higher its synthetic score, the worse it did on real pages. The gap is in the data, not the model.
Patch more aggressively. If the self-check is weak, just patch more boxes? Patching every box scored 0.643. Patches look worst exactly where erasing does too, on textured backgrounds: 50% of patches looked fine there versus 67% of erases. The best threshold-based rule I could fit scored 0.712, or 0.704 under 5-fold cross-validation. That's no improvement.

That's the evidence behind the conclusion I wrote in the project notes on 2026-09-27: hand-written rules have reached their end. Every remaining lever (flat backgrounds, the fill, the patch decision) measured at zero. The bad boxes left are large lettering on textured art, and they need either training data that looks like real pages or a fill that understands content.

What I did instead

For manga sound effects, I stopped erasing them. Splitting 65 vertical pages by background showed that dialogue inside bubbles erased cleanly. Almost all the bad boxes were sound effects drawn directly on the art. Scanlation groups solved this years ago: leave the original lettering and put a small translation next to it. On black-and-white pages that's now the default. On 84 lines from 150 held-out pages, it was good on about 67, neutral on 8 and bad on 9.

On colour pages the same rule mostly caught small bubbles and translucent ones (12 good, 21 bad), so it stays off there until I can detect bubbles reliably.

I tested inpainting properly, scoped to patched boxes only. LaMa (the Apache-2.0 ONNX export), with a DBNet text-detection mask:

Synthetic cases: leftover text went from 0.239 to 0.099 on the hardest categories (outlined, shadowed and Latin text), with damage from 0.101 to 0.022. It won 166 cases and lost 4.
Real boxes: about 15 of the 22 patched boxes I checked by eye came out better.
Speed: on a small CPU server it took 10.4 s per box (p50). Going from 1 to 2 threads didn't help at all, probably because the two vCPUs are hyperthreads of one core.

That rules out "click and wait". The next step is running it in the background for patched boxes only, and swapping the result in when it's ready. Boxes v2 already erases cleanly stay with v2, because on those LaMa was flat or occasionally worse.

What I'd keep from this

Sort failures by cause before tuning anything. "68% are flooding, 45% of boxes are vertical" told me v1's model was wrong, which no amount of threshold tuning would have shown.
Your self-check is a model too, and probably a bad one. Mine had an AUC of 0.53. Measure it before you build on it.
Measure the labeller's noise before reading its results. An LLM judge that agrees with itself 85% of the time can't tell apart versions that differ by 1%. Paired comparison on changed items is far cheaper than more labels.
Synthetic wins don't transfer by default. Two of my four failed ideas looked clearly better on synthetic data and clearly worse on real pages.
Sometimes the fix is in the product, not the algorithm. Keeping the original sound effects took one rule, and the result looks like what manga readers already expect.

The eraser is about 1,100 lines of TypeScript on Uint8ClampedArray, with no dependencies. If you're working on the same problem and want the numbers behind any row above, ask in the comments.

Why your generated Anki image occlusion cards import as plain cloze (and the one-line fix)

lxhsoluto-star — Sun, 18 Oct 2026 00:52:48 +0000

I've been building AnkiForge, a web app that turns lecture PDFs and labelled diagrams into Anki cards. For diagrams it makes image occlusion cards: every label on, say, a heart diagram gets hidden, one card per label.

Getting the masks in the right place took most of the work. Getting Anki to treat them as real image occlusion took one line of config, and finding out what that line actually does took reading Anki's source twice. Here it is, along with the parts that are still broken.

A note type named "Image Occlusion" isn't image occlusion

Anki has had image occlusion built in since 23.10. Before that it was an add-on (Image Occlusion Enhanced) with its own note type, and plenty of shared decks still use that old format. The Anki forums have a steady trickle of threads like "Anki deck with image occlusion shows no images" from people importing them.

If you generate decks in code, the obvious approach is to copy the templates and field names from the built-in IO note type. You end up with a cloze note type that happens to be called "Image Occlusion". The cards even review fine, because the template draws the masks itself.

The difference shows up when someone opens a card to fix a mask.This is how Anki desktop decides whether to show its mask editor (aqt/editor_legacy.py in the 26.9 package):

def current_notetype_is_image_occlusion(self) -> bool:
    if not self.note:
        return False

    return (
        self.note_type().get("originalStockKind", None)
        == StockNotetype.OriginalStockKind.ORIGINAL_STOCK_KIND_IMAGE_OCCLUSION
    )

Not the name, not the fields, not the template. One value in the note type's config, and ORIGINAL_STOCK_KIND_IMAGE_OCCLUSION is 6. Without it the image occlusion button stays hidden and the editor shows plain text fields, with the raw {{c1::image-occlusion:rect:...}} string where the masks should be.

In a schema 11 collection, note types live as JSON in the col table, so this is what mine looks like:

{
  name: 'Image Occlusion (AnkiForge)',
  type: 1,                // cloze
  originalStockKind: 6,   // the line that matters
  flds: [
    field('Occlusion',  0, { tag: 0, preventDeletion: true }),
    field('Image',      1, { tag: 1, preventDeletion: true }),
    field('Header',     2, { tag: 2, preventDeletion: true }),
    field('Back Extra', 3, { tag: 3, preventDeletion: true }),
    field('Comments',   4, { tag: 4, preventDeletion: false }),
  ],
  // templates copied from Anki's stock IO note type
}

The field tags and preventDeletion flags are copied from the stock note type too, so mine matches the built-in one field for field.

The occlusion field is a string, and I copied it exactly

Each mask is a cloze deletion with a shape inside it:

{{c1::image-occlusion:rect:left=.2325:top=.1104:width=.1371:height=.0412}}<br>

Coordinates are fractions of the image size. Anki formats them with a small function called floatToDisplay: four decimals, then strip leading and trailing zeros, so 0.2325 becomes .2325. "Hide all, guess one" mode adds :oi=1 to each shape.

I matched that output character for character, so a note from my export and the same note after a round trip through Anki's editor are identical. I haven't tested how forgiving Anki's parser is about other formats, and I'd rather not find out from a user.

Testing with Anki itself

I didn't want to check this by opening Anki and clicking around after every change. pip install anki gives you the same Rust backend the desktop app runs, so the test imports the generated .apkg into a fresh collection and asks Anki for the occlusions with get_image_occlusion_note, the call the mask editor uses to load a note.

While writing this post I removed the flag from a test deck to show the failure. The test still passed. The backend parses occlusions out of any note with the right field, flag or no flag:

>>> m['originalStockKind']
None
>>> col.get_image_occlusion_note(note_id=nid)
note: 3 occlusions, image 'af-test.png'

Only the desktop UI reads the flag, so a backend-only test can't see it missing. The test now checks it directly:

col = Collection(os.path.join(tmp, "collection.anki2"))
col.import_anki_package(ImportAnkiPackageRequest(
    package_path=path, options=ImportAnkiPackageOptions()))

for nt in col.models.all_names_and_ids():
    m = col.models.get(nt.id)
    nids = col.find_notes(f"mid:{nt.id}")
    if nids and m["flds"][0]["name"] == "Occlusion":
        assert m.get("originalStockKind") == 6, "editor won't treat this as IO"
        io = col.get_image_occlusion_note(note_id=nids[0])
        assert io.WhichOneof("value") == "note", io

The same script renders the cloze cards and checks that the question side hides the deletion. It runs on every change to the note types or the packer. Lesson for me: a test that passes before and after you break the thing isn't testing the thing.

Building the .apkg in the browser

An .apkg is a zip with a SQLite database (collection.anki2), a media file mapping "0", "1"... to real filenames, and the media files themselves, named 0, 1, ....

I build it in the browser with sql.js (SQLite compiled to WebAssembly) and fflate for the zip. sql.js is about 1 MB of wasm, so it's only loaded when someone clicks Download. The point is that the deck never exists on my server. Diagrams do go through the server to the OCR step, but they aren't stored, and the finished deck is assembled on the user's machine.

A few details that took more than one try:

Note GUIDs come from the content, using the same base91 scheme genanki uses. Re-exporting the same card gives the same GUID, so importing an updated deck updates notes instead of duplicating them.
Deck IDs come from the deck name, for the same reason.
Note type IDs are fixed and never change in place. If I change a template, it ships as a new note type with a new name and a new ID. Changing a published note type under the same ID means people who imported the old one get a conflict on their next import.

The same packer runs two free tools that don't call a server at all. The Anki Card CSS Styler lets you tune a card theme (colours, font, cloze highlight, night mode) against a live preview and downloads the note types as an .apkg. Since the stylesheet is part of the note type, its ID is a hash of the CSS, so a changed look ships as a new note type instead of overwriting one you already imported. The Atomic Cloze Formatter is for typing clozes by hand: select a phrase, press Alt+C, and it gets the next cloze number, or turn every bold word or number in a pasted block into its own blank.

Masks: OCR was the easy part

OCR returns lines of text with bounding boxes. Turning those lines into one mask per label is where the work was.

Labels on anatomy diagrams wrap across lines ("Superior / vena cava"), so you have to merge lines. But labels also stack in columns ("Nose / Nasal cavity / Pharynx"), and a column of short labels looks exactly like one wrapped label if all you have is geometry. Merge too little and a two-line label becomes two masks. Merge too much and one card shows five answers at once.

To measure it I built an eval set: diagrams from Wikimedia Commons with every label written down by hand. For each label the question is whether a student gets exactly one mask over it, switched on, without touching anything. I call that the one-click rate.

My first version merged by proximity and scored 64%. For comparison I ran NIOBIUM, a free CLI that does the same job with a 10 px proximity merge, through the same scorer. It got 67%. Both of us found about 93% of the labels. The whole gap was merging: 43 merged masks for NIOBIUM against 2 for my current version.

Here's one diagram from the set, same OCR engine, before and after:

![Respiratory system diagram with OCR masks. Left, merging by position only: 14 of 48 labels get their own mask, and whole columns like "Nose / Nasal cavity / Nasal conchae" sit under one red mask. Right, merging only when the text continues: 46 of 48 labels get their own mask.]

Diagram: "Respiratory system complete" by LadyofHats (Mariana Ruiz), public domain, via Wikimedia Commons.

What fixed it was making a merge depend on the text as well as the boxes. Two lines merge only if they line up and the text reads as a continuation: the next line starts lowercase or with (, the previous line ends on something like "of", or the next line is a single modifier word. Then the exceptions that the eval found:

subclavius n. above phrenic n. is two nerves, not one label. A line ending in a period ends the label, even if the next line starts lowercase.
musculo- above cutaneous n. is one word split by a hyphen and should join.
On lecture slides, the title, bullet points and page number are text too. They now get masks that are off by default.

On the 9 diagrams I tuned against (163 labels), the two OCR engines I use score 95% and 92%. On 7 diagrams I kept aside, the latest rule changes moved them from 91% to 98% and from 87% to 94%. I tuned those rule changes after looking at the failures on that held-out set, so it isn't really held out any more. The next round needs new images.

One thing I didn't expect: the newer document-parsing OCR models were much worse at this. Two of them scored 7% and 25%. They see an anatomy diagram as one figure block and don't read the text inside it. They're built to turn scanned papers into Markdown, and for that they're probably great.

The cloze side, briefly

The text-to-cloze part uses an LLM, and it got its own eval, which I wrote up on Hashnode. Two things worth knowing if you're doing something similar:

The model I use reasons before it answers, even at the lowest reasoning setting, and those tokens count against max_tokens. A sibling project had every response cut off at 400 tokens before anyone noticed. I now give it 16,000 and keep the reasoning out of the response.
Models love putting a list under one cloze number: {{c1::mitral}} and {{c1::tricuspid}}. That's a card asking you to recall a set, which is the thing spaced repetition is worst at. There's a prompt rule against it now, and the eval still catches it now and then.

What's still broken

Numbered diagrams. If the figure has "1, 2, 3" on it and a legend elsewhere, the mask hides "3" and the answer on the back is "3". The real answer is in the legend. I need to pair numbers with legend entries, and that isn't built yet.
Tiny old scans. A 409×500 plate from the 1918 Gray's Anatomy defeats every engine I've tried. Upscaling before OCR got one engine from 0 to 5 of 11 labels.
Phones. Anki's own library says the cards are image occlusion. I haven't yet opened them on a real phone in AnkiMobile or AnkiDroid.

If you're generating decks yourself and Anki's mask editor won't open your IO notes, check originalStockKind first. The tool is at ankiforge.app, and the card stylesheets I made along the way are MIT-licensed on GitHub.