Vertical Japanese broke my OCR: how I group manga text into speech bubbles on iOS

Tedy Mezaguer — Wed, 11 Nov 2026 11:15:49 +0000

Point a general camera translator at a page of raw manga and you get one of two results. A single paragraph of English in an order that makes no sense, or one tiny box per column of text, each translated on its own, so nothing reads like a sentence.

I build Yomi, an iOS app for reading manga that only exists in Japanese, and most of the work turned out not to be translation at all. It was geometry: deciding which pieces of text belong to the same speech bubble, in what order to read them, and where to put the English afterwards. This post is about that part.

Why manga pages are hard for OCR

Three things a sign or a menu never does to you:

Vertical text. Japanese bubbles are usually written in columns, top to bottom, and the columns read right to left. OCR engines tend to return each column as its own line.
Many separate blocks. A page has eight to fifteen bubbles. A translator that treats the page as one block of text merges dialogue from different characters.
Small text at page scale. A whole page photographed by a phone leaves each glyph a few pixels wide. Recognition quality collapses on small bubbles and sound effects.

Step 1: three OCR passes, because none of them is right alone

On a page I run three passes concurrently:

async let cloudPass  = ocr.recognizeLines(imageData: imageData, candidates: candidates)
async let visionPass = VisionTextRecognizer.recognizeLines(in: cg, candidateLanguages: candidates)
async let detectPass = VisionTextRecognizer.detectTextRegions(in: cg)

The cloud OCR reads Japanese best, but its overlay geometry is sometimes shifted as a whole on real pages, so a box can land beside the text it read.
Apple Vision recognition gives extra boxes on device.
Vision text rectangle detection reads nothing at all, but its geometry is trustworthy. It anchors regions where text really sits.

None of these is the answer. Together they are a set of proposals.

Step 2: read each region again, bigger

Each proposed region is cropped from the full resolution page, upscaled, flattened, and read again. This single change did more for small bubbles and vertical columns than anything else: the OCR finally sees glyphs at a size it was trained on.

Then the results are merged. Refined lines win wherever a region produced text. Regions the refine pass could not read fall back to the page level lines, so recall never drops below the naive approach. One trap: the same text read twice, once by the refine pass and once through the shifted page level geometry, creates a phantom bubble at the wrong position. So a page level line whose text was already read elsewhere is dropped.

Step 3: grouping lines into bubbles with union find

Now there is a list of lines with rectangles. Two lines belong to the same bubble if they are close. The classic way to do that is to inflate every rectangle a little and union any two that intersect:

for i in 0..<lines.count {
    for j in (i + 1)..<lines.count where inflated[i].intersects(inflated[j]) {
        union(i, j)
    }
}

My first version inflated every line by a fraction of its own size, and vertical bubbles shattered: one "bubble" per column. A vertical column is tall and narrow, so a fraction of its width never bridges the gap to the next column, which is about one column wide.

The fix is to make inflation direction aware:

let inflated = lines.map { line in
    line.isVertical
        ? Geometry.inflated(line.rect, dx: max(insetX, 1.0), dy: min(insetY, 0.3))
        : Geometry.inflated(line.rect, dx: insetX, dy: insetY)
}

Vertical lines reach a full column width sideways and stay tight vertically, so two bubbles stacked above each other do not merge. Horizontal lines (Korean webtoons, Chinese manhua) keep the gentle default. A bubble counts as vertical when at least half of its lines are.

Two small filters made a surprising difference:

Lines that are only whitespace never join a bubble. An empty bubble downstream is an invitation for the translation step to invent something.
Tiny bubbles made only of digits or punctuation are dropped. They are page numbers. Real shouts like "18!!" are far larger than the cutoff.

Step 4: reading order

Inside a vertical bubble, columns are read right to left and each column top to bottom. Inside a horizontal one, top to bottom and left to right. Across the page, bubbles are sorted top to bottom, then right to left when they sit on roughly the same row. That last sort is mostly for a stable UI, so bubbles do not reshuffle when you rescan.

Step 5: putting English back on the page

This is where "translate in place" gets honest. English does not fit in the shape Japanese leaves behind.

For wide bubbles, the original text is erased and the translation typeset inside. The erase is not machine learning inpainting, it is a color matched fill. To match the paper, I sample the pixels around each block on a downscaled copy of the frame and split them into two luminance clusters: the dominant one is paper, the minority is ink. It costs under 5 ms for a handful of blocks, and it is why a patch on a yellowed scan does not glow white.

For tall, narrow bubbles, typesetting inside fails. A horizontal sentence in a box two glyphs wide wraps every word in half. So those bubbles get a detached card beside them:

static func isNarrow(rect: CGRect, isVertical: Bool) -> Bool {
    guard rect.width > 0 else { return false }
    return rect.height > rect.width * 1.2 || (isVertical && rect.height > rect.width * 0.85)
}

Both conditions require the bubble to be at least roughly as tall as it is wide, because the OCR's vertical flag alone was too noisy to decide the look. The card is at least half the page wide, picks the side with room (right, left, below, above), stays inside the page, and draws a short connector back to its bubble so the pairing is never ambiguous.

What still does not work

Sound effects. Hand drawn, stylised, often overlapping the art. Recognition is hit and miss, mine included.
It needs a connection. The best Japanese OCR I found is in the cloud, and so is the translation. On device Vision is a helper, not the reader.
Memory in extensions. The iOS Share Extension has a tight memory budget, and a full page erase gets it killed. The extension asks for text only and skips the cleaned page.

Try it

The app is Yomi, on iPhone and iPad, with a free trial and a small library of free, legal manga and comics you can test it on. If you are curious why I bothered, I measured how much manga never reaches English: about 86% of series from 2010 to 2023.

If you have fought vertical CJK OCR yourself, I would love to hear what worked for you.

DEV Community: Tedy Mezaguer