Reading raw Japanese manga should be seamless in 2026. Yet millions of international readers still hit the same wall: paste a comic panel into standard OCR tools or generic browser extensions, and you get fragmented nonsense covered by ugly white rectangular patches.
When building AI Manga Translator (https://ai-manga-translator.com), our biggest bottleneck was not prompting translation LLMs. It was the complete failure rate of conventional computer vision when confronted with tategaki (vertical text), non-convex speech bubbles, and half-tone screentones.
Here is an architectural breakdown of why generic OCR pipelines break on comics, and how we engineered an in-browser detection and inpainting pipeline using Next.js, SegFormer, and custom client-side heuristics.
The Geometry Breakdown: Horizontal Anchor Bias vs. Tategaki
Standard OCR engines (Tesseract, PaddleOCR, Apple Vision) are tuned on standard horizontal Latin and CJK documents such as invoices, receipts, and books. They rely on horizontal projection profiles to detect text baselines.
Manga breaks every baseline assumption:
• Top-to-Bottom, Right-to-Left (Tategaki): Columns read right-to-left, but glyphs flow top-to-bottom. Horizontal clustering models frequently link glyphs across adjacent columns, merging two distinct thoughts into garbled output.
• Furigana Collisions: Japanese manga frequently prints ruby text (small phonetic furigana) right beside complex kanji. Generic bounding boxes merge furigana into the main glyph, destroying character recognition accuracy.
• Complex Non-Convex Polygons: Dialogue balloons vary from ellipses and rounded rectangles to spiked shock-bursts. Applying axis-aligned bounding boxes (AABB) captures halftone background textures, speed lines, and character hair, triggering severe false positives.
To solve the column-ordering failure mode, we implemented a custom heuristic to cluster vertical text boxes along their true semantic flow:
interface DetectedBox {
x: number;
y: number;
width: number;
height: number;
text: string;
}
export function clusterVerticalColumns(boxes: DetectedBox[], lineThreshold = 18): DetectedBox[][] {
const sorted = [...boxes].sort((a, b) => b.x - a.x);
const columns: DetectedBox[][] = [];
for (const box of sorted) {
const matchedCol = columns.find(col =>
Math.abs(col[col.length - 1].x - box.x) < lineThreshold
);
if (matchedCol) {
matchedCol.push(box);
matchedCol.sort((a, b) => a.y - b.y);
} else {
columns.push([box]);
}
}
return columns;
}
Instead of rigid bounding boxes, we deploy a comic-fine-tuned semantic segmentation backbone based on SegFormer. It outputs pixel-level probability masks separating speech bubble contours, in-bubble vertical text strokes, and sound effects (SFX) alongside ambient text overlays.
Replacing White Rectangles with Inpainting (LaMa Architecture)
The most notorious flaw of browser translation extensions is the "white patch." Standard tools draw an opaque rectangular SVG or div over the bubble, completely destroying hand-drawn screentones, line art, and atmospheric dark backgrounds.
To achieve clean scans, we treat text replacement as an image restoration problem across three explicit passes:
• Stroke Dilation Masking: Instead of blanking the whole bubble, our model generates a tight mask covering only the black ink strokes of the Japanese characters, expanded by a 2px morphological dilation to catch anti-aliasing artifacts.
• Fast Inpainting Pass: We run an optimized resolution-robust inpainting model based on Large Mask Inpainting (LaMa with fast FFT convolutions). It predicts underlying textures, such as halftone screentones or paper grain, without washing out adjacent bubble boundaries.
• Adaptive Canvas Typesetting: Using the HTML5 Canvas API on the client side, the translated target text is dynamically wrapped into the bubble's bounding polygon, automatically balancing font sizing, line height, and hyphenation to prevent text overflowing the bubble boundary.
Edge Pipeline & Performance: Beating the GPU Latency Wall
A 25-page chapter can easily choke backend inference if you stream raw 4K scans. We built an edge-optimized workflow inside our Next.js architecture to keep processing times under sub-second thresholds:
• Client-Side Offloading: For the AI Manga Translator Chrome Extension (https://ai-manga-translator.com/extension), the browser inspects DOM/Canvas elements directly on sites like Pixiv, BookWalker, and DLsite, extracting only dynamic canvas viewports rather than dumping massive full-page payloads.
• Concurrent Chunk Dispatching: Extracted regions are processed via asynchronous micro-batches. While bubble #1 is getting inpainted, bubble #2 is already streaming translation from our fine-tuned comic LLM endpoint.
• SSR Fallback with Next.js 14: On the web platform, we use Next.js server actions to handle parallel file streaming, allowing users to drop raw CBZ, ZIP, or PDF files and immediately inspect real-time progress via Server-Sent Events (SSE).
The Architecture in Action
Eliminating the friction between reading and understanding shouldn't mean turning art into flat, robotic documents.
By treating comic pages as multi-layered graphic compositions rather than flat text files, we preserve the illustrator's line work while rendering native-feeling translations in real time.
If you are curious to see how the segmentation and inpainting hold up on complex raw panels, you can test AI Manga Translator (https://ai-manga-translator.com) directly on the web with 5 free pages—no account sign-up required.
What techniques are you using to handle non-standard text directions in your vision pipelines? Drop your thoughts and benchmarks below.
Top comments (0)