How Image Translation Actually Works: From Pixels to Translated Text

zeng — Sun, 20 Sep 2026 03:13:42 +0000

Translating text on a web page is easy. The browser already knows where the text is, what each sentence says, and how it should be displayed.

Images are different.

A screenshot, restaurant menu, product label, poster, or scanned document is just a collection of pixels. Before we can translate anything, we first need to figure out where the text is, recognize what it says, translate it, and somehow put the translated text back into the image.

That makes image translation an interesting combination of computer vision, OCR, machine translation, and image generation.

Step 1: Finding Text Inside the Image

The first problem is text detection.

An image may contain multiple text blocks with different fonts, sizes, colors, and orientations. Text can also appear on complex backgrounds, signs, packaging, screenshots, or photographs.

A typical OCR pipeline first detects regions that are likely to contain text.

Instead of seeing:

Welcome to Tokyo
Platform 3
Exit A

the system initially sees something closer to:

[x1, y1, x2, y2]
[x1, y1, x2, y2]
[x1, y1, x2, y2]

Each bounding box represents an area where text probably exists.

Step 2: Recognizing the Text

Once the text regions are detected, OCR converts the pixels inside those regions into characters.

For clean screenshots, this can work extremely well.

Real-world photos are harder.

Some common problems include:

blurry images

low contrast

decorative fonts

rotated text

shadows and reflections

curved product packaging

mixed languages

This is why image quality still matters even with modern AI models.

Step 3: Translating the Content

After OCR, the recognized text can be sent to a translation model.

At this stage, context becomes important.

Imagine translating a menu containing:

Cold Brew
House Special
Market Price

Translating each phrase independently may produce technically correct results, but the wording can feel unnatural.

Modern translation systems can use surrounding text and context to produce better results.

Language detection is also useful here. Instead of asking the user to manually select both the source and target language, the application can detect the source language automatically and only ask where the text should be translated.

Step 4: Putting the Translation Back Into the Image

This is probably the most interesting part.

Returning translated text as plain text is easy.

Producing a translated image is much harder.

The system needs to remove or cover the original text and place the translated version in approximately the same location while considering:

text size

line wrapping

alignment

background

spacing

surrounding visual elements

Different languages can also have dramatically different text lengths.

A short English phrase may become a much longer sentence in German, while several Chinese characters may translate into multiple English words.

The goal is therefore not pixel-perfect reconstruction. It is preserving enough of the original visual context that the translated image remains easy to understand.

Why This Is Useful

There are many situations where keeping the translation inside the image is more useful than extracting plain text.

For example:

translating restaurant menus while traveling

reading screenshots from international communities

understanding product labels

translating posters and notices

reading foreign-language study materials

translating UI screenshots

sharing translated images with other people

You can immediately see which translated sentence corresponds to which part of the original image.

I Built a Small Tool Around This Idea

I recently experimented with this workflow and built PictureTranslator.com.

The idea is intentionally focused: upload an image, choose a target language, and receive a translated version of the image.

It currently works with common formats such as JPG, PNG, and WebP, automatically detects the source language, and lets you compare the translated result with the original.

I wanted the interaction to stay lightweight because image translation is often something you need for a single screenshot, sign, menu, or document and then move on.

It has also been a fun example of how AI can turn several traditionally separate tasks—OCR, translation, layout understanding, and image editing—into one small web workflow.

If you're experimenting with OCR, multimodal AI, or translation tools, image translation is a surprisingly interesting problem to explore.

DEV Community: zeng

How Image Translation Actually Works: From Pixels to Translated Text