Build a Self-Hosted Translation API With TranslateGemma: Code, Costs, and Trade-offs

Every translation API has the same shape: you send your users' text to someone else's servers, and you pay per character, forever. That bill scales with your growth. And for some products (internal HR tools, medical intake forms, legal chat logs) shipping that text to a third party is a compliance conversation waiting to happen.

TranslateGemma is Google's answer to both problems. It is a family of open-weight translation models that runs on a laptop, speaks 55 benchmarked languages including Bengali, and charges you nothing per character. It launched in January 2026, and it is back on the Hacker News front page this week, which makes this a good moment to walk through what it actually takes to run it yourself.

One disclosure up front: the quality numbers below come from Google's technical report and launch blog post, not from my own benchmark runs. I have linked every claim. The code is adapted from Google's official model card.

What TranslateGemma actually is

Google distilled its Gemini models into three open-weight translators built on the Gemma 3 architecture, in 4B, 12B, and 27B parameter sizes. Per the launch post and the technical report, the training was two-stage:

• Supervised fine-tuning on parallel data mixing human-translated text with synthetic translations generated by Gemini models, for broad coverage including low-resource languages.
• Reinforcement learning using an ensemble of reward models, including MetricX-QE and AutoMQM, to push translations toward contextual accuracy rather than word-for-word fidelity.

The headline result from Google's evaluation on the WMT24++ benchmark, measured with MetricX:

• The 12B model outperforms the Gemma 3 27B baseline, meaning it beats a model twice its size while using less than half the parameters.
• The 4B model rivals the performance of the larger 12B baseline, which is why Google positions it for phones and edge devices.
• The 27B model is the quality ceiling, and it fits on a single H100 GPU or TPU.

The 55 languages are the rigorously benchmarked tier, evaluated through WMT24++. Bengali is in that tier (the codes bn and bn-IN are both supported). The chat template actually ships with roughly 160 languages, and press coverage of the release noted training on hundreds of additional language pairs beyond the benchmarked set, so the 55 number is a quality guarantee, not a hard wall.

There is one more capability worth knowing: the models kept Gemma 3's vision input. You can hand them a photo of a sign or a form and get the extracted, translated text out, with no separate OCR step.

Before you install: the license is not pure open source

TranslateGemma is distributed under the Gemma Terms of Use. As The Decoder put it, that allows commercial use, modification, and redistribution, but it is not an OSI-approved open-source license. You have to pass the terms on to users, and certain applications are prohibited under Google's Prohibited Use Policy.

Practically: you must accept the license on Hugging Face while logged in before the weights download. If your CI pulls models anonymously, the download will fail with a gated-repo error. Generate a token, accept the license once, and export HF_TOKEN.

Path 1: Quick test with transformers

This is the fastest way to see output on a machine with a GPU. The image-text-to-text pipeline handles both text and image inputs:

from transformers import pipeline
import torch

pipe = pipeline(
    "image-text-to-text",
    model="google/translategemma-4b-it",
    device="cuda",
    dtype=torch.bfloat16,
)

# ---- Text translation ----
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "source_lang_code": "en",
                "target_lang_code": "bn-IN",
                "text": "Your appointment is confirmed for Tuesday at 3 PM.",
            }
        ],
    }
]

output = pipe(text=messages, max_new_tokens=200)
print(output[0]["generated_text"][-1]["content"])

The message shape looks odd, and it is deliberately odd. TranslateGemma only supports user and assistant roles, the content list must contain exactly one entry, and that entry must carry source_lang_code and target_lang_code alongside the text. The codes take an ISO 639-1 language, optionally regionalized with an ISO 3166-1 country code: en, en-US, bn, bn-IN all work.

The strict part: if you pass an unsupported code, applying the template raises an error rather than guessing. That is annoying in development and genuinely useful in production, because your API layer can catch it and return a clean 400 instead of silently translating into the wrong language.

Path 2: Serve it as an API with vLLM

For real workloads you want a persistent server with batching, and vLLM gives you an OpenAI-compatible endpoint in two commands:

pip install vllm
vllm serve google/translategemma-4b-it

Then any HTTP client can hit it. Note that the request body still uses the model's own message schema, not a plain text prompt:

curl -X POST "http://localhost:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "google/translategemma-4b-it",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "source_lang_code": "en",
            "target_lang_code": "bn-IN",
            "text": "Payment received. Thank you."
          }
        ]
      }
    ]
  }'

SGLang and Docker Model Runner are supported the same way, and the model is deployable on Vertex AI if you want managed infrastructure instead. A small FastAPI wrapper in front of the vLLM server, one that maps {"text": "...", "to": "bn"} into the message schema and validates the language codes against the template, turns this into a drop-in internal translation service in about fifty lines.

The sleeper feature: translating text inside images

The vision input survived the fine-tune even though Google did not specifically train translation on images, which means you can feed a photo and get extracted, translated text back in one call. The message shape is identical to text translation, with a URL instead of a text field:

# ---- Image extraction and translation ----
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "source_lang_code": "ja",
                "target_lang_code": "en-US",
                "url": "https://example.com/sign-photo.jpg",
            }
        ],
    }
]

output = pipe(text=messages, max_new_tokens=200)
print(output[0]["generated_text"][-1]["content"])

Think about what this replaces: a traditional pipeline is an OCR engine, a layout parser, sentence segmentation, then a translation call, four stages that each fail differently. One call does not beat that stack on accuracy for dense documents, but for a user snapping a photo of a street sign, a medicine label, or a utility bill, it removes the entire integration. The image is normalized to 896x896 and tokenized to 256 tokens, so a full-page scan of small print will lose fidelity; crop to the region that matters before sending it.

Sizing it to your hardware

Rough memory guidance (estimates for bfloat16 weights, not Google's published numbers):

• 4B: about 8 GB of VRAM, comfortable on a consumer laptop GPU, usable on CPU if you can tolerate slow speeds. This is the model Google built for phones.
• 12B: about 24 GB at full precision. On a 24 GB unified-memory laptop (Apple Silicon class), quantized versions fit more comfortably. This is the sweet spot: it is the size that beat the 27B baseline in Google's evals.
• 27B: about 54 GB at full precision, realistically a single H100, TPUs, or aggressive quantization.

The total input context is 2K tokens, images are normalized to 896x896 and cost 256 tokens each. For your use case, that means long documents must be chunked client-side before they hit the model. Build the chunking into your wrapper from day one; it is the single most common gap between a demo and a working service.

Gotchas worth knowing before you commit

• It translates, it does not converse. The fine-tune specialized the model hard. General chat or summarizing in the target language is outside its design; pairing it with a general model for anything beyond translation is the safer architecture.
• Formality and dialect are a weak spot to test. Bengali, Japanese, and Korean all encode politeness levels that a 2K-token, sentence-scale model can flatten. This is exactly what your own evaluation set should probe before rollout.
• Numbers, names, and code need protection. As with any NMT system, product names, identifiers, and code snippets inside text can get mangled. Mask them before translation and restore after; it is a standard pre/post-processing step worth the wrapper code.
• The benchmarked tier is English-centric. WMT24++ evaluates English-to-X and X-to-English. Direct pairs between two non-English languages exist in the template but sit outside the strongest published guarantees; route through English in your wrapper if quality drops.

The decision matrix: when self-hosting wins

Before you rip out a cloud API, score your situation honestly:

• High, steady volume: Self-host. At roughly $20 per million characters (Google Cloud Translation's list price), a product translating user content continuously pays for a GPU in months. Occasional low-volume calls favor the cloud API.
• Sensitive text: Self-host. Medical, legal, HR, or internal text that should not leave your infrastructure has no substitute.
• Offline or edge deployment: Self-host. The 4B model runs on-device; no API call survives a dead connection.
• Peak quality on hard pairs: Cloud still likely wins. Google's report shows TranslateGemma beating its own baseline, but the full Gemini-tier systems remain ahead on the hardest low-resource pairs. Verify on your own data before committing.
• Zero ops capacity: Cloud. A hosted API never needs a CUDA driver updated at 2 AM.

My suggested starting point: run the 4B model locally for prototyping and smoke tests, evaluate the 12B against your cloud output on a few hundred real sentences from your domain, and only reach for the 27B if the 12B measurably falls short. Validation is cheap; a wrong choice discovered in production is not.

If you evaluate it, measure it

Google's numbers were produced with MetricX and WMT24++. You do not need their harness, but you do need something more honest than reading three outputs and nodding. Back-translate a sample (translate to the target language and back, compare meaning retention), or score with an open quality-estimation metric, and keep the test set fixed so future model upgrades are comparable.

---

I write about AI infrastructure, developer tools, and backend engineering every week. Subscribe, it is free.

Have you swapped a cloud translation API for a local model, or evaluated the swap? Which language pairs pushed you one way or the other?