DEV Community

Cover image for Image Translation with a Local LLM: 30 Seconds on My Laptop's CPU, 0.7 Seconds on a GPU
maruhana
maruhana

Posted on Originally published at zenn.dev AI-assisted

Image Translation with a Local LLM: 30 Seconds on My Laptop's CPU, 0.7 Seconds on a GPU

TranslateGemma 4B in Ollama, fed one new 1920x1080 image at a time: 29–34 s on a laptop CPU, about 0.7 s on an RTX Spark GPU. The image cost is a fixed 256 tokens either way; what changed is its price. The prompt-cache trap showed up on the GPU too.

This is an English adaptation of an article I originally published in Japanese on Zenn: ローカルLLMの画像翻訳30秒は、GPUで0.7秒になった. It includes the addendum I added to the original after publication. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.

Short version: sending a camera frame straight to TranslateGemma 4B took about 30 seconds on my laptop's CPU, so I had dropped the idea. On a machine with an NVIDIA GPU, the same measurement took about 0.7 seconds per new image. With only the GPU switched on and off on that same machine, the gap was about 25×.

A 67-second video summary (narration is synthetic):

TL;DR

  • Same script, same prompt, translategemma:4b in Ollama, one new 1920x1080 image per request: 0.665–0.698 s per image on an RTX Spark N1X, versus 29–34 s on my laptop's CPU (Core Ultra 7 258V) earlier.
  • The prompt was 326 tokens every time, image included. Gemma 3-family image input is a fixed 256 tokens. The structure didn't change; the price of that fixed cost did.
  • The image-encoding step that took about 23 s on the laptop CPU took at most about 0.5 s here.
  • The prompt-cache trap appeared on the GPU too: an exact duplicate of image 1 came back in 0.151 s, more than 4× faster than a new image.
  • With matched conditions (same machine, same Ollama, GPU on vs. off), the GPU accounted for about 25×. On my laptop, a newer Ollama (0.40.11) made the integrated GPU about 1.5× faster than the CPU, but still 16–20 s.

Scope

  • What: translating English text in an image with TranslateGemma 4B (translategemma:4b, Q4_K_M, 3.3GB download) via the Ollama API, one image per request.
  • On what: a Surface Laptop Ultra (Windows 11 Arm64, NVIDIA RTX Spark N1X, CPU and GPU share memory), and for comparison my laptop (Intel Core Ultra 7 258V).
  • Measured: round-trip time plus the server's own breakdown (prompt_eval_duration, eval_duration, prompt_eval_count) for 5 new images, 1 exact duplicate, 1 text-only request, and 1 warm-up. One run per condition; no spread.
  • Not measured: real camera frames on the GPU machine (synthetic images only), translation quality, larger models.
  • Versions are as recorded in November 2026.

Background: why 30 seconds

I was trying to translate English seen through the camera of my XR glasses (VITURE Luma Cyber). One approach was to give the camera frame directly to TranslateGemma 4B, with no OCR step.

On my laptop (Core Ultra 7 258V, Ollama on the CPU), a new image took 29–34 seconds. Almost all of that was input processing before generation, and about 23 seconds of it went to turning the image into tokens (the image encoder). At the time, with Ollama 0.32.6, enabling the integrated GPU made text input processing about 5.7× faster but left those 23 seconds almost unchanged. For glasses captions, where a few seconds is the most anyone will wait, 30 seconds was out, so I went with OCR plus text translation (about 4 seconds).

I had first measured this approach at 2.7 seconds. That number came from sending the same image twice: the second request hit Ollama's prompt cache. That trap was the subject of the previous article (in Japanese, linked at the end).

One question was left open: was the 23-second image encoder a cost of the runtime, or a limit of the hardware? I had kept the code for the dropped approach, with its prompts and measurement conditions, so that I could answer it on a different GPU. A Surface Laptop Ultra with an NVIDIA RTX Spark N1X gave me that chance.

Conditions

I applied what I had learned from the cache trap:

  • A new image every time. Five synthetic 1920x1080 images (black English text on white, the camera's resolution), each with different text. Plus one exact copy of image 1 to see the cache effect, and one separate image for warm-up.
  • Prompt, temperature (0) and keep_alive the same as in the app. One image per request to the Ollama API.
  • Not just a stopwatch round trip: I recorded the server's own breakdown (prompt_eval_duration, eval_duration and so on) every time.

On the Surface I installed the Windows ARM64 build of Ollama as is. The startup log showed it using CUDA, ollama ps showed "100% GPU" after loading the model, and nvidia-smi showed about 6,500MiB more in use. No x64 emulation and no custom build. Ollama's 0.32.3 release notes list CUDA support on Windows ARM64, and it worked as described.

Result: about 0.7 seconds per new image

Input Round trip Input processing (prompt_eval) Generation (eval)
Warm-up (includes model load) 3.69 s 0.77 s 0.07 s
New image 1 0.695 s 0.59 s 0.05 s
New image 2 0.665 s 0.54 s 0.05 s
New image 3 0.688 s 0.55 s 0.10 s
New image 4 0.686 s 0.55 s 0.10 s
New image 5 0.698 s 0.55 s 0.12 s
Exact copy of image 1 0.151 s 0.05 s 0.04 s
(Reference) text only 0.329 s 0.05 s 0.26 s

All five new images came back in 0.665–0.698 s. What took 29–34 s on the laptop CPU took about 0.7 s on this GPU. The translations made sense (for example, the Japanese equivalents of "emergency exit only" and "no smoking area"). I didn't keep the CPU-side translations, so I haven't compared the output.

The prompt token count (prompt_eval_count) was 326 every time, as before. Gemma 3-family image input is designed as a fixed 256 tokens (technical report), and this matches. The structure, a fixed cost per image, didn't change at all. What changed was the price of that fixed cost.

The open question

Input processing took 0.05 s for text only and 0.54–0.59 s with an image. The difference means that on this GPU, turning the image into tokens took at most about 0.5 s (an upper bound that also includes reading the image's 256 tokens; computed the same way as the previous article, it's about 0.24–0.29 s; see the notes). On the laptop CPU this step took about 23 s.

So, runtime or hardware? From this run alone I couldn't fully say. What I could say is that the image encoder itself is not a 23-second job. Same model, same fixed 256 tokens, and on a different machine and runtime it ran in at most about 0.5 s.

This run was not an A/B comparison on one machine: the machine, the OS architecture and the Ollama version all differed. That's why I measured again with matched conditions; see "Re-measured with matched conditions" below.

The measurement trap doesn't care how fast the machine is

This is the part I most wanted to write down. Only the exact copy of image 1 came back in 0.151 s. Input processing fell from 0.59 s to 0.05 s. That's the prompt cache.

The trap that made me record 2.7 s instead of 30 s on the CPU is not a quirk of slow hardware. It shows up in exactly the same form on a fast GPU. If I had measured by sending the same test image repeatedly, I would have recorded a 0.7-second job as 0.15 seconds, more than a 4× error. That's plenty to get an adoption decision wrong.

Making a new input for every measurement is tedious. But as the duplicate row in the table shows, skip that step and the number you record is a different number.

What 0.7 seconds means

For glasses captions, the bar was "a few seconds." 30 seconds was out of the question, and I fell back to OCR plus text translation (about 4 s). With a GPU like this one, translating the image directly, skipping the OCR step entirely, works in 0.7 s. Detecting, reading and translating the text all happen in one model call.

That said, the handheld PC I actually use has no NVIDIA GPU, so this is an option for people who have that kind of machine. My own setup stays OCR plus text translation. And faster doesn't mean better: the translations are still those of a 4B model, with the same stiffness and the same mistakes. Whether a larger model translates better is a separate question.

Re-measured with matched conditions

After I published the original, a reader commented on how much difference a graphics card makes. The comparison above couldn't separate the GPU from the runtime, because the machine, the OS and the Ollama version all differed. So I measured again with the conditions matched.

Same script, images and procedure as above (5 new images, 1 duplicate, 1 text-only request, 1 warm-up; temperature 0; translategemma:4b). One run per condition.

Condition New image Of which input processing Same image, 2nd time
Surface, GPU (CUDA, Ollama 0.40.10) 0.64–0.70 s 0.53–0.55 s 0.16 s
Surface, CPU only (same PC, same version) 16.4–17.5 s 16.1–17.3 s 0.37 s
My laptop, CPU (Ollama 0.40.11) 28.2–29.7 s 27.9–29.2 s 0.48 s
My laptop, integrated GPU (same version, Vulkan) 16.3–20.0 s 15.7–19.3 s 1.01 s

For the Surface CPU-only run I kept the same Ollama and set num_gpu: 0 on each request. ollama ps showed "100% GPU" for the GPU run and "100% CPU" for the CPU run.

  • On the same PC with the same Ollama, switching only the GPU made about a 25× difference. The GPU was the biggest reason for the gap.
  • Updating my laptop's Ollama to the newer version still gave 28–30 s on the CPU, about the same as the original 29–34 s. On the CPU, the version wasn't the difference.
  • CPU vs. CPU, the Surface (20 Arm cores) took about 17 s and my laptop about 29 s. The roughly 23 s image-encoding cost was the price of paying it on my laptop's CPU.
  • "Enabling the integrated GPU didn't shrink it" was true with the old version (0.32.6). With the current version, the integrated GPU took 16–20 s, about 1.5× faster than the CPU. Still far from the GPU's 0.7 s. I haven't looked into what changed between versions.
  • The second-request trap appeared in all four conditions.

Takeaways

  • Before concluding "this step is slow," separate a fixed cost of the design from the price of the environment you pay it in. Here the structure (a fixed 256 tokens) stayed the same, and the price went from about 23 s to at most about 0.5 s.
  • The prompt-cache trap doesn't depend on the environment. Measuring on a GPU doesn't let you skip "a new input every time."
  • If you drop an approach, keep its prompts and measurement conditions in code. When the environment changes, checking again is quick. Here, setup through measurement took less than an hour (the measurement itself took minutes).

Notes

  • One run per condition (1 warm-up + 5 new images + 1 duplicate + 1 text-only); no spread.
  • GPU side: Surface Laptop Ultra (Windows ARM64), NVIDIA RTX Spark N1X (Blackwell generation, CPU and GPU share memory), Ollama 0.40.10 for Windows ARM64. The startup log read "inference compute library=CUDA compute=12.1 driver=13.5". Model: Ollama's translategemma:4b (Q4_K_M, 3.3GB download).
  • The CPU-side numbers in the main text (29–34 s, about 23 s for the image encoder, 5.7× with the integrated GPU) are from the previous article, on a Core Ultra 7 258V with Ollama 0.32.6. Machine, OS and Ollama version all differ, so the main comparison is not same-condition A/B. The matched comparison is in the re-measurement section.
  • "Input processing" is prompt_eval_duration in the response, "generation" is eval_duration. Ollama's documentation defines the former as the time spent evaluating prompt tokens that were not cached.
  • Images are synthetic 1920x1080 (same as the camera), black English text on white. I didn't measure real camera frames here. Previously, real camera frames (about 30 s) and new synthetic images (29–34 s) took about the same time, but that wasn't a direct comparison, and there's no guarantee real frames run at the same speed in this environment.
  • "At most about 0.5 s to turn the image into tokens" is the input processing with an image (0.54–0.59 s) minus text-only input processing (0.05 s), so it includes reading the image's 256 tokens. The previous article's 23 s was computed differently: it converted 326 tokens at the per-token time of text and subtracted that. Using that same conversion (50.7 ms ÷ 54 × 326 ≈ 0.31 s) gives about 0.24–0.29 s here.

About this series

This is a side story in my Japanese-language series on real-time stereo 3D for XR glasses with AI depth estimation; it tested the camera of the same VITURE glasses. Related articles:

The original of this article: ローカルLLMの画像翻訳30秒は、GPUで0.7秒になった.

Top comments (0)