I run AIFont, where we publish prompts for lettering styles and test each one on five image models: GPT Image 2.5, Nano Banana 2.1, Grok Imagine 2.0, Seedream 5.0 Pro and Qwen-Image 3.0 Pro. Every image goes into a ledger with one question answered: did the model write the word it was asked to write?
This post looks at two scripts from that ledger, Japanese and Arabic. They were checked differently, so I keep them apart.
- Japanese: 244 lettering styles, one short word each, the same prompt to all five models, first image only. 1,219 images. An AI reviewer judged all of them; a person (not a native reader) also checked 104.
- Arabic: 59 styles, first image only, 291 images. A native Arabic speaker read every one blind, without seeing the model name or our verdict.
Every prompt includes the line "write the text exactly as given, without adding or changing a single character."
Japanese: two models barely miss
| Model | Correct |
|---|---|
| GPT Image | 243 / 244 |
| Nano Banana | 239 / 244 |
| Grok Imagine | 226 / 244 |
| Seedream | 214 / 243 |
| Qwen-Image | 154 / 244 |
Of the 141 wrong images, 60 had a character replaced, 31 had extra text, 19 had one extra character, 16 gained or lost dakuten (the two small marks that turn は into ば), 14 dropped a character, and 4 were unreadable.
The dakuten errors are the easiest to miss. Grok Imagine wrote ばくはつ (explosion) as ばくはづ:
Seedream doubled a character: まほう (magic) became まほほう. At a glance it looks like a word.
Arabic: one dot is a different word
| Model | Correct |
|---|---|
| Nano Banana | 55 / 57 |
| GPT Image | 53 / 57 |
| Grok Imagine | 42 / 57 |
| Seedream | 34 / 54 |
| Qwen-Image | 8 / 59 |
(Images the native reader was unsure about are left out, which is why the totals differ.)
Arabic letters often share a shape and differ only in their dots. Six of Grok Imagine's 15 errors and seven of Seedream's 20 were a dot in the wrong place or missing. Sometimes the result is another real word. Qwen-Image was asked for فخر (pride) and wrote فجر (dawn):
Seedream turned هجوم (attack) into فجوم, which is not a word, and Grok Imagine turned ربيع (spring) into زبيغ.
The same failure in both scripts: the prompt ends up in the image
Qwen-Image adds text far more than any other model. Of its 90 wrong Japanese images, 26 had extra text; of its 51 wrong Arabic images, 21 did. When I read what it added, a lot of it came from our own prompt: the sentences describing the lettering style.
Asked for いちご (strawberry) in a cute rounded style, it also wrote ころんと, a word from the style description ("round and plump letters"):
For our italic calligraphy style, it did the same thing in both languages: instead of the one word, it wrote the style description out as a letter.
The "don't add a single character" line did not stop it. If you use Qwen-Image for text, zoom into the first image and look for words you didn't ask for, especially ones you recognise from your own prompt.
Why the Japanese numbers are not final
Before the native review, our Arabic verdicts came from an AI reviewer plus one of us, and none of us reads Arabic natively. The native reader disagreed with 23 of our 291 calls and settled 34 of the 41 images we had been unable to decide. Here is one pair: both images were asked to write شمس (sun). We passed the first and failed the second. The native reader called it the other way round.
The Japanese verdicts have not had that check yet, so I expect some of the 1,219 to move.
The data
Every image, prompt and verdict is in an open dataset under CC BY 4.0: aifont-text-rendering on GitHub (DOI 10.5281/zenodo.23815210). The comparison across seven languages is in this write-up.
If you read Japanese, Arabic or Chinese natively and have ten minutes, we are looking for native reviewers.
Top comments (0)