DEV Community: jack

I checked 2,016 AI images letter by letter. Here's how I kept the test fair

jack — Fri, 30 Oct 2026 21:53:44 +0000

GPT Image printed VENON on a death metal T-shirt. The prompt said VENOM. The last M had turned into an N, hidden in the thorns.

Our first check missed it. It was read off a contact sheet at thumbnail size and marked correct. A zoomed re-read caught it later. That one image shaped most of how we run this test, so this post is about the method, not the leaderboard.

Some context. My team runs AIFont, a site that publishes text-effect prompts for image models. Every prompt goes to five models in seven languages, and every image gets checked against the word it was supposed to write. After 59 styles that came to 2,057 first-attempt images, 2,016 of which we could score.

The results, briefly

Correct out of judged, first attempt only:

Script	GPT Image 2.5	Nano Banana 2.1	Grok Imagine 2.0	Seedream 5.0 Pro	Qwen-Image 3.0 Pro
Latin (en, de, es, pt)	235/236	235/235	229/235	230/235	213/235
Chinese	59/59	58/59	58/59	56/59	56/59
Japanese	59/59	58/59	52/59	51/59	45/59
Arabic	48/50	51/51	35/49	27/47	6/53

I give counts, not percentages, on purpose. With about 60 images per cell, one image moves a percentage by almost two points. GPT Image and Nano Banana are one image apart overall. That's a tie.

Same prompt, first image only

Two rules do most of the work.

Every model gets the identical prompt. One prompt per style per language, written in that language, sent unchanged to all five. No per-model tuning. If a model needs a special prompt to spell right, that's a result, not something to fix before measuring.

Only the first image counts. We do rerun wrong images, because the website needs a usable picture. But a retry only exists because the first try failed, and sometimes we draw extra samples after a correct one. Both lean toward correct. Count them and every model looks better than it is, and the weaker models get more second chances.

There was a less obvious trap. New styles get drafted on one model first, and the prompt changes a few times before all five run. Count every prompt version and that drafting model gets extra rows the others never saw. So scoring uses only the prompt version each style uses now. A prompt version is the MD5 of the full text sent, so "same prompt" is something you can check in the data, not something you have to take from me.

Settings were fixed per model: 2K, square, each vendor's API, October 22–29, 2026. GPT Image at xhigh quality, Grok Imagine at medium (its highest). The other three have no quality setting.

The ledger: append-only, latest review wins

Every generation writes a row, failed API calls included. Reviews go in a second table. Nothing in either table gets edited. Trimmed down, it looks like this:

-- one row per generation, success or failure
CREATE TABLE gen_attempts (
  id            bigint PRIMARY KEY,
  font_id       varchar(64),   -- style slug
  locale        varchar(10),
  word          varchar(100),  -- the text the prompt asks for
  prompt_hash   char(32),      -- md5 of the full prompt sent
  attempt_kind  tinyint,       -- 1 first, 2 retry, 3 extra sample
  model_id      varchar(64),   -- upstream model version at generation time
  ok            tinyint(1),    -- did the API return an image
  archive_url   varchar(512),  -- our copy of the original, never deleted
  gen_time      datetime
);

-- one row per review; changing your mind = a new row
CREATE TABLE gen_reviews (
  id            bigint PRIMARY KEY,
  attempt_id    bigint,
  verdict       tinyint,       -- 1 correct, 2 wrong, 3 unsure
  error_types   json,          -- fixed vocabulary
  rendered      varchar(255),  -- what the image actually says
  zoomed        tinyint(1),
  reviewer      varchar(32),
  is_native     tinyint(1),
  supersedes_id bigint,        -- the review this one replaces
  review_time   datetime
);

Three choices I'd make again.

model_id is read from the service config at the moment of generation. No hand-kept mapping table. If a vendor quietly swaps the model behind the same name, the data splits on its own.

A re-review is a new row pointing at the one it replaces. VENON's wrong "correct" is still in there, followed by the correction. You can see who changed their mind and when.

attempt_kind follows the latest reviews. If a first image flips from correct to wrong, the second image in that cell stops being an "extra sample" and becomes a "retry". It's the one field the tooling is allowed to recompute. And the content build refuses to publish a style if any of its images has no review yet.

AI reviews everything, people review where the AI might be wrong

Reading 2,000 images carefully by hand isn't realistic. I'd rather a person read 500 carefully than 2,000 tired. So there are two layers.

An AI reviewer read every image, zoomed in for Chinese, Japanese and Arabic. Then a person went through the images where the AI was most likely to be wrong:

every image the AI marked wrong or unsure
Arabic, where AI reading isn't reliable
Chinese and Japanese images the AI had only seen on a contact sheet
a random sample of 50 from everything else

The sample of 50 matched the AI on all 50, so the rest keep the AI's verdict and say so (human_reviewed: false). In the scored set, 489 images had a person look at them.

What we skipped: cases a glance settles. A paragraph of extra lettering across the image, or a word that differs only in case, which a rule decides (more below). Human time there buys nothing.

The direction of the AI's mistakes surprised me. It was too strict, not too lenient. About a third of the non-Arabic images it marked wrong were overturned by the human pass. The human pass wasn't perfect either: re-reading the disagreements changed 11 more verdicts. One read isn't enough, for a model or a person.

Wrong only if you can say what it reads as

Most of those overturned calls were the same kind: a letter that looks odd. The ぶ in しぶや drawn like a 3 with two dots. An extra slanted stroke inside 自. A capital F with a flourish that looks a bit like a J. A native reader reads all of them as the intended word without stopping.

So the rule became: a word is wrong only if you can say what it reads as instead. VENON. Konra. Eiwig. 爱の兰. If all you can say is "the shape is weird", it's correct. That's why rendered is required for every spelling error. If you can't fill it in, you don't have an error.

Three calls follow from that:

Traditional Chinese counts as correct in a Simplified prompt. A reader sees 义氣 and reads 义气. Mixed forms are normal on real signs.
Case doesn't count unless the prompt asked for it. POTION for Potion is fine.
Extra text counts as wrong only when it competes with the word: similar size, right next to it, reads like part of the title. A shop sign in the background is background.

I think this is the right rule for any text-rendering test, and I expect people to disagree. A stricter rule, every stroke matching a reference glyph, is easier to automate. It also punishes exactly the stylization people ask these models for. Nobody prompts for a blackletter logo and wants Helvetica.

Arabic: why we don't claim 100%

Nobody on my team reads Arabic natively. The AI doesn't read it reliably either. In the first human pass, the AI and the person disagreed on 17% of the Arabic images they both read.

When they disagreed, we didn't pick a winner. The record becomes unsure, gets disputed: true, keeps both calls, and stays out of the table:

{
  "id": 88,
  "language": "ar",
  "target_text": "شارع",
  "model": "Grok Imagine",
  "text_verdict": "unsure",
  "reviews": [
    { "reviewer": "ai", "text_verdict": "incorrect", "native_speaker": false, "zoomed": true },
    { "reviewer": "maintainer", "text_verdict": "correct", "native_speaker": false, "zoomed": true }
  ],
  "disputed": true
}

41 images in the scored set are like this. It changes how you should read the Arabic row. Nano Banana's 51/51 comes from 58 images; 7 are disputed. GPT Image has 8.

I'm confident in the ranking. 48 of 50 against 6 of 53 is a far bigger gap than any reviewing error. I'm not confident in the exact numbers, and they will move when native readers check them. Writing "100% human-verified" on this would be easy, and it would be false.

That one isn't disputed. The prompt asked for نمر (tiger). Seedream added a ب and a vowel mark.

Five numbers hide the error types

Overall: 99%, 100%, 93%, 91%, 79%. The ranking is right. It's still not enough to pick a model.

Qwen-Image made 86 errors. In 40 of them it spelled the word correctly and then filled the image with text nobody asked for, often a scrambled copy of the prompt.

Grok Imagine and Seedream fail on detail. Of Grok's 14 Arabic errors, 6 are a missing, extra or moved dot and 5 an extra letter. Of Seedream's 20, 9 are dots and 7 extra letters. In Japanese it's dakuten in the wrong place: Seedream turned ばくはつ into ばくばつ.

The error type predicts what a second run does:

	First image wrong	Fixed by one retry
GPT Image	3	1
Nano Banana	2	2
Grok Imagine	28	13
Seedream	36	18
Qwen-Image	86	10

Detail errors are noise: about half disappear on a second try. Qwen's extra text is a habit and survives retries. So "Seedream 91%, Qwen 79%" looks like worse and a bit more worse, but the fixes are different. With Seedream, run it again. With Qwen, rewrite the prompt; rerunning just burns money.

My take: a text-rendering benchmark that reports only an accuracy number is half a benchmark. That's why error_types in the ledger is a fixed list (extra_letter, dots_diacritics, extra_text, and so on), not a free-text note. Free-text notes don't add up.

The data is public

All 2,495 images in the ledger are in the dataset, including retries and test runs, one JSON record each:

{
  "id": 978,
  "style": "death-metal-font",
  "language": "en",
  "target_text": "VENOM",
  "prompt_hash": "035d561d24e21967dda5e64918b7d79a",
  "model": "GPT Image",
  "model_id": "gpt-image-2.5-sunburst",
  "attempt": "first",
  "purpose": "published",
  "image_url": "https://cos.aifont.ai/research/attempts/death-metal-font/death-metal-font-en-sunburst-s1-90094.jpg",
  "text_verdict": "incorrect",
  "error_types": ["wrong_letter"],
  "rendered_text": "VENON",
  "human_reviewed": true,
  "disputed": false
}

Prompts sit in a separate file keyed by prompt_hash. Images are linked, not shipped. License is CC BY 4.0, archived on Zenodo with DOI 10.5281/zenodo.23815210.

If you recompute anything, keep attempt == "first", purpose == "published", the current prompt version per style, and drop disputed. summary.json is already filtered that way.

Corrections are the part I care about most. If you read Arabic, Japanese or any language in there and a verdict is wrong, open a "Wrong verdict" issue with the record id and what the image actually says. An accepted correction goes in as a new review under your GitHub handle, same append-only rule, and the next release recomputes the verdict. Arabic readers first.

If you run a comparison like this without a native reader for some of the languages, say so in the results. We do.

The full per-language results and more examples are in our report on the best image model for text, and the data is on GitHub.

I tested 8 free background removers on the same 8 photos

jack — Mon, 19 Oct 2026 12:32:20 +0000

To find the best free background remover, I tested eight of them on the same eight photos on 19 October 2026: PixVibe, Backgroundless, Clipdrop, Pixelcut, Photoroom, Adobe Express, Canva and remove.bg. I only used what each tool gives you for free, scored every result, and checked where each tool sends your photo.

I make PixVibe, one of the eight. I'm saying that first because it matters for everything below. PixVibe also came out joint last on cutout quality, and that is in the results too.

The results

Rank	Tool	Score /100	Cutout quality /5	Free result size	Free batch	Where your photo goes
1	PixVibe	88.6	4.13	Original (2496 × 1664)	20 photos, free account	Stays on your device
1	Backgroundless	88.6	4.13	Original (2496 × 1664)	40 photos, no account	Stays on your device
3	Clipdrop	73.1	4.63	1024 × 682	No	Uploaded
4	Pixelcut	72.9	4.38	Original with a free account, 1080 × 720 without	No	Uploaded
5	Photoroom	70.9	4.38	2400 × 1600	No	Uploaded
6	Adobe Express	69.9	4.38	Original (2496 × 1664)	No	Saved to your Adobe cloud storage
6	Canva	69.9	4.38	Original (2496 × 1664)	No	Uploaded
8	remove.bg	68.9	4.38	612 × 408	No	Uploaded

The short version: Clipdrop cut out the cleanest edges. PixVibe and Backgroundless had the lowest cutout quality of the eight, and they still finished first because they are the only two that process the photo in your browser, give the full-size result for free and let you remove backgrounds from a batch of photos for free.

PixVibe runs its AI model on the user's own device. The photo never reaches a server, so PixVibe has no server cost per image, and that is how it can offer the full size and batches for free. The trade-off today is quality: in this test, the server models cut hair, glass and spokes more cleanly than the models running in a browser.

How I tested

Eight photos, each built around one thing background removers get wrong. I generated them with FontVibe (an AI image generator, mostly used for text designs) so that every one of them can be published and downloaded: no stock licence, no real person's face. Each photo is 2496 × 1664 pixels. Download any of them and run your own test:

Photo	What it tests
Curly hair	loose strands blowing across a busy street
Golden retriever	long fur on autumn leaves of almost the same color
Sneakers	a product with a soft shadow on a table
Wine glass	clear glass with a street visible through it
Bicycle	thin spokes with the background showing between them
Three friends	several people and hair against a park
Black cat	a black cat on a dark grey sofa
3D text logo	letters with holes, a glow behind them and a reflective floor

Free plans only. No trials, no paid credits. Where a tool needs an account, I used a free one.
One run per photo per tool, with the default settings. PixVibe was scored on its best free mode, Ultra; its default mode, Standard, is covered below.
Same day, same machine: 19 October 2026, Chrome on a Mac mini, home broadband.
Result size is the file a free user can actually download. I confirmed it with a downloaded file for six tools. For PixVibe I read the size of the result file the page produced, and for Pixelcut I checked the 1080 × 720 size in its download menu without an account and the original size with a free account.
Batch means more than one photo in, results out, without paying. I tried two photos at once on every tool.

How I scored

What I scored	Points	How
Cutout quality	65	Average of the eight photos (0 to 5 each), times 13
Where your photo goes	15	Processed on your device 15. Uploaded and deleted within a stated 30 days or less 9. Uploaded with no such promise 6
Free batch	8	20 or more photos per batch 8. Fewer than 20: 4. None: 0
Free result size	3	Original 3. Long side 1500 px or more 2. 1000 px or more 1. Smaller 0
No sign-up	3	A single photo works without an account 3, otherwise 0
No watermark	3	No watermark on the free download 3
Commercial use	3	The terms clearly allow it for free users 3. Unclear or contradictory 1. Not allowed 0

Each cutout got 0 to 5: 5 means I could not find a flaw when zoomed in, 4 a small flaw you would not notice in normal use, 3 a visible flaw you would fix by hand, 2 a clear mistake such as a missing piece of the subject or a large patch of background, 1 unusable, 0 no result.

I judged most cutouts from the full result on a magenta background, where anything left behind shows up. Two exceptions: Adobe Express and Pixelcut draw their result in an editor I could not export from without downloading, so I judged them from screenshots of that preview, about 750 and 600 pixels wide. remove.bg and Clipdrop were judged at the size they give for free.

Which background removers upload your photo

I watched the network requests each page made while it processed a photo. Six of the eight sent the photo to their own servers:

Tool	What the page did while processing
PixVibe	No image left the browser
Backgroundless	No image left the browser; only analytics requests
remove.bg	Sent a request to `www.remove.bg/images` and fetched the result from `o.remove.bg/downloads/`
Photoroom	Sent a request to `sdk.photoroom.com/v1/segment` about 10 seconds after the photo was chosen
Pixelcut	Sent a request to `api2.pixelcut.app/image/matte/v1`
Clipdrop	Sent a request to `api.clipdrop.co/remove-background/v2`
Canva	Showed the result from `media-transformation.canva.com`
Adobe Express	Created a project in the account for every photo and read it from Adobe's cloud storage (`platform-cs` on adobe.io)

None of the six promises to delete a free user's photos within 30 days. remove.bg says it will "delete them shortly after" (remove.bg Privacy Policy). Clipdrop keeps photos from users who have not signed up for two years (Clipdrop Privacy Policy). Backgroundless states that images are never uploaded (Backgroundless Privacy Policy).

Photo by photo

Each image below shows the same crop of the same photo from all eight tools. Magenta is what the tool removed. Adobe Express and Pixelcut appear on their own checkerboard because those crops come from screenshots of their editors. The remove.bg and Clipdrop crops are enlarged from their free sizes, so they look softer.

1. Curly hair

On the curly hair photo, Clipdrop, Pixelcut, Adobe Express, Canva and remove.bg kept the finest strands blowing away from the head. PixVibe, Backgroundless and Photoroom lost some of the outermost strands. Every tool kept the face and the main shape of the hair.

2. Golden retriever on autumn leaves

On the golden retriever photo, all eight tools kept the soft fur around the head. Two mistakes showed lower down, outside this crop: PixVibe left a few leaf fragments under the paws, and Photoroom cut off part of the right front paw.

3. Sneakers on a table

On the sneaker photo, all eight tools removed the table and the soft shadow cleanly, and every tool scored 5. Of the eight test photos, this is the closest to an ordinary product shot.

4. Wine glass

The wine glass photo separated the tools most. Clipdrop made the glass fully transparent, keeping the wine and the rim. Photoroom made it partly transparent. The other six kept the street seen through the glass as if it were printed on it. Adobe Express also kept the potted plant next to the glass as part of the subject, the only large mistake in the test.

5. Bicycle

On the bicycle photo, Clipdrop, Photoroom and Adobe Express kept every spoke and cleared the gaps between them. PixVibe, Canva and remove.bg broke some spokes. Pixelcut left small fragments of background near the saddle, handlebar and pedal. Backgroundless lost many spokes and left smears where they had been.

6. Three friends in a park

On the photo of three friends in a park, all eight tools separated the people from the park and from each other, with clean hair edges. Every tool scored 5.

7. Black cat on a dark sofa

On the black cat photo, all eight tools kept the whole cat and all eight left a strip of sofa along the bottom edge. A dark subject on a dark background was the one case in this test that no tool solved.

8. 3D text logo

On the 3D text logo, all eight tools removed the background inside the holes of the letters, the glow behind them and the reflective floor. PixVibe's default Standard mode did not: it kept the glow and the background inside the "a". Its Ultra mode removed both.

Each tool, briefly

PixVibe (88.6)

Free, processed on your device, full-size results, 20 photos per batch with a free account, and a single photo needs no account. Cutout quality 4.13 in Ultra mode. Ultra needs a browser with WebGPU and downloads about 223 MB of model data the first time. The default Standard mode scored 3.63: it failed the black cat and the 3D text. The terms allow commercial use of the results (PixVibe Terms). Try it on the free background remover.

Backgroundless (88.6)

Free, processed on your device, no account even for batches of 40 photos. Each result downloads one PNG at a time for free; a ZIP of the batch is a paid feature (Backgroundless FAQ). Cutout quality 4.13, the same as PixVibe, with a weaker result on the bicycle and a stronger one on the dog.

Clipdrop (73.1)

The cleanest cutouts in the test, 4.63, and the only tool that made the wine glass fully transparent. The free plan needs an account and scales every photo down before cutting it: my 2496 × 1664 photos came back at 1024 × 682, with an upgrade offered to keep the original. Free users get 20 removals per 24 hours (Clipdrop pricing) and process one photo at a time; the page says "Free users can only process images one by one". Clipdrop's FAQ states its terms do not limit commercial use (Clipdrop FAQ).

Pixelcut (72.9)

Cutout quality 4.38. Without an account the download is 1080 × 720; with a free account it is the original size. With a free account, the batch editor processed my two photos, then asked for Pro at "Download all". Its pricing page lists a commercial licence only on the paid plans (Pixelcut pricing), while a Pixelcut article says free results can be used commercially, so I scored commercial use as unclear.

Photoroom (70.9)

Cutout quality 4.38, with no account needed for a single photo and a 2400 × 1600 download without a watermark. It made the wine glass partly transparent, second only to Clipdrop, and it was the only tool to cut off part of the dog's paw. A free account can open Photoroom's batch editor, but the download button says Pro, and Photoroom's help centre states that batch export is limited to Pro, Max and Ultra (Photoroom Help: What is the Batch feature). Free accounts cannot be used commercially (Photoroom Help: Free accounts and commercial use). If you are looking for a Photoroom alternative for commercial work on a free plan, PixVibe, Backgroundless and Clipdrop allow it.

Adobe Express (69.9)

Needs an Adobe account. The free download was the full 2496 × 1664 with no watermark, and the result screen offers ready-made backgrounds. It takes one photo at a time; dropping two photos did nothing. Each photo became a project in the account. Adobe's pages do not agree with each other: its AI page says images generated in Adobe Express are safe for commercial use (Adobe Express AI), while its pricing FAQ lists remove background among premium features (Adobe Express pricing), even though the tool worked on my free account. I scored commercial use as unclear.

Canva (69.9)

Worked on a free Canva account from Canva's background remover page, with a full-size download and no watermark. Canva's own pages disagree on whether free accounts get it: the help centre says it is "included with paid plans" (Canva Help: Background Remover), and the feature page mentions a limited number of free downloads (Canva Background Remover). Selecting two uploads and choosing BG Remover opened a Canva Pro upgrade window. Its cutouts matched remove.bg almost exactly.

remove.bg (68.9)

Cutout quality 4.38 with no account, but the free download is a 612 × 408 preview; the 2496 × 1664 original needs payment. The website takes one photo at a time, and its bulk page sends you to Canva Pro (remove.bg Bulk Editing). The free service is for non-commercial use only (remove.bg Terms of Service).

remove.bg closes on December 1: a remove bg alternative

remove.bg's standalone website will no longer be available from 1 December 2026 at 9:00am CET, and its background removal is moving to Canva (remove.bg FAQ). In this test, Canva produced almost the same cutouts as remove.bg, at full size, but it needs an account. If you used remove.bg without an account, PixVibe, Backgroundless, Photoroom and Pixelcut also work without one, and all four gave a larger free download than remove.bg's 612 × 408 preview.

Bulk background remover: who lets you batch for free

I gave every tool two photos at once. Two of the eight processed and returned both for free:

Backgroundless: up to 40 photos per batch, no account. Download each result separately; the ZIP is paid.
PixVibe: up to 20 photos per batch with a free account, downloaded together as a ZIP.

The other six do not give you a free batch. remove.bg, Adobe Express and Clipdrop take one photo at a time. Pixelcut processed my two photos with a free account and asked for Pro at "Download all". Photoroom let a free account load photos into its batch editor, and its download button asks for Pro. Canva showed a Pro upgrade window as soon as I chose BG Remover for two photos.

Free plans compared

Tool	Account for one photo	Free download	Watermark	Commercial use on the free plan
PixVibe	No	Original size	None	Allowed by the terms
Backgroundless	No	Original size	None	Allowed by the FAQ
Clipdrop	Yes	1024 × 682	None	Allowed by the FAQ
Pixelcut	No	Original with a free account, 1080 × 720 without	None stated on its page	Pages disagree
Photoroom	No	2400 × 1600	None	Not allowed
Adobe Express	Yes	Original size	None	Pages disagree
Canva	Yes	Original size	None	Pages disagree on free access
remove.bg	No	612 × 408	None	Not allowed

The six free downloads I made had no watermark, PixVibe does not add one, and Pixelcut's background remover page states that its free download has none. Sizes are for my 2496 × 1664 test photos.

Appendix: every score

Photo	PixVibe	Backgroundless	Clipdrop	Pixelcut	Photoroom	Adobe Express	Canva	remove.bg
Curly hair	4	4	5	5	4	5	5	5
Golden retriever	4	5	5	5	4	5	5	5
Sneakers	5	5	5	5	5	5	5	5
Wine glass	3	3	4	3	4	2	3	3
Bicycle	4	3	5	4	5	5	4	4
Three friends	5	5	5	5	5	5	5	5
Black cat	3	3	3	3	3	3	3	3
3D text logo	5	5	5	5	5	5	5	5
Average	4.13	4.13	4.63	4.38	4.38	4.38	4.38	4.38

Tool	Quality 65	Photo location 15	Batch 8	Size 3	No sign-up 3	No watermark 3	Commercial 3	Total
PixVibe	53.6	15	8	3	3	3	3	88.6
Backgroundless	53.6	15	8	3	3	3	3	88.6
Clipdrop	60.1	6	0	1	0	3	3	73.1
Pixelcut	56.9	6	0	3	3	3	1	72.9
Photoroom	56.9	6	0	2	3	3	0	70.9
Adobe Express	56.9	6	0	3	0	3	1	69.9
Canva	56.9	6	0	3	0	3	1	69.9
remove.bg	56.9	6	0	0	3	3	0	68.9

First published on pixvibe.ai.

I gave 7 AI design tools the same 5 jobs on their free plans

jack — Fri, 16 Oct 2026 17:55:55 +0000

Most "best AI font generator" lists I read were written from product pages. Very few show what the tools actually produced. So I picked five ordinary design jobs, typed the exact same prompt into each tool, took the first result, and scored it. Every image in this post is the raw output, and every prompt is printed so you can run it yourself.

One thing to clear up before the results. When people search "AI font generator", most of them don't want a .ttf file. They want a word to look a certain way on a finished thing: a thumbnail, an invitation, a shirt. That's a text effect on a design, and that's what all seven of these tools make. If you need an installable font, skip to the section on font files.

The results

Rank	Tool	Average score	Text errors	Free plan, as I saw it on Oct 12, 2026
1	FontVibe	8.8	0	30 credits a day, up to 200 a month
2	Leonardo	8.7	0	150 tokens a day (8 per image in my test)
3	Recraft	8.3	0	30 credits a day, 100 a month
4	Adobe Firefly	8.0	1	3 images a day on the default model
4	Microsoft Designer	8.0	1	15 credits a month (1 per image)
6	Canva	7.8	0	5 designs used 77% of the month
7	Ideogram	7.5	3	30 credits (3 per low quality image)

"Text errors" means a broken letter, a covered word or garbled text in the final image. Scores are out of 10.

The short version: the gap at the top is small. Leonardo finished 0.1 behind FontVibe, and on two of the five jobs it scored higher. Where the tools really split was whether they did the specific lettering effect the prompt asked for. More on that below.

How I tested

Five jobs, from simple to hard. The difficulty is about the whole design: how much information it has to carry and how many layout rules it has to follow.

Bakery Instagram post (simple) Square Instagram post for my bakery announcing "Fresh Croissants Every Saturday", warm and cozy feel, golden flaky lettering, with a basket of croissants
YouTube thumbnail (simple) YouTube thumbnail for my survival challenge video, title "100 DAYS ALONE" in huge cracked-ice letters, snowy forest cabin in the background, leave the right third empty for my face
Kids' birthday invitation (medium) Birthday party invitation for my daughter, "Mia's 7th Birthday" in glossy balloon letters, "Saturday, Nov 14 · 2 PM", "Sunny Park Pavilion", "RSVP by Nov 7", bright and playful, portrait card
T-shirt chest print (medium) Front chest print for my streetwear brand, "NIGHTSHIFT" in wildstyle graffiti with drips, small "AFTER DARK CLUB · EST. 2026" underneath, a crescent moon graphic, isolated on plain white so I can send it to the printer
Gig poster (hard) A3 gig poster for my band "BURNING ORCHARD", band name in molten fire lettering as the main focus, "with special guests THE LOW TIDES", "SAT 14 NOV · DOORS 8 PM", "THE CELLAR · 22 MILL STREET", "TICKETS £12 ON THE DOOR", gritty rock atmosphere with burning trees

The rules:

Free plan features only. Default model, default settings. I only set the aspect ratio closest to the job.
One image per job. If a tool always returns more than one, I scored the best of that batch and said so.
I didn't pick fonts or styles by hand in any tool, including mine.

The scoring rubric

Every image got a score from 1 to 10 on four things. I wrote down the reason for each deduction before giving the number.

What I scored	Weight	What I looked at
Legibility	20%	First a check for hard errors: garbled letters, cut off text, text covered by other elements, broken shapes. No hard errors means 7 to 10, depending on how easily the main line reads. Any hard error means 1 to 4.
Design execution and style match	35%	Did it make the look the prompt asked for (pastry, ice, balloons, graffiti, fire), and how well: texture, colour, detail, and whether the whole image hangs together.
Layout and completeness	25%	Can you read the main message at a glance, is every line of information from the prompt there, is the order clear, and did it follow the format rules (square, empty right third, white background, A3).
Appeal	20%	Would a normal person be unable to make this themselves, and does it look like a designer made it.

What each score band means:

Score	Legibility	Design execution	Layout and completeness	Appeal
9–10	No errors, reads instantly even small	Every part of the requested look is there and well made	Clear hierarchy, all information present, careful layout	Designer quality, memorable
7–8	No errors, takes a second look	The look is mostly there, small flaws	Clear hierarchy, all information present, ordinary layout	Looks professional but like a generic template
5–6	Not used (legibility is pass or fail)	Only half done, or flat	Unclear hierarchy, or one minor line missing	Looks amateur
1–4	Any hard error	Requested look missing, or cheap looking	Key information missing or unreadable	Not usable

The total is calculated, not judged: 0.20 × legibility + 0.35 × design execution + 0.25 × layout + 0.20 × appeal, rounded to one decimal. This is the same rubric we use internally to rate designs made on FontVibe, written before this test.

Every score

Tool	Bakery	Thumbnail	Invitation	T-shirt	Poster	Average
FontVibe	9.0	9.0	8.6	8.6	8.8	8.8
Leonardo	8.7	9.0	9.0	7.7	9.0	8.7
Recraft	7.9	8.5	8.6	8.1	8.5	8.3
Adobe Firefly	7.5	8.1	7.1	8.1	9.0	8.0
Microsoft Designer	8.2	7.2	8.8	7.1	8.5	8.0
Canva	7.5	7.2	7.8	8.6	8.0	7.8
Ideogram	8.0	7.3	8.0	6.1	8.0	7.5

Tested on October 12 and 13, 2026. These tools change every few weeks, and one try per job means luck plays a part. Treat the scores as a snapshot, not a verdict.

Job by job

Job 1: Bakery Instagram post

The prompt asked for "golden flaky lettering". Only FontVibe made letters that look like baked pastry. Microsoft Designer got close. Leonardo used a script with a light pastry texture and added a shop name and a tagline on its own. Ideogram, Canva and Firefly all went with plain gold type, which looks fine but ignores the brief. Recraft's agent made four images; one spelled Saturday as "Savrdary", and the agent spotted it and dropped it itself.

Job 2: YouTube thumbnail

This job has one layout rule: keep the right third empty for a face. FontVibe, Leonardo and Recraft followed it and made convincing ice. Firefly made the best looking ice of the group but put the cabin in the right third. Canva and Designer kept the space clear but used white distressed letters with no ice at all. Ideogram's ice was good, but the L in ALONE nearly disappears into the background, so at thumbnail size it reads "A ONE".

Job 3: Kids' birthday invitation

Leonardo won this one. It put the date, place and RSVP on a card with small icons, which is what a parent would actually print. Designer was close behind. FontVibe's balloon letters were strong, but the date sits at the top and the venue at the bottom, split by a big balloon "7". Firefly wrote the date as "SATOV, 14" and invented an address, a phone number and an email. Canva used glitter type rather than balloon letters.

Job 4: T-shirt chest print

The hard part here is "isolated on plain white so I can send it to the printer". Recraft was the only tool that returned a vector file, which a print shop can use as is. Canva and FontVibe made proper wildstyle lettering with drips and the moon. Firefly's lettering was excellent, but it drew the design on a T-shirt mockup, so it isn't a print file. Designer's wildstyle is so tangled that NIGHTSHIFT is hard to read. Ideogram's drips run over the "AFTER DARK CLUB" line.

A note on FontVibe here, since I make it. My first request was routed to its single-image template mode, which produced a graffiti wall and missed the white background and the moon. I then asked in the same chat for design concepts and scored that result. The template mode isn't a design flow, so I didn't count it, but you should know it happened. We're changing the routing so a request like this goes to design concepts first.

Job 5: Gig poster

The hardest job, and the closest race. Leonardo and Firefly both nailed molten fire type, burning trees and six lines of information in a clear order. FontVibe was close; its secondary lines are all one size, so the special guests line doesn't stand out. Ideogram's layout was one of the best, but it printed a garbled word, "SSCIDU", above the date. Canva's band name is a red outline face, not fire.

Each tool, briefly

1. FontVibe (8.8)

FontVibe reads the brief and shows you three design concepts first: layout, lettering, colours and the lines of text it plans to use. You pick one, fill in anything missing (like a shop name), and it renders. It was the only tool that did the requested lettering effect on all five jobs, with no text errors.

Where it lost points: the invitation layout split the details, the poster's secondary text had no hierarchy, and the T-shirt request first went to the wrong mode. It's also not the fastest: about two minutes from prompt to image, including the concept step. A design cost me 33 credits (6 for concepts, 27 to render); with the cheapest image model it's 23. The free plan gives you 30 credits a day, up to 200 a month. Commercial use is allowed on every plan, including free.

Good for: anyone who cares that the lettering looks exactly like what they asked for.

2. Leonardo (8.7)

Leonardo's "Auto" model picked GPT Image 2 for all five jobs, and the free plan gives you that. Images came back in 30 to 40 seconds for 8 tokens each, so 150 daily tokens go a long way. It won the invitation and tied for the poster. It missed on the T-shirt, where it drew a simple black tag instead of wildstyle. Free images are public; private generation is a paid feature. The first try at the invitation was blocked by its content filter (I suspect the word "daughter"), with tokens refunded; the same prompt went through on retry.

Good for: high quality one-off images if you're happy with less control over the lettering style.

3. Recraft (8.3)

Recraft's chat agent runs one or two rounds on its own, checks spelling and layout, and tells you which image it thinks is best and why. It flagged two misspelled images on its own and told me not to use them. It's also the only tool that gave me a vector file for the shirt. The trade off is cost: each job used 6 to 13 credits because the agent decides how many rounds to run, and the free plan can't choose the image count. Recraft's own pricing page says free plan images are owned by Recraft, shown publicly and not licensed for commercial use.

Good for: logos and print work where a vector file matters, if you're on a paid plan.

4 (tie). Adobe Firefly (8.0)

Firefly's home page defaults to Google's Gemini 3.1 model (Nano Banana 2), not Adobe's own Firefly model; you have to switch in the menu if you want the one Adobe calls commercially safe. I left the default. It produced the best ice and one of the best posters, but it added information nobody asked for and broke the invitation date. The free plan gave me three images a day on this model. Adobe doesn't publish the number.

Good for: polished, photographic scenes. Check every line of text.

4 (tie). Microsoft Designer (8.0)

The simplest interface here: describe, pick a colour palette, pick one of three sizes. It was also the slowest, at 1.5 to 2 minutes per image, and it only offers square, 2:3 and 3:2, so there's no true 16:9 thumbnail. Its invitation was the second best of the group; its T-shirt lettering was hard to read. 15 free credits a month, one per image.

Good for: occasional use when you don't want to learn anything.

6. Canva (7.8)

Canva AI returns an editable Canva design, not a flat image, which is a real advantage if you want to change a word later. Layout and information were solid on every job, with no text errors. But it used ordinary fonts from its library and ignored the lettering effect in four of five jobs (pastry, ice, balloons, fire). It did make good wildstyle for the shirt. On the free plan, five designs used 77% of my monthly AI allowance.

Good for: people who already design in Canva and will edit the result by hand.

7. Ideogram (7.5)

Ideogram was the fastest, at 20 to 30 seconds, and its layouts often looked the most professional. But three of five images had text problems: a vanishing letter, drips over a line of text, and a garbled word. Its "Magic Prompt" is on by default and adds details on its own, which helped the bakery post and changed the lettering I asked for. The account showed 30 credits on the free plan, with low quality images at 3 credits each.

Good for: fast drafts where you'll check and fix the text yourself.

If you need an actual font file

None of the seven tools above make an installable font. If that's what you need, I tried Mixfont, which does.

I asked for a bold, bubbly display font for the same birthday invitation. Two minutes and 20 of my 50 free credits later, it gave me a TTF file I could install. The letters are consistent across upper case, lower case and numbers, and a long word like "Extraordinary" sets cleanly. The standard tier has 72 glyphs, so some symbols are missing (there was no & or £). An extended set with 319 glyphs costs 50 credits.

A font file and a text effect solve different problems. A font is for typing many words over and over. A text effect is for one word that has to look a particular way on one design.

What a template tool gives you

For comparison, I also tried TextStudio, a long-running template based text effect site that doesn't use AI. You pick an effect and type your words. The free "Frosty" template turned "100 DAYS ALONE" into a clean icy line of text on a snowy background. That's all it does: one line, one style, no cabin, no layout. The fire template was a premium one and covered in "PREMIUM" watermarks, and the pages carry a lot of ads.

Free plans compared

Tool	Free allowance (Oct 12, 2026)	Time per image	Notes from the test
FontVibe	20 credits without an account; with one, 30 a day, up to 200 a month. A design costs 23 to 33	About 2 min including concepts	Commercial use allowed on all plans
Leonardo	150 tokens a day; 8 per image on Auto	30 to 40 s	Free images are public
Recraft	30 credits a day, 100 a month, plus 30 bonus	30 to 60 s	Free images owned by Recraft, public, not for commercial use
Adobe Firefly	Daily free images; 3 a day on Gemini 3.1 in my test	20 to 30 s	Daily limit not published
Microsoft Designer	15 credits a month; 1 per image	1.5 to 2 min	Three sizes only
Canva	Officially up to 20 AI uses a month; 5 designs used 77%	30 to 40 s	Output is an editable design
Ideogram	30 credits on the account; 3 per low quality image	20 to 30 s	Magic Prompt on by default
Mixfont	50 credits; 20 per standard font	About 2 min	Exports a TTF file

What I'm taking back to FontVibe

Two things from this test go straight onto our list. A request with print rules, like "white background" and "a moon graphic", should go to design concepts, not the single-image template. And our secondary text needs clearer hierarchy on information heavy designs like posters and invitations. If you want to check our results against your own job, FontVibe's design mode is here.

Appendix: the full score sheet

Legibility / design execution / layout / appeal, then the weighted total.

Tool	Bakery	Thumbnail	Invitation	T-shirt	Poster
FontVibe	9/9/9/9 = 9.0	9/9/9/9 = 9.0	9/9/8/8 = 8.6	7/9/9/9 = 8.6	9/9/8/9 = 8.8
Leonardo	9/8/9/9 = 8.7	9/9/9/9 = 9.0	9/9/9/9 = 9.0	7/7/9/8 = 7.7	9/9/9/9 = 9.0
Recraft	9/7/8/8 = 7.9	9/8/9/8 = 8.5	9/9/8/8 = 8.6	7/8/9/8 = 8.1	9/8/9/8 = 8.5
Adobe Firefly	9/6/8/8 = 7.5	9/9/6/8 = 8.1	3/9/7/8 = 7.1	7/9/7/9 = 8.1	9/9/9/9 = 9.0
Microsoft Designer	9/8/8/8 = 8.2	9/5/9/7 = 7.2	9/9/9/8 = 8.8	3/8/9/7 = 7.1	9/8/9/8 = 8.5
Canva	9/6/8/8 = 7.5	9/5/8/8 = 7.2	9/6/9/8 = 7.8	7/9/9/9 = 8.6	9/6/9/9 = 8.0
Ideogram	9/6/9/9 = 8.0	3/8/9/8 = 7.3	9/8/8/7 = 8.0	3/6/8/7 = 6.1	4/9/9/9 = 8.0

The raw images, input screenshots and my written deductions for every score are kept on file. If you run one of these prompts and get something very different, email me at hello@fontvibe.ai and I'll add it to this post.

First published on fontvibe.ai.

AI cost modelling: measure your own token counts, the published table was off by 2.3x

jack — Thu, 24 Sep 2026 16:32:50 +0000

We resell image generation, so we have to know what a generation costs us before we can price it. That turned out to be a lot harder than reading the pricing page, and the two times it bit us were both cases where a number looked authoritative and was not.

Ratios taken from the published table were off by 2.3x on one tier

The provider charges per token. Tokens per image depend on resolution and on a quality setting, and the pricing page does not list tokens per image. It points you at a calculator in the docs, built from a table of token counts for a 1024×1024 image.

So we took our quality multipliers from that table. Low costs about 0.26 of Medium, High costs about 3.94 of Medium, price accordingly, move on.

To be fair to the provider, that table is not wrong. It describes one configuration, correctly. The mistake was ours: we treated ratios from a single 1024×1024 case as if they were the model's quality multipliers everywhere.

Then we measured. Nine generations, three resolutions, three quality settings, reading the token counts back out of our own usage records:

	derived from the published table	measured
low ÷ medium	0.258	0.112
high ÷ medium	3.94	4.00

High was close enough. Low cost us less than half of what we had assumed, so we had been pricing our cheapest tier as if it were 2.3 times more expensive than it actually is.

The thing that convinced me the measurement was real and not noise was that the ratios held across resolutions:

low  ÷ medium:  0.112, 0.112, 0.111
high ÷ medium:  3.998, 3.999, 3.999

Three different resolutions, same ratio to three decimal places. That is a parameter, not a sample. If those numbers had come back 0.11, 0.19, 0.14 I would have assumed measurement error and kept the published figure.

That is the check I would suggest to anybody doing this. A cost ratio that is stable across conditions you varied is a real multiplier you can build a price on. One that moves around is either noise or evidence that your model has the wrong shape.

A new model version at identical prices can still cost a quarter as much

The second one is stranger, and we caught it before it shipped rather than after.

The provider released a new generation of the same image model. Same price per token on every axis: image input, cached input, text input, output, all identical to the previous generation. Reading the pricing page, there is no change at all.

But the quality tiers went from three named levels to five, and a third-party benchmark on a fixed 2048×1152 image reported:

	output tokens
old model, `high`	1,413
new model, `high`	367

Same tier name, same price per token, about a quarter of the tokens.

I could not reproduce that directly, because at the time we had not integrated the new model. What made me believe it was that it agreed with a number we had measured ourselves for a completely different reason. Our own internal figure was high ÷ medium = 4.00 on the old model. 1,413 ÷ 3.85 lands on 367. So the third-party claim reduces to "the new model's high is roughly the old model's medium", and that is a statement our own independent measurement supports.

Worth being explicit about that reasoning, because it is the only reason I trusted a number I had not produced. A third-party benchmark you cannot reproduce is worth something if it agrees with a ratio you measured yourself for unrelated reasons. It is worth nothing on its own.

The provider did document this, sort of. Their model page says the five quality levels are not direct equivalents of the previous model's levels, and recommends evaluating with your own prompts. That sentence reads like standard hedging. It is actually a spec change with a number attached, and the number is not in it.

Why it matters more if you resell

If you are the end user of the API this is a pleasant surprise. You get the same work for less money, or you move up a tier.

If you resell, it is a trap, and the trap is shaped like a one-line config change.

Our users pick a quality tier and pay accordingly. High costs them four times what Medium costs. That multiple exists because High really did consume about four times the compute.

Swap the model ID to the new generation and nothing else, and the same button still sends quality=high, still charges four times, and now comes back with roughly what Medium used to produce. Nobody sees an error. The image still looks fine. The only thing that changed is that the customer is paying a 4× premium for something that is no longer 4× the work.

That is not a performance regression you can file and get to later. The thing you sold is not the thing you delivered. We ended up leaving the old model on its own row, untouched, and adding the new ones as separate options with their own tiers, measured separately, specifically so that nobody could make that change by editing one field.

The smaller one that nearly got us

Unrelated to models, same category of mistake.

We had a cost baseline that turned out to be wrong, $0.0021 where it should have been $0.00233, and three derived figures computed from it sitting in the same document. Fixing the baseline was a two-character edit. The three derived numbers stayed exactly where they were, looking correct, being wrong. They needed to go 4.7 / 2.4 / 2.2 to 4.3 / 2.1 / 1.9 and they were not going to do that on their own.

Derived numbers do not update themselves. They just keep being confidently wrong in a table somebody will read next quarter.

What we do now

We do not build a cost model on a vendor's published example table. It is illustrative, it is often derived from one configuration, and in our case the ratios taken from it were off by 2.3× on a tier we actually sell. Our own usage records are the only source that describes what we are actually being charged for.

Measure ratios, not absolutes, and vary something while you do it. A multiplier that holds steady across three resolutions is something you can price against. An absolute cost per image is out of date the next time anything changes.

Treat "the tiers are not direct equivalents" as a breaking change, not a disclaimer. Vendors write that sentence in the voice of a footnote.

And if you resell, put a measurement step between "new model is available" and "new model is live", because the config change that skips it is one line and reads as harmless.

We do this for fontvibe.ai, which is where the numbers came from. The reasoning is not specific to images or to any one provider.

Image prompting lessons: stop describing the image, hand the model a reference sheet

jack — Thu, 24 Sep 2026 16:10:01 +0000

Thirteen rounds, one afternoon, one word. The word was ONLINE and I wanted it drawn in a particular chunky cartoon style. I have the whole log because I was writing down what changed each round, which turned out to be the only reason I can now say anything useful about it.

Short version up front, since it isn't a twist: no amount of rewording got me there. I ended up drawing a reference sheet by hand and pointing at it.

I picked letters for this on purpose. Most image work gets judged on taste and those arguments go nowhere, but "is that an R" has an answer, and so does "is that corner curved". You can be wrong in a way somebody can check.

The scorer was happy the entire time

There was an aesthetic scorer in the loop. Every round it came back somewhere between 8.5 and 9.0. My own rating of the same images started around 3 and ended around 9.

Round 1, scored 8.5:

I had asked for small torn notches at the tips of the strokes. This has cracks across the entire body of every letter, plus a gloss I never mentioned.

The scorer was not malfunctioning. It is a decent-looking image. It answers "is this good" and I was asking "is this the thing I described", and once your model is competent enough to produce non-garbage those two questions stop being related. If I had let the score gate anything I would have stopped at round 1 and shipped a picture of broken glass.

Rounds 1 to 3 were aimed at something that was not there

Here is the part I am least pleased about.

I had looked at my reference at normal size, as a whole word, and read it as rounded letters with torn spikes poking out. So I spent three rounds describing torn spikes.

There were no spikes. When I finally cropped out single letters and blew them up, the letterforms were polygons made of straight facet cuts, and the things I had read as spikes were just the corners where two straight edges met. Nothing in that reference was torn, ripped, or broken. Three rounds of careful prompt surgery, aimed at reproducing a defect in my own eyesight.

Zoom in on one element before you write a single word about it. Whole-word at thumbnail size gives you the vibe and lies to you about the geometry.

The word you picked is arguing with you

This is the expensive one, and it happened three separate times with three different words.

I wrote	I meant	I got
`chamfer`	a flat angled corner	a 3D bevel, side faces lit and everything
`puzzle-piece-like`	letters sitting tightly together	jigsaw tabs. Actual ones.
`paper cutout`	flat, no depth	flat, with fold creases

That is round 10. I got precisely what I typed and nothing I meant.

The part that cost me rounds is that you can't fix this by pushing harder in the other direction. By round 8 the prompt carried "no 3D", "no bevel", "no side faces", "no internal lines", and the extrusion came back anyway, because chamfer and cut both point hard at "something was sliced here, show the marks". A pile of denials does not outvote the word itself. The word has to go.

So when a constraint keeps losing, stop reinforcing the constraint. Go find what it is losing to.

Say what it is, not how it got made

Two phrasings that mean the same thing to a person:

corners cut at irregular angles             # how it got made

the outer silhouette of every letter is an
irregular polygon, straight edges meeting
at sharp corners of varying length          # what it is

The first kept giving me cut marks. Seams inside the letter body, creases, and the little lit faces you would see if something really had been sliced through. The second gave me the shape and stopped.

My guess is that a verb names an event, and pictures of events come with the mess the event leaves behind. I would not push that too hard as a theory. As a habit it has held up: when junk keeps appearing inside the subject that you never asked for, go read your prompt for verbs.

Words about the room change the letters

The prompt is split into a subject half and a scene half. I had assumed that split meant something.

Material vocabulary in the scene half, describing the room, leaks onto the subject every time:

edge, core, depth used about the environment, and the flat letters grow 3D extrusion
voxel or cube, and the letter edges go blocky and stair-stepped
plastic, toy, brick, and the letters soften and round off

None of those were about the text. There's no compartment in there for "this clause is scenery". I ended up banning material words from the scene half entirely, objects and space and light only, and the drift stopped.

Giving up and drawing it

By round 11 the shape of the problem was obvious. Every fix moved something else. Tighten the silhouette, gloss comes back. Kill the gloss, the shape goes wrong. Fix the shape, a metaphor drags in creases. I was not converging, I was circling.

There is also an embarrassing middle stretch I have skipped over, where I landed a version I liked at round 5, showed it, and was told round 4 had been closer. Rounds 6 through 8 were me trying to get back to something I had already had.

So I drew the thing instead: a full A-Z and 0-9 sheet in the target style, one image. The prompt shrank to a pointer, roughly "follow the letterform skeleton in the reference, make the surface cleaner than the reference."

Eleven rounds of problems went away in one go. Not gradually. The 3D, the seams, the gloss, the jigsaw tabs, the creases, all of it, in a single generation.

Yes, drawing the sheet was work. It was less work than eleven rounds, and I still have the sheet.

One thing the reference could not do

It locked the shape, which also meant I could not change anything.

The last round wanted that exact shape with a glossier finish. Asked for in one sentence, the gloss dragged the 3D back in, same as round 8. Split across two sentences it behaved:

The 2D silhouette of every letter must stay identical to the reference.
No change to the letterform shape itself, no extrusion, no added bevel geometry.

Within that unchanged shape, push the surface material to a glossy,
high-shine polished enamel finish.

Identical information. Two sentences. I don't have an explanation for why that matters and I am slightly suspicious of it, but it has reproduced every time I h
ave needed to hold one property and move another, so I keep doing it.

If you take one thing from this: text is fine for a couple of constraints and it falls apart once you need five of them true at the same time. That is not a prompting skill issue, it's the channel. When you notice yourself circling, the question to ask is whether the thing you are describing could just be handed over instead.

This came out of building typeface generators, which live at fontvibe.ai. Nothing above needs them.

Unicode normalization is not a deduplication key

jack — Tue, 22 Sep 2026 17:29:47 +0000

I had 84,000 kaomoji to deduplicate and a key that looked obviously correct:

key = "".join(unicodedata.normalize("NFKC", text).split())

Strip the whitespace, normalize the Unicode, group by the result. Every
deduplication tutorial I have ever read says to normalize first. It took me one
afternoon of reviewing the output to realize that this key had quietly decided
(╥_╥) and (╥﹏╥) were the same record.

They are two different faces. The wavy mouth is a harder cry than the flat one.

NFKC folds four characters that matter here

NFKC is compatibility normalization. The K stands for compatibility, and the
point of it is to fold characters that exist in Unicode only for round-tripping
with older encodings: the fullwidth Latin letters, the halfwidth katakana, the
circled numbers, the ligatures. It maps them onto their "normal" equivalents so
that ｈｅｌｌｏ and hello compare equal.

That is correct when the text is prose and you want Ｅ－ｍａｉｌ to match
E-mail. It stops being correct once the characters are the content themselves.
Here is what it does to four characters that carry meaning in a kaomoji:

Codepoint	Name	NFKC result
U+FE4F	WAVY LOW LINE `﹏`	`_` (U+005F)
U+FF3C	FULLWIDTH REVERSE SOLIDUS `＼`	`\` (U+005C)
U+FF64	HALFWIDTH IDEOGRAPHIC COMMA `､`	`、` (U+3001)
U+3000	IDEOGRAPHIC SPACE	(U+0020)

The first one is the crying face. ﹏ is a wavy mouth and _ is a flat one, and
NFKC folds them together because one of them was originally a typesetting variant
of the other. The second one is a pair of arms. ＼(^o^)／ is a wide celebratory
shrug, \(^o^)/ is a narrow one, and after NFKC they are the same string.

The third one is a case where NFKC is right. A fullwidth and a halfwidth
ideographic comma really are the same punctuation mark, and (≖、≖╬) and
(≖､≖╬) really are one entry. That is what makes this hard. The normalization
is not simply wrong. It is right about maybe a third of what it touches, and you
can't tell which third without looking.

233 proposed merges, or 6,944

This is easy to measure on a published dataset, so you do not have to take my word
for it. The file below is the merged corpus, 82,109 entries, already deduplicated
on exact string match. Every collision found here is a proposed merge between
two entries that a human previously decided were different.

import gzip, json, unicodedata as ud, collections

def groups(keyfn):
    buckets = collections.defaultdict(set)
    for line in gzip.open("data/kaomoji.jsonl.gz", "rt", encoding="utf-8"):
        t = json.loads(line)["text"]
        buckets[keyfn(t)].add(t)
    g = [v for v in buckets.values() if len(v) > 1]
    return len(g), sum(len(v) for v in g)

strip_only = lambda t: "".join(t.split())
with_nfkc  = lambda t: "".join(ud.normalize("NFKC", t).split())

print(groups(strip_only))   # (233, 500)
print(groups(with_nfkc))    # (6944, 15038)

Stripping whitespace alone proposes 233 merges. Adding NFKC proposes 6,944,
covering 15,038 of the 82,109 entries. Auto-merging on that key deletes about
8,000 records, and you never find out which ones, because a merge leaves no
trace.

Some of what it finds is genuinely useful. These are real groups from that run:

(╥_╥)              (╥﹏╥)
(・∀・)ノ            (･∀･)ノ
(╯°□°)╯︵ ┻━┻       (╯°□°）╯︵ ┻━┻
(T_T)              (T＿T)          (Ｔ＿Ｔ)

The second row is a real duplicate. Halfwidth versus fullwidth katakana middle
dot is not something anyone chose, it is an artifact of which IME the author was
typing in. The third row is a table flip where one of the two parentheses came
out fullwidth, which is also an accident. The first row is two different faces.
The fourth row is three faces if you care about stroke weight and one face if you
don't.

There is no rule that separates those four rows. I tried several: character
class, width consistency within a single entry, whether the fold lands in the
mouth region. All of them got the first row wrong in one direction or the other.

Whitespace carries pose information

Stripping whitespace has the same problem on a smaller scale, because whitespace
is load-bearing in this data:

ʕ·ᴥ·ʔ          bear, facing you
ʕ·ᴥ·　ʔ         bear, looking left
ʕ　·ᴥ·ʔ         bear, looking right

( •_•)>⌐■-■    before the sunglasses
(•_•)>⌐■-■     also before the sunglasses, typed by someone else

The first three are three entries. The last two are one. Whether a given space is
a pose or a typo is not recoverable from the string, and I don't think it is
recoverable at all without knowing what the thing depicts.

Three keys, one of which is allowed to merge

key_exact = text                       # byte-identical; safe to auto-merge
key_loose = "".join(text.split())      # whitespace only; PROPOSES merges
key_nfkc  = "".join(ud.normalize("NFKC", text).split())   # output field only

key_exact collisions merge automatically with no review. key_loose collisions
go to a human queue, and 233 groups is an afternoon. key_nfkc is computed,
stored as a field in the dataset, and never consulted by the merge pipeline.
Consumers who want aggressive folding for search or for their own matching can
have it. It just doesn't get to delete anything.

That last key is the part I got wrong twice. My first instinct after it burned me
was to rip it out entirely, which would have been the opposite mistake, because
the folding is genuinely useful for recall. If somebody searches for (╥﹏╥) you
want the flat-mouthed one in the results too. So I kept computing it and took
away its ability to destroy a record.

Keep the queue small enough that you will actually read it

Normalization tells you which records might be the same. Whether they are the
same is a different question, and I am
not sure that one has an algorithmic answer.

So normalize to find candidates, keep the raw string as the only identity, and
put a human in front of anything that would destroy a record. If the human queue
is too big to review, the candidate key is too aggressive and needs tightening.
Nobody reviews 6,944 groups. They trust the key and move on, which is exactly how
the records disappear without anyone noticing.

The corpus is on GitHub with the raw JSONL and the verification scripts, and on
npm and PyPI as kaomoji-dataset if you just want the data:

npm i kaomoji-dataset
pip install kaomoji-dataset

https://github.com/Funovate/fontvibe-kaomoji

If you want to look at the entries rather than the code, the browsable version is
at fontvibe.ai/tools/kaomoji. Data is CC BY
4.0.

I merged the seven public kaomoji collections. They barely overlap.

jack — Tue, 22 Sep 2026 16:02:59 +0000

I wanted a complete list of kaomoji, the ( ͡° ͜ʖ ͡°) and ¯\_(ツ)_/¯ family. There isn't one.
So I merged the public collections instead, and the merge turned up something I
wasn't expecting: they hardly overlap at all.

Seven collections, 21 pairs, 0.3% median overlap

The ones I found:

fdw/rofimoji, the biggest English-side set
Allaman/emoji.nvim
aoguai/rime_kaomoji_dict
mtripg6666tdr/Kaomoji_proj
the Japanese IME dictionaries
kaosute
an older list of my own

Across the 21 pairwise combinations the median Jaccard overlap is 0.3%. Four
pairs share not a single entry. One pair clears 5%, rofimoji and emoji.nvim,
which are both English-side Linux pickers, so that one is no surprise.

The IME dictionaries are where most kaomoji actually live, and they share almost
nothing with any of the English sets.

Each entry in the merged file keeps the sources it came from, so you can check
this yourself:

import gzip, json, itertools, collections, statistics

by = collections.defaultdict(set)
for line in gzip.open("data/kaomoji.jsonl.gz", "rt", encoding="utf-8"):
    r = json.loads(line)
    for src in r["sources"]:
        by[src].add(r["text"])

o = [len(by[a] & by[b]) / len(by[a] | by[b]) for a, b in itertools.combinations(by, 2)]
print(statistics.median(o), max(o), sum(1 for x in o if x == 0))
# 0.003  0.079  4

None of the collections is close to complete, and after seeing those numbers I
understand why.

Coverage checked at run time

Saying "this has everything" is not worth much on its own, so
scripts/verify_coverage.py downloads each upstream file when you run it and
diffs it against the merged set. Nothing is precomputed.

Upstream	Entries	Covered	Missing
rofimoji	1,562	100.0%	0
emoji.nvim	2,016	99.9%	3
rime_kaomoji_dict	959	99.4%	6
Kaomoji_proj	2,166	97.8%	48

It prints all 57 misses, so you can look at them and disagree with me about what
counts as a kaomoji. Three rounds of that took rofimoji from 93.0% to 100.0%.

Labels, and where they fall down

Deduplicated it comes to 82,109 entries. Around 70k carry emotion, intent and
subject labels from a controlled vocabulary in en, ja, zh, es, pt and de, so
嬉しい, 开心, feliz and happy return the same entries instead of whichever English
word the original author happened to use.

Three things worth knowing before you use it:

The labels are coarse. 69,679 labelled entries share only 605 distinct label combinations, because they were applied in bundles rather than judged one at a time. Fine for pulling up a mood, not for finding one specific face.
About 15% is left unlabelled rather than guessed at.
Most of the corpus is 顔文字＋セリフ, a face plus a line of Japanese dialogue, which is not much use if you cannot read it.

That last one is why the English-language picker at
fontvibe.ai/tools/kaomoji shows 11,020, the
subset that renders safely outside Japanese contexts, while the full 82,109 sits
in the dataset and on the Japanese page.

Getting it

npm i kaomoji-dataset
pip install kaomoji-dataset

Both read a gzip off disk, so they are Node and Python only for now. They will
not bundle for a browser. The repo has the raw JSONL and a CSV as well:
github.com/Funovate/fontvibe-kaomoji

106 of the entries are not collected from anywhere. I drew them, for feelings
that had no kaomoji yet: 躺平, 社死, saudade, mamihlapinatapai. Those are CC0.

If you know of a collection I missed, say so and I will run the coverage script
against it.