I asked eight models which clubs footballers played for

ExtraTime, my football quiz site, has a database of footballers and every club they played for, checked by hand, because the games depend on it. Career Path shows you a player's clubs and you guess who he is. Five Clubs does the reverse. A wrong club in that data is a wrong answer for whoever's playing.

So when people ask why I don't just have a model generate the questions, I'd like to answer with a number. I picked 200 players from the database, asked eight models which clubs each one played for, and scored every answer against the real list.

The setup

The 200 players are 50 from each of ExtraTime's four Career Path difficulty tiers: EASY, NORMAL, HARD and EXTREME. The tiers are about how well-known a player's career is, so EASY is Fabio Grosso and Teddy Sheringham, and EXTREME is players most fans won't have heard of. Between them the 200 played for 1,292 clubs.

Every model got the same prompt, once per player:

Footballer: Fabio Grosso, born 1977-11-28.

Answer from memory, as JSON only, no other text:
{"known": true or false, "nationality": "the country he represents or is a citizen of", "clubs": ["every senior club he has played for, including loans, in order"]}

If you don't know this player, answer {"known": false} and nothing else. Don't guess clubs you aren't sure of.

The date of birth is there so the model knows which Ronaldo I mean. The last line gives every model a way out, so a model that guesses has chosen to.

All eight ran through DigitalOcean's serverless inference, one endpoint and one API key for models from OpenAI, Meta, DeepSeek, Mistral, Google and Anthropic. That made switching models a change to one string, and all 1,600 questions together cost $1.08.

Scoring each club means matching a name like "Internazionale" to ExtraTime's "Inter". I matched against all 5,451 clubs in the database and their 2,337 alternative names, then went through about 150 near-misses by hand and changed 43 of the decisions. "Dinamo Bucharest", "Cologne" and "Chealsea" count as right. "Lazio Roma" for a Roma player doesn't.

Results

Precision is the share of the clubs a model named that were really the player's. Recall is the share of the player's 1,292 real clubs that the model named. A player it declined to answer counts as zero recall. "All right" means every club listed, and nothing wrong added.

Model	Precision	Recall	All right	Said "don't know"	Nationality	Cost, 200 players	Median time
DeepSeek V4.1 Flash	76%	69%	50	6	186/194	$0.526	20.3 s
Llama 4 Maverick	75%	50%	18	13	172/185	$0.012	1.9 s
gpt-oss-120b	63%	32%	13	45	145/155	$0.090	9.9 s
GPT-5.4 mini	62%	58%	22	9	188/191	$0.060	1.3 s
Gemma 4 31B	49%	23%	4	48	133/149	$0.009	2.5 s
Claude Haiku 4.5	44%	30%	4	57	136/143	$0.081	1.4 s
gpt-oss-20b	33%	6%	1	129	67/71	$0.079	7.7 s
Ministral 3 14B	14%	9%	1	23	119/152	$0.007	1.0 s

No model is good enough to write quiz answers on its own. The best one got a quarter of its clubs wrong, and it got every club right for only 50 of the 200 players.

DeepSeek V4.1 Flash was the most accurate and the slowest by far. It's a reasoning model, and it spent a median of about 1,600 tokens thinking per answer, so the median answer took 20 seconds and one in ten took more than 66. GPT-5.4 mini named the most clubs, 1,216, and answered in 1.3 seconds. Llama 4 Maverick was as precise as DeepSeek, half as complete, and cost $0.012 for all 200 players. That's about two cents per thousand correct clubs, against 59 cents for DeepSeek.

Ministral 3 14B was wrong six times out of seven. For Grosso, it listed seven clubs, and he played for one of them. For Christian Panucci it invented "Byte CapSelect Roma". And 25 of its answers weren't valid JSON, because it wrote notes inside the array, like "Real Madrid" (loand to "PSV Eindhoven").

Fame is most of it

Precision by tier:

Model	EASY	NORMAL	HARD	EXTREME
DeepSeek V4.1 Flash	96%	91%	79%	43%
Llama 4 Maverick	92%	79%	73%	53%
GPT-5.4 mini	91%	76%	59%	28%
gpt-oss-120b	81%	58%	59%	33%
Gemma 4 31B	76%	51%	28%	16%
Claude Haiku 4.5	64%	39%	31%	35%
gpt-oss-20b	41%	33%	8%	0%
Ministral 3 14B	30%	10%	7%	7%

On the 50 best-known players, the top three were above 90%. On the 50 least known, the best was 53%. That's exactly the wrong way round for a quiz site. Famous careers are the ones I need help with least.

"Don't guess" didn't stop the guessing

Every model was told to skip players it didn't know. They mostly answered anyway, and the answers on the players they barely knew were where the wrong clubs piled up.

Claude Haiku 4.5 declined 57 players, more than most, and still named 488 clubs that weren't the player's, against 389 that were. gpt-oss-20b was the most cautious. It declined 129 of the 200, including 12 of the 50 easiest. When it did answer, two clubs in three were wrong. Declining more didn't make the answers that were left any better.

The wrong clubs are usually real clubs, which makes them harder to catch. Of the 2,820 clubs the models named that weren't the player's, 2,459 were other clubs in my database, and only 355 didn't match any club at all. A made-up name is easy to spot. Palermo on the wrong player isn't.

When the models agreed and the database didn't

If several models independently name the same club and my database doesn't have it, maybe the database is wrong. Only three player-club pairs were named by four or more of the eight models and missing from the database:

• Teddy Sheringham at Stevenage, from five models. He never played for them. He managed them in 2015–16, and according to his Wikipedia page he even registered himself as a player, at 49, without making an appearance. The models were remembering the right man at the right club in the wrong role.
• Loïc Rémy at Rennes, from four. He never played there. Four models were wrong in the same way.
• Grenddy Perozo at Deportivo Anzoátegui, from four. Here the models were right and my data was wrong: ExtraTime has the club twice, as "Anzoategui" and "Deportivo Anzoategui". His career points at one of them, and the models named the other.

So the database held up, and the test found one real bug in it.

What I got wrong

My first run gave every model 3,000 output tokens. DeepSeek ran out of tokens on 56 of the 200 players while still reasoning, and returned nothing. Those turned out to be mostly the hard players. With them missing, DeepSeek looked 88% precise. I reran the 64 truncated answers (58 DeepSeek, 6 gpt-oss-20b) with 16,000 tokens. Some needed more than 10,000, and DeepSeek's precision fell to 76%. The first attempts cost $0.22 and produced nothing.

My first club matcher counted a club as right if every word of the claim appeared in one of the player's club names. So "AC Milan" matched "Inter Milan", and Grosso picked up a club he never played for. Now an exact match with another real club wins first.

The nationality check was looser than I wanted. A model saying "Spain" for Omar Mascarell, who was born in Spain and plays for Equatorial Guinea, is marked wrong against ExtraTime's data, though it isn't really wrong. I'd treat the nationality column as rough.

Caveats

200 players and one prompt per player, at default temperature. A second run would move individual numbers by a few points. ExtraTime's data is the answer key, so where it has a gap, a correct model answer counts as wrong. The disagreement check above suggests that's rare, but it isn't zero. The tiers measure how well-known a career is for ExtraTime's players, not some general measure of fame. And this tests memory on purpose. With a search tool or the database in the prompt, every model here would do much better.

What I'd take from it

• Don't let a model write facts into a quiz unchecked. Even the best one here got a quarter of its clubs wrong.
• Use a model to draft, and check against your own data. Llama 4 Maverick at two cents per thousand correct clubs is cheap enough to draft everything and throw away what doesn't check out.
• Expect it to be worst exactly where you need help: the obscure players.
• Telling a model not to guess isn't a control. Measure what it does with the instruction.
• Give reasoning models enough output tokens, and count the ones that ran out, or your numbers will flatter them.