A list of five plausible books can look like a successful recommendation. Repeating the request tells you something else: how much of the catalogue the model actually brings into view.
For Collected's book-recommendations study, we sent the same 100 questions to seven models. We then repeated 26 questions four more times, including all 20 genre questions. The calls ran on November 3 and 4, 2026, through OpenRouter, with default settings and no web search.
Across the five runs, the median genre produced just 18 distinct titles. In 13 of the 20 genres, one book appeared in every nonempty answer.
What the repeated runs showed
Each genre had up to 35 answers: seven models, five runs apiece. Every prompt requested five books, with title and author.
| Genre | Distinct titles across all runs | Most repeated book | Answers naming it |
|---|---|---|---|
| True crime | 11 | In Cold Blood | 34 of 34 |
| Science fiction | 13 | Dune | 35 of 35 |
| Thrillers | 15 | Gone Girl | 35 of 35 |
| Fantasy | 16 | The Name of the Wind | 30 of 35 |
| Poetry | 26 | Leaves of Grass | 33 of 33 |
| Dark romance | 29 | Haunting Adeline | 35 of 35 |
The range matters. Dark romance produced more than twice as many distinct titles as true crime, yet its most repeated book still appeared in every answer. A broad combined list can coexist with a very predictable recurring recommendation.
The same pattern appeared outside the genre questions. Asked for books that would make someone laugh out loud, the models named 13 distinct titles across 35 answers. The Hitchhiker's Guide to the Galaxy appeared in all 35. Asked for beach reads, they named Beach Read in 33 of 35.
Count answers, picks and distinct books separately
These three quantities answer different questions.
An answer is a returned list. A pick is one recommendation within that list. A distinct book is a title counted once across the lists being compared.
That distinction affects the denominator. Some requests returned nothing. The 20 genre questions produced 691 nonempty answers, not 700. Of those, 627 named exactly five books and 64 named more. Treating every request as five recommendations would miscount the output; treating an empty answer as a list without the leading title would distort the repetition rate.
For this test, the useful checks were the number of distinct books across repeated runs and the fraction of returned answers containing each book. A single attractive list would not have exposed that concentration.
What this does and does not establish
A repeated recommendation can be a good one. Dune is an unsurprising answer to a broad science-fiction question. These counts do not measure whether a particular reader would enjoy it, and they do not establish why the models converge.
The questions were written by us. The first subset selected for repetition was chosen after seeing the initial results. This is a descriptive test of those questions, not a random sample of everything readers ask.
It also tests API models without web search or personal reading history. Consumer chat apps can add search, memory and other instructions, so their answers may differ. Model versions and book coverage will change.
For developers evaluating a recommendation feature, the practical lesson is to examine repeated output as well as individual answers. If the feature promises discovery, count how many different things it actually recommends. Check whether changing the prompt or rerunning it produces useful alternatives, rather than assuming variation from one good-looking result.
The full study includes the questions, model details, answers and a downloadable recommendations CSV.
This article was drafted with AI from Collected's published research; its figures were checked against the study.
Top comments (0)