An earlier project of mine built a content-based movie recommender: vectorize each movie's genre and plot summary, rank by cosine similarity, return the top 5 closest matches. It worked in the sense that it ran and printed five plausible-looking titles. What it never did was measure whether those five titles were actually good recommendations, by any definition I just eyeballed the output for two example movies and moved on. Going back to it, I built an actual evaluation, and along the way found a modeling choice that quietly made results worse, not better.
A note on the data: the original notebook used a Kaggle-hosted dataset I couldn't re-access while rebuilding this. I used the well-known public TMDB 5000 Movies dataset instead same shape (title, genres, plot overview), different specific source. The method and findings below are new work built on that substitute dataset, not a re-run of the original numbers.
The setup
4,800 movies with genre tags and plot overviews. Each movie's genres and overview text get combined into one "tags" string, vectorized, and compared by cosine similarity against every other movie. Ask for recommendations on a title, get back the 5 most similar by that measure.
The evaluation I should have had the first time
**
"Similar" isn't inherently meaningful without checking it against something. I used **genre overlap as a concrete, checkable proxy for recommendation quality: for a sample of 30 movies, pull each one's top-5 recommendations, and measure what fraction of the query movie's genres show up in each recommended movie's genres.
Then I compared that against two baselines, since a number on its own doesn't tell you much:
- Random baseline: 5 randomly chosen movies instead of the model's picks.
- Popularity baseline: always recommend the 5 most popular movies in the dataset, regardless of the query.
| Approach | Mean genre overlap (top-5) |
|---|---|
| Random baseline | 30.0% |
| Popularity baseline | 26.1% |
| Content-based (genre + overview) | 76.1% |
The content-based approach clearly beats both baselines by a wide margin — this is the actual evidence the original project never produced, and it's a real, positive result: the recommender is doing meaningfully more than chance or "just recommend whatever's popular."
*The part that surprised me
*
I also compared two design choices I hadn't questioned in the original version:
Does including genre in the text tags actually help, or would the plot overview alone do just as well?
| Tags used | Mean genre overlap |
|---|---|
| Overview text only | 42.7% |
| Genre + overview | 76.1% |
Including genre explicitly nearly doubles the genre-overlap score which sounds almost circular (of course genre-matching improves when you feed in genre), but it's still worth confirming rather than assuming, since it means the plot text alone is carrying real but limited signal on its own.
Does switching from plain word counts (CountVectorizer) to TF-IDF usually treated as the "better" default in NLP actually improve things?
| Vectorizer (genre + overview) | Mean genre overlap |
|---|---|
| CountVectorizer | 76.1% |
| TF-IDF | 49.5% |
TF-IDF did noticeably worse here. My read: TF-IDF's whole mechanism is downweighting terms that appear frequently across the corpus, but genre words like "Action" or "Drama" are exactly the short, high-value, repeated terms this system needs to weight heavily to make genre-based matches, and TF-IDF suppresses them relative to rarer overview vocabulary. The "fancier" method actively worked against the thing I was trying to optimize for. I wouldn't have caught this without measuring both instead of assuming TF-IDF was the safer default.
*Where this still falls short
*
- Genre overlap is a proxy, not ground truth. Two movies can share zero listed genres and still be a great recommendation (tone, theme, era), or share every genre and be a poor match. This measures one specific, checkable thing not "good taste."
- 30 sampled query movies is a reasonable spot-check, not exhaustive; a full evaluation across all 4,800 movies would give a more stable estimate.
- No user feedback loop this is a purely content-based, offline evaluation. Real recommendation quality ultimately needs actual user response data, which this dataset doesn't have.
*Why this mattered to actually check
*
It's easy to ship a recommender that "looks right" on a couple of manually-inspected examples and call it done, I did exactly that the first time. The gap between "looks right on 2 examples" and "measurably beats a random baseline across 30" is the difference between a demo and something you could actually trust a design decision on, and the TF-IDF result specifically shows why skipping that check can leave a worse default in place without anyone noticing.
Code
Full evaluation code: github.com
]
If you've evaluated a content-based recommender differently, or think genre overlap is the wrong proxy here, I'd like to hear your take — reply here or find me on LinkedIn.
Top comments (0)