DEV Community: Dostonbek

My Recommendation System Never Actually Measured Whether Its Recommendations Were Good - So I Checked

Dostonbek — Mon, 17 Aug 2026 13:13:36 +0000

An earlier project of mine built a content-based movie recommender: vectorize each movie's genre and plot summary, rank by cosine similarity, return the top 5 closest matches. It worked in the sense that it ran and printed five plausible-looking titles. What it never did was measure whether those five titles were actually good recommendations, by any definition I just eyeballed the output for two example movies and moved on. Going back to it, I built an actual evaluation, and along the way found a modeling choice that quietly made results worse, not better.

A note on the data: the original notebook used a Kaggle-hosted dataset I couldn't re-access while rebuilding this. I used the well-known public TMDB 5000 Movies dataset instead same shape (title, genres, plot overview), different specific source. The method and findings below are new work built on that substitute dataset, not a re-run of the original numbers.

The setup

4,800 movies with genre tags and plot overviews. Each movie's genres and overview text get combined into one "tags" string, vectorized, and compared by cosine similarity against every other movie. Ask for recommendations on a title, get back the 5 most similar by that measure.

The evaluation I should have had the first time
**
"Similar" isn't inherently meaningful without checking it against something. I used **genre overlap as a concrete, checkable proxy for recommendation quality: for a sample of 30 movies, pull each one's top-5 recommendations, and measure what fraction of the query movie's genres show up in each recommended movie's genres.

Then I compared that against two baselines, since a number on its own doesn't tell you much:

Random baseline: 5 randomly chosen movies instead of the model's picks.
Popularity baseline: always recommend the 5 most popular movies in the dataset, regardless of the query.

Approach	Mean genre overlap (top-5)
Random baseline	30.0%
Popularity baseline	26.1%
Content-based (genre + overview)	76.1%

The content-based approach clearly beats both baselines by a wide margin — this is the actual evidence the original project never produced, and it's a real, positive result: the recommender is doing meaningfully more than chance or "just recommend whatever's popular."

*The part that surprised me
*
I also compared two design choices I hadn't questioned in the original version:

Does including genre in the text tags actually help, or would the plot overview alone do just as well?

Tags used	Mean genre overlap
Overview text only	42.7%
Genre + overview	76.1%

Including genre explicitly nearly doubles the genre-overlap score which sounds almost circular (of course genre-matching improves when you feed in genre), but it's still worth confirming rather than assuming, since it means the plot text alone is carrying real but limited signal on its own.

Does switching from plain word counts (CountVectorizer) to TF-IDF usually treated as the "better" default in NLP actually improve things?

Vectorizer (genre + overview)	Mean genre overlap
CountVectorizer	76.1%
TF-IDF	49.5%

TF-IDF did noticeably worse here. My read: TF-IDF's whole mechanism is downweighting terms that appear frequently across the corpus, but genre words like "Action" or "Drama" are exactly the short, high-value, repeated terms this system needs to weight heavily to make genre-based matches, and TF-IDF suppresses them relative to rarer overview vocabulary. The "fancier" method actively worked against the thing I was trying to optimize for. I wouldn't have caught this without measuring both instead of assuming TF-IDF was the safer default.

*Where this still falls short
*

Genre overlap is a proxy, not ground truth. Two movies can share zero listed genres and still be a great recommendation (tone, theme, era), or share every genre and be a poor match. This measures one specific, checkable thing not "good taste."
30 sampled query movies is a reasonable spot-check, not exhaustive; a full evaluation across all 4,800 movies would give a more stable estimate.
No user feedback loop this is a purely content-based, offline evaluation. Real recommendation quality ultimately needs actual user response data, which this dataset doesn't have.

*Why this mattered to actually check
*
It's easy to ship a recommender that "looks right" on a couple of manually-inspected examples and call it done, I did exactly that the first time. The gap between "looks right on 2 examples" and "measurably beats a random baseline across 30" is the difference between a demo and something you could actually trust a design decision on, and the TF-IDF result specifically shows why skipping that check can leave a worse default in place without anyone noticing.

Code

Full evaluation code: github.com

]

If you've evaluated a content-based recommender differently, or think genre overlap is the wrong proxy here, I'd like to hear your take — reply here or find me on LinkedIn.

Revisiting My Capstone Project: The Metric I Trusted Was Hiding a Model That Barely Worked

Dostonbek — Mon, 03 Aug 2026 08:33:41 +0000

My Master's capstone was a churn prediction model for retail banking customers 10,000 records, the goal being to flag customers likely to leave before they actually do. At the time I reported it as a success: 86.5% accuracy from a Random Forest, best of the three models I tried. Going back to it recently, I realized that number was quietly hiding a model that was catching less than half the customers it was actually supposed to catch. This is what I found re-examining it, and what changed once I fixed it.

The original setup

The dataset: 10,000 bank customers, with features like credit score, geography, age, account balance, number of products, and whether they're an active member, and a binary target did they leave (Exited) or not.
My original capstone code (still in the repo) trained three models Logistic Regression, Decision Tree, and Random Forest with standard scaling and label encoding, and compared them on accuracy and ROC AUC. Random Forest won on both: 86.5% accuracy, 0.847 AUC.

What I missed the first time

The dataset is imbalanced about 80% of customers stayed, 20% left. I didn't account for that in the original analysis, and accuracy on an imbalanced dataset is a genuinely misleading number: a model that just predicted "stays" for everyone would already score close to 80%.

Going back and breaking the original Random Forest's performance down by class tells the real story: it only caught 45.9% of customers who actually churned (recall), while correctly flagging most of the stayers. Precision on the customers it did flag as leaving was decent (78.6%) but that's cold comfort for a churn-prevention model whose entire point is catching people before they leave. Missing more than half of them defeats the purpose. The original Logistic Regression was far worse on this front: 14.3% recall it was essentially only ever predicting "stays."

Fixing it

I re-ran the same three models with class_weight='balanced', which reweights the loss function so the minority class (churners) isn't drowned out during training, and added XGBoost with scale_pos_weight set to the actual class ratio for the same reason.

Model Accuracy Precision Recall F1 AUC Logistic Regression (original) 80.5% 58.6% 14.3% 0.229 0.771 Random Forest (original) 86.5% 78.6% 45.9% 0.580 0.847 Logistic Regression (balanced) 70.8% 38.3% 71.7% 0.500 0.774 Random Forest (balanced) 85.8% 77.3% 42.8% 0.551 0.848 XGBoost (default) 85.4% 70.6% 48.4% 0.574 0.838 XGBoost (weighted) 82.1% 55.2% 62.2% 0.585 0.831

A few things stand out:

Accuracy went down for almost every "improved" version. That's expected and, in this context, the right trade a model that's slightly less accurate overall but catches noticeably more actual churners is doing its actual job better, even though the headline number looks worse.
Balanced Logistic Regression's recall jumped from 14.3% to 71.7%, a huge swing from essentially useless to genuinely useful at catching churners, at the real cost of many more false alarms (precision dropped to 38.3%).
Weighted XGBoost gave the best overall balance (highest F1, 0.585), catching 62.2% of churners while keeping precision at a still-usable 55.2%.

Which of these is "best" isn't a purely statistical question it depends on the actual cost of a false alarm (an unnecessary retention offer to someone who wasn't leaving) versus a missed churner (losing a customer entirely). I don't have real cost figures for this bank, so I'm not going to invent a false-precision "optimal" answer but the tradeoff itself, and having several real options along that curve, is the actually useful output of this analysis, more useful than my original single accuracy number ever was.

What actually drives churn in this data

Feature importance from the balanced Random Forest:

Age (25.3%) by far the strongest single predictor.
Balance (14.3%), Estimated Salary (14.0%), Number of Products (13.6%), and Credit Score (13.4%) form a fairly even second tier.
Tenure (7.9%), Geography (3.9%), Active Member status (3.7%), Gender (2.0%), and having a credit card (1.9%) mattered comparatively little.

Age dominating this strongly, with tenure barely registering, wasn't what I expected going in I'd assumed how long someone had been a customer would matter more than it did.

Where this still falls short

I didn't have real business cost figures (cost of a retention incentive vs. value of a retained customer), so the "which model to deploy" decision here is illustrative, not a final recommendation.
The dataset is a single static snapshot rather than a time series, so this predicts churn risk at one point rather than tracking how risk changes over a customer's lifetime.
I used simple class-weighting rather than more involved resampling techniques (like SMOTE) worth comparing directly in a future pass.

Why I'm publishing the "I got it wrong the first time" version

It would have been easier to just write up the original 86.5%-accuracy result and stop there. I think the more honest and more useful piece is this one going back, checking the number I trusted at the time, and showing exactly where it broke down and what fixing it actually looked like. That's closer to how real model evaluation works than any single clean result.

Code

Original capstone notebook and the updated comparison: github.com/DostonUr/bank_churn

If you've handled class imbalance differently, or have a view on how the precision/recall trade-off should be set for a churn use case like this, I'd like to hear it reply here or find me on LinkedIn.

I Kept Doing the Same Sanity Check on Every Small Dataset, So I Turned It Into a Package

Dostonbek — Sun, 02 Aug 2026 13:42:18 +0000

After writing up two small projects - one predicting data science salaries, one checking whether Punxsutawney Phil's shadow call means anything - I noticed I'd done the same manual step both times: train a couple of models, check the test-set R², then immediately re-run everything with cross-validation before trusting that number. On a dataset with a few hundred rows, a single train/test split can make a weaker model look like the winner purely by chance, and I didn't want to keep catching that by hand.

So I turned the check into a small package: (https://github.com/DostonUr/modelcheck).

What it does

You give it your data and a dictionary of models, and it does the comparison I was doing by hand:

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from modelcheck import compare_regressors

result = compare_regressors(
    X, y,
    models={
        "Linear Regression": LinearRegression(),
        "Random Forest": RandomForestRegressor(n_estimators=300, random_state=42),
    },
    cv=5,
)
print(result)

It reports the held-out test score for each model, the 5-fold cross-validation spread (mean, std, min, max) for each, and — the part I actually built this for - it automatically flags two specific situations:

A model's cross-validation score bounces around a lot fold to fold, meaning the single test-set number shouldn't be trusted as precise.
Two models' CV score ranges overlap heavily, meaning whichever one scored higher on the held-out split isn't necessarily the real winner.

Proving it actually works, not just on a toy example

I re-ran both of my earlier projects through it instead of hand-writing the checks again.

On the salary data, it reproduced the exact numbers from the original article - Linear Regression R² 0.273 vs. Random Forest R² 0.24 on the test split - and it correctly generated the overlap warning without me writing that logic manually this time:

"Linear Regression" and "Random Forest" have heavily overlapping CV score ranges. The higher mean R2 alone is not strong evidence that "Linear Regression" is genuinely better on this data.

Then I tried something I hadn't done in the original groundhog article: instead of a hand-computed t-test, I set up a regression comparing a model that uses Phil's shadow call against a DummyRegressor that just always predicts the historical average, and ran both through modelcheck. If the shadow call carried real information, the model using it should have clearly beaten the dummy. Instead, the dummy baseline scored better (R² of ‑0.046 vs. ‑0.061), both were negative, and the tool flagged the same overlap warning again. That's the same "no real signal" conclusion from the original article, reached through a completely different comparison method — which is a more convincing confirmation than either check on its own.

Why I'm putting this out instead of keeping it as a personal script

A one-off script only proves I can write code. A package other people can install and actually use is a different kind of evidence of the work — it has to have real tests, has to handle inputs I didn't originally write it for, and has to be documented well enough that a stranger can pick it up. It's a small tool, but it's built to be used, not just read.

Try it

pip install git+https://github.com/DostonUr/modelcheck.git

Code, tests, and both worked examples: github.com/DostonUr/modelcheck

If you've hit the same problem - a small dataset making one model look better than it really is - I'd like to hear how you've handled it, or what this package is missing. Reply here or find me on
(https://www.linkedin.com/in/doston-urinov/).

Is Punxsutawney Phil Actually Any Good at Predicting Spring? I Checked 115 Years of Data

Dostonbek — Thu, 30 Jul 2026 15:07:28 +0000

Every February 2nd, a groundhog in Pennsylvania either sees his shadow or doesn’t, and half the US media treats it as a real forecast. I wanted to know if there’s anything to it, so I pulled the actual prediction record going back to the 1880s and checked it against real March temperatures. Short answer: no, and the “no” is more interesting than I expected.

The data

The dataset tracks Punxsutawney Phil’s yearly prediction (full shadow = “six more weeks of winter,” no shadow = “early spring”) alongside recorded February and March average temperatures for Pennsylvania, the Northeast, and the Midwest, going back to 1886.

After cleaning:

Dropped a summary row at the bottom of the file that wasn’t an actual year (labeled “1901–2000,” clearly an aggregate row someone left in).
Of 131 real yearly rows, 118 had an actual Full Shadow / No Shadow call - the rest were marked “No Record” (12 years) or, in exactly one case, “Partial Shadow.” I excluded those 13, since there’s no clean way to score a non-prediction.

115 of the remaining 118 had usable March temperature data to check the prediction against.

The class split is heavily lopsided: 100 Full Shadow predictions vs. only 15 No Shadow predictions - Phil calls for six more weeks of winter about 86% of the time, regardless of what actually happens. That imbalance turns out to matter a lot for how to read the accuracy number.
How I scored “correct”

Phil’s prediction is really a directional claim: Full Shadow implies a colder-than-normal March, No Shadow implies a warmer one. So for each year, I compared his call against whether that year’s actual Pennsylvania March average temperature came in above or below the long-run mean across the whole dataset (36.0°F).

Result

Accuracy: 52.2% (60 correct out of 115 years).

That sounds barely-better-than-a-coin-flip, and it is, but the more telling comparison isn’t 50%, it’s against a naive strategy that ignores Phil entirely and just always guesses “colder than average” every single year (which would nearly match his own bias toward predicting Full Shadow). That naive always-guess-colder strategy scores 49.6% statistically indistinguishable from Phil’s actual 52.2%.

I also ran a direct comparison of average March temperature in Full Shadow years vs. No Shadow years:

Full Shadow years: 35.95°F average
No Shadow years: 36.32°F average

That’s in the “correct” direction (Full Shadow years were very slightly colder), but the gap is tiny and a Welch’s t-test puts the p-value at 0.72 nowhere close to statistically significant. There’s no real signal here.

Checking it wasn’t a fluke of one region or one era

Before concluding “no skill,” I checked whether the result held up under different slices, since a null result on one narrow cut can hide something real:

Northeast temperatures: same story, 52.2% accuracy, p=0.50.
Midwest temperatures: 51.3% accuracy, p=0.73.

Pre-1960 vs. 1960-onward: 51.7% vs. 50.9%, no meaningful shift even across a period that includes real, measurable climate warming.

Consistent near-50% across three regions and two eras is a stronger “no effect” than a single test would have shown, and it’s the main reason I trust the null result here rather than writing it off as noise.

Why I think this happens

The lopsided class split explains most of it: because Phil predicts Full Shadow around 86% of the time regardless of the actual weather, his “track record” mostly just reflects how often a below-average March happens by chance which, across 115 years, is close to half the time anyway. There’s no evidence the tradition is tracking anything about the actual weather; it’s closer to a fixed, mostly-one-sided coin that happens to land near a 50–50 real-world split.

Where this could be wrong

Small sample for the “No Shadow” class (only 15 cases) means that side of the comparison has wide uncertainty a handful of different outcomes in those 15 years could shift the picture more than the same shift would in the Full Shadow group.

I used the Pennsylvania column as primary since that’s where Phil actually is, but “correct” is inherently a simplification real early-spring vs. late-winter isn’t fully captured by one month’s average temperature relative to a century-long mean.

I excluded the “No Record” and “Partial Shadow” years rather than trying to impute a guess for them, which is the more honest choice but does shrink the sample further.

Code and data

Full cleaning and analysis: github.com/DostonUr/GrounHog-prediction-Project-DS

If you’ve seen a different cut of this data or think the scoring method should be set up differently, I’d like to hear it reply here or find me on LinkedIn.

What I Learned Predicting Data Scientist Salaries From a Messy Public Dataset

Dostonbek — Wed, 29 Jul 2026 09:16:26 +0000

I've been asked some version of "how much should I expect to get paid as a data scientist" enough times that I decided to stop guessing and actually build something to answer it, using real data instead of gut feeling or whatever number shows up first on Google. This is a walkthrough of that project - what worked, what didn't, and the mistakes that taught me more than the parts that went smoothly.

The data

I used a public salary dataset covering data science and analytics roles, self-reported between 2019 and 2021 - 258 rows in total, mostly from the US, with fields for job title, years of tenure, location, education, industry, and salary.

It was not clean, and cleaning it took longer than the modeling did:

115 unique job titles for what were really about 8 actual role levels. "Senior Data Scientist," "Senior Data Scientist Lead," "Senior Predictive Analyst," and "Senor Data Analyst" (yes, a typo) were all technically different strings. I grouped them by hand into eight buckets - junior IC, analyst, mid-level data scientist, ML engineer, senior IC, manager/lead, leadership, and other - based on keyword matching against the raw title.
Salaries in inconsistent formats - some as $190k, some as "93,000" with commas and quote marks, some as bare numbers. These needed normalizing before anything downstream would work.

Missing data throughout - tenure was missing for 26 rows, education for 3, and 4 rows had no usable salary at all and had to be dropped, leaving 254 usable rows.

A handful of real outliers, not typos. Three rows - all "Data Scientist" or "Senior Data Scientist" titles on the West Coast - reported salaries between $375,000 and $475,000. I checked these individually rather than deleting them; they're plausible for senior IC/staff-level roles at large tech companies, so I kept them in, which is itself a modeling decision worth being upfront about.

After cleaning: median salary across the dataset was $114,500, with a fairly wide spread (25th percentile $85,000, 75th percentile $150,000).

Approach

Features going into the model: grouped title, years of tenure, US region, education level, and industry group.

I tried two models:

Linear Regression (baseline): R² of 0.27, mean absolute error of about $41,600.
Random Forest Regressor (300 trees, max depth 6): R² of 0.24, MAE of about $43,300 on the same held-out test split. Cross-validation across 5 folds showed the score bouncing between 0.10 and 0.34 depending on the split - a direct consequence of having only 254 rows to learn from.

The Random Forest didn't beat the simple linear baseline. I want to be honest about why, rather than quietly picking whichever number looked better: with a dataset this small, a more flexible model doesn't have enough data to find real non-linear patterns - it mostly just adds variance. The linear model's more constrained assumptions actually generalized slightly better here. That's a useful lesson on its own: more complex isn't automatically better, especially under 300 rows.

What actually mattered

Looking at feature importance from the Random Forest (grouped by category, since one-hot encoding splits each into several columns):

Job title / seniority group mattered most overall (about 30% of total importance) - unsurprising, but useful confirmation.

Region was close behind (about 24%) - West Coast roles had a median salary of $145,000 versus $86,000 for Southeast roles and $86,000 for non-US roles in this sample, the widest gap in the whole dataset.

Years of tenure alone (as a single numeric feature, not grouped) was actually the single most important individual feature - more predictive than any one title or region category.

Education showed a real gap too - PhD holders had a median of $155,000, versus $110,000 for Master's and $95,500 for Bachelor's - though I'd treat this cautiously since PhD and leadership-title rows likely overlap.
By seniority group, medians ranged from $185,000 (leadership) down to $49,000 (junior IC), with a genuinely odd result in between: the "ML engineer" group's median came out lower ($80,000) than mid-level data scientists ($110,000), which contradicts what I expected going in. With only 9 ML-engineer rows in the whole dataset, this is almost certainly a small-sample artifact rather than a real market signal, and I'm flagging it rather than smoothing it over.

Where this falls short

254 rows is small for 5 categorical features with many levels between them - several title/region/education combinations have only a handful of examples, so predictions for those specific combinations are shakier than the headline R² suggests.

The data is from 2019–2021. Tech and data science compensation has moved a lot since then; treat this as a lesson in method, not as current market pricing.

Self-reported data carries the usual bias risks - people who respond to salary surveys aren't a random sample of the whole field.
An R² of ~0.27 means the model explains roughly a quarter of the variance in salary - useful for spotting directional patterns (seniority, region, education all matter, in that rough order), not precise enough to quote a confident number to a specific person.

What I'd do differently

Next time I'd spend less effort tuning the Random Forest and more effort either finding a larger dataset or engineering better features from what little text data exists (like extracting seniority signals more carefully from free-text titles instead of keyword-matching by hand). Given how much of the "title" signal was really just noisy text, a cleaner NLP-based title-normalization step would likely have helped more than switching models did.

Code

Full cleaning and modeling code: github.com/DostonUr/DataScience_Salary

If you've worked with a similar dataset or see a flaw in how I handled the outliers or the title grouping, I'd genuinely like to hear about it - reply here or find me on LinkedIn.