A few weeks ago, while working through the NLP section of my AI Engineering internship, I built a Movie Review Sentiment Analyzer. First I tried a ready-made Hugging Face pipeline — it worked almost too well, near-perfect confidence on every test review. Then I fine-tuned my own DistilBERT model on a small subset of the IMDB dataset, and that's where things got interesting.
Not because it failed — it actually reached 86.2% accuracy. But looking closer at the training numbers, I noticed something that's apparently a very common mistake: overfitting, happening in real time, in my own results. Here's what that looked like and what it taught me.
Part 1: The easy win
I started with Hugging Face's pipeline("sentiment-analysis"), which loads a model already fine-tuned for this exact task. I tested it on 3 sample reviews, and it nailed all three with near-perfect confidence — 0.9999 on a clearly positive review, 0.9998 on a clearly negative one, and 0.9891 even on a mixed, lukewarm review. This made sense: the model had already been trained specifically for sentiment analysis on far more data than I was about to use.
Part 2: Fine-tuning my own model
Next I fine-tuned distilbert-base-uncased myself, on a 2,000-review subset of the IMDB dataset (out of 25,000 available), for 3 epochs. Here's what the training looked like:
| Epoch | Training Loss | Validation Loss | Accuracy |
|---|---|---|---|
| 1 | 0.3295 | 0.4152 | 84.6% |
| 2 | 0.2186 | 0.4945 | 86.0% |
| 3 | 0.0539 | 0.5796 | 86.2% |
At first glance this looks fine — accuracy kept climbing, 84.6% → 86.0% → 86.2%. But look at the other two columns. Training loss dropped hard (0.33 → 0.22 → 0.05), while validation loss climbed every single epoch (0.42 → 0.49 → 0.58).
Part 3: What that actually means
This is a textbook overfitting pattern. The model was getting better and better at the training data — memorizing it, essentially — while getting slightly worse at generalizing to data it hadn't seen. Accuracy still crept up because 2,000 examples aren't much for a model the size of DistilBERT, so it could still improve overall even while starting to overfit underneath.
The likely cause: too little data for too many epochs. With only 2,000 training examples instead of the full 25,000, the model ran out of new patterns to learn within just a couple epochs, and by epoch 3 it was mostly just memorizing.
Part 4: Why this mattered to me
Before this, "overfitting" was a term I understood in theory — training accuracy high, test accuracy low, textbook definition. Seeing it show up in my own loss curves was different. It's one thing to be told a lower training loss doesn't always mean a better model. It's another to watch it happen in your own numbers and realize the model that looked "best" on paper (epoch 3, highest accuracy) was actually the one most at risk of not generalizing well.
If I did this again, I'd either train on the full 25,000-review dataset, stop after epoch 1 or 2, or add early stopping based on validation loss instead of just watching accuracy climb.
Takeaway
The pretrained pipeline outperformed my fine-tuned model, and that's fine — it was trained on far more data for this exact task. But the real value of this project wasn't the accuracy number. It was catching overfitting as it happened, in my own loss curves, instead of just reading about it as a concept.
If you're learning ML too, I'd genuinely recommend printing out your validation loss alongside training loss every epoch — it's a small habit that turns an abstract warning ("watch out for overfitting") into something you can actually see.
That's it for this one. Back to Week 7 (deployment) next — will post an update once the capstone starts.
Top comments (0)