If you've trained a deep learning model from scratch, you already know the two problems that hit you immediately: you need a lot of labeled data, and even if you have it, training takes forever. I ran into both while working on a cats vs. dogs image classifier, and it's what pushed me to actually understand transfer learning instead of just using it as a buzzword.
This post walks through four experiments I ran on the same dataset - a CNN I built and trained entirely from scratch, followed by three transfer learning approaches using VGG16 - to see how much transfer learning actually helps, and why the order you apply these techniques matters.
Why not just train your own model?
Deep learning models are data-hungry. To train something reliable from scratch, you typically need thousands of labeled images - and labeling isn't free. Someone (or something) has to go through every image and mark whether it's a cat or a dog, which costs time and, at scale, money.
Even with enough data, training a CNN from scratch on a reasonably large dataset takes significant compute time. Both of these - data scarcity and training cost - are exactly what transfer learning is designed to solve.
What is transfer learning, actually?
The simplest way I can put it: transfer learning means taking a model that already learned something useful from one problem, and reusing that knowledge on a different but related problem.
It's the same idea as learning to ride a bicycle before a motorcycle - the balance and coordination your brain already built don't get thrown away, they get reused. Or if you play violin, picking up guitar is easier because you already understand musical notes, rhythm, and practice discipline.
In deep learning, this looks like: take a CNN that's already been trained on a huge, general image dataset (like ImageNet, ~1.4 million images across 1000 categories), and reuse its learned visual understanding on your own, much smaller dataset.
Why does this actually work? Because convolutional layers learn hierarchically. Early layers pick up primitive features - edges, colors, simple textures - which are common across almost any real-world image. Later layers combine those into more complex, task-specific patterns. So the early general knowledge doesn't need to be relearned every time; only the final, task-specific classification layers need to adapt to your actual problem.
My experiment: Cats vs. Dogs, from scratch vs. transfer learning
I used the Cats vs Dogs dataset from Kaggle and ran four experiments: first a CNN trained entirely from scratch as my baseline, then three transfer learning approaches, each building on lessons from the last.
Baseline: A CNN trained from scratch (not transfer learning)
Before touching transfer learning at all, I trained my own CNN architecture from scratch, with no pretrained weights involved - this is the point of comparison for everything that follows.
My first version had no regularization. Result: 99% training accuracy, but only 78% validation accuracy, with validation loss climbing past 1.3 as training progressed. That gap is a textbook overfitting signature - the model was memorizing training images rather than learning generalizable features.
Adding dropout and batch normalization changed the picture: 87% train / 76% val, with validation loss staying controlled instead of diverging. Notice train accuracy actually dropped - that's expected and healthy. The model gave up some memorization in exchange for learning patterns that generalize better.
Even with regularization, a meaningful train-val gap remained. That gap is what pushed me toward transfer learning.
Transfer Learning Approach 1: VGG16 Feature Extraction
Here's where transfer learning actually starts. I used VGG16 - a CNN pretrained on ImageNet - as a frozen feature extractor. I removed VGG16's original classification head, froze its convolutional base entirely, and attached my own dense layers on top:
model.add(Dense(256, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
Result: 98% train / 89% val accuracy - a big jump over the from-scratch baseline. But the train-val gap widened again, since the dense head was learning fast on top of already-excellent frozen features, with nothing to keep it in check.
Transfer Learning Approach 2: Feature Extraction + Data Augmentation
This was the most interesting result of the whole project. I applied the same frozen-VGG16 setup, but added image augmentation (rotation, flips, zoom) during training.
Result: 93% train / 92% val accuracy.
Look closely - train accuracy actually went down compared to the previous approach (98% → 93%), but validation accuracy went up (89% → 92%), and the train-val gap nearly disappeared entirely.
This is the clearest illustration I've seen of the difference between memorization and generalization. Augmentation forced the model to work harder on each training example (since it never sees the exact same image twice), which cost it some raw training accuracy - but that trade bought real generalization.
Transfer Learning Approach 3: Fine-tuning
For the final experiment, I unfroze VGG16's last convolutional block (block5) and trained it jointly with the dense head, using a much lower learning rate than before.
Two details mattered a lot here:
- I only unfroze block5 after the dense head was already well-trained from the earlier approaches. Unfreezing pretrained layers before the head is trained would send large, noisy gradients backward and disrupt the pretrained weights - essentially undoing the benefit of transfer learning.
- The low learning rate is deliberate. Fine-tuning pretrained weights with a normal learning rate risks catastrophic forgetting - wiping out the general knowledge the model already had.
Result: 99% train / 95% val accuracy - the best of all four experiments.
Results at a glance
| Approach | Type | Train Acc | Val Acc | Key takeaway |
|---|---|---|---|---|
| Custom CNN (dropout + batch norm) | From scratch | 87% | 76% | Overfitting still visible |
| VGG16 Feature Extraction | Transfer learning | 98% | 89% | Big jump, but wider train-val gap |
| Feature Extraction + Data Augmentation | Transfer learning | 93% | 92% | Overfitting nearly eliminated |
| Fine-tuning (unfreezing block5) | Transfer learning | 99% | 95% | Best result overall |
What I'd try next
- Swap VGG16 for a more modern backbone (ResNet50, EfficientNetB0) and compare
- Test the same pipeline with far less data, to see transfer learning's advantage more starkly in a genuinely low-data setting
- Try discriminative learning rates - smaller LR for early unfrozen layers, larger for later ones - instead of one flat rate during fine-tuning
Code
All four notebooks, plus the results and graphs above, are on GitHub:
👉 transfer-learning-cats-vs-dogs
If you're learning deep learning too, I'd genuinely love to hear what you'd try differently - drop a comment below.
Top comments (0)