DEV Community

Cover image for Overfitting, visually: zero training errors, and accuracy fell from 98.6% to 85%
Syed Abdul Hanan for Logit Solutions AI

Posted on AI-assisted

Overfitting, visually: zero training errors, and accuracy fell from 98.6% to 85%

Deep Learning, Visually #21

Watch the Short: youtube.com/shorts/K_XwgYYtXSM

A network that gets every training example right sounds like a success. It can be the opposite. In this article we train a real network, stop it at two moments, and compare them on data it has never seen.

The setup

The data is two classes shaped like crescents: 110 training points on a plane. We flip the labels of ten of them on purpose. Those ten are noise, the kind of error any real dataset has.

To judge the network, we keep 24 new points aside. It never trains on them. Their labels are clean.

The training data: filled and hollow dots are the two classes, and the ten rings mark the wrong labels.

One neuron is too simple

One neuron can only draw a straight line. Trained for 3,000 steps, it ends with 19 training errors, and it gets 3 of the 24 new points wrong. The crescents curl around each other, so no straight line can separate them.

That model isn't memorizing anything. It just can't follow the shape.

One neuron: a straight line, 19 training errors, 3 new points wrong.

A big network learns the real shape first

Now take a network with three hidden layers of 128 neurons each. That is 33,152 weights to fit 110 points, far more than it needs. We train it with Adam for 2,500 steps, and we look at it along the way.

By step 70, its line follows the two crescents. It has 9 training errors, and all 9 are wrong labels. It gets all 24 new points right. On a larger check, 5,000 more new points, its accuracy is 98.6%. This is as good as this run ever gets.

Step 70: the line follows the crescents. The nine training errors are all wrong labels, and no new point is wrong.

Keep training, and it memorizes the noise

Nothing stops the training at step 70, so gradient descent keeps working on those last nine points. To get them right, the line has to curl around each one.

By step 445 there are no training errors left: a perfect training score. By step 628, 5 of the 24 new points are wrong. At step 2,500, the same 5,000 points score 85.0%.

The network didn't forget the crescents. It added a detail for every wrong label, and those details are wrong everywhere else. This is overfitting: the network has memorized the training data, noise included, and does worse on new data because of it.

Step 2,500: no training errors, and five new points wrong. The curve snakes around every wrong label.

Seeing where it fails

The 5,000 points show this directly. In the next picture, the white dots are the ones the network gets wrong. At step 70 there are about 70 of them, close to the line. At step 2,500 there are about 750, in whole patches around the detours it took for the wrong labels.

The 5,000 new points at step 2,500: the white ones are wrong, 15% of them.

A note on the 24 points. Any sample that small is noisy, and we picked one whose counts sit near each model's true error rate, so the picture reads clearly. The 5,000 points are the honest measure.

Real networks do it too

The crescents are a toy, so here is a real run. Take 1,000 images of handwritten digits and a network with 26,432 weights, about 26 weights per image. Train it for 200 epochs with no protection against overfitting, and measure it on 10,000 images it never trains on.

  • Training accuracy reaches 100% at epoch 26 and stays there. Training loss falls to 0.0002.
  • Accuracy on new images reaches about 88% and stalls.
  • Loss on new images hits its lowest point, 0.42, at epoch 14, and then climbs to 0.81 while the training loss keeps falling.

The two loss curves tell you when it starts: from epoch 14 on, more training makes the training loss smaller and the loss on new images bigger.

Run on 1,000 digit images: the training loss falls to almost zero while the loss on new images stops falling and climbs.

What to take from this

A training score only tells you how well the network fits data it has already seen. With enough weights, it can get that score to perfect by memorizing, wrong labels included. So a perfect training score proves nothing on its own. Only new data can tell.

That raises the practical questions: how to measure on new data honestly, and how to stop a network from memorizing. Those are the next few topics in this series.

Try it

The same effect fits in six lines of NumPy. Eight noisy points lie along the line y = x, and a degree-7 polynomial has just enough room to pass through all of them, so its training error is zero.

import numpy as n
r=n.random.default_rng(8)
x=n.linspace(0,1,8)
y=x+r.normal(0,.1,8)
f=n.poly1d(n.polyfit(x,y,7))
print(f(.95).round(2))
Enter fullscreen mode Exit fullscreen mode

It prints 0.48. At x = 0.95, the real pattern says about 0.95. The curve fit every point it was given and missed the one it wasn't.

Key takeaways

  • Overfitting means a network does better and better on its training data while doing worse on new data.
  • It happens when a network has far more weights than it needs, and it gets worse with longer training.
  • The network often learns the real pattern first, then memorizes the noise.
  • A perfect training score can come with a worse network: here, 98.6% on new points fell to 85.0% as training errors fell to zero.
  • Always measure on data the network never trains on.

References

  • CS878 Deep Learning (Fall 2023), Lecture 12–13: Regularization & Transfer Learning.
  • Logit Solutions AI, Neural Networks From Scratch (full lecture), Chapter 14: Generalization. https://youtu.be/uYVk4iFb4uQ

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Top comments (0)