Why 97% Accuracy is a Lie: My Journey from ANN to CNN

Nikita Gupta — Mon, 06 Jul 2026 15:34:15 +0000

You know the feeling. Your model trains perfectly, the validation loss drops, your accuracy hits a 97.5% on the MNIST dataset and it feels like it might work fine.

Then you build a real-time UI, test it yourself, and your model completely falls apart.

That is exactly what happened with my latest project: a real-time handwritten digit recognizer built with PyTorch and Pygame.Here is how my "perfect" Artificial Neural Network (ANN) failed in the real world, and the engineering steps I took to fix it.

The Setup
The goal was simple. Build a black Pygame canvas where a user can draw a number (0-9) with their mouse, and have a PyTorch model predict the number the moment they release the click.

I started with a standard flattened ANN. It trained. But the second I drew a number on the screen, it guessed wrong almost every time.Especially numbers like 6,7,8,9.

The Problem: Domain Shift
The issue was Domain Shift.

An ANN flattens a 28x28 image into a 1D array of 784 pixels. It doesn't actually understand shapes; it just memorizes where bright pixels should be.

The Dataset: The MNIST training data has perfectly centered, thin pen strokes.
The Real World: My Pygame digital brush was thick. My hand jittered. The numbers weren't perfectly centered.

Because the lines were thicker, the pixels didn't perfectly align with the ANN's expectations, and it completely failed to recognize the numbers.

The Experiment: The "Model Arena"
To fix this, I decided to upgrade to a Convolutional Neural Network (CNN), which actually looks for edges, curves, and loops using spatial filters.

But I wanted to prove it was better. So, I built a Model Arena: a script that ran both the ANN and the new CNN side-by-side on the exact same drawing. I even built an Ensemble feature to average their probabilities.

The result? The CNN broke too. It started guessing the number 8 for almost everything.

The Fix: Normalization & Optimization
It turns out, the architecture wasn't the only issue. It was the math.

During training, I normalized the PyTorch dataset to a range of [-1, 1]. But my Pygame image pre-processing was scaling the pixels from [0, 1]. The CNN was receiving numbers it didn't expect, panicking, and defaulting to 8 (the mathematical center of the dataset).

Once I spotted the bug, the fix was incredibly satisfying. I updated the Pygame tensor math.
I also reduced the brush thickness to mimic a real pen, and dropped the random rotation in my training data augmentation from 15° to 5° so the shapes wouldn't distort too heavily.

The Takeaway
After those fixes, the CNN's accuracy skyrocketed. It easily recognized my live handwriting, completely outperforming the ANN (which I ended up deleting entirely).

The biggest lesson? A high accuracy score on a pristine dataset means absolutely nothing if your live environment doesn't perfectly match your training environment.

You can check out the final, fully optimized CNN architecture on my GitHub:

nikitagpt05 / Handwritten-Digit-Recognizer

Handwritten Digit Recognizer (PyTorch & Pygame)

This is a real-time app that guesses the numbers you draw on the screen. It is built with PyTorch and Pygame. This project shows how a machine learning model can perform perfectly on standard training data but struggle with real-world inputs, and how changing the underlying architecture fixes those issues.

How the Project Evolved

I went through three main testing phases to get the accuracy right for live, interactive drawings.

1. The First Attempt (ANN) I started by building a basic Artificial Neural Network (ANN). It scored about 97.5% accuracy on the standard MNIST dataset. But when I drew numbers with my mouse in the actual app, it failed constantly. An ANN flattens an image into a single line of pixels, so it does not actually understand shapes, edges, or curves. If I used a thick digital brush or drew the number slightly off-center…

View on GitHub

Have you ever had a model fail in production after passing all your tests? Let me know in the comments!

DEV Community: Nikita Gupta

Why 97% Accuracy is a Lie: My Journey from ANN to CNN

nikitagpt05 / Handwritten-Digit-Recognizer

Handwritten Digit Recognizer (PyTorch & Pygame)

How the Project Evolved