Deep Learning for Image Classification: Ensemble CNN Architectures with Test Time Augmentation

Accuracy: 0.99628 · Rank: 72 / 1,181 · Kaggle Digit Recognizer Competition

Problem Framing

MNIST is a solved problem in the academic sense — state-of-the-art models have exceeded human-level performance on it for years. The challenge in a competitive context is not whether a CNN can classify handwritten digits, but how much variance you can squeeze out of an already high-performing system when the ceiling is 1.0, and the marginal gains are measured in the fourth decimal place.

At 99.6%+ accuracy, a single misclassified digit per 200 samples is the difference between medal territory and the middle of the leaderboard. The solution presented here addresses this precision problem through two compounding mechanisms: ensemble diversity to reduce variance across model architectures, and Test Time Augmentation (TTA) to reduce variance across the inference distribution. The combination pushed a single-model baseline of 0.995035 to a final leaderboard score of 0.99628.

Data Preprocessing Pipeline

The preprocessing pipeline is intentionally minimal — MNIST's controlled acquisition conditions mean aggressive preprocessing adds noise rather than signal.

Normalisation scales pixel intensities from the raw [0, 255] integer range to [0.0, 1.0] float32. This is not optional — unnormalised inputs cause gradient instability during early training epochs, particularly with BatchNormalisation layers in the network.

Reshaping transforms the flat 784-dimensional pixel vectors into (28, 28, 1) tensors. The channel dimension is explicit even though MNIST is grayscale — omitting it causes shape mismatches in Conv2D layers expecting a channel axis.

Label encoding uses one-hot encoding across 10 classes, compatible with categorical cross-entropy loss.

No augmentation is applied at the preprocessing stage — augmentation is handled dynamically during training via Keras ImageDataGenerator, keeping the validation set clean and unperturbed for honest accuracy measurement.

Ensemble Architecture Design

The core hypothesis driving the ensemble design is that architecturally diverse models make uncorrelated errors. When models fail on different samples, their averaged softmax outputs smooth over individual weaknesses. Five CNN architectures were selected to span a range of representational depths and widths:

CNN-A and CNN-C (2-Block Shallow)
Conv2D(32) × 2 → MaxPool → Conv2D(64) × 2 → MaxPool → Dense(256)
Shallow networks generalise quickly and act as low-variance baselines. Their representational capacity is sufficient for MNIST's relatively simple feature hierarchy.

CNN-B and CNN-E (3-Block Deep)
Conv2D(32) × 2 → MaxPool → Conv2D(64) × 2 → MaxPool → Conv2D(128) × 2 → MaxPool → Dense(512)
The additional convolutional block captures higher-order spatial relationships — stroke intersections, curve terminations — that shallower networks approximate less precisely.
CNN-D (Wide 2-Block)

Conv2D(64) × 2 → MaxPool → Conv2D(128) × 2 → MaxPool → Dense(512)
Wider early filters increase the number of low-level feature detectors without adding depth. This architecture is architecturally distinct from both A/C and B/E, contributing a different error profile to the ensemble.

Regularisation applied uniformly across all five architectures:
BatchNormalisation after each convolutional block — stabilises activations and reduces sensitivity to weight initialisation

Dropout(0.25) after convolutional blocks, Dropout(0.5) before the final dense layer — independently drops units during training to prevent co-adaptation, MaxPooling2D for spatial downsampling and translation invariance

The choice to duplicate architectures (A/C, B/E) rather than use five entirely distinct designs is intentional — identical architectures trained from different random initialisations with different augmentation sequences produce meaningfully different weight configurations and therefore different error patterns.

Training Strategy

1. Optimiser: Adam with default parameters. Adaptive learning rate methods consistently outperform SGD with momentum on vision tasks at this scale without requiring manual schedule tuning.
*2. Real-time Data Augmentation via ImageDataGenerator: *

• Rotation range (15°): Simulates natural variations in handwritten digits
• Zoom range (15%): Accounts for differences in digit size/scale
• Width shift range (15%): Handles horizontal misalignment or centering variation
• Height shift range (15%): Handles vertical misalignment or centering variation
• Shear range (0.1): Simulates slight perspective distortions

Augmentation is applied on-the-fly during training — each epoch sees a stochastically transformed version of the training set, effectively expanding the training distribution without increasing dataset size. This is the primary mechanism preventing overfitting across 50 training epochs.

3. Callbacks:
ReduceLROnPlateau monitors validation accuracy and halves the learning rate when no improvement is observed over a patience window. This allows the optimiser to take larger steps during early training and finer steps as it approaches the loss minimum — recovering accuracy that flat learning rate schedules leave on the table.

EarlyStopping with restore_best_weights=True terminates training when validation accuracy plateaus and restores the checkpoint from the optimal epoch. This is critical in an ensemble context — each of the five models must contribute its best possible weights, not its final weights.

4. Inference: Test Time Augmentation

A trained model's prediction on a single test image is a point estimate — one forward pass through a stochastic function approximator. TTA converts this point estimate into a distributional estimate by averaging predictions across multiple augmented versions of the same image.

5. TTA protocol:

• For each test image, generate 15 augmented variants using the same ImageDataGenerator configuration used during training
• Run each variant through each of the 5 trained models
• Collect softmax probability vectors from all 80 forward passes (5 models × 16 passes: 1 original + 15 augmented)
• Average the 80 probability vectors element-wise
• Select the class index with the highest averaged probability as the final prediction

The mathematical intuition: if a model misclassifies an augmented variant of a digit, the correct class still accumulates probability mass across the other 79 passes. The averaging operation suppresses low-confidence noise and amplifies the consistent signal.

6. Quantified impact of each component:

• Single CNN baseline: ~99.54% validation accuracy, 0.99503 leaderboard score
• 5-model ensemble + TTA: ~99.57% average validation accuracy, 0.99628 leaderboard score

The leaderboard gain of +0.00125 over the single-model baseline — while appearing marginal — represents a reduction of roughly 1 misclassification per 800 test samples. At this accuracy regime, that is the practical limit of what ensemble diversity and distributional inference averaging can recover.

Execution

bash# 
Download competition data
kaggle competitions download -c digit-recognizer

# Train all five CNN architectures
python train_ensemble.py

# Run TTA inference and generate submission
python predict_ensemble.py

Analysis: Where the Remaining Errors Live
At 0.99628, the residual errors are concentrated in a small set of structurally ambiguous digit pairs — primarily (4, 9), (3, 5), and (7, 1) — where stroke topology is genuinely similar and the distinguishing feature is a single curve or termination point. These are the cases where even human annotators disagree at non-trivial rates.

Pushing beyond this threshold would require either capsule networks (which preserve spatial hierarchies that MaxPooling discards), or a larger training set sourced from distributions beyond MNIST's controlled acquisition environment. Within the constraints of this competition, 80-pass TTA across a 5-model ensemble represents a practical ceiling.

Key Findings

Three methodological conclusions generalise beyond MNIST:

• Architectural diversity outperforms depth uniformity in ensembles. Five architecturally varied models with moderate depth outperform five deep identical models — the variance reduction mechanism requires uncorrelated errors, which requires architectural differences.
• TTA is a zero-cost accuracy gain at inference time. Once models are trained, TTA costs only additional forward passes. On MNIST-scale images this is computationally trivial. On larger datasets the compute cost scales proportionally with image size and TTA count — budget accordingly.

EarlyStopping with weight restoration is non-negotiable in multi-model ensembles. Final-epoch weights frequently underperform best-epoch weights by 0.1–0.3% validation accuracy. Across five models this compounds — the ensemble ceiling is only as high as the best checkpoint of each contributor.

Full implementation available at: github.com/faissssss/kaggle-digit-recognizer
Source: kaggle.com/c/digit-recognizer