Building a Real-Time Crowd Anomaly Detector with MobileNetV2 and OpenCV

Alugam Sahasra — Mon, 07 Sep 2026 16:49:00 +0000

I recently built a real-time video anomaly detection system that classifies footage as violent or non-violent using transfer learning - and wanted to share what I learned, including the environment issues that ate up more time than the actual model building.

The Problem

Crowd monitoring systems today mostly rely on manual observation - someone watching camera feeds and reacting after something's already gone wrong. I wanted to build something that could flag anomalous behavior automatically, in real time, without needing a massive custom-trained model from scratch.

The Approach

Rather than training a CNN from zero (which needs huge datasets and compute I didn't have), I used transfer learning with MobileNetV2, pretrained on ImageNet:

Froze the base MobileNetV2 layers to keep the pretrained visual features intact
Added a custom head: GlobalAveragePooling2D → Dense(512, relu) → Dropout(0.5) → Dense(1, sigmoid)
Trained as a binary classifier: violent vs. non-violent
Total params: 2.91M, of which only 656K were trainable (the rest frozen in the pretrained base)

Dataset: 2,543 training images and 635 validation images across the two classes, resized to 150×150.

Augmentation: rotation, shifts, shear, zoom, brightness jitter, horizontal flip - since real-world footage varies wildly in lighting and camera angle.

Training safeguards: early stopping (patience=5) and a learning rate scheduler (halving LR on plateau) to avoid overfitting.

Results

Validation accuracy: 94.65% (best epoch hit 95.43%)
Training accuracy: 97.32% - close enough to validation that overfitting wasn't a major issue
Trained for 27 epochs (early-stopped from a max of 30) in ~9 minutes on CPU
Final loss: 0.088 (train) / 0.143 (val)
Live webcam inference: ~135 ms/frame, ~6.6 FPS end-to-end (tested over 1,200+ live frames)
Model size: 16.5 MB - small enough to run on modest hardware, no GPU required

What Didn't Work / What I'd Change

Environment setup was harder than the ML itself. I hit a chain of dependency conflicts - TensorFlow failing due to Windows' 260-character path limit, then h5py needing an exact version range TensorFlow was compatible with, then a NumPy version too new for SciPy. Every fix surfaced the next one. Lesson: pin your dependency versions before starting, not after debugging blind for an hour.
~6.6 FPS on CPU is usable for monitoring, not for high-frame-rate applications. For genuine real-time deployment, I'd either quantize the model (TFLite) or run inference on a GPU.
The frozen backbone limits how specialized the features are to violence detection specifically - fine-tuning the last few MobileNetV2 layers instead of freezing all of them is the obvious next experiment.

What's Next