How We Trained a 90.6% mAP50 Valve Detection Model with Iterative Pseudo-Labeling

The Problem

Underground gas infrastructure inspection generates hundreds of thousands of photos. Each valve well contains multiple valves of different types — gate valves, globe valves, ball valves, and others. Manually classifying and cataloging these is slow, expensive, and error-prone.

We needed an AI model that could automatically detect and classify valves in inspection photos. The challenge? We started with only 30 manually annotated images and a budget of zero for professional annotation.

The Solution: Iterative Pseudo-Labeling

Instead of paying for annotations, we built a self-improving pipeline:

1. Start small: Train on 30 hand-labeled images
2. Pseudo-label: Use the current model to annotate unlabeled images
3. Filter: Keep only high-confidence predictions (≥0.5)
4. Retrain: Train a new model on the expanded dataset
5. Repeat: Each iteration improves the model, which improves the labels

Results After 10 Iterations

Round	Images	Model	mAP50	mAP50-95	Notes
R1	30	YOLOv8n	28.1%	—	Hand-labeled only
R3	368	YOLOv8n	50.9%	30.2%
R4	1,626	YOLOv8n	65.5%	49.6%
R6	2,007	YOLOv8s	70.8%	52.1%	Architecture upgrade
R7	3,937	YOLOv8s	80.5%	63.6%
R8	8,506	YOLOv8s	90.6%	68.9%	Previous best
R9	18,608	YOLOv8s	81.3%	65.0%	Too much noise!
R10	9,038	YOLOv8s	90.6%	79.3%	Filtered dataset

Key Lessons

1. Model Architecture Matters More Than Data Size (Early On)

Upgrading from YOLOv8n (3M params) to YOLOv8s (11M params) at Round 6 gave a +4.3% mAP50 boost — the single largest improvement from any single change. If your model is underfitting, more data will not help until you increase capacity.

2. Data Quality > Data Quantity

When we doubled the dataset from 8,506 to 18,608 images (R9), performance actually dropped from 90.6% to 81.3% mAP50. The culprit? Low-confidence pseudo-labels introducing noise.

The fix was counterintuitive: we removed 60% of the data. By filtering to confidence ≥0.5, we reduced the dataset to 9,038 images — and mAP50 jumped to 90.6% (R10). That is a +6.9% improvement over R8 with fewer images.

Rule of thumb: A smaller, cleaner dataset beats a larger, noisier one. Always filter pseudo-labels aggressively. In our case, cutting the dataset in half while raising quality gave the biggest single-round improvement in the entire project.

3. Pseudo-Labeling Has Diminishing Returns

The biggest gains came in the early rounds:

• R2→R3: +20.7% mAP50 (from 30.2% to 50.9%)
• R3→R4: +14.4% mAP50
• R7→R8: +3.2% mAP50
• R8→R10: +6.9% mAP50 (but only after fixing data quality)

Each doubling of data yields less improvement. Beyond ~10K images, you need fundamentally better annotations (human review) or better architectures to see significant gains.

4. Per-Class Analysis Reveals Bottlenecks

Class	mAP50	Precision	Recall	Issue
Globe Valve	94.7%	89.0%	86.6%	Excellent
Gate Valve	86.1%	89.9%	66.6%	Low recall — many missed
Ball Valve	81.1%	71.9%	75.8%	Confusion with gate valve
Other Valve	72.8%	66.9%	68.0%	Scarce samples (4%)

Gate valve has great precision but poor recall — the model is too conservative. Other valve has too few training samples. These insights guide where to invest annotation effort.

5. GPS Metadata is an Underused Asset

93% of our inspection images contained EXIF GPS data. This enabled us to build an interactive inspection map showing 650+ valve well locations across three Chinese provinces. For infrastructure companies, geospatial AI is a game-changer — detection results are not just labels, they are map pins.

What is Next

We are continuing to iterate:

• R11+: Class-specific augmentation targeting the Other Valve category (only 4% of data)
• Model upgrade: Testing YOLOv8m (25M params) if memory allows
• Production deployment: ONNX export for edge devices
• Active learning: Human review of borderline predictions to further improve label quality

Try It Yourself

The model is available on Hugging Face with a commercial license. Try the live demo — upload your inspection photo and see instant results.

If you are building AI for industrial inspection, reach out — I offer custom model development starting at $500.

---

Built on Apple M4 Mac Mini with PyTorch MPS acceleration. Total training time: ~50 hours across 10 rounds.