I'm building GroundCheck, an offline-first field inspection app for construction safety managers. One of the things we bet on early was on-device hazard detection — a YOLO model that runs entirely on the phone, no internet required, spotting missing hardhats and safety vests in a photo the moment you take it. No competitor in this space does that; everyone else ships cloud-only detection that falls over the second you lose signal, which on a job site is often.
This is the story of a metric that looked great, was actually badly broken, and what it took to find out — plus a training run that got derailed twice by things that had nothing to do with the model.
A good score, hiding a broken detector
Our first real model, trained on a 10-class taxonomy (hardhats, vests, masks, cones, and a few others), hit 0.510 mAP50. That cleared our internal bar of 0.50. Model shipped, feature declared done, on to the next thing.
Then I ran it against a simple test photo — six construction workers, standard job-site shot, some in hardhats, all in hi-vis vests. The model found zero vests. Not "low confidence," zero. It also hallucinated a hardhat on someone who wasn't wearing one.
That's a bad look for a safety app, and it directly contradicted the "0.510, above target" number sitting in my training log. So what happened?
mAP50 is a metric that's easy to be misled by, if you only ever look at the macro-average. Ours was averaged across 10 classes with wildly different underlying performance:
| Class | mAP50 |
|---|---|
| Hardhat | 0.924 |
| Face-guard | 0.850 |
| Safety Vest | 0.343 |
| Mask | 0.267 |
A model that nails hardhats and face-guards can drag a broken vest detector up to a respectable-looking average. The number wasn't wrong, exactly — it just wasn't answering the question I actually cared about, which was "does this work for the classes that matter most."
The lesson: a single aggregate metric is a summary, not a verdict. If your classes aren't roughly equally easy and equally important, look at the per-class breakdown before you trust the headline number — and if you can, run the model against a real example and look at it. The zero-vest failure took me thirty seconds to spot with my own eyes; it was invisible in the metric that was supposed to be measuring exactly that.
Fixing it
Two changes: I dropped a pair of classes (NO-Safety Vest / NO-Mask) that had too little consistent training data to ever work — the app now infers "no vest" from a person detected with no vest box overlapping them, rather than trying to detect the absence directly, which turned out to be a much harder learning problem than it needed to be. Then I went and found three new open datasets specifically to beef up the underrepresented classes, roughly tripling the validation volume for the worst offenders.
Retrained. Same architecture (YOLOv8s), same target: beat 0.510, and this time, verify the classes that actually matter.
The training run that fought back
This part has nothing to do with the model and everything to do with running long training jobs on hardware that occasionally has other ideas. Two things went wrong that weren't ML problems at all:
First, the machine rebooted mid-training (unrelated maintenance), killing the process at epoch 134 of a planned 200. Recoverable — checkpoints survive a kill, just resume from last.pt.
Second, and more interesting: the auto-resume script I'd written specifically to handle crashes like that had a bug. yolo detect train resume=True model=<checkpoint> is supposed to pick up exactly where you left off. In my environment, it silently didn't — it fell back to Ultralytics' built-in 4-image toy dataset instead of my real 32,000-image training set, and happily reported success. No error, no warning. Just a training run quietly doing nothing useful for however long it took me to notice the loss curves looked suspiciously clean.
I caught it because the process count looked wrong — one training job should mean one Python process, and I was staring at twenty-five. Followed that thread back to the argument parser silently discarding my dataset config and substituting its own defaults.
The fix that actually worked: stop trusting the "smart" resume, and do a plain, boring, explicitly-specified restart every time — spell out every argument, verify the log shows my actual dataset path before walking away, don't rely on a flag that's supposed to infer everything for you. Boring and verified beats clever and silent, every time a background process fails quietly instead of loudly.
Knowing when to stop
The retrained model plateaued around epoch 177 at 0.682 mAP50, still short of a stretch goal of 0.80. I tried a short "continuation" run — warm-starting a fresh 40-epoch pass from the final weights, hoping for a few more easy points.
It made things worse, immediately and consistently — accuracy dropping every single epoch. Resetting the full learning-rate schedule on top of a model that had already converged over 178 epochs was too big a shock; instead of fine-tuning, it started forgetting. Killed it after 4 epochs once the trend was unambiguous, and kept the original checkpoint. Sometimes the right move with a training run — like with a lot of optimization problems — is recognizing you've already found a good stopping point, and further tinkering is more likely to hurt than help.
Validating for real, this time
Same two checks as before, run properly this time:
Per-class mAP50 on the full validation set:
| Class | mAP50 | Before |
|---|---|---|
| Hardhat | 0.855 | 0.924 |
| Safety Vest | 0.684 | 0.343 |
| Mask | 0.555 | 0.267 |
| vehicle | 0.425 | (new weakest) |
The test photo. Same six workers as before. This time: 4 of the visible vests correctly detected, all 3 worn hardhats correctly detected, bare heads correctly flagged. Not perfect — a couple of vests still missed, confidence scores are moderate rather than sky-high — but a real, visible fix. The classes that were quietly broken before are now genuinely functional.
The takeaway
Two unglamorous habits saved this retrain from shipping a broken detector twice: refusing to trust a single aggregate metric without a per-class breakdown, and refusing to trust an automated recovery mechanism without checking, concretely, that it actually did what it claimed to do. Neither is exciting. Both are the difference between a safety feature that works and one that just looks like it does in a dashboard nobody double-checked.
GroundCheck is an offline-first inspection app for construction safety managers, built around on-device AI hazard detection that works with zero connectivity — because job sites don't always have signal, and safety checks shouldn't wait for a cloud API.
Top comments (0)