📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram
Originally published on software-engineer-blog.com.
A detector must emit a list of unknown length. A classifier cannot—one picture in, one label out, a fixed shape. This mismatch drove twenty-five years of object detection, and the field hit it four times in four different ways.
- Mental model: Each breakthrough solved a hard constraint the previous era exposed, then revealed a new one. The real pivot wasn't architecture—it was the loss function, and whether you remove duplicates before or after training starts.
Wall 1: The Sliding Window (1990s–2011)
You cannot slide a classifier everywhere—it's too slow and it says WHAT, not WHERE. So the early field asked the easy question: "Is there an object in this 32×32 patch?" Then asked it everywhere. A 640×640 image yields roughly 145,000 overlapping windows. For each one, humans hand-drew features (edges, histograms, gradients). Each window ran through a trained classifier. Each detection got a confidence score.
The cost was computational. The ceiling was accuracy: hand-drawn features lose to learned ones.
Wall 2: Learning Features, Keeping Regions (2012–2014)
2012: AlexNet proved a neural network could learn better features than humans could design. ImageNet error dropped from ~26% to ~15%. But AlexNet is still a classifier—it sees one 224×224 image, emits one label.
R-CNN's insight (2014): Use a region proposal algorithm to narrow the search space from 145,000 windows to ~2,000 candidate regions, then run AlexNet on each one. The regions came from selective search (a hand-tuned algorithm). The mAP on PASCAL VOC 2012 jumped from 35.1 to 53.3.
The cost was speed: two stages, two forward passes per region. The ceiling was latency—video and real-time inference stayed out of reach.
Wall 3: One Stage, One Forward Pass—and a Hard Trade-off (2015–2019)
YOLO (2015) solved speed: one forward pass, one output tensor. No region proposals. No selective search.
But here is what nobody says about one-stage detectors: they do not emit a list. They emit a fixed grid of candidate boxes. A modern YOLO at 640×640 resolution emits roughly 8,400 candidate boxes—always. Before it sees the image. The shape is baked into the architecture.
Most of those boxes describe the same dog. To filter them down, every detector for twenty years ended the same way: non-maximum suppression (NMS). Greedy loop:
# Pseudocode: NMS is not learned, not differentiable
keep = []
while boxes:
best = boxes.pop(highest_confidence)
keep.append(best)
boxes = [b for b in boxes if iou(best, b) < iou_threshold] # Hand-picked threshold
return keep
NMS has three fatal flaws:
- It is not learned. No gradient flows through it during training.
- The IoU threshold is a human-picked number, retuned per dataset.
- In a crowd, it deletes real people, because real people overlap.
YOLO v1–v5 and their descendants (SSD, RetinaNet, EfficientDet) all paid this bill. Faster inference came at the cost of a post-processing hack that was brittle and blind to the actual training signal.
Wall 4: Predict a Set, Not a Grid (2020–Present)
DETR (Detection Transformer, 2020) reframed the problem: you want a set out, so predict a set.
100 learned query slots. Each slot outputs one bounding box and one class label (or the special class "no object"). Always 100 queries. Always a hundred outputs.
The variable-length problem from wall one is gone—you have a fixed shape. But now you have a different problem: how do you train it? You have 100 predictions and maybe 3 real objects. Which prediction should match which object?
The breakthrough was the loss function, not the architecture. DETR uses the Hungarian algorithm:
# Pseudocode: Hungarian matching in the loss, not post-processing
cost_matrix = compute_costs(predictions_100, ground_truth_3) # 100 Ă— 3
best_matching = hungarian_algorithm(cost_matrix) # one-to-one pairing
# Train the 3 matched predictions to be correct.
# Train the 97 unmatched predictions to output "no object".
# A second query that also finds the dog is punished during training.
# Duplicates are never created.
No hand-picked IoU threshold. No greedy loop. No post-processing that ignores the loss. Duplicates do not get removed—they are never trained to exist.
The cost was high: DETR needed 500 epochs to converge (versus ~100 for YOLO), and small objects were harder to detect. The next four years of research paid down both.
Where It Landed (Mid-2026)
By mid-2026, both families had converged on the same architectural shape:
- One-stage detector (no separate region proposal network).
- No hand-tuned anchors.
- No NMS in the default pipeline.
YOLO26 ships NMS-free by default, using a Decoupled Head with soft spatial and class matching in the loss (inspired by DETR). RF-DETR (real-time DETR) is a transformer detector with architectural tricks (RoI queries, token pruning) to match YOLO's inference speed.
On the COCO benchmark, they tie:
| Model | COCO mAP | Parameters | Approach |
|---|---|---|---|
| YOLO26x | 57.5 | 55.7M | Grid-based, learned matching in loss |
| RF-DETR-L | 56.5 | 129M | Set-based, Hungarian matching in loss |
What still differs is the backbone: the feature extraction before the head. YOLO's backbone is trained on supervised detection data. RF-DETR's uses DINOv2, a self-supervised vision model trained on a billion unlabeled images. COCO is a tie because the head (grid vs. set) is now irrelevant—the bottleneck moved downstream.
This is why COCO scores alone are the wrong lens to pick a detector. You have to ask: which backbone suits your data and your label budget?
For LLM Inference: Why This Matters to Serving
If you serve detectors as part of a larger AI system (e.g., grounding for a vision-language model), the shift from NMS to learned matching changes latency profile:
- YOLO with NMS: Inference time is deterministic. NMS is O(n log n) on the number of detections, but it runs after the network, so you cannot batch or fuse it with the next module.
- DETR / NMS-free: All computation is in the neural network. Batch processing and quantization apply uniformly. No separate post-processing stage to stall on edge devices.
- Hungarian matching (training only): Only happens during training. At inference, you just pick the 100 predictions; no matching step. But the 100 queries are learned to avoid duplicates, so you spend fewer cycles filtering.
For real-time serving, prefer NMS-free detectors (YOLO26, RF-DETR) because they fuse into quantized execution graphs and avoid the post-processing bottleneck.
Verdict
Reach for YOLO when you need simplicity and real-time speed on single-backbone setups; reach for DETR (or RF-DETR) when you want to plug in a better backbone (like DINOv2) and have latency headroom for transformer compute. The head stopped mattering in 2024. The backbone and your label budget are what differ now.
Watch the 17-minute deep dive for the full chain, the numbers, and the trade-offs.
Top comments (0)