Counting by detection works beautifully until the objects get close together, and then it does not degrade gracefully — it falls off a cliff at a spacing you can compute in advance from your object size and your suppression threshold.
Three ways to count, and what each caps out at
There are only three mechanisms in general use, and they fail at different densities. Detection and count the boxes gives you a location and a class for every object, which is what you want when the count is not the only output; it is limited by suppression and by a per-image detection cap. Segment and count connected components works when objects do not touch and fails completely when they do, for the reason worked out in semantic versus instance segmentation. Regress a density map and integrate it gives only a number and a rough spatial distribution, and it is the only one of the three that still works in a crowd.
Choosing between them is not a question of accuracy in the abstract. It is a question of what your maximum density is, and the next section turns that into arithmetic.
The density at which detection breaks, in pixels
Take a fixed camera producing 1280×720 frames, and objects that appear as roughly 40×40 pixel squares — bottles on a conveyor, say. Two neighbouring boxes offset horizontally by dx pixels overlap by (40 - dx) × 40, so:
box side 40 px, area 1600 px^2 each
dx = 32 inter = 8*40 = 320 union = 3200 - 320 = 2880 IoU = 0.111
dx = 24 inter = 16*40 = 640 union = 3200 - 640 = 2560 IoU = 0.250
dx = 20 inter = 20*40 = 800 union = 3200 - 800 = 2400 IoU = 0.333
dx = 14 inter = 26*40 = 1040 union = 3200 - 1040 = 2160 IoU = 0.481
dx = 12 inter = 28*40 = 1120 union = 3200 - 1120 = 2080 IoU = 0.538
objects that fit across a 1280 px frame at each spacing
dx = 32 -> 40 per row
dx = 20 -> 64 per row
dx = 14 -> 91 per row
dx = 12 -> 106 per row
With a non-max suppression threshold of 0.5, everything down to a spacing of 14 pixels survives and anything at 12 pixels or tighter has its neighbour deleted. So on this camera, with these objects, detection counts correctly up to about 91 objects per row and then starts losing roughly every second object — not gradually, but as a step. Below the critical spacing the count does not drift low, it approximately halves, and then quarters as the spacing tightens further and each surviving box suppresses two neighbours.
Two things follow that are worth acting on. First, the ceiling is a property of pixel size, not of object size in the world: moving the camera back so the bottles are 20 pixels across halves every number above and halves your usable density. Second, lowering the NMS threshold to keep tight neighbours is exactly the move that starts producing duplicates on every isolated object, for the reason worked out in the non-max suppression pass. The genuine fixes are more resolution, a suppression rule that decays rather than deletes, or a detector trained with a set-prediction loss that has no suppression step. If your problem is objects hidden behind each other rather than merely adjacent to each other, that is a different failure again — see detecting occluded objects.
The other ceiling: maximum detections per image
There is a second, entirely artificial cap that silently truncates counts, and it catches people constantly because it produces a plausible number rather than an error. Almost every detection pipeline keeps only the top-scoring K boxes per image after suppression.
COCO’s own evaluation protocol uses maxDets of [1, 10, 100], with 100 as the headline setting — visible in pycocotools’ cocoeval.py. A model evaluated under that protocol has never been scored on its ability to find the 101st object, so a benchmark AP tells you nothing about counting 400 items. Inference frameworks carry their own limit, usually a parameter named something like max_det or max_detections, and the default is often a few hundred. If your counts plateau suspiciously near a round number, that parameter is the first thing to check, before the model.
Density regression removes the ceiling
The crowd-counting literature takes a different route entirely. Instead of predicting objects, predict a density map: an H × W real-valued image whose integral over any region is the number of objects in that region. Training labels are constructed by placing a Gaussian at each annotated object centre and normalising each to sum to 1, so that the whole map sums to the true count. The model is trained with a pixel-wise regression loss, and the predicted count is simply the sum of the output.
Because there is no discrete object anywhere in that formulation, there is nothing to suppress and no per-image cap. Two objects two pixels apart contribute two units of mass that add rather than compete. This is the architecture behind MCNN and CSRNet and the reason counts of many thousands in a single frame are tractable at all.
The costs are real and you should take them seriously before switching. You get no boxes, so nothing downstream can act on an individual object. You need dot annotations — one click per object — which is cheaper per object than boxes but still expensive at crowd scale. And the output is not integral: a frame with three objects can return 2.7, and rounding is your problem. Most importantly the Gaussian’s width is a hyperparameter that encodes an assumption about object scale, so a scene with objects near and far needs either a perspective-aware kernel or a multi-scale architecture.
Why a vision-language model miscounts
Asking a general vision-language model “how many bottles are in this image” is the cheapest thing to try and it degrades earlier than either method above. The mechanism is worth knowing because it predicts exactly when.
These models do not look at your image. They look at a resized copy of it, cut into fixed patches — a vision transformer with 14-pixel patches over a 336×336 input has 576 patches, and each becomes one token. A 1280-pixel-wide photograph downscaled to 336 shrinks everything by a factor of 3.8, so a 40-pixel bottle becomes about 10 pixels, less than one patch. At that point the object is not small in the model’s view; it does not have a token of its own at all. Tiling schemes that process several crops at native resolution push the limit out but do not remove it. On top of that, the counting itself happens in the language head, which has no accumulator — it is producing a plausible number token, not incrementing a register, which is why answers cluster on small integers and round numbers. The general shape of that problem is covered in why vision models describe things that are not there.
If you do route counting prompts to a hosted vision model, the per-provider image handling is the variable that changes the answer, not the prompt: providers differ on maximum input resolution, on whether an image is tiled or downscaled to a single square, and on how those tiles are billed as tokens. Sending the same photograph to two providers can silently deliver two different effective resolutions. Multigrid’s single API keeps the request identical across providers and reports the image tokens each one actually charged, which is the number that tells you whether your small objects survived.
Count error is not mAP
If counting is the deliverable, report counting metrics. The crowd literature uses mean absolute error and root mean squared error on the count itself, over a test set of images: MAE tells you the typical miscount, RMSE punishes the occasional catastrophic frame, and the gap between them tells you whether your errors are uniform or spiky.
mAP is the wrong headline for this task in a specific way. A detector that finds every object but places every box a little loosely scores poorly on mAP at high IoU thresholds and counts perfectly. A detector that places beautiful boxes on 60% of the objects scores well at IoU 0.75 and undercounts by 40%. Track both if you like, but the number that goes on the dashboard should be the one that matches the decision being made, and for counting that is error in units, at the densities you actually see, measured against a human count of the same frames.
Top comments (0)