A pose model returns an ordered array of coordinates, one per named joint, each with a confidence. The ordering is the whole interface: the skeleton is not detected, it is drawn by your code from a fixed list of which index connects to which.
What the model returns
For a single person, the output is a fixed-length array. COCO’s keypoint format defines 17 joints in a fixed order — nose, left eye, right eye, left ear, right ear, then shoulders, elbows, wrists, hips, knees and ankles, left before right in each pair. Each entry is a triple: x, y and either a visibility flag in the annotations or a model confidence at inference time. In the annotation format the flag takes three values: 0 means the joint is not labelled, 1 means it is labelled but occluded, and 2 means labelled and visible. Those are not interchangeable, and evaluation treats a 0 differently from a 1.
The array is fixed length even when the person is half out of frame. There is always an entry for the left ankle; what varies is its confidence. So a consumer that draws a skeleton by connecting index pairs will happily draw a leg to a coordinate the model has no belief in, and this is the single most common reason pose overlays look broken. The edge list is also yours: COCO ships a skeleton definition, but nothing in the model output says index 5 connects to index 7.
Heatmaps, and the quantisation error nobody mentions
Almost no modern pose model regresses coordinates directly. It predicts one heatmap per joint — a low-resolution map whose value at each location is the likelihood the joint is there — and takes the argmax. Heatmaps are dramatically easier to train than direct regression because the loss is spatially local: a prediction three pixels off produces gradient at the right place, whereas a coordinate regression gives one scalar error with no indication of direction.
The cost is resolution. A network with an output stride of 4 on a 256×192 input crop produces 64×48 heatmaps, so one heatmap cell is four input pixels wide, and a bare argmax quantises every joint to a 4-pixel grid before anything else goes wrong. Implementations recover most of this by shifting the peak a quarter-cell toward its second-highest neighbour, or by fitting the local distribution properly as in the DARK decoding work. If you are comparing two pose implementations and one is mysteriously better at small people, the decode step is where to look first — it is invisible in the architecture diagram and worth several points of OKS.
Top-down and bottom-up
There are two ways to handle more than one person, and the choice is a throughput decision more than an accuracy one.
- Top-down. Run a person detector first, crop each person, resize the crop to a standard size, and run a single-person pose model on it. Accuracy is high because every person is normalised to the same scale. Cost scales linearly with the number of people, so a frame with 30 people costs 30 forward passes — and any person the detector missed has no pose at all, so detection recall is a hard upper bound on pose recall.
- Bottom-up. Predict every keypoint of every person in one pass over the whole image, then group them into people. Cost is constant in the number of people, which is what makes crowded scenes tractable at video rates. The difficulty moves entirely into the grouping: OpenPose introduced part affinity fields, which predict a vector field along each limb so that a candidate elbow and a candidate wrist can be scored on whether the field between them points from one to the other.
The failure modes differ accordingly. Top-down mixes people up when two overlap, because the crop for one contains most of the other and the single-person model assumes exactly one subject. Bottom-up produces anatomically impossible assemblies — one person’s arm attached to another’s torso — when limbs cross.
OKS, worked on a wrist and an eye
Object keypoint similarity is the pose analogue of IoU, and it is better designed than most metrics. For each labelled joint it computes a Gaussian of the distance error, scaled by both the size of the person and a per-joint tolerance, then averages over the labelled joints:
KS_i = exp( -d_i^2 / (2 * s^2 * k_i^2) ) k_i = 2 * sigma_i
OKS = mean over labelled joints of KS_i
s^2 = the person's segmented area in pixels
COCO's sigmas (from cocoeval.py, kpt_oks_sigmas / 10):
nose .026 eyes .025 ears .035 shoulders .079
elbows .072 wrists .062 hips .107 knees .087 ankles .089
Those seventeen constants are the interesting part: they were derived from how much human annotators disagree with each other about each joint. Everyone agrees where an eye is; hardly anyone agrees where a hip is, because it is inside the body. The metric therefore forgives an error on a hip that it punishes on an eye. Work it for a person occupying 12,000 pixels of segmented area, with a 20-pixel error:
s^2 = 12000 d = 20 d^2 = 400
wrist k = 2 * 0.062 = 0.124
denom = 2 * 12000 * 0.124^2 = 2 * 12000 * 0.015376 = 369.0
KS = exp(-400 / 369.0) = exp(-1.084) = 0.338
eye k = 2 * 0.025 = 0.050
denom = 2 * 12000 * 0.0025 = 60.0
KS = exp(-400 / 60.0) = exp(-6.667) = 0.00127
The same 20-pixel error scores 0.338 on a wrist and 0.00127 on an eye — a factor of 266. That is the metric behaving correctly, and it is why a model tuned for face-adjacent applications and one tuned for limb tracking can have similar mean OKS while being useless at each other’s job. AP for keypoints is then computed over OKS thresholds exactly as box AP is over IoU thresholds, with maxDets of 20 rather than 100, again from cocoeval.py.
Note also the s² term. Because the tolerance scales with the person’s area, the pixel error a small distant person is allowed is proportionally smaller. A model that looks fine on close subjects and scores badly overall is usually failing on the far ones, and a per-scale breakdown will show it immediately.
Using per-joint confidence properly
Take a returned pose with these confidences: shoulders 0.94 and 0.91, elbows 0.88 and 0.85, left wrist 0.79, right wrist 0.21, hips 0.90 and 0.89, knees 0.83 and 0.81, ankles 0.72 and 0.14. The mean over twelve joints is 0.74, which reads as a decent pose and conceals the two numbers that matter: the right wrist and right ankle are essentially not detected, most likely because that side of the body is occluded.
So never average. Gate each joint independently at a threshold, and draw a bone only when both of its endpoints pass — that single change removes most of the visual noise people try to solve with temporal smoothing. Then decide, for your application, what a missing joint means: a repetition counter that needs one wrist can carry on, and a joint-angle measurement that needs a shoulder, elbow and wrist must refuse to produce a number rather than produce a wrong one. The heatmap peak is also an uncalibrated score in the same sense discussed in classifier confidence calibration, so a threshold of 0.3 on one implementation is not a threshold of 0.3 on another.
Where 2D pose stops being enough
A 2D skeleton is a projection, and projection destroys depth irrecoverably. An arm pointing at the camera and an arm folded against the chest can produce nearly identical 2D coordinates, so any angle you compute in the image plane is wrong by an unknown amount whenever the limb is not roughly parallel to the sensor. Lifting 2D to 3D is possible and standard, but it is inference under ambiguity rather than measurement, and it inherits a well-known left-right depth flip that no amount of confidence in the 2D stage detects.
Two further limits are worth stating before anyone builds on this. The 17 COCO joints contain no spine, no hands beyond the wrist and no feet beyond the ankle, so anything about posture, grip or gait needs a different keypoint set and different training data. And frame-by-frame inference has no notion of continuity, so joints jitter between frames even when the subject is still — the fix is a temporal filter on the coordinates, chosen for the latency you can afford, not a better per-frame model.
Top comments (0)