Pose estimation has made sports video analysis look much easier than it really is.
Give a model a phone video of a tennis player, and it can return a set of body landmarks: shoulder, elbow, wrist, hip, knee, ankle. Run that across frames, and you get a moving skeleton. Add a few overlays, and suddenly the system appears to “understand” the athlete.
But keypoints are not understanding.
They are coordinates.
They tell us where parts of the body appear to be. They do not automatically tell us whether the athlete was balanced, late, loaded, recovering, drifting, or ready for the next ball.
For sports AI, this distinction matters. A tennis player does not lose a point because a wrist coordinate moved from (x1, y1) to (x2, y2). A player loses the point because the body failed to organize itself in time: the split step was late, the hips opened too early, the recovery step lagged, or the player contacted the ball while already falling out of position.
That is why a useful sports video system needs more than a pose layer.
It needs a movement state layer.
The pose layer is only the beginning
Most computer vision pipelines for sports video start with something like this:
video
-> frame extraction
-> pose estimation
-> keypoints
-> visualization
This is a good start. It gives the system a structured representation of the athlete instead of only raw pixels.
But this pipeline still treats each frame too independently.
In tennis, the interesting signal is rarely inside one frame. It is usually distributed across a short sequence:
prepare -> load -> rotate -> contact -> recover -> reset
A single frame can show that the player’s knees are bent. It cannot fully explain whether the player loaded early enough, transferred weight efficiently, or recovered in time for the next shot.
That requires temporal interpretation.
The system needs to ask questions like:
- Did the player establish balance before the swing?
- Did the lower body start the movement before the upper body?
- Did the player contact the ball while moving forward, sideways, or backward?
- How long did recovery take after contact?
- Was the player ready for the next ball?
These are not raw keypoint questions. They are movement state questions.
Coordinates need to become events
A better pipeline does not stop at pose estimation.
It turns coordinates into events.
video
-> keypoints
-> normalized body geometry
-> temporal features
-> movement events
-> movement states
For example, a tennis swing may contain events such as:
split step
unit turn
maximum racket preparation
forward acceleration
contact
follow-through
first recovery step
neutral reset
Some of these events are visible in the racket. Some are visible in the torso. Some are visible in the feet. Some are not obvious unless the system compares multiple frames before and after contact.
This is where sports video analysis becomes different from simple pose visualization.
A pose overlay can show the body.
An event layer can explain what the body is doing.
Why tennis is a good stress test
Tennis is a difficult sport for video understanding because the meaningful movement is fast, asymmetric, and context-dependent.
The player may hit the same forehand in several different body states:
balanced forehand
late forehand
open-stance forehand
defensive forehand
running forehand
recovery forehand
To a ball-tracking system, these may all look like successful shots if the ball goes in.
To a movement system, they are not the same.
A clean-looking shot can still be a bad movement pattern if the player is late, unstable, or unable to recover. The ball may cross the net, but the body may already be losing the next point.
This is the gap between shot analysis and athlete analysis.
Shot analysis describes the result.
Athlete analysis describes the body process that produced the result.
From keypoints to movement states
A movement state is a higher-level interpretation of body organization across time.
Instead of storing only:
{
"left_knee": [312, 640],
"right_knee": [420, 638],
"left_hip": [300, 520],
"right_hip": [430, 522]
}
the system should eventually infer something closer to:
{
"phase": "pre-contact",
"balance": "drifting_left",
"load": "late",
"rotation_sequence": "upper_body_first",
"recovery_risk": "high",
"next_ball_readiness": "low"
}
This second representation is not a replacement for keypoints. It is built on top of them.
The keypoints are the measurement substrate.
The movement state is the usable signal.
The hard part: normalization
Phone sports video is messy.
The camera angle changes. The player size changes. The court position changes. Lighting changes. The video may be filmed from behind, from the side, or from a diagonal angle. The player may be partially occluded.
That means raw pixel coordinates are often not enough.
Before building movement states, the system usually needs normalization layers such as:
body-relative coordinates
joint angle estimation
torso orientation
stance width
center-of-mass proxy
court-relative direction
frame-to-frame velocity
phase-relative timing
The goal is not just to know where the wrist is on the screen.
The goal is to understand what the wrist, torso, hips, legs, and feet are doing relative to the athlete’s own movement structure.
This is why sports AI cannot be only a detection problem. It becomes a representation problem.
For a deeper technical framing of single-camera movement reconstruction, we published a white paper on SpatialForm’s approach to phone sports video and 3D kinematic representation:
SpatialForm Single-Camera 3D Kinematics White Paper
A simple conceptual pipeline
A practical movement-understanding system might look like this:
1. Detect athlete body landmarks
2. Smooth and validate keypoints across frames
3. Normalize body geometry
4. Estimate temporal features
5. Detect movement phases
6. Identify movement events
7. Infer movement states
8. Present the result in a reviewable form
The last step matters.
If the system only outputs abstract numbers, most athletes and coaches will not use it. The result needs to be inspectable. The user should be able to see the movement, pause it, compare phases, and understand why the system labeled a state as late, unstable, or not ready.
In other words, the output should not be only a score.
It should be a movement representation.
This is also why we use the term Performance Form to describe the intermediate representation between raw video and athlete-side movement understanding.
Why “readiness” is more useful than isolated technique labels
Many sports AI products try to label isolated technical errors:
elbow too low
knee angle too small
shoulder rotation insufficient
These labels can be useful, but they are often too static.
In real play, the more important question is usually:
Was the athlete ready for the next action?
For tennis, this is especially important.
A player can hit a technically acceptable shot and still fail to recover. A player can make contact cleanly and still be late for the next ball. A player can win one stroke and lose the rally because the body never returned to a ready state.
That is why movement understanding should include recovery and next-action readiness, not only contact mechanics.
The point is not to judge a frozen pose.
The point is to understand the movement cycle.
Keypoints are the alphabet, not the language
Pose estimation gives sports AI an alphabet.
But an alphabet is not a sentence.
A sequence of body landmarks becomes useful only when the system learns how those landmarks form movement events, phases, and states.
For tennis video analysis, the real value is not simply drawing a skeleton on top of a player. The value is turning ordinary phone video into a structured view of how the athlete prepared, moved, hit, recovered, and reset.
That middle layer is where the hard work begins.
At SpatialForm, we describe this kind of intermediate representation as Performance Form: a reviewable movement structure built from phone sports video, designed to make athlete-side movement easier to inspect.
For readers interested in the tennis-specific side of this problem, we also wrote more about tennis swing analysis from phone sports video and the technical foundation behind single-camera 3D kinematics.
The broader lesson is simple:
Sports AI should not stop at seeing the body.
It should help us understand the movement behind the shot.
Top comments (0)