DEV Community: Nous Technology Limited

Keypoints Are Not Movement Understanding: Designing a State Layer for Sports Video AI

Nous Technology Limited — Wed, 08 Jul 2026 07:07:55 +0000

Pose estimation has made sports video analysis look much easier than it really is.

Give a model a phone video of a tennis player, and it can return a set of body landmarks: shoulder, elbow, wrist, hip, knee, ankle. Run that across frames, and you get a moving skeleton. Add a few overlays, and suddenly the system appears to “understand” the athlete.

But keypoints are not understanding.

They are coordinates.

They tell us where parts of the body appear to be. They do not automatically tell us whether the athlete was balanced, late, loaded, recovering, drifting, or ready for the next ball.

For sports AI, this distinction matters. A tennis player does not lose a point because a wrist coordinate moved from (x1, y1) to (x2, y2). A player loses the point because the body failed to organize itself in time: the split step was late, the hips opened too early, the recovery step lagged, or the player contacted the ball while already falling out of position.

That is why a useful sports video system needs more than a pose layer.

It needs a movement state layer.

The pose layer is only the beginning

Most computer vision pipelines for sports video start with something like this:

video
  -> frame extraction
  -> pose estimation
  -> keypoints
  -> visualization

This is a good start. It gives the system a structured representation of the athlete instead of only raw pixels.

But this pipeline still treats each frame too independently.

In tennis, the interesting signal is rarely inside one frame. It is usually distributed across a short sequence:

prepare -> load -> rotate -> contact -> recover -> reset

A single frame can show that the player’s knees are bent. It cannot fully explain whether the player loaded early enough, transferred weight efficiently, or recovered in time for the next shot.

That requires temporal interpretation.

The system needs to ask questions like:

Did the player establish balance before the swing?
Did the lower body start the movement before the upper body?
Did the player contact the ball while moving forward, sideways, or backward?
How long did recovery take after contact?
Was the player ready for the next ball?

These are not raw keypoint questions. They are movement state questions.

Coordinates need to become events

A better pipeline does not stop at pose estimation.

It turns coordinates into events.

video
  -> keypoints
  -> normalized body geometry
  -> temporal features
  -> movement events
  -> movement states

For example, a tennis swing may contain events such as:

split step
unit turn
maximum racket preparation
forward acceleration
contact
follow-through
first recovery step
neutral reset

Some of these events are visible in the racket. Some are visible in the torso. Some are visible in the feet. Some are not obvious unless the system compares multiple frames before and after contact.

This is where sports video analysis becomes different from simple pose visualization.

A pose overlay can show the body.

An event layer can explain what the body is doing.

Why tennis is a good stress test

Tennis is a difficult sport for video understanding because the meaningful movement is fast, asymmetric, and context-dependent.

The player may hit the same forehand in several different body states:

balanced forehand
late forehand
open-stance forehand
defensive forehand
running forehand
recovery forehand

To a ball-tracking system, these may all look like successful shots if the ball goes in.

To a movement system, they are not the same.

A clean-looking shot can still be a bad movement pattern if the player is late, unstable, or unable to recover. The ball may cross the net, but the body may already be losing the next point.

This is the gap between shot analysis and athlete analysis.

Shot analysis describes the result.

Athlete analysis describes the body process that produced the result.

From keypoints to movement states

A movement state is a higher-level interpretation of body organization across time.

Instead of storing only:

{
  "left_knee": [312, 640],
  "right_knee": [420, 638],
  "left_hip": [300, 520],
  "right_hip": [430, 522]
}

the system should eventually infer something closer to:

{
  "phase": "pre-contact",
  "balance": "drifting_left",
  "load": "late",
  "rotation_sequence": "upper_body_first",
  "recovery_risk": "high",
  "next_ball_readiness": "low"
}

This second representation is not a replacement for keypoints. It is built on top of them.

The keypoints are the measurement substrate.

The movement state is the usable signal.

The hard part: normalization

Phone sports video is messy.

The camera angle changes. The player size changes. The court position changes. Lighting changes. The video may be filmed from behind, from the side, or from a diagonal angle. The player may be partially occluded.

That means raw pixel coordinates are often not enough.

Before building movement states, the system usually needs normalization layers such as:

body-relative coordinates
joint angle estimation
torso orientation
stance width
center-of-mass proxy
court-relative direction
frame-to-frame velocity
phase-relative timing

The goal is not just to know where the wrist is on the screen.

The goal is to understand what the wrist, torso, hips, legs, and feet are doing relative to the athlete’s own movement structure.

This is why sports AI cannot be only a detection problem. It becomes a representation problem.

For a deeper technical framing of single-camera movement reconstruction, we published a white paper on SpatialForm’s approach to phone sports video and 3D kinematic representation:

SpatialForm Single-Camera 3D Kinematics White Paper

A simple conceptual pipeline

A practical movement-understanding system might look like this:

1. Detect athlete body landmarks
2. Smooth and validate keypoints across frames
3. Normalize body geometry
4. Estimate temporal features
5. Detect movement phases
6. Identify movement events
7. Infer movement states
8. Present the result in a reviewable form

The last step matters.

If the system only outputs abstract numbers, most athletes and coaches will not use it. The result needs to be inspectable. The user should be able to see the movement, pause it, compare phases, and understand why the system labeled a state as late, unstable, or not ready.

In other words, the output should not be only a score.

It should be a movement representation.

This is also why we use the term Performance Form to describe the intermediate representation between raw video and athlete-side movement understanding.

Why “readiness” is more useful than isolated technique labels

Many sports AI products try to label isolated technical errors:

elbow too low
knee angle too small
shoulder rotation insufficient

These labels can be useful, but they are often too static.

In real play, the more important question is usually:

Was the athlete ready for the next action?

For tennis, this is especially important.

A player can hit a technically acceptable shot and still fail to recover. A player can make contact cleanly and still be late for the next ball. A player can win one stroke and lose the rally because the body never returned to a ready state.

That is why movement understanding should include recovery and next-action readiness, not only contact mechanics.

The point is not to judge a frozen pose.

The point is to understand the movement cycle.

Keypoints are the alphabet, not the language

Pose estimation gives sports AI an alphabet.

But an alphabet is not a sentence.

A sequence of body landmarks becomes useful only when the system learns how those landmarks form movement events, phases, and states.

For tennis video analysis, the real value is not simply drawing a skeleton on top of a player. The value is turning ordinary phone video into a structured view of how the athlete prepared, moved, hit, recovered, and reset.

That middle layer is where the hard work begins.

At SpatialForm, we describe this kind of intermediate representation as Performance Form: a reviewable movement structure built from phone sports video, designed to make athlete-side movement easier to inspect.

For readers interested in the tennis-specific side of this problem, we also wrote more about tennis swing analysis from phone sports video and the technical foundation behind single-camera 3D kinematics.

The broader lesson is simple:

Sports AI should not stop at seeing the body.

It should help us understand the movement behind the shot.
Updated July 2026: SpatialForm applies this idea to tennis swing analysis from ordinary phone video, extending the review beyond ball flight and contact to the athlete’s timing, balance, recovery, and readiness for the next action.

What Ball Tracking Misses in Tennis Video Analysis

Nous Technology Limited — Fri, 03 Jul 2026 17:13:23 +0000

Most tennis video analysis starts with the ball.

Where did it land?

How fast did it move?

Was the shot in or out?

What was the trajectory?

Those are useful questions. But they are not the whole story.

In tennis, a player can hit a clean-looking shot and still be late for the next ball. A rally can start breaking down before the ball leaves the racket. The visible result may be the shot, but the cause often sits inside the athlete’s movement: timing, balance, recovery, and readiness.

That is the layer we are exploring with SpatialForm.

We are building SpatialForm as an early AI sports product at NOUS TECHNOLOGY LIMITED. It focuses on turning ordinary phone sports video into Performance Form — a visible movement layer for reviewing timing, balance, recovery, and next-ball readiness.

Official site: SpatialForm official site

Ball tracking explains the result

Ball tracking is valuable because it turns the most obvious part of tennis into data. It can help answer questions like where the ball landed, how fast it moved, and what happened after contact.

For players, this is useful. For coaches, it creates a clearer record of the rally.

But the ball is often the receipt, not the transaction.

By the time the ball has landed, the movement decision has already happened. A player may lose the rally because the split step was late. Because the first step went in the wrong direction. Because the body was off-balance at contact. Because recovery was slow.

None of those problems are fully explained by the ball alone.

Tennis often breaks down before contact

Many tennis mistakes look like shot problems. A forehand goes long. A backhand lands short. A clean shot does not lead to control of the point.

But if you look earlier in the sequence, the issue may not be the swing itself. It may be:

the player loaded too late
the first step was delayed
the contact point was reached under poor balance
the body was still moving away from the next ball

This is why normal replay is limited. A replay can show what happened, but it does not always make the movement structure obvious. The important frame is often not the contact frame. It may be the frame before contact. Or three frames before contact.

That is where the athlete layer starts to matter.

The athlete layer behind every shot

When we say “athlete layer,” we mean the visible movement structure behind the shot. Not medical diagnosis. Not lab-grade biomechanics. Not replacing a coach.

We mean practical movement signals that can be reviewed from video:

timing & loading
balance & transfer
contact stability
recovery & court re-entry

This is the gap between shot analysis and Performance Form. Shot tracking explains the ball. Performance Form explains whether the athlete was ready for the next action.

More on Performance Form: Performance Form

Why phone video is a tough computer vision problem

Using ordinary phone video sounds simple, but it creates several hard engineering problems. Phone videos are not recorded in controlled lab conditions.

For a computer vision system, they bring:

unstable camera angles & motion blur
changing lighting & partial occlusion
racket and limb overlap
inconsistent player distance from the camera

The harder challenge is turning noisy visual information into a useful movement review. Standard pose estimation models can struggle here because a practical tennis movement analysis system needs to understand time, not just shape.

It needs to ask: what happened before contact? Did the player recover? Was the movement sequence repeatable? A single pose frame is not enough. The value comes from the sequence.

From phone sports video to Performance Form

SpatialForm is built around the idea that ordinary sports video can reveal more than a replay. The goal is to transform phone video into a reviewable movement signal.

At a high level, that means looking beyond a single shot result and focusing on the athlete’s movement sequence:

load → prepare → contact → recover → reset

For tennis, the important question is not only: Where did the ball go?

It is also: Was the athlete ready for the next action?

Starting with tennis, the goal is to make these movement signals easier to see from the videos players and coaches already record.

Tennis Swing Analysis page: Tennis Swing Analysis

Why this matters for players and coaches

Most amateur players already record video, but it is deeply underused. A player may watch a clip and say: "I was late. My balance was off."

The problem is that these observations are often vague. They are felt, but not clearly seen. If movement signals can be made more visible, video review becomes actionable. A coach can point to a moment before contact. A player can compare recovery between two shots.

This does not remove the human coach. It gives the player and coach a clearer visual layer to discuss.

What we are building

The product direction is simple:

Phone Sports Video → Performance Form → Personal Athletic Intelligence

Instead of only asking what happened to the ball, SpatialForm asks what happened to the athlete behind the shot. The long-term goal is to help players understand movement patterns that are hard to see in normal replay.

This is still early. We are not claiming that a phone replaces a professional biomechanics lab. The goal is more practical: Can ordinary sports video become a better movement review tool?

That is the question we are exploring.

Official site: SpatialForm official site

Closing thought

Ball tracking is useful. But in tennis, the ball is only one part of the story.

The next layer of sports video analysis is not just where the ball went. It is whether the athlete was ready for the next action.

That is the layer SpatialForm is working to make visible from ordinary phone sports video.
Updated July 2026: SpatialForm applies this idea to tennis swing analysis from ordinary phone video, extending the review beyond ball flight and contact to the athlete’s timing, balance, recovery, and readiness for the next action.