Teaching a Computer to Understand Billiards: From Ball Detection to Trajectory Reconstruction

Dmitrii Sadovnik — Wed, 28 Oct 2026 10:29:32 +0000

Computer vision, YOLO, geometry, and physics-based modeling in PerformStars

Introduction

At first glance, automated billiards analysis seems like a relatively straightforward computer vision task. The table has known geometry, the balls are spherical, and their number and size are constrained by the rules of the game.

Place a camera over the table, detect the balls in each frame, and track their motion. In practice, however, the problem is considerably more complex.

Balls can move rapidly, collide with one another and with the cushions, temporarily occlude each other, disappear from the image, or become difficult to distinguish under varying lighting conditions. The camera can be positioned at an angle, introducing perspective distortion.

Detecting a ball is only the beginning. To analyze a shot, the system must determine which physical ball is moving, where it was before the shot, how its trajectory evolves, and what ultimately happened.

At PerformStars, we are developing a system that turns ordinary footage of billiards games and practice sessions into structured spatial and temporal data.

1. Why Detecting Billiard Balls Is Harder Than It Looks

Modern neural-network detectors perform well on many objects with distinct visual features. Billiards presents a different set of challenges.

First, the balls occupy only a small fraction of the image. When an entire table is filmed in Full HD, a ball may be just a few dozen pixels across.

Second, balls have nearly identical shapes and sizes, while their appearance varies with the illumination.

Third, reflections, highlights, and shadows may affect how precisely the center of a ball can be located.

Motion blur creates another problem. At a speed of 2 m/s and a frame rate of 30 fps, a ball travels approximately 6.7 cm between consecutive frames. That is more than the diameter of a standard pool ball, 57.15 mm.

As a result, correctly detecting objects in successive frames does not guarantee that we can correctly match them. When multiple balls collide, the nearest detection in the next frame may belong to a different ball.

2. Ball Detection with YOLO

PerformStars uses custom-trained models from the YOLO family to detect billiard balls. This approach offers a practical balance between inference speed and the localization of small objects.

A detector processes a video frame and returns object locations, bounding-box dimensions, and confidence scores. The estimated centers of these detections become inputs to subsequent geometric analysis.

A single universal detector, however, is not enough for every recording. Cue-sport disciplines, ball designs, lighting conditions, and viewing angles differ. The project therefore uses specialized models and training datasets for different object types and recording conditions.

Why a Neural Network Alone Is Not Enough

A detector answers the question: “Where are the balls in this frame?” It does not inherently know which ball moved a moment ago, which one took part in a collision, or where a particular ball should appear next. That requires reasoning across a sequence of frames rather than treating images independently.

3. Table Geometry and Real-World Coordinates

Image coordinates depend on camera placement. The same physical point on the table can have very different pixel coordinates in overhead and angled footage. To reason about motion, we need a consistent coordinate system attached to the table.

PerformStars uses table-surface segmentation and geometric calibration. One of the main mathematical tools is a projective transformation, also known as a homography.

Equation 1. Projective mapping between image coordinates (x, y) and coordinates (X, Y) on the table plane. H is the transformation matrix, and λ is the scale factor in homogeneous coordinates.

After calibration, ball locations can be compared in a shared coordinate system, and distances and angles can be estimated independently of the original camera viewpoint. Accuracy still depends on how well the table is localized and calibrated: a homography cannot eliminate all detection errors.

4. From Independent Detections to Trajectories

Suppose a detector identifies ten balls in one frame and ten in the next. Which detection corresponds to which physical ball?

The simplest method is to match objects using the distance between their centers:

Equation 2. Euclidean distance between two detected positions in the table coordinate system.

This method is reliable only when displacement is relatively small and the scene is unambiguous. A fast-moving ball may travel farther than its distance to a neighboring ball. During collisions, trajectories converge and detections can become difficult to distinguish.

Nearest-neighbor matching therefore needs additional safeguards. In PerformStars, detections are treated as time-dependent observations of physical objects. The analysis considers spatial proximity, displacement between frames, the geometry of the playing surface, and whether the inferred motion is consistent over time.

It is especially important to recover from temporary detection loss. A ball may disappear for several frames because of blur or occlusion without the underlying shot actually ending.

5. Ball Number Recognition and Identity Tracking

In pool, balls have different colors and markings, but recognizing a specific ball is sensitive to rotation, lighting, reflections, and image resolution.

PerformStars explores and uses specialized classifiers trained on cropped images of individual balls. Once a ball is detected, its image region can be passed to a classifier that estimates probabilities for possible identities.

Rather than relying on a single frame, the system can aggregate classification evidence across multiple observations. But this strategy is only useful while we are confident that those observations refer to the same physical ball.

If a tracking algorithm swaps two identities during a collision, previously accumulated evidence becomes attached to the wrong object. Ball-number recognition and multi-object tracking are therefore tightly connected problems.

6. Physics as an Additional Source of Evidence

Billiards is particularly interesting because ball motion follows reasonably well-understood mechanical principles. The principal interactions include motion across the cloth, ball-to-ball collisions, cushion impacts, and deceleration due to friction.

This makes physical simulation a useful tool for interpreting ambiguous trajectories. PerformStars complements motion analysis with simulations of plausible shot scenarios.

The general idea can be expressed as an optimization problem: find the model parameters that minimize the disagreement between observed and simulated positions.

Equation 3. Selecting physical-model parameters θ to best match simulated positions to observations. This is an illustrative mathematical formulation, not the complete objective function used by the production system.

A real implementation must also account for detection uncertainty, ambiguous ball assignments, incomplete trajectories, and interactions among multiple balls.

7. How Do We Know a Shot Has Happened?

A further challenge is dividing continuous video into individual shots. The system must identify the start of a shot, follow the movement and collisions, and establish when the episode has finished.

If an episode is terminated too early, residual ball movement may be incorrectly interpreted as the beginning of another shot.

For that reason, determining whether the table is at rest involves analyzing positional changes over a time interval, not making a decision from one frame. Requiring several consecutive confirmations helps reduce the impact of noise and brief detection gaps.

Once shot boundaries have been established, the system can reconstruct the initial layout, motion parameters, and outcome. This is the stage at which a stream of frames becomes a structured sporting event.

8. Validation on Real Video Footage

Strong performance on individual images does not necessarily mean that complete shots are reconstructed correctly. PerformStars has therefore been evaluated on continuous practice footage, with automated results checked manually.

In one evaluation covering 124 real shots, the system detected 93.5% of them. The accuracy of the potted/missed outcome classification among shots for which an outcome was reported was 97.2%. Outcome-decision coverage was 91.4% of detected shots.

These figures measure shot-event detection and outcome classification—not the localization accuracy of individual balls. A dedicated detection benchmark would require separate measurements such as precision, recall, mAP, and center-coordinate error.

Demonstrations: Video 1 · Video 2

9. Open Engineering Challenges

Maintaining identity through collisions. This is especially difficult when visually similar balls cross paths or disappear briefly.

Robustness across recording conditions. Camera angle, lighting, resolution, and partial occlusion of the table increase uncertainty.

Combining neural and physical models. Recognizing objects is not enough; their inferred motion should also be physically plausible.

Computational efficiency. Processing large video collections needs to remain economical, including when multiple practice sessions or matches are analyzed in parallel.

10. Conclusion

Our work on PerformStars illustrates why automated billiards analysis cannot be reduced to a single neural network. Ball detection is only the first layer.

Reliable sports analytics requires combining computer vision, geometry, object tracking, identity recognition, physics-based modeling, and temporal event analysis.

A billiards table is a compact but challenging example of a broader problem: reconstructing the dynamics of multiple interacting objects from incomplete visual observations. Similar challenges arise in robotics and industrial machine vision.

Our goal is not merely to teach a computer to see a billiards table, but to describe the game unfolding on it in mathematical terms.

PerformStars Research & Engineering · https://performstars.com/research/computer-vision-billiards

DEV Community: Dmitrii Sadovnik