I gave my coding agent my basketball videos and asked it to be my coach

Marius — Fri, 23 Oct 2026 11:15:19 +0000

I wrote this post with Claude Code and posted it myself. The project was built with Claude Code too.

I play in an adult rec basketball league. Nobody coaches us and nobody watches film. What we do have is a lot of phone video that nobody opens again.

Claude Code can look at images, so I tried giving it frames from my own shooting drill and asking what it saw. On one training session it counted my set shots at 9 of 11 and my jump shots at 7 of 19, and all 8 jump shots in the last ten minutes were misses. I hadn't noticed that at all. (That session went through the basketball plugin's training mode; the steps below use the simpler frame-based mode.)

The annoying part was everything around the review: pulling frames, cutting clips, making something my teammates would actually open on a phone. So I wrote a CLI for that part and called it sports-video-coach (MIT).

What you get back

From one phone video:

a page for your phone with what to fix, a drill and a target to check next session for each problem, and the clips that show it
a spoken version of the report, so you can listen on the way to practice
an explained video: the clip pauses where it goes wrong, a ring marks the spot, a subtitle says why

Here's a demo page and its explained video (with sound) you can open without installing anything. The demo is a synthetic drill with a pre-written review, so it shows what the output looks like, not how good the agent is at basketball. For that you need your own video.

Trying it on your own video

You need git, uv (the tool needs Python 3.11; uv fetches it), ffmpeg and ffprobe on your PATH, and a coding agent (Claude Code, Codex, Cursor ...). Then paste this into the agent:

Read https://github.com/mariusDing/sports-video-coach/blob/master/AGENTS.md, install the CLI with
`uv tool install --python 3.11 git+https://github.com/mariusDing/sports-video-coach` (it needs ffmpeg), then coach me on
my video <path to your video>: run
`sports-video-coach run <video> --sport basketball --focus "<my jersey number or what I wear>"`,
do each step it hands you (exit code 3) until it exits 0, and show me the coaching page.

The agent installs it and starts the run. Whenever the CLI needs frames looked at or a file written, it pauses, the agent does that, and the run carries on. You may still need to approve the install or fix a missing tool. It uses some of your agent plan, like any other task.

For a pickup game or a drill filmed on a phone, --sport basketball uses the frame-based review: no tracking, no GPU, no league. --focus takes a jersey number or a description like "the player in red". Film from where that player is clearly visible; the agent can only judge what it can see.

To build the demo on your own machine, run these two commands. It brings its own synthetic video and pre-written review, so no agent is involved. The audio needs internet; add --offline for just the page and clips.

uv tool install --python 3.11 git+https://github.com/mariusDing/sports-video-coach
sports-video-coach demo

Everything lands as local files. To get the page onto a phone, ask the agent for a single file with the audio and clips inside: sports-video-coach html <folder> --share --out report.html, then send yourself that file. There is also sports-video-coach run <folder> --publish for a hosted link (needs a Cloudflare account; anyone with the link can open it).

How the agent and the tool split the work

In the generic workflow the CLI makes no model API calls and needs no API key. It extracts frames, cuts clips, builds the outputs and keeps track of which steps are done. When it needs someone to watch or write, it exits with code 3 and says exactly what file to write:

$ sports-video-coach run drill.mp4 --sport basketball
frames: 4 sheets (0.5 s per tile) in .../run/frames
[run] waiting for the agent: review
...                                                   # exit 3

The agent opens the frame sheets, writes review.json, and runs sports-video-coach run <folder> to carry on.

I spent a while rewording the guides when reviews came back thin. Then I added a validator instead. If the review has fewer than three moments, the run won't build a page and says why:

review.json: needs a `moments` list with at least 3 entries

It also checks that each moment has a what and a why, that its end comes after its start, and that it doesn't end more than a second past the video. None of that tells you whether the basketball advice is right.

Where it falls short

The agent looks at frame sheets, not the video itself. By default, clips up to 30 seconds get a frame every half second, up to 3 minutes one per second, and longer videos one every two seconds. You can make it denser with --every. Very fast details, like the moment of release, can fall between frames.

Frames go to your agent's model provider, same as anything else you show your agent. The narration text goes to Microsoft's online text-to-speech (edge-tts). Use jersey numbers, not names. If you don't want anything sent to Microsoft, run with --artifacts html: you keep the page and the clips and skip the audio and the explained video.

Player tracking and box scores are in the basketball plugin, which you run from a clone; the full pipeline is meant for a CUDA GPU. Other sports work through the generic mode, and sports-video-coach new-plugin <name> gives you a starting point for your own.

Repo: https://github.com/mariusDing/sports-video-coach

I mostly use it for basketball. If you try it on something else, I'd like to hear what it gets wrong.

DEV Community: Marius

I gave my coding agent my basketball videos and asked it to be my coach

What you get back

Trying it on your own video

How the agent and the tool split the work

Where it falls short