I built an MCP server that lets AI agents investigate videos

Hey DEV! First post here 👋

I've been working on an open-source project called Urma, a local MCP server that lets AI hosts retrieve captions and frames from videos. I recently shipped its first stable release, so I thought I'd introduce it here.

The idea is quite simple. Let's say you're asking an AI agent to research how a particular feature works. It searches the documentation, goes through a few GitHub issues (including one closed in 2021 with no explanation), and eventually finds a recording where someone demonstrates the feature.

That recording could be a 40-minute conference talk, with the relevant part buried somewhere around minute 37. Naturally.

An agent with web access can find the recording itself. With Urma connected to its MCP host, it can then inspect the video, search for relevant captions, and retrieve frames around the moments it wants to examine.

I wanted that to fit naturally into the way agents already research things. Cause sometimes the most useful information is sitting inside a video, while the documentation confidently says "Coming soon."

What Urma does

Urma works through five MCP tools. An investigation begins with inspect_video, which opens a source and returns an investigation reference. The host uses that reference when requesting captions or visual evidence.

For example, suppose a tutorial includes the line:

"Now I'm turning on dark mode."

A caption search can locate where the presenter said that. But to answer whether dark mode was actually enabled, the agent needs to look at the screen.

The presenter might have turned it on. The UI might still be bright enough to illuminate the entire room. The caption alone won't settle that.

Urma can retrieve frames around that timestamp. The host can inspect the images, request additional frames if needed, and decide what the evidence supports.

This distinction influenced quite a bit of how I built the project. Urma records which source snapshot an investigation uses and which observations have been returned. A visual overview might contain 12 sampled frames, but of course those samples don't tell us everything that happened between them.

Calling 12 frames complete video coverage would be a bit like saying your tests passed because TypeScript compiled successfully.

I also spent time getting timestamp handling right. Video containers and streams can have different timing origins, and extracting the intended frame isn't always as straightforward as seeking to a number. (Literal pain)

A quick look at the five tools

• inspect_video opens a video and returns its source metadata, duration, and available caption tracks.
• search_transcript searches for literal phrases or terms in a selected caption track.
• read_transcript retrieves timestamped caption segments from a requested interval.
• get_overview provides up to 12 sampled images to help navigate a video.
• get_frames retrieves JPEG frames at specific timestamps, in short bursts, or on a fixed schedule.

The host handles video discovery and interpretation. Urma handles evidence retrieval and keeps track of what it returned.

Urma uses source-provided captions, so videos without captions can't be searched by speech. It can still retrieve visual evidence when the video source is supported.

Trying it

Urma runs locally on Windows, macOS, and Linux, with x64 and ARM64 support. It works with Node.js 22.16+ within the 22.x line, as well as Node.js 24 and 26.

If you use a coding agent that can read GitHub repositories and run commands, you can also point it to Urma's README and ask it to help with setup. Pretty fast actually, provided the agent doesn't decide your entire development environment needs refactoring first.

Setup installs and verifies the FFmpeg, ffprobe, and yt-dlp components it needs.

No need to install all three manually or play detective with environment variables. I'd rather people spend their time using Urma than figuring out which FFmpeg binary Windows has decided to use today ;)

npx -y urma-mcp@latest setup

Once setup finishes, connect Urma to your MCP host and restart the host. The README covers the configuration and has examples of the tool arguments.

You can use public video URLs that pass Urma's source checks, or local files in directories you've explicitly allowed. Remote support depends on the video provider and its available formats, so some sources may not work.

Here's the repository if you want to look through the code or try it yourself:

sajidurdev / urma

Retrieve video captions and frames through a local MCP server.

Urma

Retrieve video captions and frames through MCP.

Try Urma · Install · Tools · Configuration · Documentation

_{urma-mcp · MCP stdio · Node.js 22.16+ (22.x), 24.x, 26.x}

Urma gives MCP clients access to captions and frames from public video URLs and local video files. Search captions for a topic, browse a visual overview or retrieve frames at specific timestamps.

Urma runs locally and keeps its cache on your filesystem. Your MCP client starts the server over stdio and receives the requested text and images Remote videos require network access.

Try Urma

1. Install with npx -y urma-mcp@latest setup.
2. Connect your MCP client and restart it.
3. Give the client a video URL and a question about the video.

For example, replace <video URL> in this prompt with a public product demo:

In this demo, where does the presenter discuss dark mode, and is the toggle enabled on screen? <video URL>

…

View on GitHub

It's written in TypeScript, runs as a local stdio MCP server, and is available under the Apache-2.0 license.

I'd be interested in hearing how it works with different MCP hosts. If you run into trouble with a video source, setup, or frame retrieval, feel free to open an issue. Those are especially useful to test outside my own environment.

And if you find a video that breaks Urma, please include the URL and what went wrong. A reproducible bug report is always appreciated. My collection of mysterious failures is big enough already.

Thanks for reading my first DEV post! :)

Comments

[Arhan Canli]

The "12 sampled frames is not coverage" point is the useful one, and it could be machine-checkable rather than only documented: return the largest gap in seconds between the frames you sampled, and the caption span each frame falls in. Then a host can see "nothing sampled between 37:10 and 39:40" instead of assuming it saw everything.

On timestamps, I'd keep a fixture video whose container start_time is not zero (an MPEG-TS cut from a stream, or an MP4 with an edit list) and one with variable frame rate, and assert that the frame returned for caption time T actually shows the caption's content. Seeking before versus after the input in ffmpeg gives different frames on those files, and a zero-origin test file hides it.

Question: when the source at a URL changes after inspect_video (re-upload, or captions regenerated), does the stored snapshot let later get_frames calls detect it, for example through a content hash or etag, or do they quietly read the new version?

[Sajidur]

First of all, I loved your comment haha. Didn't expect my very first comment to be this detailed, but seriously, great points!

Urma actually tracks the largest unsampled gaps in its investigation state already, though I agree they should be surfaced directly in tool responses. The caption-span idea is interesting too.

On timestamps, I've got regression tests covering nonzero PTS and container timing differences, including actual frame comparisons. Adding a VFR fixture with caption alignment sounds worthwhile.

And you're right to question remote snapshots. Urma rechecks the source identity and selected format metadata before acquiring media, but that doesn't guarantee the bytes haven't changed. Captions have an even weaker check, so regenerated ones can slip through.

That's a real limitation I want to improve. Really appreciate you taking the time to dig into this, especially on my first post! :)

[Reid Marlow]

The operational headache I keep hitting with local MCP wrappers around ffmpeg and yt-dlp is child process lifecycle during host cancellations.

If an agent triggers an extraction and the host client hits a timeout or aborts the turn, the stdio server survives but the background ffmpeg child keeps running detached. Unless the server spawns ffmpeg in an isolated process group and explicitly reaps children on abort signals, you end up with orphan decoders chewing CPU long after the model moved on. Spawning the child with detached mode in Node and sending SIGTERM to the negative process group ID on client disconnect was the only reliable fix on my Linux boxes.

[Sajidur]

That’s a really good point! I actually spent quite a bit of time handling this in Urma.

On Linux, subprocesses run in isolated process groups, and cancellation or timeouts trigger kill(-pid, SIGKILL). I’ve also added regression tests to verify descendant cleanup.

The tricky case is when a host stops waiting without sending cancellation or closing the connection. Urma still enforces its own subprocess timeouts, but I’d like to test more real-world host behaviour.

Appreciate you bringing this up! ;)