I built an MCP server that lets AI agents investigate videos

Sajidur — Tue, 13 Oct 2026 19:10:23 +0000

Hey DEV! First post here 👋

I've been working on an open-source project called Urma, a local MCP server that lets AI hosts retrieve captions and frames from videos. I recently shipped its first stable release, so I thought I'd introduce it here.

The idea is quite simple. Let's say you're asking an AI agent to research how a particular feature works. It searches the documentation, goes through a few GitHub issues (including one closed in 2021 with no explanation), and eventually finds a recording where someone demonstrates the feature.

That recording could be a 40-minute conference talk, with the relevant part buried somewhere around minute 37. Naturally.

An agent with web access can find the recording itself. With Urma connected to its MCP host, it can then inspect the video, search for relevant captions, and retrieve frames around the moments it wants to examine.

I wanted that to fit naturally into the way agents already research things. Cause sometimes the most useful information is sitting inside a video, while the documentation confidently says "Coming soon."

What Urma does

Urma works through five MCP tools. An investigation begins with inspect_video, which opens a source and returns an investigation reference. The host uses that reference when requesting captions or visual evidence.

For example, suppose a tutorial includes the line:

"Now I'm turning on dark mode."

A caption search can locate where the presenter said that. But to answer whether dark mode was actually enabled, the agent needs to look at the screen.

The presenter might have turned it on. The UI might still be bright enough to illuminate the entire room. The caption alone won't settle that.

Urma can retrieve frames around that timestamp. The host can inspect the images, request additional frames if needed, and decide what the evidence supports.

This distinction influenced quite a bit of how I built the project. Urma records which source snapshot an investigation uses and which observations have been returned. A visual overview might contain 12 sampled frames, but of course those samples don't tell us everything that happened between them.

Calling 12 frames complete video coverage would be a bit like saying your tests passed because TypeScript compiled successfully.

I also spent time getting timestamp handling right. Video containers and streams can have different timing origins, and extracting the intended frame isn't always as straightforward as seeking to a number. (Literal pain)

A quick look at the five tools

inspect_video opens a video and returns its source metadata, duration, and available caption tracks.
search_transcript searches for literal phrases or terms in a selected caption track.
read_transcript retrieves timestamped caption segments from a requested interval.
get_overview provides up to 12 sampled images to help navigate a video.
get_frames retrieves JPEG frames at specific timestamps, in short bursts, or on a fixed schedule.

The host handles video discovery and interpretation. Urma handles evidence retrieval and keeps track of what it returned.

Urma uses source-provided captions, so videos without captions can't be searched by speech. It can still retrieve visual evidence when the video source is supported.

Trying it

Urma runs locally on Windows, macOS, and Linux, with x64 and ARM64 support. It works with Node.js 22.16+ within the 22.x line, as well as Node.js 24 and 26.

If you use a coding agent that can read GitHub repositories and run commands, you can also point it to Urma's README and ask it to help with setup. Pretty fast actually, provided the agent doesn't decide your entire development environment needs refactoring first.

Setup installs and verifies the FFmpeg, ffprobe, and yt-dlp components it needs.

No need to install all three manually or play detective with environment variables. I'd rather people spend their time using Urma than figuring out which FFmpeg binary Windows has decided to use today ;)

npx -y urma-mcp@latest setup

Once setup finishes, connect Urma to your MCP host and restart the host. The README covers the configuration and has examples of the tool arguments.

You can use public video URLs that pass Urma's source checks, or local files in directories you've explicitly allowed. Remote support depends on the video provider and its available formats, so some sources may not work.

Here's the repository if you want to look through the code or try it yourself:

sajidurdev / urma

Retrieve video captions and frames through a local MCP server.

Urma

Retrieve video captions and frames through MCP.

Try Urma · Install · Tools · Configuration · Documentation

urma-mcp · MCP stdio · Node.js 22.16+ (22.x), 24.x, 26.x

Urma gives MCP clients access to captions and frames from public video URLs and local video files. Search captions for a topic, browse a visual overview or retrieve frames at specific timestamps.

Urma runs locally and keeps its cache on your filesystem. Your MCP client starts the server over stdio and receives the requested text and images Remote videos require network access.

Try Urma

Install with npx -y urma-mcp@latest setup.
Connect your MCP client and restart it.
Give the client a video URL and a question about the video.

For example, replace <video URL> in this prompt with a public product demo:

In this demo, where does the presenter discuss dark mode, and is the toggle enabled on screen? <video URL>

…

View on GitHub

It's written in TypeScript, runs as a local stdio MCP server, and is available under the Apache-2.0 license.

I'd be interested in hearing how it works with different MCP hosts. If you run into trouble with a video source, setup, or frame retrieval, feel free to open an issue. Those are especially useful to test outside my own environment.

And if you find a video that breaks Urma, please include the URL and what went wrong. A reproducible bug report is always appreciated. My collection of mysterious failures is big enough already.

Thanks for reading my first DEV post! :)