Designing MCP tools for a stateful app: what I learned building an AI-editable video timeline

shansjl — Mon, 09 Nov 2026 17:03:20 +0000

Most MCP servers wrap something stateless: search an API, read a file, query a DB. I built one for something very stateful — a multitrack video project that a human is editing at the same time. This post is about the design choices that made that workable. The project is CutVoke, an MIT-licensed, local-first video editor.

The problem: AI video ends in an MP4

Agent-driven video pipelines usually render a file and stop. When a caption is wrong or a cut is late, you re-run the pipeline or rebuild everything in a human editor. I wanted the agent's output to be an editable project.

Architecture in one picture

A Python engine owns a persistent project (tracks, clips, captions, keyframes, effects), stored locally.
Four front doors share it: a React editor (cutvoke serve, loopback only), an MCP server (cutvoke mcp --data …, stdio), an HTTP API, and a CLI.
FFmpeg renders previews and exports. Desktop builds bundle it.
There is no model inside. The agent is whatever you already use.

Lesson 1: give the agent a lease, not a lock-free free-for-all

When a person is dragging clips and an agent is running a 15-step edit, you need a rule for who writes. CutVoke uses an editing lease: the agent calls edit_lock, renews it during long jobs, and releases it on success or failure. The UI shows that an agent is editing. Commits also carry expectedRevision, so a stale plan is rejected instead of silently clobbering human work.

Lesson 2: dry runs and batches beat "be careful" prompts

Instead of telling the model to be careful, make carefulness cheap. Supported commands can be dry-run with command_preview. Related changes go into edit.batch, which is one undo step for the human. A stable commandId makes retries idempotent.

Lesson 3: let the agent see pixels, not JSON

A timeline JSON can be perfectly valid and still look terrible: captions off-screen, a title over a face, a black frame at a cut. preview_frames returns real rendered PNGs (up to 6 timestamps per call), captions and titles included. The guide tells agents to check frames around cuts, transitions and caption intervals — and to first check whether the client can actually read tool images at all.

Lesson 4: keep the tool catalog small and lazy

A video editor has a lot of commands. Dumping every schema into context hurts smaller models. Discovery is layered: editor_help (onboarding + recipes) → commands_list({"includeSchemas": false, "query": "caption"}) → editor_help({"topic":"command","command":"caption.patch"}) for the exact payload schema.

Lesson 5: objects, not pixels, for the things people fix most

The most common post-AI fix is text. Captions are individual objects: project_lookup finds a line by its text, caption.patch changes words, timing or style. Transcript-based cutting works the same way: locate a sentence, audition it, preview the impact of deleting it, confirm, undo in one step.

Try it

Downloads (prerelease, unsigned): https://github.com/frankShan001/CutVoke/releases/tag/v0.3.5
From source: pip install -e ., build the editor, cutvoke doctor --json, then python skills/cutvoke-editor/scripts/demo.py --output .artifacts/first-edit builds a 12-second edit over real stdio MCP.
The README has two complete edits with source footage and editable projects.

It's early and I'd value feedback on the agent API more than anything: if you've built tools for stateful apps, what would you do differently?

⭐ Repo: https://github.com/frankShan001/CutVoke

DEV Community: shansjl