Most MCP servers wrap something stateless: search an API, read a file, query a DB. I built one for something very stateful — a multitrack video project that a human is editing at the same time. This post is about the design choices that made that workable. The project is CutVoke, an MIT-licensed, local-first video editor.
The problem: AI video ends in an MP4
Agent-driven video pipelines usually render a file and stop. When a caption is wrong or a cut is late, you re-run the pipeline or rebuild everything in a human editor. I wanted the agent's output to be an editable project.
Architecture in one picture
- A Python engine owns a persistent project (tracks, clips, captions, keyframes, effects), stored locally.
- Four front doors share it: a React editor (
cutvoke serve, loopback only), an MCP server (cutvoke mcp --data …, stdio), an HTTP API, and a CLI. - FFmpeg renders previews and exports. Desktop builds bundle it.
- There is no model inside. The agent is whatever you already use.
Lesson 1: give the agent a lease, not a lock-free free-for-all
When a person is dragging clips and an agent is running a 15-step edit, you need a rule for who writes. CutVoke uses an editing lease: the agent calls edit_lock, renews it during long jobs, and releases it on success or failure. The UI shows that an agent is editing. Commits also carry expectedRevision, so a stale plan is rejected instead of silently clobbering human work.
Lesson 2: dry runs and batches beat "be careful" prompts
Instead of telling the model to be careful, make carefulness cheap. Supported commands can be dry-run with command_preview. Related changes go into edit.batch, which is one undo step for the human. A stable commandId makes retries idempotent.
Lesson 3: let the agent see pixels, not JSON
A timeline JSON can be perfectly valid and still look terrible: captions off-screen, a title over a face, a black frame at a cut. preview_frames returns real rendered PNGs (up to 6 timestamps per call), captions and titles included. The guide tells agents to check frames around cuts, transitions and caption intervals — and to first check whether the client can actually read tool images at all.
Lesson 4: keep the tool catalog small and lazy
A video editor has a lot of commands. Dumping every schema into context hurts smaller models. Discovery is layered: editor_help (onboarding + recipes) → commands_list({"includeSchemas": false, "query": "caption"}) → editor_help({"topic":"command","command":"caption.patch"}) for the exact payload schema.
Lesson 5: objects, not pixels, for the things people fix most
The most common post-AI fix is text. Captions are individual objects: project_lookup finds a line by its text, caption.patch changes words, timing or style. Transcript-based cutting works the same way: locate a sentence, audition it, preview the impact of deleting it, confirm, undo in one step.
Try it
- Downloads (prerelease, unsigned): https://github.com/frankShan001/CutVoke/releases/tag/v0.3.5
- From source:
pip install -e ., build the editor,cutvoke doctor --json, thenpython skills/cutvoke-editor/scripts/demo.py --output .artifacts/first-editbuilds a 12-second edit over real stdio MCP. - The README has two complete edits with source footage and editable projects.
It's early and I'd value feedback on the agent API more than anything: if you've built tools for stateful apps, what would you do differently?
Top comments (1)
The expectedRevision guard paired with transactional batches (edit.batch as one human undo step) is the cleanest way to prevent agent drift.
One operational edge case with lease locks is agent failure. When a model hits a tool error or context limit midway through, a global project lease leaves the UI locked until the TTL expires. Adding an explicit "break lease" button in the human UI, paired with a diff view of any uncommitted fragments, keeps the operator in control.
Scoping leases to individual track IDs rather than the entire project also helps. An agent can spend twenty seconds generating and previewing captions on Track 1 while a human trims background music on Track 2 without lock contention.