Making a React Video Editor Agent-Friendly with Timeline Markers and MCP

A video editor needs to remember more than cuts. It also needs to remember why a cut matters.

“Start the second chapter here.” “Review this section before publishing.” “Leave a little more room at the end.” These instructions become much easier to act on when their timestamps and context live inside the project.

I recently added persistent timeline markers to Timeline Studio, the open-source browser video editor I maintain, and exposed them through its CLI, MCP adapter, and Agent Skill. The interesting engineering work was making those annotations precise, inspectable, and safe to edit in a batch.

You can try the editor in your browser or browse the source on GitHub. The implementation discussed here is included in Skill v1.0.7. Timeline Studio uses React and Vite and follows a local-first approach.

Give annotations their own data model

The editor supports four annotation types:

Type	Purpose	Timing
marker	A beat cue, action, or cut reference	time
chapter	The beginning of a named section	time
range	An interval to review or work on	time, endTime
note	Feedback tied to a precise moment	time

A review range looks like this:

{
  "id": "product-review",
  "type": "range",
  "time": 5,
  "endTime": 8,
  "title": "Product reveal",
  "notes": "Check whether the caption appears with the product.",
  "color": "violet"
}

Times are absolute project seconds. Titles and notes preserve Unicode, and annotations are serialized in timelineMarkers inside the portable .timeline project.

One boundary is especially useful: annotations do not contribute to rendered media duration. Adding a planning marker at 90 seconds to a 60-second edit must not produce another 30 seconds of video.

Chapter markers also do not automatically become on-screen titles or MP4 chapters. They remain editable project annotations. A marker-only task can therefore finish by writing a new project without re-encoding unchanged media.

The tradeoff is explicit timing: annotations stay at their absolute project positions during trims, reordering, and ripple edits. If they need to follow moved content, the edit plan must update them too.

Keep the React interaction compact

An always-visible annotation lane takes space away from media tracks. The default view now merges compact flags into the ruler. A toolbar chevron expands a separate lane with titles and range spans when more detail is useful.

Pressing M adds a marker at the playhead; Shift + M opens the manager. The marker UI is localized in all 13 supported interface languages.

During a drag, React holds a temporary preview. Releasing the pointer commits the edit; Escape or a canceled gesture discards the preview. This separates continuous pointer feedback from the final project mutation.

Measure snapping in pixels, then convert to time

A fixed temporal threshold such as 0.2 seconds feels very different at different zoom levels. The implementation converts a 10-pixel tolerance into project seconds:

const thresholdSeconds =
  10 / railWidth * timelineDuration;

Here, railWidth is the timeline rail width and timelineDuration is its corresponding time span. This keeps the screen-space snapping distance approximately consistent as the timeline zoom changes.

Markers reuse the shared snapping targets: the playhead, clip boundaries, and other marker edges. The editor shows a common alignment guide, and holding Alt temporarily bypasses snapping.

Ranges need an additional invariant: moving a range must preserve its length. If the end of a three-second range snaps to 12 seconds, its start must become 9 seconds.

The core calculation, simplified, is:

const nextStart = movingEdge === "end"
  ? targetTime - duration
  : targetTime;
const nextEnd = nextStart + duration;

The actual implementation checks both edges and chooses the nearer valid snap candidate. Resizing an edge is handled separately from moving the whole range.

Expose one command engine through CLI and MCP

The agent-facing architecture is small:

Agent Skill: workflow, timing evidence, verification
                         |
CLI / MCP: structured operation entry points
                         |
Shared command engine: validation, edits, semantic diff
                         |
A new .timeline project

The MCP adapter invokes the existing CLI runner. It does not maintain its own marker reducers. Fixing a validation rule therefore fixes both entry points.

From the repository root, an agent can inspect the project and its annotations:

npm run agent -- project.inspect /projects/input.timeline
npm run agent -- marker.inspect /projects/input.timeline

MCP exposes the read-only timeline_marker_inspect tool. Writes use marker.add, marker.update, and marker.delete through the existing project diff/apply workflow.

For example, suppose inspection reports revision 0 and a product-review range spanning 5–8 seconds. This plan moves it to 9 seconds and adds a note at 11 seconds:

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 0,
  "operations": [
    {
      "id": "move-product-v1",
      "type": "marker.update",
      "markerId": "product-review",
      "time": 9
    },
    {
      "id": "add-ending-note-v1",
      "type": "marker.add",
      "markerId": "ending-note",
      "markerType": "note",
      "time": 11,
      "title": "Ending rhythm",
      "notes": "Reviewer request: leave a little more breathing room.",
      "color": "rose"
    }
  ],
  "output": { "project": "/projects/output-marked.timeline" }
}

The paths, revision, IDs, and review request are illustrative. A real plan must use inspected values and actual feedback.

Two distinctions matter: id identifies an operation for retry handling, while markerId identifies the annotation. Likewise, type selects the command, while markerType selects the annotation kind.

Updating only the time of a range preserves its length. The example moves 5–8 seconds to 9–12 seconds. An explicit endTime lets the plan change the end instead.

Save the plan as /projects/markers-plan.json, validate its structure, and preview the semantic diff:

node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs /projects/markers-plan.json
npm run agent -- project.diff /projects/markers-plan.json

After reviewing the result, apply the same plan and inspect the output:

npm run agent -- project.run /projects/markers-plan.json
npm run agent -- marker.inspect /projects/output-marked.timeline

The diff reports additions, removals, and modifications under changes.markers, including before/after values. A marker-only edit should not unexpectedly modify a media track.

The Skill instructs the agent to inspect, review the diff, apply, and inspect again. The engine enforces concrete constraints:

• A mismatched revision returns REVISION_CONFLICT.
• Previously applied operation IDs are not applied again.
• A failed operation rejects the batch without writing a partially edited archive.
• Output must go to a new path; input files and existing outputs cannot be overwritten.

A revision conflict requires a fresh inspection and a reconsidered plan. Merely replacing the revision number would skip the reason for having that check.

The tricky bugs were identity and floating-point boundaries

Consider imported annotations with three identical IDs: x, x, and x. Inspection normalizes them to x, x-2, and x-3.

If normalization runs after every individual operation, deleting the first annotation can change the identities of the remaining entries. A subsequent update to x-2 may then hit the wrong annotation.

The fix was to normalize once on the transaction copy before marker writes, then resolve every operation against those fixed IDs. Duplicate-ID suffixes also reserve space within the 160-character limit so generated IDs remain valid.

The second bug involved a one-millisecond range near 1,000 seconds. Subtracting its endpoints can produce a value slightly below 0.001. A strict comparison can reject a valid move toward zero. The implementation now accounts for floating-point error at the minimum range boundary while continuing to reject genuinely invalid intervals.

Both bugs were easy to miss when testing a single ordinary drag. Batch operations and imported data made them visible.

An agent still needs evidence for its timestamps

UI snapping is a pointer interaction. CLI and MCP commands take exact seconds; they do not apply pixel-distance snapping.

For an event in source footage at constant playback speed, the mapping is:

projectTime = clipStart
            + (sourceTime - sourceStart) / playbackRate

With a speed curve, the corresponding source-time mapping is required. Dividing by an average speed can place the cue incorrectly.

Music cues have a similar boundary. The marker commands store annotations; they do not perform automatic beat detection. An agent needs supplied cue times or a verified analysis before writing a beat grid.

What was verified

The checks covered real CLI and MCP calls, plus an independent agent following the Skill to move a range from 5–8 to 9–12 seconds and add a Unicode note at 11 seconds.

Verification checked the range length, unrelated annotations, the original archive hash, embedded media bytes, media duration, and the unchanged rendering plan for annotation-only edits. Skill validation, type checking, the production build, and GitHub CI passed. Existing lint warnings remain.

The result is a project that can carry editorial intent between people and agents. A human can leave a review note, an agent can act on its exact location, and the next person can inspect the resulting project and its annotations.

If you want to try the flags and range snapping, open the Timeline Studio editor. If you are building a React editor or exposing a creative tool through MCP, the GitHub repository and marker workflow reference contain the implementation and command contract. Stars help you follow updates; reproducible issues and design feedback are welcome.

To install Skill v1.0.7 for Codex:

gh skill install MartinDelophy/ai-video-editor edit-timeline-studio --pin v1.0.7 --agent codex --scope user

The Skill installs the workflow. Running local commands still requires a Timeline Studio checkout with its Node dependencies, and the MCP adapter should run from that checkout. See the release notes for the published version.