DEV Community: MartinDelophy

Turning a README into a Playable Game Catalog

MartinDelophy — Wed, 09 Sep 2026 03:13:17 +0000

GitHub: awesome-gpt-6-astra
Live gallery: Astra Games

Building a game with AI leaves another problem to solve: helping people find it, try it, and understand how it was made.

I maintain Awesome GPT-6 Astra, a community collection of games and interactive projects. As the collection grew, a Markdown list became the content source for a visual gallery with screenshots and playable links.

This post explains a few design choices behind that workflow: structured entries, a README-driven catalog, real screenshots, and multilingual maintenance.

1. Give each entry a predictable structure

A collection becomes difficult to browse when every submission uses a different format. One project links to source code, another has a screenshot, and a third hides its demo inside a long development log.

We use a small, consistent entry format:

- Project name → playable demo
  - Creator → public profile
  - Platform → browser requirements, controls, access conditions
  - GPT-6 Astra → model contribution statement or development record
  - Resources → source code and setup instructions, when available
  - Preview → an actual gameplay screenshot

A demo answers “Can I try it?” Source code answers “Can I study or run it?” A development record answers “How was it built?” These fields serve different purposes.

Public source code is optional. A playable project can still be useful to explore, provided its creator is credited and the available information is clear.

2. Let the README supply the website's catalog

Maintaining a GitHub list and a separate frontend array creates duplicate work. A contributor can update the README while the gallery continues showing an old description—or misses the project entirely.

Our gallery reads the public upstream README and parses it into structured records:

README entries
      ↓
Parse titles, categories, authors, links, and screenshots
      ↓
Structured catalog records
      ↓
Gallery cards and playable links

Contributors can work with Markdown and pull requests without learning the frontend. The website handles presentation and periodically checks for catalog updates.

The tradeoff is that Markdown is now a content interface. Formatting conventions matter, and the parser needs to handle edits predictably.

3. Parse project entries, not every link

A README also contains navigation, language selectors, contribution instructions, licenses, and creator profiles. Treating every link as a game would produce a noisy catalog.

The parser needs to distinguish:

Project titles from navigation.
Demo destinations from source-code references.
Creator metadata from descriptions.
Games from interactive experiments.
Valid entries from incomplete or duplicate content.

The existing tests cover adding, deleting, and editing entries, duplicate handling, malformed documents, caching, and recovery from failed reads.

For a project like this, the useful guarantee is straightforward: a normal README contribution should produce the expected catalog update.

4. Make screenshots part of the entry

A short description rarely communicates how a game feels. An actual screenshot helps readers understand its visual style and interface before opening it.

We embed screenshots in the README entries so the gallery can use them as covers. For captured images stored in the repository, we also record their source, capture date, and the scope of the check.

That last detail matters. Entering the first level is not a full playthrough. A manual describing touch controls is not evidence that someone tested the game on a phone.

Keeping those distinctions explicit makes the catalog more useful to both readers and future maintainers. Screenshots should represent the running project, and third-party images retain their creators' rights.

5. Multilingual maintenance includes metadata

The collection has 12 README language versions. Translating a description is only one part of keeping them aligned.

For a new submission, we also check that every version includes the entry, uses the same demo and creator links, references an existing screenshot, and has consistent counts and maintenance dates.

Model attribution needs the same care. If one version says “awaiting creator confirmation,” another should not describe the contribution as independently verified.

The website's current catalog source is the default README. The translated READMEs are additional documentation that we maintain alongside it; translating them does not imply that every language is a separate runtime data source.

6. Describe AI involvement precisely

“Made with GPT-6 Astra” can describe very different development processes: implementing one mechanic, helping design an interface, iterating on game rules, or contributing across a larger project.

We aim to preserve those distinctions. Attribution follows creator or submitter statements, and missing confirmation is marked explicitly. Inclusion is not an independent model benchmark or an endorsement from OpenAI.

A useful submission explains:

What the player actually does.
Which parts Astra helped build.
What required iteration or manual changes.
What limitations remain in the current version.

These details give other developers something to learn from beyond a finished screenshot.

Share something people can play

The goal of this project is to give creators a place to show working games and connect their demos with useful development context.

Playable prototypes are welcome. If you have built a game with Astra, you can submit it through the repository with a demo, creator credit, a real screenshot, and a description of the model's contribution.

Disclosure: This article was drafted and translated with AI assistance from the maintainer's project notes and repository context.

Making a React Video Editor Agent-Friendly with Timeline Markers and MCP

MartinDelophy — Tue, 08 Sep 2026 06:51:06 +0000

A video editor needs to remember more than cuts. It also needs to remember why a cut matters.

“Start the second chapter here.” “Review this section before publishing.” “Leave a little more room at the end.” These instructions become much easier to act on when their timestamps and context live inside the project.

I recently added persistent timeline markers to Timeline Studio, the open-source browser video editor I maintain, and exposed them through its CLI, MCP adapter, and Agent Skill. The interesting engineering work was making those annotations precise, inspectable, and safe to edit in a batch.

You can try the editor in your browser or browse the source on GitHub. The implementation discussed here is included in Skill v1.0.7. Timeline Studio uses React and Vite and follows a local-first approach.

Give annotations their own data model

The editor supports four annotation types:

Type	Purpose	Timing
`marker`	A beat cue, action, or cut reference	`time`
`chapter`	The beginning of a named section	`time`
`range`	An interval to review or work on	`time`, `endTime`
`note`	Feedback tied to a precise moment	`time`

A review range looks like this:

{
  "id": "product-review",
  "type": "range",
  "time": 5,
  "endTime": 8,
  "title": "Product reveal",
  "notes": "Check whether the caption appears with the product.",
  "color": "violet"
}

Times are absolute project seconds. Titles and notes preserve Unicode, and annotations are serialized in timelineMarkers inside the portable .timeline project.

One boundary is especially useful: annotations do not contribute to rendered media duration. Adding a planning marker at 90 seconds to a 60-second edit must not produce another 30 seconds of video.

Chapter markers also do not automatically become on-screen titles or MP4 chapters. They remain editable project annotations. A marker-only task can therefore finish by writing a new project without re-encoding unchanged media.

The tradeoff is explicit timing: annotations stay at their absolute project positions during trims, reordering, and ripple edits. If they need to follow moved content, the edit plan must update them too.

Keep the React interaction compact

An always-visible annotation lane takes space away from media tracks. The default view now merges compact flags into the ruler. A toolbar chevron expands a separate lane with titles and range spans when more detail is useful.

Pressing M adds a marker at the playhead; Shift + M opens the manager. The marker UI is localized in all 13 supported interface languages.

During a drag, React holds a temporary preview. Releasing the pointer commits the edit; Escape or a canceled gesture discards the preview. This separates continuous pointer feedback from the final project mutation.

Measure snapping in pixels, then convert to time

A fixed temporal threshold such as 0.2 seconds feels very different at different zoom levels. The implementation converts a 10-pixel tolerance into project seconds:

const thresholdSeconds =
  10 / railWidth * timelineDuration;

Here, railWidth is the timeline rail width and timelineDuration is its corresponding time span. This keeps the screen-space snapping distance approximately consistent as the timeline zoom changes.

Markers reuse the shared snapping targets: the playhead, clip boundaries, and other marker edges. The editor shows a common alignment guide, and holding Alt temporarily bypasses snapping.

Ranges need an additional invariant: moving a range must preserve its length. If the end of a three-second range snaps to 12 seconds, its start must become 9 seconds.

The core calculation, simplified, is:

const nextStart = movingEdge === "end"
  ? targetTime - duration
  : targetTime;
const nextEnd = nextStart + duration;

The actual implementation checks both edges and chooses the nearer valid snap candidate. Resizing an edge is handled separately from moving the whole range.

Expose one command engine through CLI and MCP

The agent-facing architecture is small:

Agent Skill: workflow, timing evidence, verification
                         |
CLI / MCP: structured operation entry points
                         |
Shared command engine: validation, edits, semantic diff
                         |
A new .timeline project

The MCP adapter invokes the existing CLI runner. It does not maintain its own marker reducers. Fixing a validation rule therefore fixes both entry points.

From the repository root, an agent can inspect the project and its annotations:

npm run agent -- project.inspect /projects/input.timeline
npm run agent -- marker.inspect /projects/input.timeline

MCP exposes the read-only timeline_marker_inspect tool. Writes use marker.add, marker.update, and marker.delete through the existing project diff/apply workflow.

For example, suppose inspection reports revision 0 and a product-review range spanning 5–8 seconds. This plan moves it to 9 seconds and adds a note at 11 seconds:

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 0,
  "operations": [
    {
      "id": "move-product-v1",
      "type": "marker.update",
      "markerId": "product-review",
      "time": 9
    },
    {
      "id": "add-ending-note-v1",
      "type": "marker.add",
      "markerId": "ending-note",
      "markerType": "note",
      "time": 11,
      "title": "Ending rhythm",
      "notes": "Reviewer request: leave a little more breathing room.",
      "color": "rose"
    }
  ],
  "output": { "project": "/projects/output-marked.timeline" }
}

The paths, revision, IDs, and review request are illustrative. A real plan must use inspected values and actual feedback.

Two distinctions matter: id identifies an operation for retry handling, while markerId identifies the annotation. Likewise, type selects the command, while markerType selects the annotation kind.

Updating only the time of a range preserves its length. The example moves 5–8 seconds to 9–12 seconds. An explicit endTime lets the plan change the end instead.

Save the plan as /projects/markers-plan.json, validate its structure, and preview the semantic diff:

node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs /projects/markers-plan.json
npm run agent -- project.diff /projects/markers-plan.json

After reviewing the result, apply the same plan and inspect the output:

npm run agent -- project.run /projects/markers-plan.json
npm run agent -- marker.inspect /projects/output-marked.timeline

The diff reports additions, removals, and modifications under changes.markers, including before/after values. A marker-only edit should not unexpectedly modify a media track.

The Skill instructs the agent to inspect, review the diff, apply, and inspect again. The engine enforces concrete constraints:

A mismatched revision returns REVISION_CONFLICT.
Previously applied operation IDs are not applied again.
A failed operation rejects the batch without writing a partially edited archive.
Output must go to a new path; input files and existing outputs cannot be overwritten.

A revision conflict requires a fresh inspection and a reconsidered plan. Merely replacing the revision number would skip the reason for having that check.

The tricky bugs were identity and floating-point boundaries

Consider imported annotations with three identical IDs: x, x, and x. Inspection normalizes them to x, x-2, and x-3.

If normalization runs after every individual operation, deleting the first annotation can change the identities of the remaining entries. A subsequent update to x-2 may then hit the wrong annotation.

The fix was to normalize once on the transaction copy before marker writes, then resolve every operation against those fixed IDs. Duplicate-ID suffixes also reserve space within the 160-character limit so generated IDs remain valid.

The second bug involved a one-millisecond range near 1,000 seconds. Subtracting its endpoints can produce a value slightly below 0.001. A strict comparison can reject a valid move toward zero. The implementation now accounts for floating-point error at the minimum range boundary while continuing to reject genuinely invalid intervals.

Both bugs were easy to miss when testing a single ordinary drag. Batch operations and imported data made them visible.

An agent still needs evidence for its timestamps

UI snapping is a pointer interaction. CLI and MCP commands take exact seconds; they do not apply pixel-distance snapping.

For an event in source footage at constant playback speed, the mapping is:

projectTime = clipStart
            + (sourceTime - sourceStart) / playbackRate

With a speed curve, the corresponding source-time mapping is required. Dividing by an average speed can place the cue incorrectly.

Music cues have a similar boundary. The marker commands store annotations; they do not perform automatic beat detection. An agent needs supplied cue times or a verified analysis before writing a beat grid.

What was verified

The checks covered real CLI and MCP calls, plus an independent agent following the Skill to move a range from 5–8 to 9–12 seconds and add a Unicode note at 11 seconds.

Verification checked the range length, unrelated annotations, the original archive hash, embedded media bytes, media duration, and the unchanged rendering plan for annotation-only edits. Skill validation, type checking, the production build, and GitHub CI passed. Existing lint warnings remain.

The result is a project that can carry editorial intent between people and agents. A human can leave a review note, an agent can act on its exact location, and the next person can inspect the resulting project and its annotations.

If you want to try the flags and range snapping, open the Timeline Studio editor. If you are building a React editor or exposing a creative tool through MCP, the GitHub repository and marker workflow reference contain the implementation and command contract. Stars help you follow updates; reproducible issues and design feedback are welcome.

To install Skill v1.0.7 for Codex:

gh skill install MartinDelophy/ai-video-editor edit-timeline-studio --pin v1.0.7 --agent codex --scope user

The Skill installs the workflow. Running local commands still requires a Timeline Studio checkout with its Node dependencies, and the MCP adapter should run from that checkout. See the release notes for the published version.

What It Takes to Run an AI Video Editor Locally in the Browser

MartinDelophy — Tue, 25 Aug 2026 08:33:27 +0000

Most AI video tools begin with an upload button.

That is convenient for the application developer, but it changes the product for the user. Raw footage can be large, personal, slow to transfer, and expensive to process repeatedly. For a video editor, uploading is not a one-time action either: every trim, caption, effect, and export can create another round trip.

I wanted to explore a different constraint: how much of an AI video-editing workflow can stay inside a modern browser?

That question became Timeline Studio, an open-source, local-first AI video editor with a multi-track timeline, captions, voiceovers, visual effects, browser AI tools, and offline export.

The interesting part was not putting a familiar editor UI on a webpage. It was making media, AI inference, model delivery, timeline state, preview, and export behave like one coherent system.

Local-first is an architectural constraint

For supported workflows, project media is processed in the browser instead of being uploaded to an editing backend. This changes several engineering decisions at once:

inference needs browser-compatible runtimes;
model downloads must be lazy, resumable, cached, and versioned;
timeline operations need deterministic state rather than server-side session magic;
preview must stay responsive while export remains reproducible;
failures need to explain browser or hardware limitations without hiding them.

WebGPU is the main compute path for heavier models. WASM remains important for operations where it is more stable or broadly compatible. ONNX gives the project a practical interchange format, but converting a model is only the beginning. A model that technically loads can still be unusable because of memory pressure, unsupported operators, slow graph initialization, or inconsistent numerical output across execution providers.

The result is a mixed runtime rather than a single “run everything on WebGPU” switch.

Model delivery is part of the product

A local model still has to reach the user once.

Timeline Studio lazy-loads AI artifacts only when a feature is first used. The service worker owns the persistent model cache, while later runs reuse the cached files. Models are pinned to known revisions so an upstream change cannot silently alter an editing workflow.

The project also mirrors model artifacts across Hugging Face and ModelScope. Chinese and domestic sessions prefer ModelScope; other sessions prefer Hugging Face. If the preferred provider is unavailable, the application can fall back to the other mirror.

The important detail is that both providers map to one cache identity. A fallback should not create a second full copy of the same model on the user's device.

This sounds like infrastructure work—and it is—but it directly affects the editing experience. “Generate music” should not look like a fresh model download every time. A cache miss should not become a mysterious network exception. A pinned model should carry its license and source notes with it.

A responsive preview and a deterministic export are different jobs

Video editors need immediate feedback. Native media playback and a live canvas make trimming and positioning feel responsive, but they are not enough for final export.

Real-time playback can vary with decoding speed, dropped frames, tab scheduling, and device load. If final rendering simply records whatever happened on screen, two exports of the same project can drift.

Timeline Studio therefore treats the timeline as declarative project state and uses a separate offline rendering path for export. Clips, source trims, transforms, captions, overlays, audio, effects, and keyframes are evaluated against explicit timestamps. WebCodecs is used for MP4/WebM composition where available, with a compatibility path for browsers that need it.

The two paths have different performance goals:

Preview: respond quickly enough to edit.
Export: evaluate the same project state consistently.

They still need shared geometry, timing, interpolation, and effect definitions. Otherwise the editor develops the most damaging kind of bug: a preview that looks correct and an export that does not.

This is why small visual features can require surprisingly deep work. A timed ripple effect, for example, is not just a CSS animation. Its seeded placement, beat timing, grayscale-to-color reveal, wave propagation, and decay must all be derived from the same deterministic parameters in both preview and export.

AI output should become editable media

Another design decision was to avoid treating AI as a collection of isolated demo buttons.

Generated or analyzed results should return to the editing model:

automatic captions become timed caption segments;
generated voiceovers become audio clips;
separated vocals and instrumentals become independent audio tracks;
AI music is added to the user's assets instead of silently replacing the timeline;
repair and subject-isolation results remain reviewable and editable;
portable .timeline projects preserve the composition as the source of truth.

The browser currently supports workflows including multilingual voice generation, Whisper-based captions, local AI music, smart framing, object or watermark repair, restoration, vocal separation, and talking-avatar generation. Not every device will run every large model comfortably, so the UI has to expose setup progress, cancellation, caching, and compatibility honestly.

What did not fit the local-first story

There is a temptation to claim that moving inference into the browser removes every tradeoff. It does not.

The first model download may be substantial. GPU memory varies widely. WebGPU behavior still differs across browser and driver combinations. Mobile devices require more conservative paths. Some models become slower or less reliable after conversion, even when their quality looked good in the original research environment.

Local-first also does not mean “offline from the first visit.” The application shell and requested models must be downloaded before they can be cached. A more accurate promise is: supported editing workflows can run without uploading the user's project media to an editing backend.

That narrower statement is less dramatic, but it is testable.

Why open source it?

Browser media work sits at the intersection of WebCodecs, WebGPU, ONNX, audio processing, timeline UX, caching, and export. Many of the hardest problems are integration problems, and they benefit from reproducible bug reports and implementations that can be inspected.

Timeline Studio is available under the MIT License on GitHub. The repository includes the editor, multilingual documentation, portable project tooling, and an agent skill for inspecting, editing, diffing, and rendering timeline projects.

There is still plenty to improve: broader headless render parity, more reliable cross-device export, a larger versioned command registry, and better recovery when browser AI reaches hardware limits.

Launching today

Timeline Studio is launching on Product Hunt today.

If local-first creative software interests you, try the editor and share your honest feedback. I am especially interested in where the browser-local model feels clear, where setup is confusing, and which real editing workflow breaks first.

The Best Feature I Shipped Was Removing Friction: Rebuilding the UX of a Browser Video Editor

MartinDelophy — Mon, 24 Aug 2026 08:30:27 +0000

When I first shared Timeline Studio, it had around 80 GitHub stars.

The technical premise was already there: a local-first video editor that runs in the browser, with a multi-track timeline, voiceovers, captions, effects, and offline export.

The user experience was not.

Some early feedback was blunt:

Interesting project, but the interface feels clunky.

There are a lot of features, but some interactions feel unintuitive.

That feedback was fair. I had spent too much time proving that browser video editing was possible and not enough time asking whether ordinary editing actions behaved the way people expected.

The project has since passed 500 GitHub stars. The number is nice, but the more meaningful milestone was going back and fixing the small frustrations that made the entire editor feel harder than it needed to be.

This is a post about those fixes.

A feature can work and still be wrong

The early version had split, zoom, scrolling, track locking, audio separation, mobile controls, and export settings.

On paper, those features existed.

In practice, they often exposed the implementation instead of matching the user's mental model.

For example, the split action originally required an explicitly selected clip. A user could place the playhead in the middle of the main video, click the scissors button, and still be told to select a clip first.

The code was consistent:

selectedClip -> split(selectedClip, playheadTime)

The interaction was not.

The user was looking at the main visual under the playhead. They had already communicated enough intent.

The current rule is:

if a clip is explicitly selected:
    split that clip
else:
    find the main visual under the playhead
    select and split it

Explicit selections are still respected, but the common action now has a useful default.

This change added no new capability. It simply removed a question the editor should never have asked.

Lock the meaning of a continuous gesture

Timeline editors usually need several wheel behaviors:

vertical track scrolling
horizontal timeline panning
timeline zoom
Shift-modified panning
Ctrl/Cmd wheel zoom
trackpad pinch zoom

My initial implementation kept resolving the wheel behavior from whatever happened to be under the pointer.

That sounds reasonable until the timeline moves.

Imagine starting a vertical scroll over empty timeline space. As the tracks move, a clip slides under the stationary pointer. If the editor resolves the target again, the gesture can suddenly change from scrolling tracks to zooming the timeline.

From the user's perspective, the editor has changed its mind halfway through an action.

The fix was to treat a continuous wheel sequence as one gesture:

gestureTarget = hitTest(pointerPositionAtStart)
gestureMode = resolveMode(gestureTarget, modifierKeys)

while the gesture continues:
    keep gestureMode stable

after a short pause:
    allow the next gesture to resolve again

The resulting behavior is predictable:

start over empty timeline space to scroll track lanes
start over a clip to zoom the timeline
use Shift for horizontal panning
use Ctrl/Cmd or pinch for explicit zoom

The important part is not the exact mapping. It is that the mapping does not mutate while the user is still performing the same physical gesture.

Users manipulate clips, not source asset IDs

A split video often produces several timeline clips that reference the same source file.

Internally, that might look like this:

sourceAssetId: video-001

clip A -> source range 0s–5s
clip B -> source range 5s–12s
clip C -> source range 12s–20s

An early version of "Separate audio" relied too heavily on the shared source asset ID. That made whole-asset matching convenient, but it violated what the user had actually asked for.

When the command comes from clip B's context menu, the expected result is the audio for 5s–12s—not new mappings for every sibling clip backed by video-001.

The current behavior operates on the exact timeline segment:

preserve its source trim
preserve its playback speed
place the extracted audio on an audio lane
mute the corresponding picture-in-picture clip to prevent doubled sound
select and reveal the resulting audio
leave untouched sibling clips alone

Whole-asset matching still makes sense when an unsplit source asset is intentionally dragged onto a source-audio track. It does not make sense for a clip-scoped menu action.

The lesson was simple: internal identity is not the same thing as interaction scope.

Automatic layout should not rearrange confirmed work

Audio lane placement caused a similar problem.

When a new voice clip overlapped existing audio, an automatic placement algorithm could move an existing clip to another lane. The layout became technically valid, but the user's previously arranged work changed without permission.

Now existing clips stay where they are.

New voice or audio clips find another available lane, or a new ordinary audio lane is materialized when necessary. AI music always routes to the dedicated music track. Split audio can still be moved vertically between normal audio lanes, but adding something new never displaces something already confirmed.

Automation should solve the new placement problem, not reopen old decisions.

Keep the primary visual in view

Once a project contains overlays, captions, source audio, voiceovers, and music, vertical scrolling becomes unavoidable.

Letting every track scroll together meant the main visual could disappear while the user worked on lower lanes. That removed the most important reference point in the composition.

The main visual track is now pinned directly below the ruler. Overlay and lower tracks scroll underneath it.

The persistent scrollbar chrome is hidden, but both horizontal and vertical scrolling remain functional. The pinned track's lower edge also acts as the upper boundary for vertical drag auto-scroll, so cross-lane moves still behave correctly.

This required more than position: sticky. Pointer hit testing, drag targets, auto-scroll boundaries, and main-to-overlay or overlay-to-main moves all had to use the same geometry.

The visible result is much less dramatic: the main picture simply stops getting lost.

Mobile is not desktop with smaller buttons

The first mobile layout was essentially the desktop editor compressed into a narrow viewport.

It technically contained the same features. It was also exhausting to use.

The redesigned mobile workspace follows a different hierarchy:

preview first
timeline immediately below it
a centered playhead with the timeline moving around it
clip-specific actions after selection
properties inside a focused bottom drawer
touch-sized controls
no desktop keyboard-shortcut guide

Desktop and mobile still edit the same project model. They do not need to expose that model through the same interface.

Responsive design is not preserving every panel at every width. It is preserving the user's task.

Small state mismatches create large trust problems

Several fixes were almost embarrassingly small:

Clicking the Export button a second time now closes the export popover.
Icon-only timeline controls have localized tooltips on hover and keyboard focus.
Locked tracks retain a subdued treatment, but their clips no longer gain a dashed border that looks like selection.
"Analyze person" and "Analyze object" use the primary action style whenever they are actionable; the neutral style is reserved for "Cancel analysis."
Generated music and converted voices go to the asset library instead of silently modifying the timeline.
Repeated voice generations append after the current voiceover end instead of stacking at 0 seconds or an unchanged playhead.

None of these belongs in an impressive model benchmark.

Together, they answer a more important question: can the user predict what will happen after the next click?

The product changed when the question changed

At the beginning, I kept asking:

Can this run in a browser?

Can WebCodecs export the composition? Can WebGPU run the model? Can the timeline support another track type? Can an agent modify the project?

Those were useful questions, but they produced a technology demonstration.

The editor started becoming a product when the question changed to:

What does the user reasonably expect to happen here?

That question led to fewer interruptions, stable gestures, clip-scoped actions, conservative automation, clearer state, and a mobile interface built around touch rather than CSS breakpoints.

Timeline Studio is still evolving. It combines a real multi-track editor with local-first browser AI, portable .timeline projects, and deterministic offline export. There are still plenty of rough edges to find.

But if you tried an early version and found the UI clunky, I would genuinely like to know whether the same problems still stand—and what remains confusing today.

GitHub: https://github.com/MartinDelophy/ai-video-editor
Live editor: https://video-editor.ai-creator.top/

The most useful feedback is not "add another AI model."

It is: "I expected this action to do X, but it did Y."

AI Video Generation Is Expensive—So I Built a 15-Second Remix Workflow

MartinDelophy — Sat, 22 Aug 2026 09:11:55 +0000

Generating AI video looks effortless in a demo: write a prompt, wait a minute, and receive a cinematic shot.

In practice, the expensive part is not only the subscription or the credits. It is the uncertainty. A usable five-second clip may require several generations because the face changes, the motion breaks, the composition drifts, or the shot simply does not fit the rest of the sequence.

That led me to a different question:

What if creators did not have to generate every frame from scratch?

I tested a more practical workflow: start with video material I am authorized to use, analyze it for high-impact moments, rebuild the pacing, replace the original audio, and keep the result as an editable timeline instead of a disposable render.

The tool behind the experiment is Timeline Studio, an open-source, browser-based video editor:

👉 GitHub: MartinDelophy/ai-video-editor

The experiment

The input was simple:

one source video of roughly two minutes;
one music track of more than five minutes;
a second video containing a specific three-second ending shot.

The target was a 15-second short-form edit with:

multiple memory-like fragments;
a clear emotional build;
a transition in the middle;
a different final shot;
the original source audio removed;
a musical highlight near 1:20;
a real ending, rather than an abrupt cutoff.

The final structure looked like this:

Time	Editorial function
0–3s	Establish the character and atmosphere
3–7s	Build a fragmented memory rhythm
Around 7s	Use a short flash transition as an emotional hinge
7–12s	Increase energy while protecting readable hero frames
12–15s	Switch to the requested ending shot and fade to black

The final three seconds came from 2:48–2:51 of the second source. Its embedded audio was muted, so the entire edit used only one clean music track.

Why “highlight detection” is not random trimming

A common automation mistake is to cut a long video into equal intervals and keep the most active sections.

That produces motion, but not necessarily meaning.

For this experiment, candidate moments were evaluated with several kinds of evidence:

audio-energy change;
subject-aware motion and frame difference;
shot-change confidence;
expression change for a tracked face;
clarity around the main subject.

A practical scoring model can combine these signals:

highlight_score =
    0.30 * audio_energy_change
  + 0.25 * subject_motion
  + 0.20 * shot_change_confidence
  + 0.15 * expression_change
  + 0.10 * subject_clarity

But the score is only a candidate signal. It should not override continuity, intentional stillness, anticipation, or the most important result frame.

After saliency analysis, each retained beat still needs an editorial role: setup, rise, pre-impact, peak, aftershock, or bridge. Without that hierarchy, every shot receives equal weight and the montage feels flat.

Finding the musical high point

The user requested the highlight near 1:20, so I searched around that region instead of blindly cutting from exactly 80 seconds.

The selected range was approximately 80.5–95.5 seconds. That gave the 15-second edit a more natural musical phrase and enough room for a controlled ending.

A simplified FFmpeg extraction looks like this:

ffmpeg \
  -ss 80.5 \
  -t 15 \
  -i input.aac \
  -af "afade=t=in:st=0:d=0.15,afade=t=out:st=14.3:d=0.7" \
  -c:a aac \
  -b:a 192k \
  music-highlight.m4a

This does four important things:

starts close to the requested musical moment;
limits the result to exactly 15 seconds;
avoids a hard entrance;
fades the final 0.7 seconds so the music and picture resolve together.

The loudest passage is not automatically the best passage. A useful music highlight also needs a readable beat, a complete phrase, an emotional lift, and a point where the ending can land.

Creating the “memory fragment” feeling

A memory montage is not just a color preset.

The feeling came from combining several editorial decisions:

1. Short, non-contiguous fragments

The first 12 seconds use eight short clips instead of one continuous excerpt. The clips come from different source times and are reorganized around emotional progression.

2. A restrained flash transition

A brief flash near the middle separates two emotional phases. It is intentionally short and placed away from the decisive frame.

Transitions should clarify structure. If an effect hides the action viewers need to see, it is working against the edit.

3. A protected ending

The requested ending shot occupies exactly 12–15 seconds. The final 0.7 seconds fade to black, giving the piece a deliberate closing gesture.

4. One coherent audio bed

All embedded source audio is muted. Keeping only the selected music highlight avoids doubled sound and makes later remixing predictable.

The workflow stayed editable

A rendered MP4 is only the delivery artifact. The actual source of truth is the editable timeline.

The project contains:

eight memory fragments;
one three-second ending clip;
source-time trims for every segment;
a middle flash transition;
a final fade;
one music segment;
muted embedded audio;
normalized output dimensions and frame rate.

This matters because editing is iterative. If the ending needs to start 300 milliseconds earlier, I should not have to reconstruct a giant filter graph or repeat the full analysis.

Timeline Studio stores the edit as a .timeline project, so clips, timing, transitions, audio, and effects can continue to be adjusted visually or through automation.

The screenshot above captures part of the actual production process. The ending source ran at 30 fps, while the main sequence used 29 fps, so the render pipeline normalized the final output and fitted the new shot to the project resolution without reintroducing its original sound.

Technical validation

A video is not finished when the encoder exits successfully.

For the final result, I verified:

duration: 15.000 seconds;
resolution: 1906 × 1080;
frame rate: 29 fps;
total frames: 435;
video codec: H.264;
audio codec: AAC stereo;
full decode: successful;
the cut at 12 seconds;
the fade between 14.3 and 15 seconds;
no embedded audio leaking from the ending clip.

Basic metadata inspection can be automated with ffprobe:

ffprobe -v error \
  -show_entries format=duration \
  -show_entries stream=index,codec_name,width,height,r_frame_rate,channels \
  -of json output.mp4

I also recommend decoding the complete output once instead of trusting container metadata alone:

ffmpeg -v error -i output.mp4 -f null -

That catches broken frames and stream errors that a successful export message may miss.

Why this can be cheaper than generation-only production

This workflow does not replace generative video. It changes where generation is used.

Generation is excellent when a story requires a shot that does not exist. But when usable footage already exists, repeatedly generating near-duplicates is often a poor use of time and credits.

A hybrid pipeline can be more efficient:

reuse authorized footage where it already works;
analyze and restructure it;
generate only the missing shots;
import those shots as editable source media;
validate the finished sequence as one coherent piece.

The scarce skill is gradually moving from “Can I generate an image?” to “Can I select, structure, pace, and finish a story?”

A necessary note about copyright

“Remix” does not mean downloading and reposting somebody else’s work.

Use footage that you created, generated, licensed, purchased, or received permission to modify. Check whether the license permits derivatives and commercial use. Keep provenance and watermarks when required, and verify music, likeness, and brand rights separately.

Good second creation adds a new structure, viewpoint, explanation, or experience. Removing a watermark and uploading the same content is not a creative workflow—and may violate both rights and platform policies.

About Timeline Studio

Timeline Studio is my attempt to connect local media analysis, browser-based editing, and deterministic rendering in one workflow.

The project is useful for experiments involving:

content-aware highlight extraction;
short-form montage generation;
music highlight selection;
editable transitions and effects;
caption and voice workflows;
AI-generated footage as source media;
local-first media processing;
reusable .timeline projects.

If this direction is useful to you, take a look at the repository:

https://github.com/MartinDelophy/ai-video-editor

Issues, feedback, and contributions are welcome.

Final thought

AI video generation lowers the cost of producing footage, but more footage does not automatically create better stories.

Generation determines what material is available. Editing determines what the audience experiences.

For independent creators, the sustainable advantage may not be generating everything from zero. It may be building a repeatable system that knows what to keep, what to cut, when to peak, and how to end.

AI Video Editing Should Be More Than Auto-Cutting: Teaching an Agent Professional Post-Production

MartinDelophy — Wed, 19 Aug 2026 08:02:37 +0000

Most AI video editors can already select clips, remove silence, generate captions, add music, and cut on the beat.

That sounds impressive—until you watch the result.

The shots may be technically correct, but the edit often has no hierarchy. The pacing feels templated. Color jumps between cameras. Subject edges flicker. A product reveal passes too quickly. The editor preview looks right, but the exported file does not.

The problem is not always the model.

In many systems, the agent has learned how to operate editing controls, but not how to reason like an editor, colorist, or compositor.

We are building Timeline Studio, an open-source AI video editor focused on editable projects, local-first media workflows, and verifiable output.

In our latest update, we expanded the Timeline Studio video-editing skill with professional guidance for:

source-time speed curves
Color Wheels and shot matching
person, product, and object cutouts
subject-aware outlines
effect ordering and export validation

You can inspect the exact update in this commit.

The goal is simple: stop treating these capabilities as feature labels and start teaching the agent when to use them, why they matter, how to validate them, and when a result should be rejected.

The gap between an automatic edit and a professional edit

Automatic editing usually answers questions such as:

Which clips are relevant?
Where are the silent sections?
What captions should be generated?
Where are the music beats?
In what order should the shots appear?

Professional post-production adds a different layer of reasoning:

Where does an action actually begin?
Which frame contains the clearest consequence?
Should the approach be accelerated while the result stays readable?
Why do two adjacent shots feel visually disconnected?
Is a detected bounding box good enough for compositing?
Should the background be removed, or should the subject be emphasized while preserving context?
Did the final export retain the timing, grade, matte, and edge treatment seen in the editor?

These are not minor details. They determine whether a video feels intentionally directed or mechanically assembled.

1. Speed curves should follow the action, not a template

A common automated speed ramp looks like this:

1x -> 2x -> 0.5x -> 1x

It creates motion, but it does not necessarily create meaning.

A useful speed curve begins with the source-time structure of the shot:

action onset
-> anticipation
-> approach
-> contact or reveal
-> readable result
-> release

Only after identifying those anchors should the agent change timing.

For a product reveal, a motivated curve might be:

normal speed
-> compress low-information travel
-> slow at the first unobstructed product frame
-> hold the hero result
-> return to normal

For a software tutorial, the agent can accelerate cursor travel and waiting time, but it should return to normal speed before the click, state change, number, or result the viewer must understand.

The skill also defines rejection conditions. A speed curve should fail review if it causes unreadable action, damaged dialogue, repeated boundary frames, visible stutter, broken audio continuity, or a weaker payoff than the original shot.

The key principle is to preserve source-time reasoning even after the clip is retimed.

2. Color matching comes before a “cinematic” look

Applying the same LUT or filter to every clip does not make a sequence consistent.

Two shots can share the same preset and still disagree in white balance, exposure, black level, highlight color, skin tone, product color, saturation, and contrast.

The updated skill uses a disciplined Color Wheels workflow:

Inspect representative native frames.
Correct temperature and tint.
Establish global exposure and luminance.
Shape shadows, midtones, and highlights independently.
Adjust saturation last.
Compare adjacent shots at the actual cut boundary.

Timeline Studio exposes controls for Shadows, Midtones, Highlights, Global Offset, Temperature, Tint, and Saturation.

For portraits, the agent must preserve plausible skin, lips, teeth, eyes, and hair.

For product shots, it must protect packaging, logos, materials, and brand colors. The background can move toward a complementary palette; the product itself should not be recolored merely to create a dramatic grade.

Color parameters can be keyframed, but static correction remains the default. Animated grading should only be used when lighting or narrative intent genuinely changes.

3. Detection is not the same as a usable cutout

Modern vision models can detect a person or product quickly. But a bounding box is only a region proposal.

A production-ready cutout requires a stable alpha matte.

For people, the agent needs to inspect hair, fingers, gaps between limbs, semi-transparent fabric, motion blur, held objects, occlusion, and re-entry.

For products and objects, it needs to preserve handles, straps, holes, thin structures, transparent packaging, reflective edges, labels, logos, and the correct instance when similar objects appear.

The workflow separates three stages:

detect the target
-> generate the silhouette matte
-> validate temporal stability

For video, validation cannot stop at the first frame. The matte should be checked at fast-motion intervals, occlusions, shot boundaries, exits, re-entries, and the final frame.

The result should be rejected if it shows target switching, missing body or product parts, edge chatter, stale masks, lag, or simplified geometry that damages the subject.

Only after the raw matte passes review should the agent add a new background, depth, shadow, glow, or outline.

4. Outlines can preserve context while directing attention

Removing the background is not always the right choice.

In tutorials, interviews, product demonstrations, and documentary footage, the environment may contain information the viewer needs. A restrained subject outline can improve attention without destroying context.

A simple routing rule works well:

Need a new environment or foreground/background layering?
Use a cutout.

Need to preserve the environment but improve subject acquisition?
Use an outline.

Useful outline scenarios include presenter emphasis, product callouts, freeze-frame introductions, before/after comparisons, transitions between live footage and graphics, and small-screen attention guidance.

The outline must be derived from a verified alpha matte—not from a detection rectangle.

Its width, opacity, softness, glow, and shadow should be judged at the actual delivery size. An outline that looks subtle in a zoomed desktop preview may become overpowering on a phone, or disappear completely.

5. Professionalism comes from effect orchestration

Adding more effects does not automatically create a better video.

An agent with access to cutouts, outlines, depth, parallax, glow, Color Wheels, and speed ramps can easily over-process a shot unless every layer has an editorial job.

A product hero shot might follow this order:

technical color match
-> product matte validation
-> background composition
-> restrained shadow or outline
-> motivated speed curve around the reveal
-> readable result hold

A presenter shot may need much less:

person cutout
-> supporting graphics behind the presenter
-> subtle outline only where contrast requires it
-> normal speed during speech

Before adding an effect, the agent should be able to answer:

Does it separate the subject?
Does it clarify evidence?
Does it improve rhythm?
Does it support a narrative transition?
Does it preserve the decisive frame?

If the effect has no clear function, it should not be added.

6. The exported video—not the editor state—is the deliverable

A correct editor preview does not guarantee a correct export.

Complex projects can lose or alter grading state, alpha masks, edge treatment, time mapping, keyframe interpolation, or effect ordering.

The delivery workflow is therefore:

preserve the editable project
-> preview the complete timeline
-> export the video
-> decode the exported file
-> compare matching timestamps
-> verify color, alpha, edges, and timing

The agent should retain the editable .timeline project and verify the actual media file instead of trusting a success notification.

There is also an important implementation boundary: advanced Color Wheels, speed curves, vision-derived masks, and subject effects currently belong to the editor/browser execution path. The headless renderer does not yet provide full parity for these features.

We document that limitation instead of pretending every rendering path behaves identically.

Why encode this knowledge in a skill?

A capable model can understand video concepts, but repeatable production work still benefits from explicit operational guidance.

The skill provides routing rules, execution order, source-time reasoning, validation criteria, rejection conditions, current capability boundaries, and editable-project requirements.

This turns “the editor supports Color Wheels” into something much more useful:

Match adjacent shots, protect identity-bearing colors, compare the real cut boundary, reject clipped or unstable results, and verify the export.

The same principle applies to speed curves, cutouts, and outlines.

Features become professional capabilities only when the agent knows how to reason about them.

What comes next

Automatic trimming, captions, music matching, and beat cuts are rapidly becoming baseline features.

The next generation of AI video tools will compete on deeper questions:

Can the agent understand the visual consequence of an action?
Can it build a real hierarchy of setup, rise, peak, and release?
Can it preserve skin, products, and brand identity during grading?
Can it produce temporally stable subject masks?
Can it choose the smallest justified visual treatment?
Can it validate the final render?
Can it leave behind a project a human editor can continue modifying?

That is the direction we are exploring with Timeline Studio.

If you are working on AI agents, browser-local ML, automated editing, media pipelines, or editable video formats, take a look:

Feedback, issues, and stars are welcome.

I Connected Codex for Chrome to an Open-Source Video Editing Skill—Here’s What Changed

MartinDelophy — Mon, 17 Aug 2026 10:06:20 +0000

AI can already write scripts, generate images, synthesize voices, and summarize videos. Yet real video production still feels fragmented.

You watch a reference on YouTube. You open a product website in another tab. You copy timestamps into notes, download authorized assets, write narration, generate voice clips, and then rebuild all of that context inside an editor.

Every step may contain AI, but the steps do not share a working memory.

That is why the new Codex Chrome extension caught my attention. When combined with an editing-specific agent skill and an editor built around portable timelines, it creates something more useful than another “generate video” button:

A browser-aware agent that can turn researched context into an editable video project.

I have been exploring this workflow with Timeline Studio, an open-source, local-first browser video editor, and its edit-timeline-studio agent skill.

The missing link was not generation

Most AI video conversations focus on generation quality: longer clips, better motion, more consistent characters, or faster inference.

Those improvements matter, but editing is a different problem.

A real edit contains hundreds of decisions:

Which source moments should survive?
Where should a spoken phrase end?
Is a repeated shot building anticipation or just wasting time?
Does the subject stay readable after reframing?
Is every visible caption backed by audible speech?
Should picture timing follow the narration, or should narration be rushed to fit a predetermined cut?
Can a human revise the result without starting again?

An agent that only clicks an editor is a faster mouse. An agent that only understands a webpage is a better researcher. The interesting system appears when browser context, editorial reasoning, and a real project model are connected.

What Codex for Chrome changes

OpenAI introduced Codex for Chrome on May 7, 2026. Later browser updates added several capabilities that are unusually relevant to content workflows:

referencing open Chrome tabs;
bringing highlighted page text into a conversation;
opening Ask ChatGPT from a page's context menu;
using timestamped transcripts when captions are available on YouTube;
working with sites where the user is already signed in;
inspecting DOM, styles, console output, network traffic, and performance through Developer Mode.

The important change is not simply that an agent can open a URL. It can work with the browser environment in which research is already happening.

Open a reference video and the transcript can become structured evidence. Open a product website and the agent can verify which capabilities are actually visible. Open an authorized asset source and the agent can organize candidates without repeatedly asking the user to copy page context into chat.

Browser understanding is the perception layer. It still needs an editing layer.

An editing skill, not a mega-prompt

Timeline Studio is a local-first AI video editor that runs in the browser. It combines a multi-track timeline with captions, voiceovers, music, visual effects, transitions, and offline export.

The repository also ships an agent skill named edit-timeline-studio. The skill is not a single prompt that says “make this cinematic.” It defines how an agent should inspect, plan, execute, and verify an edit.

Among other things, it requires an agent to:

inspect duration, dimensions, media type, and audio before editing;
analyze representative frames, speech, OCR, motion, and subject regions;
classify the job as a talking-head cleanup, tutorial, highlight reel, multi-speaker edit, promotion, or reference reconstruction;
record keep, remove, shorten, and reorder decisions against source time;
build narration from short physical voice assets and lock the audio spine before timing the picture;
verify captions against audible speech;
inspect boundaries, transitions, audio behavior, and the final decoded render;
deliver both a rendered video and a reopenable .timeline project.

The editable project remains the source of truth. The render is an output, not the only artifact.

A three-layer video agent

The combined architecture is straightforward:

Codex Chrome extension
  authorized tabs, signed-in pages, selected text, timestamped transcripts
                              ↓
edit-timeline-studio skill
  evidence analysis, narrative planning, edit decisions, execution, validation
                              ↓
Timeline Studio
  tracks, clips, captions, voiceovers, music, effects, .timeline project, render

Each layer has a distinct responsibility.

Chrome answers: What is happening in the user's web context?

The skill answers: What editorial decisions should be made, and how should they be verified?

Timeline Studio answers: Where do those decisions live so that a human can continue editing?

This separation matters. It prevents browser automation from being mistaken for editorial judgment, and it prevents AI output from collapsing into an opaque one-shot render.

Workflow 1: reconstruct why a reference video works

Asking an AI to “summarize this video” rarely produces an actionable edit plan. A serious reference reconstruction needs to identify:

shot and sub-shot boundaries;
repeated source families;
speed ramps, holds, and reversals;
transition anatomy;
subject center, scale, and movement path;
changes in look or filter state;
the relationship between retained audio, narration, captions, and picture;
setup, rise, pre-impact, peak, aftershock, and bridge beats.

With the reference open in Chrome, the agent can use the available transcript and page context as one evidence lane. Local analysis can then inspect the actual media for timing, motion, composition, and audio.

A task can begin like this:

@Chrome analyze the reference video in my current tab and use its timestamped
transcript when available. Identify the narrative sections and verified page
context.

Then use the edit-timeline-studio skill to reconstruct the shot order,
repetitions, speed changes, transitions, subject movement, and tension curve.
Use only media I provide or media with an explicit platform-provided download
and suitable reuse rights. Deliver an editable .timeline project and a verified
render. Mark anything that cannot be established from evidence.

The result is no longer just an explanation of the video. It is an explanation expressed as an editable timeline.

Workflow 2: turn a product website into a narrative, not a feature reel

Many AI-generated product videos are feature lists with animated typography. Motion is present, but no transformation occurs.

A better promotional structure gives each example a complete loop:

context → friction → product action → visible result → consequence

The Chrome extension can help an agent inspect an authorized product journey across the homepage, feature pages, documentation, and the actual interface. The agent can build a coverage manifest and distinguish verified behavior from marketing claims.

The editing skill can then organize multiple visually distinct examples into a larger arc:

problem → transformation → proof → payoff → call to action

Narration is created as short breath-group assets first. Captions and picture timing are derived from the accepted speech sequence, rather than globally speeding up a voice to hit an arbitrary runtime.

This is especially useful for open-source projects. A README tells people what a project contains. A narrative video can show which part of someone's workflow actually changes.

Workflow 3: close the loop between editor development and visual QA

The Chrome integration is also useful while building Timeline Studio itself.

With Developer Mode, Codex can inspect:

DOM and applied styles after timeline interactions;
WebGPU or ONNX Runtime initialization failures;
worker messages, model requests, and caching behavior;
runtime errors in the console;
responsive inspector layouts;
performance problems during preview or export.

At the same time, Codex can work with the local repository and terminal. That creates a tight loop:

observe the rendered problem
→ inspect runtime evidence
→ locate the implementation
→ make a scoped change
→ reopen and verify the UI

For a browser application that combines a multi-track timeline, media decoding, WebGPU inference, workers, and offline rendering, that shared loop is much more useful than debugging from a screenshot alone.

Two execution paths instead of pretending every click is deterministic

Timeline Studio uses two automation paths.

The versioned command runner directly inspects and modifies portable .timeline archives. It supports semantic diffs and transactional operations for the commands it registers.

npm run agent -- project.inspect /projects/demo.timeline
npm run agent -- project.diff /projects/edit-plan.json
npm run agent -- project.run /projects/edit-plan.json

The browser path handles AI generation, visual preview, rich rendering, effects, and operations that are not yet available through the command contract.

The distinction is deliberate: deterministic project operations belong in the command layer; visual or UI-only operations belong in the browser layer and require visible verification.

Why the editable project matters

If an AI video tool returns only an MP4, every correction becomes a regeneration problem.

Timeline Studio treats .timeline as a portable project archive containing project structure and media. After the agent finishes, a human can still:

adjust a cut;
replace one voice segment;
move a caption;
change music;
revise a transition;
replace media and export again.

AI should remove repetitive setup work without removing authorship.

Powerful browser access needs narrow boundaries

Browser-aware agents also create real security and rights-management questions. A practical workflow should:

grant site access per task or domain rather than allowing every site by default;
enter credentials only in the browser, never in chat;
treat page content as untrusted input;
use web footage only through publisher- or platform-provided downloads with suitable item-level rights;
preserve watermarks and provenance;
require human confirmation for publishing, paid generation, permission changes, or deletion;
keep media processing local when the capability supports it.

The browser is a powerful perception and action surface. It should not become an invisible permission bypass.

Try it

GitHub: MartinDelophy/ai-video-editor
Live editor: video-editor.ai-creator.top
Skill directory: skills.sh/MartinDelophy/ai-video-editor

Install the skill through skills.sh:

npx skills add MartinDelophy/ai-video-editor --skill edit-timeline-studio

Or install it for Codex with GitHub CLI:

gh skill install MartinDelophy/ai-video-editor edit-timeline-studio \
  --agent codex --scope user

The next step is not a bigger Generate button

The next generation of AI editing will not be defined only by faster models or longer generated clips.

It will also depend on whether an agent can enter the creator's real information environment, understand why material matters, make explainable editorial decisions, validate the result, and leave those decisions on a timeline that a human still controls.

Codex for Chrome gives the agent browser context. The Timeline Studio skill provides an editing discipline. Timeline Studio turns those decisions into a portable project.

That combination is the interesting part: not AI replacing the timeline, but AI finally learning how to work inside one.

If you are interested in browser media, WebCodecs, WebGPU, ONNX Runtime, agent skills, or editable AI workflows, I would love your feedback and contributions on GitHub.

References

Building a Safe Video-Editing Agent for DeepSeek Harness

MartinDelophy — Sat, 15 Aug 2026 09:34:03 +0000

Large language models can explain how a video should be edited. But safely editing a real project is a different problem.

If you ask an agent to “convert this project to 9:16, keep the captions, save a copy, and render an MP4,” the hard part is not understanding the sentence. The hard part is making sure the agent:

selects the correct project, track, and clips;
does not overwrite the original;
does not execute the same operation twice after a retry;
does not write from a stale project revision;
cannot read or write outside the approved workspace;
and verifies that the rendered video actually exists and is valid.

To explore this problem, I built and open-sourced dsh-timeline-studio-plugin, a community plugin that connects DeepSeek Harness to Timeline Studio’s deterministic .timeline command layer.

Plugin: MartinDelophy/dsh-timeline-studio-plugin
Editor: MartinDelophy/ai-video-editor

This is a community integration for Timeline Studio. It is not part of DeepSeek Harness core and is not an official DeepSeek project.

What the plugin does

The plugin is not another video editor UI. It is an automation bridge between an agent and the editor.

Timeline Studio remains responsible for the visual timeline, media processing, browser-local AI features, preview, and the final creative experience. dsh-timeline-studio-plugin exposes a small, deterministic tool surface that DeepSeek Harness can call.

The workflow looks like this:

Natural-language request
  ↓
DeepSeek Harness
  ↓
Read-only project inspection
  ↓
Structured edit plan
  ↓
Semantic diff
  ↓
Transactional apply to a new .timeline project
  ↓
Render and validate MP4

The model decides what should happen. The command layer decides whether that operation is allowed and valid for the current project state.

What the user sees

The user first adds a local workspace containing a .timeline project and its media assets.

A safe first prompt is deliberately read-only:

Inspect the Timeline Studio project in this workspace. Tell me its duration, aspect ratio, tracks, and media. Do not modify any files yet.

After reviewing the result, the user can continue with an editing request:

Change the project to 9:16. Show me the planned diff first. If it is valid, save it as a new project without overwriting the original, then render an MP4.

The plugin has no separate visual panel. Its work appears in Harness tool calls and in the resulting .timeline and MP4 files. Rich visual inspection and manual refinement still happen in Timeline Studio.

Seven tools, three responsibilities

The first version exposes seven model tools:

Tool	Responsibility
`timeline_studio_project_inspect`	Inspect revision, duration, ratio, tracks, media inventory, and warnings
`timeline_studio_track_inspect`	List clips on a track in timeline order
`timeline_studio_clip_inspect`	Inspect source mapping, timing, transforms, and relationships
`timeline_studio_transcript_inspect`	Inspect captions, word timing, speakers, and audio links
`timeline_studio_project_diff`	Validate an edit plan against the real command registry without writing
`timeline_studio_project_apply`	Apply a validated plan transactionally
`timeline_studio_project_render`	Render and validate an H.264/AAC MP4

Why the agent does not rewrite project JSON directly

The shortest implementation would be to send the entire project JSON to the model and ask it to return a modified copy.

That is also the least reliable approach.

Syntactically valid JSON can still be semantically invalid for the editor. The model might reference a deleted clip, place media on the wrong track, invent an unsupported property, or repeat an operation that was already committed.

Instead, the agent produces a structured edit plan:

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 0,
  "dryRun": false,
  "operations": [
    {
      "id": "set-ratio-001",
      "type": "project.set_ratio",
      "ratio": "9:16"
    }
  ],
  "output": {
    "project": "/projects/output.timeline"
  }
}

The plan is checked against Timeline Studio’s real command registry before it is allowed to write anything.

Diff before apply

Every edit goes through two distinct stages:

timeline_studio_project_diff
  ↓ only after a successful diff
timeline_studio_project_apply

The diff stage does not write the output project. It checks questions such as:

Does the command exist?
Are the arguments valid?
Does the target clip still exist?
Does baseRevision match the current project?
Has the operation ID already been used?
Do all input and output paths stay inside the approved roots?

A wrong chat response can be regenerated. A wrong local write can damage a user’s project. That is why the preview step is a required safety gate rather than an optional UX enhancement.

Making retries safe with revisions and operation IDs

Agent tool calls can be retried because of cancellations, process failures, network interruptions, or replanning.

The plugin uses two mechanisms to make those retries predictable.

Project revision

Inspection returns the current revision. The edit plan must include it as baseRevision. If the project has changed since inspection, a new write based on the stale revision is rejected.

Idempotent operation ID

Every operation has a stable ID. Resubmitting an ID that has already been applied becomes a no-op instead of adding the same clip or modification twice.

Together, these checks provide optimistic concurrency control and idempotent execution for agent-driven edits.

File access boundaries belong in code, not prompts

Video editing requires access to project files, source media, and render outputs. A system prompt saying “do not leave the workspace” is not a security boundary.

The plugin requires explicit allowedRoots configuration:

- name: 'dsh-timeline-studio-plugin'
  config:
    timelineStudioRoot: /absolute/path/web_player
    allowedRoots:
      - /absolute/path/projects

Projects, plans, imported media, and output files must all resolve inside those roots. The implementation also blocks symbolic-link escapes.

This is a general rule I find useful for agent tooling: if a restriction can be enforced deterministically, enforce it in code instead of asking the model to remember it.

Installation

The currently verified environment is:

DeepSeek Harness 0.1.0-rc.6 Developer Preview
Node.js 22.20+ or 24+
a local Timeline Studio repository with dependencies installed
FFmpeg and ffprobe

Install the GitHub repository as a DSH Web-profile bundle:

dsh plugin --profile web add \
  "github:MartinDelophy/dsh-timeline-studio-plugin#main"

Then start Harness with absolute paths for Timeline Studio and the project workspace:

TIMELINE_STUDIO_ROOT=/absolute/path/web_player \
TIMELINE_PROJECTS_ROOT=/absolute/path/projects \
dsh --profile web

The bundle stays disabled when TIMELINE_STUDIO_ROOT is missing, so an incomplete installation does not break an existing Harness profile.

After restarting, open Settings → Plugins → Plugin list and search for timeline.

When the configuration is enabled and the Cordis status is mounted, the plugin is ready. Users do not call the mount identifier manually; they simply describe the editing task in a Harness conversation.

What was tested end to end

Registering tools is not enough. The project also exercises the real DeepSeek Harness and Cordis pipeline.

The verified path covers:

DSH bundle installation;
automatic Cordis mounting;
registration of all seven tools;
inspection of a real .timeline project;
confirmation that diff performs no project write;
revision and idempotency checks during apply;
allowedRoots and symbolic-link boundary enforcement;
cancellation propagation to the Timeline Studio subprocess;
reinspection of the generated project;
MP4 rendering and output validation.

The local checks are:

npm run check
TIMELINE_STUDIO_ROOT=/absolute/path/web_player npm run test:e2e

DeepSeek Harness is still a Developer Preview, so breaking changes are possible. The plugin intentionally keeps the Harness adapter thin and leaves editing behavior in Timeline Studio. If the host interface changes, the integration layer can evolve without rewriting the editor’s command engine.

Why the editable project matters

Many AI video systems return only a final render. That works for one-shot generation, but it makes small follow-up changes expensive:

extend a shot by half a second;
correct one caption;
lower the music;
create a vertical variant;
or reuse the same project for another campaign.

This workflow returns both the video and an editable .timeline project. The agent can handle repetitive, structured, verifiable work, while the creator keeps control over the final cut.

Closing thoughts

The interesting part of an agent that edits video is not just whether the model understands editing vocabulary. The real engineering questions are:

What can the agent access?
Which project revision is it acting on?
Was the operation already executed?
Can the edit be previewed before writing?
Can the result be validated afterward?
Can a human continue editing the result?

dsh-timeline-studio-plugin is an open-source attempt to answer those questions with a small deterministic tool layer around a real editor.

Issues, feedback, and contributions are welcome.

How Four Color Wheels Work in Video Editing—and How I Built Them in the Browser

MartinDelophy — Wed, 12 Aug 2026 08:11:53 +0000

I built this in an open-source browser video editor

I recently added a complete desktop color-wheels workflow to Timeline Studio, an open-source, local-first video editor that runs in the browser.

GitHub: https://github.com/MartinDelophy/ai-video-editor
Live editor: https://video-editor.ai-creator.top

The implementation includes:

separate wheels for shadows, midtones, highlights, and global offset;
temperature, tint, and global saturation controls;
hue, saturation, and luminance controls for every wheel;
independent keyframes for all 15 grading properties;
shortest-path hue interpolation;
the same animated grade in preview, transitions, and final export;
a desktop-focused interface, while the mobile web editor keeps its simpler speed workflow.

If you have ever opened a professional video editor and wondered why it displays four colorful circles, this article explains what they do. The second half looks at the engineering problems behind implementing them in a browser.

What is a color wheel?

A color wheel is a two-dimensional controller for expressing both a color direction and an adjustment strength.

Moving the control point toward red adds a red bias.
Moving it toward blue adds a blue bias.
Moving farther from the center increases the strength.
Leaving it in the center adds no directional color bias.

This is often more intuitive than controlling red, green, and blue with three unrelated sliders. If an image feels too cold, you can move toward orange. If it feels too warm, you can move toward blue.

In a useful grading tool, however, the wheel does not simply add the same color to every pixel. A frame contains dark, medium, and bright regions, and they usually need different treatment. That is why professional editors provide multiple wheels.

Why are there four wheels?

One frame may contain almost-black hair, naturally exposed skin, a white shirt, and a bright window at the same time.

With only one global color control, making the dark areas cooler would also make the skin and window cooler. Color-grading tools therefore divide their influence primarily by luminance range.

Wheel	Main area of influence
Shadows	Dark clothing, hair, night backgrounds, and unlit areas
Midtones	Skin, products, walls, and most normally exposed subjects
Highlights	Windows, sky, lamps, reflections, and bright surfaces
Offset	The overall color balance of the frame

Shadows

Moving the shadows slightly toward blue or cyan can make a shot feel colder and deeper. This is one ingredient commonly associated with a cinematic look.

It is also easy to overdo. Excessive adjustment can turn black areas visibly blue, damage natural hair color, and make low-light regions look dirty or posterized. Subtle adjustments tend to work better than large, obvious movements.

Midtones

Midtones often contain the most important part of the image: a person's skin or the main product.

If a face looks pale, moving the midtones slightly toward a warm color may help. Pushing too far can make skin yellow or red and contaminate neutral clothing and backgrounds.

A safer order of operations is:

Correct exposure.
Correct white balance.
Check whether skin and neutral objects look natural.
Use the midtone wheel for a small creative adjustment.

A color wheel is excellent for refinement, but it should not hide a fundamentally incorrect exposure or white balance.

Highlights

Highlights affect bright areas such as sky, windows, lamps, reflections, white clothing, and the lit side of a face.

Moving highlights toward yellow or orange can strengthen the feeling of sunlight or sunset. Moving them toward blue can create a colder night or technology-oriented look.

A frequent creative choice is to keep shadows slightly cool and highlights slightly warm. This creates color separation and additional depth. It does not mean that every video should use an aggressive teal-and-orange preset. Interviews, food, products, and natural scenes all have different requirements for color accuracy.

Offset

Offset affects the whole frame rather than a single luminance range.

It is useful when:

the entire clip has a green cast;
a shot is globally too cold or too warm;
several cameras need a common starting balance;
you want to establish a gentle global direction before regional adjustments.

If only the shadows are green while the highlights are correct, a global offset is the wrong tool. The important question is not only which color should change? but also which luminance range should change?

What is the curved control beside the wheel?

Many color-wheel interfaces place a curved control next to the circle. It usually controls luminance for that range.

A complete wheel therefore represents three important values:

Hue: which color direction to use;
Saturation: how strong that color direction should be;
Luminance: whether the affected range should become brighter or darker.

The disc handles hue and saturation, while the curved control changes luminance. Together they provide much more useful control than a decorative color picker.

Color wheels are not the same as filters

A filter is usually a predefined group of adjustments. It is excellent for reaching a recognizable style quickly.

A color wheel is a manual correction and grading tool.

Filter	Color wheel
Produces a quick result	Provides detailed control
Hides many values in a preset	Exposes luminance-specific decisions
Can impose the same style on every source	Can preserve the character of each source
Works well for rapid exploration	Works well for correction and refinement

They can also work together. A user can apply a filter as a starting point and then repair skin, shadows, or highlights with the wheels.

Video color needs keyframes

A still image can use one set of parameters. Video changes over time.

A subject may walk from indoors to sunlight. Stage lighting may change from blue to red. A memory sequence may gradually lose saturation. A sunset shot may become warmer as it progresses.

Keyframes allow a clip to store different grading states at different times:

0s: neutral color;
2s: midtones begin moving warmer;
4s: highlights gain a small yellow bias;
6s: global saturation decreases.

The editor interpolates between these states. Color grading stops being a fixed filter and becomes an animation that follows the content.

In Timeline Studio, the keyframeable properties are:

temperature, tint, and global saturation;
hue, saturation, and luminance for shadows;
hue, saturation, and luminance for midtones;
hue, saturation, and luminance for highlights;
hue, saturation, and luminance for offset.

That produces 15 independently keyframeable properties.

Ordinary interpolation breaks hue

Hue is circular, not linear.

For example, 350° and 10° are only 20 degrees apart on a color wheel. Ordinary numeric interpolation may travel from 350° through 180° to 10°, passing through many unrelated colors.

The desired transition crosses zero:

350° -> 0° -> 10°

A shortest-path interpolation can be implemented like this:

function interpolateHue(currentHue, nextHue, progress) {
  const delta = ((nextHue - currentHue + 540) % 360) - 180;
  return (currentHue + delta * progress + 360) % 360;
}

progress ranges from 0 to 1. The normalized delta stays between -180 and 180, so the transition follows the shorter direction around the wheel.

This is a small mathematical detail with a very visible result. Without it, animated color may unexpectedly cycle through green, cyan, or purple between two nearby red hues.

A practical data model

A simplified base grade can be represented as follows:

const colorGrade = {
  temperature: 0,
  tint: 0,
  saturation: 0,
  shadows:    { hue: 0, saturation: 0, luminance: 0 },
  midtones:   { hue: 0, saturation: 0, luminance: 0 },
  highlights: { hue: 0, saturation: 0, luminance: 0 },
  offset:     { hue: 0, saturation: 0, luminance: 0 },
};

Each animated property can use a path such as:

colorGrade.temperature
colorGrade.shadows.hue
colorGrade.highlights.luminance

At render time, the resolver needs to:

Find an exact keyframe at the current time if one exists.
Otherwise find the nearest previous and next values for the property.
Interpolate ordinary numeric properties linearly.
Interpolate hue using the shortest circular path.
Use the base grade before the first keyframe or when a property has no keyframes.
Normalize and clamp the resolved values.

Property-level keyframes are important. A user may want to animate highlight luminance without freezing every other color property into the same keyframe object.

Preview and export must use the same resolver

Drawing attractive wheels is only the UI portion of the feature. A video editor is not correct unless the final export matches the editor preview.

If preview and export use separate color logic, users may encounter:

different colors after export;
color jumps during transitions;
keyframes that work in the editor but disappear from the final video;
different results for images and video clips.

Timeline Studio resolves the grade from the clip's keyframes and local time, then sends that result through the preview, transition, and offline export paths.

The next clip in a transition must resolve its own local grading state as well. Otherwise the outgoing clip may be correct while the incoming clip temporarily displays a static or incorrect grade.

This shared resolver was one of the most important parts of the implementation. The wheels are not decorative UI; they are editable parameters that reach the final rendered artifact.

Filter thumbnails should use the user's source

Another small but important detail is filter preview imagery.

A filter grid should not replace the user's image with an unrelated stock thumbnail. For an image clip, every filter card can reuse the selected image. For a video clip, the editor can reuse an extracted frame from the selected clip.

Only the preview filter should change. The source image must remain the same across cards, so users can compare color treatments rather than compare different subjects.

Why the full UI is desktop-only

Four wheels, basic controls, value readouts, reset actions, and 15 keyframe buttons form a dense interface.

Desktop users have enough space and a precise pointer. Copying the same layout directly to mobile web would create tiny wheels, accidental touches, an extremely long inspector, and competition with the timeline for screen space.

For this release, Timeline Studio exposes the complete color-wheels workflow on desktop. Mobile web does not show the wheels and keeps the more focused video-speed workflow.

Feature consistency does not always require identical UI on every device. The interface should reflect the precision and space available on each platform.

A simple grading order for beginners

If you are new to color wheels, this order is a useful starting point:

Disable creative filters and inspect the original source.
Correct temperature and tint.
Check exposure and preserve useful shadow and highlight detail.
Inspect midtones, especially skin or the main product.
Add subtle separation to shadows and highlights.
Use offset only when the whole frame needs a global correction.
Toggle the grade on and off frequently because human vision adapts quickly.
Add keyframes when lighting changes over time instead of forcing one static grade onto the whole clip.

Closing thoughts

The four wheels can be remembered simply:

Shadows: dark regions;
Midtones: subjects and skin;
Highlights: bright regions and the character of light;
Offset: the entire frame.

Good grading is rarely about pushing every control to a dramatic value. It usually comes from several small, intentional adjustments that preserve the identity of the footage while supporting its mood.

The complete implementation is open source. You can inspect the UI, property keyframes, shortest-path hue interpolation, preview composition, transition handling, and export integration in the repository:

GitHub: https://github.com/MartinDelophy/ai-video-editor
Live editor: https://video-editor.ai-creator.top

If browser-based video editing, React, WebCodecs, local AI, or deterministic media export interests you, issues and pull requests are welcome. If the project is useful, a GitHub star also helps more developers discover it.

Building a Browser-Native AI Video Editor: Timeline Synchronization, WebGPU, and Worker Architecture

MartinDelophy — Tue, 11 Aug 2026 08:41:24 +0000

Modern browsers can now do work that once required a native desktop application. WebGPU exposes GPU compute, WebAssembly brings mature runtimes to the web, WebCodecs provides lower-level media primitives, and Web Workers let us move expensive tasks away from the UI thread.

But getting an AI model to run once in a browser is very different from building a video editor that remains predictable during a long editing session.

I recently completed a substantial infrastructure update to Timeline Studio, a local-first browser AI video editor:

GitHub: MartinDelophy/ai-video-editor
Release: v1.0.2
Live demo: https://video-editor.ai-creator.top

The release changed 37 files and added roughly 1,400 lines. The work focused on:

stable dragging, splitting, reordering, and cross-track movement;
synchronization between project time and media time;
reusable timeline frame data;
explicit high-performance WebGPU adapter selection;
persistent AI workers and reusable inference sessions;
centralized model caching through a service worker.

This article explains the engineering decisions behind that update.

1. A timeline is a constrained data model, not a row of draggable rectangles

A first timeline prototype can position clips with a simple pixel conversion:

const left = startTime * pixelsPerSecond;
const width = duration * pixelsPerSecond;

Dragging appears equally straightforward:

const nextStartTime =
  originalStartTime + deltaX / pixelsPerSecond;

A production clip, however, contains more than a visual position:

const segment = {
  id: "segment-001",
  trackId: "visual-track",
  startTime: 30,
  duration: 10,
  trimStart: 10,
  trimEnd: 20,
  sourceDuration: 60,
  playbackRate: 1,
  locked: false
};

A single move may need to enforce all of these rules:

the clip cannot begin before zero;
it cannot exceed the project boundary;
its position may snap to other edit points;
a locked track cannot accept it;
it may not overlap another clip on the same track;
its media type must be compatible with the target track;
related caption and audio state must remain valid.

When each React component implements its own version of those rules, dragging, duplication, cutting, and reordering eventually disagree.

The update moves timeline decisions into a shared domain layer:

pointer coordinates
       ↓
timeline coordinates
       ↓
candidate start time
       ↓
bounds and snapping constraints
       ↓
track routing and collision checks
       ↓
atomic project-state update

The UI collects input and renders a preview. Domain logic determines the final valid result.

2. New clips must not rearrange existing work

A common way to handle overlapping audio is to redistribute every clip until no overlap remains. The algorithm succeeds, but the user's earlier track arrangement changes unexpectedly.

The new rule is deliberately asymmetric:

A new clip may search for an available track, but existing clips keep their lanes.

function findTrackForNewAudio(newClip, tracks) {
  for (const track of tracks) {
    if (!hasOverlap(track.clips, newClip)) {
      return track.id;
    }
  }

  return createAudioTrack();
}

Asset semantics also matter. AI-generated music belongs on the dedicated music track rather than whichever voice track happens to be empty:

function resolveAudioTrack(asset) {
  if (asset.type === "ai-music") {
    return MUSIC_TRACK_ID;
  }

  return findAvailableVoiceTrack(asset);
}

A track is not merely a visual group. It may define mixing behavior, caption relationships, mute and solo behavior, volume defaults, and export rules.

Automatic routing should therefore preserve both user intent and media meaning.

3. Separate project time from media time

A video editor has at least two clocks:

Project time describes where a clip appears in the final composition.
Media time describes which position in the source file should be decoded.

Suppose seconds 10–20 of a source video are placed at seconds 30–40 of the project. At project time 33, the video element should display source time 13.

function getMediaTimeAtTimelineTime(segment, timelineTime) {
  const localTime = timelineTime - segment.startTime;
  const mediaTime =
    segment.trimStart +
    localTime * segment.playbackRate;

  return Math.max(
    segment.trimStart,
    Math.min(segment.trimEnd, mediaTime)
  );
}

Keeping this transformation independent from the DOM makes it reusable for trimming, splitting, playback-rate changes, clip movement, and multiple clips referencing the same source asset.

4. Do not write video.currentTime on every frame

The obvious synchronization operation is:

video.currentTime = targetTime;

Doing this continuously forces the browser to seek repeatedly. That can cause decoder churn, visual jitter, black frames, and unnecessary CPU use.

Instead, compare the current media position with the calculated target:

const drift = Math.abs(video.currentTime - targetTime);

if (isSeeking || drift > MAX_ALLOWED_DRIFT) {
  video.currentTime = targetTime;
}

During normal playback, the media element advances on its own and small drift is tolerated. During timeline scrubbing, exact frame feedback matters more, so synchronization happens immediately.

Both paths are called “synchronization,” but they optimize for different outcomes:

playback optimizes for continuity;
scrubbing optimizes for precision.

5. Make timeline frames part of the media asset

Thumbnail frames are not just decoration. They help users identify content and find cut points.

If those frames live only in temporary component state, they are easy to lose after splitting, copying, or moving a clip. A generated video may even appear as a blank block or one stretched cover frame.

The asset now carries compact sampled frames:

const videoAsset = {
  id: "video-001",
  duration: 12,
  trackFrameDuration: 0.5,
  trackFrames: [
    { time: 0, image: "..." },
    { time: 0.5, image: "..." },
    { time: 1.0, image: "..." }
  ]
};

The same data can be reused by:

the media-library card;
the main visuals track;
overlay tracks;
multiple clips created from a split;
browser-generated video assets.

Rendering code selects frames based on the clip's trim range and visual width without rewriting the source frame data.

6. WebGPU does not necessarily choose the fastest GPU

A typical WebGPU setup begins with:

const adapter =
  await navigator.gpu.requestAdapter();

On a dual-GPU machine, the browser may prefer an integrated adapter to save power. That can be reasonable for ordinary UI rendering but expensive for ONNX Runtime and generative workloads.

For compute-heavy paths, the project now uses an explicit default:

const adapter =
  await navigator.gpu.requestAdapter({
    powerPreference: "high-performance"
  });

An explicit caller override must still win:

function normalizeAdapterOptions(options = {}) {
  return {
    ...options,
    powerPreference:
      options.powerPreference ?? "high-performance"
  };
}

This provides a high-performance default without breaking callers that intentionally request low-power.

The policy is shared across AI music, speech, face processing, video repair, and super-resolution workers.

7. Handling requestAdapter calls inside third-party runtimes

Not every adapter request is made by application code. A pinned runtime may internally call requestAdapter() without options.

When upgrading the dependency immediately would introduce compatibility risk, a narrowly scoped initialization wrapper can provide the missing default:

const originalRequestAdapter =
  navigator.gpu.requestAdapter.bind(navigator.gpu);

navigator.gpu.requestAdapter = (options = {}) =>
  originalRequestAdapter({
    powerPreference: "high-performance",
    ...options
  });

Two details are essential:

Keep the patch scoped to the relevant worker or initialization phase.
Spread explicit options after the default so the caller retains control.

The original method should be restored after initialization. This is a compatibility strategy, not a preferred permanent API.

8. AI latency is more than inference time

The delay users experience usually contains several stages:

download artifacts
       ↓
write cache
       ↓
read model files
       ↓
create inference sessions
       ↓
preprocess inputs
       ↓
run inference
       ↓
postprocess outputs

Optimizing only model execution may leave most of the perceived delay untouched.

Download independent artifacts in parallel

const artifacts = await Promise.all(
  modelFiles.map(downloadModelFile)
);

Create large GPU sessions serially

Initializing several large WebGPU sessions at once can create a sharp memory and GPU-resource peak. Parallel downloading combined with serial session creation is usually a safer balance.

Keep initialized workers alive

Terminating a worker after every generation discards the expensive model sessions:

first use
   ↓
start worker
   ↓
initialize models
   ↓
generation 1
   ↓
generation 2
   ↓
release when the page closes

The UI should also distinguish model setup from content generation. A repeated generation should not look like another model download.

9. Give one component ownership of persistent model caching

If the page, inference workers, and service worker all write to Cache Storage, a large model can be stored more than once.

Timeline Studio makes the shared service worker the only persistent cache writer:

inference worker
       ↓ model request
service worker
       ├─ normalize provider URLs
       ├─ resolve immutable model identity
       ├─ preflight storage capacity
       ├─ evict stale model families
       └─ write Cache Storage

Hugging Face and ModelScope may use different URLs for equivalent artifacts. Those URLs are normalized to one cache identity based on the model, immutable revision, and file path.

A cache write failure also does not necessarily mean the current inference must fail. If the artifact is already available in memory, the task can continue; only the next session may need to download it again.

That distinction separates a performance degradation from a functional failure.

10. Never expose “Failed to fetch” as the complete error

A browser's generic Failed to fetch message is nearly useless to an end user.

The application layer should distinguish at least:

no network connection;
an unavailable model mirror;
insufficient browser storage;
unavailable WebGPU support;
model initialization failure;
user cancellation.

function toUserFacingError(error) {
  if (isNetworkError(error)) {
    return "The model server is unavailable. Check your connection and retry.";
  }

  if (isStorageError(error)) {
    return "Browser storage is full. This run will try an in-memory fallback.";
  }

  if (isAbortError(error)) {
    return "The operation was cancelled.";
  }

  return "The model could not be initialized.";
}

A useful error should answer three questions:

What failed?
Can the current task continue?
What can the user do next?

Resulting architecture

The refactored flow is easier to reason about:

user interaction
       ↓
timeline commands and constraints
       ↓
project state
       ├─ segment placement
       ├─ track routing
       ├─ caption relationships
       └─ sampled track frames
                ↓
media synchronization
       ├─ project time
       ├─ source-media time
       └─ drift correction
                ↓
AI workers
       ├─ model lifecycle
       ├─ WebGPU adapter policy
       └─ cancellation
                ↓
centralized model cache

The most important outcome is not the number of new features. It is the clearer separation of responsibilities:

timeline components no longer invent their own placement rules;
media-time conversion is independent from UI state;
WebGPU, worker lifecycle, and model caching form shared infrastructure.

Validation

The release was checked with:

npm run check

This runs ESLint, TypeScript checking, and the Vite production build. The release completed with:

zero ESLint errors;
successful TypeScript validation;
successful production build;
GitHub release v1.0.2;
successful production deployment.

Final takeaways

The difficult part of a browser-native AI video editor is not running a single model. It is keeping timeline state, media elements, React state, workers, GPU sessions, and local caches consistent throughout a real editing session.

The most reusable lessons from this update are:

Treat the timeline as a constrained domain model.
Do not let new assets silently rearrange existing work.
Separate project time from source-media time.
Use different synchronization policies for playback and scrubbing.
Explicitly request a high-performance WebGPU adapter for compute workloads.
Download artifacts in parallel but initialize large GPU sessions serially.
Reuse initialized workers and inference sessions.
Give the service worker exclusive ownership of persistent model caching.

Browsers are becoming local compute platforms, not just document viewers. Turning those capabilities into a dependable creative tool requires careful boundaries around time, state, storage, and GPU resources.

Project links

Running Chinese TTS Fully in the Browser: Migrating from Piper to Kokoro 1.1 FP16

MartinDelophy — Sat, 08 Aug 2026 09:22:18 +0000

When an AI video editor needs text-to-speech, the easiest solution is usually a hosted API: send the script to a server, wait a few seconds, and download the audio.

For Timeline Studio, we deliberately took a harder route. We want voice generation to run on the user's device, inside the browser, alongside the editable timeline.

In Timeline Studio v1.0.0, we replaced our previous Chinese Piper/VITS ONNX voices with a browser-ready FP16 build of Kokoro multi-lang v1.1. The new path provides two female and two male voices, handles Chinese text with inline English, and still performs synthesis locally through sherpa-onnx WASM.

GitHub: MartinDelophy/ai-video-editor
Live editor: video-editor.ai-creator.top
Hugging Face Space: haixin/timeline-studio

If browser AI, WebAssembly, ONNX, or open-source video editing interests you, a GitHub star is greatly appreciated.

Why move away from Piper for Chinese?

Piper was a pragmatic starting point. Its ONNX models are relatively compact, the browser deployment path is well understood, and it allowed us to build a local TTS workflow without depending on a metered cloud service.

But the quality bar changes when TTS becomes part of a video editor.

It is no longer enough for a model to simply pronounce a sentence. Product demos, tutorials, explainers, and narrative videos need more natural pacing, clearer voice choices, and reliable handling of modern Chinese copy that often contains English names and technical terms.

Consider this sentence:

使用 Timeline Studio，让 AI video editing 直接在浏览器里完成。

Splitting it into separate Chinese and English clips creates avoidable problems: speaker identity can change, pacing can jump, and the sentence loses its natural context. We wanted one speaker to generate the complete utterance.

Kokoro multi-lang v1.1 gave us a better foundation for that experience.

Piper has not disappeared from Timeline Studio. It still powers browser voices for German, Spanish, French, Italian, and Brazilian Portuguese. This migration only replaces the Chinese Piper route.

A browser model is more than an ONNX file

Choosing a model was the easy part. Turning it into a dependable browser feature required much more work.

A server can assume a controlled filesystem, abundant memory, a long-running process, and predictable model storage. A browser must deal with:

first-use download size;
memory and bandwidth pressure;
main-thread responsiveness;
storage quotas;
interrupted or incomplete downloads;
regional model availability;
immutable model versions;
cache migration across application releases.

A model that produces one successful sample on a developer machine is a demo. A product feature must download, verify, initialize, cache, reuse, upgrade, and fail clearly.

Converting Kokoro 1.1 to FP16

We converted the selected Kokoro multi-lang v1.1 bundle to FP16 and packaged it for the sherpa-onnx WASM TTS runtime.

The resulting browser pipeline looks like this:

User script
    ↓
Chinese/English text normalization
    ↓
Voice ID → speaker ID
    ↓
Kokoro multi-lang v1.1 FP16
    ↓
sherpa-onnx WASM in a Web Worker
    ↓
Float32 PCM samples
    ↓
WAV encoding in the browser
    ↓
Timeline Studio asset library

FP16 reduces storage, transfer, and runtime memory-bandwidth pressure relative to FP32 while preserving the voice quality we need.

It is not magic. The first run still requires a substantial model download, and synthesis speed depends on the device, browser, available memory, and script length. The goal was not to pretend the cost had disappeared, but to make a higher-quality multilingual TTS model practical in a local browser workflow.

Keeping inference off the main thread

Model initialization and synthesis are compute-heavy operations. Running them on the main thread would make an editor feel frozen.

Timeline Studio creates a dedicated worker:

worker = new Worker("/kokoro-multilang.worker.js");

worker.postMessage({
  type: "init",
  baseUrls,
});

The UI thread handles the script, selected voice, progress display, and timeline state. The worker downloads and verifies the runtime bundle, initializes sherpa-onnx WASM, creates the offline TTS session, and generates the samples.

A synthesis request is intentionally small:

worker.postMessage({
  type: "generate",
  requestId,
  text,
  sid,
  speed,
});

The generated sample buffer is returned as a transferable object:

self.postMessage(
  {
    type: "result",
    requestId: message.requestId,
    samples: audio.samples,
    sampleRate: audio.sampleRate || tts.sampleRate,
  },
  [audio.samples.buffer],
);

Transferring the underlying ArrayBuffer avoids copying a potentially large block of PCM data.

The worker stays alive after initialization, so repeated generations in the same editing session reuse the warm runtime instead of downloading and initializing the model again.

Four voices, one shared model

We selected four speakers with distinct roles:

Timeline Studio voice	Upstream speaker	Character
Qinglan / 晴岚	`zf_001`	Natural, clear female voice
Ruoxi / 若溪	`zf_073`	Softer female voice
Yunzhou / 云舟	`zm_009`	Steady, natural male voice
Jingche / 景澈	`zm_010`	Younger, brighter male voice

Internally, product voice IDs map to the four speaker slots:

const SPEAKER_IDS = Object.freeze({
  zh_f_qinglan: 0,
  zh_f_ruoxi: 1,
  zh_m_yunzhou: 2,
  zh_m_jingche: 3,
});

All four voices share the same FP16 model. Switching speakers does not trigger another full model download.

We also provide a real preview generated by the matching speaker for every selectable voice. A voice card should never play a placeholder sample from a different speaker.

Keeping mixed Chinese and English in one utterance

Before synthesis, Timeline Studio normalizes the text while preserving Han characters, Latin characters, numbers, whitespace, and common punctuation:

const text = normalized
  .replace(
    /[^\p{Script=Han}\p{Script=Latin}0-9\s，。！？；：、,.!?;:()\-]/gu,
    "",
  )
  .replace(/[ \t]+/g, " ")
  .replace(/ *\n+ */g, "。")
  .replace(/[。]{2,}/g, "。")
  .trim();

The important product rule is that mixed-language copy remains one linguistic utterance. We do not split a Chinese sentence simply because it contains "WebGPU", "API", or an English product name.

That preserves speaker identity, punctuation-driven pauses, and editing simplicity.

Parallel downloads with integrity checks

The browser bundle includes the WASM binary, JavaScript runtime, wrapper code, model data, and supporting resources. Large data is divided into parts described by a manifest.

The worker downloads manifest entries in parallel:

const entries = [
  ...manifest.runtime.files,
  ...manifest.runtime.data.parts,
];

const resources = await Promise.all(
  entries.map((entry) =>
    fetchAndVerify(baseUrl, entry, onChunk),
  ),
);

Each entry includes its expected byte length and SHA-256 digest. The worker verifies individual downloads, reassembles the model data in manifest order, and verifies the completed payload again before creating the TTS session.

A successful HTTP status is not enough. Proxies, partial caches, and interrupted connections can all return incomplete model data. Integrity verification prevents corrupted weights from reaching inference.

Hugging Face and ModelScope mirrors

Model availability is a product issue, especially for a browser application serving users in different regions.

We mirror the voice bundle in repositories we control on both Hugging Face and ModelScope, pinned to immutable provider revisions.

Timeline Studio prefers ModelScope for Chinese and domestic sessions and Hugging Face elsewhere. If the preferred route fails, it attempts the other mirror.

The two providers use different URLs and revisions for the same artifacts, so we canonicalize them to one internal cache identity. Otherwise, a browser could store two copies of the same large model after a network route changes.

The source may change; the model identity should not.

Cache migration matters

Timeline Studio also uses local models for captions, music, vision, voice conversion, and other features. Browser storage cannot be treated as unlimited.

Before loading the Kokoro bundle, the application preflights available storage. During upgrades, it:

removes legacy FP32 Kokoro files;
avoids duplicate Piper caches;
migrates unchanged files to the new canonical revision;
deletes changed manifest or model parts;
evicts stale voice families when capacity is tight.

This is not glamorous work, but it separates a one-release demo from a browser AI application that can continue evolving.

TTS is only the beginning of the workflow

Timeline Studio is not a standalone text-to-speech page.

A generated WAV becomes an editable media asset. The user can audition it, regenerate it, place it on the voiceover track, trim and move it, generate captions, mix it with source audio and music, and export the final project to MP4 or WebM.

The current voice routing is deliberately model-specific:

Language or use case	Browser model
Chinese and mixed Chinese/English	Kokoro multi-lang v1.1 FP16
English	Kokoro 82M ONNX
German, Spanish, French, Italian, Brazilian Portuguese	Piper/VITS ONNX

We prefer specialized, verified paths over claiming that one model is best for every language.

What this migration delivered

The final change included more than replacing two entries in a voice picker:

Kokoro multi-lang v1.1 replaced the Chinese Piper route;
the model was converted and packaged as FP16;
sherpa-onnx WASM runs synthesis locally;
a persistent Web Worker keeps the editor responsive;
two female and two male voices share one model;
mixed Chinese/English scripts remain one utterance;
model parts download in parallel and are SHA-256 verified;
Hugging Face and ModelScope provide pinned fallback mirrors;
both providers share one cache identity;
storage preflight and cache migration support future releases;
generated audio enters the editable video timeline instead of ending as a demo file.

Closing thoughts

Running AI in the browser is not simply a matter of moving an ONNX file to the frontend.

A production-quality local feature needs a model delivery system, an isolated runtime, integrity checks, regional routing, cache ownership, upgrade behavior, honest progress reporting, and a clear place in the user's workflow.

That engineering work is less visible than a model benchmark, but it is what turns local inference into a usable creative tool.

Timeline Studio is open source under the MIT License:

GitHub: https://github.com/MartinDelophy/ai-video-editor
Try it online: https://video-editor.ai-creator.top/

If this work is useful to you, please consider starring the repository, opening an issue, or contributing.

From Natural Language to an Editable Video: Inside Timeline Studio's Agent Skill

MartinDelophy — Fri, 07 Aug 2026 08:51:59 +0000

One instruction, one rendered video, and one editable .timeline project.

Repositories

Examples and reproducible workflows — start here

Timeline Studio Skills Handbook

The Handbook contains reference-video remakes, product promotions, multilingual localization, narrative marketing videos, and science explainers. Each case includes as much reproducible evidence as possible: the prompt, result video, editing decisions, and an editable .timeline project.

Main repository

MartinDelophy/ai-video-editor

The main repository contains the browser video editor, Agent Skill, timeline command protocol, local AI integrations, and build and deployment configuration.

Install the Skill with:

npx skills add MartinDelophy/ai-video-editor \
  --skill edit-timeline-studio

Why video production needs more than another generation API

Generative AI has made it easy to create text, images, speech, music, and captions independently. Producing one complete video is still a fragmented process.

A short video may require all of the following:

inspecting and organizing source media;
selecting useful shots;
designing the narrative and pacing;
generating or cleaning narration;
creating and synchronizing captions;
adding music, overlays, transitions, and effects;
exporting and decoding the result;
preserving a project that can still be edited later.

Most AI tools solve one item on this list. The creator still moves files between services and manually rebuilds timing relationships.

Timeline Studio's edit-timeline-studio Skill treats the entire process as an executable workflow. Its goal is not merely to produce an MP4. It also preserves an editable .timeline project that a human can reopen and refine.

1. Starting from a natural-language brief

The user describes the intended result rather than a sequence of UI operations:

Turn these product demo clips into a roughly 40-second vertical promo. Preserve the real product interaction, add English narration and captions, and deliver both the video and an editable project.

The Skill decomposes that request into a controlled sequence:

inspect the named assets;
analyze video, audio, speech, and text;
record editorial decisions;
build a declarative edit plan;
apply timeline operations;
run browser-local AI or rich editor features when required;
validate the project and rendered output.

Natural-language brief
        ↓
Media analysis → Editorial decisions → Edit plan
        ↓
Transactional timeline operations
        ↓
AI speech, captions, music, and effects
        ↓
Project validation → Video validation
        ↓
MP4/WebM + editable .timeline

The Skill is therefore more than a prompt template. It defines what the Agent must inspect, how it should execute the edit, and what evidence is required before the task can be called complete.

2. Analyze before cutting

A common automatic-editing failure is to start cutting before understanding the media.

Silence does not necessarily mean that a section is disposable. It may contain a product result, a useful reaction, intentional tension, or the preparation for an important action.

Depending on the task, Timeline Studio can inspect:

duration, resolution, frame rate, and audio presence;
representative frames;
speech transcripts and OCR text;
shot-change evidence;
subject position, scale, and motion;
global and subject-region optical flow;
audio-energy changes;
visual clarity.

The analysis is converted into a source-time decision record rather than immediately mutating the project:

{
  "assetId": "product-demo",
  "sourceStart": 12.8,
  "sourceEnd": 19.4,
  "decision": "shorten",
  "keepStart": 14.1,
  "keepEnd": 18.2,
  "reason": "Preserve the complete interaction and remove repeated setup",
  "confidence": 0.92,
  "protectSourceAudio": true
}

This makes keep, remove, shorten, and reorder decisions explainable and reviewable.

3. Turning the brief into declarative timeline commands

Once the plan is ready, the Agent uses stable project, track, asset, and clip identifiers wherever the command registry supports the requested operation.

npm run agent -- project.inspect /path/to/project.timeline
npm run agent -- project.diff /path/to/edit-plan.json
npm run agent -- project.run /path/to/edit-plan.json

project.inspect reads the project, tracks, clips, media, and captions.
project.diff validates the plan and calculates a non-writing semantic diff.
project.run transactionally applies the validated plan.

A simplified edit plan looks like this:

{
  "version": 1,
  "projectRevision": 8,
  "operationId": "update-product-intro-v2",
  "operations": [
    {
      "type": "timed.move",
      "clipId": "voice-intro",
      "start": 2.5
    },
    {
      "type": "caption.update",
      "clipId": "caption-intro",
      "text": "Let AI handle the repetitive editing work.",
      "start": 2.5,
      "end": 5.8
    }
  ]
}

4. Making Agent edits safe

Video editing needs many of the same guarantees as other stateful systems. A plan may update visuals, captions, narration, and music together. A partial result is often unusable.

The command layer therefore uses several safeguards:

Revision checks: reject a plan if the project changed after the plan was created.
Preconditions: verify that clips, tracks, media, and expected state still exist.
Transactions: apply one user-visible intent as a complete unit.
Idempotency: use a stable operationId so retrying a request does not duplicate media or captions.
Semantic diffs: show meaningful project changes before writing the new archive.

These mechanisms separate a reliable production Agent from a UI automation demo.

5. Two execution paths

Timeline Studio does not pretend that every editor feature is already a mature headless API. It uses two complementary paths.

Path	Typical work
Versioned command layer	Project inspection, media import, clip timing, captions, track operations, portable archive output
Browser editor	AI speech, automatic captions, advanced effects, digital humans, rich preview and full export

Stable operations run through the command registry. Features that have not yet moved into that registry remain available through the local or hosted browser editor.

After either path, the Skill reopens and verifies the project instead of assuming that a successful click or command means the edit is correct.

6. Why `.timeline` matters

Many AI video systems return only an MP4. That is a delivery file, not an editable source of truth.

Timeline Studio uses a portable .timeline archive:

project.timeline
├── project.json
└── media/
    ├── visual-001.mp4
    ├── visual-002.png
    ├── voice-001.wav
    └── music-001.wav

The project can preserve:

the contiguous main Visuals track;
timed picture-in-picture overlays;
captions and stickers;
voiceover, source audio, and music;
transforms, masks, filters, animations, and keyframes;
media identity and source-time mapping;
canvas ratio and track state.

An Agent can produce a first edit, while a human can continue working in a familiar visual timeline. Changing one caption or replacing one shot does not require regenerating the entire video.

7. Captions must map to audible speech

Automatically generated videos often contain captions with missing, duplicated, or misaligned speech.

Timeline Studio applies a simple invariant:

If captions are enabled, every visible caption must map to exactly one audible speech clip for its full interval.

Transcribed captions bind to existing source dialogue. Agent-authored explanation or narration receives a generated or recorded voice clip instead of becoming silent text.

{
  "captionId": "caption-result",
  "audioClipId": "voice-result",
  "start": 18.4,
  "end": 22.8,
  "text": "One instruction can produce a project you can still edit."
}

Final verification checks caption boundaries, real audible output, duplicate narration, loudness differences, and unexpected stereo-channel offsets.

8. Browser-local AI

Timeline Studio follows a local-first architecture. Supported inference runs in the browser with technologies including WebGPU, ONNX Runtime Web, Web Workers, Cache Storage, and WebCodecs.

Current editor capabilities include:

Whisper Small Q8 ONNX automatic captions;
Piper/VITS Chinese voices and Kokoro English voices;
multilingual browser TTS;
Stable Audio 3 Small Q4 ONNX music generation;
YOLOS Tiny subject detection;
MODNet portrait matting;
MI-GAN object removal;
NanoVSR image and video enhancement;
vocal and accompaniment separation;
JoyVASA and LivePortrait digital-human workflows.

Models are loaded only when a feature needs them and are reused from browser caches. Owned Hugging Face and ModelScope mirrors can provide pinned model revisions while sharing a provider-independent cache identity where possible.

Local-first execution reduces remote inference cost and helps keep unpublished media on the creator's device.

9. Completion means more than “export succeeded”

A completed editing request normally produces both artifacts:

output/
├── result.mp4
└── result.timeline

Before delivery, the Skill verifies:

Visuals continuity and clip ordering;
transition and overlay timing;
caption-to-speech relationships;
the presence of real audible audio;
output dimensions, duration, and container;
full video decoding;
the absence of repeated tails or stalled boundaries;
successful reopening of the .timeline archive;
archived media resolution and first-frame preview.

The render and editable project must both be usable. A single opaque output file is not considered sufficient for a complete editing task.

10. Current boundaries

The browser editor already provides multi-track editing, AI captions, speech, music, visual processing, and rich composition.

The command runner can inspect and modify .timeline projects and render its documented portable Visuals + Voiceover + Music subset. Rich captions, stickers, complex overlays, some effects, and AI generation may still require the browser editor.

The current architecture is best summarized as:

Stable command layer + browser compatibility layer

This boundary is intentional. The project does not describe UI automation as deterministic headless execution, and the command registry can expand without blocking users from accessing editor features today.

Conclusion

Timeline Studio's Agent Skill is not primarily about teaching an Agent to click through a video editor. It defines a production protocol for understanding media, recording editorial decisions, applying safe timeline mutations, and verifying the result.

Its key ideas are:

multimodal media analysis;
content-specific editing workflows;
declarative timeline operations;
revision checks, transactions, and idempotency;
browser-local AI inference;
portable, editable .timeline projects;
validation of both the rendered video and the underlying project.

AI does not need to remove the creator from the process. It can remove the repetitive work while preserving human control over facts, taste, and final editorial decisions.

Explore the reproducible examples:

Timeline Studio Skills Handbook

Install, run, deploy, or contribute:

Timeline Studio main repository