DEV Community: MartinDelophy

Building editable beat templates for a browser video editor

MartinDelophy — Mon, 09 Nov 2026 06:13:03 +0000

A beat template in Timeline Studio starts with a set of media slots and ends with clips you can select on the timeline. In the Memory Vortex example, photos and video stills first appear as panes of a rotating glass tunnel. At 2.70 seconds, the tunnel cuts to the first full-screen shot. The next 48 shots use the same set of media bindings. Release notes

That reuse creates an interesting constraint. If a video shot starts partway through a source file or plays at a different rate, the pane in the opening should show a frame from the shot that actually follows.

One set of bindings, two outputs

The template stores a cut schedule measured on a 30 fps reference. The reference tempo is 106 BPM. Applying the template scales the measured offsets by the ratio between the reference tempo and the selected tempo, so the gaps between cuts and their markers move together. At the default tempo, frame 81 becomes the first full-screen shot.

const secondsPerReferenceFrame = (106 / selectedBpm) / 30;
const timelineTime = start + referenceFrame * secondsPerReferenceFrame;

The template resolves its media slots into actual shot ranges before preparing the opening. The same resolver supplies the full-screen clips. Those clips, the opening effect, beat markers, and optional soundtrack are added as ordinary editor objects. The smoke project for the example contained 49 visual clips, one opening effect, and one music track. Shot preparation and timing · Cut and transition schedule

Prepare the stills once

For a video slot, the atlas builder converts the midpoint of the timeline shot into source time. When multiple slots refer to the same source, it groups them, sorts their requested timestamps, and removes duplicates before extracting frames. The extraction path tries WebCodecs and can fall back to seeking a video element. Photos are decoded and center-cropped into their assigned tiles.

The result is a WebP texture atlas with 8 columns × 6 rows. Each tile is 224 × 280, which gives a 1792 × 1680 image. A decoded RGBA copy of that image is about 11.5 MiB. This bounds the size of the texture the effect samples, while the timeline clips still reference their original media. Atlas builder · Frame extraction

During playback or export, the Canvas 2D effect computes its pane geometry from the current time within the clip. It clips to each visible polygon and uses the source-rectangle form of drawImage() to paint the corresponding atlas tile. The drawing step can then render a frame at any requested timestamp, including a jump during scrubbing, using the prepared image and the effect settings. Canvas renderer

Keep the next beat in place when replacing a clip

A replacement is a local edit. If the timeline clip lasts two seconds and the user selects three seconds from another video, the replacement plays at 1.5×. The editor keeps the clip ID and timeline duration, then updates its media and source range. Reapplying the template with new bindings is the separate operation that rebuilds the opening atlas. Clip replacement

To reproduce the sequence, open Templates → Memory Vortex Beats in the browser editor, keep the sample media and default tempo, and apply the template. At 30 fps, inspect frames 80 and 81 in the export to see where the opening hands off to the first shot. The Timeline Studio repository contains the editor and the template definitions.

Disclosure: AI drafted this article using the released source, project records, and exported frames.

Building ChatCut: Conversational Video Editing in the Browser with Cloud Models

MartinDelophy — Fri, 06 Nov 2026 10:20:45 +0000

“Shorten the second shot, then move the narration and its captions together.”

That request sounds simple until you try to connect a language model to an actual video project. The model needs to identify the right clips, understand their timing, preserve links, and avoid applying a plan after the user has changed the timeline.

I am building ChatCut, the conversational editing workspace in the Anna edition of my open-source project, Timeline Studio. The goal is to let users edit through conversation without installing desktop editing software or setting up their own model inference environment.

GitHub repository · Anna integration notes

This is an architecture walkthrough, including cloud video generation and the music integration under development. Availability depends on the Anna release, account grants, and selected model. The standalone editor and the Anna edition have different capabilities.

What runs in the browser, and what runs in the cloud?

ChatCut combines a conversation panel with the live preview and editable timeline. The cloud model interprets requests and chooses tools. The browser owns the editing state, validates operations, previews changes, and performs composition/export.

User request
     |
Browser: inspect project and available assets
     |
Cloud LLM: choose a supported tool
     |
Browser: validate and execute
     |                        |
Timeline edit          Cloud generation job
     |                        |
Review / apply         Download and decode result
     |                        |
Editable timeline <--- Explicit placement request
     |
Browser composition and export

“No desktop install” does not mean “every operation runs on a server.” This division keeps direct editing and playback close to the user while using platform-hosted models for planning and supported generation tasks.

A bounded tool loop around the editor

ChatCut uses Anna's llm.complete host API. The browser maintains the tool loop; it does not create a separate cloud Agent session or expose an unrestricted standard MCP server.

Each turn starts by inspecting the project. The model receives a filtered catalog of registered editor tools, their input schemas, bounded conversation context, and the user's request.

Examples include:

timeline_project_inspect
timeline_track_inspect
timeline_clip_inspect
timeline_assets_inspect
timeline_edit_preview
timeline_video_generate
timeline_export_prepare

The loop has a hard limit of 16 model calls per turn. Only registered operations are dispatched. Model-generated JavaScript, shell commands, and arbitrary media URLs are not executed as editing instructions.

The model proposes actions. The application enforces their meaning.

Protect the edits made while the model is thinking

A conversation turn captures an initial project state token. Any generated edit preview must still match that state before it can be committed.

If the user trims a shot while the model is planning, the old result becomes stale. Re-reading the project inside the same turn does not silently authorize a plan based on a different initial state.

The shared review engine checks asset and clip IDs, supported operations, timing, media availability, and locked tracks.

When automatic application is enabled, ordinary validated edits can be committed by the host, with inline Undo and View changes receipts. New narration and audio-processing results require listening review and remain pending. Historical receipts cannot apply or undo an unrelated transaction.

This is what makes the conversation part of an editable application rather than a detached suggestion box.

Visual evidence has a budget

A filename is weak evidence about a video. ChatCut can inspect selected source frames with permission, using an imported asset ID and explicit source times.

The decoder is independent of the visible preview, so sampling does not move the user's playback position. A call can return at most six bounded JPEG frames, with eighteen frames allowed per conversation request.

These are source frames. They exclude the editor's rendered captions, overlays, and effects.

Sparse samples can help answer a local question about composition. They cannot establish complete scene understanding, exact cut detection, or audio transcription. The model should not claim to have watched the whole video when it has only seen a handful of frames.

Video generation is an asynchronous media pipeline

The Anna connector uses the real video.generate and video.getJob host APIs.

Explicit generation request
        |
Validate model parameters and app permission
        |
Submit once -> retain job ID -> poll existing job
        |
Download actual media bytes
        |
Decode and commit to My assets

An app-level media grant is required. Selecting a video model alone does not establish permission, and submitting a job consumes the user's Anna CU.

A recorded live text-to-video check submitted a five-second, 480p, 16:9 request. It returned a playable 832x480 video, approximately 5.184 seconds long, with H.264 video and AAC audio. The downloaded asset was stored and the existing timeline was unchanged.

That validates one concrete path. Image-conditioned generation and other model/resolution combinations need their own checks; the adapter's existence is not evidence that every provider combination works.

Stopping local waiting also does not imply that a remote generation job was canceled or refunded.

Generation and placement are separate decisions

A generated video might need another iteration. Committing it directly to the timeline would conflate “the provider returned a file” with “the user accepted the result.”

Completed video outputs are downloaded, decoded, and saved in My assets. Placement uses a separate inspected editing plan when the user requests it.

This separation preserves the original footage and makes recovery easier. Download failure, generation failure, and placement failure become distinct states instead of one ambiguous “try again” button.

Music generation starts with model capability discovery

The Anna music connector queries the official catalog:

llm.catalog({ serviceType: 'music_gen' });

The selected model determines prompt length, lyrics support, instrumental mode, available formats, and whether duration is adjustable. A fixed-duration model receives no invented duration parameter. Lyrics go through the official lyrics field when supported.

No provider secret is embedded in the frontend; requests use the platform's permissioned host interface.

The current ChatCut music tool prepares a prompt and opens the official generation form. It does not silently submit a billed music job: the user clicks Generate in that form. Conversation-driven music preparation and beat-aware montage are working-draft features, not a promise that every published edition already includes them.

Recover a music result without composing—and paying—again

The implemented connector persists the exact request and an idempotency key before submission. An ambiguous outcome keeps its receipt; recovery looks for the same request rather than immediately inventing a new key.

If a successful job's signed download URL expires and returns 403, the connector refreshes the result through getJob. It does not call generate again.

Downloaded audio is decoded to establish duration and waveform data, marked as music, and committed to My assets for explicit Music-track placement. Playback uses the downloaded object URL rather than a short-lived signed URL.

The repository documents build, contract, and local-browser checks for the music connector. Those checks do not establish live paid-generation audio quality, which still needs an authorized end-to-end run.

Music-aware cutting needs evidence beyond BPM

The montage work under development treats templates as starting points. Uploaded and generated music should feed the same analysis path: decoded waveform, beat and accent candidates, then a model-planned sequence of cuts.

Music -> waveform and timing evidence
                    |
Template + source clips + user direction
                    |
              Cut plan
                    |
Validate times, source IDs, and available media
                    |
            Editable sequence

A constant BPM grid alone does not prove that the cuts follow the track's actual accents or sections. This is why analysis, planning, and validation are separate stages.

Export through the existing compositor

ChatCut can prepare export of the full timeline, a selected visual or overlay clip, or an explicit time interval. The visible export action starts rendering after the user clicks it.

Exports reuse the editor's composition pipeline, including the relevant captions, effects, and audio. This avoids maintaining a separate renderer for conversational edits.

The browser still does real work. Performance depends on the source media, browser capabilities, device, and composition complexity; cloud planning does not remove those constraints.

Where to explore the code

Start with the open-source repository, and the Anna implementation on the codex/anna-validation branch:

src/lib/annaChatCut.js — bounded conversation loop and state guards
src/components/ChatCut.jsx — conversation UI and generation handoffs
src/lib/annaRuntime.js — host API adaptation
src/plugins/generation/providers/anna/ — official generation adapters
docs/anna-chatcut.md — execution contract and workflow
docs/anna-video-generation.md — video integration and live validation
docs/anna-music-generation.md — model capabilities and recovery behavior

The standalone browser editor lets you explore Timeline Studio's editing foundation. ChatCut and the platform-hosted generation paths described here belong to the Anna edition; use the integration notes to understand their release and permission requirements.

What interests me most about this architecture is the connection between an accessible conversation and a durable editing model. Users can ask for a change, inspect the result, undo it, and continue manually—all inside the same browser workspace.

If you build agent tools, browser media applications, or generation pipelines, I would welcome feedback and reproducible failure cases through the repository.

Disclosure: I maintain the project discussed here. This article was drafted by an AI assistant using the current source code and implementation records.

Turning Video into Line Art in the Browser with WebGL 2 and Sobel

MartinDelophy — Wed, 04 Nov 2026 06:40:49 +0000

Depth maps and line art preserve different parts of a video. Depth describes spatial relationships; contours emphasize shape, composition, and visible changes in pose.

After adding depth-video conversion to Timeline Studio, my open-source browser video editor, I added a line-art asset generator. This version runs locally with WebGL 2 and Sobel edge extraction. It does not download or run a neural line-art model.

Source code · Try the editor

Here is how the image-processing pipeline connects to actual video editing: previews, trimmed clips, speed curves, source audio, encoding, and cancellation.

Two passes on the GPU

The pipeline is deliberately small:

Image or decoded video frame
          |
   Grayscale + 3x3 smoothing
          |
   Intermediate GPU texture
          |
   Sobel + soft threshold
          |
    Output canvas
          |
      PNG or WebM

The smoothing pass renders into a texture attached to a framebuffer. The edge pass samples that texture and draws to the output canvas. Intermediate pixels stay on the GPU rather than being read back into JavaScript between passes.

Shader programs, buffers, textures, and the framebuffer are reused across frames. That reduces setup work, although decoding and encoding still contribute to total processing time.

Convert to luminance, then smooth

The shader computes a weighted RGB luminance:

float gray(vec2 p) {
  return dot(
    texture(source, p).rgb,
    vec3(0.2126, 0.7152, 0.0722)
  );
}

The first pass uses this weighted 3x3 kernel, divided by 16:

1  2  1
2  4  2
1  2  1

Smoothing suppresses some high-frequency texture before edge extraction. It is a tradeoff: small details can also become weaker.

Because detection operates on luminance, different colors with similar luminance may produce weak boundaries. Conversely, a shadow or reflection can produce a strong edge without marking an object boundary.

Extract Sobel gradients

The second pass estimates horizontal and vertical gradients:

Gx                  Gy
-1  0  1            -1 -2 -1
-2  0  2             0  0  0
-1  0  1             1  2  1

Their combined magnitude becomes the edge strength:

float gradient = length(vec2(
  c + 2.0 * f + i - a - 2.0 * e - g,
  g + 2.0 * h + i - a - 2.0 * b - c
));

The letters represent neighboring luminance samples. Each output pixel performs the same local calculation, which maps naturally to a fragment shader.

Make the parameters useful

The inspector exposes detail, line weight, and inversion. Defaults are 55% detail, 1x line weight, and black lines on white.

Detail maps to an edge threshold:

float threshold = mix(0.42, 0.035, detail / 100.0);
float ink = smoothstep(threshold * 0.5, threshold, gradient);
float value = inverted ? ink : 1.0 - ink;

Higher detail lowers the threshold, retaining weaker edges. The soft threshold avoids an abrupt binary transition around the cutoff. For busy footage, increasing detail can make the result less useful by retaining background texture.

Line weight changes the sampling distance. It uses the longest dimension as a 1080px reference:

const scaledThickness =
  thickness * Math.max(width, height) / 1080;

This helps keep the edge-sampling scale comparable across output sizes. It is not morphological dilation: changing the sampling distance also changes which details respond. The initial blur itself remains a one-pixel 3x3 neighborhood.

Preview and asset generation share one renderer

Both paths call the same implementation:

const renderer = createLineArtRenderer(canvas, width, height);
renderer.draw(media, options);
// Release resources when finished:
renderer.dispose();

The preview redraws a sampled frame when parameters or the local playhead time change. Asset generation traverses output frames through that same renderer.

Sharing the implementation reduces differences in thresholds, sampling, and inversion between preview and generated assets. It does not make processing instantaneous.

Output dimensions preserve aspect ratio, cap the longest edge at 1080 by default, avoid enlarging smaller media, and normalize dimensions to even numbers.

Video has to follow the timeline's clock

A selected clip may already be trimmed or retimed. Sampling the original file from zero would produce the wrong result.

There are separate coordinates for output time, clip-local time, and source-media time. For constant speed:

sourceTime = sourceIn + clipLocalTime * speed

For speed curves, the implementation reuses the editor's mapping:

getVisualSourceTime(segment, localTime)

The generator outputs 24 fps. Each local sample time is i / 24; it is mapped to source time before decoding the frame. The processed frame is then encoded at its output timestamp.

Output frame time
       |
Source-time mapping, including trim and retiming
       |
Decode source frame -> draw contours -> encode output

Mediabunny handles media input and timestamp-based canvas sampling. The final frame duration is limited to the remaining clip duration.

24 fps is the output rate, not a claim of real-time generation speed. Actual throughput depends on the browser, hardware, source media, and encoder.

Normalize still-image orientation

One integration detail had nothing to do with Sobel: image orientation.

The image path decodes a Blob to an ImageBitmap, then draws it to a 2D canvas before uploading it to WebGL. ImageBitmap does not follow the same UNPACK_FLIP_Y_WEBGL behavior as the other upload path, so the intermediate canvas normalizes orientation.

Without that step, still images and video frames could disagree about which side is up.

Encode a reusable asset, with corresponding source audio

Images become PNG. Videos become WebM using VP9 at approximately 4 Mbps. When corresponding source audio exists, the output uses Opus at 192 kbps.

Audio preparation reuses the editor's source-audio and offline-mixing helpers so trimming and retiming follow the selected segment. This preserves the segment's corresponding source sound; it does not mix every music and narration track from the whole project.

The completed asset goes to My assets. It is not automatically inserted into the timeline, making it possible to retain the original and compare versions before placement.

Cancellation includes cleanup

An AbortController carries cancellation through source reading, frame processing, and output finalization. The renderer releases its textures, framebuffer, shaders, programs, and vertex buffer. Media input resources are disposed, and encoding is canceled when necessary.

Frame processing reports progress up to 99%. Finalization and the asset commit must finish before the UI reports completion. Processing the last frame is not the same as having a usable saved file.

What structural reference means here

Depth video and contour video provide complementary reference assets:

Asset	Emphasizes
Depth video	Relative depth and foreground/background layout
Line-art video	Visible contours, composition, and changes in shape

A downstream generation workflow can use these only if it accepts the corresponding conditioning input and its preprocessing, timing, resolution, and frame-rate requirements are satisfied.

This feature creates the reference asset. It does not imply automatic integration with every image or video generator, nor guarantee that a generator will preserve the desired structure.

The current Sobel route also lacks semantic simplification and explicit temporal tracking. Fine textures, noise, and motion can make lines fluctuate across frames. A learned line-art model or additional temporal processing would need separate evaluation rather than being claimed as part of this implementation.

Try it or read the code

Open Effects -> Line art, select an image or video clip, adjust parameters, and generate the asset. The feature includes 13 interface languages and a mobile layout.

Implementation entry points:

src/lib/lineArt.js: renderer, source sampling, encoding, and cleanup
src/components/LineArtPanel.jsx: preview, controls, and job state
src/i18nLineArt.js: localized copy

If you work on browser graphics or video tools, I would welcome difficult examples: busy backgrounds, small contours, and fast motion are particularly useful for understanding where this approach needs improvement.

GitHub repository · Open the editor

Disclosure: I maintain the project discussed here. This article was drafted by an AI assistant using the current repository source and implementation notes.

I Built an Editable Timeline for AI Agents: Skills, MCP, and Transactional Video Editing

MartinDelophy — Tue, 03 Nov 2026 07:20:35 +0000

An AI agent can generate an FFmpeg command. But what happens when the next request is: “Shorten the second shot, move the voiceover and its captions together, and give me a project I can keep editing”?

That is the problem I have been working on in Timeline Studio, an open-source, local-first browser video editor with an Agent Skill and a shared command layer.

This post focuses on the editing architecture: how an agent turns a request into an inspectable plan, how that plan changes project state, and where rendering still has explicit limits.

Start with an editable project

A rendered MP4 does not preserve the decisions needed for the next edit: clip identity, source ranges, timing, transforms, and caption/audio relationships.

Timeline Studio uses an editable .timeline project as the source of truth. In the local command workflow, the archive contains project.json and embedded media. Asset import probes necessary metadata, computes SHA-256 integrity information, and embeds the media bytes.

This gives both the user and the agent a concrete state to return to. The user can reopen the project in the editor. The agent can inspect it before making another change.

Separate editorial decisions from state changes

The model can decide that a voiceover should start at three seconds. The application should enforce whether the clip exists, whether the time is valid, and how linked captions follow it.

The architecture looks like this:

User request + media evidence
             |
     Skill: analyze and plan
             |
     Structured edit plan
             |
 CLI / local MCP / browser WebMCP
             |
 Shared command registry and reducers
             |
 Editable project + verified output

The transports have different environments. The CLI operates on local archives. The local STDIO MCP adapter exposes that command runner to an agent. Browser WebMCP, when exposed by the host, operates on the open editor and commits through its normal history.

The important constraint is to reuse editing rules rather than implement a separate editor for each transport.

An edit plan needs preconditions

Here is a simplified plan that moves an existing voiceover to project time 3s:

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 12,
  "dryRun": false,
  "operations": [
    {
      "id": "move-voice-001",
      "type": "timed.move",
      "track": "audio",
      "clipId": "voice-1",
      "start": 3
    }
  ],
  "output": {
    "project": "/projects/output.timeline"
  }
}

The revision and clip ID are illustrative: they must come from inspecting the actual project.

Each field has a job:

baseRevision checks the state on which the plan was based.
Operation IDs support idempotent handling of retries.
clipId identifies an existing project object.
type restricts the action to a supported operation.
output.project declares a new output artifact.

For this audio move, still-linked captions move by the same delta. That behavior belongs in the editing rules.

Preview a semantic diff before applying

The workflow is:

inspect -> plan -> validate -> diff -> apply -> inspect

The CLI exposes it directly:

npm run agent -- project.inspect /projects/input.timeline

node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs \
  /projects/plan.json

npm run agent -- project.diff /projects/plan.json
npm run agent -- project.run /projects/plan.json

The standalone validator checks transport shape, finite time fields, and duplicate operation IDs. It does not prove that a plan is valid for a particular project.

project.diff is the project-aware dry run. It uses the same command registry and reducers as application, without writing an archive. It checks supported operations and preconditions, then reports project fields, clip additions/removals, changed fields, ordering, and duration changes.

This makes accidental scope expansion visible. A request to move narration should not quietly change the visual sequence duration.

Retries and concurrent edits are normal cases

Three mechanisms matter here.

Idempotency

If an operation completes but its response is lost, retrying the same operation ID should not insert the same clip twice. Previously applied IDs are treated as no-ops.

Revision checks

If the agent inspected revision 12 and the user subsequently edited the project, a stale plan containing new operations must fail with a revision conflict. The agent needs to inspect again and reconsider the plan.

Transactional application

A failed batch should not leave a partially edited output archive. Local CLI and MCP writes also require a new output path and reject overwriting the input or an existing output.

These are familiar application-engineering techniques. They become especially useful when the caller is an agent that may retry or reason from outdated state.

Video editing has more than one clock

A timestamp can refer to source media, a clip-local position, or the final project.

For a constant-speed clip:

sourceTime = sourceIn + (timelineTime - clipStart) * speed

With a speed curve, source-time mapping must accumulate the changing playback speed over time.

An agent identifying an action at source time 8s cannot assume that the cut belongs at project time 8s. Trimming, splitting, keyframes, thumbnails, and effect analysis all depend on maintaining that mapping.

Tracks also have different semantics. Main visuals form a contiguous sequence; narration, captions, and overlays have explicit timing. Whether a duration change shifts other tracks depends on ripple mode, links, and track locks.

These rules need to be explicit application behavior.

Skills and MCP serve different purposes

The Skill describes the workflow: inspect material, plan the edit, choose an execution path, and verify the result.

The MCP adapter exposes concrete tools, including:

timeline_project_inspect
timeline_track_inspect
timeline_clip_inspect
timeline_project_diff
timeline_project_apply
timeline_project_render

The local adapter is a thin transport over the existing command runner. It does not duplicate timeline editing logic.

To install the Skill:

npx skills add MartinDelophy/ai-video-editor \
  --skill edit-timeline-studio

To explore the repository and local MCP server:

git clone https://github.com/MartinDelophy/ai-video-editor.git
cd ai-video-editor
npm install
npm run mcp

Installing the Skill does not install the complete runtime. Local MCP needs the checkout and Node dependencies; media operations need their corresponding tools. The Skill includes a host dependency-check workflow.

An edited project is not yet a verified video

The current local project.render path supports the documented portable Visuals, Voiceover, and Music subset. It uses FFmpeg to produce H.264/AAC MP4 and ffprobe to check dimensions, duration, and audio-track presence.

It rejects unsupported composition features rather than silently dropping visible captions, overlays, transitions, or effects. Richer compositions require the browser renderer.

Verification needs several levels: inspect the project, reopen it and resolve archived media, check the output container, and actually play the important visual and audio boundaries. A valid project structure alone cannot prove that the final mix sounds right.

Where to read the implementation

Useful entry points in the repository:

src/lib/projectCommandEngine.js — shared project command engine
scripts/timeline-command.mjs — local archive operations and rendering entry point
skills/edit-timeline-studio/mcp/server.mjs — STDIO MCP adapter
src/hooks/useWebMcpEditor.js — browser editor integration
skills/edit-timeline-studio/SKILL.md — agent workflow

The command reference documents the current contract and limits.

The larger lesson for me is that agent reliability depends on the application interface: explicit state, constrained operations, reviewable changes, and verifiable outputs.

If you are building agent tools or working on browser media, I would welcome feedback on the architecture. Real editing failures and reproducible cases are particularly useful. You can explore the source, try the editor, or browse the editing handbook.

Writing disclosure: this article was drafted with AI assistance using the project's source code and documentation.

Rehearsing AI Video in the Browser: A Three.js Blockout Workspace

MartinDelophy — Tue, 20 Oct 2026 07:35:21 +0000

A generated video can look beautiful and still tell the wrong story. A character enters from the wrong direction. A close-up replaces the shot that should establish the room. A prop removed in one scene reappears in the next.

In Timeline Studio, our open-source, local-first video editor, we built a usable browser-based 3D blockout workspace to make those directing decisions visible before final generation.

Source code on GitHub · Try the Anna edition

This article covers the architecture and lessons from a separate AI short-film experiment that helped motivate it. The experiment's blockout footage was made independently; the editor did not generate that finished film in one click.

What a blockout makes explicit

A blockout is a simplified rehearsal: basic characters, primitive props, camera positions, and timed movement. Materials and detailed appearance can wait while we check where people stand, what the camera sees, and whether an action has enough time to finish.

“Walk to the desk, put down the milk, and return a hair clip” hides several decisions: which side of the desk, when the hand releases the carton, where the handoff happens, and when to cut to the other character's reaction.

The goal is to give these decisions an editable representation.

The workspace is directly editable. The voice-binding area shown in this screenshot is future work.

Separate scene data from rendering

The workspace uses Three.js and loads separately from the main editor. Its scene is a versioned, serializable document with objects, timed motions, camera shots, script segments, and a total duration.

A simplified view of the data model is:

{
  revision,
  duration,
  objects,
  motions,
  shots,
  script
}

Three layers share responsibility:

Layer	Responsibility
Scene commands	Validate operations and produce the next scene snapshot
Three.js viewport	Selection, transforms, playback, and camera-view output
Workspace and chat	Editing state, timeline, script, and proposal review

The scene command layer is the common boundary for manual controls and AI proposals. The renderer displays the scene; it does not define a second set of editing rules.

One clock for actors, cameras, and script

At playback time t, the workspace computes actor poses, selects the current camera shot, and finds the corresponding script segment.

For a linear move between P0 and P1 over [t0, t1]:

u = clamp((t - t0) / (t1 - t0), 0, 1)
P(t) = P0 + (P1 - P0) * u

Preview and video export use the same actor-pose and camera-selection helpers. That reduces the risk of a preview showing one arrangement while an export uses another.

Clicking an action, shot, or script segment seeks to its start. Script text and time ranges remain editable. Changing the total scene duration scales timed content together. Script timing is an editable estimate, not automatic alignment to recorded speech.

AI proposes edits through a constrained command layer

In the Anna integration, the model receives the current scene metadata and a scene-edit catalog. It submits structured proposals rather than arbitrary executable code.

The flow is:

Inspect scene snapshot
→ Propose structured operations
→ Validate commands
→ Review the proposed edit
→ Check that the scene still matches
→ Apply as one undoable edit

Validation checks object identities, values, and supported timing constraints. Unknown objects, unsupported operations, arbitrary code, and external assets are rejected. If the scene changes while a proposal is pending, that old proposal cannot overwrite the newer edits.

This keeps natural-language assistance connected to an inspectable, editable project. Manual editing works locally; remote planning needs a working Anna host session.

Export a rehearsal you can actually watch

The workspace exports scene JSON and a real 1280 × 720 WebM camera-view video, with progress and cancellation. Video capture uses the browser canvas stream and MediaRecorder; available codecs depend on the browser.

Scene changes persist per project in the same browser. This does not imply cross-device synchronization. The blockout workspace is also separate from the main video timeline.

What we learned from a short-film experiment

In a campus short-film experiment, we compared independent blockout footage against AI-generated shots. The comparison images accompanying this article show corresponding narrative moments sampled proportionally from clips of different lengths. They are not exact timestamp matches or proof of frame-by-frame adherence. All realistic character images are AI-generated.

G02 r5: blockout above, AI-generated image below. Proportional narrative-moment sampling, not frame-exact alignment.

1. Similar composition does not mean the whole shot passed

Comparisons made it easier to inspect character positions, desk layout, occlusion, and camera direction. But a matching still did not establish that a whole sequence followed the rehearsal.

One generated segment preserved the broad order—enter, approach, put down milk, return the hair clip—while changing medium shots into close-ups or over-the-shoulder views and delaying a reaction shot. We did not mark it as strict adherence.

2. Sometimes the rehearsal contains the mistake

A character needed to return to his seat before a segment ended. Initially, we blamed generation for not completing the action. Reviewing the blockout showed that the original staging itself had left too little time.

We moved the return approximately 0.7 seconds earlier and adjusted the ending shot. The lesson was practical: inspect the reference before adding stronger wording to the prompt.

3. Prop interactions need motion review

Putting down milk requires a stable contact with the table before release. Returning a hair clip requires an actual handoff and the correct resulting prop state.

A still helps locate a problem. It cannot establish whether the fingers, object, and movement remain coherent throughout the action. Full playback is still necessary.

4. Continuity references can propagate errors

Using the previous clip's real final frame helped carry appearance and pose forward. It also carried an incorrect hair clip into the next segment. A conflicting environment reference contributed to a window being replaced by a blackboard.

We preserved the original final frame, made a corrected reference copy, and removed the conflicting environment image.

Reference	Main information
Blockout video	Space, movement, cameras, and pacing
Character image	Appearance and clothing
Previous final frame	State at the end of the preceding segment
Environment image	Setting and layout

More references help only when their information is compatible.

What this contributes—and what remains open

The clearest benefits we observed were earlier staging checks, a concrete comparison baseline after generation, and more precise discussions about revisions.

We have not measured a reliable percentage reduction in retries or cost. A blockout also cannot guarantee that a generation model will execute its camera plan.

The browser workspace already supports direct editing, timed actions and shots, script timing, reviewed AI edits, and JSON/WebM export. Voice binding, complex facial animation, collision and IK, external mesh imports, and full integration with the video timeline remain unfinished. There is no automatic end-to-end blockout-to-final-video pipeline.

The direction is to keep directing decisions visible and revisable before committing to expensive final imagery.

If you work on browser 3D, AI filmmaking, or agent-assisted editors, I'd welcome a concrete shot-control problem or an implementation discussion.

Feature availability can differ between the public editor and the Anna edition.

Building an AI Video Editor: Reviewable Commands, State Tokens, and Browser Export

MartinDelophy — Wed, 14 Oct 2026 09:51:33 +0000

Disclosure: I work on Timeline Studio, the open-source editor discussed below. This article was prepared with AI assistance.

A generated video shot can be beautiful and still need another cut. The opening lingers, the narration arrives too late, or the sequence needs a different order.

We are building Timeline Studio around that next creative decision: a browser video editor where generated media and agent-assisted operations become part of an editable multitrack project.

Here are the architectural choices behind its command interface, with a small example you can follow in the repository.

Repository screenshot; interface details may differ in the current release.

1. Separate a source asset from a timeline clip

A source video can appear several times in one project, each time with a different trim, playback speed, transform, or effect. The asset identifies the media; the clip identifies one use of it.

When a user says “move the second shot to the beginning,” an agent first needs to resolve that description against the current sequence. Execution then targets a stable clip ID, rather than a filename or screen coordinate.

Timeline Studio exposes project, track, and clip inspection. An agent can read the overall structure and then request the relevant objects. Metadata inspection is distinct from understanding the picture or transcribing speech; those require their own processing routes.

2. Turn intent into an explicit plan

The local command interface accepts a versioned JSON plan. This example is illustrative: the revision, clip ID, and source range must match an inspected project.

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 12,
  "dryRun": false,
  "operations": [
    {
      "id": "trim-001",
      "type": "visual.trim",
      "clipId": "visual-123",
      "sourceIn": 2,
      "sourceOut": 6
    }
  ],
  "output": {
    "project": "/projects/output.timeline"
  }
}

The envelope states which project revision the plan assumes, identifies operations, and specifies a new output path. The operation states exactly which source interval to retain.

Planning interprets the request. Execution enforces the supported operations and project constraints. This boundary also lets non-conversational tools produce the same plans.

3. Validate meaning, not just JSON

Valid JSON can describe an impossible edit. The clip may no longer exist, the source range may be invalid, or the track may be locked.

We separate transport-shape validation from project-aware validation. A semantic dry run uses the shared command registry and state transformations to predict the actual result without writing an archive.

The local workflow is:

npm run agent -- project.inspect /projects/input.timeline
npm run agent -- project.diff /projects/plan.json
npm run agent -- project.run /projects/plan.json

The diff reports predicted changes to project fields, clips, timing, and duration. It gives review a concrete connection to execution. The local writer requires a new output path, preserving the input project.

4. Reject stale plans in a live editor

There is a concurrency problem even with a single human user:

The agent inspects the project.
The user splits or moves a clip.
The agent returns a plan based on the earlier state.

The browser interface uses a state token to bind preparation to the inspected state. A reviewed edit receives a preview ID. Applying it checks that the editor still matches the reviewed snapshot; otherwise, the agent must inspect again and replan.

Successful edits enter the editor's normal history mechanism, so the user can continue adjusting and undoing them.

5. Reuse editing rules across transports

The project provides a CLI, a bundled STDIO MCP adapter, and a browser WebMCP interface. Separate implementations of trimming, sorting, and link handling would invite inconsistencies.

The CLI and MCP adapter use the same application service. The browser adapter compiles reviewed operations into the shared command engine and commits through the editor.

Capability boundaries still matter. The browser has live media, inference jobs, and the full editor export workflow. The local renderer supports a defined portable subset and rejects unsupported compositions.

6. Keep generation and placement separate

Generation providers differ in authorization, job protocols, and output formats. Source-integrated adapters handle service-specific transport; a shared host receives downloaded media and commits it to My assets.

Completed generation does not automatically insert footage into the timeline. A creator may want to compare candidates first. This also prevents a long-running job from unexpectedly modifying an active edit.

Track submission, generation, download, and asset commit separately. A successful generation followed by a failed download needs a different recovery path from a failed generation. Stopping polling also does not by itself cancel a remote task.

7. Put preview and export on the same timeline clock

A source interval from 10 to 18 seconds, played at 2×, occupies four timeline seconds. For constant speed:

source time = source in + clip-local time × playback speed

Speed curves require accumulating the changing rate over time. Preview, splitting, thumbnails, and export need to agree on that mapping.

Timeline Studio uses native media paths for responsive playback and a separate deterministic offline rendering path for export, with WebCodecs-based composition and a compatibility fallback. Availability depends on the browser and device.

Resource scheduling matters too: thumbnail preparation is progressive, visible regions receive priority, and scrubbing pauses background refinement. Bounded processing queues keep throughput improvements from becoming unbounded memory growth.

What is available now?

ChatCut is usable, and the Anna edition includes video-generation integrations. Its setup and experience entry points are documented in the repository. Hosted generation is distinct from browser-local processing.

The command contract and browser adapter are useful starting points if you are building a similar system. The goal is to preserve an understandable project at every handoff: from request to plan, plan to reviewed operations, and editable state to rendered output.

Explore the Timeline Studio source and documentation on GitHub.

What would you put in an agent-facing editing contract first: timing, media identity, or transaction handling?

Cutting Browser Video Export Time by 31%: An 8-Minute WebCodecs Case Study

MartinDelophy — Mon, 12 Oct 2026 05:54:01 +0000

An eight-minute video is long enough to expose work that is easy to miss in short browser-editor demos: audio preparation, repeated frame conversions, background thumbnail decoding, and idle time between decode, composition, and encoding.

We recently optimized these paths in Timeline Studio, an open-source browser video editor. In a controlled comparison against the code before the first optimization round, average export time fell from 192.0 seconds to 131.9 seconds—a 31.3% reduction, saving about 60 seconds.

Source code on GitHub · Try the editor

Benchmark: what we actually measured

Both versions ran on the same machine with the same approximately eight-minute synthetic stress-test video. The input was H.264 at 1440 × 1080 and approximately 29.88 fps, with stereo AAC audio at 44.1 kHz.

Output settings were identical:

H.264, 1440 × 1080, 30 fps
10 Mbps target video bitrate
AAC audio at 192 kbps
No effects or overlays in this benchmark

Timing started immediately before embedded-audio preparation and ended when export completed. Import and timeline-thumbnail preparation were excluded.

We ran old → new → new → old, starting a fresh browser process for each run.

Version	First run	Second run	Mean total
Before the first optimization	203.7 s	180.4 s	192.0 s
Current implementation	141.6 s	122.3 s	131.9 s

Stage	Before, mean	After, mean
Audio preparation	9.3 s	0.9 s
Video export stage	182.7 s	131.0 s

These are two runs per version, with visible runtime variation. The input is synthetic, so the 31.3% result should not be generalized to every camera recording, device, or effect stack. We also checked decoding and compatibility with a native Sony ZV-1 4K camera sample, but that sample is not the source of the eight-minute benchmark result.

1. Avoid decoding the entire audio track just to prepare it

The old preparation path used FFmpeg to decode embedded audio into WAV before subsequent processing. For long inputs, this adds whole-track decoding, PCM allocation, and intermediate-data handling.

For eligible AAC tracks, the new path extracts encoded packets and remuxes them instead. Final output audio is still encoded according to export settings; this change removes unnecessary work in the preparation stage.

Timestamp handling matters. AAC can contain preroll and negative timestamps. Simply copying packets while discarding that information can shift audio relative to the picture. The implementation preserves and compensates for the source offset and distinguishes storage duration from logical segment duration.

Other formats retain the fallback path. In this benchmark, mean audio preparation dropped from 9.3 seconds to 0.9 seconds.

2. Remove an intermediate frame conversion only when it is safe

At 30 fps, an eight-minute output contains approximately 14,400 frames. Even a small repeated operation is worth examining at that scale.

The new eligible path sends decoded VideoFrame objects directly to the shared compositor, avoiding an additional intermediate Canvas rasterization step.

Eligibility is deliberately narrow: original-size, unrotated, square-pixel frames with no relevant transformations or effects that require the existing rasterization path. Scaling, grading, animation, masks, and other affected cases keep their compatible Canvas path.

This restriction came from validation: different rasterization routes produced small pixel differences when resizing. We therefore did not extend the direct-frame optimization to those cases.

3. Overlap work with bounded queues

A fully serial loop can leave resources waiting:

Decode frame → Compose frame → Submit/wait for encoding → Repeat

The updated pipeline overlaps a limited amount of work:

Prepare one decoded frame ahead.
Compose the next frame while a video submission is in flight, subject to backpressure.
Process audio and video concurrently.
Release frames promptly after use.

The queue bounds are essential. Unrestricted prefetching can turn a throughput problem into a memory problem on long videos.

The exporter also probes supported hardware-encoding configurations and falls back when necessary. Actual hardware acceleration depends on the browser, operating system, and device.

Cancellation needs the same discipline: stop submitting new work, let bounded in-flight operations settle, and then release output resources. Closing everything immediately can leave asynchronous waits stranded.

4. Give playback priority over background thumbnails

Export was only part of the original problem. Playback and seeking also competed with background timeline work.

We changed scheduling so that:

Playback pauses background thumbnail refinement.
Scrubbing cancels background thumbnail work for the pointer gesture.
Releasing the pointer resumes viewport-prioritized refinement.
Starting playback prioritizes the latest seek instead of stale scrub requests.
The playback clock pauses while decoding is seeking or buffering, reducing cases where the timeline advances while the picture is frozen.

For live thumbnail updates, confirmed decoded frames are drawn directly rather than repeatedly going through JPEG encoding and decoding. Source timestamps remain aligned with timeline sampling.

For locally imported media processed in the browser, these are client-side resource and coordination issues worth measuring alongside network behavior.

Validation: speed with the same output settings

We kept output resolution, frame rate, and target bitrate unchanged. Validation covered:

Output video/audio tracks, duration, and dimensions.
AAC packet extraction and source-offset handling.
Pixel comparisons for eligible direct-frame cases.
Canvas fallback behavior for scaling, grading, and animation.
Encoded-packet consistency for a fixed effects case across the pipeline scheduling change.
Cancellation and resource cleanup.

This does not establish byte-identical output between the original version and the final version, or cover every possible project. It checks the specific paths changed by this optimization.

Explore the implementation

The main gains came from reducing intermediate work and overlapping bounded tasks while retaining the processing paths required by effects.

If you reproduce a performance issue with your own footage, include your device, browser version, source codec parameters, export settings, and effects in a GitHub issue. Separating preparation time from export time makes the bottleneck much easier to discuss.

Video Depth Estimation in the Browser: Building with WebGPU and Depth Anything V2

MartinDelophy — Thu, 08 Oct 2026 06:15:19 +0000

An ordinary video records color and motion, but it does not explicitly tell an editor which pixels are near the camera and which belong to the background.

I added browser-local depth estimation to Timeline Studio, my open-source video editor, to turn that missing information into an editable resource. The current implementation supports depth-aware blur, depth-based relighting, and exporting an independent grayscale depth video.

This post covers the implementation and what happened when I used the resulting depth videos as references for AI video generation.

What the model actually predicts

The implementation uses Depth Anything V2 Small to estimate relative depth from RGB images. In our default visualization, brighter pixels generally represent nearer surfaces, and darker pixels represent farther ones. The display can also be inverted.

Relative depth is not a distance measurement. A grayscale value does not tell us that an object is exactly two meters from the camera. It is useful for visual layering and effects, but it is not a calibrated 3D reconstruction.

The browser pipeline

The processing flow is:

Video segment
  -> map timeline time to source time
  -> decode and resize a frame
  -> run depth inference in a Web Worker
  -> normalize and refine depth pixels
  -> preview an effect or encode a depth video

The worker loads the model through Transformers.js with WebGPU and a Q4F16 configuration. The essential setup looks like this:

const estimator = await pipeline(
  "depth-estimation",
  modelPath,
  {
    device: "webgpu",
    dtype: "q4f16",
    progress_callback: reportProgress,
  },
);

Production setup also handles pinned model revisions, mirrored downloads, progress reporting, optional caching, and failures.

A worker keeps inference work off the main JavaScript thread. It does not eliminate resource contention: decoding, inference, and editor rendering still share the device. That is why bounded work matters.

Video frames stay local during this depth-analysis path. Sending the exported depth video to a remote generation service is a separate operation that uploads reference material.

Timeline time is not source time

This was one of the most important integration details.

An editor segment may be trimmed, sped up, slowed down, or assigned a speed curve. Sampling the original file at the same numerical timestamp as the timeline would produce incorrect depth frames.

The analysis uses the editor's shared source-time mapping:

const sourceTime = getVisualSourceTime(segment, localTime);

Preview and depth-video output use the same mapping. Analysis cache signatures include the source, trim, playback rate, speed curve, quality setting, and model revision. Relevant changes invalidate the old analysis.

A good depth estimate attached to the wrong video frame is still a broken effect.

Bounding inference and memory

The current analysis presets are:

Preset	Target sample rate	Maximum input dimension
Fast	8 fps	392 px
Balanced	16 fps	504 px
Quality	24 fps	518 px

These are sampling settings, not measured real-time inference speeds. Actual throughput depends on the browser, GPU, and workload.

Each analysis has a 720-sample budget. Longer clips reduce their effective sampling density to cover the segment within that budget.

The implementation overlaps inference for the current frame with preparation of the next frame. It keeps only a limited number of tasks in flight instead of decoding a whole video into memory.

Independent depth-video output runs at 24 fps. Fast and Balanced modes use RIFE interpolation between depth samples. Interpolation can make transitions smoother, but it does not add genuine depth observations and can still fail around rapid motion or occlusion.

Turning depth into an editable effect

For depth-aware blur, the renderer prepares a sharp layer, a blurred layer, and a depth-derived mask.

Pixels close to the selected focus depth stay sharp. Pixels outside the focus band progressively mix in the blurred layer, with a smooth transition at the boundary.

This is a compositing approximation rather than a physically exact lens simulation. Hair, transparent materials, and complex occlusion remain challenging.

The depth output also receives a small edge-guided smoothing pass. Neighboring pixels contribute less when their depth or source-color differences are large. This reduces unwanted mixing across foreground/background boundaries; image texture guides smoothing rather than becoming new geometry.

Exporting a reusable depth video

The editor can encode depth results as a new asset in My assets. The current browser path uses WebM with VP9 video, preserves the source aspect ratio, and caps the longest output dimension at 1080 pixels. Where source audio exists, the export prepares audio matched to the segment timing.

Output size and inference resolution are different. Encoding a larger image does not recover geometry the model never predicted.

Experiment: depth references for AI video generation

I also tested a workflow combining three inputs:

A depth video for silhouettes, relative depth, occlusion, and framing changes.
Character images for appearance, clothing, and materials.
Text describing the action stages, environment, and visual style.

There is an important qualification: these experiments submitted the depth video as an ordinary reference video, not through a dedicated depth-conditioning API. The generation service had to interpret the grayscale reference itself.

Some broad movements survived, including turning toward the camera and moving into a closer shot. But the results were not frame-exact. In one experiment, a foreground limb shape was interpreted as a different body action, and head/shoulder proportions drifted from the reference.

A separate two-character sleeve-transition experiment followed the main action structure more closely when I supplied the original RGB video instead. Sampled-frame inspection showed the sleeve raise, full-frame occlusion, and subsequent close-up, while hand details still differed.

The practical lesson is to choose reference inputs according to the task. Depth emphasizes spatial structure, while RGB retains semantic information about gestures, fabric, and objects.

A relative depth map also does not directly provide a calibrated camera FOV. Framing fidelity needs to be checked through screen-space proportions, crop boundaries, and camera changes. Uploading a depth reference alone is not proof of accurate reconstruction.

What is working, and what still needs work

The browser implementation connects depth analysis to editable effects and independent video output. The engineering work is as much about timing, cancellation, progress, cache validity, and bounded resources as it is about running a model.

The generation experiments are useful evidence of possibilities and limitations, not a promise of precise motion transfer. Better temporal stability, difficult boundaries, and dedicated depth-conditioning integrations remain areas to improve.

If you are building browser AI tools or experimenting with depth-guided editing, I would be interested in how you handle these issues.

Try it or explore the source

Timeline Studio is free and open source. Feedback, reproducible issues, and contributions are welcome.

Depth-Assisted Video Relighting in the Browser with WebGPU and WebGL

MartinDelophy — Sun, 04 Oct 2026 14:53:58 +0000

I added an editable video-relighting effect to Timeline Studio, my open-source browser video editor. It lets you move a virtual light around a scene, adjust its color and softness, and compare the result with the original while keeping the effect editable on the timeline.

The useful part is the division of work: a depth model analyzes the footage, then a small GPU shader handles lighting adjustments. Moving the light does not require another inference pass.

Try Timeline Studio · Explore the source

Light-position artwork from the editor. The interactive control uses a sphere to make light direction easier to understand.

From cinematic depth to editable lighting

The editor already used depth estimation for its cinematic depth-of-field effect and grayscale depth-map videos. Relighting builds on that same scene-depth information.

The pipeline is:

Decode sampled frames from the selected clip.
Run Depth Anything V2 Small locally through Transformers.js and a WebGPU inference path.
Normalize and refine the depth maps using source-image edges.
Blend neighboring depth samples at the current source time.
Render the original frame and the depth textures through a WebGL lighting shader.

WebGPU runs the model; WebGL renders the relighting effect. Keeping those responsibilities separate lets the expensive analysis be reused while lighting parameters remain interactive.

Turning a depth map into an approximate surface

A grayscale depth map tells us which visible regions are nearer or farther away. Differences between neighboring depth pixels provide an approximate surface orientation.

The shader estimates a normal from those depth gradients. In simplified GLSL:

vec3 normal = normalize(vec3(-dx * 0.12, -dy * 0.12, 1.0));
vec3 lightDirection = normalize(lightPosition - surface);
float diffuse = max(0.0, dot(normal, lightDirection));

The implementation combines that directional response with a spatial falloff controlled by the light's range and distance. Color and strength determine the added illumination; softness changes the response and the neighborhood used to estimate the surface.

It also approximates a rear-light rim using foreground depth discontinuities. A single camera view does not provide the hidden geometry behind the subject, so this is a screen-space silhouette estimate.

The shader adds a restrained fill to the existing image rather than replacing its original shading. It uses an approximate gamma-to-linear conversion before adding light, then converts the result back for display.

A sphere instead of mysterious coordinates

The inspector exposes a draggable light-position sphere, horizontal and vertical angles, distance, strength, color, range, and softness. Warm, neutral, and cool presets provide quick starting points.

Internally, the sphere's angles become a three-dimensional light position:

const azimuth = light.x * Math.PI;
const elevation = -light.y * Math.PI / 2;
const position = [
  Math.sin(azimuth) * Math.cos(elevation) * light.distance,
  Math.sin(elevation) * light.distance,
  Math.cos(azimuth) * Math.cos(elevation) * light.distance,
];

The effect also has a hold-to-view-original button. Comparison is transient preview state: it does not change the saved effect, create an undo entry, or disable lighting in an export.

The controls are localized in all 13 interface languages supported by the editor.

Video needs source-time alignment

A still image needs one depth result. Video needs the right result at the right source time, including after trimming or changing playback speed.

The analysis signature includes the source, trim information, duration, playback rate, speed curve, quality, and model revision. Changing the source-time mapping invalidates reuse of an incompatible analysis.

At playback time, neighboring depth maps are blended. This softens sample-to-sample steps, although it is not motion-compensated interpolation and cannot guarantee flicker-free results on every clip.

Depth refinement uses a small edge-guided smoothing pass. Source-image edges discourage mixing across boundaries, while a depth-difference gate helps preserve foreground/background separation.

The seek bug that mattered for editing

A lighting shader can be correct and still display the wrong video frame.

Seeking a video element is asynchronous. Updating the playhead time does not mean the target frame has finished decoding. Painting too early can leave the effect canvas showing the previous frame after a drag.

The preview now avoids painting while the source is seeking and refreshes after decoded-frame events. The main video also uses video-frame callbacks where available. That connects repainting to the actual media frame rather than relying only on a UI state update.

Preview and export both call the shared relighting renderer. They use the same lighting math, though resolution, decoding, and encoding can still affect the final appearance.

Try it on your own footage

Open Timeline Studio and add a video to the timeline.
Select the clip and open Effects → Video lighting.
Analyze the depth, then drag the light on the sphere.
Adjust strength and color first; use range, softness, and distance to refine the look.
Hold the original-comparison button to judge the change, then export when you are happy with it.

For this effect, footage and depth inference stay in the browser. The model files need to download, and the current analysis route requires WebGPU support. Speed depends on the browser, GPU, clip length, and analysis quality.

What this approach can and cannot do

This is an editable approximation based on visible scene depth. It does not reconstruct a complete 3D scene, recover material properties, remove baked-in shadows, or simulate physically accurate cast shadows and reflections.

Its practical role is adding a directional fill, a color accent, or a restrained edge-light impression to existing footage. Strong motion, difficult depth boundaries, and reflective or transparent surfaces can expose the approximation.

The implementation is open source. Start with the relighting renderer, depth rendering integration, and analysis hook.

This article was prepared with AI writing assistance and checked against the feature's source code.

Timeline Studio: From Approved Storyboard to Editable Video with Sanity

MartinDelophy — Sat, 26 Sep 2026 04:08:38 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange.

What I Built

A storyboard is useful only if the editor can turn it into an actual sequence of shots. I wanted to connect those two steps without making creators upload all their footage to a cloud service.

I develop and maintain Timeline Studio, an open-source browser video editor. For this challenge, I built a Sanity-powered storyboard workflow: write a shot plan, review it, approve a specific version, and turn that version into editable timeline clips using local media.

Sanity stores the shared creative decisions. The browser keeps the source footage and the edit.

The workflow is straightforward:

An author creates a brief and ordered shots, each with a duration, visual direction, optional narration draft, and asset hint.
A reviewer requests changes with a note or approves the saved version.
Approval creates a separate snapshot of that version.
In the editor, the creator chooses an approved storyboard and matches its shots to local images or videos.
The editor checks that approval is still current, then appends editable clips and shot annotations. One undo reverses the import.

The new workbench uses Astro with an interactive React/Sanity Studio custom tool. It shares the schema and workflow components with the editor integration. The editor retains its Vite build, and the two outputs are assembled into one static deployment. / opens the editor directly; /storyboard/ opens the workbench.

Demo

The demo is a 13-second editable sequence, with three shots lasting 4, 5, and 4 seconds. Its local media are screenshots of the editor. Open the downloaded .timeline file through the editor's Open project action to inspect the clips and annotations. This is a project archive, not an encoded video or a recording of the review process.

To exercise the cloud handoff, upload local media, open File → Storyboards, load the approved demo board, match each shot, and append it. Reading approved boards and editing locally do not require a Sanity login.

Writing and reviewing in the workbench require an account authorized for the selected Sanity project. The reviewer guide also explains how to use your own project and dataset through the connection form. No shared owner password is needed.

Code

Source code and feature branch

The main workflow lives in src/sanity/. The Astro entry at storyboard-site/pages/storyboard/index.astro mounts the shared workbench. Approval and timeline import each have one implementation, rather than separate copies for different frontends.

My Build Process

I built this integration with Codex, iterating on the data contract, interface, and behavior in the actual browser.

My initial instruction was to integrate Sanity on an independent branch. The useful follow-up constraints were concrete: keep media local, preserve the direct editor experience, import only approved content, and protect existing timeline edits. These are a summary of the working constraints, not a verbatim prompt transcript.

Make the handoff explicit

A cloud storyboard cannot assume that a creator's local file has a particular URL or duration. We made media matching a deliberate step. For video, a source in-point and shot duration must fit the source; the importer does not quietly loop or stretch footage.

The append operation uses the editor's existing ripple-editing behavior and undo system. Shot direction and narration become range-marker notes. They remain editable and do not extend the rendered media duration. Importing a narration draft does not automatically generate speech or captions.

Treat approval as a versioned decision

A status label alone was insufficient. A reviewer might approve one version while another browser still holds older changes.

We used revision-guarded writes and a separate approved snapshot. Conflicting saves preserve the user's local edits, and unsaved drafts can recover after reload. Revising an approved board removes it from new imports until it is approved again.

Verify the real integration

Isolated checks helped exercise revision conflicts and withdrawn approvals, but those checks did not establish that authentication and real cloud writes worked.

We then used my signed-in Chrome session to create, save, submit, request changes, recover an unsaved edit after reload, resubmit, and approve a real board. The editor fetched the actual approved snapshot, matched local assets, and created three clips and three annotations. The exported project reopened in a fresh browser.

The deployed preview was checked separately: the Astro workbench loaded the approved board, and anonymous cloud read → local import → undo passed without intercepting network responses.

Work through build issues

The first Astro configuration used publicDir: false, which the build rejected. A dedicated public directory fixed that. An Astro 7 build then exposed a cookie-module compatibility error. Pinning Astro 6.4.8 with its matching React integration produced a working build.

The combined production build and TypeScript check pass. Lint reports zero errors; the repository still has pre-existing warnings. The Sanity schema has also been deployed through the official CLI.

Sanity Project Details

Project ID: rgq98xsq
Dataset: production — public
Demo board: ts-board-19e4f5ec-b9be-4a0b-9a8d-8f577f929e12
Approved snapshot: ts-release-37ad30b2-04d4-4a10-a23a-652efb239341

The schema has two document types:

Document	Responsibility
`tsStoryboard`	Brief, language, ordered shots, review note, workflow status, actor/time, and approved-release reference
`tsStoryboardRelease`	Approved content snapshot, source revision, storyboard reference, and approval actor/time

Each shot has a stable key, duration, visual direction, narration draft, and asset hint. Approval creates the release and updates the board in one transaction guarded by ifRevisionId.

The editor reads through the non-CDN API and rechecks the approved snapshot immediately before appending. This establishes approval at fetch time; it is not a distributed lock against a later cloud revision. The imported annotations retain the release identity for provenance.

Design Choices and Limits

This uses ordinary Content Lake documents and an application-defined review workflow. It does not require paid workflow features, and it is a Studio custom tool rather than an App SDK application.

The dataset is public, including demo script drafts and review notes, so it contains only shareable material. Source images, videos, voice profiles, and local project archives are not uploaded by this feature. Official Sanity Studio handles authentication; no developer write token is embedded in the editor.

Approval is an application workflow, not an access-control barrier against a dataset administrator. Imported timelines are independent local edits: later storyboard changes do not silently rewrite a creator's project.

The integration's controls support 13 languages. The result is a small but complete bridge from a shared, reviewed shot plan to a local, editable video sequence.

Agent Session

I used Codex for implementation, debugging, validation, and assistance preparing this writeup. No public agent-session transcript is attached; the code and reviewer guide document the implementation and reproducible workflow.

Turning a README into a Playable Game Catalog

MartinDelophy — Wed, 09 Sep 2026 03:13:17 +0000

GitHub: awesome-gpt-6-astra
Live gallery: Astra Games

Building a game with AI leaves another problem to solve: helping people find it, try it, and understand how it was made.

I maintain Awesome GPT-6 Astra, a community collection of games and interactive projects. As the collection grew, a Markdown list became the content source for a visual gallery with screenshots and playable links.

This post explains a few design choices behind that workflow: structured entries, a README-driven catalog, real screenshots, and multilingual maintenance.

1. Give each entry a predictable structure

A collection becomes difficult to browse when every submission uses a different format. One project links to source code, another has a screenshot, and a third hides its demo inside a long development log.

We use a small, consistent entry format:

- Project name → playable demo
  - Creator → public profile
  - Platform → browser requirements, controls, access conditions
  - GPT-6 Astra → model contribution statement or development record
  - Resources → source code and setup instructions, when available
  - Preview → an actual gameplay screenshot

A demo answers “Can I try it?” Source code answers “Can I study or run it?” A development record answers “How was it built?” These fields serve different purposes.

Public source code is optional. A playable project can still be useful to explore, provided its creator is credited and the available information is clear.

2. Let the README supply the website's catalog

Maintaining a GitHub list and a separate frontend array creates duplicate work. A contributor can update the README while the gallery continues showing an old description—or misses the project entirely.

Our gallery reads the public upstream README and parses it into structured records:

README entries
      ↓
Parse titles, categories, authors, links, and screenshots
      ↓
Structured catalog records
      ↓
Gallery cards and playable links

Contributors can work with Markdown and pull requests without learning the frontend. The website handles presentation and periodically checks for catalog updates.

The tradeoff is that Markdown is now a content interface. Formatting conventions matter, and the parser needs to handle edits predictably.

3. Parse project entries, not every link

A README also contains navigation, language selectors, contribution instructions, licenses, and creator profiles. Treating every link as a game would produce a noisy catalog.

The parser needs to distinguish:

Project titles from navigation.
Demo destinations from source-code references.
Creator metadata from descriptions.
Games from interactive experiments.
Valid entries from incomplete or duplicate content.

The existing tests cover adding, deleting, and editing entries, duplicate handling, malformed documents, caching, and recovery from failed reads.

For a project like this, the useful guarantee is straightforward: a normal README contribution should produce the expected catalog update.

4. Make screenshots part of the entry

A short description rarely communicates how a game feels. An actual screenshot helps readers understand its visual style and interface before opening it.

We embed screenshots in the README entries so the gallery can use them as covers. For captured images stored in the repository, we also record their source, capture date, and the scope of the check.

That last detail matters. Entering the first level is not a full playthrough. A manual describing touch controls is not evidence that someone tested the game on a phone.

Keeping those distinctions explicit makes the catalog more useful to both readers and future maintainers. Screenshots should represent the running project, and third-party images retain their creators' rights.

5. Multilingual maintenance includes metadata

The collection has 12 README language versions. Translating a description is only one part of keeping them aligned.

For a new submission, we also check that every version includes the entry, uses the same demo and creator links, references an existing screenshot, and has consistent counts and maintenance dates.

Model attribution needs the same care. If one version says “awaiting creator confirmation,” another should not describe the contribution as independently verified.

The website's current catalog source is the default README. The translated READMEs are additional documentation that we maintain alongside it; translating them does not imply that every language is a separate runtime data source.

6. Describe AI involvement precisely

“Made with GPT-6 Astra” can describe very different development processes: implementing one mechanic, helping design an interface, iterating on game rules, or contributing across a larger project.

We aim to preserve those distinctions. Attribution follows creator or submitter statements, and missing confirmation is marked explicitly. Inclusion is not an independent model benchmark or an endorsement from OpenAI.

A useful submission explains:

What the player actually does.
Which parts Astra helped build.
What required iteration or manual changes.
What limitations remain in the current version.

These details give other developers something to learn from beyond a finished screenshot.

Share something people can play

The goal of this project is to give creators a place to show working games and connect their demos with useful development context.

Playable prototypes are welcome. If you have built a game with Astra, you can submit it through the repository with a demo, creator credit, a real screenshot, and a description of the model's contribution.

Disclosure: This article was drafted and translated with AI assistance from the maintainer's project notes and repository context.

Making a React Video Editor Agent-Friendly with Timeline Markers and MCP

MartinDelophy — Tue, 08 Sep 2026 06:51:06 +0000

A video editor needs to remember more than cuts. It also needs to remember why a cut matters.

“Start the second chapter here.” “Review this section before publishing.” “Leave a little more room at the end.” These instructions become much easier to act on when their timestamps and context live inside the project.

I recently added persistent timeline markers to Timeline Studio, the open-source browser video editor I maintain, and exposed them through its CLI, MCP adapter, and Agent Skill. The interesting engineering work was making those annotations precise, inspectable, and safe to edit in a batch.

You can try the editor in your browser or browse the source on GitHub. The implementation discussed here is included in Skill v1.0.7. Timeline Studio uses React and Vite and follows a local-first approach.

Give annotations their own data model

The editor supports four annotation types:

Type	Purpose	Timing
`marker`	A beat cue, action, or cut reference	`time`
`chapter`	The beginning of a named section	`time`
`range`	An interval to review or work on	`time`, `endTime`
`note`	Feedback tied to a precise moment	`time`

A review range looks like this:

{
  "id": "product-review",
  "type": "range",
  "time": 5,
  "endTime": 8,
  "title": "Product reveal",
  "notes": "Check whether the caption appears with the product.",
  "color": "violet"
}

Times are absolute project seconds. Titles and notes preserve Unicode, and annotations are serialized in timelineMarkers inside the portable .timeline project.

One boundary is especially useful: annotations do not contribute to rendered media duration. Adding a planning marker at 90 seconds to a 60-second edit must not produce another 30 seconds of video.

Chapter markers also do not automatically become on-screen titles or MP4 chapters. They remain editable project annotations. A marker-only task can therefore finish by writing a new project without re-encoding unchanged media.

The tradeoff is explicit timing: annotations stay at their absolute project positions during trims, reordering, and ripple edits. If they need to follow moved content, the edit plan must update them too.

Keep the React interaction compact

An always-visible annotation lane takes space away from media tracks. The default view now merges compact flags into the ruler. A toolbar chevron expands a separate lane with titles and range spans when more detail is useful.

Pressing M adds a marker at the playhead; Shift + M opens the manager. The marker UI is localized in all 13 supported interface languages.

During a drag, React holds a temporary preview. Releasing the pointer commits the edit; Escape or a canceled gesture discards the preview. This separates continuous pointer feedback from the final project mutation.

Measure snapping in pixels, then convert to time

A fixed temporal threshold such as 0.2 seconds feels very different at different zoom levels. The implementation converts a 10-pixel tolerance into project seconds:

const thresholdSeconds =
  10 / railWidth * timelineDuration;

Here, railWidth is the timeline rail width and timelineDuration is its corresponding time span. This keeps the screen-space snapping distance approximately consistent as the timeline zoom changes.

Markers reuse the shared snapping targets: the playhead, clip boundaries, and other marker edges. The editor shows a common alignment guide, and holding Alt temporarily bypasses snapping.

Ranges need an additional invariant: moving a range must preserve its length. If the end of a three-second range snaps to 12 seconds, its start must become 9 seconds.

The core calculation, simplified, is:

const nextStart = movingEdge === "end"
  ? targetTime - duration
  : targetTime;
const nextEnd = nextStart + duration;

The actual implementation checks both edges and chooses the nearer valid snap candidate. Resizing an edge is handled separately from moving the whole range.

Expose one command engine through CLI and MCP

The agent-facing architecture is small:

Agent Skill: workflow, timing evidence, verification
                         |
CLI / MCP: structured operation entry points
                         |
Shared command engine: validation, edits, semantic diff
                         |
A new .timeline project

The MCP adapter invokes the existing CLI runner. It does not maintain its own marker reducers. Fixing a validation rule therefore fixes both entry points.

From the repository root, an agent can inspect the project and its annotations:

npm run agent -- project.inspect /projects/input.timeline
npm run agent -- marker.inspect /projects/input.timeline

MCP exposes the read-only timeline_marker_inspect tool. Writes use marker.add, marker.update, and marker.delete through the existing project diff/apply workflow.

For example, suppose inspection reports revision 0 and a product-review range spanning 5–8 seconds. This plan moves it to 9 seconds and adds a note at 11 seconds:

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 0,
  "operations": [
    {
      "id": "move-product-v1",
      "type": "marker.update",
      "markerId": "product-review",
      "time": 9
    },
    {
      "id": "add-ending-note-v1",
      "type": "marker.add",
      "markerId": "ending-note",
      "markerType": "note",
      "time": 11,
      "title": "Ending rhythm",
      "notes": "Reviewer request: leave a little more breathing room.",
      "color": "rose"
    }
  ],
  "output": { "project": "/projects/output-marked.timeline" }
}

The paths, revision, IDs, and review request are illustrative. A real plan must use inspected values and actual feedback.

Two distinctions matter: id identifies an operation for retry handling, while markerId identifies the annotation. Likewise, type selects the command, while markerType selects the annotation kind.

Updating only the time of a range preserves its length. The example moves 5–8 seconds to 9–12 seconds. An explicit endTime lets the plan change the end instead.

Save the plan as /projects/markers-plan.json, validate its structure, and preview the semantic diff:

node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs /projects/markers-plan.json
npm run agent -- project.diff /projects/markers-plan.json

After reviewing the result, apply the same plan and inspect the output:

npm run agent -- project.run /projects/markers-plan.json
npm run agent -- marker.inspect /projects/output-marked.timeline

The diff reports additions, removals, and modifications under changes.markers, including before/after values. A marker-only edit should not unexpectedly modify a media track.

The Skill instructs the agent to inspect, review the diff, apply, and inspect again. The engine enforces concrete constraints:

A mismatched revision returns REVISION_CONFLICT.
Previously applied operation IDs are not applied again.
A failed operation rejects the batch without writing a partially edited archive.
Output must go to a new path; input files and existing outputs cannot be overwritten.

A revision conflict requires a fresh inspection and a reconsidered plan. Merely replacing the revision number would skip the reason for having that check.

The tricky bugs were identity and floating-point boundaries

Consider imported annotations with three identical IDs: x, x, and x. Inspection normalizes them to x, x-2, and x-3.

If normalization runs after every individual operation, deleting the first annotation can change the identities of the remaining entries. A subsequent update to x-2 may then hit the wrong annotation.

The fix was to normalize once on the transaction copy before marker writes, then resolve every operation against those fixed IDs. Duplicate-ID suffixes also reserve space within the 160-character limit so generated IDs remain valid.

The second bug involved a one-millisecond range near 1,000 seconds. Subtracting its endpoints can produce a value slightly below 0.001. A strict comparison can reject a valid move toward zero. The implementation now accounts for floating-point error at the minimum range boundary while continuing to reject genuinely invalid intervals.

Both bugs were easy to miss when testing a single ordinary drag. Batch operations and imported data made them visible.

An agent still needs evidence for its timestamps

UI snapping is a pointer interaction. CLI and MCP commands take exact seconds; they do not apply pixel-distance snapping.

For an event in source footage at constant playback speed, the mapping is:

projectTime = clipStart
            + (sourceTime - sourceStart) / playbackRate

With a speed curve, the corresponding source-time mapping is required. Dividing by an average speed can place the cue incorrectly.

Music cues have a similar boundary. The marker commands store annotations; they do not perform automatic beat detection. An agent needs supplied cue times or a verified analysis before writing a beat grid.

What was verified

The checks covered real CLI and MCP calls, plus an independent agent following the Skill to move a range from 5–8 to 9–12 seconds and add a Unicode note at 11 seconds.

Verification checked the range length, unrelated annotations, the original archive hash, embedded media bytes, media duration, and the unchanged rendering plan for annotation-only edits. Skill validation, type checking, the production build, and GitHub CI passed. Existing lint warnings remain.

The result is a project that can carry editorial intent between people and agents. A human can leave a review note, an agent can act on its exact location, and the next person can inspect the resulting project and its annotations.

If you want to try the flags and range snapping, open the Timeline Studio editor. If you are building a React editor or exposing a creative tool through MCP, the GitHub repository and marker workflow reference contain the implementation and command contract. Stars help you follow updates; reproducible issues and design feedback are welcome.

To install Skill v1.0.7 for Codex:

gh skill install MartinDelophy/ai-video-editor edit-timeline-studio --pin v1.0.7 --agent codex --scope user

The Skill installs the workflow. Running local commands still requires a Timeline Studio checkout with its Node dependencies, and the MCP adapter should run from that checkout. See the release notes for the published version.