Project links
GitHub: https://github.com/MartinDelophy/ai-video-editor
Live demo: https://video-editor.ai-creator.top/
Timeline Studio is a local-first AI video editor that runs in the browser. I recently added an experimental Optical Flow Tracking capability to it.
The goal was not to cover the frame with impressive-looking arrows. The feature first detects a person or object, calculates optical flow only inside that semantic region, aggregates the local vectors into a motion cohort, and accumulates the cohort into a visible trajectory.
When processing finishes, the browser renders a reusable WebM result and adds it to My assets. The generated result is a real editing asset rather than a temporary analysis preview.
Why raw optical flow was not enough
Optical flow describes pixel displacement between adjacent frames:
v = (deltaX, deltaY)
Drawing every vector gives us a familiar motion-field visualization, but those vectors do not understand the scene.
They may come from:
- camera shake;
- moving background textures;
- subtitle changes;
- lighting variation;
- compression artifacts;
- hair and clothing deformation.
Clustering all of those vectors can easily turn background texture into a supposed “motion group.”
So I changed the question from:
Which pixels moved?
To:
Which person or object moved, in which direction, and along what trajectory?
The processing pipeline
The browser-local pipeline looks like this:
Selected video clip
↓
Detect people and objects
↓
Create large semantic regions
↓
Calculate local optical flow inside each ROI
↓
Aggregate vectors into motion cohorts
↓
Accumulate trajectories over time
↓
Render on clear source frames
↓
Encode a WebM result
↓
Add the result to My assets
Frame extraction, model inference, flow calculation, Canvas rendering, and video encoding all happen in the browser. The source video does not need to be uploaded to an analysis server.
Detecting the subject before calculating flow
The first sampled frame acts as the semantic anchor. The implementation uses bounded local fallbacks:
- NanoDet for lightweight person and object proposals;
- MediaPipe segmentation to confirm a person region;
- YOLOS as another detection fallback;
- a conservative centered semantic prior when the model runtimes fail on an otherwise valid video.
A “group” does not require multiple people. One clearly visible person or one main object is already a valid semantic motion cohort.
Close-up footage also needs special handling. A person may occupy almost the complete frame, so rejecting every large detection box would incorrectly fail on normal portrait composition.
Representing a semantic motion cohort
Each detected subject becomes a large region that can be propagated through time.
const cohort = {
id: "cohort-1",
label: "person",
box: {
xmin: 0.2,
ymin: 0.1,
xmax: 0.8,
ymax: 0.95
},
center: {
x: 0.5,
y: 0.525
},
dx: 0,
dy: 0,
confidence: 0,
stability: 0
};
The box is normalized to the 0–1 range. This makes it possible to calculate motion on a small analysis frame and later map the result onto a higher-resolution rendering canvas.
Nearby people may be merged into a larger cohort when their regions and motion agree. A single principal object remains its own cohort.
Calculating flow inside the semantic ROI
The sampled video frames are converted to grayscale. A local block-matching pass searches for the best displacement between the current frame and the next one.
For a candidate displacement, the matching error can be described as:
E(dx, dy) = average(abs(frameA(pixel) - frameB(pixel + displacement)))
The displacement with the lowest error becomes a candidate motion vector.
Not every candidate is accepted. The implementation rejects vectors that are:
- too small to represent meaningful motion;
- too uncertain;
- outside the semantic ROI;
- too far from the cohort’s dominant direction;
- produced by a poor block match.
The important rule is that background vectors cannot create a person or object cohort on their own.
Aggregating local vectors into one group direction
Inside a semantic region, the accepted local vectors are combined using their confidence as a weight:
cohortVector = sum(confidence[i] * vector[i]) / sum(confidence[i])
A simplified implementation looks like this:
function aggregateVectors(vectors) {
let dx = 0;
let dy = 0;
let totalWeight = 0;
for (const vector of vectors) {
const weight = vector.confidence;
dx += vector.dx * weight;
dy += vector.dy * weight;
totalWeight += weight;
}
if (!totalWeight) {
return { dx: 0, dy: 0 };
}
return {
dx: dx / totalWeight,
dy: dy / totalWeight
};
}
I also compare how consistently the local vectors agree with the group direction. That becomes the cohort’s stability score.
The UI exposes several useful diagnostics:
- valid vector count;
- motion cohort count;
- dominant direction;
- trajectory stability;
- active detector;
- processed frame count;
- analysis and encoding progress.
These metrics make the experiment easier to understand and help identify weak tracks.
Turning frame-by-frame motion into a trajectory
One optical-flow step only explains the motion at that instant. To show movement from point A to point B, the cohort center is updated after each sampled frame.
const nextPoint = {
x: previousPoint.x + cohortVector.dx,
y: previousPoint.y + cohortVector.dy,
time: frameTime
};
track.points.push(nextPoint);
The visible trail is a configurable window over the accumulated points:
const visiblePoints = track.points.slice(-trailLength);
This supports short motion tails as well as longer experiment-style trajectories.
Rendering the visualization
The result Canvas contains:
- the original video frame;
- a subtle analysis grid;
- cyan local-flow arrows;
- unlabeled cohort bounds;
- distinct accumulated trails;
- the main cohort direction;
- an experiment timecode in the upper-right corner.
I intentionally removed the text label from the cohort boundary. On close-up footage, a “Person · 1” label covered the subject and made the generated video feel more like a debug screen than a finished experiment.
The cohort count and detector information still appear in the inspector, where they do not obscure the image.
A simplified arrow renderer uses the Canvas 2D API:
function drawArrow(ctx, startX, startY, endX, endY, color) {
const angle = Math.atan2(endY - startY, endX - startX);
const head = 5;
ctx.strokeStyle = color;
ctx.fillStyle = color;
ctx.beginPath();
ctx.moveTo(startX, startY);
ctx.lineTo(endX, endY);
ctx.stroke();
ctx.beginPath();
ctx.moveTo(endX, endY);
ctx.lineTo(
endX - head * Math.cos(angle - Math.PI / 6),
endY - head * Math.sin(angle - Math.PI / 6)
);
ctx.lineTo(
endX - head * Math.cos(angle + Math.PI / 6),
endY - head * Math.sin(angle + Math.PI / 6)
);
ctx.closePath();
ctx.fill();
}
Adding the timecode
A clip-relative timecode is burned into the upper-right corner of each generated frame.
const timecode = formatTime(frame.time);
ctx.font = "700 16px ui-monospace, monospace";
const padding = 10;
const width = ctx.measureText(timecode).width + padding * 2;
const x = canvas.width - width - 16;
const y = 16;
ctx.fillStyle = "rgba(3, 10, 13, 0.78)";
ctx.fillRect(x, y, width, 34);
ctx.fillStyle = "#55f3e1";
ctx.textAlign = "center";
ctx.textBaseline = "middle";
ctx.fillText(timecode, x + width / 2, y + 17);
Besides improving readability, this gives the output the feel of a completed motion-analysis experiment.
Low-resolution analysis, high-resolution rendering
The first version stored the 192-pixel-wide analysis frames and enlarged them during result generation. The motion vectors looked correct, but the underlying video became visibly blurry.
The fix was to separate the analysis and rendering paths.
Analysis path
- resize the frame to a small working resolution;
- convert it to grayscale;
- calculate local flow;
- update cohorts and trajectories.
Rendering path
- retain clear samples from the source video;
- cap the render width at 1280 pixels;
- map analysis coordinates to the render Canvas;
- redraw vectors, bounds, trails, and timecode;
- encode the high-resolution result.
The coordinate mapping is straightforward:
const scaleX = renderWidth / analysisWidth;
const scaleY = renderHeight / analysisHeight;
const renderX = vector.x * scaleX;
const renderY = vector.y * scaleY;
This keeps the optical-flow calculation inexpensive without sacrificing the clarity of the generated asset.
Encoding the result with WebCodecs
After analysis, every sampled moment is rendered to a Canvas frame. The frames are then encoded as WebM using WebCodecs.
const resultBlob = await encodeFrames(
renderedFrameBlobs,
renderWidth,
renderHeight,
sampleRate,
keyframeTimes,
duration,
{
signal: abortController.signal,
onProgress(value) {
updateProgress(92 + Math.round(value * 8));
}
}
);
The same abort signal is checked during frame rendering and encoding, so cancellation stops further processing and releases the encoder resources.
Generating real timeline frames
A generated video should not become a blank rectangle when it is placed on the timeline. Compact timeline frames are therefore created from the rendered output frames.
const trackFrames = await createVideoTrackFramesFromBlobs(
renderedFrameBlobs,
{
duration,
width: renderWidth,
height: renderHeight,
signal
}
);
The asset stores a matching track-frame duration:
const asset = {
id: crypto.randomUUID(),
type: "video",
src: URL.createObjectURL(resultBlob),
blob: resultBlob,
duration,
width: renderWidth,
height: renderHeight,
trackFrames,
trackFrameDuration: duration,
generatedBy: "optical-flow-tracking",
diagnostics: {
detector,
sampleRate,
vectors: summary.vectorCount,
cohorts: summary.cohortCount,
dominantAngle: summary.dominantAngle,
stability: summary.stability
}
};
Adding the result to My assets
When encoding finishes, the result is added to the media library and selected automatically.
setUserAssets(current => [asset, ...current]);
setSelectedLibraryAssetId(asset.id);
setActiveTool("media");
setMediaTab("mine");
It is deliberately not inserted into the timeline automatically. The user can inspect the result first, then decide whether it belongs on the main visual track or as an overlay.
A small browser benchmark
I tested the pipeline with a two-second, 852×480 video at a 4 fps sampling rate.
The result contained:
- 9 sampled analysis frames;
- 268 valid motion vectors;
- 1 semantic motion cohort;
- a dominant direction of roughly 235 degrees;
- a trajectory stability score of roughly 78%;
- an 852×480 WebM output.
Because low-resolution analysis and clear-frame rendering are separated, the result preserves the source dimensions instead of enlarging the analysis frames.
Benefits and limitations of browser-local processing
The complete pipeline runs locally:
- video frame extraction;
- person and object detection;
- person-region estimation;
- optical-flow calculation;
- cohort tracking;
- Canvas rendering;
- WebM encoding;
- timeline thumbnail generation.
This provides several advantages:
- the source video does not need to be uploaded;
- there is no backend inference queue;
- parameter changes can be tested immediately;
- the result becomes an editing asset right away;
- server-side video storage and processing costs are reduced.
There are also real limitations. Long clips, high sampling rates, and 4K sources increase memory use and processing time. Heavy motion blur, long occlusion, and scene cuts can reduce tracking stability. Browser support for WebCodecs and local model runtimes also varies.
Conclusion
The most important change was not a more complicated arrow renderer. It was giving optical flow a semantic boundary and a complete product workflow.
Person/object detection
+
Semantic ROI
+
Local optical flow
+
Cohort vector aggregation
+
Trajectory accumulation
+
High-resolution Canvas rendering
+
WebCodecs encoding
+
Reusable media asset
Three decisions made the largest difference:
- Do not create people or object groups from background vectors.
- Separate low-resolution analysis from high-resolution output rendering.
- Treat analysis as complete only after it becomes a reusable video asset.
The result feels less like a raw computer-vision demo and more like a finished browser-based motion experiment that can continue through the normal editing workflow.
If you are interested in browser AI, WebCodecs, video editing, or computer vision, take a look at the repository:
Top comments (0)