The Timeline Was Split, but the Audio Wasn't: Fixing Source Offsets in a Browser Video Editor

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

Timeline Studio is a local-first AI video editor that runs in the browser. It combines a multi-track timeline with captions, AI voiceovers, music, visual transforms, and deterministic offline export.

Like most editors, it uses non-destructive editing: splitting a clip should not rewrite the underlying media. Each timeline segment instead stores where it appears in the project and which range of the original source it represents.

That distinction exposed a subtle but very audible bug.

Bug Fix or Performance Improvement

Voiceover clips could be split at the playhead, and the UI correctly produced two timeline segments. Their positions, durations, and waveform slices looked right. However, playback of the second segment started from the beginning of the original audio.

For example, splitting a four-second source after 1.5 seconds should create these views of the same source:

first segment:  timeline 2.0s -> 3.5s, source 1.0s -> 2.5s
second segment: timeline 3.5s -> 6.0s, source 2.5s -> 5.0s

The old playback path effectively calculated only this:

audio.currentTime = timelineTime - segment.start;

That is correct for an unsplit clip whose source starts at zero. It is wrong for the second half of a non-destructively split clip. The result was especially confusing because the timeline geometry and waveform suggested a clean edit while the user heard repeated audio.

The same data-model mismatch also affected explicit seeking and offline export. Fixing only real-time playback would have left the exported file different from the editor preview.

Code

The fix is available in commit 8287b90.

The core split logic now preserves a source offset for the second segment:

const firstDuration = time - source.start;
const secondDuration = source.duration - firstDuration;

const first = {
  ...source,
  duration: firstDuration,
  peaks: source.peaks.slice(0, peakSplit),
};

const second = {
  ...source,
  start: time,
  duration: secondDuration,
  sourceStart: (source.sourceStart || 0) + firstDuration,
  peaks: source.peaks.slice(peakSplit),
};

Every consumer then resolves source time with the same rule:

const sourceTime = (segment.sourceStart || 0)
  + getTimelineTrackLocalTime(timelineTime, segment.start, segment.duration);

Offline mixing receives the same offset and trimmed duration:

{
  sourceOffset: Math.max(0, item.sourceStart || 0),
  sourceDuration: Math.max(0, item.duration || 0),
  playbackRate: 1,
}

My Improvements

1. I made the timeline segment model explicit

A timeline clip now carries two different coordinates:

• start: where the segment begins in the edited project
• sourceStart: where its content begins in the original audio

Keeping these concepts separate is the foundation of non-destructive editing. A split changes timeline ranges and source views, not the original Blob.

2. I fixed every playback path, not only the visible symptom

The source offset is now honored when:

• playback crosses into a segment;
• the user seeks with the playhead;
• the media synchronization effect corrects drift;
• the deterministic offline audio mixer prepares the export;
• the compatibility export path schedules decoded audio buffers.

This keeps preview and export behavior consistent.

3. I preserved waveform continuity

The waveform peak array is split at the proportional position of the edit. The first segment keeps the peaks before the cut and the second keeps the remaining peaks. This makes the visual representation agree with the source range users actually hear.

4. I preserved linked-caption behavior

When a voiceover is linked to a caption, the original caption stays linked to the first segment and its end is clamped to the split time. The new audio segment receives its own identity, preventing ambiguous links between one caption and two independently editable clips.

5. I added a regression test for the complete invariant

The test starts with a segment that already has a non-zero sourceStart, splits it at the playhead, and verifies:

• both timeline ranges;
• the accumulated source offset of the second segment;
• both waveform slices;
• fade-boundary cleanup;
• caption relinking;
• selection of the newly created segment;
• safe replacement of object URLs.

expect(audioSegments[0]).toMatchObject({
  start: 2,
  duration: 1.5,
  sourceStart: 1,
  peaks: [0.1, 0.2],
});

expect(audioSegments[1]).toMatchObject({
  start: 3.5,
  duration: 2.5,
  sourceStart: 2.5,
  peaks: [0.3, 0.4],
});

The important part is sourceStart: 2.5: the original offset of 1 second plus the 1.5 seconds retained by the first piece.

What I Learned

Media editors often display one timeline while internally managing several clocks: project time, clip-local time, source-media time, and sometimes playback-rate-adjusted time. Bugs appear when a shortcut that is valid for an unsplit clip is reused after editing creates a non-zero source offset.

The most useful invariant for this fix was:

source time = source offset + clip-local timeline time

Once that rule was encoded consistently across playback, seeking, synchronization, and export, the editor stopped merely looking correct and began producing the correct audio as well.

Comments

[Siyu]

The "fix every consumer, not just the visible symptom" discipline is what separates a real fix from a patch. Most people would have corrected the playback handler, shipped it, and then discovered the export was still wrong three weeks later.The coordinate framing is useful beyond media editors. Any system where the same underlying entity gets viewed through different lenses runs into this. The shortcut that works in one context silently breaks in another, and the UI looks correct the whole time.I applied the same principle in Opportunity Skill with perspective isolation. A person can be both hiring and offering services, and mixing those signals in one data path contaminates matching on both sides. The separation lives at the model level, not as a query-time filter. Same lesson. Encode the invariant where the data is defined, not where it happens to be consumed.Nice regression test, by the way. Testing the accumulated offset across a re-split is exactly the invariant that would silently regress.

[MartinDelophy]

Exactly—that’s the lesson I’m taking from this as well. Once an entity can be interpreted through multiple coordinate systems or perspectives, the distinction has to be represented in the model rather than reconstructed by individual consumers.
Your Opportunity Skill example is a great parallel: hiring and offering services may belong to the same person, but they’re different perspectives with different matching semantics. Keeping that isolation at the model level prevents every downstream query from having to rediscover the distinction.
And thank you—the re-split test felt important because it validates the invariant across composition, not just the immediate split result. That should make future changes to playback, export, or rate handling much safer.

[Wren Calloway]

The invariant you landed on is correct, but there's a fourth clock hiding in playbackRate: 1 that will re-break this the moment you support speed changes. The rule source time = source offset + clip-local timeline time only holds when 1 second of timeline equals 1 second of source. As soon as a clip plays at 1.5x, clip-local timeline time and source time diverge, and your split math has to divide firstDuration by the rate before adding it to sourceStart — otherwise the second segment's offset drifts by exactly the accumulated speed factor.

Worth pinning in the regression test now while it's cheap: add a case with playbackRate !== 1 and assert the second segment's sourceStart, even if the editor doesn't expose rate yet. The bug you just fixed is the class of thing that comes back through a new axis, and the test you wrote asserts the whole invariant except the one dimension you hardcoded to 1. That hardcoded literal is the tell.

[MartinDelophy]

Good catch on the missing rate dimension. I agree that the regression test should cover playbackRate !== 1, and I’ll add that case now.
One clarification on the math: if playbackRate follows Web Audio semantics, the second segment’s offset should be sourceStart + firstDuration * playbackRate, because one second of timeline at 1.5x consumes 1.5 seconds of source audio. Division would apply only if our stored rate represents the inverse duration scale. I’ll confirm the project’s convention and encode it explicitly in the test so the invariant can’t become ambiguous later.

[Nazar Boyko]

The invariant grows a term once clips can change speed: source time = offset + local time * rate. I'm guessing that's why the export path pins playbackRate to 1 for now?

[MartinDelophy]

Yep, once clips can change speed, the time mapping becomes offset + local time * rate instead of a simple linear offset. So pinning playbackRate to 1 in the export path makes sense as a temporary simplification to keep the export logic and synchronization stable.

[Frank]

Interesting problem! Did you find yourself needing to re-create AudioBufferSourceNode instances for each

[MartinDelophy]

Yes. Since an AudioBufferSourceNode is single-use—after start() is called, it can’t be started again—I create a new instance for each playback. The decoded AudioBuffer itself is reused, so there’s no need to decode the audio again

[Wren Calloway]

@martindelophy You're right, and I had the direction of the multiply backwards. If playbackRate is Web Audio semantics — rate as a source-consumption multiplier — then 1.5x eats 1.5s of source per timeline second, so it's sourceStart + firstDuration * playbackRate. My "divide" only holds if the stored field is a timeline-stretch factor (source seconds per timeline second's inverse), which is the opposite convention. Good catch back at me.

Which is exactly why your last line is the important one: pin the convention in the test, not just the case. The math is trivial once you know whether the number means "source per timeline" or "timeline per source" — the bug is that nobody wrote that down, so the same firstDuration and rate produce two defensible answers. Assert the direction with a comment that names the semantics, and the ambiguity can't quietly flip on the next person who reads the field name and guesses.