Keeping a browser video and a locally processed audio track in sync — and the 0.2s leak that broke it

We build [HaramLite](https://haramlite.com), a desktop app that removes music and
instrumentals from video and audio on the user's own machine — no uploads, no
accounts, no cloud. This is the story of a sync bug in its browser side, because
the fix is a nice illustration of a class of bug that is easy to ship and hard to
see.

## The setup

The browser extension never re-encodes anything. When you watch a YouTube video
with the music removed, it does two things:

1. mutes the page video element, and
2. plays a second `<audio>` element — the file the desktop app produced — and
   keeps the two in step.

The audio file is not the same length as the video, because the stretches that
held only music were *cut out* of it. So the timeline is compressed, and we keep
a map of the ranges that survived: `kept = [[0,10],[12,20]]` reads as "seconds
0–10 and 12–20 of the video exist in the audio".

Whenever the video enters a stretch that is missing from the audio, we jump the
**picture** forward over it:

js
const target = skipVideoGaps(video.currentTime, kept); // full timeline
if (Math.abs(target - video.currentTime) > 0.15) {
video.currentTime = target;
// ...and the audio has to follow, or it keeps playing content that
// belongs after the skip.
}

## The bug

The audio was re-anchored **only** when a hold flag was set, which happened only
when a seek had landed inside a removed stretch. On an ordinary jump the picture
moved and the sound did not — for about two tenths of a second. The user heard
the first word of the next sentence twice: once from the tail of the stretch that
should have been skipped, then again when the audio finally landed in the right
place.

Why did nothing correct it? There *is* a periodic drift corrector. It runs every
second and fixes the audio when it has drifted more than **0.35s**. The leak was
**0.20s** — comfortably inside the tolerance, so it was never corrected. It only
snapped back once leaks accumulated past the threshold.

## Two details that matter in the fix

**Compute from the value you just seeked to, not from the element.** A media
seek is asynchronous; reading `video.currentTime` immediately after assigning it
is unreliable. The old code re-derived the audio position from
`video.currentTime`, which could read stale. The fix maps the *target* instead:

js
const reanchorAudio = (fullT) => {
const want = clamp(mapFullToCut(fullT, kept), 0, audio.duration - 0.05);
if (Math.abs(audio.currentTime - want) <= 0.05) return;
// mute across the seek so the fragment is inaudible, unmute on 'seeked'
audio.muted = true;
audio.currentTime = want;
audio.addEventListener('seeked', unmute, { once: true });
setTimeout(unmute, 120); // fallback if 'seeked' never arrives
};

**Keep the wide tolerance for the periodic corrector, tighten only at the jump.**
The 0.35s window exists so the corrector does not fight the player's own seeks.
Loosening or removing it globally trades one bug for another. The jump path uses
its own 0.05s threshold, because there we know exactly where the audio should be.

## Prove it without a browser

The mapping functions are pure, so the fix can be demonstrated arithmetically —
no autoplay policies, no headless quirks, no flaky test. Extracting the real
functions from the shipped file and running the case above:

plaintext
words 0-10, removed 10-12, words 12-20
audio drifted to 10.20s during the gap
expected after the jump 10.00s
error 0.20s < 0.35s tolerance => never corrected
new re-anchor threshold 0.05s => corrects 10.20 to 10.00

Sixteen assertions, including the boundary cases: no map means no change, a
0.1s jump does not trigger the skip, past the last range the video stays put.

## Three things I would tell my past self

1. **A tolerance window silently hides every error below it.** If your corrector
   has a threshold, ask what *small persistent* error it permits — ours was 0.2s
   of audible audio belonging to the wrong moment.
2. **Media elements are asynchronous.** Never derive a new position by reading
   an element you have just written to.
3. **Extract the pure logic and test it.** The interesting part of a media bug is
   usually the arithmetic around it, and that part needs no browser at all.

The app is open source (Tauri v2 + Rust on the desktop side, ONNX Runtime with
UVR-MDX-NET for the separation itself), and the extension is plain JavaScript:

- Project and source: https://github.com/SMSMy/HaramLite
- Site and guides: https://haramlite.com

If you have shipped a similar sync problem — subtitles, karaoke, dubbed audio,
anything where two media elements have to agree on where "now" is — I would like
to hear how you handled the drift window.

Comments

[Raknaos]

The 0.20s leak sitting under a 0.35s tolerance is the whole story here, and I think it generalises further than sync code: any periodic corrector has a blind spot exactly the size of its own threshold, and that spot is where the bug lives. Your fix of tightening only the jump path instead of the global window is the right call — I've seen the opposite (lowering the corrector to 0.1s) and it just makes the corrector fight the player's own seeks, which is a different bug wearing the same clothes.

The detail I'd steal even if I never touch media: re-anchoring from the target you just seeked to rather than re-reading video.currentTime. Media seeks are async, so video.currentTime right after assignment can read the old position — deriving state from the element you just commanded instead of the command itself is how you get regressions that only appear under load. Does the mapping function have a pure/testable form, or is it entangled with the element APIs? Curious how you prove it without a browser, since that's usually the part that makes a fix like this hard to keep fixed.