DEV Community: Neural Sound

Why Millisecond Timing Errors Can Ruin a Music Separation Benchmark

Neural Sound — Wed, 05 Aug 2026 18:00:10 +0000

Disclosure: This article is based on a benchmark conducted by the NeuralSound team. AI was used to help edit and structure the draft; the code, methodology, and technical claims were reviewed by the team before publication.

A music source-separation model can sound good and still receive a surprisingly poor score.

One common reason is not the model itself. It is timing. NeuralSound

A cloud-based AI vocal remover may add a short delay, trim a few samples, pad silence, or export a file that is slightly shorter than the original. To a listener, a 10–30 ms offset may sound almost irrelevant. To sample-by-sample metrics such as SI-SDR, that same offset can create a large penalty.

This tutorial explains how to detect and correct timing offsets before evaluating an AI stem splitter, vocal remover, acapella extractor, or background music remover.

The examples come from our public AI vocal-remover benchmark, where the same five songs were processed by NeuralSound, Moises, and Fadr.

Play the full benchmark with 35 audio samples

Why timing matters

Suppose the reference vocal is:

reference[n]

and the estimated vocal is identical except that it starts 1,000 samples later:

estimate[n] = reference[n - 1000]

At 44.1 kHz, 1,000 samples are only about 22.7 ms.

The audio may still sound almost identical. But a metric comparing reference[n] with estimate[n] sees different values at nearly every sample.

That creates a false conclusion:

“The separator is inaccurate.”

The more accurate conclusion is:

“The separator output and reference are not synchronized.”

Core rule: synchronize first, score second.

What should be normalized before scoring?

A fair audio benchmark should make these conditions consistent:

sample rate;
channel layout;
start time;
duration;
stem definition;
output format;
quality tier.

For our two-stem benchmark, every evaluated output was synchronized at 44.1 kHz before scoring. The reference vocal came from the dataset’s ground-truth vocal stem, and the instrumental reference was calculated from the mixture minus vocals.

Step 1: Resample and convert to mono

For a controlled experiment, every waveform should use the same sample rate.

from __future__ import annotations

from math import gcd

import numpy as np
from scipy import signal


def to_mono(audio: np.ndarray) -> np.ndarray:
    """Convert mono or channel-last audio to one float64 waveform."""

    audio = np.asarray(audio, dtype=np.float64)

    if audio.ndim == 1:
        return audio

    if audio.ndim == 2:
        return audio.mean(axis=1)

    raise ValueError("Expected mono audio or channel-last stereo audio")


def resample_audio(
    audio: np.ndarray,
    source_rate: int,
    target_rate: int,
) -> np.ndarray:
    """Resample with a polyphase filter."""

    if source_rate <= 0 or target_rate <= 0:
        raise ValueError("Sample rates must be positive")

    if source_rate == target_rate:
        return np.asarray(audio, dtype=np.float64)

    factor = gcd(source_rate, target_rate)
    up = target_rate // factor
    down = source_rate // factor

    return signal.resample_poly(audio, up=up, down=down)

For a production benchmark, preserve the original stereo files separately. Mono conversion makes the timing example easier, but it does not evaluate spatial imaging or stereo artifacts.

Step 2: Estimate the lag with cross-correlation

Cross-correlation asks:

At what shift do these two signals match most strongly?

A practical implementation should limit the search window. Songs contain repeated beats and repeated sections, so searching across the entire duration can match the wrong chorus or drum pattern.

def standardize(audio: np.ndarray) -> np.ndarray:
    """Remove the mean and scale variance for correlation only."""

    audio = np.asarray(audio, dtype=np.float64)
    centered = audio - np.mean(audio)
    scale = np.std(centered)

    if scale < 1e-12:
        raise ValueError("Cannot align a silent or near-silent waveform")

    return centered / scale


def estimate_lag(
    reference: np.ndarray,
    estimate: np.ndarray,
    sample_rate: int,
    max_lag_seconds: float = 0.5,
) -> int:
    """
    Return the sample shift to apply to the estimate.

    Positive return value: shift estimate right.
    Negative return value: shift estimate left.
    """

    if sample_rate <= 0:
        raise ValueError("sample_rate must be positive")

    reference = standardize(reference)
    estimate = standardize(estimate)

    correlation = signal.correlate(
        reference,
        estimate,
        mode="full",
        method="fft",
    )
    lags = signal.correlation_lags(
        reference.size,
        estimate.size,
        mode="full",
    )

    max_lag = int(round(max_lag_seconds * sample_rate))
    allowed = np.abs(lags) <= max_lag

    if not np.any(allowed):
        raise ValueError("No candidate lags were available")

    candidate_corr = correlation[allowed]
    candidate_lags = lags[allowed]

    best_index = int(np.argmax(candidate_corr))
    return int(candidate_lags[best_index])

The function returns the shift that should be applied to the estimate.

A negative lag means the estimate is delayed and should be moved left.

A positive lag means it should be moved right.

Step 3: Apply the shift without changing content

Alignment should correct timing only.

It should not denoise, equalize, compress, or otherwise “improve” one product’s output.

def shift_to_reference(
    estimate: np.ndarray,
    lag_samples: int,
    reference_length: int,
) -> np.ndarray:
    """Shift an estimate and return exactly reference_length samples."""

    estimate = np.asarray(estimate, dtype=np.float64)

    if reference_length <= 0:
        raise ValueError("reference_length must be positive")

    if lag_samples > 0:
        shifted = np.pad(estimate, (lag_samples, 0))
    elif lag_samples < 0:
        advance = -lag_samples
        shifted = estimate[advance:]
    else:
        shifted = estimate

    if shifted.size < reference_length:
        shifted = np.pad(
            shifted,
            (0, reference_length - shifted.size),
        )

    return shifted[:reference_length]

Now combine the steps:

def align_for_evaluation(
    reference: np.ndarray,
    estimate: np.ndarray,
    sample_rate: int,
    max_lag_seconds: float = 0.5,
) -> tuple[np.ndarray, int]:
    """Align estimate to reference and report the applied lag."""

    reference = to_mono(reference)
    estimate = to_mono(estimate)

    lag = estimate_lag(
        reference=reference,
        estimate=estimate,
        sample_rate=sample_rate,
        max_lag_seconds=max_lag_seconds,
    )

    aligned = shift_to_reference(
        estimate=estimate,
        lag_samples=lag,
        reference_length=reference.size,
    )

    return aligned, lag

Step 4: Verify the alignment

Never assume the largest correlation peak is correct.

Check:

the lag in samples;
the lag in milliseconds;
the correlation before and after;
a short waveform overlay;
a listening test around transients.

def lag_in_milliseconds(
    lag_samples: int,
    sample_rate: int,
) -> float:
    return 1000.0 * lag_samples / sample_rate


aligned, lag = align_for_evaluation(
    reference=reference_vocal,
    estimate=estimated_vocal,
    sample_rate=44_100,
)

print("Applied lag:", lag, "samples")
print("Applied lag:", lag_in_milliseconds(lag, 44_100), "ms")

If the reported shift is unexpectedly large, do not continue automatically. Investigate whether:

the wrong file was loaded;
the stem starts at a different song section;
the export was trimmed;
there is a sample-rate mismatch;
the signal contains long silence;
repetitive music created a false correlation peak.

Why alignment can still fail

Repeated beats

A steady kick pattern can create several strong peaks. Restrict the lag window and use a section with distinctive transients.

Long silence

Silence contains little information. Skip silent intros or align on an active excerpt.

Different source content

If one tool removes part of the vocal or damages the transient structure, correlation may become less reliable.

Stereo phase differences

Averaging stereo channels can cancel information. For stereo evaluation, consider aligning channels separately or using a carefully chosen mono reference only for timing estimation.

Variable delay

A constant shift cannot correct clock drift, time stretching, or nonlinear resampling. If the offset changes through the song, inspect the sample rates and processing pipeline.

Where SI-SDR fits

After alignment, SI-SDR can answer the intended question:
How closely does the estimated stem match its reference after accounting for scale?
SI-SIR adds information about unwanted source leakage, while SI-SAR focuses on artifacts introduced by separation.

A strong music-separation benchmark should report several metrics and also provide playable audio. One overall number cannot explain missing cymbals, metallic textures, damaged reverb, or vocal residue.

Our five-song benchmark reported:

NeuralSound: 15.80 dB overall SI-SDR;
Moises: 14.47 dB;
Fadr: 12.59 dB.

Those numbers are limited to the tested songs and configuration. They are not a universal ranking.
You can inspect the per-track audio and methodology in the NeuralSound vs Moises vs Fadr comparison.

A reproducibility checklist

Before comparing an AI vocal remover or stem separator, record:

input file hash;
sample rate and channel count;
product and plan;
separation mode;
test date;
output format;
estimated lag;
duration correction;
metric implementation;
failures and retries.

Also publish the raw results when possible.

neuralgpt407 / ai-vocal-remover-benchmark-2026

NeuralSound AI Vocal Remover Benchmark 2026

NeuralSound vs Moises vs Fadr

A playable five-song comparison of AI vocal removal and two-stem music separation.

About NeuralSound

NeuralSound is an AI vocal remover, online music separator and multi-stem splitter available on the web, Android, iPhone and iPad. It can separate a mixed song or video into clean vocals, drums, bass, guitar, piano and other instrument stems, helping musicians, DJs, singers, producers and content creators work with individual parts of a recording.

Use NeuralSound to remove vocals from a song, create an instrumental or backing track, extract an acapella, split music into stems, reduce music behind a voice with the background music remover, or prepare practice tracks with the AI karaoke maker. NeuralSound also supports synchronized lyrics, pitch and tempo controls, stem mixing, audio/video input and downloadable separated tracks.

This repository publishes original benchmark evidence for people…

View on GitHub

The live benchmark provides the playable inputs and outputs, while the GitHub repository contains the public project resources.

Product context without turning the article into an ad

This post focuses on evaluation rather than product claims.

For readers unfamiliar with the terms:

an AI vocal remover separates vocals and instrumental;
an AI music separator can additionally isolate drums, bass, guitar, piano, and other stems .

The engineering lesson applies to any provider:

Synchronize first. Score second. Publish enough detail for others to reproduce the result.

Final takeaway

A few milliseconds can make a good model look bad.

Before trusting SI-SDR, SI-SIR, or SI-SAR:

standardize the sample rate and channels;
estimate a bounded timing offset;
apply timing correction only;
verify the shift visually and audibly;
calculate multiple metrics;
publish enough detail for others to reproduce the result.

That process creates a more honest music source-separation benchmark—and prevents timing errors from being mistaken for model errors.

References

MUSDB18 dataset documentation
SDR — Half-baked or Well Done?
Interactive NeuralSound benchmark

How We Benchmarked 3 AI Vocal Removers with SI-SDR, SI-SIR, and SI-SAR

Neural Sound — Fri, 31 Jul 2026 23:03:10 +0000

Disclosure: NeuralSound designed and conducted this benchmark. Moises and Fadr did not review or approve the test. The same input files and evaluation pipeline were used for all three services.

Comparing AI vocal removers is harder than uploading one song and deciding which result sounds louder or cleaner.

Cloud services may introduce timing offsets, export different file lengths, apply different output levels, and update their models without exposing a version number. A fair comparison therefore needs consistent inputs, reference stems, time alignment, objective metrics, and audible examples.

We compared NeuralSound, Moises, and Fadr on the same five songs in two-stem mode:

isolated vocals
instrumental

The full interactive benchmark includes 35 playable previews and the complete per-track results:

▶️ Play the full benchmark

The evaluation question

For each product, we wanted to answer three separate questions:

How closely does the estimated stem match the reference?
How much of the unwanted source remains?
How many artifacts were introduced by the separation process?

That is why we used three related metrics instead of relying on one score:

SI-SDR for overall reconstruction quality
SI-SIR for unwanted source interference
SI-SAR for processing artifacts

Test setup

We used the first five songs in the valid folder of the MUSDB18-HQ source used for this study.

For every track:

The identical mixture was uploaded to all three services.
Each service produced a vocal stem and an instrumental stem.
The original vocal stem was used as the vocal reference.
The instrumental reference was calculated as:
instrumental_reference = mixture - vocals
Evaluation signals were converted to mono at 44.1 kHz.
Estimated outputs were aligned with the references.
Timing and length differences were corrected only for synchronization.
No denoising, EQ, or post-processing was applied before scoring.

Why alignment matters

Even a good separation can score poorly if the estimate is shifted by a few milliseconds.

A simplified alignment step looks like this:

from __future__ import annotations

import numpy as np
from scipy import signal


def align_estimate(
    reference: np.ndarray,
    estimate: np.ndarray,
) -> np.ndarray:
    """Align an estimated mono waveform to a mono reference waveform."""

    if reference.ndim != 1 or estimate.ndim != 1:
        raise ValueError("reference and estimate must be mono waveforms")

    if reference.size == 0 or estimate.size == 0:
        raise ValueError("waveforms must not be empty")

    correlation = signal.correlate(
        reference,
        estimate,
        mode="full",
        method="fft",
    )
    lag = int(np.argmax(correlation) - (estimate.size - 1))

    if lag > 0:
        aligned = np.pad(estimate, (lag, 0))
    elif lag < 0:
        aligned = estimate[-lag:]
    else:
        aligned = estimate

    if aligned.size < reference.size:
        aligned = np.pad(aligned, (0, reference.size - aligned.size))

    return aligned[: reference.size]

This is only the alignment stage, not a complete source-separation evaluator. In a production benchmark, also validate sample rate, channel layout, clipping, silent references, and file integrity.

What the metrics measure

SI-SDR: overall reconstruction quality:
SI-SDR measures how closely an estimated stem matches its reference after accounting for a simple difference in scale.

Higher is better.

SI-SIR: unwanted source leakage:
SI-SIR focuses on interference from the wrong source.

For an isolated vocal, a higher score generally means less accompaniment remains in the vocal. For an instrumental, it generally means less vocal residue remains in the music.

SI-SAR: processing artifacts:
SI-SAR focuses on artifacts introduced by the separation system.

Listeners may hear these as metallic textures, unstable reverb, watery sounds, missing transients, or robotic vocal edges.

The SI-SDR formulation was proposed as a simpler and more robust alternative to commonly misused SDR implementations in source-separation evaluation.

NeuralSound produced the highest average measured result in this five-song test.

That statement is intentionally narrow. It does not mean NeuralSound will perform best on every song, genre, model version, or export setting.

Averages hide track-level variation

The per-track results were not equally difficult.

The So So Glos — Emergency produced the lowest average SI-SDR for all three services:

NeuralSound: 12.31 dB
Moises: 11.52 dB
Fadr: 10.13 dB

The Wrong’Uns — Rothko produced the highest average SI-SDR for all three:

NeuralSound: 19.88 dB
Moises: 18.11 dB
Fadr: 14.97 dB

This is one reason a single aggregate number is not enough. Separation quality depends heavily on the source mix, vocal reverb, instrument overlap, distortion, and arrangement density.

What we learned

1. Synchronization is part of evaluation

A timing offset can change the score even when the audio sounds similar. Alignment must be documented rather than treated as an invisible cleanup step.

2. One metric cannot describe the whole result

A system can reduce interference while introducing artifacts. SI-SDR, SI-SIR, and SI-SAR should be interpreted together.

3. Listening tests still matter

Objective metrics make the comparison reproducible, but they do not fully represent human preference.

Listeners should still check:

accompaniment inside the vocal
lead-vocal residue inside the instrumental
missing cymbals or guitar attacks
phasey stereo effects
unstable ambience
damaged vocal texture and reverb

Limitations

This benchmark has several important limitations:

only five songs were tested
only two-stem separation was evaluated
there was no blind listening panel
cloud services may update their models
account tier and export format may affect results
NeuralSound conducted the study
the result is not an official MUSDB18 leaderboard A stronger follow-up should include more tracks, more genres, equivalent lossless exports, repeated processing runs, and a blind listening test.

Reproduce or inspect the full benchmark

The interactive page includes:
five original mixtures
15 vocal outputs
15 instrumental outputs
per-track SI-SDR, SI-SIR, and SI-SAR
methodology and limitations

▶️ Open the NeuralSound vs Moises vs Fadr benchmark

References

MUSDB18 dataset documentation