Neural Sound

Posted on Jul 31

How We Benchmarked 3 AI Vocal Removers with SI-SDR, SI-SIR, and SI-SAR

#machinelearning #python #podcast #ai

Disclosure: NeuralSound designed and conducted this benchmark. Moises and Fadr did not review or approve the test. The same input files and evaluation pipeline were used for all three services.

Comparing AI vocal removers is harder than uploading one song and deciding which result sounds louder or cleaner.

Cloud services may introduce timing offsets, export different file lengths, apply different output levels, and update their models without exposing a version number. A fair comparison therefore needs consistent inputs, reference stems, time alignment, objective metrics, and audible examples.

We compared NeuralSound, Moises, and Fadr on the same five songs in two-stem mode:

isolated vocals
instrumental

The full interactive benchmark includes 35 playable previews and the complete per-track results:

▶️ Play the full benchmark

The evaluation question

For each product, we wanted to answer three separate questions:

How closely does the estimated stem match the reference?
How much of the unwanted source remains?
How many artifacts were introduced by the separation process?

That is why we used three related metrics instead of relying on one score:

SI-SDR for overall reconstruction quality
SI-SIR for unwanted source interference
SI-SAR for processing artifacts

Test setup

We used the first five songs in the valid folder of the MUSDB18-HQ source used for this study.

For every track:

The identical mixture was uploaded to all three services.
Each service produced a vocal stem and an instrumental stem.
The original vocal stem was used as the vocal reference.
The instrumental reference was calculated as:
instrumental_reference = mixture - vocals
Evaluation signals were converted to mono at 44.1 kHz.
Estimated outputs were aligned with the references.
Timing and length differences were corrected only for synchronization.
No denoising, EQ, or post-processing was applied before scoring.

Why alignment matters

Even a good separation can score poorly if the estimate is shifted by a few milliseconds.

A simplified alignment step looks like this:

from __future__ import annotations

import numpy as np
from scipy import signal


def align_estimate(
    reference: np.ndarray,
    estimate: np.ndarray,
) -> np.ndarray:
    """Align an estimated mono waveform to a mono reference waveform."""

    if reference.ndim != 1 or estimate.ndim != 1:
        raise ValueError("reference and estimate must be mono waveforms")

    if reference.size == 0 or estimate.size == 0:
        raise ValueError("waveforms must not be empty")

    correlation = signal.correlate(
        reference,
        estimate,
        mode="full",
        method="fft",
    )
    lag = int(np.argmax(correlation) - (estimate.size - 1))

    if lag > 0:
        aligned = np.pad(estimate, (lag, 0))
    elif lag < 0:
        aligned = estimate[-lag:]
    else:
        aligned = estimate

    if aligned.size < reference.size:
        aligned = np.pad(aligned, (0, reference.size - aligned.size))

    return aligned[: reference.size]

This is only the alignment stage, not a complete source-separation evaluator. In a production benchmark, also validate sample rate, channel layout, clipping, silent references, and file integrity.

What the metrics measure

SI-SDR: overall reconstruction quality:
SI-SDR measures how closely an estimated stem matches its reference after accounting for a simple difference in scale.

Higher is better.

SI-SIR: unwanted source leakage:
SI-SIR focuses on interference from the wrong source.

For an isolated vocal, a higher score generally means less accompaniment remains in the vocal. For an instrumental, it generally means less vocal residue remains in the music.

SI-SAR: processing artifacts:
SI-SAR focuses on artifacts introduced by the separation system.

Listeners may hear these as metallic textures, unstable reverb, watery sounds, missing transients, or robotic vocal edges.

The SI-SDR formulation was proposed as a simpler and more robust alternative to commonly misused SDR implementations in source-separation evaluation.

NeuralSound produced the highest average measured result in this five-song test.

That statement is intentionally narrow. It does not mean NeuralSound will perform best on every song, genre, model version, or export setting.

Averages hide track-level variation

The per-track results were not equally difficult.

The So So Glos — Emergency produced the lowest average SI-SDR for all three services:

NeuralSound: 12.31 dB
Moises: 11.52 dB
Fadr: 10.13 dB

The Wrong’Uns — Rothko produced the highest average SI-SDR for all three:

NeuralSound: 19.88 dB
Moises: 18.11 dB
Fadr: 14.97 dB

This is one reason a single aggregate number is not enough. Separation quality depends heavily on the source mix, vocal reverb, instrument overlap, distortion, and arrangement density.

What we learned

1. Synchronization is part of evaluation

A timing offset can change the score even when the audio sounds similar. Alignment must be documented rather than treated as an invisible cleanup step.

2. One metric cannot describe the whole result

A system can reduce interference while introducing artifacts. SI-SDR, SI-SIR, and SI-SAR should be interpreted together.

3. Listening tests still matter

Objective metrics make the comparison reproducible, but they do not fully represent human preference.

Listeners should still check:

accompaniment inside the vocal
lead-vocal residue inside the instrumental
missing cymbals or guitar attacks
phasey stereo effects
unstable ambience
damaged vocal texture and reverb

Limitations

This benchmark has several important limitations:

only five songs were tested
only two-stem separation was evaluated
there was no blind listening panel
cloud services may update their models
account tier and export format may affect results
NeuralSound conducted the study
the result is not an official MUSDB18 leaderboard A stronger follow-up should include more tracks, more genres, equivalent lossless exports, repeated processing runs, and a blind listening test.

Reproduce or inspect the full benchmark