A Classification Model Suddenly Started Getting Audio Wrong

#ai #debugging #llm #machinelearning

Accuracy was 0.91 for six months. On Tuesday it was 0.62, on the same model binary, with no deployment and no code change. That specific shape — a step, not a slope, with the model untouched — points at the input, and the most common single cause is clipping in the capture chain.

The symptom, and what it rules out

The step is the diagnostic. Genuine data drift is gradual: the world changes over weeks, and accuracy sags rather than falls off a cliff. A discontinuity on a specific day means something discrete changed, and if it was not your model, it was your data. The usual list is a firmware update on the recording device, a new hardware revision in the field, a change in the ingest or transcoding path, or a gain setting somebody adjusted.

Three further observations narrow it to clipping specifically:

Loud classes suffer most. Clipping only affects samples that exceed full scale, so quiet classes are untouched while loud ones collapse. If your per-class accuracy table shows the drop concentrated in the loudest categories, that is close to conclusive.
Predictions collapse toward one or two classes. Clipped audio acquires broadband harmonic energy that resembles whichever training class was noisiest, so errors are not scattered — they pile up on a specific wrong answer.
The audio still sounds basically fine. Moderate clipping is not obvious to a listener on laptop speakers, especially on already-loud material. “I listened to a few files and they were OK” does not rule this out, which is why the check below is numerical.

The check that finds it in one pass

Clipping is a flat top on the waveform: consecutive samples pinned at the maximum representable value because the true signal went past it. The naive test — count samples at full scale — produces false positives, because a single sample can legitimately hit the maximum. The reliable test adds run length: a plateau of consecutive samples at or near full scale does not occur in unclipped audio.

import numpy as np
import soundfile as sf

def clipping_report(path, near=0.999, min_run=3):
    """Fraction of samples at full scale, and the longest plateau."""
    x, sr = sf.read(path, dtype="float32", always_2d=True)
    x = x.mean(axis=1)                       # mono, average not sum

    limit = near * 1.0                       # soundfile scales to [-1, 1]
    at_rail = np.abs(x) >= limit
    frac = float(at_rail.mean())

    # longest run of consecutive at-rail samples
    longest, run = 0, 0
    for flag in at_rail:
        run = run + 1 if flag else 0
        longest = max(longest, run)

    # count of plateaus, which is the number of clipped events
    padded = np.concatenate(([False], at_rail, [False]))
    starts = np.flatnonzero(~padded[:-1] & padded[1:])
    ends = np.flatnonzero(padded[:-1] & ~padded[1:])
    events = int(np.sum((ends - starts) >= min_run))

    return {"rail_fraction": frac, "longest_run": longest,
            "clip_events": events, "sample_rate": sr}

Read the output like this. A rail_fraction below about 1e-5 with a longest_run of 1 or 2 is normal. A longest_run in the tens or hundreds is unambiguous clipping. A rail_fraction above 0.001 means roughly one sample in a thousand is pinned, which at 16 kHz is 16 clipped samples a second and is already destroying your features.

Then find the change point, which is the step that turns a suspicion into a cause. Run the report across a sample of files from each ingest day for the last few months and plot the median rail_fractionagainst date. If it jumps on one day, you now have the date, and the change log for that date has the answer. Group the same numbers by device ID as well — a firmware rollout that reached 30% of the fleet shows as a bimodal distribution rather than a step, and a date plot alone will make it look like a gradual drift.

Run the check at the right stage of the pipeline. In float32, values beyond ±1.0 are representable and have not clipped yet; the damage happens at conversion to int16. If your pipeline holds float audio in memory, also check for max(abs(x)) > 1.0, which is the same problem one step earlier and is still repairable at that point.

Why clipping destroys a spectrogram model

Hard limiting is a memoryless nonlinearity. Feed a sine wave through one and the output tends toward a square wave, and a square wave contains the odd harmonics of the fundamental at slowly decaying amplitude. Feed a complex signal through it and you get every harmonic of every component plus intermodulation products at every sum and difference frequency — energy spread across the entire band, generated from nothing.

On a log-mel spectrogram that appears as a broadband lift, strongest in the high mel bands where the original signal had least energy and the added harmonics have most. Every feature a classifier depends on moves: spectral centroid rises, spectral flatness rises, harmonic-to-noise ratio falls, and the mel band ratios that encode timbre are rewritten. The model is not seeing a degraded version of its training distribution, it is seeing a different distribution.

The consequence people find hardest to accept is that this is irreversible. Peak-normalising a clipped file rescales it so that its maximum is 1.0 and its waveform looks well-behaved, and it does nothing at all, because the flat tops are still flat — the samples that exceeded full scale were never recorded and no scaling brings them back. De-clipping algorithms exist and reconstruct plausible values by interpolation or sparse recovery, but they are inference, not recovery, and on short plateaus they help while on long ones they invent.

Where the clipping came from

A gain or sensitivity change on the device. A new microphone with higher sensitivity, or a preamp gain someone raised to fix a “too quiet” complaint. This is the most common single cause.
Automatic gain control. AGC raises quiet input, and when a loud event arrives before the control loop reacts, the attack period clips. This produces clipping concentrated at event onsets — exactly the part of the signal a detector cares about most.
Summing stereo to mono by addition. Two correlated channels each peaking at 0.9 sum to 1.8. Average instead of adding. This one appears the day someone changes a downmix and affects every file at once.
Float to int16 conversion without headroom. Any processing that can produce values beyond ±1.0 — a gain stage, a filter with resonance, a resampler whose interpolation overshoots — will saturate or, worse, wrap at conversion. Wrapping is louder and more destructive than saturation and appears as full-scale samples of the opposite sign.
Lossy decode overshoot. A file mastered to 0 dBFS can decode from MP3 or AAC to values slightly above full scale, because the codec does not guarantee the reconstruction stays inside the range. A pipeline that decodes straight into int16 clips a small fraction of an otherwise fine file.

The fix, in order

Quantify before changing anything. Run the report above over a stratified sample — a few hundred files per ingest day, per device model — and record rail_fraction, longest_run and clip_events. You need the baseline to prove the fix worked.
Locate the change. Plot the median against ingest date and against device identifier. A step in date points at a pipeline or firmware change; a split by device points at hardware. Cross-reference with deployment logs for that day.
Fix the capture, not the data. Reduce input gain until peaks sit around −12 dBFS in normal operation, which leaves headroom for the loud events that matter. If AGC is responsible and cannot be disabled, a limiter with a fast attack ahead of the converter is less damaging than hard clipping, because it compresses rather than truncates.
Add an ingest guard. Reject or flag any file whose rail_fraction exceeds a threshold you set from the clean baseline, and emit it as a metric with an alert. This is the change that stops the same incident recurring, and it is a few lines.
Quarantine the affected data. Do not retrain on it. Clipped audio in a training set teaches the model that the artefact is a feature, which appears to fix accuracy on clipped inputs and degrades it on clean ones. Exclude the affected window, verify the model recovers on newly captured clean audio, and only then decide whether anything needs retraining.
If clipping is unavoidable in production, model it explicitly. Some sources genuinely arrive clipped and cannot be re-recorded. In that case augment training with deliberately clipped copies at the observed rates, so the model sees both conditions — and expect a lower ceiling, because information that was never captured cannot be recovered by training.

The same check belongs at the front of any acoustic pipeline whose features depend on spectral shape. Band-ratio methods are especially exposed, since clipping adds energy to precisely the high-frequency bands they read — the envelope analysis used for machinery faults will report a fault that is entirely an artefact of the microphone input stage.