A stereo gotcha in audio_decode 0.3.1: toFloat32() is interleaved, not per-channel

audio_decode (pub.dev, v0.3.1) decodes MP3 and Ogg Vorbis to PCM. PcmAudio.toFloat32() gives you the samples as floats in [-1.0, 1.0]. On stereo audio, that array holds left and right samples interleaved: L0, R0, L1, R1, L2, R2, .... Any code that walks it with a fixed window and treats consecutive elements as consecutive time steps of one signal (an envelope follower, RMS-per-window, peak detection, zero-crossing pitch estimation, a waveform downsampler for a UI) runs, produces numbers, and is wrong, silently, with no exception and no obviously garbled output.

The shape of samples

From the source, lib/src/audio_decode_base.dart:

/// [samples] holds signed 16-bit samples interleaved by channel, so for stereo
/// the order is left, right, left, right and so on. A single audio frame is one
/// sample per channel; [frameCount] is the number of frames and [duration] is
/// how long they play at [sampleRate].
class PcmAudio {
  final int sampleRate;
  final int channels;
  final Int16List samples;

  int get frameCount => channels == 0 ? 0 : samples.length ~/ channels;

And toFloat32():

Float32List toFloat32() {
  final out = Float32List(samples.length);
  for (var i = 0; i < samples.length; i++) {
    out[i] = samples[i] / _int16Scale;
  }
  return out;
}

It is a straight per-element divide. No deinterleaving happens. The doc comment on toFloat32() even says "interleaved by channel like samples", but nothing about the method name suggests that, and it's easy to write time-domain code against it without ever opening the source.

The bug: windowing an interleaved array

Here is the natural way to write an envelope follower if you don't know about the interleaving:

List<double> naiveEnvelope(PcmAudio audio, int windowSize) {
  final samples = audio.toFloat32();
  final envelope = <double>[];
  for (var start = 0; start + windowSize <= samples.length; start += windowSize) {
    var peak = 0.0;
    for (var i = start; i < start + windowSize; i++) {
      final abs = samples[i].abs();
      if (abs > peak) peak = abs;
    }
    envelope.add(peak);
  }
  return envelope;
}

This compiles, runs, and returns a plausible-looking list of peak values. On mono audio it's correct. On stereo audio, each "window" of 512 array elements is actually 256 frames of L and R interleaved, half the real time you meant to cover, and each element inside it alternates between two different channels.

The fix: channel(index) or toMono()

PcmAudio has two methods that actually deinterleave. channel(index) returns one channel, one value per frame:

Float32List channel(int index) {
  if (index < 0 || index >= channels) {
    throw RangeError.range(index, 0, channels - 1, 'index');
  }
  final frames = frameCount;
  final out = Float32List(frames);
  for (var i = 0; i < frames; i++) {
    out[i] = samples[i * channels + index] / _int16Scale;
  }
  return out;
}

toMono() averages all channels per frame into a new single-channel PcmAudio, if you want one combined signal instead of picking a channel:

PcmAudio toMono() {
  if (channels <= 1) return this;
  final frames = frameCount;
  final mono = Int16List(frames);
  for (var i = 0; i < frames; i++) {
    final base = i * channels;
    var sum = 0;
    for (var c = 0; c < channels; c++) {
      sum += samples[base + c];
    }
    mono[i] = (sum / channels).round();
  }
  return PcmAudio(sampleRate: sampleRate, channels: 1, samples: mono);
}

The corrected envelope function is the same loop, just fed a deinterleaved array:

List<double> channelEnvelope(PcmAudio audio, int channelIndex, int windowSize) {
  final samples = audio.channel(channelIndex);
  final envelope = <double>[];
  for (var start = 0; start + windowSize <= samples.length; start += windowSize) {
    var peak = 0.0;
    for (var i = start; i < start + windowSize; i++) {
      final abs = samples[i].abs();
      if (abs > peak) peak = abs;
    }
    envelope.add(peak);
  }
  return envelope;
}

What actually happens on real stereo audio

I ran both versions against the package's own test fixture, sine_44100_stereo_1s.mp3, a 1.07 second, 44.1 kHz stereo clip, with windowSize = 512:

import 'dart:io';
import 'package:audio_decode/audio_decode.dart';

int countPeaks(List<double> envelope) {
  final maxAbs = envelope.fold<double>(0.0, (m, v) => v > m ? v : m);
  final threshold = 0.5 * maxAbs;
  var count = 0;
  for (var i = 1; i < envelope.length - 1; i++) {
    if (envelope[i] >= threshold &&
        envelope[i] >= envelope[i - 1] &&
        envelope[i] >= envelope[i + 1]) {
      count++;
    }
  }
  return count;
}

void main(List<String> args) {
  final audio = decodeMp3(File(args.first).readAsBytesSync());
  print('channels: ${audio.channels}, frameCount: ${audio.frameCount}, '
      'samples.length: ${audio.samples.length}');

  const windowSize = 512;
  final naive = naiveEnvelope(audio, windowSize);
  final correct = channelEnvelope(audio, 0, windowSize);

  print('naive windows: ${naive.length}, correct windows: ${correct.length}');
  print('naive peaks: ${countPeaks(naive)}, correct peaks: ${countPeaks(correct)}');
}

Output:

channels: 2, frameCount: 47232, samples.length: 94464
naive windows: 184, correct windows: 92
naive peaks: 66, correct peaks: 36

Same windowSize, same clip, two different answers. The naive version produces exactly twice the window count (184 vs 92, a 2.00x ratio matching channels == 2), because it is walking 94464 interleaved elements instead of 47232 per-channel frames. The peak count differs by more than that factor: 66 vs 36, because each "window" now spans a different amount of real audio:

	naive, toFloat32(), 512 elements	correct, channel(0), 512 elements
elements per window	512 interleaved (L,R,L,R,...)	512 single-channel
frames covered	256	512
real time per window	5.805 ms	11.610 ms
windows for this clip	184	92
peaks (>= 0.5x max)	66	36

Why this fixture didn't visibly break, and when it will

One honest caveat: in this fixture, left and right are bit-identical, a mono sine duplicated into both channels (max abs(L[i] - R[i]) over all 47232 frames is 0.000000). So the error here comes from the window grid being wrong, not from mixing two different signals. If you eyeballed a plot of it, the windows would just be built from a shorter time span than you think, already enough to throw off a beat-detection threshold tuned in milliseconds, or a UI waveform that's supposed to represent one second of audio in a fixed number of bars.

On stereo content where the channels actually differ (any real music or voice recording panned even slightly), the same naive loop does something worse: it averages or aliases L and R samples together inside a window that thinks it's looking at one channel. A zero-crossing pitch detector counting sign changes across L0, R0, L1, R1, ... will count a crossing every time L and R differ in sign, regardless of the fundamental frequency of either channel. That failure mode doesn't show up on this fixture because there is nothing for the channels to disagree about.

Deinterleave before you window

If you're writing time-domain analysis against PcmAudio (envelope, RMS, peak, zero-crossing, downsampling for a waveform view), check channels first. On anything other than mono, call channel(index) for a specific channel or toMono() for a combined one before you start iterating with a fixed window. toFloat32() is the right call when you're about to hand the array to something that already expects interleaved multi-channel data (a WAV-style consumer, a resampler that knows about channels), and the wrong call the moment your window size means "N samples of time" instead of "N array elements."