I fixed an unbounded audio queue in a live translation server before it could take the process down

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

The project is a live speech translation app.

A browser client captures microphone audio, converts it to 16 kHz PCM, and streams it to a NestJS server over Socket.IO. The server opens one WebSocket session to a local ML service, forwards the audio frames, and relays translated segments back to listeners.

The important part for this bug: audio starts arriving from the browser before the ML WebSocket is guaranteed to be open.

That is normal for realtime systems. The dangerous part was what happened when the ML service was unavailable, slow to handshake, or stuck behind a bad connection.

Bug Fix or Performance Improvement

The MlSession class buffered audio frames while waiting for the ML WebSocket to open:

sendAudio(pcm: Buffer) {
  if (this.closed) return;
  if (this.open) this.ws.send(pcm);
  else this.queue.push(pcm);
}

That looks harmless until you remember what the data is: raw PCM audio.

At 16 kHz, mono, 16-bit PCM, the stream is roughly 32 KB per second per active speaker. If the ML WebSocket never opens, every audio event keeps appending another Buffer to queue. A stalled ML service could quietly turn one active audio source into an unbounded memory growth path.

The server did not have to be under heavy traffic for this to matter. One broadcaster with a stuck ML backend was enough to create an ever-growing in-memory queue.

Code

I added an explicit cap for queued audio before the ML connection opens, tracked queued bytes, reset the counter once the connection opens, and released queued buffers on close/error.

const MAX_QUEUED_AUDIO_BYTES = Number(
  process.env.ML_MAX_QUEUED_AUDIO_BYTES || 512 * 1024,
);

The send path now fails fast instead of buffering forever:

sendAudio(pcm: Buffer) {
  if (this.closed) return;
  if (this.open) this.ws.send(pcm);
  else {
    if (this.queuedBytes + pcm.byteLength > MAX_QUEUED_AUDIO_BYTES) {
      this.finish(new Error("ML websocket audio queue overflow before connection opened"));
      try {
        this.ws.close();
      } catch {}
      return;
    }
    this.queue.push(pcm);
    this.queuedBytes += pcm.byteLength;
  }
}

When the socket opens, queued audio is flushed and the byte counter goes back to zero:

this.ws.on("open", () => {
  this.ws.send(JSON.stringify({ type: "config", ...config }));
  this.open = true;
  for (const buf of this.queue) this.ws.send(buf);
  this.queue = [];
  this.queuedBytes = 0;
});

Close/error paths also clear the queue:

close() {
  this.closed = true;
  this.queue = [];
  this.queuedBytes = 0;
  try {
    this.ws.close();
  } catch {}
}

private finish(err?: Error) {
  if (this.closed) return;
  this.closed = true;
  this.queue = [];
  this.queuedBytes = 0;
  this.onClose(err);
}

My Improvements

The fix changes the failure mode.

Before:

• ML service unavailable or stuck
• browser keeps streaming audio
• server queues every PCM frame
• memory grows until the process is restarted or killed

After:

• ML service unavailable or stuck
• server accepts only a bounded amount of pending audio
• session closes with a specific overflow error
• existing gateway cleanup runs and listeners get the normal ended signal
• memory is released immediately

I kept the default cap at 512 KB, which is around 16 seconds of 16 kHz mono PCM audio. That is enough for normal connection delay but small enough to prevent a single stalled session from becoming a memory leak.

The value is configurable:

ML_MAX_QUEUED_AUDIO_BYTES=1048576

That lets production tune the buffer based on expected network conditions without changing code.

Verification

The normal server build still passes:

npm run build

Output:

> live-translation-server@0.1.0 build
> tsc -p tsconfig.json

I also ran a smoke test that accepts the TCP connection but intentionally never completes the WebSocket handshake. Then I sent audio chunks into MlSession with a small queue limit.

Expected result: the session should close with the overflow error instead of keeping the buffers forever.

Observed result:

{"closed":true,"errMessage":"ML websocket audio queue overflow before connection opened"}

That is the behavior I wanted: when the ML backend is not actually ready to receive audio, the realtime server fails the session explicitly instead of becoming a quiet memory sink.

What I learned

Realtime systems make small buffers feel safe.

This was not a dramatic bug in the happy path. The app worked when the ML WebSocket opened quickly. The bug lived in the waiting room between "the browser is already sending audio" and "the ML service is ready."

That is exactly where production bugs like to hide: not in the main algorithm, but in the glue code that assumes the next component will be ready soon.

The fix was not a bigger abstraction. It was a boundary:

audio may wait briefly, but it may not wait forever.

That one rule turns an unbounded failure into a controlled one.

Comments

[Luis Cruz]

I appreciate how you've addressed the issue of unbounded audio queue growth by introducing a cap on queued audio bytes before the ML connection opens. The addition of MAX_QUEUED_AUDIO_BYTES as a configurable environment variable is particularly useful for adapting to different network conditions. I'm curious to know if you've considered implementing any monitoring or alerting mechanisms to detect when the queue is approaching its limit, allowing for proactive intervention before the ML websocket audio queue overflow error occurs.

[Alex Shev]

Yes, monitoring is the next layer I would add before calling this production-complete. I would track queued audio bytes, queue age, time-to-ML-open, overflow count, and session close reason. The alert should probably fire before the hard cap, because the cap is the last safety stop; the useful signal is repeated near-cap pressure or slow ML connection establishment.

[JustSoftLab]

The line that stuck with me: "audio may wait briefly, but it may not wait forever." That's the whole discipline in one sentence, and it generalizes far past this one queue. Every producer/consumer seam where you can't pause the producer has this bug latent in it.
Once the bound is in, the interesting question is what to shed when you hit it. You chose fail-the-session, which is right when the ML backend is dead. The other tool worth keeping for realtime audio: drop oldest frames and keep the session alive. Stale PCM is worthless once you're behind, so for a transient stall rather than a dead backend, dropping the front of the queue can save the call instead of ending it. Fail-fast vs drop-oldest really comes down to whether the stall is fatal or temporary, and sometimes you want both: drop-oldest up to a threshold, then fail if it persists.

[Alex Shev]

Exactly. The shed policy is the product decision hiding inside the queue. Dropping, failing the session, summarizing, or backpressuring all protect the process differently. The important thing is choosing that behavior before the system is hot.

[JustSoftLab]

"Choosing that behavior before the system is hot" is the whole thing. The shed policy always exists. The only question is whether you picked it or the incident picked it for you. Summarizing is the underrated option on your list here, a partial transcript beats a dropped second of audio when the goal is comprehension over fidelity. Enjoyed this one

[Alex Shev]

Exactly. Summarizing is easy to forget because it is not a pure queue operation, but for realtime audio it can preserve the user outcome better than pretending every frame has equal value.