Stream FreeSWITCH call audio to a WebSocket for real-time ASR (mod_ws_media)

#freeswitch #webrtc #ai #c

If you're building anything that listens to live phone calls — real-time transcription, an AI voice agent, call analytics, compliance — you eventually hit the same wall: how do you get the audio out of FreeSWITCH, decoded, in real time, and into your ASR/LLM?

You don't want to deal with RTP, jitter buffers, or per-codec decoding in your app. You want a clean PCM stream over a socket you already know how to consume. That's what I built mod_ws_media for.

What it does

mod_ws_media is a FreeSWITCH module that taps a call leg and streams its audio to any WebSocket server in real time. A few things that make it easy to consume:

Decoded L16 PCM — your server never touches the RTP codec. Whether the call is PCMU, PCMA, G.722 or Opus, you get plain signed 16-bit PCM at the channel's native rate.
One WebSocket per leg. On connect the module sends a small JSON start frame, then streams binary PCM. That's the whole protocol.
Non-intrusive. v1 is tap-only (read-only) — it forks the audio out and never modifies the call.
No heavy deps. The module ships its own RFC 6455 client on top of OpenSSL — no libwebsockets, no gRPC.

Try it in 5 minutes

Build it out-of-tree against an installed FreeSWITCH:

git clone https://github.com/luoyumin/mod_ws_media
cd mod_ws_media
make
sudo make install
fs_cli -x "load mod_ws_media"

The repo ships a zero-dependency example server (pure Python stdlib, no pip install) so you can see audio flowing without wiring up an ASR backend yet:

python3 example/echo_server.py        # listens on ws://0.0.0.0:8080/

Then tap a live call — it needs a real media path (echo, playback, bridge, etc., not a parked or bypassed channel):

uuid_ws_media <uuid> start ws://127.0.0.1:8080/media in=stereo role=agent
# ...talk, then hang up...
# -> ws_media_recordings/<call_id>.wav

The example server prints the start frame and writes the streamed PCM to a playable WAV. Swap it for your own server and you're streaming straight to your ASR.

Speaker separation for free

For transcription you almost always want to know who said what — agent vs. customer. mod_ws_media gives you that at the media layer with in=stereo:

left channel = read = this leg's own party (its mic — what it says)
right channel = write = the far party (what this leg hears)

Tap the agent leg in stereo and you get the agent on the left, the customer on the right — clean diarization without a diarization model. You can also grab a single direction with in=read / in=write, or a summed mono mix with in=mixed.

The wire protocol

On connect, one JSON text frame:

{
  "event": "start",
  "call_id": "abc123",
  "media_format": { "encoding": "L16", "sample_rate": 8000, "channels": 2, "ptime": 20 },
  "capture": {
    "mode": "stereo",
    "tracks": [
      { "ch": 0, "source": "read",  "role": "agent" },
      { "ch": 1, "source": "write", "role": "peer" }
    ]
  }
}

Then binary frames: raw signed 16-bit little-endian PCM. For stereo, samples are interleaved [L R L R …]. Treat it as a byte stream — a binary message isn't guaranteed to be exactly one 20 ms frame.

Config is per-call (URL, capture mode, role, call_id, metadata) via command args or channel variables, so different calls can stream to different backends.

When the backend hiccups

Long-lived taps need to survive backend restarts. The module reconnects automatically, and if the server stays unreachable it drops into a recoverable bypass mode — the call continues untouched, and the tap resumes when the backend comes back. TLS (wss://) with optional certificate verification and SNI is supported too.

Scope and what's next

v1.0.x is tap only — capture audio out. That already covers real-time transcription, recording and analytics, where you push results to a screen or a pipeline rather than back into the call.

On the roadmap is injection — putting processed audio back into a chosen leg. That's what you need for real-time translation, prompts, or agent whisper. The design is written up in docs/DESIGN.md.