DEV Community: Ahad Khan

How to Build a SIP-to-OpenAI Voice Gateway in C#

Ahad Khan — Wed, 11 Nov 2026 09:01:36 +0000

If you already have a SIP phone system, you can give an AI voice agent its own extension. A caller dials the extension, speaks normally, and hears the OpenAI Realtime model’s reply on the same call.

I’m affiliated with VaxVoIP, and this walkthrough uses our VaxVoIP SIP Call OpenAI SDK for Windows.

What the gateway does

SIP phone or PBX ↔ C# gateway ↔ OpenAI Realtime

SIP sets up and ends the phone call, and RTP carries the audio. The SDK handles the SIP stack, bridges audio in both directions, and connects to OpenAI Realtime. You can build the gateway without writing WebRTC code or implementing a SIP stack.

Your C# application decides which calls to accept, what instructions to give the voice agent, and which application functions the agent can use.

Try it with the C# sample

Watch the C# SIP-to-OpenAI gateway demo on YouTube

Download the C# sample project, extract it on Windows, and open the project in Visual Studio.
Configure your OpenAI API key in the product-order demo. Load the key from an environment variable or a secret store to keep it out of source control.
Run the sample, register a SIP softphone with the gateway, and dial 101. Ask the product-order agent whether it supports laptops and listen for the spoken reply.

After preparing the prompt, model, and tools, the sample’s incoming-call handler accepts the call with one SDK call.

base.AcceptCallSession(30, KeyOpenAI, ModelPrompt, Model, "coral", 1.0);

The sample also shows how to connect the voice agent to your application data. When the caller asks about a product, the model can request a lookup. The C# application checks the product and returns the result so the model can answer the caller.

Once the softphone test works, route a test extension from your PBX to the gateway. For an internal help desk, you could change the prompt and replace the product lookup with a function that retrieves support hours or checks ticket status.

VaxVoIP SIP Agent OpenAI SDK and documentation

What you can build

AI receptionists that greet callers, find out what they need, and route them to the right person or department.
Customer support agents that answer common questions, check order or ticket status, and hand off to a human when needed.
Telemarketing agents that call opted-in leads, ask qualifying questions, and book follow-up calls.
Smart AI autodialers that make reminder calls, return callbacks, run post-service surveys, and log call outcomes.
Smart IVRs that let callers explain what they need in their own words and route them to the right queue or self-service option.
Appointment booking agents that check availability, book or reschedule appointments, and confirm the details over the phone.
Order-taking agents that check product availability, collect order details, and read them back for confirmation.
Multilingual voice agents that answer questions in the caller’s preferred language and transfer calls when human help is needed.
Internal help desk agents that check ticket status or gather the information needed to open a new ticket.

How I Trained My Own On-Premises Speech-to-Speech AI System (Fully Offline)

Ahad Khan — Tue, 29 Jul 2025 12:35:27 +0000

In this post, I’ll share how I trained and deployed a complete speech-to-speech AI pipeline that runs entirely offline, without any dependency on third-party cloud APIs.

Using open-source models and GPU acceleration, I built a multilingual voice communication engine that can listen, understand, translate, and speak in real time.

What I Built

The system captures live audio, detects speech, transcribes it, processes it through a local language model, translates it into another language, and finally speaks it back — all in milliseconds.

Here’s a summary of the steps:

Capture incoming voice audio in raw PCM format
Detect voice activity and trim silence
Transcribe speech using Whisper
Generate intelligent responses using a fine-tuned LLaMA model
Translate back to the original language using NLLB
Convert the translated text to speech using MMS TTS
Send the audio response back to the user in real time

Tech Stack

Language: CSharp, Python
Audio Format: 16kHz, mono, 16-bit PCM
Speech Recognition: Whisper (OpenAI)
VAD: Silero VAD
LLM: LLaMA 3.2B / 8B (fine-tuned)
Translation: Meta NLLB-200 3.3B
Text-to-Speech: MMS TTS (Facebook)
Hardware: NVIDIA H200 GPU (Blackwell), also tested with H100

Pipeline Overview

Audio Input
Voice Activity Detection (Silero Model)
Speech Recognition (Whisper)
Language Model (LLaMA)
Translation (NLLB)
Text-to-Speech (MMS TTS)
Audio Output

Hardware Recommendations

To achieve real-time performance, I recommend:

Primary GPU: NVIDIA H200 (Blackwell, 141GB HBM3e)
Alternative: NVIDIA H100 (80GB HBM3)

What’s Next

I’m currently extending this system to include:

Language-based routing for SIP calls
Real-time speaker sentiment detection
WebRTC Integration for Website-Based AI Voice Agents

To test and develop an AI-powered support agent, I connected the pipeline with our SIP Server by using SIP SDK to handle real-time RTP audio, enabling full duplex voice conversations between callers and the AI agent over standard VoIP calls.

Related Resources

WebRTC and SIP Integration for Web to VoIP Solutions
VoIP Development with OpenAI SDK
Building a Speech-to-Speech AI Engine

How I Integrated OpenAI with a SIP Server and Built a Multilingual Smart AI Agent for IP Telephony

Ahad Khan — Mon, 28 Jul 2025 15:26:46 +0000

In this post, I’ll share how I integrated OpenAI’s real-time API with our SIP Server to create an intelligent voice agent capable of live transcription and AI-generated voice replies — all over a VoIP network.

This project demonstrates how traditional SIP-based telephony can be enhanced using powerful AI services like GPT-4o, without needing to replace existing infrastructure.

What We Built

Using VoIP SIP SDK, I developed a multilingual AI voice agent for VoIP systems. It captures real-time SIP audio in PCM format, streams it to OpenAI’s speech-to-speech WebSocket API, and receives back AI-generated audio replies in real time.

The system performs the following tasks:

Captures live SIP call audio in raw PCM format
Streams the audio to OpenAI’s real-time speech-to-speech API via WebSocket
Receives the AI-generated audio response from OpenAI
Sends the synthesized audio back to the caller via RTP

Tech Stack

Language: C#, VBNet
VoIP SDK: VoIP SIP Server SDK
AI Services: OpenAI GPT-4o (realtime Speech-to-Speech APIs)
Audio Handling: Named Pipes, PCM audio streaming

How the Integration Works

Incoming SIP Call

A SIP call is received on the server and accepted using SIP SDK.
Audio Captured in Real Time

The SDK provides access to the raw PCM audio stream from the RTP packets.
Streaming to OpenAI WebSocket

A client connects to OpenAI’s real-time speech-to-speech API and streams the PCM audio over WebSocket.
AI-Generated Audio Response

OpenAI processes the audio, understands the intent, and returns an AI-generated audio reply.
Audio Playback to Caller

The received audio is injected directly into the SIP call as RTP media, allowing real-time AI voice interaction.

Real-World Use Cases

Smart IVR Systems that answer customer queries
AI-based call routing by understanding caller intent
Language translation over SIP calls
VoIP chatbots that feel human

Related Resources

What’s Next

We’re actively expanding this system to support:

Multilingual translation across real-time SIP calls
Emotion detection to adapt responses based on caller sentiment
Integration with WebRTC gateways to enable AI voice interaction for website visitors
Open-source plugins and developer tools for easier integration and customization

If you're working on a similar VoIP+AI project, or want to integrate OpenAI with your SIP platform, feel free to connect or ask questions below. I’d love to collaborate or share more details.

Tags:

DEV Community: Ahad Khan

How to Build a SIP-to-OpenAI Voice Gateway in C#

What the gateway does

Try it with the C# sample

What you can build

How I Trained My Own On-Premises Speech-to-Speech AI System (Fully Offline)

What I Built

Here’s a summary of the steps:

Tech Stack

Pipeline Overview

Hardware Recommendations

What’s Next

Related Resources

How I Integrated OpenAI with a SIP Server and Built a Multilingual Smart AI Agent for IP Telephony

What We Built

Tech Stack

How the Integration Works

Real-World Use Cases

Related Resources

What’s Next

voip #openai #sip #ai