DEV Community

Harpreet Singh Seehra
Harpreet Singh Seehra

Posted on

Browser Phone Calls with Real-Time AI Coaching

Click-to-Call WebRTC with AI Assist — a split-screen browser app where you make real phone calls from your browser and get live AI coaching tips in a sidebar. WebRTC telephony credentials, browser SpeechRecognition, and Telnyx AI Inference. One API key for calls and AI. No phone needed.

The Problem: Sales Calls Without Feedback

You are on a sales call. You are pitching, handling objections, trying to close. You have no idea if you are talking too fast, missing buying signals, or failing to ask for the next step. Your manager is not on the line. There is no coach whispering in your ear. You are flying blind.

The existing solutions are clunky. Call recording platforms store the audio for later review, but "later" means after the deal is lost. AI note-takers join as a third party on the call, which creeps out prospects and sometimes breaks compliance. Real-time coaching tools exist, but they require integrations with your dialer, your CRM, and your conferencing platform. Three vendors, three API keys, three bills.

The Click-to-Call WebRTC with AI Assist example fixes this in 75 lines of Python. Open a browser tab. Enter a phone number. Click Call. You are talking to a real phone number from your browser, and an AI coach is watching your transcript and feeding you actionable tips every 8 seconds in a live sidebar.

What It Does

Open http://localhost:5000 in your browser. You see a split-screen UI: a call panel on the left and an AI coaching sidebar on the right.

Enter a phone number and click Call. The backend creates a WebRTC telephony credential via POST /v2/telephony_credentials. The frontend uses the Telnyx WebRTC SDK to connect via SIP. The WebRTC call connects — you are now talking to a real phone number from your browser, no desk phone, no softphone app, just the browser.

While you talk, the browser SpeechRecognition API transcribes your speech in real time. Every 8 seconds, the transcript is sent to the /coaching endpoint, which calls Telnyx AI Inference and returns one actionable coaching tip. The tip appears in the right sidebar with a timestamp — live AI coaching during the call.

Click Hang Up to end. The call disconnects, transcription stops, coaching stops.

Step What happens Tech
1 User clicks Call Frontend sends number to backend
2 Backend creates WebRTC credential POST /v2/telephony_credentials
3 Frontend connects via WebRTC @telnyx/webrtc SDK, SIP registration
4 Call connects WebRTC audio to PSTN
5 Browser transcribes speech SpeechRecognition API, real-time
6 Every 8s, transcript sent to AI POST /v2/ai/chat/completions
7 AI returns coaching tip Model: moonshotai/Kimi-K2.6
8 Tip appears in sidebar DOM update with timestamp
9 User clicks Hang Up WebRTC disconnect, cleanup

The Architecture

Everything lives in one Flask file (~75 lines) and one HTML template. No database, no Redis, no WebSocket server. The browser handles speech recognition. The backend handles WebRTC credentials and AI inference.

Browser opens http://localhost:5000
        ↓
User enters number, clicks Call
        ↓
Frontend POST /webrtc/token → Backend creates telephony credential
        ↓
Backend returns SIP username, password, caller_number
        ↓
Frontend TelnyxRTC client.connect() via WebRTC
        ↓
WebRTC call connects to PSTN number
        ↓
Browser SpeechRecognition starts → live transcript in left panel
        ↓
Every 8s: transcript POST /coaching → AI Inference → coaching tip
        ↓
Tip appears in right sidebar with timestamp
        ↓
User clicks Hang Up → client.disconnect() → cleanup
Enter fullscreen mode Exit fullscreen mode

The Backend: Flask in 75 Lines

The backend has four endpoints. Here is the full app.py:

import os, time, requests
from flask import Flask, request, jsonify, render_template

app = Flask(__name__)

TELNYX_API_KEY = os.getenv("TELNYX_API_KEY")
AI_MODEL = os.getenv("AI_MODEL", "moonshotai/Kimi-K2.6")
WEBRTC_CONNECTION_ID = os.getenv("WEBRTC_CONNECTION_ID")
CALLER_NUMBER = os.getenv("CALLER_NUMBER")
INFERENCE_URL = "https://api.telnyx.com/v2/ai/chat/completions"

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/webrtc/token", methods=["POST"])
def webrtc_token():
    """Create a WebRTC telephony credential and return SIP credentials."""
    resp = requests.post(
        "https://api.telnyx.com/v2/telephony_credentials",
        headers={"Authorization": f"Bearer {TELNYX_API_KEY}",
                 "Content-Type": "application/json"},
        json={"connection_id": WEBRTC_CONNECTION_ID,
              "expires_secs": 3600},
        timeout=10)
    resp.raise_for_status()
    data = resp.json()["data"]
    return jsonify({
        "sip_username": data["sip_username"],
        "sip_password": data["sip_password"],
        "caller_number": CALLER_NUMBER
    })

@app.route("/coaching", methods=["POST"])
def coaching():
    """Send transcript to AI Inference and return a coaching tip."""
    transcript = request.json.get("transcript", "")
    if not transcript:
        return jsonify({"tip": "Start speaking to get coaching tips."})

    messages = [
        {"role": "system", "content":
            "You are a sales coach. Review the call transcript and give ONE "
            "actionable tip to improve the call. Be specific, concise, and "
            "practical. Focus on: asking better questions, handling objections, "
            "closing techniques, or tone adjustments. Return only the tip, no preamble."},
        {"role": "user", "content": f"Call transcript so far:\n{transcript}"}
    ]

    resp = requests.post(INFERENCE_URL,
        headers={"Authorization": f"Bearer {TELNYX_API_KEY}",
                 "Content-Type": "application/json"},
        json={"model": AI_MODEL, "messages": messages,
              "max_tokens": 150, "temperature": 0.7},
        timeout=15)
    resp.raise_for_status()
    tip = resp.json()["choices"][0]["message"]["content"]
    return jsonify({"tip": tip, "timestamp": time.strftime("%H:%M:%S")})

@app.route("/health")
def health():
    return jsonify({"status": "ok"})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=int(os.getenv("PORT", 5000)))
Enter fullscreen mode Exit fullscreen mode

That is the entire backend. Four endpoints, one inference call, one credential creation call. No session management, no database, no background workers.

WebRTC Telephony Credentials

The /webrtc/token endpoint creates a temporary SIP credential via POST /v2/telephony_credentials:

resp = requests.post(
    "https://api.telnyx.com/v2/telephony_credentials",
    headers={"Authorization": f"Bearer {TELNYX_API_KEY}",
             "Content-Type": "application/json"},
    json={"connection_id": WEBRTC_CONNECTION_ID,
          "expires_secs": 3600},
    timeout=10)
Enter fullscreen mode Exit fullscreen mode

The connection_id links the credential to your Telnyx WebRTC connection (configured in the portal). The credential expires in 1 hour. The response contains sip_username and sip_password, which the frontend uses to register with Telnyx's SIP server over WebRTC.

This is the key abstraction: Telnyx handles the SIP infrastructure, the STUN/TURN servers, the NAT traversal, and the PSTN gateway. Your code just asks for a credential and passes it to the WebRTC SDK.

AI Coaching Endpoint

The /coaching endpoint sends the transcript to Telnyx AI Inference and returns a coaching tip:

messages = [
    {"role": "system", "content":
        "You are a sales coach. Review the call transcript and give ONE "
        "actionable tip to improve the call. Be specific, concise, and "
        "practical. Focus on: asking better questions, handling objections, "
        "closing techniques, or tone adjustments. Return only the tip, no preamble."},
    {"role": "user", "content": f"Call transcript so far:\n{transcript}"}
]

resp = requests.post(INFERENCE_URL,
    headers={"Authorization": f"Bearer {TELNYX_API_KEY}",
             "Content-Type": "application/json"},
    json={"model": AI_MODEL, "messages": messages,
          "max_tokens": 150, "temperature": 0.7},
    timeout=15)
Enter fullscreen mode Exit fullscreen mode

The system prompt is specific: one tip, actionable, no preamble. Temperature is 0.7 — high enough for creative, varied coaching tips, low enough that they stay relevant. Max tokens 150 — enough for a sentence or two, not enough for the model to ramble.

The default model is moonshotai/Kimi-K2.6, but you can override via the AI_MODEL environment variable. Any model available on Telnyx AI Inference works.

The Frontend: Browser WebRTC + SpeechRecognition

The frontend is a single HTML template served by Flask. It loads the Telnyx WebRTC SDK from CDN and uses the browser's native SpeechRecognition API.

WebRTC Connection with TelnyxRTC 

<script src="https://unpkg.com/@telnyx/webrtc@2/dist/telnyx.js"></script>
<script>
const client = new TelnyxRTC({
    login: sipUsername,
    password: sipPassword,
    ringtoneFile: null,
    ringbackFile: null
});

client.on('telnyx.ready', () => {
    statusBadge.textContent = 'Ready';
    statusBadge.className = 'badge ready';
});

client.on('telnyx.error', (error) => {
    statusBadge.textContent = 'Error: ' + error.message;
});

async function startCall() {
    const resp = await fetch('/webrtc/token', {method: 'POST'});
    const creds = await resp.json();
    sipUsername = creds.sip_username;
    sipPassword = creds.sip_password;
    callerNumber = creds.caller_number;

    client.login = sipUsername;
    client.password = sipPassword;
    client.connect();

    client.on('telnyx.ready', () => {
        const call = client.newCall({
            destinationNumber: document.getElementById('phoneNumber').value,
            callerNumber: callerNumber
        });
        activeCall = call;
        startTranscription();
        startCoachingLoop();
        startTimer();
    });
}
</script>
Enter fullscreen mode Exit fullscreen mode

The TelnyxRTC client handles SIP registration, ICE negotiation, and media setup. You pass it the SIP username and password from /webrtc/token, call connect(), and wait for the telnyx.ready event. Then you create a new call with newCall({destinationNumber, callerNumber}).

Browser SpeechRecognition for Live Transcript 

const recognition = new (window.SpeechRecognition || window.webkitSpeechRecognition)();
recognition.continuous = true;
recognition.interimResults = true;
recognition.lang = 'en-US';

let transcript = '';

recognition.onresult = (event) => {
    let interim = '';
    for (let i = event.resultIndex; i < event.results.length; i++) {
        const result = event.results[i];
        if (result.isFinal) {
            transcript += result[0].transcript + ' ';
            document.getElementById('transcript').textContent = transcript;
        } else {
            interim += result[0].transcript;
        }
    }
    document.getElementById('interim').textContent = interim;
};

function startTranscription() {
    transcript = '';
    recognition.start();
}
Enter fullscreen mode Exit fullscreen mode

The SpeechRecognition API is built into Chrome and Safari. It streams partial results in real time (interimResults: true) and finalizes phrases when the speaker pauses. The transcript accumulates in a string that is displayed in the left panel and sent to the coaching endpoint every 8 seconds.

The AI Coaching Loop 

let coachingInterval;

function startCoachingLoop() {
    coachingInterval = setInterval(async () => {
        if (!transcript.trim()) return;
        const resp = await fetch('/coaching', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({transcript})
        });
        const data = await resp.json();
        addCoachingTip(data.tip, data.timestamp);
    }, 8000);
}

function addCoachingTip(tip, timestamp) {
    const sidebar = document.getElementById('coachingSidebar');
    const div = document.createElement('div');
    div.className = 'coaching-tip';
    div.innerHTML = `<span class="timestamp">${timestamp}</span> ${tip}`;
    sidebar.appendChild(div);
    sidebar.scrollTop = sidebar.scrollHeight;
}
Enter fullscreen mode Exit fullscreen mode

Every 8 seconds, the current transcript is sent to /coaching. The AI reviews the conversation so far and returns one tip. The tip is appended to the sidebar with a timestamp. The sidebar auto-scrolls so the latest tip is always visible.

The 8-second interval is a tradeoff. Shorter intervals mean more responsive coaching but more API calls and higher latency (the AI needs context to give good tips). Longer intervals mean fewer calls but delayed feedback. Eight seconds hits the sweet spot for sales calls: enough context for meaningful tips, frequent enough to feel live.

Call Timer and Status Badges 

let callStartTime;
let timerInterval;

function startTimer() {
    callStartTime = Date.now();
    timerInterval = setInterval(() => {
        const elapsed = Math.floor((Date.now() - callStartTime) / 1000);
        const mins = String(Math.floor(elapsed / 60)).padStart(2, '0');
        const secs = String(elapsed % 60).padStart(2, '0');
        document.getElementById('timer').textContent = `${mins}:${secs}`;
    }, 1000);
}
Enter fullscreen mode Exit fullscreen mode

The call timer starts when the call connects and updates every second. Status badges show Ready, Connecting, On Call, and Ended states. These are small UX touches that make the app feel like a real dialer.

One API Key for WebRTC and AI

The entire app uses a single TELNYX_API_KEY:

  • WebRTC credentials — POST /v2/telephony_credentials (via requests.post with Bearer token)
  • AI Inference — POST /v2/ai/chat/completions (via requests.post with Bearer token)
  • Call delivery — handled by Telnyx's PSTN gateway (no additional API calls)

The SDK handles the WebRTC media layer. The two requests.post calls handle credentials and AI directly. No third-party transcription service, no separate LLM provider, no conferencing platform. One network, one key, one bill.

Environment Variables 

TELNYX_API_KEY=your_api_key_here
AI_MODEL=moonshotai/Kimi-K2.6
WEBRTC_CONNECTION_ID=your_connection_id
CALLER_NUMBER=+12155551234
PORT=5000
Enter fullscreen mode Exit fullscreen mode
  • TELNYX_API_KEY — your Telnyx API v2 key
  • AI_MODEL — any model on Telnyx AI Inference (default: moonshotai/Kimi-K2.6)
  • WEBRTC_CONNECTION_ID — the connection ID from your Telnyx Portal WebRTC configuration
  • CALLER_NUMBER — the caller ID for outbound calls (must be a Telnyx number on your account)
  • PORT — HTTP port (default: 5000)

Try It Yourself 

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/click-to-call-webrtc-with-ai-assist-python
cp .env.example .env   # add TELNYX_API_KEY, WEBRTC_CONNECTION_ID, CALLER_NUMBER
pip install -r requirements.txt
python app.py           # starts on http://localhost:5000
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:5000 in Chrome or Safari. Enter a phone number. Click Call. Start talking. Watch the coaching tips appear.

Key links:

Top comments (0)