Muhammad Afzal Kalwar

Posted on Nov 1

Building an AI Outbound Calling Agent on Google Voice with Python, Groq and FastAPI

#python #ai #automation #fastapi

Most AI voice-agent tutorials begin the same way: sign up for a telephony API, get a webhook when the call connects, then stream audio. google-voice-dispatch-agent works differently. It runs real outbound calls through Google Voice in a Chrome browser, and that changes which problems are hard.

This post walks through how it's built, based on the project's own README. Code: https://github.com/mafzalkalwardev/google-voice-dispatch-agent

Why a browser makes this harder

A telephony API tells you when a call is answered. A browser doesn't. You read the page (the DOM), and the page is noisy: banners from the last call stay on screen, call controls can show up before anyone picks up, and voicemail cues can appear while the phone is still ringing.

So the AI is the easy part. The hard parts are:

detecting call state reliably
coping with DOM drift when Google Voice changes its markup
routing audio between the browser and the agent
getting voicemail timing right
staying stable over long unattended runs

The project is built around a few rules:

Never speak until the call is actually connected.
Never treat a burst of DOM noise as a finished call.
Respect a minimum ringing time.
Detect voicemail only after there is evidence of a connected call or a greeting.
Log everything so every call can be audited.
Run preflight checks before any live dialing.

The call-state gate

The most important logic is in src/google_voice.py. It's a conservative state gate:

Ringing can't jump straight to ended. A min_ring_seconds window has to pass before the agent accepts a connected, ended or voicemail transition.
Early signals are held, not trusted. If call controls appear too soon, the agent logs them and stays in the ringing state. Stale "ended" banners are ignored during the minimum ringing window, and so are voicemail cues while the phone is still ringing.
Connected needs stable evidence, such as answered-call controls or a running call timer.
The opening line plays only after a confirmed connection.
No answer is a timeout. max_ring_seconds ends the attempt only after the configured ring timeout.

Every decision is logged: current state, elapsed ringing time, DOM cues, answered controls, call-active status, voicemail cues, the audio classifier's result and the timeout reason. When something goes wrong, you can see exactly why the agent did what it did.

The session model and its CallState transitions are kept in their own file, call_session.py, so state handling can be tested apart from the browser.

The realtime conversation loop

Once a call is confirmed connected, conversation_loop.py takes over:

Capture: audio_capture.py records input and splits it into frames for voice activity detection (VAD), so the agent knows when the other person is talking.
Speech-to-text: speech goes to Groq STT.
LLM response: ai_groq.py generates the reply with Groq.
Text-to-speech: realtime_tts.py plays the reply with Edge TTS, with caching, and falls back to pyttsx3.
Routing: audio_routing.py finds Windows audio devices and sends playback into the call. VB-CABLE, sounddevice, soundcard and soundfile handle the audio plumbing.

Two details cut down awkward silence. The opening line and the voicemail message are generated before dialing, so they're ready the moment the state gate allows them. And voicemail_detector.py runs an audio classifier in the first seconds of a connected call, listening for greeting, beep and live-call signals. When it confirms voicemail, the loop hands off to voicemail playback instead of trying to hold a conversation with a recording.

One call, end to end

The README describes this runtime flow:

Load .env, dialer_config.json, the contacts and the CRM data.
Run preflight checks for Groq, the Chrome profile, the contacts, the callback number and the audio devices.
Launch Google Voice with a persistent Chrome profile (Selenium + webdriver-manager).
Prepare the opening-line and voicemail audio.
Dial one contact.
Poll detect_call_state() until the call is connected, voicemail, no answer, ended or failed.
Start the conversation loop only after a confirmed connection.
Watch for voicemail in the early connected window and hand off if it's confirmed.
Save the call log, transcript, recording, CRM update and call archive.
Wait out a cooldown, then move to the next contact.

The operator console (FastAPI)

web_app.py is a FastAPI app with Jinja2 templates and plain HTML, CSS and JavaScript. From one console an operator can:

edit settings and run preflight checks
test audio
start and watch live runs
browse logs, transcripts and recordings
manage leads and a carrier CRM

crm.py handles CRM enrichment and keeps separate archives for connected and failed calls. Contacts come from CSV or XLSX files and settings from JSON.

Packaging for Windows

The agent is meant to run on a Windows desktop, so:

desktop_app.py is the PyInstaller entry point, with a spec file in packaging/
PowerShell scripts in scripts/ install a desktop shortcut and a Windows Startup shortcut
there's an optional Inno Setup installer config in installer/

Tests

The tests/ folder covers state handling, mock dialing, the CRM, audio helpers, web routes and the realtime loop. It runs with pytest on GitHub Actions. With a browser-driven agent, mock dialing tests are what let you change the state gate without placing real calls.

Project layout

src/

  main.py               CLI runner, safe test, batch calling

  google_voice.py       Selenium automation + call-state detection

  call_session.py       Call session model and CallState transitions

  conversation_loop.py  Capture, VAD, STT, LLM, TTS, voicemail handoff

  voicemail_detector.py Greeting/beep/live-call classifier

  ai_groq.py            Groq responses

  realtime_tts.py       Edge TTS playback + cache

  audio_capture.py      Input capture and VAD framing

  audio_routing.py      Windows device discovery and routing

  crm.py                CRM, archives, transcripts, recordings

  web_app.py            FastAPI operator console

  desktop_app.py        PyInstaller entry point

Takeaways

With browser automation, state detection is the product. Be conservative and log every decision.
Prepare audio before you dial so the first words come out right away.
Classify voicemail with audio, not only page cues, and only after the call is connected.
Preflight checks catch missing keys, profiles and audio devices before a live run.

Whether calling through Google Voice fits your use case is up to you: Google's terms, consent and calling rules in your region are yours to check.

Source: https://github.com/mafzalkalwardev/google-voice-dispatch-agent
More about the call-state logic: https://dev.to/mafzalkalwardev/how-my-google-voice-ai-agent-knows-when-to-talk-call-state-detection-groq-and-tts-e5a

Muhammad Afzal Kalwar, Python Automation Engineer & Founder, INDUS Web Agency. Portfolio: https://mafzalkalwardev.github.io

DEV Community