Giving "Hands" to AI Voice: Building a Self-Hosted, Tool-Capable Agent with Asterisk

Haider Jarral — Mon, 02 Feb 2026 03:53:13 +0000

Most AI voice platforms today are "SaaS black boxes." You send your audio to a proprietary API, pay a per-minute tax, and pray the latency stays low. As developers, we lose three things: Privacy, Visibility, and Control.

I’ve been working on the Asterisk AI Voice Agent—an open-source project designed to bring the "Home Assistant" philosophy to telephony. In my latest release, I’ve moved beyond simple "chatting" bots to create an agent that can actually do work using a tri-stage tooling architecture.

The Architecture: Why Asterisk?
For the Linux-savvy developer, the stack is straightforward but powerful:

Asterisk: The battle-tested telephony engine.
Python/Node.js Logic: Handling the bridge between audio streams and LLMs.
REST Webhooks: The universal language for "Skills."

By running this locally (or on your own VPS), you eliminate the per-minute SaaS cost and keep your voice data within your own network.

The New Feature: Tool-Calling (Pre, In, and Post-Call)
The real "magic" happens when the AI can interact with your existing stack. I’ve implemented three specific hooks using standard REST webhooks:

1. Pre-Call (Data Enrichment)
Before the agent even says "Hello," it performs a lookup. If the Caller ID matches a record in your CRM or database, the agent is initialized with that context.

Example: "Hi [User Name], I see your last order was [Order #]..."

2. In-Call (Real-Time Agency)
During the conversation, the LLM can decide to trigger a tool. Since these are standard REST calls, your agent can:

Query a live calendar.
Toggle a smart home device.
Check shipping status via an external API.

3. Post-Call (The Automation Hand-off)
Once the call ends, the agent summarizes the interaction and pushes it to an endpoint like n8n, Make, or a custom webhook. I personally use this to send call summaries directly to WhatsApp via an n8n workflow.

See the Implementation
I’ve documented the full setup and a demo of these tools in action here:

Join the Project (Open Source)
This is a project for builders. If you are tired of proprietary voice wrappers and want to contribute to a universal open-source voice AI platform, come join us.

GitHub Repo: hkjarral/Asterisk-AI-Voice-Agent

Status: Looking for contributors to help build more "Skills" and refine the core engine.

I’d love to hear your thoughts: What is the first "Skill" you would give a self-hosted voice agent? A private receptionist? A voice-activated server monitor? Let’s discuss in the comments.

Building a Production-Ready AI Call Center with Asterisk in 15 Minutes

Haider Jarral — Thu, 06 Nov 2025 18:32:56 +0000

Traditional IVR (Interactive Voice Response) systems are dying. For decades, they have been the frontline of customer service, but their rigid menus and frustrating "press 1 for sales" loops are a relic of a bygone era. Customers today expect instant, intelligent, and natural interactions. If your business still relies on a traditional IVR, you are likely losing customers to competitors who have already embraced conversational AI.

The good news is that building a powerful, AI-driven voice agent is no longer the exclusive domain of large corporations with massive budgets. Thanks to the vibrant open-source community and the power of modern AI, you can build and deploy a production-ready AI call center with Asterisk in about 15 minutes. This guide will show you how.

The Problem with Traditional IVRs

Before we dive into the solution, let's be clear about the problem. Traditional IVRs fail because they are not intelligent. They are based on a simple, tree-like logic that cannot handle the complexity of human conversation. This leads to a number of common frustrations:

Limited Options: Callers are forced to choose from a predefined menu, which often doesn't include their specific issue.
No Natural Language Understanding: IVRs cannot understand the nuances of human speech, leading to misunderstandings and repeated requests.
High Abandonment Rates: Frustrated callers often hang up and seek support elsewhere, leading to lost revenue and customer churn.
Poor Customer Experience: A frustrating IVR experience can damage your brand reputation and erode customer loyalty.

In short, traditional IVRs are a bottleneck, not a solution. It's time for a change.

The Solution: Asterisk + AI

By combining the power of Asterisk, the world's most popular open-source PBX, with modern AI technologies, we can create an intelligent voice agent that overcomes the limitations of traditional IVRs. Our open-source project, Asterisk AI Voice Agent, provides the bridge between your Asterisk server and the world of conversational AI.

How It Works: The Architecture

Our voice agent uses a modular, real-time architecture to process calls. Here's a high-level overview:

Asterisk Call Handling: An incoming call is received by your Asterisk server.
ARI & ExternalMedia: Asterisk's REST Interface (ARI) and the externalMedia application are used to send the call's audio to our Python-based voice agent in real-time.
Speech-to-Text (STT): The agent transcribes the caller's speech into text using either a cloud-based service (like Deepgram) or a local, open-source model (like Whisper).
Large Language Model (LLM): The transcribed text is sent to a large language model (like OpenAI's GPT-4o or a local Llama 3 model) to generate an intelligent response.
Text-to-Speech (TTS): The LLM's response is converted back into natural-sounding speech using a TTS engine (like ElevenLabs or a local Piper model).
Real-time Audio Streaming: The synthesized audio is streamed back to the caller through Asterisk, creating a seamless, conversational experience.

This entire process happens in near real-time, with a typical round-trip latency of under a second, making the conversation feel natural and fluid.

Quick Start: Your AI Call Center in 15 Minutes

Ready to build your own? Here's a quick-start guide to get you up and running. This tutorial assumes you have a working Asterisk server and are familiar with basic Linux commands.

Step 1: Clone the Repository (2 minutes)

First, clone the Asterisk AI Voice Agent repository from GitHub:

git clone https://github.com/hkjarral/Asterisk-AI-Voice-Agent.git
cd Asterisk-AI-Voice-Agent

Step 2: Configure Your Environment (5 minutes)

Next, you'll need to configure your environment variables. We provide a simple .env.example file to get you started.

cp .env.example .env
nano .env

You will need to fill in the following:

Asterisk ARI credentials
Your choice of STT, LLM, and TTS providers (we support both cloud and local options)
API keys for your chosen providers

Step 3: Run with Docker Compose (5 minutes)

We've made deployment incredibly simple with Docker Compose. From the project's root directory, simply run:

docker-compose up -d

This will build the Docker image, install all dependencies, and start the voice agent in the background. Your AI call center is now running!

Step 4: Configure Your Asterisk Dialplan (3 minutes)

Finally, you need to tell Asterisk to send calls to your new AI agent. Add the following to your extensions.conf dialplan:

exten => 1234,1,Answer()
same => n,Stasis(voice-agent)
same => n,Hangup()

Now, when someone calls extension 1234, they will be connected to your AI voice agent. It's that simple!

Real-World Use Cases

An AI-powered voice agent can transform your business in a number of ways:

24/7 Customer Support: Provide instant, intelligent support to your customers, any time of day or night.
Automated Appointment Booking: Allow callers to book, reschedule, or cancel appointments without human intervention.
Intelligent Call Routing: Understand the caller's intent and route them to the correct department, every time.
Lead Qualification: Automatically qualify inbound sales leads and schedule follow-up calls.
Healthcare Triage: Help patients with common medical questions and direct them to the appropriate level of care.

What's Next? Join the Community!

This is just the beginning. The world of open-source voice AI is moving at an incredible pace, and we are constantly improving the Asterisk AI Voice Agent project. We invite you to join our growing community of developers and help us build the future of conversational AI.

⭐ Star us on GitHub: https://github.com/hkjarral/Asterisk-AI-Voice-Agent
💬 Join our Discord: https://discord.gg/CAVACtaY
📝 Read our Blog: https://hkjarral.github.io/Asterisk-AI-Voice-Agent/

Together, we can finally put an end to the frustrating IVR and create a world where every phone call is an intelligent, natural, and productive conversation.

DEV Community: Haider Jarral