DEV Community

Sanchita Sunil
Sanchita Sunil

Posted on

Building a fully autonomous AI Receptionist

Quick links.
Code: https://github.com/murf-ai/murf-cookbook/tree/main/examples/agents/reception-agent
Video Walkthrough:https://youtu.be/eCbejEt78sw

We have spent years building interfaces that require people to learn them – apps, chatbots, dashboards – and the entire time, the most natural interface we have ever had, the phone call, has been sitting there, mostly automated by hold music and press-one-for-billing menus.

Voice is the interface that needs no manual – it works across every age group, every literacy level and every level of technical comfort. And for most people, when something actually matters – a doctor's appointment, a flight change, a question that needs a real answer – they pick up the phone.

But the reality of picking up the phone is mostly frustrating automated systems. The gap between those and a real conversation is what we’ll be bridging today.

This tutorial builds a fully autonomous AI voice receptionist. You call a real Twilio phone number, an AI answers, recognizes you if you've called before, books appointments into a real database, and answers questions from a local knowledge base. The whole thing runs in a few hundred lines of Python.

I’ve built it as a receptionist for a clinic, but you can adapt this for a restaurant, a salon, a support desk or any other business, just by making a few minor tweaks.

We’ll be building this in three layers:
-The Core Voice Loop: Giving our agent a voice and wiring it up to answer phone calls
-Memory and Booking: Giving it memory and the ability to book appointments
-RAG: Adding a RAG to make sure the agent doesn’t hallucinate any details
In the end we’ll also go over some additions you can make to make the agent even better (like Google Calendar blocking, Whatsapp Confirmations and Call Transfer), some common errors and how to adapt this for any business.

We’ll dive into each component and their setups first and then get into the code. By the end of it, we will have a fully autonomous AI voice receptionist that you can call, talk to and hang-up with a booking confirmed.

Table of Contents

The Stack

There are 5 components:

Architecture

Livekit

When we wire up a voice agent ourselves - transcribing an input audio, putting it through an LLM and then synthesizing a response through a TTS, it adds about 2-4 seconds of delay. We could use a Websocket, but while websockets work well with text and data, it could cause stutters and pauses with voice.

Livekit is a streaming infrastructure that helps us bring this latency down to what feels natural in a conversation, without the issues caused by a Websocket. It uses WebRTC, which is designed specifically to stream conversations.

Setup:

  1. Create an account at Livekit and create a project.
  2. From your project settings, copy the url, api key, api secret and sip uri and drop them into your .env file.
  3. On the console, go to Telephony > SIP Trunks and create a new trunk.
  4. Give the trunk a name, select inbound, set the allowed IP addresses(set to 0.0.0.0/0 if you want to allow all IP addresses)
  5. Leave the phone numbers blank for now, we’ll fill that in once we’ve created the twilio account.
  6. If you switch to the JSON editor, your json should look something like this:
{
   “name”: “trial”,
   “allowedAddresses”: [
    0.0.0.0/0
   ]
}
Enter fullscreen mode Exit fullscreen mode
  1. Go to Telephony > Dispatch Rules and create a new dispatch rule.
  2. Give your rule a name, a prefix and an agent name. Remember your agent name, it must match the one in your code, or the call will connect but all you’ll hear is silence.
  3. If you switch to the json editor, your dispatch rule should look something like this:
{
  "sipDispatchRuleId": "SDR_abc123",
  "rule": {
    "dispatchRuleIndividual": {
      "roomPrefix": "clinic-"
    }
  },
  "trunkIds": [
    "ST_abc123"
  ],
  "name": "clinic",
  "roomConfig": {
    "agents": [
      {
        "agentName": "clinic-agent"
      }
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

Copy this and save it as dispatch-rule.json in your project root directory.

Twilio

Connecting your software to a phone line or sending an SMS would mean negotiating contracts with multiple telecom carriers, configuring physical servers, and managing telecom protocols - a grueling process that could take months.

Twilio is a cloud communications platform that turns the global telecom network into simple software APIs. It acts as a digital bridge between the internet and phone networks, abstracting all of that hardware and routing complexity behind the scenes, allowing us to make real phone calls, send messages or even build an entire customer support system with just a few lines of code.

Setup:

  1. Create an account at console.twilio.com or 1console.twilio.com. Copy your account sid, auth token and your phone number and paste them into the .env file.
  2. On the console, go to TwiML bins and create a bin.
  3. Give it a name and put this in the TwiML section:
<?xml version="1.0" encoding="UTF-8"?>   
<Response>
  <Dial>
    <Sip>sip:phone_number@sip_uri;transport=tcp</Sip>
  </Dial>
</Response>
Enter fullscreen mode Exit fullscreen mode

Replace phone_number with your twilio phone number and sip_uri with you SIP URI.
For example:
If your phone number is +1234567 and your sip uri is sip:abc123.sip.livekit.cloud, then your TwiML section would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Dial>
    <Sip>sip:1234567@abc123.sip.livekit.cloud;transport=tcp</Sip>
  </Dial>
</Response>
Enter fullscreen mode Exit fullscreen mode
  1. On 1console.twilio.com, go to Products & Services > Numbers & Senders, click on your phone number and in the Voice and emergency calling section, click on edit configuration details.
  2. Select your configuration method as the one with TwiML bins in it, set your primary method as TwiML bins and select the TwiML bin you just created.
  3. Now that your Twilio number is active and routing via TwiML, go back to your LiveKit SIP Trunk > Configure trunk and add this phone number to the "Phone Numbers" field so LiveKit knows to accept its calls. If you phone number is +1234567, when you switch to the json editor, it should look something like this now:
{
  "numbers": [
    "+1234567",
    "1234567"
  ],
  "allowedAddresses": [
    "0.0.0.0/0"
  ]
}
Enter fullscreen mode Exit fullscreen mode
  1. (Optional) To get the confirmation messages on Whatsapp, use the number given in the .env.example(+14155238886 - it must be written in the .env file as “whatsapp:+14155238886”). Then go to console.twilio.com, Messages > Send a Whatsapp message, and follow the instructions. The connection resets every 5 days, so the recipients must send the join message again to receive messages. For production, get a Whatsapp Business number from Twilio and use that instead, the sandbox is free and only meant for testing.

Speech-To-Text(STT)

Whisper Realtime(OpenAI’s latest realtime speech-to-text model: gpt-realtime-whisper)

OpenAI recently expanded its ecosystem by introducing a dedicated streaming speech-to-text model built directly into its real-time infrastructure. If you’ve used previous Whisper models, you know the frustrating delay caused by waiting to process an entire block of input audio at once. This new model allows for true continuous streaming - meaning the input audio is transcribed into text word-by-word as it arrives, rather than waiting for the speaker to finish before transcription begins.

In this project, Realtime Whisper is used as our primary STT model. Note that you will require a paid OpenAI API key to use it, as it isn't available on the free tier. You can get your API key at Openai.

You also have the option to use Deepgram’s Nova-3 as your STT for this project.

Deepgram(Nova-3)
If you want to make the switch, all you have to do is change the STT_PROVIDER in the .env file to deepgram. Deepgram offers free credits of $200 when your account is created, which is more than enough to test and run this project.

Another advantage that comes with using Deepgram is that it is optimized to handle noisy environments, allowing it to distinguish between the caller’s voice and the background noise. You can get your api key at Deepgram.


LLM

You will require a Large Language Model to act as the central reasoning engine for your voice agent. This model is responsible for parsing the transcribed text, deciding how to respond, and triggering database or calendar actions.
This project is built modularly and configured to support two major providers. To choose yours, you simply need to set the LLM_PROVIDER variable in your .env file to one of the following:
-openai (Requires a paid API key)
-gemini (An excellent choice to test with, Google offers a free tier)
If you wish to use a different provider, the code is structured so you can wire in any custom LLM.

Text-To-Speech(TTS)

Once our LLM decides exactly what to say, we need to convert that text back into natural human speech to stream down the phone line. In a live conversation, timing is everything. If the Text-to-Speech engine takes even a second to generate the audio, the interaction breaks down.

Murf Falcon

For this project, we are using Murf Falcon, the consistently fastest TTS model(130ms TIme To First Audio) built specifically for real-time conversational applications. Falcon is optimized to generate lifelike, natural-sounding human speech on the fly.
It achieves this by supporting continuous chunked audio streaming - The moment our LLM streams its first few words, Falcon immediately starts converting those tokens into audio packets and pushing them back into the LiveKit media stream. This ensures that the caller hears a seamless, expressive response without any awkward pauses.
Go to Murf, create an account and get your API key.

Supabase

For an AI receptionist to be genuinely useful, simply talking isn’t enough, it needs to be able to remember. We need a secure place to store and identify returning callers, existing appointments and log new appointments.

Supabase is an open-source Firebase alternative built on top of PostgreSQL. It eliminates that backend friction by giving you a fully managed, highly scalable Postgres database with an instant API.

In this project, Supabase acts as our system of record. Every time the Reception Agent takes a call, it queries Supabase to check if the caller's phone number is already in the system, logs the conversation context, and securely writes the new appointment details.

Setup:

  1. Create an account at Supabase and create a project.
  2. From your project dashboard, copy your project url and anon key and paste them into your .env file.

Google Calendar*(optional)*

Human staff need visibility into the actions the agent takes. We can achieve this by wiring our agent to mirror its bookings directly onto Google Calendar.

In this project, Google Calendar acts strictly as a write-only visual layer for clinic staff. It does not drive the core availability or booking logic, that is handled entirely by our Supabase database. If Google Calendar goes down, the agent will still book callers successfully because it updates Supabase first, then fires off an asynchronous background task to update Google Calendar without making the caller wait on the line.

Setup:

  1. Create a Project on Google Cloud Console.
  2. In the Project API settings, enable Google Calendar.
  3. Go to IAM & Admin > Service Accounts and click Create Service Account. 4. Give it a descriptive name and click Done.
  4. Select your service account and create a json key.
  5. Save this json key to the project root as service-account.json
  6. Go to Google Calendar, under Other Calendars, create new calendars. In this project, we create a calendar each for the doctors.
  7. In the created calendars, go to the Shared With section and Add People - add the service account ID and make sure to give it access to make changes to events.
  8. In the Integrate Calendar section, copy the calendar ID and drop it into the .env.

We’ve gone over all the major components for our project. Now that we have everything configured and all our credentials in place, let’s dive into building the agent.

The Core Voice Loop

We will start by getting the agent to successfully connect to a call, speak, and adopt a persona.

Before we write the logic for our voice agent, let’s start by installing the necessary packages and prepping our environment.

Run these in your terminal:

python -m venv venv
Enter fullscreen mode Exit fullscreen mode

On MacOS/Linux:

source venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

On Windows(powershell):

venv\Scripts\Activate.ps1
Enter fullscreen mode Exit fullscreen mode

Create a requirements.txt file in your project root and add the following packages:

livekit-agents>=1.0.0
livekit-plugins-deepgram>=0.7.0
livekit-plugins-openai>=0.10.0
livekit-plugins-google>=0.6.0
livekit-plugins-silero>=0.7.0
livekit-murf>=0.1.0
python-dotenv>=1.0.0
twilio>=9.0.0
livekit-api>=0.7.0
supabase>=2.0.0
lancedb>=0.6.0
sentence-transformers>=3.0.0
google-api-python-client>=2.100.0
google-auth-httplib2>=0.2.0
google-auth-oauthlib>=1.2.0
pytz>=2024.1
Enter fullscreen mode Exit fullscreen mode

This covers Livekit, Twilio and all our chosen plugins.
Now, run the command below to install all of them:

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

System Prompt

Let’s define how our receptionist should behave. In the root of your project, create a folder named prompts and create system_prompt.py within that folder. By keeping the prompt in a separate file, we can easily make tweaks to the persona or change the it as a whole if we want to.

prompts/system_prompt.py:

SYSTEM_PROMPT = """
You are Matthew, the AI receptionist for The Clinic.
Always identify yourself as an AI at the start of every call.
Use a warm, calm, professional tone.

Your opening line at the start of every call:
Hello, thank you for calling The Clinic. I'm Matthew, your AI receptionist. How may I help you today?

Voice rules — follow on every response:
- Keep responses to 1–3 sentences
- No lists, bullet points, or markdown — this is a phone call
- Speak naturally
- Never say "Absolutely!" or "Great question!" — say "Of course" or "Certainly" instead

You can answer general questions about the clinic and help callers.
If you don't know something, say you'll have the team call them back.
""".strip()

def build_system_prompt(patient: dict | None = None) -> str:
    return SYSTEM_PROMPT
Enter fullscreen mode Exit fullscreen mode

Core Logic

Coming to the core logic. Create agent.py in your project’s root, this will be the file that wires together and controls the core functionalities of the agent.

agent.py:

from __future__ import annotations
import os
import asyncio
import logging


from dotenv import load_dotenv
from livekit import rtc
from livekit.agents import (
    Agent,
    AgentSession,
    AutoSubscribe,
    JobContext,
    JobProcess,
    WorkerOptions,
    cli,
)


from livekit.plugins import deepgram, google, openai, silero, murf
Enter fullscreen mode Exit fullscreen mode

Import our receptionist's personality instructions:

from prompts.system_prompt import build_system_prompt
Enter fullscreen mode Exit fullscreen mode

1. ENVIRONMENT & CONFIGURATION

Load the .env file so we have access to our API keys without hardcoding them.

load_dotenv()
Enter fullscreen mode Exit fullscreen mode

Set up logging so we can see what the agent is doing in the terminal:

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("clinic-agent")
Enter fullscreen mode Exit fullscreen mode

Fetch our environment variables:

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
MURF_API_KEY = os.getenv("MURF_API_KEY")
Enter fullscreen mode Exit fullscreen mode

2. AGENT DEFINITION

In a normal phone call, the person picking up always speaks first. If the AI waits for the user, the caller will think the line is dead. We hardcode the opening line so we can trigger it immediately on connection:

OPENING_LINE = (
    "Hello, thank you for calling The Clinic. I'm Matthew, your AI receptionist. "
    "How may I help you today?"
)
Enter fullscreen mode Exit fullscreen mode

We pass in the instructions from our system prompt so the LLM knows it is acting as a clinic receptionist. Later on, this is where we will attach specific tools (like calendar booking):

class ClinicAgent(Agent):
    def __init__(self, instructions: str) -> None:
        super().__init__(instructions=instructions)
Enter fullscreen mode Exit fullscreen mode

AI models suffer from 'Cold Starts'. If we wait until the user speaks to load the Voice Activity Detection (VAD) model into memory, it adds a massive delay to the first response. This function pre-loads the Silero VAD weights the moment the server starts, eliminating that lag:

def prewarm(proc: JobProcess) -> None:
    proc.userdata["vad"] = silero.VAD.load()
Enter fullscreen mode Exit fullscreen mode

LiveKit rooms generated by Twilio SIP trunks usually start with a specific prefix. We check this so the agent knows whether it's talking to a real phone line or just a local web-testing environment:

def _is_phone_room(room_name: str) -> bool:
    return room_name.startswith("clinic-") and not room_name.startswith("clinic-test-")
Enter fullscreen mode Exit fullscreen mode

3. THE CORE WORKER LOOP

This is the engine of the voice agent. Every time a new phone call comes in, LiveKit fires up a new instance of this entrypoint to handle the conversation:

async def entrypoint(ctx: JobContext) -> None:
    is_phone = _is_phone_room(ctx.room.name)
Enter fullscreen mode Exit fullscreen mode

Connect to the LiveKit room. Since it's a phone call, we only need audio:

        await ctx.connect(
        auto_subscribe=AutoSubscribe.AUDIO_ONLY if is_phone else AutoSubscribe.SUBSCRIBE_ALL,
    )
Enter fullscreen mode Exit fullscreen mode

A: Speech-To-Text (Ears)
I’m going to be using Whisper Realtime. (To use Deepgram: stt = deepgram.STT(model="nova-3", language="en", api_key=DEEPGRAM_API_KEY)

            stt = openai.STT(
            model="gpt-realtime-whisper",
            use_realtime=True,
            language="en",
            api_key=OPENAI_API_KEY,
        )
Enter fullscreen mode Exit fullscreen mode

B: Large Language Model (Brain)
I’m going to be using OpenAI with gpt-4o-mini, you can swap this for any other LLM.
For example, to use Gemini: llm = google.LLM(model=”gemini-2.5-flash”, api_key=GEMINI_API_KEY)

llm = openai.LLM(model="gpt-4o-mini",api_key=OPENAI_API_KEY)
Enter fullscreen mode Exit fullscreen mode

C: Text-To-Speech (Mouth)
Murf Falcon converts our LLM's text back into audio. We select a voice id based on our requirement. I’m going to use Matthew:

    tts = murf.TTS(voice="en-US-matthew", locale="en-US")
Enter fullscreen mode Exit fullscreen mode
  1. The Session Conductor

AgentSession takes our Ears, Brain, and Mouth and wires them into a single
continuous streaming loop. It automatically handles turn-taking, so if the
user interrupts, the session stops the TTS instantly and starts listening again:

    session = AgentSession(
        stt=stt,
        llm=llm,
        tts=tts,
        vad=ctx.proc.userdata["vad"],
    )
Enter fullscreen mode Exit fullscreen mode

Load our receptionist instructions and start the session in the room:

    prompt = build_system_prompt()
    await session.start(ClinicAgent(prompt), room=ctx.room)
Enter fullscreen mode Exit fullscreen mode

4. CALL EXECUTION & GREETING

Wait up to 20 seconds for the Twilio SIP trunk to bridge the audio stream. If no one connects, we gracefully log the error and shut down this instance:

    if is_phone:
        try:
            participant = await asyncio.wait_for(ctx.wait_for_participant(), timeout=20.0)
            session.room_io.set_participant(participant.identity)
        except asyncio.TimeoutError:
            logger.error("No caller joined within 20s")
            return
Enter fullscreen mode Exit fullscreen mode

Play the opening line immediately. Setting allow_interruptions=False ensures the agent finishes saying "Hello" completely before it starts listening for the caller's response:

        handle = session.say(OPENING_LINE, allow_interruptions=False)
        await handle.wait_for_playout()
Enter fullscreen mode Exit fullscreen mode

Keep the worker process alive as long as the phone call is active:

    while ctx.room.isconnected():
        await asyncio.sleep(0.25)
Enter fullscreen mode Exit fullscreen mode

5. SERVER INITIALIZATION

Start the LiveKit worker. The agent_name MUST match your LiveKit Dispatch Rule otherwise Twilio will route the call, but this script won't know to pick it up, so all you'll hear is silence:

if __name__ == "__main__":
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
            agent_name="clinic-agent",
            num_idle_processes=1,
        )
    )
Enter fullscreen mode Exit fullscreen mode

We can now run the agent using the following command:

python agent.py start
Enter fullscreen mode Exit fullscreen mode

Watch the logs on the terminal. Once you see registered worker, pick up your phone and dial your Twilio phone number, your agent will answer and hold a conversation with you.

Memory & Appointment Booking

Having a receptionist who only speaks to you, can’t take any action, and immediately forgets the conversation the second you hang up wouldn’t be of much use to a business.

In this layer, we are going to give our agent memory and actionable capabilities so it can recognize returning callers, greet them by name, and securely lock time slots without double-booking.

Database Setup

First, let's create tables where our agent will store the details. In your Supabase dashboard, navigate to the SQL Editor and run the following query:

create extension if not exists "pgcrypto";

-- 1. Caller Memory
create table if not exists customers (
    phone                   text primary key,
    name                    text,
    preferred_doctor        text,
    last_booking_id         text,
    last_appointment_date   text,
    last_appointment_time   text,
    call_count              integer not null default 1,
    first_seen              timestamptz not null default now(),
    last_seen               timestamptz not null default now()
);

-- 2. Availability + Bookings
create table if not exists slots (
    id              uuid primary key default gen_random_uuid(),
    doctor          text not null,
    iso_date        date not null,
    iso_time        time not null,
    status          text not null default 'available',
    booking_id      text,
    patient_name    text,
    phone           text,
    reason          text,
    cancelled_at    timestamptz,
    created_at      timestamptz not null default now(),
    constraint slots_status_check check (status in ('available', 'booked')),
    -- CRITICAL: If you change the doctors in your Python code, you MUST update them here!
    constraint slots_doctor_check check (
        doctor in ('Dr. Sarah Lin', 'Dr. James Cole')
    ),
    constraint slots_unique_doctor_datetime unique (doctor, iso_date, iso_time)
);

-- 3. Booking Audit Log
create table if not exists appointments (
    id               text primary key,
    phone            text not null,
    doctor           text,
    date             text,
    time             text,
    reason           text,
    booking_id       text,
    status           text not null default 'confirmed',
    rescheduled_from text,
    created_at       timestamptz not null default now(),
    constraint appointments_status_check check (
        status in ('confirmed', 'cancelled', 'rescheduled')
    )
);

-- Indexes for Fast Querying
create index if not exists idx_slots_available
    on slots (doctor, iso_date, iso_time) where status = 'available';
create index if not exists idx_slots_phone_booked
    on slots (phone, iso_date, iso_time) where status = 'booked';

-- Seed 14 days of sample slots (Mon–Sat, 09:00–13:00 and 17:00–20:00)
do $$
begin
    insert into slots (doctor, iso_date, iso_time, status)
    select d.doctor, days.slot_date, t.slot_time, 'available'
    from (values ('Dr. Sarah Lin'), ('Dr. James Cole')) as d(doctor)
    cross join lateral (
        select (current_date + gs.i)::date as slot_date
        from generate_series(0, 13) as gs(i)
    ) as days
    cross join lateral (
        select time '09:00' as slot_time union all select time '09:30'
        union all select time '10:00' union all select time '10:30'
        union all select time '11:00' union all select time '11:30'
        union all select time '12:00' union all select time '12:30'
        union all select time '17:00' union all select time '17:30'
        union all select time '18:00' union all select time '18:30'
        union all select time '19:00' union all select time '19:30'
    ) as t
    where extract(isodow from days.slot_date) between 1 and 6
    on conflict (doctor, iso_date, iso_time) do nothing;
end $$;
Enter fullscreen mode Exit fullscreen mode

Note: These have been named and structured for a clinic, they can be adjusted to the business of your choice.

Adding the Memory

Create a folder in your project root called tools and create memory.py inside it.

This file handles our database connection. Because the standard Supabase Python client is synchronous (blocking), we use asyncio.to_thread to ensure our database queries don't freeze the real-time audio stream while they fetch data.

memory.py:

Part A: Setup and Phone Normalization

Telephony systems are notoriously messy with how they format phone numbers. Before we can look up a caller in our database, we need to normalize their number into a standard E.164 format (e.g., +1234567890):

from __future__ import annotations 
import os 
import asyncio 
import logging 
import re from datetime 
import datetime, timezone 
from supabase import Client, create_client 
from dotenv import load_dotenv 

load_dotenv() 

logger = logging.getLogger(__name__) 

SUPABASE_URL = os.getenv("SUPABASE_URL") 
SUPABASE_KEY = os.getenv("SUPABASE_KEY") 
Enter fullscreen mode Exit fullscreen mode

Initialise the Supabase Client:

def get_client() -> Client:
    return create_client(SUPABASE_URL, SUPABASE_KEY)
Enter fullscreen mode Exit fullscreen mode

Strips weird characters and normalizes the phone number:

def _normalize_phone(phone: str) -> str:
    cleaned = re.sub(r"[\s\-()]", "", phone.strip())
    if cleaned.startswith("+91"):
        return cleaned
    if cleaned.startswith("0"):
        return f"+91{cleaned[1:]}"
    if len(cleaned) == 10 and cleaned.isdigit():
        return f"+91{cleaned}"
    if cleaned.startswith("91") and len(cleaned) == 12 and cleaned.isdigit():
        return f"+{cleaned}"
    if cleaned.startswith("+"):,
        return cleaned
    return f"+91{cleaned}"
def _now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()
Enter fullscreen mode Exit fullscreen mode

Note: Adjust the default country code (+1, +91, etc.) based on your region.

Part B: Fetching and Writing Data

Fetching a caller:

async def get_patient(phone: str) -> dict | None:
    def _fetch():
        client = get_client()
        response = client.table("customers").select("*").eq("phone", _normalize_phone(phone)).limit(1).execute()
        return response.data[0] if response.data else None


    try:
        return await asyncio.to_thread(_fetch)
    except Exception as exc:
        logger.error("get_patient failed: %s", exc)
        return None
Enter fullscreen mode Exit fullscreen mode

Updating a caller’s profile after they’ve booked an appointment:

async def upsert_patient(phone: str, name: str, doctor: str, booking_id: str, date: str, time: str):
    def _upsert():
        client = get_client()
        normalized = _normalize_phone(phone)
        existing = client.table("customers").select("call_count").eq("phone", normalized).limit(1).execute()
        call_count = (existing.data[0].get("call_count") or 0) + 1 if existing.data else 1

        client.table("customers").upsert({
            "phone": normalized, "name": name, "preferred_doctor": doctor,
            "last_booking_id": booking_id, "last_appointment_date": date,
            "last_appointment_time": time, "call_count": call_count, "last_seen": _now_iso(),
        }, on_conflict="phone").execute()


    try:
        await asyncio.to_thread(_upsert)
       return True
    except Exception as exc:
        logger.error("upsert_patient failed: %s", exc)
       return False
Enter fullscreen mode Exit fullscreen mode

Tracking the call count:

async def increment_call_count(phone: str):
    def _inc():
        client = get_client()
        norm = _normalize_phone(phone)
        res = client.table("customers").select("call_count").eq("phone", norm).limit(1).execute()
        if res.data:
            client.table("customers").update({
                "call_count": (res.data[0].get("call_count") or 0) + 1,
                "last_seen": _now_iso()
            }).eq("phone", norm).execute()

    await asyncio.to_thread(_inc)
Enter fullscreen mode Exit fullscreen mode

Adding an appointment to the database:

async def log_appointment(phone: str, doctor: str, date: str, time: str, reason: str, booking_id: str):
    def _log():
        client = get_client()
        client.table("appointments").insert({
            "id": booking_id, "phone": _normalize_phone(phone), "doctor": doctor,
            "date": date, "time": time, "reason": reason, "booking_id": booking_id,
        }).execute()

    try:  
await asyncio.to_thread(_log) 
return True 
    except Exception as exc: 
logger.error("log_appointment failed: %s", exc) 
return False 
Enter fullscreen mode Exit fullscreen mode

Create booking.py, in the tools folder.
booking.py:

from __future__ import annotations 
import asyncio 
import logging 
from datetime import date, datetime 
from typing import Any 
from tools.memory import get_client 

logger = logging.getLogger(__name__) 

VALID_DOCTORS = ("Dr. Sarah Lin", "Dr. James Cole") 

def _format_iso_time(value: str) -> str: 
parts = str(value).split(":") 
return f"{parts[0]}:{parts[1]}" if len(parts) >= 2 else str(value)
Enter fullscreen mode Exit fullscreen mode

Translate messy database timestamps (2026-05-29 15:00) into natural spoken English (Friday the 29th of May, 3:00 PM):

def _spoken_from_iso(iso_date: str, iso_time: str) -> tuple[str, str]:
    try:
        dt = datetime.strptime(f"{iso_date} {_format_iso_time(iso_time)}", "%Y-%m-%d %H:%M")
        return dt.strftime("%A the %d of %B"), dt.strftime("%I:%M %p").lstrip("0")
    except ValueError:
        return iso_date, iso_time
Enter fullscreen mode Exit fullscreen mode

Scan the database for the next open slot for a particular doctor:

async def find_available_slot(doctor: str, iso_date: str | None = None, iso_time: str | None = None) -> dict[str, Any]:
    if doctor not in VALID_DOCTORS:
        return {"available": False, "error": f"Unknown doctor: {doctor}"}

    def _find():
        client = get_client()
        query = (
            client.table("slots")
            .select("id, doctor, iso_date, iso_time, status")
            .eq("doctor", doctor)
            .eq("status", "available")
            .order("iso_date")
            .order("iso_time")
        )
        if iso_date: query = query.eq("iso_date", iso_date)
        if iso_time: query = query.eq("iso_time", iso_time)

        response = query.limit(1).execute()
        if not response.data: return {"available": False}

        row = response.data[0]
        return {
            "available": True,
            "slot_id": row["id"],
            "doctor": row["doctor"],
            "iso_date": row["iso_date"],
            "iso_time": _format_iso_time(row["iso_time"])
        }


    try:
        result = await asyncio.to_thread(_find)
        if not result.get("available"):
            return {"available": False}

        date_spoken, time_spoken = _spoken_from_iso(result["iso_date"], result["iso_time"])
        return {**result, "date": date_spoken, "time": time_spoken}
    except Exception as exc:
        logger.error("find_available_slot failed: %s", exc)
        return {"available": False, "error": str(exc)}
Enter fullscreen mode Exit fullscreen mode

Making an appointment(reserving a slot):
ATOMIC LOCK: The most important function here. By requiring .eq("status", "available") in our update query, the database ensures that if two callers try to book the exact same slot simultaneously, only the first one will succeed.

async def reserve_slot(slot_id: str, patient_name: str, phone: str, doctor: str, booking_id: str, reason: str) -> bool:
    def _reserve():
        client = get_client()
        res = client.table("slots").update({
            "status": "booked", "booking_id": booking_id, "patient_name": patient_name, "phone": phone, "reason": reason
        }).eq("id", slot_id).eq("doctor", doctor).eq("status", "available").select("id").execute()
        return bool(res.data)
    return await asyncio.to_thread(_reserve)
Enter fullscreen mode Exit fullscreen mode

Finally, a function that runs on startup to warn if the slots are running out in the database and need to be added:

async def check_slot_coverage() -> None:
    def _check():
        res = get_client().table("slots").select("iso_date").gte("iso_date", date.today().isoformat()).eq("status", "available").order("iso_date", desc=True).limit(1).execute()
        if not res.data: return 0
        return (date.fromisoformat(str(res.data[0]["iso_date"])[:10]) - date.today()).days
    days = await asyncio.to_thread(_check)
    if days <= 0: logger.critical("SLOT COVERAGE: No future slots available — booking will fail")
Enter fullscreen mode Exit fullscreen mode

Equipping the AI

The agent does not know how to run python scripts to take actions by itself. Large Language Models don't automatically know how to run Python scripts. We have to wrap our logic in LiveKit's @function_tool decorator. This acts as a bridge, exposing the function to the LLM so it can trigger it natively during a conversation.

Create appointment.py in the tools folder:
appointment.py:

from __future__ import annotations


import asyncio
import logging
import random
from typing import Any


from livekit.agents import function_tool


from tools.booking import find_available_slot, reserve_slot
from tools.memory import log_appointment, upsert_patient


logger = logging.getLogger("clinic-agent.tools")
Enter fullscreen mode Exit fullscreen mode

A function to check if a slot is available before asking the caller to confirm if they want to book their appointment for the slot:

@function_tool
async def check_availability(date: str, time: str, doctor: str) -> dict[str, Any]:
    slot = await find_available_slot(doctor)
    if slot.get("available"):
        return {"available": True, "confirmed_slot": f"{slot.get('date', date)} {slot.get('time', time)}", "iso_date": slot.get("iso_date"), "iso_time": slot.get("iso_time")}
    return {"available": False}
Enter fullscreen mode Exit fullscreen mode

A function to generate a booking ID, block a slot and book an appointment once the caller has explicitly confirmed the booking:

@function_tool
async def book_appointment(patient_name: str, phone: str, date: str, time: str, doctor: str, reason: str, iso_date: str | None = None, iso_time: str | None = None) -> dict[str, Any]:
    booking_id = f"TC-{random.randint(1000, 9999)}"
    slot = await find_available_slot(doctor, iso_date, iso_time)

    if not slot.get("available"):
        return {"status": "failed", "message": "No available slots. Please try another day/doctor."}

    reserved = await reserve_slot(slot["slot_id"], patient_name, phone, doctor, booking_id, reason)
    if not reserved:
        return {"status": "failed", "message": "That slot was just taken. Please try another time."}   
    logger.info("Booking confirmed: %s for %s", booking_id, patient_name)
    return {"patient_name": patient_name, "phone": phone, "date": date, "time": time, "doctor": doctor, "status": "confirmed"}
Enter fullscreen mode Exit fullscreen mode

Function to get the list of available doctors, when the caller needs help picking a doctor:

@function_tool
async def get_doctor_list() -> list[dict[str, str]]:
    return [{"name": "Dr. Sarah Lin", "specialty": "General Physician"}, {"name": "Dr. James Cole", "specialty": "Physician"}]
Enter fullscreen mode Exit fullscreen mode

Updating the System Prompt

Now we update the prompt. We instruct the LLM on the booking rules, and if the user is found in our database, we inject their history directly into the prompt so the agent already knows their name and preferred doctor before the call even begins.

Update the contents of prompts/system_prompt.py:

Add this to the end of SYSTEM_PROMPT:

"""
BOOKING (New Appointment)
Collect these details one or two at a time: Patient's name, phone, date, time, doctor (Dr. Sarah Lin or Dr. James Cole), and reason.
If the caller asks about a slot, use check_availability first.
Read back the details and ask: "So that's an appointment for [name] at [time] on [date] with [doctor] for [reason] — shall I go ahead and book that?"
ONLY call book_appointment after the patient clearly says yes.
"""
Enter fullscreen mode Exit fullscreen mode

Update build_system_prompt to check if the current caller is a returning caller and update the memory accordingly:

def build_system_prompt(patient: dict | None = None) -> str:
    if patient is None:
        return SYSTEM_PROMPT

    last_appt = patient.get("last_appointment_date", "unknown")
    last_time = patient.get("last_appointment_time", "unknown")
    last_doctor = patient.get("preferred_doctor", "unknown")

    memory_block = f"""

## Caller memory
You already know this caller. Do not ask for their name or phone number again.
- Name: {patient["name"]}
- Preferred doctor: {last_doctor}
- Last appointment: {last_appt} at {last_time}


Greet them warmly by name. If they want to re-book, use these details!
"""
    return SYSTEM_PROMPT + memory_block
Enter fullscreen mode Exit fullscreen mode

Wiring it all together

And finally, to bring all of the additions we just made to life, we make a few changes to agent.py.

Add these imports and the helper functions to the top of agent.py:

from livekit import rtc
from tools.memory import get_patient, increment_call_count
from tools.booking import check_slot_coverage
from tools.appointment import book_appointment, check_availability, get_doctor_list
Enter fullscreen mode Exit fullscreen mode

Function to extract the caller’s phone number from Twilio’s SIP stream:

def _sip_caller_phone(participant: rtc.RemoteParticipant) -> str | None:
    if participant.kind != rtc.ParticipantKind.PARTICIPANT_KIND_SIP: return None
    return participant.attributes.get("sip.phoneNumber") or participant.identity
Enter fullscreen mode Exit fullscreen mode

Function to swap the generalised opening line with a personalised one for returning callers:

def _opening_line_for_patient(patient: dict | None) -> str:
    if patient and patient.get("name"):
        return f"Hello {patient['name']}, welcome back to The Clinic. I'm Matthew, your AI receptionist. How can I help you today?"
    return OPENING_LINE
Enter fullscreen mode Exit fullscreen mode

Add the newly added tools to the agent:

class ClinicAgent(Agent):
    def __init__(self, instructions: str) -> None:
        super().__init__(
            instructions=instructions,
            tools=[book_appointment, check_availability, get_doctor_list],
        )
Enter fullscreen mode Exit fullscreen mode

Add the slot coverage checking to the prewarm function:

def prewarm(proc: JobProcess) -> None:
    proc.userdata["vad"] = silero.VAD.load()
    try: asyncio.run(check_slot_coverage())
    except Exception: pass
Enter fullscreen mode Exit fullscreen mode

Replace the entrypoint function with the below. We are intercepting the SIP connection, getting the phone number, checking if it is a return caller, fetching the data and generating the personalised greeting and the prompt before the session connection is established:

async def entrypoint(ctx: JobContext) -> None:
    is_phone = _is_phone_room(ctx.room.name)
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY if is_phone else AutoSubscribe.SUBSCRIBE_ALL)

    # Keep your STT, LLM, TTS initialization code here exactly as it was

    # --- 1. RESOLVE THE CALLER ---
    caller_phone = None
    if is_phone:
        # Check if Twilio passed the phone number immediately
        for p in ctx.room.remote_participants.values():
            caller_phone = _sip_caller_phone(p)
            if caller_phone: break

        # If not immediately available, wait briefly for the network handshake
        if not caller_phone:
            try:
                p = await asyncio.wait_for(ctx.wait_for_participant(), timeout=10.0)
                caller_phone = _sip_caller_phone(p)
            except asyncio.TimeoutError: pass

    # --- 2. FETCH MEMORY ---
    patient_memory = None
    if caller_phone:
        patient_memory = await get_patient(caller_phone)
        # Update the analytics call count in the background
        asyncio.create_task(increment_call_count(caller_phone))

    # --- 3. INJECT CONTEXT ---
    prompt = build_system_prompt(patient_memory)
    opening_greeting = _opening_line_for_patient(patient_memory)

    # --- 4. START SESSION ---
    session = AgentSession(stt=stt, llm=llm, tts=tts, vad=ctx.proc.userdata["vad"])
    await session.start(ClinicAgent(prompt), room=ctx.room)


    if is_phone:
        try:
            participant = await asyncio.wait_for(ctx.wait_for_participant(), timeout=20.0)
            session.room_io.set_participant(participant.identity)
        except asyncio.TimeoutError: return


        # Greet the caller (Uses the personalized greeting if they exist in the DB)
        handle = session.say(opening_greeting, allow_interruptions=False)
        await handle.wait_for_playout()


    while ctx.room.isconnected():
        await asyncio.sleep(0.25)
Enter fullscreen mode Exit fullscreen mode

Now you can start the agent and call it. On the second call, the agent will recognise and greet you by name and if you’ve placed an appointment, it will have details of the appointment as well.

RAG

RAG Pipeline

The agent can now book appointments and recognise returning callers, but if a caller asks for any static information on the business such as clinic hours or about parking, it could confidently hallucinate a random answer.

To ensure that this does not happen we can use Retrieval-Augmented Generation(RAG). Instead of relying on cloud-based vector databases that add network round-trip latency, we are going to build a local RAG pipeline. The agent will chunk, embed, and search a markdown document on your local machine using LanceDB to answer specific questions instantly and accurately.

The Knowledge Base

In your project root, create a folder named knowledge and inside the folder, create clinic_faq.md.
Structure the file using ## (H2) headings. Our python script will use these headings to break the document into searchable chunks.

This has been written for a clinic, you can restructure it and add your own data to use it for any business.

clinic_faq.md:

## About The Clinic


The Clinic is a general practice clinic in Maplewood, founded in 2018


## Clinic hours


We are open Monday to Saturday from nine in the morning to one in the afternoon,
and from five in the evening to eight at night. We are closed on Sundays and
public holidays. The last appointment is thirty minutes before closing.


## Consultation fees


A general consultation costs fifty dollars. A follow-up within two weeks
costs thirty-five dollars. A specialist consultation costs eighty dollars.


## Our doctors


Dr. Sarah Lin is a General Physician with fifteen years of experience,
specialising in diabetes, hypertension, and thyroid conditions.


Dr. James Cole is a Physician with ten years of experience, specialising
in respiratory conditions, infectious diseases, and general medicine.


## Location and how to find us


Our address is 14 Birch Lane, Suite 2, Maplewood. Free parking in the building lot.


## Booking and cancellation policy


Appointments are thirty minutes. Cancel at least two hours before. Rescheduling
is free once; a second reschedule in the same week carries a small admin charge.


## Payments we accept


Cash and all major debit/credit cards. We work with most major health insurers.


## Lab and diagnostic services


In-clinic: blood count, blood sugar, HbA1c, lipid profile, thyroid panel, LFT,
KFT, urine analysis. Most results same day by six in the evening.
No X-ray, ECG, or ultrasound on-site — we refer to a nearby diagnostic centre.


## Emergencies and urgent care


The Clinic is not an emergency facility. For emergencies, call your local
emergency number or go to the nearest ER.


## Contact


Phone: 1234567. WhatsApp available on the same number for confirmations.
Email: hello@theclinic.example — response within 24 hours on working days.
Enter fullscreen mode Exit fullscreen mode

The local RAG engine

In your tools folder, create faq.py. We will use sentence-transformers to turn our text into math (vectors), and LanceDB to store and search those vectors without ever leaving your machine.

Part A: Setup and Index Building
Starting with what happens when the server boots up. It breaks the markdown file into chunks and saves them in the database.

tools/faq.py:

from __future__ import annotations

import asyncio
import logging
import re
from pathlib import Path

import lancedb
from livekit.agents.llm import function_tool
from sentence_transformers import SentenceTransformer

logger = logging.getLogger(__name__)

_ROOT = Path(__file__).resolve().parent.parent
DB_PATH = str(_ROOT / ".lancedb")
TABLE_NAME = "clinic_faq"
FAQ_PATH = _ROOT / "knowledge" / "clinic_faq.md"
MODEL_NAME = "all-MiniLM-L6-v2"
VECTOR_METRIC = "cosine"
DISTANCE_THRESHOLD = 0.83
MAX_RESPONSE_CHARS = 500
_SEARCH_LIMIT = 4
Enter fullscreen mode Exit fullscreen mode

We split the document at every H2(##) heading and each section becomes a chunk:

def _chunk_markdown(text: str) -> list[dict]:
    chunks: list[dict] = []
    for part in re.split(r"\n(?=## )", text.strip()):
        part = part.strip()
        if not part.startswith("## "):
            continue
        lines = part.split("\n", 1)
        heading = lines[0].strip()
        body = lines[1].strip() if len(lines) > 1 else ""
        if len(body) >= 30:
            chunks.append({"heading": heading, "body": body, "full": f"{heading}\n\n{body}"})
    return chunks
Enter fullscreen mode Exit fullscreen mode

Build a LanceDB vector index from clinic_faq.md:

def build_index() -> None:
    if not FAQ_PATH.is_file():
        logger.error("FAQ file not found: %s", FAQ_PATH)
        return

    chunks = _chunk_markdown(FAQ_PATH.read_text(encoding="utf-8"))
    if not chunks:
        logger.error("No chunks produced from FAQ file")
        return

    model = SentenceTransformer(MODEL_NAME)
    vectors = model.encode([c["full"] for c in chunks])

    data = [
        {
            "heading": c["heading"],
            "body": c["body"],
            "full": c["full"],
            "vector": vectors[i].tolist(),
        }
        for i, c in enumerate(chunks)
    ]

    db = lancedb.connect(DB_PATH)
    if TABLE_NAME in db.table_names():
        db.drop_table(TABLE_NAME)
    db.create_table(TABLE_NAME, data)
    logger.info("FAQ index built: %d chunks", len(chunks))
Enter fullscreen mode Exit fullscreen mode

Part B: Searching

Now we come to the actual tool the LLM will call.

Sometimes, when a vector database is searched, the intent matching could go wrong. For example, if 'fees' and 'price' are not considered under consultation fees. To avoid this, we add heading boosts, to boost relevance based on keywords in the query matching a section heading. In faq.py, add:

_HEADING_BOOSTS: list[tuple[tuple[str, ...], str]] = [
    (("timing", "hour", "open", "closed", "sunday", "holiday"), "clinic hours"),
    (("fee", "cost", "price", "charge", "follow-up", "consultation"), "consultation fees"),
    (("insurance", "payment", "card", "cashless"), "payments"),
    (("x-ray", "ecg", "ultrasound", "lab", "blood test"), "lab and diagnostic"),
    (("emergency", "urgent"), "emergencies"),
    (("doctor", "sarah", "james", "lin", "cole"), "our doctors"),
    (("address", "location", "find", "directions", "where"), "location"),
    (("cancel", "reschedule", "book", "appointment", "walk-in"), "booking"),
    (("phone", "whatsapp", "email", "contact"), "contact"),
]
Enter fullscreen mode Exit fullscreen mode

Adds a mathematical boost to the search score if the keyword matches a section heading:

def _heading_boost(query: str, heading: str) -> float:
    q, h = query.lower(), heading.lower()
    for keywords, hint in _HEADING_BOOSTS:
        if hint in h and any(k in q for k in keywords):
            return 0.25
    return 0.0
Enter fullscreen mode Exit fullscreen mode

Make sure that very long blocks of text are not returned, which would be too long to be spoken at once:

def _trim_at_sentence(text: str, max_chars: int) -> str:
    if len(text) <= max_chars:
        return text
    cut = text[:max_chars]
    boundary = cut.rfind(". ")
    return cut[: boundary + 1] if boundary >= int(max_chars * 0.55) else cut.rstrip() + "..."
Enter fullscreen mode Exit fullscreen mode

The function the LLM calls to actually perform the search:

@function_tool()
async def search_faq(query: str) -> str:
    """
    Search the clinic knowledge base to answer patient questions.
    Call this for ANY factual question about the clinic: hours, location,
    fees, doctors, lab services, parking, payments, cancellation policy,
    pharmacy, or emergencies.
    Do not guess — always call this tool first.
    query: the patient's question exactly as they asked it.
    """
    def _search(q: str) -> str:
        try:
            model = SentenceTransformer(MODEL_NAME)
            vec = model.encode([q])[0].tolist()


            db = lancedb.connect(DB_PATH)
            if TABLE_NAME not in db.table_names():
                return ""


            results = (
                db.open_table(TABLE_NAME)
                .search(vec)
                .metric(VECTOR_METRIC)
                .limit(_SEARCH_LIMIT)
                .to_list()
            )


            # Filter by distance threshold, apply heading boost, pick best
            relevant = [r for r in results if r.get("_distance", 2.0) < DISTANCE_THRESHOLD]
            if not relevant:
                return ""


            ranked = sorted(
                relevant,
                key=lambda r: r.get("_distance", 2.0) - _heading_boost(q, r.get("heading", "")),
            )
            best_body = ranked[0]["body"]
            return _trim_at_sentence(best_body, MAX_RESPONSE_CHARS)


        except Exception as exc:
            logger.error("FAQ search error: %s", exc)
            return ""


    result = await asyncio.to_thread(_search, query)
    if not result:
        return (
            "I don't have specific information on that. "
            "Let me have someone from our team call you back with the answer."
        )
    return result
Enter fullscreen mode Exit fullscreen mode

Equipping the Agent to use the RAG to answer clinic questions

In prompts/system_prompt.py, add this to the SYSTEM_PROMPT:

"""
ANSWERING CLINIC QUESTIONS
For any factual question about the clinic — hours, location, fees, doctors,
lab services, parking, payments, cancellation policy, pharmacy, or
emergencies — call the search_faq tool before answering. Never guess or
answer from memory. If search_faq returns nothing useful, say:
"Let me have someone from our team call you back with that information.
May I take your number?"
"""
Enter fullscreen mode Exit fullscreen mode

Next, in agent.py,

At the top, add the imports:

from tools.faq import build_index, search_faq
Enter fullscreen mode Exit fullscreen mode

To the prewarm function, add:

    build_index()
Enter fullscreen mode Exit fullscreen mode

This ensures the vector index is built on startup.

In ClinicAgent, add search_faq to the tools, to establish it as a tool the agent can use:

class ClinicAgent(Agent):
    def __init__(self, instructions: str) -> None:
        #!Layer 2: Memory and Tools
        super().__init__(
            instructions=instructions,
            tools=[book_appointment, check_availability, get_doctor_list, search_faq],
            )
Enter fullscreen mode Exit fullscreen mode

Change the if __name__ == “__main__”: section at the end of agent.py to this:

if __name__ == "__main__":
    import sys
    if len(sys.argv) >= 2 and sys.argv[1] == "download-files":
        build_index()
    else:
        cli.run_app(
            WorkerOptions(
                entrypoint_fnc=entrypoint,
                prewarm_fnc=prewarm,
                agent_name="clinic-agent",
                num_idle_processes=1,
            )
        )
Enter fullscreen mode Exit fullscreen mode

Run:

python agent.py download-files
Enter fullscreen mode Exit fullscreen mode

This downloads the embedding model before startup, so run it before starting the agent.

When you start the agent and make a call, it should answer any questions on the clinic(or whichever business you’ve adjusted it for) based on the content in the markdown file.

Some Optional Additions to Make the Agent Better

Google Calendar Blocking

After a successful booking, we create an event on the doctor’s/business’s calendar - this is only for visibility, the agent never reads from it. We’ve already covered all the credentials and configurations in the setup section.

The code for this can be found in tools/calendar_mirror.py on Github([repo link]).

In tools/appointment.py, import the new function:

from tools.calendar_mirror import create_calendar_event
Enter fullscreen mode Exit fullscreen mode

Scroll to the book_appointment function, the part where we create the Supabase background tasks and add create calendar as a background task:

    reserved = await reserve_slot(slot["slot_id"], patient_name, phone, doctor, booking_id, reason)
    if not reserved:
        return {"status": "failed", "message": "That slot was just taken. Please try another time."}
# -------------------------- ADD THIS ------------------------------------
    asyncio.create_task(create_calendar_event(
        patient_name, phone, doctor, reason, booking_id, slot["iso_date"], slot["iso_time"]
    ))
# ------------------------------------------------------------------------

    logger.info("Booking confirmed: %s for %s", booking_id, patient_name)
    return {"patient_name": patient_name, "phone": phone, "date": date, "time": time, "doctor": doctor, "status": "confirmed"}
Enter fullscreen mode Exit fullscreen mode

Whatsapp Confirmation

After booking an appointment, we can configure the agent to send the caller a confirmation on their Whatsapp. We’ve covered the configuration and credentials in the Twilio section.

The code for this can be found in tools/notification.py on Github([repo link]).

In tools/appointment.py:
Add the import to the top of the file:

from tools.notifications import send_whatsapp_confirmation
Enter fullscreen mode Exit fullscreen mode

Add the whatsapp confirmation as a background task in the book_appointment function:

asyncio.create_task(send_whatsapp_confirmation(phone, patient_name, doctor, date, time, booking_id, reason))
Enter fullscreen mode Exit fullscreen mode

Cancelling and Rescheduling

A fully autonomous receptionist doesn't just book appointments, it manages the calendar. However, modifying or deleting existing appointments introduces a critical safety risk: we do not want the AI accidentally canceling a booking just because a patient asked, "What is your cancellation policy?"

To solve this, we are going to implement a Two-Step Confirmation Pattern. The LLM must first call our tool with a confirmed=False flag to look up the appointment and read it back. It is strictly instructed to only call the tool again with confirmed=True after the patient explicitly says "Yes."

You can find the code for this on Github([repo link]) in tools/cancellation.py.

In agent.py,
Import the new functions at the top:

from tools.cancellation import cancel_appointment reschedule_appointment
Enter fullscreen mode Exit fullscreen mode

Add the two new tools to ClinicAgent:

class ClinicAgent(Agent):
    def __init__(self, instructions: str) -> None:
        super().__init__(
            instructions=instructions,
            tools=[book_appointment, check_availability, get_doctor_list, search_faq, transfer_to_human, cancel_appointment, reschedule_appointment],
            )
Enter fullscreen mode Exit fullscreen mode

Adapting To Your Use Case

The clinic persona is a thin configuration layer on top of a general-purpose call agent. The voice pipeline, slot system and memory are business-agnostic. Here is what to change:

What to change Which File Update
Agent name and persona prompts/system_prompt.py Identity block, opening line and instructions
Staff/provider names sql/create_tables.sql slots_doctor_check constraint
Booking Flow prompts/system_prompt.py Booking intent section
Business hours and slots sql/create_tables.sql supabase/functions/seed-slots/index.ts Seed times and working days
FAQ content knowledge/clinic_faq.py Replace entirely with content about you business, keep the ## heading structure
Calendar names .env GOOGLE_CALENDAR_ID_*
Handoff number .env CLINIC_PHONE_NUMBER
Voice agent.py Murf voice ID – see murf.ai/voices

Example Prompts

Hair Salon
System Prompt:

You are Zara, the AI receptionist for Curl & Cut salon, Indiranagar, Bangalore.
Opening line: Hello, thanks for calling Curl & Cut. I'm Zara, your AI assistant. How can I help?
Enter fullscreen mode Exit fullscreen mode

Booking flow: service type (haircut / colour / blowout), stylist preference, date, time.
sql/create_tables.sql — update the provider constraint:
constraint slots_stylist_check check (doctor in ('Aisha', 'Priya', 'Riya'))
Replace knowledge/clinic_faq.md with your services, pricing, and cancellation policy.

Legal Intake
System Prompt:

You are Alex, the AI intake assistant for Mehta & Associates. Collect: caller name,
contact number, matter type (civil / criminal / family / property), and a brief description.
Then schedule a callback with a solicitor.
Enter fullscreen mode Exit fullscreen mode

For callback-only intake, replace book_appointment with a lighter tool that logs the inquiry and records a preferred callback time. Memory, transcripts, and WhatsApp all still work as-is.

Restaurant
System Prompt:

You are Anaya, an AI receptionist who takes reservations for The Spice Room. Collect: guest name, contact number,
date, time, party size, and any dietary requirements.
Enter fullscreen mode Exit fullscreen mode

The slots table works naturally for table-time pairs. Update the provider constraint to table names:
constraint slots_table_check check (doctor in ('Table 1', 'Table 2', 'Table 3', 'Terrace'))
Update supabase/functions/seed-slots/index.ts for your opening hours, days, and booking interval.

Errors

Error Cause Fix
Required environment variable 'X' is not set Missing .env value Copy .env.example to .env and fill in the variable
Agent answers but stays silent Dispatch rule has no agents block Edit the rule in LiveKit Cloud — add agentName: clinic-agent to roomConfig.agents
DuplexClosed in logs, call drops mid-greeting dev mode restarts on file save Use python agent.py start for all phone testing
Call drops immediately TwiML Bin not reachable, or URI missing ;transport=tcp Check the URI in the TwiML Bin and add ;transport=tcp at the end
ERROR: relation "slots" does not exist Ran only part of create_tables.sql Select the full file (Ctrl+A) and run it again from the top
Table "slots" is missing at seed time Same as above Same fix — the seed block at the bottom requires the tables above it
401 or 403 from Murf or Deepgram Wrong or expired API key Re-check MURF_API_KEY and DEEPGRAM_API_KEY in .env
WhatsApp message not delivered Recipient has not joined the sandbox Send join from the recipient's WhatsApp to the sandbox number
Calendar events not appearing Service account not shared with the calendar Go to each calendar's settings and share it with edit permissions to the service account email
Slot coverage warning on startup Fewer than 14 days of available slots ahead Run the manual seed SQL or POST to the Edge Function URL

The complete, ready-to-run source code for this project is available on GitHub. Clone the repo, swap out the system prompt, adjust the database schema, and build a voice agent for your own use case.

If you build something cool using this stack, I’d love to hear it, let me know what your agent is booking!

Top comments (0)