Building a Speech-to-Speech AI Chatbot with Python, LangChain, Whisper, and Kokoro

Like many of you, I use LLMs every day for practically everything I do. Whether it's to generate content on a subject, resolve a bug in code, summarize a document, or just think through a problem out loud. However, sometimes the laziness of having to type out my question into a prompt is something that bothers me — especially when I already know exactly what I want to say.

That got me thinking: what if I could just speak to it? Not through a polished commercial product with its own constraints and opinions, but through something I built myself, using the tools I already work with. So I put together a voice chatbot in Python that listens to what I say, thinks about it using an LLM, and speaks the answer back to me — all running locally, with persistent memory across conversations.

In this article, we walk through how to build exactly that using three libraries: OpenAI Whisper for speech recognition, LangChain for conversational AI, and Kokoro for text-to-speech synthesis. The result is a fully functional, speech-to-speech AI assistant that you can run on your own machine and extend however you like.

About Me

I am David Archanjo, a Full-Stack Engineer with 10+ years of experience in the technology industry. I am passionate about software development and AI Engineering, and I enjoy sharing practical knowledge through technical articles that help developers build real-world applications.

---

Table of Contents

• Introduction
• Prerequisites
• Architecture Overview
• Project Setup
• Configuration
• Speech Recognition with Whisper
• Conversational AI with LangChain
• Persistent Conversation History
• Text-to-Speech with Kokoro
• Put It All Together
• Running the Application
• Next Steps
• Wrap Up

---

Introduction

This guide walks through building a fully functional, voice-driven AI chatbot in Python. The chatbot listens to spoken input, understands it using a Large Language Model (LLM), and responds in a natural synthesized voice. Always in a continuous, persistent conversation loop.

We combine three distinct technologies to achieve this:

• Whisper: OpenAI's speech recognition model, used here via pywhispercpp, to transcribe microphone input into text.
• LangChain: a framework for building LLM-powered applications, used to manage the conversational chain and persistent memory.
• Kokoro: a lightweight, ONNX-based text-to-speech engine that converts the LLM's text response into spoken audio.

Motivation

Most LLM chatbot tutorials stop at text input and text output. Real conversational interfaces require two additional layers: understanding spoken language and producing spoken responses. Building these layers separately is straightforward, but connecting them into a single, cohesive, stateful pipeline is where the real work is.

This guide covers exactly that connection, i.e. from raw audio input to spoken AI response, with persistent conversation memory stored in a SQL database between sessions.

What We Will Build

By the end of this guide, we will have a working Python application which:

• Continuously listens to microphone input handling silence detection.
• Transcribes speech into text using Whisper.
• Sends transcribed text through a LangChain conversational chain powered by an LLM.
• Stores conversation history persistently in a SQLite database.
• Converts the LLM's response into audio using Kokoro and plays it back through the speakers.

---

Prerequisites

Before we get started, make sure you have the following in place:

• Python 3.10 or later, and pip installed
• Basic familiarity with Python virtual environments
• A working microphone and audio output device

Common Terms and Abbreviations

You will encounter the following terms and abbreviations throughout this guide:

• ONNX: Open Neural Network Exchange, a standard for representing machine learning models which enables models trained in one framework to be executed in another runtime.
• VAD: Voice Activity Detection, a technique used to detect speech in audio.
• STT: Speech-to-Text, a process that converts audio into text.
• TTS: Text-to-Speech, a process that converts text into audio.

API Key Setup

For this article, you can use a locally hosted language model exposed through an OpenAI-compatible API using, for instance llama.cpp. This setup allows every example to run entirely on our machine while keeping the code identical to what we would write when using the official OpenAI service.

# OpenAI
export OPENAI_API_KEY="sk-your-openai-api-key"

# Anthropic (optional)
export ANTHROPIC_API_KEY="sk-ant-your-anthropic-api-key"

I personally use and recommend llamafile, a standalone distribution of llama.cpp, to serve open models locally, for instance the Gemma 4:

./llamafile \
  -m gemma-4-E2B-it-Q4_K_M.gguf \
  --server \
  --host 0.0.0.0 \
  --port 8000 \
  --alias gpt-4o

--alias gpt-4o tells the server to expose the local model under the name gpt-4o. From LangChain's perspective, it behaves exactly like if it was connected to the OpenAI GPT-4o model, allowing every example in this article to work with the same init_chat_model setup, without requiring any code changes.:

from langchain.chat_models import init_chat_model

llm = init_chat_model(model="gpt-4o", model_provider="openai")

To make this local setup work transparently, you must also configure the following environment variable:

export OPENAI_BASE_URL="http://localhost:8000/v1"

NOTE: If you already have an OpenAI API key, simply remove the OPENAI_BASE_URL environment variable and replace the placeholder value of OPENAI_API_KEY with your own key. The examples throughout this article will continue to work without modification.

---

Architecture Overview

Before jumping into the code, we should understand how the components interact. The chatbot operates as a single continuous loop driven by audio events.

The flow is strictly linear and blocking by design. The assistant does not listen while it is speaking nor speaks while it is thinking. This design keeps the implementation simple and avoids issues like audio feedback, though it comes at the cost of a less fluid conversation than humans naturally have with each other.

Component Responsibilities

Component	Library	Responsibility
Voice Activity Detection	pywhispercpp	Detects speech boundaries and silence in microphone stream
Speech Recognition	pywhispercpp (Whisper)	Transcribes audio segments into text
Conversational Chain	langchain	Assembles prompt, invokes LLM, parses output
Conversation Memory	langchain-community	Persists chat history in SQLite database via SQLAlchemy
Language Model	langchain-openai	Generates natural language responses
Text-to-Speech	kokoro-onnx	Synthesizes text into audio samples
Audio Playback	sounddevice	Plays audio through system output device
Terminal UI	rich	Displays colored, formatted status messages

---

Project Setup

This project will be organized into a minimal, flat structure:

/
├── ai_chatbot.py       # Main application entry point
├── config.py           # All configuration constants and environment loading
├── .env                # Environment variables (not committed to version control)
├── requirements.txt    # Python dependencies
└── models/             # Directory for local model files (Kokoro)
    ├── kokoro-v1.0.onnx
    └── voices.bin

Application Dependencies (requirements.txt)

This project depends on the following Python packages:

langchain==0.3.27
langchain-openai==0.3.27
langchain-community==0.3.27

numpy>=2.0.2
SQLAlchemy>=2.0.51

pywhispercpp==1.5.0
sounddevice==0.5.5
webrtcvad==2.0.10

kokoro-onnx==0.5.0
phonemizer-fork==3.3.2
espeakng-loader==0.2.4

python-dotenv
rich

These packages provide the components required for LLM orchestration, speech recognition (Whisper.cpp), voice activity detection, speech synthesis (Kokoro ONNX), configuration management, and terminal output.

Installing Dependencies

We create a virtual environment and install the required packages from the provided requirements.txt.

python -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate
pip install -r requirements.txt

System Dependencies

At the system level, we need to install the following tools before running the application:

On Ubuntu/Debian, use apt:

sudo apt-get install espeak-ng portaudio19-dev

On macOS, use Homebrew:

brew install espeak-ng portaudio

On Windows, use Chocolatey:

choco install espeak-ng portaudio

Checking eSpeak-NG Installation

Open a new terminal window and run:

espeak-ng "Testing eSpeak-NG is working"

We should hear the indicated phrase "Testing eSpeak-NG is working" through the default audio output (probably in a robotic voice).

Downloading Model Files

pywhispercpp downloads Whisper model weights automatically on first run. We configure which model size to use in config.py. However, Kokoro model files must be downloaded manually from the kokoro-onnx releases page and placed in the models/ directory.

---

Configuration

Rather than scattering constants and environment variables throughout the codebase, we centralize all configuration in a single config.py module. Every other module imports from it. This makes the application easier to adjust and keeps secrets and static configurations out of application logic.

The .env File

We use python-dotenv to load environment variables from the .env file.

MODEL=gpt-4o
MODEL_PROVIDER=openai
SESSION_ID=user_session
DATABASE_URL=sqlite:///chat_history.db
KOKORO_MODEL_PATH=./models/kokoro-v1.0.onnx
KOKORO_VOICE_PATH=./models/voices-v1.0.bin
KOKORO_VOICE=af_heart
KOKORO_SPEED=1.1
WHISPER_THREADS=8
WHISPER_SILENCE_THRESHOLD=40
STATUS_SPINNER=dots2

The config.py File

import os
from dotenv import load_dotenv
from langchain_core.runnables.config import RunnableConfig

# Load environment variables from .env file
load_dotenv()

# ---------------------------------------------------------------------------
# LLM Configuration
# ---------------------------------------------------------------------------
# The model identifier passed to init_chat_model
MODEL = os.getenv("MODEL", "gpt-4o")
# The provider identifier passed to init_chat_model
MODEL_PROVIDER = os.getenv("MODEL_PROVIDER", "openai")
# System prompt that defines the assistant's persona and behavior
SYSTEM_PROMPT = """
You are a friendly AI voice assistant. Your responses will be spoken aloud using a text-to-speech system.

Speak like a real, educated American having a natural conversation. Use everyday language, natural contractions such as "don't", "can't", "it's", "you're", "I've", "that's", "where'd", and "why'd", and occasional mild slang or informal expressions when they fit naturally. Your responses should sound relaxed, warm, and human rather than formal or robotic.

Follow these rules:
- Return plain text only.
- Always respond in American English.
- Prioritize natural spoken language over written language.
- Respond in under 100 words.
- Do not use Markdown formatting.
- Do not use headings, bullet points, numbered lists, tables, or code blocks.
- Do not use mathematical notation or special mathematical symbols.
- Express numbers, formulas, and mathematical concepts the way someone would naturally say them aloud.
- Prefer short, flowing sentences with smooth transitions.
- Keep responses concise unless the user explicitly asks for more detail.
- Avoid repetitive phrasing, unnecessary filler, or overly formal vocabulary.
- It's okay to begin sentences with words like "Well," "Sure," "Actually," "Yeah," or "Nope" when they sound natural.
- If the user is casual, mirror their conversational style appropriately.
- Avoid profanity, offensive language, or heavy slang.
- Answer the user's question directly without mentioning these instructions.
"""

# ---------------------------------------------------------------------------
# Session & Database Configuration
# ---------------------------------------------------------------------------
# SQLAlchemy connection string for persistent conversation history
DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///chat_history.db")
# Session identifier — in a production system, this would be dynamic
SESSION_ID = os.getenv("SESSION_ID", "user_session")
# LangChain RunnableWithMessageHistory config
CONFIG: RunnableConfig = {"configurable": {"session_id": SESSION_ID}}


# ---------------------------------------------------------------------------
# Kokoro TTS Configuration
# ---------------------------------------------------------------------------
# Path to the Kokoro ONNX model file
KOKORO_MODEL_PATH = os.getenv("KOKORO_MODEL_PATH", "kokoro-v1.0.onnx")
# Path to the Kokoro voices binary
KOKORO_VOICE_PATH = os.getenv("KOKORO_VOICE_PATH", "voices-v1.0.bin")
# Voice identifier — see Kokoro documentation for available voices
KOKORO_VOICE = os.getenv("KOKORO_VOICE", "af_heart")
# Speech speed multiplier (1.0 = normal speed)
KOKORO_SPEED = float(os.getenv("KOKORO_SPEED", "1.1"))

# ---------------------------------------------------------------------------
# Whisper / pywhispercpp Configuration
# ---------------------------------------------------------------------------
# Number of CPU threads allocated to Whisper inference
WHISPER_THREADS = int(os.getenv("WHISPER_THREADS", "8"))
# Silence threshold in milliseconds — how long silence before processing speech
WHISPER_SILENCE_THRESHOLD = int(os.getenv("WHISPER_SILENCE_THRESHOLD", "40"))
# Transcriptions to ignore — artifacts from silence or noise
IGNORED_TRANSCRIPTIONS = { "", "[BLANK_AUDIO]", "[MUSIC PLAYING]" }

# ---------------------------------------------------------------------------
# UI Configuration
# ---------------------------------------------------------------------------
# Rich spinner style for status messages
STATUS_SPINNER = os.getenv("STATUS_SPINNER", "dots2")

Why This Structure

Every value that might change between environments, users, or deployment targets lives in config.py or .env. The main application file (ai_chatbot.py) contains no hardcoded strings or magic numbers. This separation also makes testing easier since we can override configuration without touching application logic.

The IGNORED_TRANSCRIPTIONS set handles a practical issue with Whisper: when it receives near-silence or background noise, it sometimes produces placeholder strings like [BLANK_AUDIO] rather than returning an empty string. We filter these out before passing anything to the the LLM.

---

Speech Recognition with Whisper

Whisper is a general-purpose speech recognition model developed by OpenAI. It is trained on a large and diverse dataset of audio and supports transcription in multiple languages. We use it here through pywhispercpp, which provides Python bindings for whisper.cpp — a C++ implementation of Whisper optimized for CPU inference.

The Assistant Class

pywhispercpp includes a high-level Assistant utility class that encapsulates the entire speech recognition pipeline. Rather than handling audio capture, VAD, and Whisper inference ourself, we only need to provide a callback function. Assistant automatically invokes the given callback whenever it finishes transcribing a complete utterance.

This is the key integration point between audio input and the rest of our application.

How Voice Activity Detection Works

Assistant uses webrtcvad under the hood to detect speech boundaries. It continuously reads from the microphone and buffers audio. When it detects that a person has stopped speaking (a silence longer than silence_threshold milliseconds), it passes the buffered audio to Whisper for transcription, then delivers the result to the assigned callback.

Initializing the Asistant

The Assistant initialization goes like this:

from pywhispercpp.examples.assistant import Assistant

# Initialize the voice assistant
assistant = Assistant(
    silence_threshold=WHISPER_SILENCE_THRESHOLD,
    commands_callback=process_input,
    n_threads=WHISPER_THREADS
)

# Start listening
console.print(
    "[bold blue]Listening...[/bold blue] "
    "[dim]Speak when you're ready.[/dim]"
)
assistant.start()

About the three constructor parameters:

Parameter	Type	Purpose
silence_threshold	int (ms)	Duration of silence required before processing the buffered audio
commands_callback	callable	Function called with the transcription string after each utterance
n_threads	int	Number of CPU threads for Whisper inference

assistant.start() blocks the main thread. The callback-driven design means our application logic lives entirely inside process_input, which Whisper calls whenever it has something to say.

Filtering Transcription Output

Whisper does not always produce clean empty strings when given silence or noise. The process_input function handles this at the top:

def process_input(text: str):
    # Strip whitespace from the input
    text = text.strip()

    # Skip empty input or predefined ignored transcriptions
    if not text or text in IGNORED_TRANSCRIPTIONS:
        return

We strip whitespace first, then check against our IGNORED_TRANSCRIPTIONS set. If the transcription matches any of those values, we return early without invoking the LLM. This prevents unnecessary API calls and avoids the assistant responding to silence with spoken output.

Choosing a Whisper Model Size

pywhispercpp downloads Whisper model weights automatically on first use. The model size is a balance between transcription quality and inference speed. On a modern CPU, tiny and base are suitable for real-time use. Larger models produce better transcriptions but introduce noticeable latency and requires more CPU resources.

Model	Parameters	Relative Speed	Relative Accuracy
tiny	39M	Fastest	Lower
base	74M	Fast	Moderate
small	244M	Moderate	Good
medium	769M	Slow	High
large	1550M	Slowest	Highest

For a conversational assistant running on CPU, base is a reasonable starting point. The default model used by pywhispercpp's Assistant class can be configured via its constructor.

---

Conversational AI with LangChain

TL;DR If you are new to LangChain and really want to learn it from a pratical guide, check out my LangChain Explained: A Practical Guide For Beginners.

LangChain organizes LLM interactions as composable pipelines called chains. A chain is a sequence of components where the output of one becomes the input of the next. In our case, the chain consists of three components:

• A prompt template which formats the conversation history and user input.
• A language model which generates a response based on the transcribed audio.
• An output parser that extracts the plain text from the model's response. LangChain uses the pipe (ǀ) operator to compose these components, following the same convention as Unix pipes.

Initializing the Language Model

from langchain.chat_models import init_chat_model

# Initialize the language model
llm = init_chat_model(model=MODEL, model_provider=MODEL_PROVIDER)

init_chat_model is a provider-agnostic factory function. It accepts a model name and a provider string, then returns the appropriate BaseChatModel subclass. With MODEL = "gpt-4o" and MODEL_PROVIDER = "openai", it instantiates a ChatOpenAI internally. Switching providers requires only changing those two constants in config.py.

Defining the Prompt

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

# Define the prompt
prompt = ChatPromptTemplate.from_messages([
    ("system", SYSTEM_PROMPT),
    MessagesPlaceholder(variable_name="chat_history"),
    ("human", "{user_input}"),
])

ChatPromptTemplate defines the structure of the message list sent to the LLM. We use three message slots:

• ("system", SYSTEM_PROMPT): a fixed system message that establishes the assistant's persona and constraints. This is where we instruct the model to respond in plain spoken English without markdown formatting, since responses will be read aloud by Kokoro.
• MessagesPlaceholder(variable_name="chat_history"): a dynamic slot where the conversation history will be injected. The variable_name must match the history_messages_key we configure later.
• ("human", "{user_input}"): the current user message. The {user_input} placeholder must match the input_messages_key we configure later.

Assembling the Chain

from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import Runnable

# Initialize the output parser
parser = StrOutputParser()

# Create the LangChain chain
chain: Runnable = prompt | llm | parser

The StrOutputParser extracts the content field from the AIMessage returned by the LLM, giving us a plain Python string. Without it, chain.invoke() would return an AIMessage object rather than a string, and we would need to access .content manually everywhere we use the response.

The type annotation chain: Runnable is not strictly required but documents the expected interface. Every component in a LangChain expression, e.g. prompts, models, parsers, implements the Runnable interface, which defines .invoke, .stream, .batch, and their async equivalents.

Invoking the Chain

response = chain.invoke(
    {"user_input": text},
    config=CONFIG
)

Inside process_input, we invoke the chain with the current user input. The config dictionary carries session metadata (we will cover its role in the next section). At this point, chain has been wrapped with history management, so .invoke will automatically load conversation history before calling the LLM and save the new exchange afterward.

---

Persistent Conversation History

LLMs are stateless, i.e. each call to the model is independent, as the model has no memory of previous turns unless we explicitly include them in the prompt. For a chatbot, this means we must load prior messages and append them to every request.

LangChain addresses this through RunnableWithMessageHistory, a wrapper that automatically retrieves conversation history before each invocation and saves new messages afterward. The underlying storage mechanism is configurable.

SQLChatMessageHistory

We use SQLChatMessageHistory from langchain-community package, which persists conversation history in a SQL database via SQLAlchemy.

On every invocation, it:

• Reads all previous messages for the given session_id from the database.
• Injects them into the chat_history placeholder in the prompt.
• After the chain completes, writes the new human message and AI response back to the database.

from langchain_community.chat_message_histories import SQLChatMessageHistory

# Define the function to retrieve the persistent session history
def get_session_history(session_id: str):
    return SQLChatMessageHistory(session_id=session_id, connection=DATABASE_URL)

get_session_history is a factory function responsible for returning the chat history associated with a given session. On every invocation, RunnableWithMessageHistory calls this function, passing the session ID extracted from the config dictionary. For example, if the history is stored in a SQLite database configured as DATABASE_URL = "sqlite:///chat_history.db".

Wrapping the Chain

from langchain_core.runnables.history import RunnableWithMessageHistory

# Add persistent conversation history
chain = RunnableWithMessageHistory(
    chain,
    get_session_history,
    input_messages_key="user_input",
    history_messages_key="chat_history"
)

The four arguments:

Argument	Value	Purpose
runnable	chain	The base chain to wrap
get_session_history	Our factory function	Called to retrieve or create a history store
input_messages_key	"user_input"	Must match the input key in chain.invoke()
history_messages_key	"chat_history"	Must match the MessagesPlaceholder variable name in the prompt

The key names must be consistent across three places: the prompt template, the RunnableWithMessageHistory constructor, and the chain.invoke() call. A mismatch in any of these will cause a runtime error.

Session Management

The config dictionary passed to chain.invoke() carries the session identifier:

# config.py
SESSION_ID = "default-session"
CONFIG = {"configurable": {"session_id": SESSION_ID}}

In this application, we use a single fixed session for learning purpose, but in a production context with multiple users, each user or conversation would have its own session_id, keeping conversation histories isolated. The session_id can be any string, for instance a UUID, a username, or any identifier that uniquely scopes a conversation.

Execution Flow with History

---

Text-to-Speech with Kokoro

Kokoro is a text-to-speech model distributed in ONNX format. It runs entirely on CPU without requiring a GPU or an external API call. The kokoro-onnx Python package loads the ONNX model and provides a simple interface to synthesize speech from text.

Kokoro's ONNX format enables efficient CPU inference using ONNX Runtime, without requiring PyTorch or TensorFlow. This eliminates the need to install heavyweight deep learning frameworks while still providing high-performance inference.

Initializing Kokoro

from kokoro_onnx import Kokoro

# Initialize Kokoro
kokoro = Kokoro(model_path=KOKORO_MODEL_PATH, voices_path=KOKORO_VOICE_PATH)

We initialize Kokoro once at startup and reuse the instance for every speak() call. Loading the ONNX model has a non-trivial startup cost, so we avoid re-initializing it per request.

The voices_path points to a binary file that contains pre-computed speaker embeddings. Kokoro uses these embeddings to control the vocal characteristics of the synthesized speech. Each named voice (like "af_heart") is an identifier into this file.

Generating and Playing Speech

import sounddevice as sd

# Generate and play speech
def speak(text: str):
    # 1. Generate speech samples from text
    samples, sample_rate = kokoro.create(text, voice=KOKORO_VOICE, speed=KOKORO_SPEED)

    # 2. Ensure array shape and dtype are correct for sounddevice
    wav = samples.squeeze().astype("float32")

    # 3. Play audio synchronously
    sd.play(wav, samplerate=sample_rate, blocking=True)

kokoro.create() returns a NumPy array of audio samples and the corresponding sample rate (typically 24000 Hz). The samples represent normalized floating-point audio in the range [-1.0, 1.0].

We call .squeeze() to remove any extra dimensions that Kokoro may add to the array shape, and .astype("float32") to ensure the dtype matches what sounddevice expects.

sd.play() with blocking=True means the function does not return until the audio has finished playing. This is intentional as we do not want the assistant to start listening again before it has finished speaking, right?.

Voice Options

Kokoro ships with several pre-trained voices. The voice is controlled by the voice parameter in kokoro.create(). The KOKORO_VOICE constant in config.py can be changed to any voice included in the voices.bin file. Check out the Kokoro's HuggingFace page for the full list of available voice identifiers.

Suppressing Phonemizer Logs

Kokoro internally uses phonemizer to convert text into phoneme sequences before synthesis. phonemizer is verbose by default. We suppress its logs in the application:

import logging
logging.getLogger("phonemizer").setLevel(logging.ERROR)

This line only affects the phonemizer logger, leaving all other loggers at their default level.

---

Put It All Together

Now that we've covered each component individually, it's time to walk through the complete chatbot implementation. Each section of the file initializes a component or defines a callback, while the assistant.start() call at the end brings everything together.

# ai_chatbot.py

from config import *
import logging
import sounddevice as sd
from kokoro_onnx import Kokoro
from langchain.chat_models import init_chat_model
from langchain_community.chat_message_histories import SQLChatMessageHistory
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables import Runnable
from langchain_core.runnables.history import RunnableWithMessageHistory
from pywhispercpp.examples.assistant import Assistant
from rich.console import Console

# Configure phonemizer's logging to suppress verbose output
logging.getLogger("phonemizer").setLevel(logging.ERROR)

# Initialize Rich console for formatted terminal output
console = Console()

# Defines a function to get a persistent conversation history
def get_session_history(session_id: str):
    return SQLChatMessageHistory(session_id=session_id, connection=DATABASE_URL)

# Initialize the LLM
llm = init_chat_model(model=MODEL, model_provider=MODEL_PROVIDER)

# Defines a parser to extract the LLM response
parser = StrOutputParser()

# Defines a prompt template for the LLM
prompt = ChatPromptTemplate.from_messages([
    ("system", SYSTEM_PROMPT),
    MessagesPlaceholder(variable_name="chat_history"),
    ("human", "{user_input}"),
])

# Defines a runnable chain for the LLM
chain: Runnable = prompt | llm | parser

# Defines a runnable chain with a persistent conversation history
chain = RunnableWithMessageHistory(
    chain,
    get_session_history,
    input_messages_key="user_input",
    history_messages_key="chat_history"
)

# Initialize Kokoro once at startup
kokoro = Kokoro(model_path=KOKORO_MODEL_PATH, voices_path=KOKORO_VOICE_PATH)

# Defines a function to speak text
def speak(text: str) -> None:
    """Convert text to speech and play it through the audio output device."""
    # Synthesize speech samples from text
    samples, sample_rate = kokoro.create(text, voice=KOKORO_VOICE, speed=KOKORO_SPEED)

    # Normalize array shape and dtype for sounddevice
    wav = samples.squeeze().astype("float32")

    # Play audio synchronously — blocks until playback is complete
    sd.play(wav, samplerate=sample_rate, blocking=True)

# Defines a callback for the Whisper Assistant
def process_input(text: str) -> None:
    """
    Callback invoked by the Whisper Assistant after each transcribed utterance.
    Filters noise, invokes the LangChain chain, and speaks the response.
    """
    # Remove leading/trailing whitespace
    text = text.strip()

    # Discard empty strings and known Whisper noise artifacts
    if not text or text in IGNORED_TRANSCRIPTIONS:
        return

    # Display the recognized user input
    console.print(f"\n[bold cyan]You:[/bold cyan] {text}")

    # Invoke the LLM chain with a loading indicator
    with console.status("[bold yellow]Thinking...[/bold yellow]", spinner=STATUS_SPINNER):
        try:
            response = chain.invoke(
                {"user_input": text},
                config=CONFIG
            )
        except Exception as e:
            console.print(f"[red]Error:[/red] {e}")
            speak("Sorry, something went wrong.")
            return

    # Display the assistant's text response
    console.print(f"[bold green]Assistant:[/bold green] {response}")

    # Synthesize and play the response with a status indicator
    with console.status("[bold magenta]Speaking...[/bold magenta]", spinner=STATUS_SPINNER):
        try:
            speak(response)
        except Exception as e:
            console.print(f"[red]Speech error:[/red] {e}")

    # Signal that the assistant is ready for the next input
    console.print(
        "\n[bold blue]Listening...[/bold blue] "
        "[dim]Speak when you're ready.[/dim]"
    )

# Initializes the Whisper-backed voice assistant
assistant = Assistant(
    silence_threshold=WHISPER_SILENCE_THRESHOLD,
    commands_callback=process_input,
    n_threads=WHISPER_THREADS
)

# Prints the initial listening prompt and begins the audio capture loop
console.print(
    "[bold blue]Listening...[/bold blue] "
    "[dim]Speak when you're ready.[/dim]"
)
assistant.start()

Breaking Down the Code

1. We imported the required libraries, including LangChain, Whisper, Kokoro, Rich, and our application configuration.
2. We configured logging to suppress unnecessary messages from the phonemizer library.
3. We created a Rich console to display colored messages and loading animations.
4. We defined get_session_history(), which tells LangChain how to load and store conversation history in a SQLite database.
5. We initialized the language model using init_chat_model().
6. We created a StrOutputParser to extract the model's response as plain text.
7. We built a prompt template containing the system prompt, the conversation history, and the user's latest message.
8. We assembled a LangChain pipeline that sends the prompt to the language model and parses the generated response.
9. We wrapped the pipeline with RunnableWithMessageHistory so previous conversations are automatically loaded and persisted.
10. We initialized the Kokoro text-to-speech engine once at startup.
11. We implemented speak(), which converts text into speech and plays it through the speakers.
12. We implemented process_input(), the callback responsible for processing each Whisper transcription, generating a response, displaying it, and speaking it aloud.
13. We initialized the Whisper Assistant, configuring it to invoke process_input() whenever speech is recognized.
14. Finally, we displayed a listening message and started the assistant, allowing it to continuously listen for the user's voice.

The Execution Flow

---

Running the Application

With the virtual environment activated and the .env file populated:

python ai_chatbot.py

On the first run, pywhispercpp downloads the configured Whisper model. This is a one-time operation. Kokoro loads the ONNX model from the local models/ directory.

Expected Terminal Output

Listening... Speak when you're ready.

You: What is the capital of France?
Assistant: The capital of France is Paris.

Listening... Speak when you're ready.

You: And what is it known for?
Assistant: Paris is known for the Eiffel Tower, world-class cuisine, fashion, art museums like the Louvre, and its rich history as a cultural capital of Europe.

Listening... Speak when you're ready.

All responses are printed to the terminal, and their corresponding audio is played through the system's default audio output device.

Verifying Conversation Persistence

After interacting with the chatbot through several conversations, you can inspect the SQLite database to verify that the conversation history has been persisted correctly. The database should look like to the following:

Stopping the application with Ctrl+C and restarting it will resume the conversation from the stored history. The LLM will have access to all prior messages from the same session_id.

---

Next Steps

This implementation intentionally focuses on the core concepts. As a result, it has a few limitations, such as supporting voice input only (no text input), processing requests sequentially (listen → think → speak), and using a single conversation session.

Possible improvements include:

• Stream the LLM response as it's generated
• Stream speech synthesis for lower latency
• Add tool calling
• Support multiple conversation sessions
• Add a wake word
• Replace SQLite with PostgreSQL

Wrap Up

We have built a complete, voice-driven AI chatbot from raw audio input to spoken response. The system is composed of four layers: voice capture and transcription, conversational reasoning, persistent memory, and speech synthesis. Each component is implemented by a specific library and connected through a clean, callback-driven architecture.

This guide has an accompanying GitHub repository, making it easy to run, modify, and extend the chatbot on your own.

Happy AI coding 🚀