🎙️ Building a Voice-Controlled AI Agent with Tool Execution

In this article, I’ll walk through how I built a voice-controlled AI agent that can understand user commands, decide what action to take, execute tools like file creation or code generation, and respond naturally — all through a simple web interface.

---

🚀 Overview

The goal of this project was to move beyond a basic chatbot and build something closer to an agentic system — where the model doesn’t just respond, but decides what to do.

The agent supports:

• 🎤 Voice input (record or upload audio)
• 🧠 Speech-to-text using OpenAI Whisper
• 🤖 LLM-based decision making (no hardcoded intent rules)
• 🛠️ Tool execution (file creation, code generation)
• 💬 Natural language responses
• 🖥️ Interactive UI using Streamlit

---

🏗️ Architecture

At a high level, the system looks like this:

User (Voice Input)
        ↓
Speech-to-Text (whisper-1)
        ↓
Agent (LLM decides action)
        ├── If "final" → respond normally
        └── If tool → execute tool
                    ↓
              Tool Result
                    ↓
              Sent back to LLM
                    ↓
          Final Natural Response
                    ↓
            Streamlit UI

---

🧠 Agent Design (Core Idea)

Instead of using traditional intent classification, I implemented an agent loop:

1. Send user input to the LLM
2. LLM returns structured JSON like:

   {
     "action": "write_code",
     "input": { ... }
   }

1. If it's a tool → execute it
2. Feed tool result back to the LLM
3. LLM generates a final natural response

This approach removes rigid pipelines and makes the system fully flexible and extensible.

---

🛠️ Tools Implemented

Currently, the agent supports:

• 📁 create_file → Create files in a restricted folder
• 💻 write_code → Generate and write code files
• 💬 General chat fallback

All file operations are sandboxed inside an output/ directory for safety.

---

🤖 Models Used

• Speech-to-Text: whisper-1
• LLM: gpt-4o-mini

💡 Why gpt-4o-mini?

Due to limited token budget (I paid for OpenAI credits myself), I had to choose a cost-efficient model.

However, I do have hands-on experience working with:

• GPT-4 / GPT-5 series (up to GPT-5.4)
• Claude Sonnet / Opus models
• Other flagship LLMs

So the choice here was purely practical, not due to lack of exposure.

---

⚠️ Challenges Faced

1. Streamlit (Biggest Pain Point)

Honestly, the hardest part of this project was not the AI — it was Streamlit.

Problems I faced:

• Session state management is unintuitive
• Frequent unwanted reruns
• Hard to control UI flow
• Debugging is painful

2. Audio Input Handling

Handling both:

• 🎤 Recorded audio
• 📤 Uploaded audio

…was surprisingly tricky.

Issues included:

• Audio not updating correctly
• Previous audio persisting
• Preview disappearing unexpectedly
• Send button not triggering properly

Getting this right required careful control of:

• session_state
• widget keys
• rerun timing

---

🧩 Key Learning

The biggest takeaway:

Building AI systems is not just about models — it’s about managing state, UI behavior, and system flow.

The agent logic was actually straightforward.

The real complexity came from:

• UI framework limitations
• State synchronization
• Event-driven behavior

---

🔥 Example Flow

User:

"Create a Python file with a retry function"

System:

1. Audio → transcribed to text
2. LLM decides → write_code
3. Tool generates Python code
4. File saved in output/
5. LLM explains what was done

---

🚀 Future Improvements

• Streaming responses (ChatGPT-like typing effect)
• More tools (API calls, DB queries, etc.)
• Better UI framework (possibly replacing Streamlit)
• Multi-step reasoning chains
• File preview & download in UI

---

🧠 Final Thoughts

This project was a great exercise in building a real-world agent system.

While LLMs make reasoning easy, the real engineering challenge lies in:

• system design
• tool orchestration
• UI-state synchronization

And surprisingly… debugging Streamlit 😄