Talk to the Web: Building a Hands-Free Form Assistant with Agora Voice Agent SDK

Think about the last online form you filled out - a government services portal, a bank application, or a simple signup page. Name, email, phone number, multi-select dropdowns, and checkboxes hidden behind tiny scrollable windows.

For many of us, it's a minor inconvenience. But for the elderly, individuals with disabilities, or those who aren't tech-savvy, a complex online form isn't just annoying - it's a digital wall. Whether someone is navigating a tremor, managing low vision, using one hand, or simply trying to fill out a field while multi-tasking, these traditional interfaces often lead to frustration and abandoned pages.

Accessibility on the web shouldn't be an afterthought. What if, instead of typing and clicking, you could simply talk and have the page respond?

From Hackathon Idea to Reality

That question originally sparked GabAI - a concept my team pitched during a hackathon. The vision was straightforward: build a voice assistant that lives in your browser side panel, filling out web forms with you in real time out loud, and confirming details before anything is submitted.

At the time, building a truly responsive, low-latency Voice AI agent felt like a massive undertaking, and we didn't get to complete it during the hackathon. But with the rapid advancement of real-time conversational AI tools, revisiting this idea became a whole lot easier.

Powered by Agora

Using the Agora Agents SDK and Agora Skills, the heavy lifting of real-time audio orchestration and conversational state management was already handled. What used to require complex custom voice pipelines can now be wired directly into a web interface.

The result is a working Chrome extension that enables hands-free, accessible form filling right inside your browser side panel.
It's still a demo, and I'll be completely transparent about its current limits later in this post. But building GabAI showed me just how accessible creating Voice AI agents has become - and how much power Agora gives developers to bridge digital accessibility gaps. Let's dive into how it works!

Demo Video

drive.google.com

Why Agora?

A voice agent that feels like a conversation has a lot of moving parts. You need low-latency audio in both directions, speech-to-text, an LLM, text-to-speech, and turn-taking: knowing when you've finished a sentence, and stopping mid-reply if you interrupt. Then you need a way to show a live transcript and agent state in the UI.

With Agora, I didn't build any of that plumbing. The backend describes the agent, and Agora's cloud runs it:

llm = OpenAI(
   model="gpt-4o-mini",
   greeting_message=GREETING,
   mcp_servers=[{
       "name": "gabaitools",
       "endpoint": mcp_endpoint,
       "transport": "streamable_http",
       "headers": {"Authorization": f"Bearer {mcp_secret}"},
       "allowed_tools": ["*"],
       "timeout_ms": 90000,
   }],
)
stt = DeepgramSTT(model="nova-3", language="en")
tts = MiniMaxTTS(model="speech_2_6_turbo", voice_id="English_captivating_female1", speed=1.2)

agent = AgoraAgent(
   client=client,
   instructions=GABAI_PROMPT,
   greeting=GREETING,
   turn_detection={...},
   advanced_features={"enable_rtm": True, "enable_tools": True},
).with_stt(stt).with_llm(llm).with_tts(tts)

That's the speech pipeline: a speech-to-text vendor, an LLM and a text-to-speech vendor, chained with with_stt, with_llm and with_tts. I used the managed defaults from Agora's official quickstart, so there were no separate vendor accounts to set up.

Three things stood out while building:

1. Turn-taking is configuration, not code.

How long the agent waits after you stop talking, and how quickly it yields when you interrupt, are a few numbers in turn_detection. I exposed the silence window as an environment variable (GABAI_SILENCE_MS, default 450) so it can be tuned without touching code. Lower is snappier but may cut you off mid-thought. That trade-off is a setting you can tune.

2. Tools are first-class.

Setting enable_tools and pointing the LLM at an MCP server is how the agent gets hands. Agora's cloud agent connects to my endpoint, lists the tools, and calls them when the model decides to. I watched this happen in the backend log: initialize, notifications/initialized, tools/list, then a tools/call for each action. The default Agora-managed gpt-4o-mini handled MCP tool calls in my setup. I found no mention of that in the docs, so I mention it here as something I observed rather than something guaranteed.

3. The client side is also covered.

The side panel uses Agora's RTC SDK for the mic and speaker, RTM for the live transcript and agent state, and the agora-agent-client-toolkit to tie them together. The panel shows whether GabAI is listening, thinking or speaking without any custom signalling from me.

The tricky part: the cloud can't reach your browser

Here's the catch. The Agora agent runs in the cloud. Your Chrome tab is on your laptop. The agent decides "fill the email field", but there is no route from the cloud to your page.

The fix is a relay. Reading left to right:

The extension opens a WebSocket outward to my local backend. When the agent calls a tool, Agora sends it to the backend's /mcp endpoint through a Cloudflare quick tunnel. The backend forwards the call down the WebSocket to the side panel, the panel hands it to the extension's service worker, and the worker passes it to the content script on the authorized tab. The result travels back the same way and the LLM turns it into speech.

What else becomes possible

GabAI is one small shape of a bigger idea: a voice agent with a few well-chosen, tightly limited tools. The same pattern (Agora for the real-time voice loop, MCP for the hands) fits many other places.

These are ideas, not things I've built:

• Hands-busy work. A cook asking for the next step and a timer, a mechanic pulling up a torque spec, a lab worker logging a result without taking off gloves.
• Data entry in the field. A clinician or inspector dictating into a structured form and confirming it at the end, instead of typing on a small screen.
• Accessibility as a default. People with motor, visual or cognitive disabilities who find keyboards and mice hard. A voice layer that can read, fill and confirm could sit in front of many existing sites.
• Voice for the long tail of old software. Internal tools that will never get a redesign could get a talking front door.

The common ingredient is a conversation that can do something, with a human always holding the final yes.

Build your First Voice AI Agent

Using Agora Skills and Agora Agents SDK, building Voice Agents has never been easier.

Prompt your coding assistant to install Agora Skills using:

npx skills add AgoraIO/skills

Then, refine your project idea first in a separate AI e.g. ChatGPT.
After that, prompt your AI coding assistant to build your project idea with the use of Agora Skills and Agora Agents SDK:

Use Agora Skills and Agora Agents SDK to help me build [PROJECT_IDEA]
Check the official docs first, choose the right starter, and use the Agora CLI for setup