DEV Community

anusha
anusha

Posted on

I Built a Website Voice Assistant That Can Actually Control the Page

#telnyx #ai #devrel

A lot of website AI demos stop at answering questions.

The assistant can explain where something is, but it does not actually take you there. It can tell you to open settings, but it does not open settings. It can ask you to fill out a form, but it does not touch the form.

That is the difference I wanted to show with Telnyx AI Assistant client-side tools.

The Telnyx code example is here:

https://github.com/team-telnyx/telnyx-code-examples/tree/main/ai-assistant-client-side-tools-nextjs

The process is clone the sample, configure the same tool names and JSON schemas on your Telnyx AI Assistant, set the public assistant ID in the browser app, and then test the UI actions from the website call button.

The Idea

Client-side tools let a Telnyx AI Assistant invoke predefined JavaScript functions in the browser during a voice or chat conversation.

That sounds small, but it changes the demo.

Instead of saying:

Go to the AI Assistants page and click Create Assistant.
Enter fullscreen mode Exit fullscreen mode

the assistant can call:

navigate_to_section
open_create_assistant_modal
Enter fullscreen mode Exit fullscreen mode

The browser updates immediately because those functions are normal React state updates.

What I Built

The sample is a fictional SaaS dashboard called PolarForge AI. It is built with Next.js, React, TypeScript, and Tailwind CSS.

It has a sidebar, several dashboard sections, light and dark themes, an AI Assistants page, a Create Assistant modal, and an activity panel that logs every tool call.

The assistant can use five tools:

  • set_theme
  • navigate_to_section
  • open_create_assistant_modal
  • get_form_state
  • update_assistant_form

The interesting one is get_form_state, because it has to read current React state. It is not returning a hard-coded object. If the modal is open and the form says the assistant name is "Enterprise Concierge", the tool returns that visible state.

Why Client-Side Instead of Webhook?

Webhook tools are still useful. If the assistant needs to create an account, charge a card, update a CRM record, or call something with privileged credentials, that belongs on the backend.

But a lot of website actions are local:

  • switch to dark mode
  • open a modal
  • navigate inside a single-page app
  • fill a visible form
  • read page state
  • call APIs the browser is already authenticated to call

That is where client-side tools are a better fit. The assistant does not need a backend webhook just to change React state.

The Integration

The app uses @telnyx/ai-agent-lib:

<TelnyxAIAgentProvider agentId={agentId}>
  <ToolRegistrar executeTool={executeTool} />
  <TelnyxWidget />
</TelnyxAIAgentProvider>
Enter fullscreen mode Exit fullscreen mode

Inside React, the tools are registered with:

client.registerClientTool("navigate_to_section", async (args) => {
  return executeTool("navigate_to_section", args);
});
Enter fullscreen mode Exit fullscreen mode

The matching tool definitions still need to exist on the Telnyx AI Assistant. The names and JSON schemas need to match what the browser registers.

The Demo Moment

The clean version of the demo is:

User: switch to dark mode
Assistant: calls set_theme
Page: changes to dark mode

User: take me to AI Assistants
Assistant: calls navigate_to_section
Page: switches section without reload

User: create an assistant named Enterprise Concierge
Assistant: opens the modal and updates the name field
Page: shows the form update
Activity panel: logs every tool call
Enter fullscreen mode Exit fullscreen mode

That is the proof: the assistant is not just talking about the app. It is controlling the app through approved browser functions.

Security Boundary

The browser does not get a Telnyx API key.

The sample uses public NEXT_PUBLIC_* configuration for the AI Agent Lib connection. If you need to provision an assistant or call the Telnyx REST API, do that from a secure server-side environment or through the Telnyx Portal.

The sample also validates every tool call. Unknown sections fail. Unknown form fields fail. Invalid voices or languages fail. Updating the form while the modal is closed fails.

That is the shape I would keep in a production app: small explicit tools, strict validation, and no privileged secrets in client-side code.

Resources

Top comments (0)