DEV Community

TAMSIV
TAMSIV

Posted on

How I Gave My Voice AI the Ability to Read Your Schedule (Not Just Write To It)

#ai #reactnative #supabase #productivity

For five months, my voice AI assistant could create tasks, memos, and calendar events. It could write. But it couldn't read.

If you asked "What do I have tomorrow?", it had no idea. It only knew write commands — create, update, delete. Your actual schedule? Completely opaque.

This week I changed that. Here's how.

The Problem

My app TAMSIV uses a voice pipeline: audio → Deepgram STT → OpenRouter LLM → function calling → OpenAI TTS. The LLM has 7 function tools: create_task, update_task, create_memo, update_memo, create_calendar_event, ask_clarification, end_conversation.

All write operations. The AI could create things in your schedule, but couldn't tell you what was already there.

The Solution: An 8th Tool

I added query_agenda(startDate, endDate) — a read tool.

When you ask "What's on my plate this week?", the LLM:

  1. Parses your intent and calls query_agenda with the right date range
  2. A Supabase RPC function queries three tables: privat.tasks, privat.calendar_events, and privat.memos
  3. Returns structured data: tasks with due dates, events with times, memos with timestamps

But raw data isn't useful as a voice response. Nobody wants to hear "Task ID 47, title grocery shopping, due date 2026-03-24T10:00:00Z."

The 2nd LLM Call

This is where it gets interesting. The orchestrator makes a second LLM call — taking the raw agenda data and generating a natural vocal summary:

"Tomorrow you have three things. In the morning, grocery shopping. At 2pm, a dentist appointment. And you still have that memo about calling the insurance company."

Natural. Conversational. Adapted for voice.

The Architecture Change That Surprised Me

The FunctionExecutor — the component that runs tools — was synchronous and had no database access. It only returned instructions that the frontend would execute.

For query_agenda, the backend needed to query Supabase directly. So the FunctionExecutor became async, with access to the Supabase client and userId.

// Before: synchronous, no DB access
executeTool(name: string, args: object): ToolResult

// After: async with DB context
async executeTool(
  name: string, 
  args: object, 
  supabase: SupabaseClient, 
  userId: string
): Promise<ToolResult>
Enter fullscreen mode Exit fullscreen mode

This opened the door for future read tools — the AI can now query anything server-side.

The RPC Function

The Supabase RPC crosses three tables in a single query:

CREATE OR REPLACE FUNCTION get_user_agenda(
  p_user_id UUID,
  p_start_date TIMESTAMPTZ,
  p_end_date TIMESTAMPTZ
) RETURNS JSON AS $$
  SELECT json_build_object(
    'tasks', (SELECT json_agg(...) FROM privat.tasks WHERE ...),
    'events', (SELECT json_agg(...) FROM privat.calendar_events WHERE ...),
    'memos', (SELECT json_agg(...) FROM privat.memos WHERE ...)
  );
$$ LANGUAGE sql SECURITY INVOKER;
Enter fullscreen mode Exit fullscreen mode

Using SECURITY INVOKER means RLS policies are enforced — users only see their own data, even when the backend calls this function.

Frontend: No PendingCreation

Every other tool creates a PendingCreation — a preview the user validates before it's saved to the database. But query_agenda doesn't create anything. The frontend just needs to handle the agenda_queried action by... doing nothing. The voice response is the result.

The Bigger Picture

This changes the nature of the voice assistant fundamentally. Before, it was an input device — you talk, it writes. Now it's a conversational partner — you ask, it answers.

"What do I have this week?" → vocal summary
"Create a task: buy flowers for Friday" → creates task
"Actually, what else is on Friday?" → checks and responds
"Add a reminder for 5pm" → creates event

The conversation flows naturally because the AI can both read and write.

700+ commits. Solo dev. Production in 5 days.

If you're building voice AI features, the key insight is: read tools change everything. An AI that can only write feels like a command interface. An AI that can read and write feels like an assistant.

Top comments (0)