An AI agent that makes phone calls (OxHack ‘26)

This is a submission for the MLH x DEV Writing Challenge

What I Built

Smart Connect gives AI agents something they usually don't have: the ability to pick up the phone.

Ask most AI assistants to book you a haircut and they'll give you a phone number. You still have to make the call, wait on hold and go back and forth over times. We wanted an agent that actually does it.

In Smart Connect you create your own agents and choose exactly what each one is allowed to do: phone calls, email, calendar, payments. Then you just ask in plain English:

"Book a dentist appointment tomorrow at 3pm. If that doesn't work, 2 or 2:30 is fine."

The agent plans the call, a voice agent phones the business and has a real conversation, and you get back the transcript, a short summary and the confirmed booking. It only offers the alternatives you allowed, and it never agrees to pay a deposit without your approval.

The biggest thing I learned: with voice agents, less is more. Every time a call sounded robotic, our instinct was to add another rule to the prompt, and every rule made it worse. Short, simple instructions are what made it sound human.

Demo

Code: https://github.com/JamilHassiba/oxhack26

What a call looks like in the app:

1. You send your request in the chat.
2. The card shows the call happening: Calling → Ringing → On the call, with a timer and a Hang up button.
3. When the call ends, the full transcript appears, followed by a summary and a result table (business, date, time, details).

Partner Technologies

Gemini API: the brain

Gemini runs each agent. We used function calling so it can turn a message into a place_call with the business name, the goal, any constraints, and the alternative times you said were fine.

The tricky part: a phone call takes minutes, so Gemini can't just wait for an answer. We pause the function-calling loop when the call starts, save its state, and resume it when the call ends. Gemini then reads the transcript, extracts a structured result with JSON output (booked? what time? what else was offered?), and decides what to do next. That might be calling back once with an offered time you allowed, adding the event to your calendar, or emailing you a confirmation.

ElevenLabs: the voice

The phone conversation itself is an ElevenLabs Conversational AI agent, placed over Twilio. For every call we pass it dynamic variables: who it's calling for, the goal, the constraints and the allowed alternatives. When the call ends, ElevenLabs sends the full transcript to our backend through a signed post-call webhook (we verify the HMAC signature). We also gave it a server tool for asking the user to approve a deposit mid-call.

What surprised us: ElevenLabs only gives you an agent call's transcript after the call ends. We spent a while building a "live transcript" that polled for it mid-call, before realising the data simply isn't there yet. So we switched to following the call's status through Twilio and built an animated "on the call" view instead.

Tiger Data: every call as a time-series

Our whole database is Tiger Data (Postgres + TimescaleDB). Each call is stored as a time-series of events (dialling, ringing, answered, every transcript line, ended) in a hypertable. The chat streams those events to the browser with server-sent events, and that's what moves the call card from "Calling" to "On the call" in real time. Modelling a phone call as time-series data turned out to be a really natural fit.

Vultr: hosting

We deployed on a Vultr VM in London: FastAPI running under systemd, behind Caddy for HTTPS, which also serves the React frontend. Having a real public URL mattered, because ElevenLabs needs somewhere to send its webhooks.

Hackathon Experience

We built Smart Connect at OxHack '26 in Oxford.

What broke at 3am:

• The clocks went back. UK time switched from BST to GMT during the hackathon weekend, and the model kept guessing the wrong offset, so "6pm" bookings could land an hour out. We ended up attaching the UK time offset ourselves.
• The voice agent read times like a robot. It said "eighteen hundred", then "Monday 26 October 2026", and was far too cheerful about everything.
• Twilio wouldn't let us call. A trial account can't call unverified numbers, so getting the first real call through took some creative problem-solving.

We started as a team of three and finished as two after a teammate had to leave, so we re-planned mid-hackathon and kept going.