DEV Community: Dhruv

No-Show Desk: TabPFN spots the motel guests likely to cancel, and Gemma asks them first

Dhruv — Sun, 18 Oct 2026 00:06:15 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

A small motel loses a night's room every time a guest silently doesn't turn up and when no deposit was taken, nothing comes back. No-Show Desk is a morning assistant for small motels and B&Bs. It picks the arriving guests most likely to cancel then says why in plain words and drafts a short "are you still coming?" message that the owner sends or skips. Scored on 30 real past mornings, learning only from earlier bookings, its picks caught 111 of 172 cancellations, against about 64 for random picks of the same size.

Here is the demo morning (real 2017 bookings; guest names are made up). At 8 a.m. a Telegram message lands on the owner's phone:

Good morning. 7 guests arriving in the next 3 days might not show: $1,599.20 of bookings. Start with Chloe Nguyen (74%), Daniel Okafor (67%) and Ana Silva (61%), plus 4 more.

They tap through to one page. Chloe's row opens: room only, booked 72 days ahead, and "34% of past room-only bookings cancelled, against 26% of all past bookings without a deposit." They press Draft a message. A few seconds later there's a short note asking Chloe to confirm she's still coming or to say if her plans changed. Under it, in green: Dates, nights and total match the booking. They press Send by email. The owner decides everything; nothing is sent on its own.

I'd been pitching motels a voice agent that answers the phone when nobody's at the desk. When this week's theme asked for something a small business would actually pay for, motels were the business I knew, so I built something smaller: one list and one message each morning, simple enough for a few owners to try.

TabPFN, a model already trained to learn from small tables, scores every upcoming booking from the motel's own past bookings.
Gemma 4, Google's open-weight model, writes the words on the owner's own computer, right inside the browser.
Plain code works out every date, night count and total, and checks every draft before the owner can send it.

On real past bookings, outcome hidden until scoring	Measured
Cancellations the desk caught ahead of time, of all that happened, over 30 test mornings	111 of 172 (random picks of the same size: about 64)
Drafts with a wrong date, night count or total	13 of 100 when Gemma worked them out, 0 of 100 when code handed them over
Seconds per message, Gemma in the browser against Ollama (a program that runs models locally) on the same Mac	3.25 against 31.4 (median of 100; two builds of the same small model)

The rule I built everything around: code computes, the model phrases.

Who pays. The owner of a small motel or B&B, who pays for a booking system but has no data team. On the demo morning, 4 of the 7 flagged guests did cancel, holding $776.60 of bookings, and a single night there is $67 to $107. The value is hearing sooner, while the room can still be resold, so the desk has to cost less than one resold night a month. Gemma costs nothing per message.

Try it live: no-show-desk.onrender.com. Every visitor gets their own copy of the demo morning. On a computer, the first Draft asks before downloading Gemma (2 GB), then writes each message in a few seconds (tested in Chrome). Phones get the pre‑written message from Gemma, checked the same way. The free server can take up to a minute to wake. The 08:00 Telegram message comes from the version that runs on my computer.

Today: the guests who might not show, riskiest first. A row opens into the booking, the reasons, Draft and Leave (no message, with a few seconds to undo). Edit a draft and get a fact wrong, and the desk flags it as you type.
Replay: pick any of 30 real past mornings, see who the desk would have messaged, then press Reveal what happened. On the demo morning, Chloe (74%) really did cancel, and so did 3 of the other 6 flagged guests; Tara, at 26%, was below the 30% line the desk uses to decide who to flag and cancelled anyway. Drag the line or press Replay all 30 mornings to add it up against random picks.

The demo lives on a fixed day in early 2017 because the data is real bookings that end in 2017. Guest names and contacts are made up. Prices are the hotels' own numbers and the data doesn't name the currency (the hotels are Portuguese, so probably euros) plus the page shows them with a $ sign. Send writes to a file and says demo: not delivered.

Code

My-CMDhub / no-show-desk

Find out which guests won't turn up, days before they don't. A morning no-show desk for small motels: TabPFN flags risky bookings, local Gemma drafts a reconfirm message, code checks every fact.

Know who won't turn up, before they don't.

A morning assistant for small motels and B&Bs. It flags the guests most likely to cancel or not show,
says why in plain words, and drafts a short "are you still coming?" message for the owner to send.

Try the live demo · your own copy of the demo morning · Gemma writes in your browser on a computer

Why

A small motel loses a night's income every time a guest silently doesn't arrive. Many of those cancellations can be heard about earlier, if someone asks. No-Show Desk picks who to ask each morning, so the owner spends a minute on it, and a room that would have stood empty can be offered to someone else.

How a morning works

08:00, a Telegram message. "7 guests arriving in the next 3 days might not show: $1,599.20 of bookings. Start with Chloe Nguyen…

View on GitHub

noshow/: the app. Risk, reasons, facts, drafting, the checker, the agent, the page and the Telegram bot. tests/: 61 tests for the logic.
scripts/ and results/: every measured result in this post comes from a script in the first and is written to the second, including every stored draft. The few things I only observed are said as such.
render.yaml and results/demo_snapshot.json: the live demo.

How I Built It

The data, and the columns that give the answer away

There's no public motel data, so I used the Hotel booking demand dataset (Antonio, Almeida & Nunes, 2019): 119,390 real bookings from two Portuguese hotels, 2015 to 2017. All 1,207 no-shows are marked cancelled, so the desk predicts both. Every test works on a motel-sized slice: about 3,000 past bookings and 20 arrivals a morning.

Some columns are only filled in after the guest arrives, so they give the answer away: bookings with a parking space recorded cancelled 0.0% of the time because the space is recorded when the car turns up. I dropped six such columns and every test learns only from bookings that arrived before a date and is scored on bookings after it, the way the desk is used.

The desk only scores bookings without a deposit: a guest who prepaid and then cancels costs the owner nothing (non-refundable bookings in this data "cancel" 99.4% of the time). All the percentages below are among bookings without a deposit.

TabPFN at motel scale

A 20-room motel has a few thousand past bookings at most. TabPFN was trained in advance on a huge number of synthetic tables, so it learns a new one in a single step, with no training run of my own. I gave it, gradient boosting (the usual tool for tables) and a plain logistic regression 300, 1,000 or 3,000 past bookings, three histories each and scored the same 5,000 bookings from 2017. The score is ROC-AUC: how often a guest who cancelled is ranked above one who didn't.

TabPFN ranked best in all 9 runs, with gradient boosting (default settings) furthest behind. The plain regression came much closer than I expected (0.699, 0.741 and 0.749), so the honest summary: TabPFN wins at every size, by 0.02 to 0.03 over the regression.

Checking the cut-off on a year it never saw

My first version flagged anyone over 50%. On my first demo morning it flagged 1 guest of 20, who didn't cancel, and missed all 4 who did: with only about 3 in 10 guests cancelling, few scores ever reach 50%. So I picked 30% on 30 mornings from 2017 then, because choosing and reporting on the same mornings is peeking, checked it on 30 mornings from 2016 that the choice never saw. Two measures: cancellations caught, of all that happened, and how many flagged guests really cancelled.

Flag when	2017 (picked here): cancellations caught	flagged guests who did cancel	2016 (never seen): cancellations caught	flagged guests who did cancel
5 riskiest	84 of 172	56%	81 of 134	54%
25% or more	129 of 172	49%	105 of 134	52%
30% or more	111 of 172	53%	92 of 134	56%
50% or more	70 of 172	65%	66 of 134	77%

On 2016, which the choice never saw, 30% still caught more than the 5-riskiest rule or the 50% line, and 50% left 2 mornings with nobody flagged. 25% catches more, but fewer of its flagged guests cancel. Replay in the demo recomputes these rows in browser and they match the results files exactly.

Is it better than a rule I could follow by hand?

All seven guests flagged on the demo day booked through the same travel agent (#9; the data numbers agents), so could an owner just follow a rule? Each rule got the same number of messages per morning as the desk:

Same number of messages each morning	Cancellations caught, of 172
TabPFN, 30% line	111
Logistic regression, fitted on each morning's history	106
Online travel agent first, then longest lead time	99
Travel agent #9 first, then longest lead time	94
Longest lead time first	81

The best rule I could think of finds 99. TabPFN finds 111 by weighing everything together, 5 more than the regression.

Reasons a person can read

TabPFN gives a probability, not a reason, so the reasons come from plain counting. A guest's group (lead time, travel agent, channel, meal plan) is named only if it cancelled at least 5 percentage points more often than all bookings without a deposit, over at least 30 of them. The reasons describe the group; the ranking is TabPFN's.

Code computes, the model phrases

I gave Gemma the same 50 bookings twice, two drafts each. Once as the raw record: arrival 2017-08-23, 2 weekend nights, 5 week nights, average daily rate 160.2. Once with every fact worked out by my code: arrive Wednesday 23 August, leave Wednesday 30 August, 7 nights, $1,121.40.

From that very record, Gemma wrote to Grace: "your booking is confirmed for August 23 to August 25, 2017. That is 2 nights for a total of $320.40." It's 7 nights and $1,121.40. Another offered "cancel for free until April 0th." All of them were fluent and friendly.

A plain-Python checker reads every draft's dates, night counts and dollar amounts against the booking. If a draft is wrong, the owner sees exactly what's wrong before deciding. Three times, reading the drafts myself caught something no count showed:

My checker was wrong first. Its first count said 25 bad drafts from the raw record against 1 from code's facts. I read all 26 flagged drafts, and 13 were correct drafts in formats it couldn't read yet ("2/13", "Feb 23", a total without "$"). I taught it those formats, test first, and rescored the same stored drafts: 13 against 0.
The desk offered dates that had passed. The demo motel's policy (my choice; the data has none) ends free cancellation 2 days before arrival, so for almost every guest it messages, that date was gone. Code now leaves a passed date out.
My own instruction leaked into a message. I had told Gemma to mention the free-cancel date "if you are given one". It wrote "You can cancel for free until [Insert Date Here]" once in 6 test drafts. Every number was right, so the checker passed it. The fix was the same principle: code decides whether that sentence exists at all and the checker now catches placeholders.

An agent that waits for a person

Each guest is one run of a small LangGraph graph, a library for agents that can stop and wait for a human. It pauses twice for the owner: Draft or Leave, then Send or Reject once the draft is checked. Leave and Reject are stored as different outcomes, because "no message needed" and "this draft was wrong" mean different things.

Gemma inside the browser

On my 8 GB Mac, Gemma through Ollama takes about 30 seconds a draft. Google also publishes a 2 GB build of the same small Gemma 4 that runs inside a web page, on the computer's graphics chip (through WebGPU). So, I gave it the same 100 drafts and the same checker.

None of its 100 drafts had a wrong fact, at a median of 3.25 seconds against 31.4 (the two builds aren't identical files). Reading all found something the checker isn't built for: a few drafts leave a fact out, like the night count on a one-night stay, 6 in the browser and 5 through Ollama. None stated anything false.

So the page drafts in the browser and it asks before the 2 GB download, says what the file is and where it comes from, and shows real progress. In a browser on my Mac the download took about 3 minutes and loading about 10 seconds. Real devices changed the page twice. My iPhone 15 Pro downloaded the whole model, then Safari reloaded the page, most likely out of memory, so phones now get the recorded message instead. On my 8 GB Mac the browser froze for a while just after Gemma loaded but only on one test.

Why Does Open Innovation Matter?

The message stays with the motel. A no-show message holds a guest's name, dates and money. Gemma writes it on the owner's own computer, so none of that leaves your disk to be phrased.

There's no bill per message. The desk flagged about 7 guests a morning on the test days; with a paid writing API, that cost grows every morning. Here the model is a file on the owner's disk.

Small businesses have small data. TabPFN learns from a few thousand rows with no training run and no data scientist, which is the size of a real motel's history.

I could read and change every step that writes or checks a message. When Gemma leaked "[Insert Date Here]", I could see the exact instruction, change one line, and rerun 100 drafts.

One step isn't local yet: TabPFN scoring runs through the API of Prior Labs, so booking details (never names; this data has none) leave the computer for that one call each morning. TabPFN's v2 weights can run on computer instead and their licence allows commercial use with attribution. I checked: on my 8 GB Mac, at about 23 seconds a morning, v2 caught the same 111 of 172 on the 30 test mornings, flagging 205 guests instead of 210 (ROC-AUC 0.787 against the API's 0.792). That's the path for a paying version.

My First User

Not yet. I told a motel owner about the desk: someone I'd earlier pitched a voice agent to. He said he'd use it if he had more cancellations, and that it's handier, saves money and is easier to set up than the voice agent I'd pitched him. He doesn't have many cancellations right now, so he hasn't tried it and I can't tell you what an owner would pay. A second owner, introduced through a friend, is next. The first question I'll ask is whether a reply ever arrived early enough to resell the room.

Prize Categories

TabPFN. The risk step inside an agent. TabPFN scores each morning's bookings from only earlier bookings and a LangGraph run per guest acts on the score, pausing for the owner. It was tested the way it's used: on 30 later mornings it caught 111 of 172 cancellations, against about 64 at random and the 30% line held on a 2016 year it was never chosen on (92 of 134). It beat gradient boosting at every history size, a plain regression by 0.02 to 0.03 ROC-AUC and the best hand rule by 12 cancellations. Run on an 8 GB Mac with TabPFN's v2 weights, it caught the same 111, in about 23 seconds a morning.
Gemma. Gemma 4 E2B, the smallest of Gemma 4's five sizes, writes every message, in the visitor's browser or through Ollama on the owner's Mac. On the same bookings, wrong dates or totals fell from 13 to 0 of 100 when code handed Gemma the facts. The browser build drafted the same 100 in a median 3.25 seconds against 31.4 through Ollama on the same Mac, with no wrong fact. A checker reads every draft and the owner's own edits before send.
Render. Render runs the agent's front end from a render.yaml blueprint. Every visitor gets their own desk, where each guest is an agent run on the server that waits for Draft or Leave, then Send or Reject. The TabPFN scores were worked out earlier on my Mac and saved, and a test fails if the saved morning drifts from what the desk works out live.

What's real, and what isn't

Real: the bookings, TabPFN's scores, the reasons, Gemma's drafts and every check. The demo day is the earliest of the 30 test mornings that flagged 7 guests, the average.
Not measured: whether a message gets a guest to reply or a room resold. That's the part only a real motel can test and it's what I'd build next: the desk running every morning on a live booking system, with the replies counted.

Changing what my phone agent was shown beat changing the model

Dhruv — Wed, 07 Oct 2026 00:19:31 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

On 18 September 2026, two turns after its own lookup had returned a booking's real email, my AI phone receptionist told the caller no email was on file. Then it read one back anyway: "Dhruv Patel at gmail dot com". That's my name, at an address that exists nowhere.

I didn't change the model. I changed its notes, the facts my code hands it on every turn of the call. Across six models on Kaggle, invented emails went from 17 of 216 runs to none.

The same move cut two more mistakes from real calls, and a fourth I found by probing. On one of them, the weakest model after the fix (14 of 18 runs right) beat the strongest model before it (4 of 18).

The three mistakes from real calls. The fourth, found by probing, is in Finding 4. A "run" is one model answering one scripted call once.

The pattern, in three rules: withhold what must never be said, carry what will otherwise be guessed, compute what it can't work out.

My agent answers the phone for a motel. It finds bookings, resends confirmations and puts callers through to reception. Its worst mistakes never crash anything. They sound polite and fluent, and they are wrong. So my itch was simple: when it gets something wrong, is that the model, or what I showed the model?

Two words I'll use throughout. A tool is a function the model can call, like "look up a booking"; whatever the tool returns, the model reads. Notes are facts my agent's code adds to its instructions during a call, such as the booking it has already found.

How I built it:

A copy of my agent's real instructions (about 7,000 tokens) and its 12 tools, set in a fictional motel.
Each mistake became a pair of tasks: the same calls, the same models, one change, before and after the fix.
The tools give fixed, scripted replies, so the only thing that varies is the model's choices.
Control tasks where the caller asks directly ("put me through", "bye", "here's my new email"), to check a fix didn't just make the agent passive.
Scoring in plain Python, no AI judge. Every call runs three times per model, and I read every reply the scorer flagged.

The fixes for the three from real calls are public (here's the email one).

Models Tested

Six models from Kaggle's hosted list: GPT-5.6 Luna, GPT-5.4 mini, Gemini 3.7 Flash, Gemma 4 31B, Claude Haiku 4.5 and Claude Sonnet 5.

Why these: three model makers (OpenAI, Google, Anthropic), including Google's open-weights Gemma, and a mix of small, fast models and larger ones. A phone call can't wait long for an answer, so fast models matter here. These runs are not my agent itself. They answer a different question: does the design change work whichever model reads it?

Gemma ran with its provider's default reasoning, because it rejects "off". The other five ran with reasoning off.

This is not a model ranking. Each model is compared with itself, before and after.

Findings

1. A tool's reply is a prompt

The caller's booking can't be found, for the third time. My agent's lookup tool used to answer: "I couldn't find a booking linked to your number or the details I have here - want me to put you through to reception?"

The models took the hint. In 97 of 108 runs the agent pushed the caller toward reception instead of asking them to spell their name. A typical reply, from Gemma 4 31B:

"I'm still not seeing it. Want me to put you through to reception?"

Then I deleted only the question. That dropped it to 17 of 108. Today's wording (spell the surname, ask for the booking reference, "Do not escalate to a human yet") brought it to 5 of 108.

Model choice mattered far less than that one tool reply. Before the fix, no model got more than 4 of 18 runs right. After it, the worst got 14 of 18.

Prompt-injection benchmarks such as AgentDojo plant hostile text in tool results. Nothing here was hostile: my own tool gave honest, well-meant advice, and it still steered every model.

2. A rule next to a secret isn't a lock

The caller has only said hello. Their number matches a booking, so before the fix the agent's notes held the guest's name, with the rule "Do not say the name, the reference, or any of these details until the caller has told you who they are…"

Five models kept the secret. Claude Haiku 4.5 said it in 14 of 36 runs, spread over 9 of the 12 calls:

"I have a booking here for Linh Tran, checking in 30 October, checking out 1 November—is that you?"

The fix keeps the name out of the notes in code; the lookup returns it only once the caller gives it. After that: 0 of 216 runs said a name or booking reference.

3. Leaving a fact out makes models invent it

This is the mistake from the opening. The caller has said who they are and their booking was found earlier. They ask which email it's under. Before the fix, the notes carried the booking but not its email, on the theory that the agent could look it up again.

Some didn't look. 17 of 216 runs said or used an address that isn't on the booking:

"I have priya.nair@gmail.com on file — is that correct?" (Claude Haiku 4.5)

The booking says priya.nair@example.com. Fifteen of the 17 invented addresses were the guest's own name at gmail.com, the same shape as "Dhruv Patel at gmail dot com". With the email carried in the notes: 0 of 215 (one run was lost to a provider error).

4. Having every input isn't working it out

This one came from probing my agent, not from a real call. A caller asks to move a paid booking whose stay has already ended. Today's date is in the agent's instructions; the stay dates are on the booking.

In two separate benchmark passes, 0 of 108 replies said the stay was over. Most sent the caller to reception because the booking was paid, which is safe but for the wrong reason:

"Your payment's already processed. I'll need to put you through to reception to modify the booking. Shall I connect you?" (GPT-5.6 Luna)

Ten replies in the second pass (fifteen in the first), all Claude Haiku 4.5, offered to move a stay that had already happened.

My proposed fix (my agent doesn't do this yet) has code work out two facts and place them next to the booking: "this stay ended on Sunday 4 October 2026. It is over and can't be moved or changed", plus the payment rule. Offers to move a finished stay went from 10–15 to 0, and 50 of 108 replies now said the stay was over. Gemini 3.7 Flash did it 18 of 18 times; Haiku, never. The unpaid control stayed at 108 of 108, so the facts didn't make agents refuse changes they're allowed to make.

Would a model swap have done the same? For the name leak and the offers to move a finished stay, yes: only Haiku made those. For invented emails, mostly: two models never made one. For the push toward reception, no: every model did it in at least 14 of 18 runs. Changing what the model was shown cut the harmful outcome of all four, on every model that made it.

Across all of this, the fixes didn't make agents passive. On the controls (put me through, goodbye, here's my name, here's my new email): 144 of 144 runs right before the fixes, 143 of 144 after, and every "put me through" call was put through both times.

What I got wrong

Before I saw the first pass's results, I predicted at least one small model would dial a transfer unprompted in one run in five. No model did, in any of the 324 booking-not-found runs. They pushed in words and waited. So I switched to scoring what they said, any push toward reception, and every result in this post uses that rule.

I predicted inventing an email would get worse when the question came five exchanges after the lookup. It didn't: 11 inventions came straight after the lookup, 6 later.

Limits

Six models, three repeats per call, provider-default temperature, so counts shift by a few runs between passes; the direction never did. I changed scorers after reading replies: the transfer score after the first pass, and the move-a-booking scorer twice. The proposed move-a-booking fix ran in one pass only. Kaggle's own score for the "Move a booking" proposed-fix task sits slightly below my final scorer's count, because its built-in version missed six replies that said the stay was over in other words.

What it changed

I used to treat a wrong answer as a model problem and reach for a bigger model. Now I check what the model was shown, and three rules come first:

Withhold what must never be said. Keep it out of the context in code; a written rule is a strong default, not a lock.
Carry what will otherwise be guessed. A missing fact doesn't get fetched. It gets made up.
Compute what it can't work out. Today's date plus a stay's dates isn't enough. "This stay is over" is.

What I'd measure next

Ship the stay-timing fix in my agent and re-run it, then aim at the 58 replies that still answered by the payment rule with "this stay ended" two lines away. I'd also audit all 12 tool replies for sentences that steer, and run these pairs through real speech-recognition errors on longer calls.

My Benchmark

Ovela Phone Agent: Before and After the Fix

Read it in pairs. Each cell is the share of runs where the agent did the right thing; 22.2% means 4 of 18 runs. The number under each model's name averages every task, before and after together, so it isn't a ranking. On "Move a booking" the score also counts the upcoming and unpaid calls, so 67% for today's design (four models; Gemma 65%, Haiku 39%) is the ceiling for an agent that never says a finished stay is over: every finished-stay reply scored a miss.

The pairs:

Booking not found: before, question removed, after
Unknown caller: before, after
Email question: before, after
Controls: before, after
Move a booking: today's design, proposed fix

Four words of Python killed every sentence after first

Dhruv — Fri, 21 Aug 2026 01:54:55 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The bug report was: "it says hello, then it goes deaf."

My AI receptionist answers a real landline. Caller dials, agent greets them properly, caller says "hi, I'd like to check my booking" and then nothing. Forever ♾️. The call stays connected. No error, no exception, no warning. The WebSocket to our speech provider sits there wide open, healthy, receiving audio, and the agent has simply stopped existing.

Turn one worked. Turn two never happened. Every time.

The wrong week

I spent it on the network, because that is where the evidence pointed. A speech socket goes quiet, check the socket. I logged every frame in and out where frames were flowing. I checked for silent disconnects, proxy timeouts, Heroku's 55s idle timeout, mu-law framing, buffer starvation. I added heartbeats.

Then I added reconnect logic, which "fixed" it in the sense that the agent came back four seconds later having forgotten the conversation.

That reconnect was the tell and I ignored it. Something was closing a connection nobody had asked to close. I read it as "the network is flaky" because that was the story I had already decided to believe.

The four words

async def receive_audio_events(self):
    try:
        async for message in self.ws:
            yield parse(message)
    finally:
        self.is_connected = False

Read it the way the author did. This generator drains the speech socket for the lifetime of the call. When it finishes, the call is over, so mark the bridge disconnected. Clean, defensive, obviously right.

Now read how the caller uses it:

async for event in bridge.receive_audio_events():
    if event.type == "EndOfTurn":
        break          # got the turn end, go run the model

break on an async generator runs its finally.

Not when the socket dies. Not when the call ends. Every single turn — the moment the caller stopped talking and we broke out to go think — Python unwound that generator, ran the cleanup, and set is_connected = False on a socket in perfect health.

Turn two arrives. The code checks is_connected, sees False, and politely declines to listen. The socket is fine. The audio is arriving. The flag is a lie, and the flag is what we trusted.

Four words. One finally: clause that was correct for the loop I had in mind, and fatal for the loop that actually existed.

Why nothing caught it

This is the part I keep thinking about.

There was no exception to catch, because nothing failed. No error to log, because nothing errored. No alert to fire, because every health check we had asked a component whether it was up, and every component was up. The socket was open, the process was fine, memory was flat. The agent had quietly agreed with itself to stop.

The tests were green too. They tested one turn.

The fix, and the better fix

The fix is to move the cleanup to where the connection genuinely ends, rather than where a consumer got bored:

async def receive_audio_events(self):
    # No `finally` here. This generator is broken out of every turn, and
    # `break` runs `finally`. Connection lifetime is closed by close().
    # A consumer that stops iterating is not a closed connection.
    async for message in self.ws:
        yield parse(message)

The better fix was structural, and it is the one I would actually recommend to anyone debugging a live provider integration: I stopped using a human as the regression suite.

For weeks the loop had been — someone calls the number, describes what they heard, I dig through logs and guess. That is a debugging cycle measured in hours per iteration with a person as the test harness, and it is why one finally clause survived a week.

What replaced it was unglamorous: a standalone script that opens the provider's WebSocket directly, sends the real prompt and the real tool definitions, and prints every single message that comes back.

async with websockets.connect(
    DEEPGRAM_URL,
    subprotocols=["token", DEEPGRAM_API_KEY],
) as ws:
    await ws.send(json.dumps(settings_msg))   # exactly what the handler sends
    async for message in ws:
        logger.info(message)                  # exactly what it says back

logger.info(message) — no filtering, no parsing, no if/elif. That last line is the entire trick, and it is the line the production code was missing.

Because the moment I stopped interpreting the provider's messages and just printed them, the thing our handler had been silently discarding for a week was sitting there in the terminal:

{"type": "Error", "code": "UNPARSABLE_CLIENT_MESSAGE",
 "description": "unknown field `processors`, expected `thresholds`"}

A hundred and fifty lines, most of it the payload. I should have written it in week one, and the reason I didn't is that building the harness always feels like the thing you do instead of fixing the bug.

What I took away

I stopped trusting flags.

is_connected was a variable claiming to know something about the world, and for a week it lied to me while every socket, every process and every health check was fine. A flag can only ever tell you what some earlier code believed.

Now I measure the thing itself. Did audio come back? How long did it take?

It took four words to break it and a week to find them.

Six silent bugs in a voice AI phone line

Dhruv — Fri, 21 Aug 2026 01:53:57 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

The agent said goodbye and then stayed on the line for 566 seconds. That was the loudest of six bugs that all returned successfully, logged an OK, and kept running, six PRs, and the traces for each.

Project Overview

Ovela answers the real phone line. A guest dials a landline, and five things have to happen before the agent can say a word:

the phone network hands me the caller's audio (Twilio)
something decides when the caller has actually stopped talking, as opposed to pausing (Deepgram)
a model decides what to say, and which of 12 tools to call (OpenAI)
something turns that text into speech (Cartesia)
it goes back down the phone line

All of it inside 800 ms. Past about a second, people say "hello?" and start talking over the agent. There is no spinner on a phone call, and no way to say "still thinking".

Two words you will need. A turn is one round trip - caller speaks, agent answers and it is the unit everything here is measured in. Barge-in is the caller talking over the agent, which has to cut the agent off mid-word, the way a human would stop talking if you interrupted them.

That deadline is why these bugs were expensive, and the deadline is also why none of them looked like bugs.

Bug Fix or Performance Improvement

Start with the one that would not let a caller off the phone.

The agent has a tool for ending a call. It returned this:

{"action": "hangup", "message": "Thanks for calling, goodbye!"}

Nothing read action.

When a model calls a tool, you run the function and hand the return value back to the model as text, and it carries on from there. That is the tool loop, and mine passed this whole dict straight through. The model found message, said goodbye — warmly, in the right tone, at the right moment. Then the line stayed open.

Here is one call, laid out from Twilio's record and Sentry's turn timeline:

01:06:15   call answered
01:07:46   1:31  caller says goodbye, agent says goodbye back
              …  475 seconds of an open line. Three turns fire in that
                 stretch, all on room noise, none of them a conversation.
01:15:41   9:26  the caller gives up and hangs up the phone themselves

Sentry has the hole. Each row is one turn; the last column is the start time in seconds, so the spacing is the conversation:

b82b0208   1.48s    1,786,756,066.3304   ← the goodbye
aa847306   1.48s    1,786,756,379.7518   ← the next thing that fired
                    ─────────────────────
                              313 seconds

The rows above those two are spaced 12, 11, 8, 12, 12, 7, 13, 7 and 3 seconds apart. Then 313.

566 seconds, and the agent ended none of it. transfer_to_staff had the identical defect — it promised a transfer that never dialled.

Read the transcript and the call is perfect. Read the logs and everything succeeded. The only places it looks wrong are the carrier's billing record and a five-minute hole in a column of timestamps.

Reconstructing that call took a Twilio record and thirteen unrelated Sentry traces, because in August each turn was its own trace with nothing tying them together. That is exactly what PR #11 fixes: the same call today arrives as one conversation, in one view.

That is the shape of every serious bug in this system. They return, they log an OK, they keep running. Here are four more of the same family, all from the provider boundary:

What the provider told me	What I did with it	What the caller heard
`UNPARSABLE_CLIENT_MESSAGE: unknown field 'processors', expected 'thresholds'`	dropped	25 seconds of silence
`400 unsupported encoding for raw: mulaw` (correct: `pcm_mulaw`)	dropped	only the cached greeting was ever real speech
`sonic-english` sunsetted → a second `400` behind the first	dropped	same
`TurnInfo` carries its state in `event`; I read `type`	matched nothing	barge-in never fired, and neither did the early nudge to the model that starts it thinking before the caller finishes

Each of those arrived on a WebSocket as an ordinary message and hit an if/elif chain with no else. Parsed, matched nothing, discarded. The socket stayed open. Audio kept flowing in. Nothing came back, and every health check in the system stayed green, because every component genuinely was up.

Two more, once the telemetry was honest enough to find them:

The tenant lookup 404'd on every single call. It queried by document ID; the rows are keyed by slug. It failed, fell through to the correct query, succeeded, and logged success — while spending 4,316 ms on the critical path of the first turn. A slow path that is also a working one is the hardest kind to notice.

One booking was looked up eight times in a single call, because nothing remembered the answer to a question the caller had not changed. One database round trip is about 250 ms; a single lookup_booking makes up to three of them in sequence includes phone, then name, then a hundred-row scan to redo the name match case insensitively - so a miss costs 1.1 to 1.6 seconds.

Code

Six PRs, stacked, each one reviewable on its own:

PR
#7 Make the provider bridges speak the APIs the providers actually run	the four dropped errors, and a `finally` that ran on every `break`
#8 Stop the agent interrupting itself	barge-in armed before any audio existed; chunks from an abandoned turn
#9 Drive the turn from the orchestrator	a design change, not a fix
#10 Execute control-flow tool actions, kill the cold start	the 566-second call; 4,316 ms → 12 ms
#11 Repair the telemetry, group the call as one conversation	a span measuring 0.01 ms; a span hiding 84% of a turn
#12 Stop paying for work already done	prompt cache missed every turn; eight identical lookups

#9 is the odd one out, and its commit message says so in the first line: it is not a bug fix. It moves the live conversation off a multi-agent graph and into a direct streaming call. The graph routes work between a Manager and two workers — worth its latency for background jobs, not for someone waiting mid-sentence. Under the real payload it cost 2,810 ms to first token against 1,040 ms direct.

Everything else in that table is the same bug wearing different clothes: a message arrived, nothing was listening, and the system reported success. A dropped provider error, an action field with no reader, a span opened and closed on the same line, an environment tag never set. Same shape, six times.

My Improvements

Hanging up without cutting off the goodbye.

The obvious fix is to terminate the call the moment the tool fires — and then the caller never hears the farewell, because the line drops mid-word. So the intent is recorded, the audio finishes draining, and only then does the call end:

if result.get("action") == "hangup":
    self._pending_hangup = True

finally:
    interrupted = self.state != ConversationState.AGENT_SPEAKING
    if self._pending_hangup:
        self._pending_hangup = False
        if interrupted:
            logger.info("🛑 Hangup aborted — user spoke during farewell")
        else:
            await self._hangup_call()

If the caller starts speaking during the goodbye, they have changed their mind, and hanging up on them would be worse than the original bug. The termination is also idempotent — because a tool with no observable effect gets retried by the model, which is how the bug announced itself in the first place.

Caching the task, not the result.

The repeated booking lookup got a per-call memo. The detail that matters is one line:

task = self._cache.get(key)
if task is None:
    # Cache the task, not the result: the prefetch and the first tool call
    # race for the same row, and the loser must join the in-flight request
    # instead of issuing a second one.
    task = asyncio.ensure_future(self._db.lookup_motel_reservation(**kwargs))
    self._cache[key] = task

If you cache the finished value, both racers find an empty cache and both hit the database. Caching the unfinished task makes the second one wait on the first. Invalidation uses a read-only allowlist, so a tool added next month that writes a booking will be slow before it is wrong.

Two lines at the front of a prompt.

The other half of that PR is smaller and I nearly left it out. A volatile header — the current date and time — sat at the very start of a ~9,000-token system prompt. The prefix therefore changed on every single turn, so none of it could ever be reused. Moving it to the end was a two-line diff:

- return f"""{context_header}You're the AI receptionist for Coal Creek Motel...
+ return f"""You're the AI receptionist for Coal Creek Motel...
+ {context_header}"""

Cached prompt tokens are billed at a discount and, more to the point here, skip re-processing — so this is both cheaper and quicker to first token. Sentry reads the result straight off the model call: 178 new tokens, 8.9K served from cache, 32 out — 98% of the prompt reused on a turn budgeted at 800 ms end to end.

What the memo did, stated precisely. Two live calls, one either side of the fix, paired trace by trace:

	before (8 calls)	after (7 calls)
Queries that actually ran	8 of 8	4 of 7
Resolved from an in-flight task	0	3, at ~0.1 ms
Median of the queries that still ran	1,263 ms	1,136 ms
Total query wait per call	11,049 ms	4,496 ms

Look at the third row, because it says what this fix is not. A memo cannot make a database query faster, and it didn't. The 59% saving is entirely the three queries that stopped happening.

Elsewhere: cold start 4,316 ms → 12 ms — that pair came from Heroku startup logs at the time, which have since rotated, so it is the one number here with no artifact behind it. Conversational turns land at 400–890 ms, median 790 ms across 28 consecutive turns. There is no before-figure for that median, and I am not going to manufacture one — before these fixes the pipeline never reached a second turn, so there was nothing to time.

Best Use of Sentry

Sentry found the latency bugs, and one I shipped myself halfway through. It is not the place I reported them from afterwards.

The span that was lying to me. A span is a stopwatch around one stage of a request; nest them and a turn arrives as a waterfall instead of a single number. Mine covered "user speech ended → tool executed", which turned out to be far too much ground for one stopwatch. Across the 25 pre-fix turns still inside Sentry's retention window it held a median 84% of turn time. Here is one of them: 5.29 seconds of a 5.52 second turn in a single bar, with the only other span reading 0.01 ms, because it was opened and closed on the same line.

I read that 84% and concluded the model was slow, because the model was what I already suspected. Splitting it into llm.stream per model round and gen_ai.execute_tool per tool inverted the answer immediately:

A 2.02 s turn: 538 ms model round, 506 ms tool call, 407 ms second model round, 268 ms of speech synthesis. (Different turn, different length — compare the shape, not the totals.) I had spent a day blaming the language model. It was a database round trip.

One phone call, one conversation. The pipeline emits Sentry's gen_ai.* conventions with the Twilio CallSid as the conversation id:

from sentry_sdk.ai import set_conversation_id

if self.call_sid:
    set_conversation_id(self.call_sid)

One line, and the dashboard stopped being a latency chart. This is a real call - the caller's words, the tool the agent reached for, the arguments it passed, and the cost of each:

lookup_booking  guest_name: "Drew Patel"    1.72 s     ← the real query
lookup_booking  guest_name: "Drew Patel"    0.08 ms    ← memo
lookup_booking  guest_name: "Drew Patel"    0.09 ms    ← memo

That screenshot also contains a bug I have not fixed. Look at the turn where the caller spells their name — B h r u v p a t e l — and the very next lookup still goes out as "Drew Patel". The memo did its job: same question in, cached answer back. The model never updated the argument after being corrected. A cache is only ever as right as the question it is asked, and that one is next on my list as UX work.

Sentry also caught me. While adding those tool spans I shipped this:

transaction.start_child(op="gen_ai.execute_tool", attributes={...})
# TypeError: Span.__init__() got an unexpected keyword argument 'attributes'

start_child() takes no attributes. My unit test passed because it used a MagicMock, and a MagicMock accepts every keyword argument you invent. Someone called the number ten minutes later and every tool call on that call failed.

I found it in the issue before the call had ended, and rewrote the test against a real Transaction subclass — then proved the new test fails against the broken version. A mock at a provider boundary tests your assumption. The provider's contract is somewhere else, and it does not care what your mock agreed to.

Also in Sentry, also mine: sentry_sdk.init() never passed an environment, so every event arrived tagged production — including the ones my own test suite raised on every run. PR #7 sets it and blanks the DSN in conftest.py. That is why every screenshot here reads environment: demo: it is the live service, labelled correctly for the first time.

I could not run Seer Autofix. I asked Sentry support, including about the bugsmash26 credits but needs a paid plan and the credits do not unlock it on a free-tier org. So there is no RCA screenshot here. The conversation view above is what I used instead, and it is the artifact that showed me the tool firing three times.

Best Use of Google AI

Gemini 2.5 Flash reads the traces after every call, without being asked. When Twilio posts a completed callback, a watchdog waits two minutes for spans to land, pulls them from Sentry's API, aggregates them per span name, and hands the numbers to Gemini with the pipeline architecture as context. It replies in strict JSON.

Here is a real run: the whole file is in the repo, including the two root causes I trimmed for space:

{
  "bottleneck_span": "Span 1: User Speech Ended -> First Token Yielded",
  "bottleneck_share_pct": 88,
  "root_causes": [
    "Excessive latency in specific tool executions: The `perform_live_search`
     tool has an extremely high median latency of 2126.73 ms, and other tools
     like `check_availability` and `lookup_booking` also add ~500 ms when
     invoked."
  ],
  "verdict": "The pipeline currently does not meet the sub-800ms TTFA target,
              with a median `user_voice_turn_transaction` of 952.18 ms."
}

Here is the honest shape of this system, stated once: a turn with no tool call lands at ~790 ms median across 28 turns. A turn that hits the database lands around 950 ms. The 800 ms budget is met on conversation and missed on lookups — which is exactly what the analyser flags, unprompted, after every call, because that window mixes both kinds of turn together.

It also flagged perform_live_search at 2,126 ms — on two samples, so a median of two points is just the slower one. The analyser did not mention that, and I will.

What Gemini does not do here is fix files, and it has not yet found a root cause I did not already have. It reads production telemetry and says where to look, unprompted, after every call. That step used to cost me a day per bug. A span that regresses more than 40% against the previous run raises a Sentry message; nobody has to open a chart.

Google ADK still runs the cold path, and that is exactly why PR #9 exists. The graph was driving the live conversation, and it was never built for that: it routes work between a Manager, a BookingWorker and an InfoWorker, which is worth its latency in the background and not worth it to someone waiting mid-sentence. Measured under the payload actually shipped a ~9,500-token prompt with 12 tool definitions — the direct streaming call reached first token in 1,040 ms against 2,810 ms. On a bare "hello" the ranking reverses, which is precisely why the benchmark that counts is the real payload, and why a day of my optimisation went the wrong way before I understood that.

So Gemini does the analysis and the background reasoning, and does not drive the conversation, because the measurement said not to.

Every number above was read out of Sentry after a call placed over the phone network📞.

here's what the latency sounds like: click to hear

*Small update: several numbers here have since been re-measured on the server that actually answers the calls
*

From Voice Demo to Operational Voice Assistant: Reviving Ovela AI

Dhruv — Wed, 03 Jun 2026 15:17:28 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

Ovela AI started as a side project driven by a question that kept pulling me back:

What would it take for a business to genuinely trust a voice AI system?

At first, I thought the answer was simple: make conversations sound natural.

The original prototype could answer calls, respond to questions, and carry a conversation reasonably well. From a technical perspective, it looked impressive.

But after speaking with accommodation providers and small business owners, I realized I was focused on the wrong problem.

Businesses don't trust a system because it sounds human.

They trust it because it behaves reliably.

Can it check availability correctly?

Can it update reservations safely?

Can it collect payments?

Can it transfer a call when confidence is low?

Can staff see exactly what happened afterward?

That realization changed the direction of the project completely.

Ovela AI evolved from a voice demo into an operational voice assistant designed to help businesses handle real customer interactions while keeping humans in control of important decisions.

Today, Ovela can:

Handle inbound phone calls
Check room availability
Create reservations
Process payments through Stripe
Answer property and local information questions
Transfer calls when needed
Keep staff synchronized through a management dashboard

More importantly, every improvement is guided by a simple principle:

AI should support human operations, not blindly replace them.

Demo

📞 Live Demo (Australia)

Phone: +61 3 4823 6219

Due to abuse protection and testing limits, availability may occasionally be restricted.

Try asking:

"Do you have any rooms available this weekend?"
"Can I make a reservation?"
"What attractions are nearby?"
"What's the weather like today?"

🌐 Website: https://ovela.dev

🐙 GitHub Repository:https://github.com/My-CMDhub/Ovela-AI

The Comeback Story

Like many side projects, Ovela reached a point where the prototype worked well enough to demonstrate the idea.

Then it sat untouched not because the project failed but because other priorities took over.

Months later, after more conversations with business owners and more exposure to real operational challenges, I came back to the project with a very different perspective.

The biggest lesson was surprisingly non-technical.

The challenge isn't making AI speak.

The challenge is making AI behave appropriately within human workflows.

A real receptionist doesn't simply answer questions.

They:

Recognize interruptions
Acknowledge requests before acting
Handle uncertainty
Understand when information is missing
Escalate sensitive situations
Maintain context across an entire conversation

Most voice demos don't fail because speech recognition is poor.

They fail because the operational behavior doesn't match what people expect from a trusted assistant.

That became the focus of the revival.

What Changed

Multi-Agent Architecture

The original system relied on a much simpler flow.

The new version uses a multi-agent architecture built around Google's Agent Development Kit (ADK), allowing different agents to handle reservations, business operations, and information requests independently.

Lower Latency Conversations

Voice interactions are highly sensitive to delays.

Several architectural bottlenecks were removed to improve response times and reduce awkward pauses during calls.

Stronger Context Awareness

One of the most interesting challenges was interruption handling.

People interrupt constantly during real conversations.

The system now maintains awareness of what information has already been spoken, allowing it to continue naturally instead of restarting or losing context.

Operational Reliability

Reservation workflows, payment handling, availability checks, and dashboard synchronization were rebuilt to behave more like real business processes rather than isolated AI actions.

Abuse Protection

Real phone systems attract misuse.

Rate limits, call protections, and operational safeguards were added to prevent abuse while keeping legitimate usage frictionless.

My Experience with GitHub Copilot

Returning to a codebase that has sat inactive for months is often harder than starting a brand new one. You inherit your own past decisions without fully remembering why you made them.

For the revival of Ovela AI, I didn't use GitHub Copilot as a simple autocomplete tool to write boilerplate code. Instead, I used it as a high-level engineering partner and data auditor to manage complex architectural shifts and harden my system’s reliability.

Here are the two major ways Copilot helped me cross the finish line, backed by real-world interaction during my workflow development:

1. Translating Complex Systems into Architecture Diagrams

As Ovela AI transitioned to a multi-agent setup, mapping out component connections, telephony triggers, and dashboard synchronization endpoints became a major cognitive bottleneck. I leveraged Copilot within my workspace as a principal solutions architect. By feeding it my core file dependencies, it mapped out a clean, production-ready system workflow directly in Mermaid.js syntax for the repository documentation.

2. Eval Hardening (The Supreme Judge)

Building a reliable voice AI requires robust testing. I simulate conversations between two LLMs and dump the evaluation telemetry into local .json log files. However, default automated grading scripts are notoriously prone to false positives (e.g., grading a hallucinated response highly simply because it sounded polite).

I utilized Copilot as a Supreme AI Evaluation Auditor. I passed it raw JSON conversation objects, prompting it to critically audit the automated scores, spot misleading feedback, and generate an adjusted "Supreme Score" with a bulleted logical justification. This drastically reduced noise in my evaluation pipeline.

Operational Code Polish & Balancing

Beyond these two core pillars, Copilot served as an excellent "cleanup crew" throughout this journey even after hitting rate limits ✋. It assisted in tracking down legacy typing issues, reviewing asynchronous edge cases, and generating clean inline documentation.

Ultimately, the biggest value Copilot provided wasn't just writing lines of code faster but it was accelerating complex architectural decisions and data validation when reviving a stale codebase.

What I Learned

The most valuable lesson wasn't technical.

It was understanding the difference between a convincing demo and a useful product.

A demo succeeds when the AI says the right thing.

A business system succeeds when the right thing actually happens afterward.

That distinction changed how I think about voice AI.

Natural conversation matters.

Latency matters.

Speech quality matters.

But trust matters more.

Trust comes from reliability, transparency, and knowing when humans should remain part of the process.

I don't believe current voice AI systems perfectly replicate human interaction, and that's not really the goal.

What interests me is the space between humans and AI:

How can AI handle repetitive operational work while humans remain responsible for judgment, relationships, and important decisions?

Reviving Ovela helped me explore that question far more deeply than when I first started the project.

And honestly, that's what made finishing it worthwhile.