DEV Community: horio

Five ways a voice agent tells you it booked a table when it didn't

horio — Tue, 22 Sep 2026 03:24:48 +0000

I maintain Oathra, an open-source runtime for AI agents that make
phone calls. Its one opinion is that the model does not get to decide whether the call succeeded. A
separate piece of code reads the other party's words and decides.

That sounds like belt and braces until you look at what clerks actually say. Here are five replies to
the same request. Only one of them is a booking. A model asked "did this succeed?" says yes to at least
four, because all five sound like a yes.

The request, every time:

AI: Could I book a table for two at 7:30 pm on September 25? The name is Tanaka.

1. The pencil

Them: Sure, I will pencil you in for September 25 at 7:30 pm and call you back to confirm.

Verdict: incomplete, { date, time, partySize } extracted, confirmed missing.

"Sure" is agreement. The date, the time and the party size are all there, and they are all correct.
The only thing missing is the thing you called about. A completion check that counts filled fields
passes this. A check that requires the callee to have committed does not.

2. The explicit hold

Them: We can hold September 25 at 7:30 pm for now, but it is not confirmed yet.

Verdict: incomplete, nothing extracted at all.

This one is interesting because the clerk is being maximally clear, and a naive extractor still walks
away with time = 19:30. The clause carrying the time is the one being negated. If you extract values
per-utterance instead of per-clause, "not confirmed yet" and "7:30 pm" end up in different variables and
the negation is lost on the way.

3. The counter-offer

Them: The only thing left that evening is 9 pm. Would that do?

Verdict: incomplete, nothing extracted.

A number was said. It was not offered as your booking; it was offered as a question. The trap here is
that the agent's next turn is usually "9 pm works, thank you" — and if your extractor took 21:00
from the clerk's turn and your agent then says something agreeable, you have a fully populated booking
that nobody ever agreed to. The clerk's turn has to stay a proposal until the caller accepts it and the
clerk acknowledges the acceptance.

4. The retraction

Them: Yes, we have you down for two at 7:30 pm on September 25.
Them: Sorry, that day is fully booked after all.

Verdict: incomplete, { date, time, partySize } extracted, confirmed gone.

This is the one I got wrong. Until yesterday, Oathra reported this as completed.

My retraction pattern required the refusal to name the booking — "we cannot take the reservation",
"the booking is cancelled". A clerk who simply says the slot is gone does not phrase it that way. They
say the table is taken, the day is private-hire, they are closed that day. The booking is equally dead
and my code called it a success.

The fix is narrow on purpose: an availability word from the callee (full, private hire, closed,
"fully booked") revokes a confirmation spoken strictly earlier. Not any refusal — "we can't take cards"
after a booking is a payment remark, not a cancellation. And only earlier, so the ordinary "7 pm is
full but 7:30 is free" that happens before a booking is untouched.

It went out as PR #37 with tests in both directions.

5. The actual yes

Them: You are all set for September 25 at 7:30 pm, party of two.

Verdict: completed, { date: 2026-09-25, time: 19:30, partySize: 2, confirmed: true }.

This one also failed until today, in a way I find more embarrassing than #4: it returned nothing at
all. Not "unconfirmed" — empty.

The engine had two separate ideas, "the callee agreed to a value the caller proposed" and "the callee
confirmed the booking", and "you are all set" was in the second list but not the first. So nothing the
caller had proposed was ever verified; and because a confirmation is bound to the values that were
settled when it was spoken, a confirmation with no settled values behind it is stale and gets dropped.
Two lists that should have overlapped, and the result is a blank screen for a perfectly normal sentence.

I only found it because I pasted an English log into my own public checker while writing a comment on
someone else's thread. Three of five natural English confirmations worked. The Japanese side, which I
use daily, was fine. The lesson is not about regexes; it is that the language you don't test in is the
language that's broken.

How this is kept honest

Every release runs 10,000 seeded adversarial dialogues — the five shapes above plus voicemail,
transfers, hold-then-reply, dialect confirmations, wrong restatements — against a hard gate:

False Completion: 0 / 10000 adversarial runs

A false completion is when the runtime reports "completed" and the callee's own ground truth says they
never committed. Zero is the only passing number. It does not prove the checker is right about
everything; it proves the specific ways I know a call can lie are all covered, and it fails loudly the
moment a change reopens one.

The inverse error — reporting "incomplete" when the table really was booked — is allowed to happen. It
is a phone call you make again. The other direction is a customer standing outside a restaurant.

Watch the verdict change

Twenty-six seconds of the checker being used, no sound. Each time the shop's reply is swapped, the same
code runs again and the fields re-settle. The camera moves, the cursor and the captions were added in the
edit; the screen and the verdicts were not.

Try it on your own log

If you run a voice agent, the fastest version of this is to paste one of your own transcripts into the
checker. No install, no key, nothing leaves the tab:

https://forifor.github.io/oathra/en/check.html

One turn per line, with a speaker:

AI: Could I book a table for two at 7:30 pm on September 25? The name is Tanaka.
Them: Sure, I will pencil you in and call you back to confirm.

If it reads your clerk's yes as a no, that is a bug I want — the supported phrasings are a list, and
lists are always short somewhere. The repo is Apache-2.0:
FORIFOR/oathra.

Four regexes and a staleness rule: how my phone agent refuses to call 'probably fine' a booking

horio — Sun, 13 Sep 2026 15:38:00 +0000

In the launch post I said Oathra decides whether a call succeeded from what the callee said, not from the model. This is the part that does it. It lives in packages/evidence and uses no LLM.

The three lines that fooled the model

"Probably fine, but it's not confirmed yet" — a hedge
"7 pm is full. 7:30 is open" — a refusal and an offer in one breath
"Got it. But the price will be 23,500 yen" — acceptance, then a change of terms

Asked "was the reservation made?", an LLM would say yes to 1 and 3 often enough to matter. The conversation flows like a yes. So completion moved out of the model into four rules.

Rule 1: split into clauses before reading values

"7 pm is full but 7:30 works" as one sentence yields two candidate times. The clause is split right after a contrast word (Japanese ですが/ますが/けど, English but), and a clause carrying a negative (full, unavailable, can't…) never produces an offer.

const sub = c.text.split(/(?<=ですが|ますが|けど|けれど|but\s)/);

7:00 drops out. 7:30 survives as a callee offer, which stays pending until the agent accepts it.

Rule 2: one hedge word disqualifies the whole utterance

export const HEDGE_RE =
  /と思います|たぶん|多分|おそらく|かもしれません|確認します|調べてみ|probably|maybe|perhaps|I think|let me check|not sure|I'll check|might be/i;

Agreement is AGREEMENT_RE && !REFUSAL_RE && !HEDGE_RE, so "probably fine" is not an agreement even though "fine" matches. The same guard sits in front of the confirmation check: "I think… you're booked" never sets confirmed.

The calling side follows the same rule. The scripted agent used to restate its request after a hedge until it gave up. Since v0.1.1 it asks, at most twice, "can you confirm the reservation for the 12th, 7 pm, two people, or is it still tentative?"

Rule 3: `confirmed` has exactly two entry points

The callee says an explicit confirmation ("your table is booked", 「ご予約承りました」) — CONFIRMATION_RE.
The agent asks a yes/no confirmation question ("can you confirm the booking?") and the callee's reply starts with an affirmative — CONFIRM_REQUEST_RE then AFFIRMATIVE_RE.

A reply containing a question mark is not an answer. A re-quote ("a non-smoking room would be 19,900 a night") is not an answer. And nothing the agent says counts: only nodes whose speaker is callee are read.

Rule 4: a confirmation is bound to the terms at the moment it was spoken

When "you're booked" is followed by "the rate is 23,500", the earlier confirmation goes stale. The engine snapshots the settled values at the time of the confirmation plus whatever the callee restated in that utterance, and drops confirmed if any of them differs from the final state.

const stale = Object.entries(snapshot).some(([field, v]) => v !== undefined && out[field] !== v);
if (!stale) out.confirmed = true;

The agent then asks for the confirmation again.

What it does with the three lines

Fed straight into the engine (log):

Callee line	Result
"Probably fine, but it's not confirmed yet"	incomplete, no `confirmed`
"7 pm is full. 7:30 is open"	incomplete, 7:00 never taken, 7:30 pending
…then "7:30 it is" / "booked for two at 7:30"	complete, time=19:30
"Got it. But the price will be 23,500"	incomplete
"Booked at 18,000" then "sorry, 23,500"	incomplete, confirmation stale

You can try them in the browser: npx oathra demo, pick Play (you answer the phone), and the lines are one-click buttons under the input.

The honest limit

This depends on phrase coverage. CI requires 0 false completions over 10,000 mutated callees, but that is "zero within the mutations I wrote", not a real-call number. If you find a phrasing that slips through, open an issue. It is usually one more alternation in a regex, and it is the most useful contribution.

I built an AI team that ships real work — and shows you the conversation

horio — Sat, 12 Sep 2026 15:44:18 +0000

Most "multi-agent" demos are bots narrating to each other. The log is the product; the work is secondary. I wanted the opposite: one request in, real files out, and a trail I could audit afterwards.

Agent Team is an open-source (MIT), local-first runtime where a Master plans, a Researcher / Builder / Reviewer actually do the work, and you get the artifacts plus the real bot-to-bot messages, a timeline, and verification bound to each artifact revision.

Repo: https://github.com/FORIFOR/Multibot · Site + 59s intro: https://forifor.github.io/Multibot/

What makes it different

No scripted chat. The chat panel is a projection of message.sent events — messages that were really delivered to another bot's mailbox. A question from the Builder wakes the Researcher, which answers with reply_to. Acknowledgements never wake a model.
Evidence, not vibes. Checks and review verdicts are events bound to an artifact revision hash. The final report is compiled from the event log; a model summary cannot upgrade "started" to "done".
Runtime-enforced limits. Tool scope, write scope, budget reservation (spent + reserved for in-flight calls), approvals with hash + nonce, cancel / resume / fork — enforced in code, not prompt wording. Unknown model prices refuse to start.
Per-bot configuration. Each bot inherits a default connection and model and can override endpoint, model, effort and system prompt (lockable). Configured vs. provider-reported model are both shown. No silent fallbacks.
Replay never calls a model. Fork from a checkpoint with a different model for one bot and compare. Export the whole run as JSONL.

No API key needed (Claude Code)

The default connection runs every agent session through the local claude -p. Claude Code owns the loop for one session; the team's tools (send_message, publish_artifact, run_check, …) are exposed to it as MCP tools through a small stdio proxy that forwards each call to the runtime's ToolGateway. Policy, budget and the event log are identical to the API path; Claude Code's own built-in tools are disabled for these sessions. Cost and usage are read from the CLI's JSON result, and the model it actually used comes from modelUsage, never from the model's own claims.

Claude API (official SDK), any OpenAI-compatible chat endpoint and local Ollama work the same way, per bot.

A real run, unedited

On 2026-09-13 I ran this request through the local Claude Code CLI: "From this product description, build a Japanese launch page and three social-post drafts. Record assumptions for anything missing. Stop before publishing. Have the Reviewer verify."


Model (reported by the provider)	`claude-opus-5`
Plan	Master chose 1 builder task + 1 reviewer task and skipped the researcher; 8 recorded assumptions (no prices, no invented numbers, placeholder URLs only)
Deliverables	`index.html` (single-file page), `posts.md`, `HANDOFF.md`, `final-report.md`
Verification	10 programmatic checks → all pass; reviewer verdict 6/6, plus 4 optional findings sent back as a real message
Usage	39 model turns · 35 tool calls · $1.66 list-price equivalent · 18 min 37 s
Status	completed

The generated files, the final report and the 77-event JSONL log are committed under docs/evidence/ without edits. Four runs total: run 1 finished partial and exposed a bug (a reviewer verifying two tasks had only its last verdict applied — fixed), runs 2–3 hit limits that were then tuned. One request is not a benchmark, and the prompts are original seeds.

The demo video on the site drives the real UI and runtime with a scripted test provider (labelled on screen) so it is deterministic and free to reproduce.

How it works

One request → a checked plan. The Master turns the request into deliverables, assumptions and a task DAG (structured output). The runtime validates schema, cycles, owners, tools, write scopes and limits before anything runs.
Independent bots, real delivery. Each bot has its own conversation state, mailbox, task scope, tools and workspace. A worker receives its task, input artifact refs and its own inbox — not the whole history.
Verify, revise, report from evidence. The Reviewer runs checks against a specific revision. Fail → the Builder revises → re-review, bounded. The final report is compiled from the event log.

Stack: Python 3.12 / FastAPI / SQLite (WAL, append-only events, SSE), React + TypeScript. macOS seatbelt sandbox for builder commands (no network, writes only inside the task workspace); elsewhere a plain subprocess that says so in every result.

Try it

git clone https://github.com/FORIFOR/Multibot && cd Multibot
cd backend && uv venv .venv --python 3.12 && uv pip install --python .venv/bin/python -e '.[dev]'
cd ../frontend && pnpm install && pnpm build && cd ../backend
.venv/bin/agentteam probe    # real capability check through `claude -p`
.venv/bin/agentteam serve    # http://127.0.0.1:8787

I'd love feedback on two things: whether the review → revise → re-review loop bound to revisions holds up on your tasks, and where "fewer messages, more evidence" (workers get their task, artifact refs and inbox — not the history) breaks down.

I let an AI make phone calls, then took the word "booked" away from it

horio — Sat, 12 Sep 2026 15:39:30 +0000

Getting an AI to place a phone call takes an evening. The trouble starts after that.

It said "your table is booked" when the restaurant had said no such thing.
It talked to a voicemail greeting for two minutes, politely asking for a reservation.
It accepted a price above the budget because the conversation had a nice flow.

All three happened on real calls to my own phone. So I built a runtime that takes the completion decision away from the model and gives it to code.

https://github.com/FORIFOR/oathra

What it does

npx oathra demo

Two AIs start a phone call in your browser, no API key. One plays a restaurant, the other wants a table. 7 pm is full, the restaurant offers 7:30, the caller takes it, gives a name, and only when the restaurant says "you're all set" does the run become MISSION COMPLETE.

The result is not a summary. Every field is anchored to something the other party said.

{
  "field": "time",
  "value": "19:30",
  "source": "callee",
  "transcript": "7 is fully booked, but we do have 7:30.",
  "span": "7:30",
  "explicit": true,
  "verified": true
}

Completion is a formula, not an opinion:

complete = connected ∧ date.verified ∧ time.verified ∧ partySize.verified ∧ confirmed.verified ∧ constraints hold

If the caller says "great, that's confirmed", it is recorded and ignored. Only the callee's "you're booked" fills confirmed.

The evidence rules

This part is deterministic parsers and rules. No LLM.

"7 is full, but 7:30 works" → no evidence for 7:00 (negated clause)
"the 13th, sorry, the 14th" → the 14th survives, with a supersede trail
"probably fine" → neither agreement nor confirmation
price changes after "you're booked" → the confirmation goes stale and must be re-obtained
"OK, I'll look elsewhere" → not an acceptance

There is a command that generates ten thousand hostile clerks to break this:

oathra eval --adversarial 10000
# never-confirm / wrong-restate / negate-then-offer / silent-hangup / caller-echo-trap
# False Completion: 0 / 10000 adversarial runs

The fuzzer caught real bugs. The first one: "¥21,100" was split at the comma, became "¥100", and the agent happily accepted. More recently I let GPT-4o mini and Gemini play the clerk (oathra eval --callee openai), which found refusals that quoted a number being read as offers, and product names being parsed as serial numbers.

Real phone calls

The same runtime drives real calls.

oathra setup phone      # pick a voice engine and a carrier
oathra phone doctor     # which layer is broken
oathra call --to +81…   # dial

Carriers (Twilio direct, or Plivo / your own SIP through a LiveKit gateway) and voice engines (GPT-Live, OpenAI Realtime, Deepgram + LLM + TTS) are chosen independently.

Calls to my own phone:

#	Stack	What happened
1	Deepgram + GPT-4o-mini + TTS	11.8 s to first reply. It kept asking "can you hear me?"
2	Same, streaming TTS	2.2 s. Talked to voicemail for 2 minutes; noise interrupted it 13 times
3	GPT-Live (full duplex)	~0.4 s. A 6 min 25 s chat with no errors

Call three was when it felt usable. GPT-Live is $0.05/min for the session; Twilio to a Japanese mobile is ¥28.78/min.

Where it honestly stands

Twilio direct and GPT-Live are verified on real calls
Plivo and custom SIP are implemented from provider docs; PSTN not yet verified
MCP server is v0.2

Every number above was measured by me and can be reproduced with the commands in the README. Scenarios are YAML and welcome as PRs.

https://forifor.github.io/oathra/en/