I built CallForMe, a remote MCP server that lets an agent (Claude, ChatGPT, Codex, Cursor) phone a business: it works the phone menu, waits on hold, asks the agent mid-call when the business needs something, and returns answers plus a transcript.
Agents started using it for real errands in October. Here are nine things that broke, none of which showed up in testing.
1. ChatGPT hides your link until its turn ends
The first real call needs a one-time link where the user adds credit. Our tool told every agent: show the link, then call wait_for_setup, which long-polls until payment.
That works in Claude, which streams text before a tool call. ChatGPT doesn't show the user anything until its turn is over. So it called wait_for_setup six seconds after getting the link, waited up to 50 seconds, and the user never saw a link to click. Our numbers: almost nobody from ChatGPT ever opened one.
Fix: the tool result now depends on the client. ChatGPT is told to reply with the link and end its turn; it waits only after the user says they've paid. And because a user who pays may never say "done", the server now places the waiting call by itself 20 seconds after payment if no agent has.
Lesson: an MCP tool's instructions are part of your UX, and clients differ in when the user sees anything.
2. One-time codes are the account holder's job
Airlines, banks and carriers increasingly text a code to the account holder mid-call and ask for it back. Our assistant will never read a code to a business, even if the user hands it over, because reading someone's code for them is exactly how account-takeover scams work.
The bug was the guidance around it. When an agent tried to pass a code, we answered with the advice for card numbers ("the customer will handle payment"). That told the agent nothing useful about what to do next.
Fix: a voice rule (never say a code; if transfer_to is set, connect the account holder; otherwise get a reference number for a callback), a different next_step for codes, and plan_call now flags loyalty and award bookings as account-holder calls up front, so the transfer number is there from the first call.
3. "Is it open?" needs more than the phone number
A user called a car dealer's sales line three times after hours. Each call sat through a 45-second recorded privacy notice and a menu before reaching voicemail.
We already warned "looks closed right now" in our preview tool, by looking the number up on Google Places. But REST users skip the preview, and the sales line wasn't on any listing: only the dealer's main number was.
Fix: place_call now does the lookup while the call dials (cached, never waits more than 1.5 seconds), and falls back to the business name when the number isn't listed, but only trusts a match in the same area code.
4. Race conditions in "what happened"
When a business answers in a language the call wasn't set up for, we call back once in that language for free and replace the first call's summary. An agent long-polling get_call woke up on the first write and read "we could not confirm the closing time", a moment before the summary changed to "calling back in Vietnamese".
Fix: decide about the callback before saving the outcome, and write the final summary the first time.
5. Your opener is a product decision
Every call opens "an assistant calling for (name), on a recorded line", and the assistant says it's an AI whenever asked. When a request had no name, mid-call lines came out as "let me check with a customer". Small thing; it sounds like a robocall. Now it's "my customer", and the tool warns the agent to pass the user's name.
The first sentence matters more than anything after it. When calls opened "this is an AI assistant", nearly half of the businesses that picked up hung up within 40 seconds. Opening with who the call is for, and saying it's an AI the moment anyone asks, brought that to about one in ten. Same honesty, different order.
6. Agents guess your tool names
Agents call tools that don't exist: make_call, check_call, end_call, /v1/call instead of /v1/calls. The MCP SDK's answer is "Tool make_call not found", and a missing argument comes back as "expected string, received undefined at phone". Both leave the agent guessing again.
Fix: answer with the closest real name ("Did you mean place_call? Call it with the same arguments.") and the full tool list; rewrite argument errors into the field, its type and what to do ("No number yet? Call find_business first."); on REST, a did_you_mean field on every 404. Cheap to build, and it turns a dead end into a retry.
7. The transcript said "214-555-0142". The audio didn't.
Our voice model wrote phone numbers as numerals, and the text-to-speech engine ran them together. The call transcript looked perfect. When we transcribed the recording itself with Whisper, "214-555-0142" came back as "2455-042" on one take and "214-555-042" on another. A business writing down a callback number would have gotten it wrong, and nothing in our logs would have shown it.
Fix: the prompt now says to write any number it speaks as words, one digit at a time, with commas between groups ("two one four, five five five, zero one four two"), while tool arguments keep digits so the saved answer stays clean. Voicemail text we generate server-side gets the same treatment. Two takes later, every digit was there.
8. Your prompt's examples leak into other languages
Our system prompt is in English and full of quoted example lines ("Let me check on that, one moment."). On a test call to a Spanish-speaking bakery, the assistant switched to Spanish as designed, then said exactly that English line while it checked with its user. The host answered "Perdón, no hablo inglés."
Fix: one sentence telling the model that every quoted line is an English example, and in another language it should say the meaning in that language. Both re-runs stayed in Spanish.
9. When a prompt rule keeps losing, move it
Three bugs from one evening of real calls. The assistant said "I'll wait for the menu to finish" out loud into phone menus, and a bank's speech menu took it as an answer. It pressed 0 on menus that never offered 0, and they hung up. And on a "don't book, just ask" call, it stopped to ask its user about the restaurant's booking offer, so the call sat waiting.
We fixed all three in the prompt first, and each failed again on repeat test calls. Another prompt rule was winning every time. What worked was moving the instruction somewhere else. "Silence is how you wait for a menu" went into the description of the stay-silent tool: 0 narrations in 7 calls since. "Only press keys the menu offered" went into the menu rule's fallback, and the re-prompt test went from 1 of 5 to 4 of 5. The booking case moved into the server: when the goal says not to book and the model asks its user about booking, the server answers for them. That went from 1 of 4 to 4 of 4.
Fix, in general: run each voice check several times before and after a change (one pass proves nothing), and when a rule keeps losing, put it in a tool description or in code.
What we'd tell anyone building on voice
- Test with the agents people really use, not just your own harness. Each client has different turn and display behavior.
- Treat every string your tools return as copy the end user will see.
- Log the funnel per client: links created, opened, paid. The ChatGPT problem was invisible until we split it out.
- Listen to your recordings, or transcribe them yourself. The text log shows what the model wrote, not what the caller heard.
You can try the free demo line from any MCP client: add https://callforme.tel/mcp and ask your agent to call Maple Street Pizza at (469) 770-7412, a fictional restaurant we run for demos (no account, up to 6 free calls a day). Real calls are $1 per answered call including the first 2 minutes, then $0.40 a minute, and nothing if nobody picks up.
I build and run CallForMe. This post was drafted with help from Claude.
Top comments (0)