Every other part of my stack comes with a safety net I never asked for. TypeScript refuses to compile. A test goes red. A pull request makes somebody ask why I touched that line.
Then there is the prompt.
A client sends a note saying the agent came across as a bit pushy on that last call. I open the prompt, soften one sentence, hit publish. No build. No diff. No test suite. The next caller gets the new behaviour immediately, in every branch of the conversation, including the four branches I was not thinking about while I rewrote that sentence.
That is a production deployment. It just does not look like one, and for longer than I am comfortable admitting I treated it like editing a config value.
This is the change discipline I run on agents already answering real phones. Most of it applies to any LLM feature you have shipped to users, because the missing pieces are the same.
A prompt is global state written in prose
There is no scoping in a prompt. A sentence you add to soften the booking flow is also read during a refusal, during a transfer, and during the part where a caller is slowly spelling out a registration number. You cannot mark it private. Everything sees everything, always.
Two regressions that genuinely happened to me on live agents:
I added a line to make an agent more forthcoming about pricing. It became more forthcoming in general, and started answering an adjacent question it was specifically supposed to decline. Nothing in my change mentioned that topic.
I tightened the confirmation step to cut mistakes on email addresses. The agent became slower and more repetitive on every other capture in the call, which surfaced as callers hanging up mid-booking rather than as an error anywhere.
Neither was visible in the text I changed. Both were visible in behaviour, days later, in numbers nobody was watching closely. The calls completed. The transcripts read fine. Nothing reported a problem.
Four safety nets that are simply not there
There is no deterministic replay. I cannot re-run last Tuesday's call against the new prompt. The audio, the timing, the interruptions and the model's own sampling are gone. Coming from years of real-time work in Unity, this was the hardest adjustment by a distance. Games hand you a repro. Calls hand you a story.
The output is a conversation, not a value. Assert on exact wording and your test is worse than useless, because the wording is supposed to vary.
Production config is edited live in a browser. Most platforms let anyone with a login rewrite the agent in a text box and publish on the spot. There is no review step, and the change history is whatever the vendor felt like keeping.
Part of the stack updates without you. The model version, the voice, the recognition behaviour. You ship nothing and the agent changes anyway. More on that at the end.
None of that discipline comes from the platform, so it has to come from outside.
Build a small regression set out of real calls
The most useful artifact I keep for any live agent is a frozen list of twelve to twenty call scenarios, pulled from real traffic rather than imagined at a desk. It covers:
- the two or three highest-volume intents, in their plain form
- every refusal the agent is supposed to make, one scenario each
- one messy capture: an awkward email, a surname heard three different ways, a postcode
- one caller who interrupts, and one who changes their mind mid-booking
- one call that must end in a transfer, and one where the transfer destination does not answer
- one out-of-scope question that should end in an honest "I do not know, let me get someone"
It is not exhaustive coverage. It is the set where a regression is most likely to be both plausible and expensive.
Assert on outcomes, never on phrasing
For each scenario I write down, in a line or two:
- which exit the call must reach (booked, message taken, transferred, refused)
- which tools must be called, and with what arguments
- what must exist afterwards: the calendar event, the CRM record, the fields captured correctly
- what must never appear: a price quoted, a diagnosis offered, a claim of being human
People forget that last category, and it is the one that gets you in trouble. Pass or fail per assertion, not an overall impression of whether the call felt good. "Felt good" is not a test result.
Automate the structure, listen to the rest
I split every run in two, because the halves catch different regressions.
The automated half drives the agent through the scenarios, captures the transcripts and the resulting records, and checks the assertions. It reliably catches structural regressions: a tool that stopped being called, a field that stopped being captured, a refusal that stopped happening, an exit that now lands somewhere else.
The human half is a handful of calls listened to on a real phone line. It catches what a transcript cannot: pace, a business name mispronounced after a voice update, a confirmation read out as a wall of digits, an agent that now sounds curt. In every one of those the transcript contains the correct string, which is exactly why it cannot be the only thing you look at.
Two tiers keeps it affordable. A wording fix inside one knowledge base answer gets a three-scenario smoke run. Anything touching the prompt, the flow, the tools or the model gets the full set.
Make the change reversible before you make it
Config lives in version control, not only in the platform. I export the prompt, the flow, the tool definitions and the automation workflows into the repo and commit them with the reason in the message. The platform's history is a convenience. The repo is the audit trail, and it is what tells you six weeks later why that odd sentence is in there.
One person publishes. Editing is fine. Publishing is not a group activity. Nearly every unexplained "the agent got worse this week" I have investigated turned out to be two people making individually reasonable edits two days apart.
Rollback is one action, and it has been rehearsed once. Restoring the previous version should take under a minute while the phone is ringing. That habit came out of shipping mobile releases at RAQTS through App Store and Play Console review, where the only real protection against a bad build is how fast you can pull it.
And ship one change at a time. Batch six edits into one publish and when the transfer rate doubles you have six suspects and no way to attribute.
Ship into the quietest traffic you have
There is no staging environment with real callers, so the closest thing is a smaller slice of real traffic. Publish at the start of a quiet window instead of nine on a Monday, or put the change on one agent when a client runs several, or expose it first on an overflow line. Then I read the first fifteen or twenty live calls myself. Not a random sample next week. The first ones, immediately.
After that the signal is a change in shape, not an error. Transfer rate up. Capture accuracy on one field down. Average call length up by twenty seconds. Resolution rate down while volume is flat. Those are the fingerprints of a regression, and not one of them throws an exception.
The change you did not make
This is the part clients find surprising, and the reason the set is worth keeping rather than running once.
The model version changes. The voice provider updates a voice and the business name is suddenly pronounced differently. Recognition behaviour shifts. A client's staff member edits the knowledge base, entirely correctly, and creates a contradiction with a line in the prompt. A CRM renames a field. Any of these can alter agent behaviour with no deployment on your side at all.
So pin versions wherever the platform lets you, treat any provider change notice as a trigger to re-run the full set, and keep a dated note of when the set last passed. When a client says the agent has got worse and you can answer that it passed everything eleven days ago, and here is precisely what changed since, the conversation becomes an investigation instead of an argument.
The short version
Keep the set small and real. Assert on outcomes, not wording. Version the config outside the platform, publish through one person, rehearse the rollback. One change at a time, into quiet traffic.
None of this is sophisticated. It is the release discipline any production system gets, applied to a system whose test failures arrive as a polite customer who did not get a call back.
I write more about building production voice agents and the telephony and automation layer behind them at nabeelbaghoor.com/blog.
Top comments (0)