We build Dograh, an open source voice agent platform, and the tightest requirement anyone has handed us arrived from a bank fraud team as a single sentence. The agent must never ask a customer to reveal their card number.
Easy. Put it in the system prompt. Never request the full card number, never ask for a passcode, never ask for a date of birth. Run the evals. It passes. It keeps passing through staging, through the pilot, through a few thousand calls.
Then a customer answers the confirm-or-deny question with something slightly sideways, and the agent says: "no problem, can you just confirm the rest of the digits for me?"
The forbidden phrase never appeared anywhere in that sentence. The forbidden meaning did.
Prompts leak by paraphrase
This is the failure mode people underestimate when they move a policy into a system prompt. You are not installing a constraint. You are expressing a preference to a model that generates the most plausible continuation of a conversation, and plausibility is exactly what gets you into trouble when a customer goes off script.
The model learned what you banned at the level of wording. It did not learn what you banned at the level of intent, so it reroutes. "Confirm the last four" becomes "confirm the rest." "Verify your identity" becomes a slow slide toward a date of birth. Every one of those survives a keyword check on your transcripts, which is how the problem reaches production intact.
On most outbound calls this would be a policy violation you clean up later. On a fraud-alert call it is worse than that, for a reason specific to the domain.
The domain makes it expensive
The FTC reported that Americans lost 3.5 billion dollars to imposter scams in 2025, with the highest losses of all going to people pretending to be from a bank. Their own description of how those scams open is the uncomfortable part: the costliest ones start with a fake security alert, often from a bank.
So the legitimate call and the most profitable fraud pattern in the country share an opening move. The only thing separating them, from the customer's side of the phone, is what happens next. A scammer asks you to hand something over. A real fraud-alert call should only tell you what the bank already knows: the merchant, the amount, roughly when.
Which means the moment your agent asks for a digit, two things break at once. You have failed a policy check, and you have taught a customer that their bank's real calls behave like the fake ones. The next attacker who calls that person has a warmer target and a better script.
Stop asking the model nicely
The fix is to take generation out of the path at the moments that matter.
Split the call into explicit nodes. At the sensitive ones, the agent has a small set of exact utterances it can emit, written by a human and reviewed like any other piece of production code. The model's job at that point is routing, deciding which node comes next based on what the customer said. It does not get to compose the sentence.
Everywhere else, let it generate. Small talk and acknowledgements and a customer confused about a merchant name all benefit from a model. The identity-handling moment does not.
The best line in the whole flow costs nothing to implement. The agent should be able to stop and tell the customer to hang up and call the number printed on the back of their card. No scammer can survive that instruction, which is precisely why saying it is the strongest signal of legitimacy the call has.
Proving it afterwards
Here is where an architecture decision turns into a hosting decision.
You cannot prove a negative about a generative system. There is no test run that establishes your model will never say a thing. What you can prove is traversal. If every call emits a record of the nodes it passed through, you can demonstrate that no call in the last quarter entered a state where sensitive information was requested. That is an auditable claim rather than an assurance.
Traversal proof only works if the traces are yours. Trace a fraud-alert call through a typical hosted build and count the companies involved: speech recognition at one, the language model at a second, synthesis at a third, telephony at a fourth, orchestration at a fifth. Each sees some slice of a name, a merchant, an amount, and the fact that this person is expecting a call from their bank right now. That last detail is close to a complete script for an attacker.
Banks are working under the GLBA Safeguards Rule here, which requires choosing providers that can maintain safeguards and then reassessing them over time. The FTC is blunt about where responsibility lands, which is with you. Adding a vendor does not subtract an obligation.
Running the stack yourself collapses most of that. The workflow engine, the transcripts, and the traces sit on infrastructure the bank already audits. Two caveats worth stating plainly, because vendors blur them. Telephony is only yours if you run Asterisk ARI against your own PBX, since hosted carriers hold call audio like any third party. Closed commercial weights cannot be self-hosted at all, so the honest best case is same-region providers, which shortens the network path without reducing how many companies hold the data.
If you are starting one of these, build the identity-handling node first. That one node decides whether your call earns belief or sounds like the thing it exists to prevent.
We go further into the vendor math and the latency budget in the full write-up. Read it on Dograh.
Top comments (0)