I've been thinking about this problem for a while. Every company with a phone line has the same nightmare: the carrier goes down and nobody can call you. For most businesses that's annoying. For a bank, a hospital, or an emergency notification service, it's existential.
The standard fix is a runbook. A human gets paged, logs into the carrier dashboard, manually reroutes traffic to a backup connection, and hopes nobody called during the gap. Fifteen to forty-five minutes of lost calls, every single time.
So I built a circuit breaker for voice infrastructure. Not the software kind — the telecom-native kind.
Here's the demo: https://github.com/team-telnyx/telnyx-code-examples/tree/main/auto-failover-voice-routing
The Scenario
I used a fraud alert line for a fictional bank. Here's how it works: the bank's system detects a suspicious transaction. It calls the customer. An AI voice says "We detected a purchase of $1,240.50 at an electronics retailer in Miami. Press 1 if this was you. Press 2 and we'll block your card immediately." The customer presses a button. The card is confirmed or blocked. The customer gets an SMS receipt with a case number.
Now imagine the primary carrier fails during this process. Without a circuit breaker, the customer gets dead air. With one, the same call happens over the backup connection — same voice, same alert, same flow — with one additional sentence: "Heads up: we're running on our backup systems right now."
That one sentence is the only difference the customer notices.
How the Circuit Breaker Works
The system has two independent Telnyx Call Control connections. A router sits in front and watches for failure signals. Every call outcome arrives as a signed webhook from the carrier. When failures accumulate — busy, no-answer, timeout — the breaker trips at a threshold and all new calls route through the backup connection.
After a cooldown period, the breaker goes half-open. The next call is a live probe of the primary. If the probe connects without a failure code, traffic flows back automatically. If the probe fails, backup keeps handling calls.
The key insight is that the carrier tells you about every call outcome in real time. You don't poll for health. You don't guess. The webhooks are the failure detection layer, and they're signed — so nobody can fake a failure and trick your system into switching to backup.
Why This Matters More for Voice Than for HTTP
When your API goes down, clients retry. When your phone line goes down, callers hear dead air and hang up. There's no retry. There's no queue. The call is gone.
That changes the math on failover. In a microservices setup, thirty seconds of downtime means some requests get retried and eventually succeed. In voice, thirty seconds of downtime means customers who called and got nothing. They might try again. They might not.
The Technical Bits
The whole thing runs on Telnyx Edge Compute using the Agent SDK. One durable actor — FailoverAgent — owns the circuit breaker state and the call flow. It uses two Call Control applications as primary and backup connections, monitors carrier webhooks for failure signals, and manages the fraud alert conversation: text-to-speech announcement, keypress collection, SMS receipt.
The state is simple: a failure counter, a last-failure timestamp, and a tripped flag. The actor increments the counter on every failure webhook, trips the breaker at a threshold (default: three), and resets after a successful probe. The routing decision is one function: closed routes primary, open routes backup, half-open probes primary.
What makes it interesting is the customer experience design. The backup announcement doesn't hide the outage — it says "we're running on our backup systems right now." That one sentence builds trust. The customer knows something happened, knows the bank is handling it, and gets on with their day.
What I Learned
Building this taught me that failover for voice is not the same problem as failover for HTTP. The failure modes are different. The recovery is different. And the cost of getting it wrong is different — you're not losing a request, you're losing a conversation.
The circuit breaker pattern has been around forever in distributed systems. Applying it to voice infrastructure, with carrier webhooks as the failure signal and two independent Call Control connections as the failover path, is the telecom-native version of a pattern every backend engineer already knows.
The sample is open source and includes a demo trigger endpoint so you can test the breaker without waiting for a real carrier outage. Clone it, wire up two connections, and your phone line becomes outage-proof.
Top comments (0)