DEV Community: Rokas Remeika

Scheduled Outbound Calls Fire An Hour Early After Daylight Saving Transitions

Rokas Remeika — Thu, 05 Nov 2026 15:03:00 +0000

By Rokas Remeika. Published 22 October 2026.

Scheduling outbound calls using naive local time arithmetic breaks twice a year. A spring-forward gap creates a wall-clock time that does not exist, causing date libraries to throw, while a fall-back overlap creates a time that exists twice, causing unpredictable dialing an hour early. To fix this, ask your date library if a constructed local time is valid to detect gaps, and snap to the next existing instant. For overlaps, consistently select the earlier instant. Finally, validate that the projected call end falls within the calling window using the estimated conversation duration.

Symptoms

An outbound call goes out at 07:00 local time instead of the expected 08:00 on the Monday following a clock change.
A scheduled dial executes legally at 19:50, but the conversation is still running at 20:20, pushing past the strict 20:00 cutoff the customer configured.
A scheduling job throws a runtime exception on a specific date because the underlying date library asserts the constructed local time does not exist.

What is actually happening

These present as three separate errors that all look like timezone bugs.

The first error stems from deriving a timezone exclusively from a phone number's country code. This approach works for most countries, but it is simply wrong for countries that span several zones. Guessing the local time for a contact in a country spanning several zones based purely on the country code is a coin flip. Eventually, your scheduler will guess incorrectly and place a call while the recipient is eating breakfast.

The second error comes from checking window membership at the start instant only. A calling window constrains the entire conversation rather than the moment of dialing. When you evaluate a scheduled task by comparing the dial instant to the cutoff time, a half-hour call started ten minutes before the cutoff violates the constraint for twenty minutes. The system believes it acted legally because the dispatch cleared the gate, but the recipient experiences an out-of-hours intrusion.

The third error is raw arithmetic failing across a daylight-saving transition. When calculating future schedules, a spring-forward gap creates a wall-clock time that does not physically exist. A naive local-time construction will immediately throw an exception. Conversely, a fall-back overlap creates a time that exists twice in the same day. Without explicit instructions, date libraries silently pick one of the two instants, usually placing them exactly an hour apart from what you intended. The library makes an arbitrary decision, and your scheduler executes on it.

How to confirm it

Search your application exceptions for invalid date or time errors clustered specifically on the exact day of a spring-forward transition. Expect to see date library validation errors during the spring-forward gap where the target wall-clock time does not exist. This confirms your code is constructing a non-existent wall-clock time and the library is throwing a validation error.
Query your call logs for the conversation end time against the allowed window cutoff for any complaints about late calls. Expect the start time to be legally inside the window, but the duration pushing the end time past the limit. This confirms the scheduler only validates the dial instant and ignores the call duration.
Check the timezone assignment logic for multi-timezone regions in your database. Look at a random sample of numbers from multi-zone countries. Expect to find a default offset being blindly applied. This confirms derivations are being made where explicit user selection is required, proving the system is guessing rather than knowing.

How it works

I built my scheduling layer to treat local time complexity as a strict boundary. The first rule I enforce is to refuse to guess. The scheduler derives the country from the parsed phone number and maps that country to an IANA zone only where that mapping is unambiguous. For countries that span multiple zones, I require an explicit selection rather than defaulting. The customer configuring the campaign knows which zone their contact lives in, and I do not. If the explicit selection is missing, the schedule is rejected.

To enforce the calling window, I require the projected call end to fit entirely inside the allowed window rather than checking only the dial start time. The system calculates the projected end by adding the estimated call duration to the target dispatch time. The scheduler already carries this duration estimate for capacity accounting. When deciding admission inside one transaction, as detailed in /blog/enforcing-maximum-concurrent-sip-calls-without-race-conditions/, the system allocates a slot based on how long the call will occupy it. I feed that same duration metric into the window evaluation. This projection relies on the estimated call duration that the scheduler already tracks for capacity accounting. If the total expected duration pushes the end of the call past the cutoff, the dispatch is declined.

For daylight-saving transitions, I handle both directions explicitly rather than trusting library defaults. In a spring-forward gap, I detect the missing time by asking the date library whether the constructed local time is valid at all. I use this validity check instead of hardcoding transition dates. When it returns invalid, the system snaps to the next instant that actually exists. In a fall-back ambiguity where the time exists twice, I configure the library to choose the earlier of the two instants consistently. This makes the behavior a documented rule rather than whichever branch the library happens to prefer on a given language version.

Finally, I instrument every scheduling decision. The scheduler logs the resolved IANA zone and records a boolean flag indicating how it was resolved: explicitly selected or safely derived. I maintain a strict count of gap snaps and ambiguity resolutions, which is the only way you find out your DST handling ran correctly during a transition weekend. I also track a distinct count of dispatches declined specifically because the projected end fell outside the window, separating these rejections from capacity-based delays.

What I would check first

Review how your date library handles ambiguous local times during fall-back overlaps. Expect it to default to one of the two instants silently without warning. Check whether early dispatches happen only during transition hours or persist on standard days to distinguish overlap ambiguity from invalid timezone offsets.
Inspect the mathematical validation logic for your calling windows. Expect the code to evaluate only the current timestamp against the cutoff limit. If the check lacks an addition of the projected call duration, this rules out complex timezone math errors and confirms a logic gap in your boundary enforcement.
Look at your timezone mapping dictionary or function for parsed phone numbers. Expect single-zone countries to map cleanly. Correlate out-of-hours complaints with recipient country codes to separate multi-zone derivation errors from daylight saving arithmetic bugs.

Making daylight saving time an explicit decision

This architecture generalises to any scheduler that acts on someone else's local time. Whether you manage outbound calls, SMS campaigns, appointment reminders, or delivery windows, the local time belongs to the recipient.

Three rules carry across all these domains. First, never infer a timezone you can ask for. A mapping function is a convenience for single-zone regions, not a replacement for domain knowledge. Second, constrain the end of the activity rather than its start. The user constraint always applies to the entire duration of the intrusion. Third, make both DST directions explicit decisions with a written rule.

Every mature date library exposes a validity check that doubles as a gap detector. You must use it rather than building a transition calendar. Hardcoding transition dates guarantees your system will fail the moment a local government changes their DST observance policy, which happens frequently. By asking the library if the constructed time is valid, you offload the political complexity to the underlying tz database updates, while your application code safely snaps to the next available instant.

Common questions

Why did a scheduled call dial an hour early on Monday?

Your scheduler performed local-time arithmetic across a fall-back daylight saving transition. Naive local-time arithmetic across the clock change calculated an instant an hour off from the intended local hour.

How do I handle a scheduled time that does not exist during spring forward?

Ask your date library whether the constructed local time is valid instead of hardcoding transition dates. When it returns invalid, snap the scheduled time to the next instant that actually exists.

Should I derive the timezone from the phone number?

Only for countries that do not span multiple timezones, mapping the country to an unambiguous IANA zone. For countries spanning several zones, require an explicit selection instead of defaulting.

Why did my call run past the calling window cutoff?

Your scheduler checked the window membership using only the dialing start instant. A window constrains the conversation, so you must validate the projected call end using the estimated duration.

Canonical URL: https://www.dreamchasers.lt/blog/scheduled-outbound-calls-fire-an-hour-early-after-daylight-saving-transitions

The Realtime Session Dies On Connect With A Websocket 1007 Close

Rokas Remeika — Tue, 03 Nov 2026 14:43:17 +0000

By Rokas Remeika. Published 20 October 2026.

The websocket closes with code 1007 because the realtime plugin built a session-setup frame using a configuration schema the provider recently deprecated. The provider treats the outdated shape as malformed payload data and drops the connection immediately instead of returning a validation error. Because the plugin constructs this frame internally before returning control to your application, you cannot fix it in your configuration. You must patch the plugin's setup construction code at runtime to emit the current schema.

Symptoms

The agent joins the LiveKit room and the participant shows as connected.
The worker takes the job successfully.
The session produces no audio at all.
The provider closes the backend websocket almost immediately.
The close code is 1007, with no validation message and no useful error text.
The room state is entirely misleading. From the client's point of view, everything is healthy, but nobody speaks.

What is actually happening

The failure happens entirely on the backend leg between your agent worker and the realtime model provider. The WebRTC connection to the caller is perfectly healthy, which is why the room looks fine.

When a realtime speech session starts, the provider expects a session-setup frame over the websocket. This frame dictates the voice, the temperature, the endpointing configuration, and the system prompt. In a managed framework, you do not build this JSON frame yourself. The LiveKit Agents realtime plugin builds the setup frame for you, translating your application-level configuration into the exact JSON shape the provider expects.

The 1007 close code means "invalid frame payload data". The provider deprecated a specific configuration schema block in its API, specifically a generation-parameters block whose key naming changed between API versions. The realtime plugin was still building the setup frame using the old schema.

When the provider receives the outdated shape, it does not respond with a field-level validation error. It simply treats the frame as malformed and terminates the websocket immediately with a 1007 close code. Nothing in your application code can prevent this natively, because your application never sees the final frame. The plugin constructs it internally and puts it directly on the wire.

How to confirm it

Record the websocket close code and close reason on every session end, not just on errors. Expect to see exactly 1007 logged milliseconds after the connection opens. A 1007 close code confirms the provider rejected the frame payload rather than terminating cleanly.
Patch the provider client's session-connect entry point to log the full outbound setup payload at the boundary, exactly once, before the connect call. Expect to see the raw JSON dictionary the plugin is about to send.
Compare the logged outbound payload against the provider's current realtime API documentation. Expect to find a key in the generation-parameters block that the documentation no longer lists, or a new required key that your logged payload lacks.

The fix

Because I work in Python on a pinned SDK version, the solution was to patch the plugin's setup-config construction directly at runtime to emit the current schema.

You cannot just change your application config dict, because the plugin will map your inputs back into its hardcoded internal representation. You must intercept the step where the plugin formats the outbound frame.

I wrote a monkey patch for the plugin's setup frame builder. The patch wraps the builder method, calls the original implementation to get the base dictionary, and then mutates the dictionary keys to match the provider's new API requirements before returning it.

Apply this as an idempotent runtime patch. Guard the replacement with a sentinel attribute on the patched class so re-import or double application is a no-op. Check for this attribute before applying the wrapper. This ensures that if the module is re-imported, or if a queue worker forks and runs the patch logic a second time, the double application is a safe no-op.

Gate the schema change on the exact model generation you are dialling. The provider only expects the new schema for the newer models. The older generation you were already working with keeps the exact bytes it was working with. If you rewrite the keys unconditionally, you will break older model generations that expect the deprecated schema.

What I would check first

The exact websocket close code. A 1007 means the payload structure was rejected by the parser. A 1007 close code confirms the provider rejected the frame payload data as malformed. If you do not have the close code logged, you are guessing.
The exact bytes of the outbound setup payload. This rules out your own application logic. Inspecting the logged frame reveals whether the plugin emitted the deprecated generation-parameters schema before going on the wire.
The model version string in the request. This rules out using a new model with an old SDK. If the provider shipped a new model generation last week, their API boundary for that model likely expects the new setup shape.

Why documentation will lie to you

When a provider deprecates a config shape, you get a close code rather than a validation error, and the only place to fix it is the layer that builds the frame.

If you read the provider's API documentation, you will see the correct, current schema. If you read the plugin's documentation, you will see the configuration shape for the newest release. Neither of these helps you debug the live incident.

You must read the installed SDK source on your worker. The documentation describes the version they shipped today. Your worker is running the version you pinned in your dependencies. Opening the local site-packages directory and reading the frame-builder function is the only way to prove what your system is actually doing.

Gating the patch by model generation

The fix must be conditional. Realtime providers version their APIs tightly with their model generations. An older speech-to-speech model will reject the new generation-parameters block just as aggressively as the new model rejected the old one.

I have hit a similar schema rejection later in the call loop, exactly at the ten-minute mark, when the provider recycles the session. In that case, the realtime plugin reconnects and replays the conversation so far into the new session's setup frame. If you see a 1007 close code after ten minutes instead of on connect, you are hitting the history replay variant. You can read the full breakdown of that failure in my post on why a realtime session reconnect fails with 1007 at ten minutes.

If the reconnect opens a new session entirely without context, causing the agent to greet the caller again mid-conversation, you are missing the continuation token in this same setup frame. I covered that fix in my post on why a voice agent goes silent after a websocket reconnect.

Common questions

Why does the realtime websocket close with code 1007?

Code 1007 indicates invalid frame payload data. The realtime model provider is rejecting the initial session-setup JSON frame because its schema is malformed or deprecated.

Why is the LiveKit room healthy but there is no audio?

The WebRTC transport between the caller and your server is completely separate from the backend websocket to the AI provider. The provider rejected the setup frame and closed its leg, leaving the caller in a silent room.

Can I fix the 1007 error by updating my agent config?

No. The SDK plugin rebuilds the session-setup frame internally before sending it. You must patch the plugin's internal builder at runtime to emit the current schema.

How do I see the frame causing the 1007 error?

You must intercept the provider client's session-connect entry point. Log the full outbound setup payload just before it goes on the wire to see the exact bytes being rejected.

Canonical URL: https://www.dreamchasers.lt/blog/the-realtime-session-dies-on-connect-with-a-websocket-1007-close

Enforcing maximum concurrent SIP calls without race conditions

Rokas Remeika — Sun, 01 Nov 2026 13:50:10 +0000

By Rokas Remeika. Published 18 October 2026.

To prevent concurrent workers from overcommitting a SIP trunk, capacity checks and claims must occur inside the same atomic database transaction. Reading a live-session count and writing a claim in separate steps allows workers to interleave and breach the limit. For scheduled calls, point-in-time checks fail because calls occupy time, not just an instant. Project capacity over a grid of five-minute slots spanning the call's estimated duration, and admit the call against the worst overlapping interval rather than the current count.

Symptoms

Your SIP trunk carrier rejects outbound legs outright because you exceeded your licensed channel capacity.
Logs show two separate dispatch workers inspecting the capacity limit at the same second, passing the check, and dialing simultaneously.
The capacity count looks perfectly safe at the moment of dispatch, but the concurrency ceiling is breached ten minutes later.
A single destination phone number exhausts its capacity limit and fails while the overall SIP trunk remains quiet and well below its ceiling.

What is actually happening

This is a scheduling failure driven by two distinct bugs wearing the same costume.

The first is a classic read-then-write race condition. A dispatcher reads the live-session count from the database, decides there is room, and writes its claim in a second step. When two workers run at the same time, they interleave between the read and the write. Both read a safe number, both admit the call, and your system hands the carrier one more leg than it is licensed to carry.

The second bug is treating a scheduled call as a point in time. A voice agent call is not an instantaneous event. It occupies a SIP channel for its estimated duration. A capacity check that counts what is live right now systematically under-counts what will be live in fifteen minutes. When several long calls admitted at different moments are all still up, they overlap. The ceiling is crossed by calls that were each individually admissible at the exact moment they were evaluated.

How to confirm it

Query your dispatch logs for the timestamps of the capacity read and the capacity write. Expect them to be identical in a healthy atomic system. If they are milliseconds apart, you have a read-modify-write gap where another worker can slip in.
Plot the estimated duration of all admitted calls on a timeline grid. Expect the peak concurrency to match your point-in-time metrics. If the timeline shows a peak that exceeds your ceiling, your admission logic is blind to call duration.
Look at the rejection timestamps from your carrier. Expect them to cluster during periods where multiple long-running calls overlap, rather than right at the start of a busy minute.

How it works

I designed and operate an admission control system to make this breach unreachable. It handles capacity at the application layer, running on a single VM, and bounds the concurrency before a call ever starts. No ceiling breach has been observed here because nothing has been run at a volume that could produce one, but the design prevents it structurally.

The core rule is that the whole admission decision happens inside one atomic transaction. The transaction re-reads the capacity configuration and the live-session set after it opens. Trusting anything read before the transaction began is unsafe, because an operator can change the ceiling between the probe and the commit.

To fix the point-in-time counting problem, the system projects capacity over a grid. I tuned this to a five-minute slot granularity. Instead of looking at a single point count, the system reduces peak concurrency over these five-minute slots spanning the call's estimated duration. A new call is admitted against the worst slot it overlaps. If a call is estimated to last fifteen minutes, it must clear the available headroom across all three of the upcoming five-minute slots.

This transaction enforces two limits simultaneously: a global ceiling for the entire trunk, and a per-number ceiling. Both are evaluated together. A single phone number can easily exhaust its own specific concurrency limit while the overall trunk remains quiet, so checking only the global limit is insufficient.

To stop transactions from deadlocking or constantly retrying due to contention, I serialize admissions per capacity pool. This sits behind an exclusive lock in the database. When a dispatcher wants to place a call, it acquires the lock for that specific pool. This ensures pools do not interleave read-modify-write operations on shared counters. Calls in unrelated pools proceed in parallel without waiting on each other.

What I would check first

Correlate carrier capacity rejections with dispatch timestamps to see if failures match simultaneous dispatch bursts. An exact match rules out a read-then-write race condition. A gap of a few milliseconds means you are evaluating capacity outside the transaction.
Compare trunk occupancy metrics at dispatch time against occupancy ten minutes later when long calls are still active. A query that only uses the current instant rules out duration awareness, confirming that overlapping long calls will breach your limits.
Check whether outbound placement failures are isolated to a single destination phone number or affect the entire trunk. The absence of a per-number ceiling means a single outbound campaign can exhaust the pool, which rules out targeted rate limiting and points to a global trunk exhaustion issue.

Why application-level counters fail

When engineers first spot the capacity breach, the instinct is often to introduce an in-memory counter in the application layer. If you run a single VM, as I do here, an in-memory counter seems like a straightforward way to track live sessions. You increment the counter when the call starts, and decrement it when the call completes.

This fails because it decouples the capacity state from the scheduling state.

Even backed by a document store, a simple counter only tracks the calls that have already dialed. It cannot see the calls that are scheduled to start in ten seconds.

This is why the database must be the source of truth for capacity. The scheduling data and the capacity claim must reside together, protected by the same exclusive lock.

Instrumenting the admission control path

When you move capacity checks inside an atomic transaction with exclusive locks, you introduce a new bottleneck. The transaction itself becomes the expensive resource. To maintain visibility, you need to instrument exactly what happens around that lock.

The first metric to track is the time spent waiting on the per-pool admission lock. If your dispatch pipeline slows down, this wait time is the first thing to move. It tells you immediately whether the delay is in the database contention or somewhere else in the application layer.

I also emit the admission outcome per attempt. This must be typed and explicitly record which ceiling bound the decision. You need to know if it hit the global SIP trunk ceiling or the per-number ceiling.

Finally, the system logs the peak occupancy per five-minute slot alongside the forecast the admission decision used. Emitting these together allows the forecast to be compared against reality later, though no such comparison has been run here yet.

Applying limits to finite external resources

This pattern generalizes beyond SIP channels and voice agents. Any system that admits work against a finite external resource requires the same structural protection. Whether you are dealing with carrier ports, licensed software seats, or GPU slots, the constraint is identical.

The rule remains the same whether you use Postgres or a document store. The check and the claim must be the same atomic act. If the work has a duration, you have to admit against the worst overlapping interval rather than the current count.

Common questions

Why are outbound calls exceeding the SIP trunk channel limit?

Dispatch workers are likely interleaving a read-then-write race condition. Two workers read the capacity count, see available headroom, and dial simultaneously before either writes their claim.

How do I enforce maximum concurrent calls across concurrent dispatch workers?

You must move the admission decision into a single atomic database transaction. Use an exclusive lock per capacity pool to serialize the dispatchers before they check the available limits.

Why does peak concurrency spike above the limit later in the call?

Your capacity check only evaluates the current point in time. Because calls have duration, multiple long calls admitted safely at different times will overlap and breach the ceiling.

How should scheduled calls be counted against capacity?

Reduce peak concurrency over a time grid covering the call's estimated duration. Admit the new call against the highest concurrency interval it overlaps, not the current moment.

Canonical URL: https://www.dreamchasers.lt/blog/enforcing-maximum-concurrent-sip-calls-without-race-conditions

Realtime Session Reconnect Fails With 1007 At Ten Minutes

Rokas Remeika — Fri, 30 Oct 2026 13:34:29 +0000

By Rokas Remeika. Published 16 October 2026.

Your realtime provider is recycling long sessions on its own schedule, usually at the ten-minute mark. When the plugin reconnects, it replays the conversation so far into the new session setup frame. This includes model-role turns. Newer model generations reject that specific history shape outright and close the new connection with a 1007 invalid payload code. To fix this, patch the setup frame to carry the initial replay as client content, which the newer generation accepts. Gate this change on the model generation so older models retain their expected behavior.

Symptoms

Every call ends around the ten-minute mark. The caller is speaking or listening to the agent, and suddenly the agent is gone while the caller is still on the line.

Short calls are perfect. Audio flows cleanly in both directions on calls under ten minutes. The system survives all testing because developer tests and evaluation runs rarely simulate a single continuous ten-minute conversation.

The session log shows a normal provider-initiated close. Immediately following that close, there is a reconnect attempt that fails with a websocket 1007 code. After the 1007, the agent process cleans up the activity and drops out of the room.

What is actually happening

The disconnect at the ten-minute mark was not a bug. The provider recycled long realtime sessions on its own schedule, issuing a server-initiated session-recycle notice before closing the connection cleanly.

My plugin correctly received the notice and triggered a reconnect. The reconnect is a completely different code path from the initial connect, and it carries state that the initial connect did not have. To ensure the new session had the context of the last ten minutes, the plugin replayed the conversation so far into the new session's setup frame. This history replay included model-role turns, meaning the setup frame told the provider exactly what the model had already said.

A newer model generation rejected that history shape outright. The schema rules had tightened, making model-role turns in a setup frame illegal. Rather than returning a field-level error indicating which key was wrong, the provider simply treated the entire payload as malformed and closed the new connection as invalid, issuing a 1007 code.

The disconnect itself was routine and expected. The failure was that the reconnect path sent a payload the first connect never sent, and that payload was now fatal.

How to confirm it

Record wall-clock call duration as a field on every call end event. Expect the durations of the failed calls to pile up at a round number, such as 600 seconds. A tight clustering at a round number confirms this is a provider-scheduled event rather than a random network coincidence.
Record every reconnect as a first-class event, including the close code that preceded it. Expect to see a standard recycle close code followed immediately by a 1007 on the reconnect. This sequence confirms the crash is happening on the resumption attempt.
Log the setup frame payload sent on the initial connect and compare it to the payload sent on the reconnect. Expect the reconnect payload to contain model-role messages in its history array. Finding these model-role turns confirms the plugin is sending the shape the newer generation rejects.

The fix

I configured the history so the initial replay was carried as client content. The newer model generation accepts client-role content in the setup frame without rejecting the connection. I applied this fix at the same setup-frame patch point inside the plugin where schema corrections live.

Because I occasionally route traffic to older models, I gated this behavior on the model generation. The older generation kept the original behavior it was already working with, and only the newer generation received the modified client-content shape.

I verified the fix using a synthetic session-recycle injector. Sitting on ten-minute calls is an unacceptable debug loop. I built an injector that raised the exact same reconnect path a real session recycle raises, but timed it to fire after five seconds. This turned a ten-minute edit-test loop into a five-second one.

What I would check first

Check the call durations on the dropped sessions. Expect exactly ten minutes. This rules out random network instability and rules in a provider-enforced boundary.
Check the payload structure on the second connection attempt. Expect model-role messages in the history array. This rules out missing authentication credentials and confirms a schema rejection.
Check whether session continuation is configured in your setup frame. Expect it to be missing if the agent greets the user again instead of dropping the call entirely. If the agent greets the caller a second time, you are hitting the failure described in /blog/voice-agent-goes-silent-after-a-websocket-reconnect/ where the session restarts cold.

Why the reconnect path lacks test coverage

Reconnect is the path least covered by tests. Reaching it honestly takes as long as the provider's session lifetime.

Any long realtime session has this shape, whichever vendor you are on. The client must maintain context across physical connections, which means it will attempt to serialize the state of the room and send it upstream. If the upstream provider changes its schema validation for history payloads, the connection will drop.

Patching plugin internals in production

When the SDK itself is the blocker, you have to patch the plugin internals in production. The plugin builds the session-setup frame for you, abstracting away the raw JSON payload.

I wrote the patch against the installed SDK source. I intercepted the setup-config construction, inspected the target model generation, and mutated the history array before it went over the wire.

Common questions

Why does my voice agent drop the call at exactly ten minutes?

Your realtime provider recycles long sessions around the ten-minute mark. Your agent attempts to reconnect but sends a payload the provider rejects, killing the call.

What does websocket close code 1007 mean during a reconnect?

Code 1007 means invalid frame payload data. During a reconnect, your agent is sending a conversation history shape in the setup frame that the model considers illegal.

How do I test a ten-minute session timeout without waiting?

Build a fault injector at your session event boundary. Create a timer that triggers the exact session-recycle notice and reconnect path after five seconds.

Why does the reconnect path fail when the first connect succeeds?

The reconnect path carries state the initial connect did not have. It replays conversation history, including model-role turns, which the newer schema rejects on session setup.

Canonical URL: https://www.dreamchasers.lt/blog/realtime-session-reconnect-fails-with-1007-at-ten-minutes

Inbound SIP Calls Produce No Logs, Or Trunk Provisioning Creates Duplicates

Rokas Remeika — Wed, 28 Oct 2026 13:36:27 +0000

By Rokas Remeika. Published 14 October 2026.

Trunk creation APIs generally create resources rather than upserting them. Any provisioning path run twice produces duplicate trunks, leaving your outbound dialer pointing to a stale ID. Separately, raw carrier trunks use an IP allowlist for inbound authenticity. If the carrier's signalling CIDR blocks are missing from that list, inbound INVITEs are dropped before your application exists. To fix both, wrap provisioning in a name-keyed create-or-update script, and pin the exact published carrier CIDRs on the inbound trunk.

Symptoms

You run your infrastructure provisioning script, redeploy your environment, or retry a failed setup step. Shortly after, you open your provider console and see multiple SIP trunks sharing the exact same name but holding entirely different identifiers. Nobody on the team can confidently say which trunk ID the production phone numbers are actually attached to.

When your application attempts to dial out, outbound calls fail abruptly. Outbound calls fail unpredictably because your stored outbound trunk ID points to a duplicate trunk instance that lacks the expected phone number mapping.

Alternatively, your outbound calls work fine, but inbound calls never reach your application code. There are absolutely no logs, no webhooks firing, and no participant joining the session. The carrier insists they are sending the traffic. The carrier insists it is dispatching the calls, while your system records total silence. Because the INVITE is dropped before your application exists in the call path, no log, webhook, or participant is ever generated.

What is actually happening

There are two distinct mechanisms at play here: one governing the duplicate trunks, and one governing the silent inbound failures.

Trunk creation APIs generally create resources; they do not upsert them. SIP trunking platforms often treat trunks as logical containers and do not enforce uniqueness on the display name. When your provisioning script issues a creation request, the API happily provisions a second logical container with the same name. Any provisioning path that runs more than once produces duplicates. This happens during a redeploy, a rerun after a partial failure, or when a second engineer follows the same runbook.

When the API creates this duplicate, the outbound trunk ID stored next to your phone numbers in your database becomes stale. The stored outbound trunk ID may point to any of the duplicates, leaving your dial path disconnected from the trunk associated with the phone numbers. Outbound dials using that stale ID will fail. I wrote about a similar state loss in LiveKit SIP trunk does not exist after a container recreate or host reboot, where transient state loss breaks the exact same routing path.

Separately, on a raw carrier trunk, the inbound authenticity check relies on an IP allowlist of the carrier's signalling addresses. SIP is a raw network protocol, and anyone can send an INVITE to an open port. On a raw carrier trunk, the inbound authenticity check is an IP allowlist of the carrier's signalling addresses. This allowlist sits at the very edge of the trunk configuration.

If those CIDR blocks are wrong or incomplete, incoming SIP INVITEs are dropped immediately at the boundary. This drop happens before your application even exists in the call path. The failure produces total silence rather than an application-level error because the traffic is rejected at the network transport layer. Your application cannot log a webhook for a network packet it never saw.

How to confirm it

Query your trunk provider API to list all configured trunks by name. Expect to see exactly one inbound and one outbound trunk per environment. Seeing two or more with identical names confirms your provisioning path lacks idempotency and is blindly creating duplicates.
Read the outbound trunk ID stored in your database and compare it against the list of active trunk IDs in the provider console. Expect them to match exactly. A mismatch confirms your application is dialing through a ghost trunk while the real phone numbers sit on a different instance.
Trigger an inbound test call and inspect your ingress layer or host firewall logs for dropped UDP or TCP traffic on port 5060 or 5061. Expect to see dropped packets originating from carrier IP addresses. A dropped INVITE confirms an allowlist mismatch, not an application routing failure.
Cross-reference the source IP of that dropped packet with the carrier's published documentation. Expect the source IP to fall within a CIDR block that is entirely missing from your trunk's configured allowlist.

How it works

I run provisioning as a name-keyed create-or-update operation for both directions. The script lists existing trunks, matches them on a stable name, updates them in place if they exist, and creates them only when they are absent. This is the only safe shape for infrastructure code that runs multiple times. Provisioning code is run more than once by definition, so name-keyed upserts prevent split-brain routing.

For inbound authenticity, I pin the carrier's signalling CIDR blocks on the inbound trunk directly from the carrier's own published documentation. I never populate this list by observing source addresses during staging tests. Carriers routinely route production traffic from entirely different subnets or failover datacenters. If you only allowlist the single IP you saw during a test call, the first production call from a different switch will hit your port and get dropped immediately. For outbound trunks, I set the carrier's SIP address explicitly.

I give this provisioning system a single idempotent entry point with two operating modes.

The first is an explicit mode. It provisions the exact trunks it is told about via configuration files. This handles the initial setup, deliberate configuration changes, and testing new routing rules.

The second is a self-heal mode. It sources the active phone numbers from the database of record, provisions both the inbound and outbound trunks to match, and then writes the fresh outbound trunk ID back to the exact location where the dial path reads it. When an infrastructure restart or a container recreate occurs, recovery is one command rather than a multi-step manual runbook. The system boots, detects its state, provisions exactly what is missing, and binds the correct IDs automatically.

To keep this system honest, I check the trunk count by name on a schedule. Anything above one trunk per direction constitutes drift. I also explicitly separate inbound INVITEs rejected at the trunk from those that reach the application. This ensures an allowlist error is instantly distinguishable from a routing error in my telemetry. I record every self-heal provisioning run and its outcome, giving me a clear audit trail of when the system had to rebuild its own trunk bindings.

What I would check first

Look for raw network traffic on the SIP port. If the carrier claims a call is active but your application is silent, run a packet capture or check your host firewall logs. Seeing packets arrive and get dropped rules out a carrier routing issue and points directly to your IP allowlist. Seeing no packets at all means the carrier is sending them to the wrong destination IP entirely.
Check the trunk configuration for the inbound IP allowlist. Compare it against the carrier's official documentation. An exact match rules out the allowlist as the culprit and points to a misconfigured local firewall or a crashed application listener. A missing CIDR block confirms the trunk itself is rejecting the calls.
Read the trunk ID out of your database and compare it to the API response. If the ID in your database does not exist in the active trunk list, this confirms your provisioning script created a new trunk without updating the application state. It rules out a temporary network glitch as the reason for outbound call failures.
Inspect trunk telemetry to separate rejected inbound INVITEs from traffic reaching the application. Seeing INVITEs dropped at the trunk confirms an allowlist mismatch, whereas packets reaching the application rule out the allowlist and isolate the issue to downstream routing. This ensures an allowlist rejection is immediately distinguishable from an application dispatch failure.

Common questions

Why do inbound SIP calls fail without any logs?

Incoming SIP traffic must pass an IP allowlist on the raw carrier trunk. If the carrier's signalling IP is missing from that list, the INVITE is dropped before reaching your application. You see no logs because the traffic is rejected at the network boundary.

How do I prevent duplicate SIP trunks during deployment?

Stop using create-only APIs in your deployment scripts. Implement a script that lists existing trunks, matches them by a stable name, and updates them in place. Create new trunks only when the name search returns zero results.

Where should I get the IP addresses for my SIP allowlist?

Always use the official signalling CIDR blocks published in your carrier's technical documentation. Never rely on the source IP addresses you observe during staging tests, as production calls will route from different subnets.

How do I recover lost trunk IDs after an infrastructure restart?

Build a self-heal mode into your provisioning tool that sources numbers from the database of record, provisions both trunks, and writes the fresh outbound trunk ID back where the dial path reads it.

If the boundary this note describes is one you would rather own, a self-hosted deployment you control moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/inbound-sip-calls-produce-no-logs-or-trunk-provisioning-creates-duplicates

LiveKit Agent Opener Answers a Phantom Turn and Reply Requests Time Out

Rokas Remeika — Mon, 26 Oct 2026 13:35:27 +0000

By Rokas Remeika. Published 12 October 2026.

The framework forces the model to speak first by pushing a one-character text placeholder into the session, then waits for the provider to acknowledge it. A speech-to-speech model reads this punctuation as a real user turn and shapes its greeting as a reply to it. Because it is a client-initiated kickoff, the provider does not reliably emit the expected generation-created event, causing explicit reply requests to time out while the audio is already playing. The fix is patching the plugin to use an XML-shaped internal control marker on a private message channel, and resolving the reply future immediately upon enqueueing.

Symptoms

The agent's first spoken line sounds like a disjointed reply to something the caller said, even though the caller has not spoken yet.
The transcript carries a spurious one-character user turn right before the agent's greeting.
Explicit reply requests raise a timeout error waiting for generation_created or a similar provider acknowledgment.
The timeout fires on calls where the model demonstrably does speak and the audio reaches the caller.

What is actually happening

To make a realtime model start a turn without user audio, the framework needs something for the model to respond to. It pushes a one-character placeholder text message into the session to simulate an initial prompt. The framework then waits for the provider's generation-created event before it resolves the reply future and considers the turn successfully started.

This approach was written for text-first LLM providers, and on a speech-to-speech provider, both halves of the strategy are wrong.

First, the placeholder is treated as real conversational content. The model reads a lone punctuation character as the caller's actual turn. The model reads a lone punctuation character as the caller's turn and shapes its opening greeting as a response to it. This is why the greeting sounds subtly off, and why the first transcript line belongs to a phantom user instead of the agent.

Second, the provider does not reliably emit a generation-created event for this specific type of client-initiated text kickoff. The provider begins generating audio and streaming it down the websocket, but the explicit server event the framework expects never arrives. Because this event never arrives, the pending generation future remains unresolved while generation is in fact under way. Eventually, the explicit reply call raises a timeout exception in your application code, even while the model generation is in fact under way and the caller is hearing the greeting.

How to confirm it

Log every conversation item the framework creates on a failing call, including the items your code did not explicitly author. Expect to find a one-character text item injected right at the start of the session. If this item is present, the framework is forcing the turn.
Time your explicit reply request from the function call to the exact millisecond the timeout exception fires, and compare it against the arrival time of the first agent audio frame. Expect the audio frame timestamp to fall within the timeout window. An audio frame arriving while the future is still pending confirms the provider is generating speech but skipping the event acknowledgment.
Search the installed framework source code for the payload it injects to start a turn. Expect to find a hardcoded string containing a single punctuation mark. Check which public API method it uses to send this string; if it uses the public turn API, it will inevitably appear in the conversation transcript.

The fix

I patched the plugin's kickoff path directly in the installed source. The goal was to remove the false conversational context while still forcing the model to generate a response.

I replaced the punctuation placeholder with an explicit internal control marker. This marker was XML-shaped rather than something that reads as human speech, ensuring the model would treat it as a system directive rather than a user utterance. I pushed this payload onto the plugin's own internal message channel rather than passing it through the public turn API. Routing it this way meant it never became a conversation item, keeping the public transcript clean.

I also updated the framework's future resolution logic on the kickoff path. I modified the kickoff path to resolve the pending generation future as soon as the kickoff payload was enqueued on the transport, instead of waiting on a server event that might never arrive for this specific execution path.

Finally, I wrapped these changes as an idempotent runtime patch. I gated the patch on the model generation state, exactly like the other realtime patches in my infrastructure, so a re-import or a double application would be a safe no-op.

What I would check first

When a reply timeout fires or an agent starts with a bizarre greeting, I run through this triage order to isolate the boundary responsible.

Check if the timeout is accompanied by dead air. If the caller hears absolutely nothing, you might be dealing with a severed media path or a failed audio subscription, not a state machine hang. If the caller hears a greeting while the logs show a timeout, the transport is fine and the framework's event loop is waiting on a missing acknowledgment.
Check the source stream for the spurious transcript character. A speech-to-speech session carries two separate transcription streams. If the spurious character comes from the inbound caller audio stream, verify whether the plugin is reading from the wrong side's transcription stream rather than an injected turn. If it appears instantly without any caller audio, the framework is injecting it locally.
Check the reconnect logic. If this timeout only happens halfway through a call rather than at the very beginning, inspect your websocket re-drive mechanisms. A reconnect restores the transport, not the turn. As I covered in a previous write-up, a voice agent goes silent after a websocket reconnect because the model waits for input it already consumed. Issuing a new explicit reply request there can trigger this exact same timeout if the framework relies on the same flawed kickoff logic.
Check the installed framework's turn-start contract. Ask the installed source what server event it waits on before it considers the turn started. The framework was written for a text-first provider, whereas speech-to-speech providers do not reliably emit generation-created events for client-initiated text kickoffs.

Generalizing the State Machine Mismatch

This failure class generalizes to almost any agent framework that kicks off the first turn on your behalf. Ask two questions of your installed source: what content does it inject to start the turn, and what server event does it wait on before it considers the turn started.

A framework designed for text-first APIs assumes that pushing text into a session will reliably trigger a symmetrical server event. Speech-to-speech providers often treat text inputs, audio inputs, and control messages through distinct pipelines with different event guarantees.

I time each explicit reply request from the initial call down to the first agent audio frame, and I track that metric completely separately from the provider's event acknowledgment. When the audio frame latency remains normal but the provider acknowledgment metric spikes to a hard timeout, the metrics prove the audio pipeline is active while the event machine is stuck.

Common questions

Why does my voice agent transcript start with a single dot?

The framework injected a one-character text placeholder to force the model to start the conversation. Because it goes through the public turn API, it appears as a user message in your transcript.

Why does generate_reply time out when the agent is actually speaking?

The framework awaits a generation-created event from the provider to resolve the reply future. For a client-initiated kickoff, speech-to-speech providers often omit this event, leaving the future pending until it times out.

How do I force a speech-to-speech model to start the call?

Push an explicit XML-shaped control marker through the plugin's internal message channel instead of a punctuation character. Then, resolve your reply future as soon as the kickoff is enqueued.

Canonical URL: https://www.dreamchasers.lt/blog/livekit-agent-opener-answers-a-phantom-turn-and-reply-requests-time-out

Outbound Call Answers to Dead Air Before the Agent Joins

Rokas Remeika — Thu, 22 Oct 2026 12:23:03 +0000

By Rokas Remeika. Published 6 October 2026.

Getting the order of outbound placement wrong causes dead air, unresolvable webhooks, or leaked rooms. The fix is ordering three operations strictly. First, create the room with your custom routing metadata so webhooks can resolve the tenant. Second, dispatch the agent so the worker is fully joined before the phone rings. Finally, create the SIP participant to dial the number, using a 45 s ringing timeout. Wrap the entire flow in a block that catches any error after the room exists and explicitly deletes the room before returning.

Symptoms

The callee answers the phone but hears two seconds of dead air before the agent starts speaking.
Your webhooks arrive with a room identifier but no custom metadata, leaving you no way to tell which tenant or phone number the call belongs to.
A failed SIP dial leaves an empty room behind, and a worker sits idle in that room until the empty timeout expires.

What is actually happening

Outbound SIP placement in a realtime system requires three distinct operations: creating the room, dispatching the agent, and creating the SIP participant. The sequence you use to execute these operations is load-bearing. Getting the order wrong causes race conditions at the start of the call and leaks infrastructure when calls fail.

If you dispatch the agent after the SIP participant is created, the dial is already in flight. A carrier network can route and connect a SIP call very quickly. By the time the callee picks up the phone, the agent worker can still be starting. The greeting gate logic cannot help you here, because there is nobody in the room to evaluate the gate. The callee says hello into a void, waits two seconds, and then the agent abruptly joins and starts speaking.

Additionally, creating the room without metadata means the platform's native room and participant webhooks carry only platform identifiers. Platform native room and participant webhooks carry only platform identifiers. They know the internal room identifier, not your domain model. Any downstream service that attempts to resolve a tenant or phone number from a webhook event will fail because it has nothing to resolve from.

Finally, resource ownership is strict. From the moment a room exists, you own it. Any failure after that point requires explicit cleanup. If you experience a placement error, a classification failure, or an unexpected exception before the dial succeeds, that room is left hanging. A worker will occupy it and wait for activity until the default empty timeout expires. This leaks a room that a worker will occupy until the empty timeout expires.

How to confirm it

Log the time difference between the agent dispatch creation and the worker joining the room, then compare it against the time between SIP placement and the callee answering. Expect the dispatch-to-join duration to be strictly shorter than the placement duration. If the worker joins after the answer event, your placement order is causing the dead air.
Query your webhook receiver logs for events that fail to resolve a tenant or session identifier. Expect to find these unmatched events correlated with outbound calls where the room was created without metadata.
Search your application logs for room creation events without a corresponding room deletion event on a failed call branch. Expect to find leaked rooms that remained open until the generic worker empty timeout finally destroyed them.

How it works

The fix is to enforce a strict ordering for resource creation and to establish explicit ownership over the room lifecycle. I organize the outbound placement pipeline to always execute in this exact sequence.

First, I create the room explicitly. This is the only way to carry the routing metadata the webhooks will need. By passing a JSON payload containing the tenant identifier, session identifier, and destination number into the room creation request, every subsequent webhook emitted by the platform will include this context. Your webhook handlers no longer need to hit a database to figure out who the call belongs to.

Second, I create an explicit agent dispatch. Initiating the dispatch immediately after the room is created ensures a worker is assigned and joins the room before the phone ever rings. The agent is already sitting in the room, waiting for the media tracks to arrive.

Third, I create the SIP participant. I configure this with wait-until-answered enabled, dial tone playback on, and a 45 s ringing timeout. I also assign a deterministic participant identity derived directly from the application session id. This means every later event involving this participant can be correlated directly back to the database row without any secondary lookups.

Crucially, I wrap everything after the room creation in an error handling block. Because the system acquires ownership of the room in the first step, every failure branch must delete the room before returning the error. If the agent dispatch fails, if the SIP participant rejects the dial, or if an internal API throws an exception, the catch block intercepts it, issues a room deletion request, and then bubbles the error up.

What I would check first

Check the timestamp of your agent dispatch request against your SIP participant creation request. This discriminates between an initialization race and a media subscription delay. If the dispatch request fires after the SIP creation request, the agent is late to the room and you must fix the ordering. If the dispatch happens first, the dead air is likely an audio subscription timing issue.
Inspect the metadata field on the earliest webhook event you receive for a new call. This isolates whether you are relying on implicit or explicit room creation. If the metadata field is empty, the room was created implicitly by the platform and you must switch to explicit room creation to pass your domain identifiers.
Search your application logs for room creation events on failed calls and check whether a deletion event was emitted. This discriminates between an unhandled exception during placement and an orphaned worker caused by a downstream teardown failure. If the logs show a room being created but no subsequent deletion log within the same request lifecycle, your error handling is leaking infrastructure.

Why the greeting gate cannot save a late agent

You might already have logic that gates the initial greeting to prevent it from playing into a cold connection. I use a 2.5 s answer grace window combined with an audio track subscription trigger to solve the problem where an outbound call greets before the callee's audio is subscribed. However, that gate solves a signaling issue, not an agent lifecycle issue.

If the SIP participant is created before the agent dispatch, the agent process is simply not running when the SIP leg connects. The greeting gate logic lives inside the agent code. If the agent is not in the room yet, it cannot evaluate track subscriptions, it cannot enforce a 2.5 s grace period, and it cannot observe the active signaling state. By the time the agent finally joins, the initial SIP events have already passed. Because the worker is still starting when the callee picks up, no agent is present to execute the greeting gate. This results in the callee hearing two seconds of absolute silence before the greeting suddenly fires.

Deriving identity for deterministic correlation

Generating identifiers at the correct boundary reduces operational overhead when debugging a failed call. If you allow the realtime platform to generate the participant identity for the SIP leg, you are forced to store that identifier and use it as a lookup key for all future webhook events.

Instead, I derive the SIP participant identity directly from the application database session id. During the third step of the outbound placement pipeline, I format the session id into the participant identity field. When a webhook arrives reporting that the SIP participant disconnected, the webhook payload includes that deterministic identity. The webhook receiver correlates the event directly to the session without an external lookup. It never needs to query a mapping table.

Protecting capacity with strict teardown

An unhandled exception during outbound placement might seem like a minor transient error, but in a realtime voice stack, it directly impacts capacity. A room created without a SIP participant is an active resource rather than an inert record. It actively holds a worker assignment.

When the agent dispatch succeeds but the SIP dial fails, the worker process is commanded to join the room. The worker joins, observes that it is alone, and waits for instructions. Without explicit cleanup, that worker will sit idle until the platform's empty room timeout eventually evicts it. If a misconfigured destination number causes a burst of placement failures, you can exhaust your entire worker pool with idle agents sitting in leaked rooms.

Treating resource creation as an acquisition of ownership solves this. The function that requests the room must guarantee its destruction on every exit path.

Common questions

Why does the person hear dead air when answering my outbound call?

The LiveKit agent was dispatched after the SIP dial was initiated. By the time the person answered, the agent process was still initializing and not yet in the room to speak.

Why are my LiveKit webhooks missing the tenant metadata?

The room was created without metadata. You must explicitly create the room first and pass your domain metadata in the creation request.

Why are empty rooms staying open after failed outbound dials?

An error occurred during outbound placement after the room was created, but before the dial succeeded. You must catch these placement errors and explicitly delete the room.

If the boundary this note describes is one you would rather own, a self-hosted deployment you control moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/outbound-call-answers-to-dead-air-before-the-agent-joins

Voice Agent Prompt Change Improved One Behavior but Broke Another

Rokas Remeika — Tue, 20 Oct 2026 12:08:35 +0000

By Rokas Remeika. Published 4 October 2026.

A system prompt change that improves one voice agent behavior will almost always trade away attention from another. You only see this trade if you measure the untouched behavior in the same run, against a threshold written before you see the data. To stop regressions from reaching production, register a falsifier for every hypothesis before you run it: a specific metric that kills the change if it drops, regardless of how good the headline number looks. In my own environment, this pre-registered falsification forced me to revert three of my own prompt fixes before they shipped.

Symptoms

A system prompt change measurably improves the specific behavior you were targeting.
The change ships to production because the headline metrics look positive.
Something else the agent used to do reliably quietly stops working.
Nobody notices the regression until a live call goes badly in a way that has nothing to do with the original change.
The misleading observation: local testing of the targeted scenario shows a clear win, tricking you into believing the prompt change was an isolated fix with no side effects.

What is actually happening

Behavioral edits to a voice agent trade failure classes against each other. When you add instruction text to tighten one specific behavior, that new text spends the model's attention. That attention is finite, and it was previously holding up another behavior elsewhere in the conversation.

The model has no way to tell you which behavior it dropped to accommodate your new rule. You only see the trade if the behavior you did not touch is measured in the exact same run. That measurement must be evaluated against a threshold written before the run even started, on a sample size large enough that a real move is distinguishable from statistical noise.

If you only score the agent on the problem you are trying to fix, every prompt expansion looks like a success. The regression hides in the unmeasured branches of the conversation state machine.

How to confirm it

Run the new prompt and the old prompt against a baseline regression suite of at least a hundred recorded calls using a synthetic caller. Expect the target metric to improve; expect at least one unrelated metric to shift.
Compare the failure rates of mandatory agent behaviors (like a required decline or a strict data-collection step) between the old and new prompt. Expect the failure rate on the unrelated mandatory step to increase.
Score the new prompt against an adversarial arm targeting behaviors adjacent to the change, not just the change itself. Expect the adversarial arm to expose edge cases the new prompt instruction inadvertently opened up.
Check the timing of your threshold decisions. Ask the team when the passing score for the run was decided. Expect to find that the threshold was agreed upon after the data landed, which confirms the evaluation was contaminated by the desire to ship the fix.

The fix

To stop these silent regressions, I stopped treating prompt changes like isolated bug fixes. For every hypothesis, I registered a falsifier first. This was a specific measurement that killed the change if it moved the wrong way, regardless of how good the headline number looked.

Then, I ran a matrix large enough to detect the regression. Here, that was 123 scripted conversations driven by the synthetic caller. The synthetic caller joins as a real RTC peer and speaks real audio, with no mocks in the media path. The synthetic caller joins as a real RTC peer to drive scripted dialogue against the agent without mocks in the audio pipeline. I repeated each of the 123 conversations three times against a pristine control, scored by the same evaluator with full denominators.

Alongside this main matrix, I added a 48-run adversarial arm aimed specifically at the behaviors most likely to be collateral damage from the prompt edit.

Following this strict methodology, three separate prompt-level fixes were designed, built, measured, and reverted by their own pre-registered falsifiers. A fourth was built and also falsified.

The most instructive hypothesis improved the behavior it targeted from 20 bad cases in 123 down to 16. The headline metric was an obvious win. However, it completely destroyed the agent's ability to decline an invalid request. A scenario where the agent is strictly required to refuse fell from 6 out of 6 successful refusals to just 2 out of 6.

Because I had registered the falsifier beforehand, the revert was forced by that stage's own logic. It was not argued afterwards in a meeting. The change died in testing, and the production agent was saved from a catastrophic behavioral regression.

What I would check first

Check the sample size of the evaluation run. If the run relies on an unrepeated sample rather than a repeated evaluation matrix, rule out the run as statistical noise. You need a repeated matrix of 123 conversations across three repeats to reliably detect regressions.
Check the scope of the scenarios tested. If the evaluation only included the scenario the prompt tweak was meant to fix, rule out the test as a complete measure of agent health. The regression is hiding in the paths you did not test.
Check whether the apparent regression stems from transcript scoring brittleness rather than agent behavior. If the team picked the passing threshold after looking at the numbers, rule out the evaluation as objective. Verify the failure against raw transcripts before concluding the prompt logic regressed.
Check the denominator of the scoring logic. If the test only measures the count of successful passes without measuring the total count of valid attempts, rule out the metric as a reliable indicator. You need full denominators to know if a behavior actually improved or if the agent just avoided the scenario.

The illusion of isolated prompt edits

In a traditional codebase, a localized patch to a pure function has isolated effects. You write a unit test for the function, verify the output, and ship the patch.

A system prompt is not a pure function. It is a shared, globally weighted context window. Every word you add shifts the relative importance of every other word.

This is why traditional unit testing methodologies do not map to voice agents. The entire persona, including its constraints and refusal mechanisms, is recomputed on every single audio turn. Testing one branch requires re-verifying the structural integrity of every other branch.

Structuring the evaluation matrix

Catching these cross-pollinating failures requires a specific testing topology. A handful of transcribed text chats will not expose the regression. You need an evaluation matrix built for scale and objectivity.

In my environment, this meant building a synthetic caller that actually negotiates WebRTC media and speaks real audio into the agent's microphone track. The agent does not know it is talking to a test harness. The matrix executes 123 synthetic-caller conversations. Because LLM behavior is non-deterministic even at low temperatures, a single pass is insufficient. Every call in the matrix is repeated three times to establish a reliable failure rate rather than a single anecdotal pass or fail.

The critical component is the pristine control. Every new prompt is scored simultaneously against the unmodified production prompt. The scorer evaluates both arms using the same deterministic criteria. If the treatment arm improves the targeted behavior but the control arm retains a higher success rate on mandatory refusals, the falsifier triggers.

The psychology of the pre-registered revert

If you wait until after the evaluation run finishes to decide what an acceptable regression looks like, human nature will compromise the agent.

The pre-registered falsifier removes this negotiation entirely. By writing down the exact metric that kills the change before the run starts, you bind the team to the data. It shifts the discussion from "is this regression acceptable" to "the prompt failed the contract, try another approach". It is the strongest available answer to a client asking how they know your changes will not break their live agent: because three of mine did, and the falsifiers killed them before they shipped.

Common questions

How do I know if a prompt change broke unrelated behaviors?

You run the new prompt against a regression suite of synthetic calls that test behaviors you did not touch. If the failure rate on those unrelated paths increases, the change caused a silent regression.

What is a pre-registered falsifier in prompt testing?

It is a specific metric and threshold written down before you evaluate a prompt change. If the run fails to meet this threshold, the change is automatically discarded regardless of other improvements.

Why did my voice agent stop declining invalid requests?

Adding new instructions to the prompt diluted the weight of the instruction telling the agent to decline. The model traded away the refusal behavior to spend attention on your new rule.

Why is testing the targeted scenario not enough?

Because system prompts are not isolated functions. Changing one rule shifts the weight of all other rules, meaning regressions will hide in the conversation paths you chose not to measure.

Canonical URL: https://www.dreamchasers.lt/blog/voice-agent-prompt-change-improved-one-behavior-but-broke-another

Deterministic Inbound SIP Routing When Tenants Share A Phone Number

Rokas Remeika — Sun, 18 Oct 2026 11:45:39 +0000

By Rokas Remeika. Published 2 October 2026.

When multiple tenants share an inbound phone number, the caller's phone number is the only routing key available. If a caller exists in multiple accounts, a database query that takes the first match will route the call non-deterministically based on index order. The fix is to gather every matching record across all tenants assigned to the dialed number, then sort them using an explicit four-level tie-break chain ending in the unique record ID. Writing a routing decision log for every outcome makes these transient decisions auditable days later.

Symptoms

A caller dials your shared inbound number and reaches the wrong tenant's voice agent.
The routing changes over time: the same person calls from the same device an hour later and reaches a completely different record.
A caller is rejected and disconnected, but you verify later that a valid record for them exists in the datastore.
When a complaint arrives days later, there is no way to reconstruct which records existed at the time or why one of them was chosen. The raw E.164 numbers are usually correct, but the application is guessing what to do with them.

What is actually happening

When a call arrives from the carrier over a raw SIP trunk, the INVITE contains two critical pieces of routing information. The To header tells you which number was dialed (the shared DID). The From header contains the caller's phone number. On a multi-tenant platform where several tenants share a single inbound number, the caller's number is the only routing key you have.

The failure occurs because caller numbers are not globally unique to a single record. The same person might exist as a customer under two different tenants assigned to that shared number. They might even have multiple records within a single tenant (for example, an active profile and an archived one).

If your routing logic executes a database query that stops at the first match, you are delegating the routing decision to the database storage engine. A query without a strict total order will return records based on index traversal order, page boundaries, or insertion history. This behaves non-deterministically. The database might return tenant A today and tenant B tomorrow, simply because a background process rewrote a page on disk. The symptom is invisible in standard application logs because the application correctly routes the call to the record it was handed.

Because the interesting information is the set of matches at that exact instant, the context is gone by the time anyone asks. When a complaint arrives days later, there is no way to reconstruct which records existed at the time or why one of them was chosen.

How to confirm it

Query your datastore for the caller's number across all tenants sharing the dialed DID. Expect to find more than one active record; this confirms a routing collision is mathematically possible.
Review the routing logic handling the inbound INVITE. Expect to see a simple query limit or a first-match ORM fetch rather than an exhaustive gather-and-sort operation.
Ask the database for the same caller number multiple times from an interactive shell while simulating background writes to those records. Expect the returned record to occasionally swap, proving the query ordering is unstable.

How it works

The solution requires taking the routing decision out of the datastore's hands and enforcing a strict mathematical total order in application memory.

First, gather every matching record for the caller across every tenant assigned to the dialed number. You must retrieve the entire set of potential matches.

Once all matching records are in memory, rank them with an explicit, documented tie-break chain rather than letting the datastore decide. I settled on a four-level deterministic ranking chain:

Status first. An active or expected record always outranks a stale one.
The expected-call timestamp. A record with an expected-call timestamp outranks records without one.
Creation time. Creation time acts as the third tie-breaker before the final unique record ID.
Record ID. This is the final total order. By ending with a globally unique identifier, the ranking guarantees a deterministic outcome even when everything else ties.

After sorting, the system routes the call to the top record. However, routing is only half the fix. You must write a routing decision record for every outcome: matched, ambiguous, and rejected.

An ambiguous outcome means that more than one plausible record survived the initial status ranking, forcing the system to rely on timestamps or IDs to break the tie. The caller still connects, but the ambiguity is permanently logged. A rejected outcome means nothing matched, or the caller is not permitted.

Because inbound complaints are always retrospective, you need these records to reconstruct any single call afterwards. Caller numbers must be hashed before they reach any log to protect privacy.

What I would check first

Find the inbound INVITE in your SIP trace and extract the exact From header. Expect the number to match records across multiple tenants sharing the dialed DID, confirming that the caller ID is not a unique routing key.
Search your application logs for that exact caller number hash at the time of the call. Expect to see whether the application found zero records or multiple records. Finding zero records indicates the caller had no matching record or was not permitted, whereas multiple records confirm a routing collision.
Check the database indexing for the table holding the caller records. Expect a composite index on the shared DID and the caller number. If the query fetches a single record without an explicit order clause, the datastore returns records based on storage index traversal and page boundaries.

Treating rejections as system defects

When a call is rejected because the system finds no matching record, the caller is disconnected. In a multi-tenant environment, a rejection represents a dropped interaction.

I review rejections that turn out to have had a valid record as system defects. If a tenant reports that a known customer called in and was rejected, the routing decision log is the only way to prove what happened. The decision record provides the audit trail needed to prove whether an active record existed when the caller was rejected.

Generalizing across shared identifiers

Any multi-tenant inbound channel keyed by a shared identifier has this problem. Whether you are operating a shared phone number, a shared email inbox, or a shared SMS short code, the mechanism remains identical.

Two rules carry over directly. First, make the ranking a total order, ending in something globally unique, so the answer never depends on storage internals. Second, log the decision, not just the result. You cannot re-run yesterday's set of matches, so the system must freeze its exact reasoning at the moment the event arrived.

Splitting numbers based on ambiguity

Shared numbers work perfectly until the user bases overlap too heavily. The failure mode of a shared number is ambiguity.

Because I record the ambiguous outcome explicitly on the decision record, I can aggregate these over time. Repeated ambiguity on one number is the leading indicator that a shared DID needs splitting. When a specific tenant begins generating constant collisions, you have concrete data justifying the migration to a dedicated number.

Common questions

Why does an inbound call route to the wrong tenant?

When tenants share a phone number, the caller's number is the only routing key. If the caller exists in multiple tenants, a basic database query returns the first match it finds, which depends on unpredictable disk indexing.

How do I make inbound call routing deterministic?

Fetch all matching records into application memory and apply a strict sorting chain. End the sort with a globally unique record ID so the answer never depends on storage internals.

Why log routing decisions if I already log errors?

Inbound complaints arrive days after the call. The datastore state will have changed, so you cannot re-run the query to see what matches existed at the time. A routing decision record freezes that instant.

If the boundary this note describes is one you would rather own, a self-hosted deployment you control moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/deterministic-inbound-sip-routing-when-tenants-share-a-phone-number

Agent Audio Cuts Off When Calling A Function Tool In The Same Turn

Rokas Remeika — Fri, 16 Oct 2026 12:02:42 +0000

By Rokas Remeika. Published 30 September 2026.

On a speech-to-speech model, a single turn produces one atomic generation. Calling a function tool terminates that generation early, which truncates the concurrent audio or stops it from being emitted at all. No framework guards against this because the SDK does not know your specific tool's semantics. The fix is to add an explicit named rule to the tool's description instructing the model to call the tool in a turn containing no speech. This places the constraint exactly where the model makes the decision, avoiding the context decay of the system prompt.

Symptoms

The agent said a line and the caller heard nothing. Sometimes, the caller heard just the first syllable of a word, followed instantly by silence. The issue appeared intermittent and unrelated to the tool, because the rest of the conversation turns functioned normally.

However, the failure isolated to very specific interactions. It happened only on the turns where the model also called a function tool.

The most misleading observation was the text transcript. When I pulled the session logs, the output text transcript was complete. It showed that the agent generated the correct phrase alongside the tool call. The logs indicated the turn completed and the tool fired, masking the dropped audio from automated log filters. The model had output the text transcription internally, but no corresponding audio reached the caller.

What is actually happening

On a speech-to-speech model, reasoning, text generation, and audio synthesis operate together behind a single provider connection. A turn produces exactly one generation.

When the model decides to invoke a tool, it emits a function call token. That function call terminates the generation phase for that turn. If the model was synthesizing audio at the same time, the audio generation is abruptly cut off. The audio currently in flight is either truncated mid-word or flushed completely before it ever reaches your infrastructure.

In a speech-to-speech architecture, audio generation cannot be decoupled from the model's turn. When the model invokes a function tool, the single generation for that turn terminates immediately. Audio synthesis and model reasoning share a single execution lifecycle on the provider.

In a speech-to-speech model, there is no text-to-speech plugin running on your server. Hearing, reasoning, and speaking sit behind one model and one websocket. The provider handles the audio synthesis internally. When the model makes a choice to emit a tool call, that action signals the end of its response phase. The provider stops synthesizing audio immediately. The audio track is cut off at the source. If the model generated full text internally but only synthesized one syllable of audio before calling the tool, that single syllable is all that arrives over the network.

No framework can guard against this for you. The LiveKit SDK receives the media stream exactly as the provider sends it. If the provider terminates the generation, the SDK stops receiving audio frames. The SDK does not raise an error because it has no way of knowing this was unintended. Whether this truncation matters depends entirely on your tool's semantics. Nothing in the SDK knows that this particular tool is fatal to concurrent speech.

How to confirm it

Log every turn that contains both a tool call and generated audio. Expect to see this exact combination exclusively on the turns where the audio dropped. That combination is the defect.
Score whether the intended speech for a tool-calling turn appears in the agent's own output transcription for that turn. Expect the text transcript to contain the intended phrase, confirming the model attempted to speak before the generation terminated.
Put a human on the line to trigger the specific tool, and listen to the output. Expect the human to hear silence or a clipped syllable. Confirming the audio itself reached the caller needs a human on the line, since there is no recording in the realtime media path and no second recognizer to check the final audio output for you automatically.

The fix

To solve this, I changed how I prompted the tools. The model has complete control over when a tool is invoked. Your code cannot force the model to speak first and call the tool later. The only enforcement surface you have is the text the model reads during its generation phase.

I put the turn-level contract directly into the tool description. I added an explicit, named rule stating that the tool must be called in a turn containing no speech.

I did not put this rule in the system prompt. In a system prompt, behavioral constraints compete with persona instructions, safety guardrails, and the growing conversation history. As the context window fills, a system prompt rule decays and the model forgets it. Function-tool descriptions avoid this because they are the most reliable per-tool instruction channel a model has. They are read in the exact context where the model evaluates its available tools and makes the decision. Anything the model must do around calling a tool belongs in the description.

I gave the rule a specific name, like RULE_SILENT_TURN, so I could search transcripts and prompts for it later. Without a name, you cannot easily query your logs to see if the model even acknowledged the constraint.

Once the description was updated, I verified it behaviorally on real runs. A tool description is a request, not a guaranteed constraint. The model can still fail to isolate the turn. I set up my evaluation runs to score whether the intended line appeared in the agent's own output transcription for that turn, rather than just checking whether the tool fired. Scoring the tool execution tells you nothing about the audio failure. You must score the turn isolation.

That is as far as an automated run can take you. Automated scoring confirms that the model emitted the transcript text, but cannot detect whether synthesised audio was dropped before playout. Live listening on test calls remains the only verification that the provider emitted the audio track to the caller.

What I would check first

Check the event sequence in your provider logs for the failing turn. Expect the function call event to arrive in the exact same generation block as the terminated speech. This confirms the generation was killed by the tool call, ruling out a network disconnection.
Check if the tool in question tears down the session. If the tool is designed to hang up, you are dealing with a lifecycle race rather than just a generation limit. A function tool that tears the session down ends the call while the closing audio is still being generated or played out. If this is the case, defer the teardown to the next caller turn boundary. I have documented this specific failure in /blog/voice-agent-end-call-tool-truncates-the-final-sentence-or-loses-the-callers-last/. This observation discriminates between a standard silent turn and a teardown race.
Check whether the silent turn coincided with a websocket reconnect edge rather than a function tool execution. Expect a reconnect edge to leave both sides waiting silently because in-flight generation died with the dropped connection, which points to a missing reconnect redrive watchdog rather than a tool-turn defect. If the failure occurs on clean transport without reconnect events, you have ruled out transport drops and isolated the failure to concurrent tool calling.

Common questions

Why does my voice agent transcript show text but play no audio?

The model generated text but terminated the turn before synthesizing the audio. This happens on speech-to-speech models when a tool call interrupts the generation phase. The audio being synthesised is truncated or never emitted at all, leaving the text intact while the caller hears silence.

Can I fix silent tool turns in the system prompt?

No, because system prompt instructions decay as the conversation history grows. The rule competes with persona constraints and loses attention weight. You must place the constraint directly in the tool description, which is evaluated at the moment the model chooses to call the tool.

How do I know if the model ignored my tool calling rule?

You must verify the turn behaviorally on real runs by scoring the text transcripts. Score whether the intended line appears in the output transcript, and verify the tool-calling turn contains zero speech tokens. Scoring whether the tool fired is not enough.

Does the LiveKit SDK catch aborted audio generations?

No, because the SDK has no way to know if concurrent speech with a specific tool is an error. The SDK simply stops receiving audio frames when the provider terminates the stream. You must handle this constraint at the model prompting layer.

Canonical URL: https://www.dreamchasers.lt/blog/agent-audio-cuts-off-when-calling-a-function-tool-in-the-same-turn

Opaque RPC errors hide the SIP response codes you need for outbound call retries

Rokas Remeika — Wed, 14 Oct 2026 12:24:41 +0000

By Rokas Remeika. Published 28 September 2026.

When an outbound SIP call fails, the client frequently throws an opaque RPC error string that hides the underlying SIP response code. Without that code, you cannot distinguish a busy signal from a disconnected number, and your retry policy collapses. To fix this, build a cycle-safe recursive function that walks the error's metadata and trailers to recover the three-digit protocol code. Map that code to a closed set of typed outcomes (such as no-answer, busy, or invalid number). Pass only the typed outcome to your retry logic so it can safely retry busy numbers without hammering dead ones.

Symptoms

Failed outbound placements all surface as the same opaque RPC error string in your application logs.
You cannot separate a normal "they were busy" condition from "the number is dead" or "our trunk credentials broke".
Your task queue retry policy collapses into an all-or-nothing toggle.
If you configure the queue to retry everything, you end up hammering disconnected numbers and burning your carrier relationship.
If you configure the queue to retry nothing, you drop calls that would have easily connected on the second attempt.

What is actually happening

When a carrier rejects or fails to complete an outbound call, they respond with a standard three-digit SIP status code. However, the RPC client that your application uses to interface with the calling platform frequently drops this protocol code from the top-level exception it throws.

The SIP response code still exists, but it lives buried in the RPC error's metadata or trailers. It appears under one of several possible key spellings, depending on the specific trunk and the vendor routing the call. Sometimes the code is nested a level or two down inside a dictionary. Sometimes it is only embedded within a human-readable text message string.

Without recovering this underlying protocol code, your application is forced to classify failures by running substring matches against the error message text. This approach degrades rapidly because the wording of an error message is prose. A vendor is completely free to change their error text without telling you, which silently breaks substring matching and causes recoverable calls to be dropped.

How to confirm it

Inspect the raw error object from a failed placement attempt. Log the exception's top-level attributes and inspect its metadata and trailers dictionaries directly. Expect to see a generic top-level exception message, but a dictionary of trailers or metadata containing the real SIP code buried a level or two down.
Trigger a deliberate auth failure by changing a working trunk credential to an invalid string, then attempt a call. Expect your system to log the exact same top-level RPC error type as it does for a busy signal. This confirms your current exception handling cannot tell an auth failure from a busy callee.
Search your logged exception strings for three-digit numbers. Expect to find protocol codes like "486" or "404" embedded in sentences like "Call failed with SIP response 486". This proves the protocol data is present but arriving completely unstructured.

How it works

To build a working retry policy, you have to recover the underlying protocol code first, map it to a small closed set of kinds you own, and let only that typed kind reach your retry logic. Codes are a spec. Strings are prose.

I handle this by writing a recursive function that walks the error's metadata and trailers. This walk must be depth-limited and cycle-safe, utilizing a visited set. The exception objects thrown by these RPC clients frequently hold self-references (such as an exception pointing to its own cause). A naive recursive walk will immediately hang the process.

Inside the walk, I probe the object for plausible key spellings rather than relying on one canonical name. The useful detail is not in the same place on every trunk.

If the key lookups fail to find a code, I fall back to running a regular expression over the human-readable message. This regex requires a "sip" proximity anchor. I look for the word "sip" near a three-digit number, rather than just matching any three digits in the text. Without the anchor, a three-digit area code that happens to be present in the error string will be mistakenly harvested as a SIP response code, leading to bizarre retry behavior.

Once I successfully extract the three-digit code, I map it onto a small closed set of typed kinds, and I branch my logic on the kind. The mapping handles the distinct failure domains:

408, 480, and 487 map to no-answer. I return this to the queue with instructions to retry later.
486 and 603 map to busy. I instruct the queue to retry this later, but with a significantly shorter backoff time than a no-answer.
404 and 410 map to invalid-number. This is a terminal failure. The system stops calling the number immediately.
5xx series codes map to carrier-reject. I retry these, but I escalate if the condition is sustained across multiple calls.
401 and 403 map to auth-failure. This is terminal for the entire queue run. It represents a human problem where the trunk is broken, not the destination number. Continuing to dial under a 401 just wastes compute.

Isolating retry decisions to this small closed set ensures carrier wording changes cannot alter placement scheduling. Pre-placement markers prevent retrying tasks from double-dialing in-flight calls while classification evaluates earlier attempts. If you are struggling with retries that hit the same customer twice, you can read more about how I solve that in my post on /blog/task-queue-retry-called-the-customer-twice-and-an-exists-check-will-not-stop-it/.

What I would check first

Count your unmapped errors by string across a batch of failed placements. A high volume of identical strings arriving at your fallback logic rules out random network drops. It points directly to a missing key spelling in your scraper that you need to add to the probe list.
Cross-reference a specific failed placement in your application logs with the logs provided by your carrier. If the carrier logged a 486 Busy but your application logged an opaque RPC error string, it rules out a carrier routing issue and confirms the RPC layer is masking the response.
Check whether unmapped errors contain status codes in the message text rather than the metadata. If the message contains a phrase like 'SIP 486' but metadata extraction returned nothing, your key probe list missed the trailer spelling and you need the proximity-anchored regex.

Instrumentation and observability

I record the typed kind on every single failed placement. This allows failures to be counted by kind rather than by string. Grouping errors by busy or invalid-number gives a much clearer picture of list quality than reading through hundreds of slightly different exception sentences.

I log every failure where no SIP code could be recovered by any path. Each one is a sign the scraper needs another key spelling. A new carrier or an updated RPC library version will eventually introduce a new metadata shape, and logging the unmapped errors is how you catch the change before it impacts your retry metrics.

Finally, I record terminal classifications directly against the target phone number in the database. If a systematic misclassification occurs in the pipeline, it shows up as a dense cluster of terminal states against otherwise healthy numbers, allowing me to flag the run and investigate the carrier response.

Common questions

Why is the SIP code missing from the error message?

RPC clients often encapsulate carrier responses inside generic exceptions. The specific protocol code is usually pushed down into the exception's metadata or trailer dictionaries, leaving only an opaque string at the top level.

How do I tell if a call failed because the user was busy?

Extract the SIP response code from the error metadata. A 486 or 603 code indicates a busy signal, which you should classify as a temporary failure and retry with a short backoff.

Why shouldn't I just search the error string for a 404?

Blindly searching for a three-digit number will cause false positives if the target phone number or request ID happens to contain 404. You must use a proximity anchor like the word 'sip' to ensure you are matching an actual status code.

What should happen if I get a 401 or 403 on an outbound call?

Treat it as an authentication failure. This is terminal for the current run because your trunk credentials are broken. Retrying the number will not fix a trunk configuration problem.

If the boundary this note describes is one you would rather own, moving this layer off LiveKit Cloud moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/opaque-rpc-errors-hide-the-sip-response-codes-you-need-for-outbound-call-retries

Voice Agent Goes Silent After A Websocket Reconnect

Rokas Remeika — Thu, 08 Oct 2026 14:03:26 +0000

By Rokas Remeika. Published 24 September 2026.

A reconnect restores the transport connection, not the conversational turn. Whatever generation was in flight when the session dropped dies with the old connection. Afterwards, the model waits for input it believes it already consumed, and the caller waits for a reply that no longer exists. Because silence is a stable state, no timeout fires and no error is raised. You fix this by building a watchdog that detects the reconnect edge and issues an explicit reply request to the provider. When building this detector, you must latch on the observed absence of the session to avoid the ABA polling trap.

Symptoms

The session reconnects successfully and then nothing happens. The room is alive, audio frames are flowing both ways, the caller says something and the agent does not answer. No error is raised and no timeout fires.
Around ten minutes into a successful call, the agent spontaneously restarts the conversation. It re-introduces itself, asks its opening question again, and has no memory of anything already discussed. The call never dropped.
The caller interrupts the agent mid-sentence. The agent stops correctly, and then never speaks again. About 90 s later, something else fires and the call either recovers or ends. During this wedge, frames flow in both directions and nothing has errored.

What is actually happening

These failures all stem from a mismatch between transport health and turn state. A realtime connection can be perfectly healthy while the conversation sitting on top of it is entirely dead.

When a websocket connection drops and reconnects, the plugin restores the transport. It does not restore the turn. Whatever generation was in flight when the network dropped died with the old connection. Once the transport is back up, neither side is driving the conversation. The model is waiting for input it believes it already consumed. The caller is waiting for a reply that no longer exists. The system is structurally idle, which is why no component throws an error. Silence is a stable state, so it persists until something external breaks it.

The mid-call greeting is caused by provider session management. The provider closes long realtime sessions with a server-initiated session-recycle notice on a roughly ten-minute cadence. The plugin reconnects, but unless session continuation is configured in the setup frame, the reconnect opens a brand-new session with no prior context. The system instructions are still there, so the model does the only thing a cold start allows: it greets. To the caller this reads as the agent forgetting the entire conversation. To the logs, it reads as a completely successful reconnect.

The post-interruption wedge is a failure in generation finalisation. The receive path is gated on the current generation reaching a terminal state. On a caller barge-in, the provider stops generating audio, but the turn-complete event for that generation does not always arrive. The plugin keeps treating the dead generation as in flight. Because the system thinks a response is still active, inbound caller audio is never turned into a new generation request. Nothing errors because the client is simply waiting, and waiting is exactly what it is programmed to do.

How to confirm it

Inspect the setup frame dispatched by the client during the websocket connection phase. Look for the session continuation configuration block. If it is absent, expect the agent to re-greet the caller upon the first session recycle.
Log the generation open-to-terminal duration per turn, and count generations that never reach a terminal state. Expect these permanently open generations to correlate directly with the calls that fall silent immediately after a caller barge-in.
Record the time from a reconnect edge to the first agent audio on each call. This makes a run with the watchdog off directly comparable to one with it on. Expect to see an infinite gap where the transport is connected but agent audio never follows, confirming the turn is dead.

The fix

For the reconnect silence, you need a watchdog that detects the reconnect edge. If no agent audio follows within a defined bound, the watchdog issues an explicit reply request to restart the turn.

For the mid-call context loss, configure session continuation in the setup frame. The provider will issue a continuation token the reconnect can present, allowing the reconnect to carry on the same conversation rather than starting a new one. On the plugin version I run, that configuration had to be re-added at the setup-frame patch point because the plugin was not sending it. Then, handle the edge case where the provider rejects the continuation. When it reports that the continuation did not take, clear the per-session state you were carrying and re-drive the conversation deliberately rather than letting the model improvise a second opener.

For the barge-in wedge, implement a finalisation watchdog. This watchdog tracks the open generation and the time since its last activity. If the generation remains open with no activity past a bound, the watchdog finalises it locally and releases the gate. Keep this decision logic as a pure function of timestamps so the boundary cases are unit-testable.

What I would check first

I checked if the silence immediately followed a caller interruption. Because it did, I ruled out a network drop and investigated the finalisation state machine for a missing turn-complete event.
I checked the call duration when the context reset. Because the agent re-greeted consistently around the ten-minute mark, I checked the setup frame for a missing session continuation token.
I checked the state-change detection logic in the supervisor loop. Because the reconnect detector compared the current session handle to the previous one, I rewrote the detector to fix the ABA problem.

The polling trap in reconnect detection

When building the reconnect re-drive watchdog, the obvious implementation is a trap. You poll the session handle to detect when it changes. The easiest way to write that detector is to check if the current handle differs from the last one seen. This passes every test, because in a test the reconnect happens once and stays.

Under a real recycle, the network drops and restores rapidly. The handle can go present, absent, and present again between two ticks of the supervisor loop. If the observable value on tick three is identical to the one on tick one, a differs-from-last detector concludes nothing changed. It never issues the re-drive, and the call just sits there in silence.

This is the ABA problem, and here it fails entirely silently. The fix is to latch on the gap. You must record that you observed the absence, and treat the next presence as a new session even when the observable value is identical to the one before it. This requires three lines of state management rather than a single equality check.

Being able to reason about this polling failure came entirely from having the reconnect on a fault injector. I had event logs with monotonic timestamps to replay against, which allowed me to step through the state machine ticks rather than reading the loop and hoping it worked.

Re-driving is not free

Resumption alone is not enough. A resumed session still needs someone to restart the turn, which is the entire purpose of the re-drive watchdog. If the model managed to resume on its own and you issue a re-drive anyway, the agent will talk over itself. The bound you set before re-driving has to be strictly longer than the provider's worst post-reconnect first-token latency. You have to measure that latency in production before choosing the number.

You also need to instrument the outcome. Emit an event for the reconnect edge, an event for the re-drive decision, and an event for the result. This instrumentation makes a re-drive that talked over a live model distinguishable from a re-drive that successfully rescued a dead call. Record the time from reconnect to first agent audio on each call so a run with the watchdog disabled is directly comparable to one with it enabled.

Tuning the finalisation bound

The finalisation watchdog requires setting a timeout bound on open generations, and that number is a real tradeoff in both directions. Make it too tight, and you cut off a model that is genuinely thinking or producing slowly. Make it too loose, and you have simply reproduced the 90 s wedge with extra steps.

I observed that when a generation's terminal event never arrives on real calls, something else eventually fires and ends the silence after approximately 90 s. Your local finalisation bound must be significantly shorter than that.

Testing this requires an injector. If you rely on organic barge-ins failing, you will never get enough data to tune the bound safely. You need a test harness that deliberately swallows the turn-complete event on a chosen conversational turn, so the wedge reproduces in seconds rather than waiting for a live incident.

Barge-in is the primary trigger for this failure, which means this bug concentrates on exactly the calls where the caller is most engaged. It is also completely invisible to any health check that looks at the room, the transport, or the participant, all of which remain perfectly fine throughout the wedge. The only observable that tells the truth is the time since the agent last produced audio, and you have to watch it explicitly.

Common questions

Why does the voice agent stop responding after a websocket reconnect?

The reconnect restores the network transport but abandons the conversational turn. The generation in flight died, so the model waits for input and the caller waits for a reply.

Why does the AI agent greet the caller again ten minutes into a call?

Realtime providers recycle long sessions around the ten-minute mark. If you do not configure session continuation in the setup frame, the reconnect opens a brand-new session and the model defaults to its greeting.

Why does the agent go silent for 90 seconds after I interrupt it?

The provider stops generating audio but fails to send the turn-complete event. The client waits for the dead generation to finish, blocking new input until something else fires roughly 90 seconds later.

How do I safely detect a reconnect to redrive the agent?

Poll the session handle and latch on the absence. If you only compare the current value to the previous one, you fall into the ABA trap and miss a reconnect that drops and restores between two ticks.

Canonical URL: https://www.dreamchasers.lt/blog/voice-agent-goes-silent-after-a-websocket-reconnect