DEV Community: Rokas Remeika

Agent Audio Cuts Off When Calling A Function Tool In The Same Turn

Rokas Remeika — Fri, 16 Oct 2026 12:02:42 +0000

By Rokas Remeika. Published 30 September 2026.

On a speech-to-speech model, a single turn produces one atomic generation. Calling a function tool terminates that generation early, which truncates the concurrent audio or stops it from being emitted at all. No framework guards against this because the SDK does not know your specific tool's semantics. The fix is to add an explicit named rule to the tool's description instructing the model to call the tool in a turn containing no speech. This places the constraint exactly where the model makes the decision, avoiding the context decay of the system prompt.

Symptoms

The agent said a line and the caller heard nothing. Sometimes, the caller heard just the first syllable of a word, followed instantly by silence. The issue appeared intermittent and unrelated to the tool, because the rest of the conversation turns functioned normally.

However, the failure isolated to very specific interactions. It happened only on the turns where the model also called a function tool.

The most misleading observation was the text transcript. When I pulled the session logs, the output text transcript was complete. It showed that the agent generated the correct phrase alongside the tool call. The logs indicated the turn completed and the tool fired, masking the dropped audio from automated log filters. The model had output the text transcription internally, but no corresponding audio reached the caller.

What is actually happening

On a speech-to-speech model, reasoning, text generation, and audio synthesis operate together behind a single provider connection. A turn produces exactly one generation.

When the model decides to invoke a tool, it emits a function call token. That function call terminates the generation phase for that turn. If the model was synthesizing audio at the same time, the audio generation is abruptly cut off. The audio currently in flight is either truncated mid-word or flushed completely before it ever reaches your infrastructure.

In a speech-to-speech architecture, audio generation cannot be decoupled from the model's turn. When the model invokes a function tool, the single generation for that turn terminates immediately. Audio synthesis and model reasoning share a single execution lifecycle on the provider.

In a speech-to-speech model, there is no text-to-speech plugin running on your server. Hearing, reasoning, and speaking sit behind one model and one websocket. The provider handles the audio synthesis internally. When the model makes a choice to emit a tool call, that action signals the end of its response phase. The provider stops synthesizing audio immediately. The audio track is cut off at the source. If the model generated full text internally but only synthesized one syllable of audio before calling the tool, that single syllable is all that arrives over the network.

No framework can guard against this for you. The LiveKit SDK receives the media stream exactly as the provider sends it. If the provider terminates the generation, the SDK stops receiving audio frames. The SDK does not raise an error because it has no way of knowing this was unintended. Whether this truncation matters depends entirely on your tool's semantics. Nothing in the SDK knows that this particular tool is fatal to concurrent speech.

How to confirm it

Log every turn that contains both a tool call and generated audio. Expect to see this exact combination exclusively on the turns where the audio dropped. That combination is the defect.
Score whether the intended speech for a tool-calling turn appears in the agent's own output transcription for that turn. Expect the text transcript to contain the intended phrase, confirming the model attempted to speak before the generation terminated.
Put a human on the line to trigger the specific tool, and listen to the output. Expect the human to hear silence or a clipped syllable. Confirming the audio itself reached the caller needs a human on the line, since there is no recording in the realtime media path and no second recognizer to check the final audio output for you automatically.

The fix

To solve this, I changed how I prompted the tools. The model has complete control over when a tool is invoked. Your code cannot force the model to speak first and call the tool later. The only enforcement surface you have is the text the model reads during its generation phase.

I put the turn-level contract directly into the tool description. I added an explicit, named rule stating that the tool must be called in a turn containing no speech.

I did not put this rule in the system prompt. In a system prompt, behavioral constraints compete with persona instructions, safety guardrails, and the growing conversation history. As the context window fills, a system prompt rule decays and the model forgets it. Function-tool descriptions avoid this because they are the most reliable per-tool instruction channel a model has. They are read in the exact context where the model evaluates its available tools and makes the decision. Anything the model must do around calling a tool belongs in the description.

I gave the rule a specific name, like RULE_SILENT_TURN, so I could search transcripts and prompts for it later. Without a name, you cannot easily query your logs to see if the model even acknowledged the constraint.

Once the description was updated, I verified it behaviorally on real runs. A tool description is a request, not a guaranteed constraint. The model can still fail to isolate the turn. I set up my evaluation runs to score whether the intended line appeared in the agent's own output transcription for that turn, rather than just checking whether the tool fired. Scoring the tool execution tells you nothing about the audio failure. You must score the turn isolation.

That is as far as an automated run can take you. Automated scoring confirms that the model emitted the transcript text, but cannot detect whether synthesised audio was dropped before playout. Live listening on test calls remains the only verification that the provider emitted the audio track to the caller.

What I would check first

Check the event sequence in your provider logs for the failing turn. Expect the function call event to arrive in the exact same generation block as the terminated speech. This confirms the generation was killed by the tool call, ruling out a network disconnection.
Check if the tool in question tears down the session. If the tool is designed to hang up, you are dealing with a lifecycle race rather than just a generation limit. A function tool that tears the session down ends the call while the closing audio is still being generated or played out. If this is the case, defer the teardown to the next caller turn boundary. I have documented this specific failure in /blog/voice-agent-end-call-tool-truncates-the-final-sentence-or-loses-the-callers-last/. This observation discriminates between a standard silent turn and a teardown race.
Check whether the silent turn coincided with a websocket reconnect edge rather than a function tool execution. Expect a reconnect edge to leave both sides waiting silently because in-flight generation died with the dropped connection, which points to a missing reconnect redrive watchdog rather than a tool-turn defect. If the failure occurs on clean transport without reconnect events, you have ruled out transport drops and isolated the failure to concurrent tool calling.

Common questions

Why does my voice agent transcript show text but play no audio?

The model generated text but terminated the turn before synthesizing the audio. This happens on speech-to-speech models when a tool call interrupts the generation phase. The audio being synthesised is truncated or never emitted at all, leaving the text intact while the caller hears silence.

Can I fix silent tool turns in the system prompt?

No, because system prompt instructions decay as the conversation history grows. The rule competes with persona constraints and loses attention weight. You must place the constraint directly in the tool description, which is evaluated at the moment the model chooses to call the tool.

How do I know if the model ignored my tool calling rule?

You must verify the turn behaviorally on real runs by scoring the text transcripts. Score whether the intended line appears in the output transcript, and verify the tool-calling turn contains zero speech tokens. Scoring whether the tool fired is not enough.

Does the LiveKit SDK catch aborted audio generations?

No, because the SDK has no way to know if concurrent speech with a specific tool is an error. The SDK simply stops receiving audio frames when the provider terminates the stream. You must handle this constraint at the model prompting layer.

Canonical URL: https://www.dreamchasers.lt/blog/agent-audio-cuts-off-when-calling-a-function-tool-in-the-same-turn

Opaque RPC errors hide the SIP response codes you need for outbound call retries

Rokas Remeika — Wed, 14 Oct 2026 12:24:41 +0000

By Rokas Remeika. Published 28 September 2026.

When an outbound SIP call fails, the client frequently throws an opaque RPC error string that hides the underlying SIP response code. Without that code, you cannot distinguish a busy signal from a disconnected number, and your retry policy collapses. To fix this, build a cycle-safe recursive function that walks the error's metadata and trailers to recover the three-digit protocol code. Map that code to a closed set of typed outcomes (such as no-answer, busy, or invalid number). Pass only the typed outcome to your retry logic so it can safely retry busy numbers without hammering dead ones.

Symptoms

Failed outbound placements all surface as the same opaque RPC error string in your application logs.
You cannot separate a normal "they were busy" condition from "the number is dead" or "our trunk credentials broke".
Your task queue retry policy collapses into an all-or-nothing toggle.
If you configure the queue to retry everything, you end up hammering disconnected numbers and burning your carrier relationship.
If you configure the queue to retry nothing, you drop calls that would have easily connected on the second attempt.

What is actually happening

When a carrier rejects or fails to complete an outbound call, they respond with a standard three-digit SIP status code. However, the RPC client that your application uses to interface with the calling platform frequently drops this protocol code from the top-level exception it throws.

The SIP response code still exists, but it lives buried in the RPC error's metadata or trailers. It appears under one of several possible key spellings, depending on the specific trunk and the vendor routing the call. Sometimes the code is nested a level or two down inside a dictionary. Sometimes it is only embedded within a human-readable text message string.

Without recovering this underlying protocol code, your application is forced to classify failures by running substring matches against the error message text. This approach degrades rapidly because the wording of an error message is prose. A vendor is completely free to change their error text without telling you, which silently breaks substring matching and causes recoverable calls to be dropped.

How to confirm it

Inspect the raw error object from a failed placement attempt. Log the exception's top-level attributes and inspect its metadata and trailers dictionaries directly. Expect to see a generic top-level exception message, but a dictionary of trailers or metadata containing the real SIP code buried a level or two down.
Trigger a deliberate auth failure by changing a working trunk credential to an invalid string, then attempt a call. Expect your system to log the exact same top-level RPC error type as it does for a busy signal. This confirms your current exception handling cannot tell an auth failure from a busy callee.
Search your logged exception strings for three-digit numbers. Expect to find protocol codes like "486" or "404" embedded in sentences like "Call failed with SIP response 486". This proves the protocol data is present but arriving completely unstructured.

How it works

To build a working retry policy, you have to recover the underlying protocol code first, map it to a small closed set of kinds you own, and let only that typed kind reach your retry logic. Codes are a spec. Strings are prose.

I handle this by writing a recursive function that walks the error's metadata and trailers. This walk must be depth-limited and cycle-safe, utilizing a visited set. The exception objects thrown by these RPC clients frequently hold self-references (such as an exception pointing to its own cause). A naive recursive walk will immediately hang the process.

Inside the walk, I probe the object for plausible key spellings rather than relying on one canonical name. The useful detail is not in the same place on every trunk.

If the key lookups fail to find a code, I fall back to running a regular expression over the human-readable message. This regex requires a "sip" proximity anchor. I look for the word "sip" near a three-digit number, rather than just matching any three digits in the text. Without the anchor, a three-digit area code that happens to be present in the error string will be mistakenly harvested as a SIP response code, leading to bizarre retry behavior.

Once I successfully extract the three-digit code, I map it onto a small closed set of typed kinds, and I branch my logic on the kind. The mapping handles the distinct failure domains:

408, 480, and 487 map to no-answer. I return this to the queue with instructions to retry later.
486 and 603 map to busy. I instruct the queue to retry this later, but with a significantly shorter backoff time than a no-answer.
404 and 410 map to invalid-number. This is a terminal failure. The system stops calling the number immediately.
5xx series codes map to carrier-reject. I retry these, but I escalate if the condition is sustained across multiple calls.
401 and 403 map to auth-failure. This is terminal for the entire queue run. It represents a human problem where the trunk is broken, not the destination number. Continuing to dial under a 401 just wastes compute.

Isolating retry decisions to this small closed set ensures carrier wording changes cannot alter placement scheduling. Pre-placement markers prevent retrying tasks from double-dialing in-flight calls while classification evaluates earlier attempts. If you are struggling with retries that hit the same customer twice, you can read more about how I solve that in my post on /blog/task-queue-retry-called-the-customer-twice-and-an-exists-check-will-not-stop-it/.

What I would check first

Count your unmapped errors by string across a batch of failed placements. A high volume of identical strings arriving at your fallback logic rules out random network drops. It points directly to a missing key spelling in your scraper that you need to add to the probe list.
Cross-reference a specific failed placement in your application logs with the logs provided by your carrier. If the carrier logged a 486 Busy but your application logged an opaque RPC error string, it rules out a carrier routing issue and confirms the RPC layer is masking the response.
Check whether unmapped errors contain status codes in the message text rather than the metadata. If the message contains a phrase like 'SIP 486' but metadata extraction returned nothing, your key probe list missed the trailer spelling and you need the proximity-anchored regex.

Instrumentation and observability

I record the typed kind on every single failed placement. This allows failures to be counted by kind rather than by string. Grouping errors by busy or invalid-number gives a much clearer picture of list quality than reading through hundreds of slightly different exception sentences.

I log every failure where no SIP code could be recovered by any path. Each one is a sign the scraper needs another key spelling. A new carrier or an updated RPC library version will eventually introduce a new metadata shape, and logging the unmapped errors is how you catch the change before it impacts your retry metrics.

Finally, I record terminal classifications directly against the target phone number in the database. If a systematic misclassification occurs in the pipeline, it shows up as a dense cluster of terminal states against otherwise healthy numbers, allowing me to flag the run and investigate the carrier response.

Common questions

Why is the SIP code missing from the error message?

RPC clients often encapsulate carrier responses inside generic exceptions. The specific protocol code is usually pushed down into the exception's metadata or trailer dictionaries, leaving only an opaque string at the top level.

How do I tell if a call failed because the user was busy?

Extract the SIP response code from the error metadata. A 486 or 603 code indicates a busy signal, which you should classify as a temporary failure and retry with a short backoff.

Why shouldn't I just search the error string for a 404?

Blindly searching for a three-digit number will cause false positives if the target phone number or request ID happens to contain 404. You must use a proximity anchor like the word 'sip' to ensure you are matching an actual status code.

What should happen if I get a 401 or 403 on an outbound call?

Treat it as an authentication failure. This is terminal for the current run because your trunk credentials are broken. Retrying the number will not fix a trunk configuration problem.

If the boundary this note describes is one you would rather own, moving this layer off LiveKit Cloud moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/opaque-rpc-errors-hide-the-sip-response-codes-you-need-for-outbound-call-retries

Voice Agent Goes Silent After A Websocket Reconnect

Rokas Remeika — Thu, 08 Oct 2026 14:03:26 +0000

By Rokas Remeika. Published 24 September 2026.

A reconnect restores the transport connection, not the conversational turn. Whatever generation was in flight when the session dropped dies with the old connection. Afterwards, the model waits for input it believes it already consumed, and the caller waits for a reply that no longer exists. Because silence is a stable state, no timeout fires and no error is raised. You fix this by building a watchdog that detects the reconnect edge and issues an explicit reply request to the provider. When building this detector, you must latch on the observed absence of the session to avoid the ABA polling trap.

Symptoms

The session reconnects successfully and then nothing happens. The room is alive, audio frames are flowing both ways, the caller says something and the agent does not answer. No error is raised and no timeout fires.
Around ten minutes into a successful call, the agent spontaneously restarts the conversation. It re-introduces itself, asks its opening question again, and has no memory of anything already discussed. The call never dropped.
The caller interrupts the agent mid-sentence. The agent stops correctly, and then never speaks again. About 90 s later, something else fires and the call either recovers or ends. During this wedge, frames flow in both directions and nothing has errored.

What is actually happening

These failures all stem from a mismatch between transport health and turn state. A realtime connection can be perfectly healthy while the conversation sitting on top of it is entirely dead.

When a websocket connection drops and reconnects, the plugin restores the transport. It does not restore the turn. Whatever generation was in flight when the network dropped died with the old connection. Once the transport is back up, neither side is driving the conversation. The model is waiting for input it believes it already consumed. The caller is waiting for a reply that no longer exists. The system is structurally idle, which is why no component throws an error. Silence is a stable state, so it persists until something external breaks it.

The mid-call greeting is caused by provider session management. The provider closes long realtime sessions with a server-initiated session-recycle notice on a roughly ten-minute cadence. The plugin reconnects, but unless session continuation is configured in the setup frame, the reconnect opens a brand-new session with no prior context. The system instructions are still there, so the model does the only thing a cold start allows: it greets. To the caller this reads as the agent forgetting the entire conversation. To the logs, it reads as a completely successful reconnect.

The post-interruption wedge is a failure in generation finalisation. The receive path is gated on the current generation reaching a terminal state. On a caller barge-in, the provider stops generating audio, but the turn-complete event for that generation does not always arrive. The plugin keeps treating the dead generation as in flight. Because the system thinks a response is still active, inbound caller audio is never turned into a new generation request. Nothing errors because the client is simply waiting, and waiting is exactly what it is programmed to do.

How to confirm it

Inspect the setup frame dispatched by the client during the websocket connection phase. Look for the session continuation configuration block. If it is absent, expect the agent to re-greet the caller upon the first session recycle.
Log the generation open-to-terminal duration per turn, and count generations that never reach a terminal state. Expect these permanently open generations to correlate directly with the calls that fall silent immediately after a caller barge-in.
Record the time from a reconnect edge to the first agent audio on each call. This makes a run with the watchdog off directly comparable to one with it on. Expect to see an infinite gap where the transport is connected but agent audio never follows, confirming the turn is dead.

The fix

For the reconnect silence, you need a watchdog that detects the reconnect edge. If no agent audio follows within a defined bound, the watchdog issues an explicit reply request to restart the turn.

For the mid-call context loss, configure session continuation in the setup frame. The provider will issue a continuation token the reconnect can present, allowing the reconnect to carry on the same conversation rather than starting a new one. On the plugin version I run, that configuration had to be re-added at the setup-frame patch point because the plugin was not sending it. Then, handle the edge case where the provider rejects the continuation. When it reports that the continuation did not take, clear the per-session state you were carrying and re-drive the conversation deliberately rather than letting the model improvise a second opener.

For the barge-in wedge, implement a finalisation watchdog. This watchdog tracks the open generation and the time since its last activity. If the generation remains open with no activity past a bound, the watchdog finalises it locally and releases the gate. Keep this decision logic as a pure function of timestamps so the boundary cases are unit-testable.

What I would check first

I checked if the silence immediately followed a caller interruption. Because it did, I ruled out a network drop and investigated the finalisation state machine for a missing turn-complete event.
I checked the call duration when the context reset. Because the agent re-greeted consistently around the ten-minute mark, I checked the setup frame for a missing session continuation token.
I checked the state-change detection logic in the supervisor loop. Because the reconnect detector compared the current session handle to the previous one, I rewrote the detector to fix the ABA problem.

The polling trap in reconnect detection

When building the reconnect re-drive watchdog, the obvious implementation is a trap. You poll the session handle to detect when it changes. The easiest way to write that detector is to check if the current handle differs from the last one seen. This passes every test, because in a test the reconnect happens once and stays.

Under a real recycle, the network drops and restores rapidly. The handle can go present, absent, and present again between two ticks of the supervisor loop. If the observable value on tick three is identical to the one on tick one, a differs-from-last detector concludes nothing changed. It never issues the re-drive, and the call just sits there in silence.

This is the ABA problem, and here it fails entirely silently. The fix is to latch on the gap. You must record that you observed the absence, and treat the next presence as a new session even when the observable value is identical to the one before it. This requires three lines of state management rather than a single equality check.

Being able to reason about this polling failure came entirely from having the reconnect on a fault injector. I had event logs with monotonic timestamps to replay against, which allowed me to step through the state machine ticks rather than reading the loop and hoping it worked.

Re-driving is not free

Resumption alone is not enough. A resumed session still needs someone to restart the turn, which is the entire purpose of the re-drive watchdog. If the model managed to resume on its own and you issue a re-drive anyway, the agent will talk over itself. The bound you set before re-driving has to be strictly longer than the provider's worst post-reconnect first-token latency. You have to measure that latency in production before choosing the number.

You also need to instrument the outcome. Emit an event for the reconnect edge, an event for the re-drive decision, and an event for the result. This instrumentation makes a re-drive that talked over a live model distinguishable from a re-drive that successfully rescued a dead call. Record the time from reconnect to first agent audio on each call so a run with the watchdog disabled is directly comparable to one with it enabled.

Tuning the finalisation bound

The finalisation watchdog requires setting a timeout bound on open generations, and that number is a real tradeoff in both directions. Make it too tight, and you cut off a model that is genuinely thinking or producing slowly. Make it too loose, and you have simply reproduced the 90 s wedge with extra steps.

I observed that when a generation's terminal event never arrives on real calls, something else eventually fires and ends the silence after approximately 90 s. Your local finalisation bound must be significantly shorter than that.

Testing this requires an injector. If you rely on organic barge-ins failing, you will never get enough data to tune the bound safely. You need a test harness that deliberately swallows the turn-complete event on a chosen conversational turn, so the wedge reproduces in seconds rather than waiting for a live incident.

Barge-in is the primary trigger for this failure, which means this bug concentrates on exactly the calls where the caller is most engaged. It is also completely invisible to any health check that looks at the room, the transport, or the participant, all of which remain perfectly fine throughout the wedge. The only observable that tells the truth is the time since the agent last produced audio, and you have to watch it explicitly.

Common questions

Why does the voice agent stop responding after a websocket reconnect?

The reconnect restores the network transport but abandons the conversational turn. The generation in flight died, so the model waits for input and the caller waits for a reply.

Why does the AI agent greet the caller again ten minutes into a call?

Realtime providers recycle long sessions around the ten-minute mark. If you do not configure session continuation in the setup frame, the reconnect opens a brand-new session and the model defaults to its greeting.

Why does the agent go silent for 90 seconds after I interrupt it?

The provider stops generating audio but fails to send the turn-complete event. The client waits for the dead generation to finish, blocking new input until something else fires roughly 90 seconds later.

How do I safely detect a reconnect to redrive the agent?

Poll the session handle and latch on the absence. If you only compare the current value to the previous one, you fall into the ABA trap and miss a reconnect that drops and restores between two ticks.

Canonical URL: https://www.dreamchasers.lt/blog/voice-agent-goes-silent-after-a-websocket-reconnect

Voice Agent End-Call Tool Truncates The Final Sentence Or Loses The Caller's Last Answer

Rokas Remeika — Tue, 06 Oct 2026 13:40:13 +0000

By Rokas Remeika. Published 20 September 2026.

A function tool that tears the session down when called ends the call while the closing audio is still being generated or played out, so the caller hears half a sentence. Deferring teardown to the next caller turn boundary fixes truncation but creates a second failure. If a pacing watchdog force-arms the end while the caller is answering, the teardown trigger consumes the answer and it never reaches the transcript. The fix is an armed closure that carries a flag for whether it may fire on the next turn. Tool-initiated ends set it true, and force-armed ends set it false.

Symptoms

The agent signs off, but the connection drops before the audio finishes playing. The caller hears half a sentence or nothing at all.
The caller answers the agent's final question, the call ends politely, but the caller's final answer is entirely missing from the transcript.
The agent says its closing line, but the caller hears absolute silence on that turn.
The conversation is supposed to be bounded, but the agent ignores time limits. Calls regularly overrun until a hard transport or provider limit drops them.

What is actually happening

If a tool ends the session, the teardown begins while the text generation or audio synthesis for that same turn is still in flight. The connection closes while the closing audio is still being generated or played out, resulting in truncated audio.

Moving the teardown to fire on the next caller turn boundary fixes the truncation, allowing the agent's speech to play completely. However, this introduces a second failure. If something else arms the teardown, such as a pacing watchdog forcing a close because the time budget ran out, the very next caller turn is the caller's final answer. If the code fires the teardown right on that boundary, it consumes the answer as the teardown trigger. The answer never reaches the transcript entirely.

If the agent is going silent on the turn where it calls the tool, the issue lies at the generation level. On a speech-to-speech model, a turn produces one generation. A function call terminates that generation. If the model attempts to generate speech and call an end-tool in the exact same turn, the audio being synthesised is truncated or never emitted. No framework guards against this because nothing in the SDK knows this specific tool is fatal to concurrent speech.

The call overruns happen because nothing in a realtime session tells the model how much of its budget it has used. Elapsed time is not in the context window. Prompt-level pacing instructions compete with everything else in the prompt and decay as the conversation grows.

How to confirm it

Log the turns that contain both generated audio and a tool call. Expect to find that every time the agent goes silent on its closing line, the model attempted to invoke the end tool concurrently with speech.
Record the specific path that triggered the teardown on every call missing a final transcript. Expect the transcript loss to correlate perfectly with force-armed closures happening on a caller turn boundary, not tool-armed closures.
Inspect the agent's own output transcription for the final turn. Expect the intended sign-off text to be present in the output transcription, confirming the model attempted to speak but the function call terminated the concurrent generation.

How it works

End tools do not perform a teardown; they arm a pending closure. The armed closure carries a flag indicating whether it may fire on the next caller turn.

Tool-initiated ends set this flag to true. The closing line plays, the caller gets their beat to respond, and the following turn boundary provides a clean place to tear down the connection. Force-armed ends set this flag to false. They are driven by a different path, ensuring the caller's answer is completely recorded and transcribed first, with the teardown executing only after the answer completes.

I guarded the arm and fire state with a lock that was held briefly and had no asynchronous await inside it. Arming the closure races with the turn boundary that fires it. Putting an await inside that lock simply reintroduces the exact race condition you just closed.

The constraint that the closing line and the arming call must be separate turns is enforced in the tool description, not in code. The model decides which turn to call the tool in, so the only enforcement surface is the text the model reads.

What I would check first

Check whether the missing transcript correlates with a watchdog timeout or an agent tool call. An agent-initiated end rules out the watchdog, meaning your turn boundary logic is consuming the answer. A watchdog timeout means you need a separate path for force-armed teardowns.
Check if the closing line exists in the model's output generation payload. If the text is there but the caller heard nothing, it rules out a logic failure and confirms the tool execution is destroying the concurrent audio generation.
Record whether each call closed by agent decision, by pacing force-arm, or by a hard cap to isolate the exact limit being hit.

Pacing watchdogs and pure functions

To fix calls overrunning the budget, I run a pacing watchdog as a separate async task alongside the session. Each tick computes how much of the budget remains and decides whether the agent should begin closing. When it should, the watchdog injects a closing instruction at a turn boundary rather than mid-generation. If the agent fails to wrap up, the watchdog force-arms the teardown.

Two structural rules govern this approach. First, the decision logic is a pure function of elapsed time and session state. It takes timestamps and state, returning a decision. This means the boundaries can be unit tested exhaustively without a live model or a room. The async task is just a thin loop that gathers state, calls the function, and acts.

Second, the watchdog is default-on but gated by an environment kill switch. Watchdogs act on live calls unprompted, so a misfiring watchdog must be disabled in seconds, not in a full deploy. The operational question is not whether it works, but how fast you can stop it when it is wrong.

Putting behavioural rules in tool descriptions

Naming the rule inside the tool description is a small structural choice that pays off. The model decides when to call a tool. Any turn-level contract, such as instructing the model that this tool must be called in a turn containing no speech, belongs in the description.

Do not put it in the system prompt where it competes with everything else. Give the rule a name so you can search transcripts and prompts for it later. The description is the most reliable per-tool instruction channel the model has, read right in the context where the decision occurs.

Verify it behaviourally on real runs. A description is a request, not a constraint. That is as far as an automated run can take you. Confirming the audio itself reached the caller requires a human on the line, since there is no recording and no second recognizer to check it for you. Both halves of this failure came from sitting on my own test calls and listening to them end badly. The truncated sign-off is obvious to anyone on the line and completely invisible in logs. The eaten answer is the opposite: it sounds perfect to a listener, but exists only as an absence in the transcript. You only notice it if something downstream depends on that last answer.

Common questions

Why does the agent audio cut off when it calls the end tool?

On a speech-to-speech model, a function call terminates the generation for that turn. If the model attempts to generate speech and call a tool concurrently, the audio synthesis is truncated or destroyed entirely.

Why is the caller's last answer missing from the transcript?

Your teardown logic is firing exactly on the final caller turn boundary. By executing the close at that moment, the system consumes the answer as the teardown trigger before it reaches the transcript.

How do I enforce a strict time limit on a voice agent?

A realtime model has no clock, so prompt instructions decay over time. You must run a pacing watchdog as a separate asynchronous task that observes the clock, injects closing instructions, and force-arms a teardown if necessary.

Where should I put instructions about how a tool is used?

Put behavioral rules in the tool description as an explicit named rule, not in the system prompt. The description is the most reliable per-tool instruction channel, read exactly when the tool choice is made.

Canonical URL: https://www.dreamchasers.lt/blog/voice-agent-end-call-tool-truncates-the-final-sentence-or-loses-the-callers-last

Outbound Calls Fail With Trunk Does Not Exist After A Restart

Rokas Remeika — Sun, 04 Oct 2026 12:52:45 +0000

By Rokas Remeika. Published 18 September 2026.

When outbound dialing fails with a 'trunk does not exist' error after a restart, the SIP trunk state store has likely lost its data. The platform mints new trunk IDs on boot, rendering any cached IDs in your database or application memory invalid. The immediate fix is to catch this specific validation error, evict the stale ID, resolve the new ID by its stable name, and retry the placement exactly once. For a durable fix, implement an idempotent boot-time provisioning script that rebuilds the trunk state from your database of truth.

Symptoms

Every outbound call failed with a "trunk does not exist" error.
The error referenced a trunk id that was configured and that used to work.
Inbound was still fine.
Nothing was deployed. The only event was a restart of the SIP or SFU stack, or a host reboot.
The blast radius was total. Outbound was completely dead, and the only evidence was an error string that read like a configuration typo.

What is actually happening

SIP trunk state is held in a store that is not necessarily durable across restarts. A common shape is a Redis instance deployed without persistence configured. When the platform comes back online, it has forgotten all previously provisioned trunks and mints new trunk IDs for everything it creates.

Every ID you cached in process memory, wrote into a static configuration file, or persisted next to a phone number now points at a dangling pointer. The failure remains completely invisible until the application attempts to dial out.

Because the error is a validation error rather than a call outcome, it is easy to classify as a bad number instead of broken infrastructure. A team that has not seen it before will spend the first hour checking credentials. Any resource identifier minted by a stateful service that you cache or persist elsewhere is a restart away from being a dangling pointer.

How to confirm it

Check your event logs for self-heal events carrying both the old and the new trunk id per restart.
Look for placement failures classified specifically as trunk-not-found, separating them from carrier-side rejections.
Measure the time from stack restart to the first successful outbound call.

How it works

Fixing this durably requires handling the failure at two layers. You need a narrow in-flight self-heal to catch the first dead dial, and an idempotent boot-time provisioner to rebuild state without duplication.

The in-flight self-heal targets this specific validation error. When an outbound placement fails with the "trunk does not exist" string, the dial path intercepts the failure. It evicts the cached ID, re-resolves the trunk by its stable name at the edge, and retries the placement exactly once. The freshly resolved ID is then written back to the database of truth so the next process does not repeat the discovery. Without this write-back step, the heal never converges, and every worker process re-learns the same thing.

Retrying a telephony placement is normally unsafe because you cannot tell whether the first attempt reached the carrier. It is safe here only because this specific error is raised during request validation, before any INVITE leaves the server. A retry provably cannot ring a person twice. Widen this error match, and you have built a double-dial.

The in-flight self-heal covers the gap, but the durable fix is boot-time provisioning. Trunk creation APIs generally create resources. They do not upsert. Any provisioning script that runs more than once produces duplicate trunks, leaving you guessing which one the phone number is actually attached to.

I make provisioning create-or-update by name for both directions. The boot-time unit lists existing trunks, matches on the stable name, updates the configuration in place, and creates the trunk only when absent. It runs as a one-shot unit that waits for the stack, authenticates, and re-provisions trunks from the database of record with bounded retries. Recovery after an infrastructure restart is then automated, rather than a runbook executed by hand under pressure.

What I would check first

Examine the exact error string and metadata on the failed outbound placement. An explicit "trunk does not exist" points to state drift. Separate placement failures classified as trunk-not-found from carrier-side rejections.
Inspect the persistence configuration on the Redis instance backing your SIP stack. A Redis instance without persistence configured is the common shape, so the platform mints new trunk ids when it comes back. Check that trunk ids cached in process memory have not become dangling pointers.
Check your application database for duplicate trunk names. Multiple active trunks with the exact same name mean your provisioning script is not idempotent. Check trunk count by name on a schedule, because anything above one per direction is drift.

Why an existence check will not prevent a double-dial

When you implement retry logic for failed telephony placements, you run into the limits of your task queue. An at-least-once queue retries the dial task if the worker takes too long to acknowledge it. If this happens, the same person is called twice within a minute. The queue shows one logical task, but the carrier shows two outbound INVITEs.

The guard most people reach for is checking whether the call record already exists in the database. If it exists, they abort the dial. This either never fires, or it fires on the first attempt and blocks a call that was never placed.

Two separate problems stack here. First, the call record is not authored by one writer. The platform native room-started webhook can create it before the dial RPC even returns. The record existing is not evidence that a dial happened. An existence check therefore reads a marker somebody else wrote. Second, the dangerous window is between the moment we start placing the call and the moment we record the placement. Ringing takes tens of seconds. A queue that retries on an acknowledgement timeout re-enters exactly there, when nothing has been written yet.

I use two markers with different jobs. A post-placement marker is written only by the dial path after the placement RPC returns. This is the proof that a dial actually happened. Check that field, never bare document existence.

Sizing the in-flight lock window

A pre-placement attempt marker, written before the RPC, closes the in-flight window. Any attempt that starts inside a bounded validity window after an existing attempt marker is refused. Too short, and a retry fires while the phone is still ringing. Too long, and a failed attempt locks a number out of being called for minutes.

I size this window to a 45 s ringing timeout plus the task queue acknowledgement budget. This ensures that a genuinely in-flight attempt blocks a retry, while a dead attempt eventually releases the lock instead of wedging the number forever. Enforce this same window at webhook-processing time so the guard does not depend on which path runs first. Take both numbers from configuration rather than from memory.

Inbound allowlists and total silence

If your provisioning script runs twice and misconfigures the inbound trunk, the symptom is total silence. You will see no logs, no webhook, and no participant, while the carrier insists it is sending the calls.

On a raw carrier trunk, the inbound authenticity check is an IP allowlist of the carrier signaling addresses. If those CIDRs are wrong or incomplete, INVITEs are dropped before your application even exists.

I pin the carrier published signaling CIDR blocks on the inbound trunk directly from their documentation, rather than relying on observed source addresses. When inbound produces no telemetry whatsoever, suspect the allowlist before suspecting your application logic. Your application is not involved yet.

Raw trunking is more work than a CPaaS SDK, and it is also where the interesting failures live. You own the allowlist, the trunk lifecycle, and the URI format, none of which a CPaaS lets you see. The trade is control and cost against a large amount of undifferentiated setup. The engineering claim here is idempotent, self-healing provisioning against a raw carrier trunk, not a carrier abstraction layer.

Traceability and instrumentation

To make these failures visible, you need per-call event-sourced traceability with a structured failure taxonomy. When the self-heal fires, emit an event carrying the old and the new trunk ID, and count them per restart.

Separate placement failures classified as trunk-not-found from carrier-side rejections. Recovering SIP codes buried in RPC metadata allows you to build a typed SIP failure taxonomy driving retry-versus-terminal branching. A trunk missing error is a local cache invalidation signal.

Measure the time from stack restart to the first successful outbound call.

Common questions

Why do outbound calls fail with trunk not found after rebooting?

The SIP stack stores trunk IDs in a memory store like Redis. If persistence is disabled, a reboot wipes the state, forcing the stack to mint new IDs.

Is it safe to automatically retry a failed telephony placement?

Retrying is only safe if the failure happens before the signaling leaves the server. A local validation error like a missing trunk ID meets this criteria.

How do I prevent duplicate SIP trunks after rerunning setup scripts?

Use a create-or-update approach keyed by the stable trunk name. List existing trunks, update them in place if they exist, and create them only if they are missing.

Why does my at-least-once task queue call the same person twice?

If a queue redelivers a dial task while the first attempt is still ringing, two INVITEs are sent. You need a pre-placement attempt marker to lock the number during the ringing phase.

If the boundary this note describes is one you would rather own, taking ownership of the infrastructure underneath moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/outbound-calls-fail-with-trunk-does-not-exist-after-a-restart

LiveKit SIP Trunk Does Not Exist After A Container Recreate Or Host Reboot

Rokas Remeika — Fri, 02 Oct 2026 13:10:14 +0000

By Rokas Remeika. Published 16 September 2026.

SIP trunks and dispatch rules live in Redis, not on disk. Recreating a stock Redis container wipes this state. Your provisioning path recreates the trunk but receives a brand-new identifier. Any trunk ID you cached in your database is now stale, and outbound calls will fail silently with a missing trunk error. Fix this by enabling append-only persistence on a mounted Redis volume. Stop treating trunk IDs as durable keys: resolve trunks by name, cache the ID, and re-resolve on that specific missing-trunk error before retrying once.

Symptoms

After a host reboot or a container recreate, the media stack comes back and appears healthy.
Browser calls connect without issue.
Inbound phone calls might still route correctly.
Outbound dialling fails quietly. The error logged in your application names a SIP trunk that does not exist.
Nothing failed during the restart sequence itself. The stack looks fully operational until the next outbound call attempt.

What is actually happening

When you deploy a self-hosted realtime stack, the SIP trunks and dispatch rules do not live in the media server's memory, nor do they live on disk. They live in Redis. A stock Redis container ships with no persistence configured. When you recreate the container, the entire store is emptied.

Your infrastructure comes back up. The media server is healthy. At this point, your stack has no telephony configuration. Your automated provisioning path runs against the control plane to recreate the missing trunks. The API obliges, creates the trunk, and hands back a brand-new identifier.

The breakage happens because your application's database of record is now out of sync with the media stack. Every trunk ID you cached in your own database is stale. A stale trunk ID only surfaces when a component tries to place an outbound call with it. The restart was silent, the health checks are green, and the state that routes your phone calls has been quietly reset.

When people say they self-host LiveKit, they usually mean one binary. Production is actually four co-located containerised services: the SFU, the SIP service, Redis, and a TLS terminator on port 443. These four fail independently. Only the SFU has a health check that most setups bother to probe. You are observing a silent failure in the SIP dependency while looking at a green light from the SFU.

How to confirm it

Check your application logs for the failed outbound call and extract the specific trunk ID it attempted to use. The logged error will specifically name a trunk that does not exist.
Resolve the specific trunk by name against the media server. Expect to find the trunk you want by name, but with an entirely different ID than the one your application just tried to dial with.
Inspect the Redis container from the host system to see if the data directory is actually mounted. Expect to find an empty or non-existent volume mount on the host file system.

How it works

Fixing this requires two distinct changes. Persistence handles the common case. Resolving by name survives the case where persistence was not enough.

First, you must turn on append-only persistence in Redis with a real mounted data directory. A container recreate will then keep the store intact. Since a realtime stack running on host networking has no container network namespace to hide behind, you must also bind Redis to the loopback interface with protected mode on.

Second, stop treating a SIP trunk ID as a durable key. The underlying mistake here is writing a service-minted identifier into your own database. The moment you do that, you create a coupling between your persistence and someone else's volatile cache, with no mechanism to notice when they diverge. The trunk name is the stable thing. The ID is just a cache.

I resolved trunks by name, cached the resulting ID, and placed the call. If that call failed, I matched on the narrow "trunk does not exist" error. On that specific error, the application evicted the cached ID, re-resolved the trunk by name, and retried exactly once.

You must keep the retry's error match deliberately narrow. That specific missing-trunk error is raised before any SIP INVITE leaves the box. This is the only reason retrying is safe. A broader match that catches failures after the INVITE has been sent turns a self-heal into a second real phone call to a real person.

What I would check first

Check if the Redis data directory has survived the recreate. A fresh data directory points to a missing volume mount, which rules out an application-layer cache invalidation bug.
Check the exact text of the dial failure. The exact 'trunk does not exist' error is raised before any INVITE leaves the box, confirming the state mismatch.
Check the open-file descriptors held by the stack against the systemd unit's configured limit. A ceiling hit here manifests as refused network connections, which rules out missing trunks and points to resource starvation.
Check if the SIP container is currently restarting in a loop while the systemd unit shows as active. This discriminates between a partially failed compose project and a completely crashed systemd service.

Boot-time SIP reprovisioning without hanging the host

A host reboot is the most common trigger for this silent outage. The core issue is that the machine starts up with no telephony configuration. Every restart, planned or not, requires a human to log in and run a provisioning step by hand unless you automate it.

I used a systemd oneshot bound to the stack unit, so it re-ran whenever the media stack did. It polled the stack's health endpoint for up to two minutes before doing anything. A unit ordered "after" the stack unit only knows the process started, not that it is actively serving. Provisioning against a media server that has not finished coming up returns errors that look like configuration problems. Polling prevents this.

Once healthy, the script took a short-lived identity token from the platform the host runs on. This token was scoped to the control plane it was calling. I never used a stored credential for this. A long-lived key file on a disk is a durable credential in the least defensible location you have. A platform-issued token binds to the machine's own identity, leaving nothing to rotate and nothing to leak in a disk snapshot.

The script then called the control plane's provisioning endpoints with eight bounded retries. The provisioning logic itself must be an idempotent create-or-update by name, not a plain create. A boot unit that runs on every restart against a create-only endpoint will produce drifting duplicate trunks. Since duplicate trunks route traffic, you will end up with multiple paths, one of which nobody is maintaining.

Crucially, every network request in this script carries both a connect timeout and a total timeout. A TCP connection that establishes and then goes quiet will sit there past any connect timeout. In a boot unit, that means the machine's startup sequence is now waiting on a dead socket. You must bound the whole request and bound the retry count to let the unit fail loudly rather than hang indefinitely.

Restart ordering and the host network namespace

A production realtime topology sits on host networking. The media services need roughly ten thousand UDP ports for RTC (documented as 50000 to 60000), the SFU websocket on 7880, TCP fallback on 7881, TURN on UDP 3478 and TLS 5349, plus SIP signalling on 5060 with its own RTP range from 10000 to 20000. You cannot sensibly publish this many ports through the container runtime's port mapping.

Running in the host network namespace means services can collide on ports. Restart ordering has to be explicit because two versions of the same container cannot both hold the same signalling port at the same time.

I manage the compose project with a single systemd unit. The unit carries an ExecStartPre directive that brings the entire project down before bringing it up. A restart is exactly one deterministic action. Without this down-before-up step, a partially failed start leaves old containers alive that the next start cannot replace. You end up with a half-old, half-new set of containers fighting over exclusive host ports.

Because a realtime process holds several sockets per participant, plus relay sockets, it walks past the default Linux open-file limit long before it runs out of CPU. The systemd unit must raise the open-file limit well past the stock default. The failure mode when you hit this limit is not a crash; it is refused connections, which reads as a networking problem. The raised limit is a reasoned hardening decision taken before the ceiling is ever hit.

Instrumenting the silent failures

The thing that makes this entire class of failure expensive is the silence. There is no error logged at restart time. The only broken path is the one that needs a trunk ID, which is the path you exercise least often in automated testing and care about most in production.

I recommend two specific additions to surface this state mismatch before a real caller experiences it. First, add a boot-time check that verifies the trunk IDs held in your own database of record still resolve correctly on the media server. This turns a silent breakage into a noticed one at the moment the mismatch happens, rather than delaying the discovery until the next call attempt.

Second, count the self-healed placements. The narrow retry logic is there to save the call, but it also hides the fact that a cache eviction was necessary. Emitting a metric every time the application successfully falls back to resolving the trunk by name tells you whether the Redis persistence is actually doing its job. A rising count of self-healed calls means the persistence layer is failing, and your volume mounts or configuration have drifted.

Common questions

Why did my SIP trunk ID change after a restart?

The trunk configuration lives in Redis. If the Redis container is recreated without persistence, the state is lost. Re-provisioning the trunk mints a new identifier.

How do I wait for LiveKit to be healthy before reprovisioning?

Use a systemd oneshot that polls the stack's health endpoint. Set a maximum wait time of two minutes before making the first provisioning call.

Why does my startup script hang indefinitely?

You are likely missing a total timeout on your network request. A connect timeout alone will not catch a socket that establishes and then goes quiet.

Does the LiveKit server health check include SIP?

No. The media server answers and is marked healthy even if the separate SIP service is down or completely missing its configuration.

Canonical URL: https://www.dreamchasers.lt/blog/livekit-sip-trunk-does-not-exist-after-a-container-recreate-or-host-reboot

Outbound call greets before the callee's audio is subscribed

Rokas Remeika — Wed, 30 Sep 2026 13:01:38 +0000

By Rokas Remeika. Published 14 September 2026.

The SIP leg reporting active is only a hint that the call is up, not a guarantee that audio is flowing. If your agent greets on the active signal, it speaks into a cold pipe before the callee's audio track is actually subscribed. The first thing the user says arrives before the agent worker is listening, so it never reaches transcription. Fix this by explicitly gating the greeting on the inbound audio publication being subscribed, using the active signal only to arm a bounded 2.5 s fallback grace window.

Symptoms

On outbound PSTN calls, the first thing the person says is missing from the transcript. The agent talks over them, asks something they already answered, or the whole conversation runs one turn out of step. Browser calls into the same agent are fine, which sends everyone looking at the model instead of the call setup. The symptom is intermittent because nothing guarantees the carrier timing is the same on the next call or the next route, which makes it read as a probabilistic model failure rather than a deterministic timing race.

What is actually happening

Any outbound voice agent on any carrier has two distinct events at the start of a session. The first event is the call coming up, where the SIP participant attribute reports the leg is active. The second event is audio actually flowing into your process, where the callee's inbound audio publication is subscribed by the agent worker.

The active signal flips a beat before the audio publication is actually subscribed. If your system triggers the agent greeting based on the leg reporting active, it plays the opening line into an input pipe that nothing is reading yet. The person on the other end hears the greeting and answers, but their first answer arrives before the agent worker is fully listening. The utterance never reaches the transcriber or the model.

The gap between these two events is short.

How to confirm it

Record the milliseconds between the leg reporting active and the first audio track subscription, per call. The gap between the two signals is the single number that tells you whether this is your bug, and it costs one timestamp subtraction per call to collect.
Count calls where the first user turn is empty, grouped by SIP versus browser transports.
Record which of the two paths released the greeting: track subscription or grace expiry.

How it works

The correct approach is to gate the greeting on an audio publication actually being subscribed, not on the leg reporting active. The active signal is only a hint.

The obvious fix of adding a fixed sleep after the active signal is wrong. A fixed sleep is a guess at a distribution you have not measured. If you guess too short, the symptom stays. If you guess too long, you add dead air to every call and make the agent feel slow.

Instead, the active signal should only arm a bounded grace window of 2.5 seconds. The moment the actual track subscription fires, it preempts that window and releases the greeting immediately. If the subscription never arrives, the window expires and releases the greeting anyway, so the call fails open rather than hanging forever in silence. I settled on 2.5 seconds as a bounded fallback chosen to sit comfortably above the gap a normal SIP trunk shows. You should instrument the per-call gap first, so the number can be tuned from your own real data.

Clamping this grace window matters more than it looks. The call needs an overall answer budget, which I set to 45 seconds for ringing and answering combined. If a leg reports active at 44 seconds into that 45 second budget, a 2.5 second grace window pushes the greeting past the deadline. A call that was about to succeed is recorded as a timeout. To prevent this, clamp the grace window to whatever remains of the answer budget. The grace can only ever consume time that already belongs to the call.

Finally, remove both event handlers in a finally block. A retry cannot be allowed to stack listeners, and if the handler registrations themselves fail, the code must drop through to the greeting rather than leaving the worker permanently waiting.

What I would check first

When an outbound agent misses the first thing the caller says, the subscription gate is not the only candidate. Here is the order I check things to isolate the exact cause.

Check if the agent worker is actually present in the room before the phone rings. If your code dispatches the agent after creating the SIP participant, expect the worker to still be starting when the callee picks up. This rules out the audio gate, because there is no agent in the room to gate yet.
Check the echo-cancellation warm-up duration on SIP legs. If the greeting waits for the audio track perfectly but a short "hello" still vanishes, expect the stock 3.0 second WebRTC warm-up to be substituting silence for the caller's first word.
Compare the time from dispatch created to worker joined against the time from placement to answer. If the worker join time is later than the answer time, the worker was still starting when the callee picked up.

Creating the room and dispatching before dialing

Agent presence before answer is a precondition for the subscription gate to work, not an alternative to it. Outbound placement consists of three operations whose order is load-bearing.

If you dispatch the agent after the SIP participant is created, the worker can still be starting when the callee picks up. The greeting gate cannot help you, because there is nobody to greet yet. If you create the room implicitly without metadata, the platform's native webhooks carry only platform identifiers. They know the room identifier, but they have no access to your database to resolve the tenant or number.

The correct order fixes both the timing and the ownership. First, create the room explicitly, carrying the routing metadata the webhooks will need. Second, create an explicit agent dispatch so a worker is sitting in the room before the phone ever rings. Third, create the SIP participant last, with wait-until-answered, dial tone playback, a 45 s ringing timeout and a deterministic participant identity derived from the session id so every later event can be correlated without a lookup.

From the moment you create that room, you own it. Any failure after that point (a placement error, a classification failure, or an unexpected throw) leaks a room that a worker will occupy until the empty timeout expires. Wrap everything after room creation in a block that guarantees every failure branch deletes the room before returning the error.

Echo-cancellation warm-up on telephony legs

Even with the subscription gate perfectly implemented, the echo-cancellation warm-up default can still erase a short first answer on telephony legs. Fixing one without the other leaves the symptom in place, which is the usual reason a team concludes the subscription gate did not work.

The warm-up substitutes silence for the far-end audio during the agent's first turn so the agent does not hear itself arriving back through the echo path. The stock window is 3.0 seconds, which is sized for browser audio. On a phone leg, the person answers inside that window, so their utterance is replaced by silence before anything downstream can process it.

Setting the warm-up to zero breaks the agent in the opposite direction. Late-subscribing carrier echo reaches the input, the agent treats its own greeting as user barge-in, and the turn wedges. I cover this exact failure in my post on why echo-cancellation warm-up eats a short first answer on phone calls.

The fix is to make the warm-up transport-dependent. A value of 2.0 seconds on SIP legs is long enough to cover the carrier echo path and short enough that a normal phone answer lands outside it. Keep the stock 3.0 seconds on WebRTC, where the browser's own echo cancellation changes the problem entirely. Implement it as a single override point that returns no override at all for non-SIP sessions. Because the override returns nothing for browser calls, the WebRTC path runs exactly the stock code it ran before, making any regression unambiguously attributable to the SIP branch.

Common questions

Why is the caller's first word missing on outbound SIP calls?

The agent is greeting based on the SIP leg reporting active, before the audio track is subscribed. The caller replies into a cold pipe, so the transcription never captures their first word.

Should I add a sleep timer before the agent greeting?

No. A fixed sleep is a guess at a variable network delay. Too short and you still miss words; too long and you add dead air to every call. Gate the greeting on actual track subscription instead.

Why does my agent interrupt its own greeting on phone calls?

If you disable the echo-cancellation warm-up completely, carrier echo feeds the agent's voice back into its own input. The agent treats its own echo as user barge-in and stops talking.

In what order should I place outbound AI calls?

Create the room with routing metadata first, then dispatch the agent explicitly. Create the SIP participant last so the worker is waiting in the room before the phone rings.

If the boundary this note describes is one you would rather own, running the SFU yourself moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/outbound-call-greets-before-the-callees-audio-is-subscribed

LiveKit SIP trunks gone after a VM reboot

Rokas Remeika — Mon, 28 Sep 2026 14:28:19 +0000

By Rokas Remeika. Published 12 September 2026.

When a voice stack restarts, volatile stores wipe SIP trunks. Browser calls work because the SFU is healthy, but telephony routing fails silently. The fix is a systemd oneshot unit that polls the stack's health endpoint for up to two minutes, then re-provisions trunks automatically. The script must use a short-lived platform token rather than a stored credential, enforce both connect and total timeouts to prevent a stalled socket from hanging the boot sequence, and use idempotent create-or-update requests so successive reboots do not duplicate routing rules.

Symptoms

The host reboots or the media containers are recreated.
The stack comes back online, health checks pass, and browser WebRTC sessions connect normally.
Outbound phone calls fail silently. The error logged states the SIP trunk does not exist.
Inbound routing may still work while outbound dialling fails.
A manual run of your provisioning step fixes the routing.

What is actually happening

"Self-hosted LiveKit" is not a single binary. Production requires four co-located containerised services: the SFU, the SIP service, Redis, and a TLS terminator on port 443. TURN is not a fifth process. It is a capability enabled inside the media server, exposed on its own ports and reached through the layer-4 TLS route, which is exactly why people forget to think about it. These services fail independently, and only the SFU usually has a health check that anyone bothers to probe.

SIP trunks and dispatch rules do not live in the media server's configuration files on disk. They are derived state written into Redis. A stock Redis container ships with no persistence configured. When the host reboots or the container is recreated, the store empties completely.

Your database of record holds the canonical telephony configuration, but the realtime node has forgotten it. The node is running, the SFU is accepting browser connections, and the SIP service has no routing rules.

If you have a naive startup script ordered after the stack unit, it is likely firing before the SIP service is genuinely serving traffic. The script executes against an endpoint that is not yet ready, returns an error that looks like a configuration problem, and exits. The boot finishes, but telephony is unprovisioned.

If you successfully re-provision the trunks later, the service mints brand-new identifiers. Any identifiers you cached in your database are now stale. You only discover this silent reset when something tries to place a call with the stale trunk id.

How to confirm it

Pull the active trunk list from the running service API immediately after a restart. Expect the response to omit your trunks, which confirms the volatile store was wiped.
Attempt an outbound dial using the trunk identifier cached in your database. Expect the specific "trunk does not exist" error, confirming the application is holding a stale identifier rather than failing at the SIP transport layer.
Inspect the host filesystem outside the container for the Redis data mount. Expect an empty directory or missing volume; this confirms that append-only persistence is either disabled or writing to an ephemeral container path.

How it works

I run a systemd oneshot bound to the main stack unit, so it executes every time the stack restarts. Three specific details in this unit prevent it from failing under edge conditions.

The provisioning call must wait for readiness. A unit ordered "after" the stack unit only knows the system process started, not that it is handling requests. The oneshot polls the stack's health endpoint with a ceiling of up to two minutes before making its first configuration attempt.

It authenticates to the control plane using identity from the platform. It requests a short-lived token scoped to the control plane and to the machine's own identity. There is no long-lived secret stored in a file on disk, which means there is nothing to rotate and nothing to leak in a snapshot.

The network call carries both a connect timeout and a total timeout. This is the detail most scripts skip. A TCP connection that establishes and then goes quiet will sit there past any connect timeout. In a systemd boot path, the machine's startup sequence is now waiting indefinitely on a dead socket. Bounding the total request and capping bounded retries to eight attempts lets the unit fail loudly rather than hang the host.

The configuration payload must be idempotent. The provisioning logic relies on create-or-update by name. A startup script firing blindly against a create-only endpoint leaves duplicate drifting trunks on every restart. Duplicates still route, leaving you with identical active trunks, only one of which you are actually maintaining.

What I would check first

Curl the SIP service health endpoint separately from the SFU health endpoint. Expect to see whether the SIP signalling is answering, because folding SIP health into overall stack health is an open gap.
Review the logs of the provisioning systemd unit for the retry attempt count. Expect to see the count incrementing. A climbing count is the sign that the health poll is finishing before the stack is genuinely ready.
Compare the trunk identifier returned from a manual API provisioning call against the one in your database. Expect the strings to differ entirely. This confirms your system is persisting a transient cache key rather than resolving the stable name.

The network namespace constraints

Realtime media requires publishing roughly ten thousand UDP ports, plus SIP and TURN ports. You cannot sensibly publish that range through container port mapping.

The containers run in the host network namespace. This means they are no longer isolated from each other or the host network. Redis is exposed directly. I bind Redis to loopback with protected mode enabled, because on host networking there is no container network to hide behind.

The stack binds to explicit host ports: 7880 for the websocket, 7881 for TCP fallback, 50000 to 60000 for RTC UDP, 5349 TLS and 3478 UDP for TURN, 5060 for SIP signalling, and 10000 to 20000 for RTP.

Because host ports are exclusive, a restart must be a single deterministic action. I configure the supervising systemd unit with an ExecStartPre command that explicitly tears the compose project down before bringing it up. Without this down-before-up ordering, a partial failure leaves stale containers holding host ports, preventing the new containers from starting. You end up with a half-old stack fighting itself.

Treating identifiers as caches

A container recreate mints new trunk identifiers. Treating a service-minted identifier as durable is the underlying mistake. The moment you write one into your database of record, you couple your persistence to a volatile third-party cache with no mechanism to notice when they diverge.

The trunk name is the stable identifier. The identifier string is a cache.

I resolve trunks by name and cache the identifier locally. When outbound dialling encounters the narrow "trunk does not exist" error, I evict the cache, re-resolve by name, and retry exactly once.

The error match for the retry is deliberately narrow. That specific error is raised before any INVITE leaves the box. A broader error match risks catching a failure that occurs after the INVITE is sent, which turns a self-heal into a second real phone call to a real person.

Append-only persistence on a mounted directory handles the standard container recreate path. The resolve-by-name self-heal catches the cases where the volume was missing, restored from an older state, or started fresh. A boot-time check that the trunk ids held in your database still resolve would turn a silent breakage into a noticed one at the moment it happens. Counting self-healed placements would tell you whether persistence is actually doing its job, because a rising count means it is not. Neither is built, but both are cheap.

Open file descriptor exhaustion

A realtime process hits the stock open-file limit long before it runs out of CPU. Every participant holds signalling sockets, media sockets, and relay sockets.

When a process hits this ceiling, it does not crash. It simply refuses new connections. This looks identical to a networking problem.

I raise the open-file limit well past the stock default directly on the supervising systemd unit. This is reasoned hardening ahead of the ceiling, not a ceiling I have measured under sustained load.

Common questions

Why do SIP trunks disappear after restarting LiveKit?

SIP trunks and dispatch rules live in Redis, not on disk. If Redis lacks append-only persistence on a mounted volume, a restart wipes the memory store and destroys the configuration.

How do I prevent a curl startup script from hanging the boot?

You must explicitly set both a connect timeout and a total maximum time limit. A socket that connects but stalls indefinitely will block the systemd sequence unless a total timeout terminates it.

Why are my outbound phone calls failing with trunk does not exist?

Your application is caching a trunk identifier that was minted before a restart. When the trunk was re-provisioned, the service assigned it a new identifier, rendering your cached key stale.

Can I use Docker port mapping for the media UDP range?

No, you cannot sensibly publish ten thousand UDP ports through standard container port mapping. The media services must run in the host network namespace to operate properly.

Canonical URL: https://www.dreamchasers.lt/blog/livekit-sip-trunks-gone-after-vm-reboot

Echo-cancellation warm-up eats a short first answer on phone calls

Rokas Remeika — Thu, 24 Sep 2026 12:13:10 +0000

By Rokas Remeika. Published 10 September 2026.

The echo-cancellation warm-up substitutes silence for the far-end audio during the agent's first turn to prevent it from hearing its own greeting. The stock 3.0 s window is sized for browsers. On a phone leg, a person answers inside that window, so their first utterance is replaced by silence. Setting this to zero stops the erasure but lets late-subscribing carrier echo trigger a false interruption. The fix is a transport-specific 2.0 s warm-up for SIP calls.

Symptoms

A one-word first answer like "yes", "speaking", or "hello" never reaches the agent on telephony legs. Longer first answers survive, but their opening word is clipped from the transcript.

Browser calls into the exact same agent work perfectly, which typically sends teams looking at the model instead of the call setup.

If you set the echo-cancellation warm-up to zero to fix the missing word, the agent immediately starts interrupting its own greeting. The call begins with the person asking whether anyone is there, because the agent stopped talking mid-sentence.

On outbound PSTN calls, the first thing the person says might be missing entirely. The agent talks over them, asks something they already answered, or the whole conversation runs one turn out of step.

What is actually happening

The echo-cancellation warm-up substitutes silence for the far-end audio during the agent's first turn. This erasure prevents the agent from hearing its own voice arriving back through the echo path.

The stock window is 3.0 s, which is sized for browser audio. On a phone leg, the person answers inside that 3.0 s window. Their first utterance is replaced by silence before anything downstream can see it.

Zero is the tempting alternative value. It fails in production rather than in testing because carrier echo depends entirely on the path and only shows up on some routes. Setting the window to zero removes the erasure entirely. Late-subscribing carrier echo reaches the input, the agent treats its own greeting as a barge-in, and the turn wedges.

There is a second race condition hiding behind the warm-up. The SIP participant attribute that reports the leg as active flips a beat before the callee's audio publication is actually subscribed by the agent worker. Greet on that active signal, and the opener plays into an input pipe nothing is reading yet. The first answer arrives before anything is listening.

How to confirm it

Log every first user turn that arrives empty, tagged with the transport. Expect SIP and browser to show entirely different failure rates; a SIP-exclusive failure points straight to the transport boundary.
Measure the time from the end of the greeting to the first inbound audio frame that survives the warm-up. Expect this gap to match your current warm-up window if the cancellation logic is eating the audio.
Count barge-in events attributed to the agent's own greeting. Expect zero on a healthy configuration; a high count means the warm-up is either missing or too short to cover the carrier echo path.
Record the milliseconds between the leg reporting active and the first audio track subscription, per call. Expect a measurable gap. If your greeting fires before this gap closes, the agent is speaking before it can hear.

How it works

The fix is making the warm-up transport-dependent instead of global. I settled on a 2.0 s echo-cancellation warm-up on SIP legs. This is long enough to cover the carrier echo path and short enough that a normal phone answer lands outside it.

I kept the stock 3.0 s warm-up on WebRTC, where the browser's own echo cancellation changed the problem. I implemented this as a single override point that returned no override at all for non-SIP sessions. Because the override returned nothing, the browser path stayed byte-equivalent to stock. Any regression is unambiguously attributable to the SIP branch.

For the audio subscription race, the fix is gating the greeting on an audio publication actually being subscribed, not on the leg reporting active. The active signal only arms a bounded grace window of 2.5 s. Track subscription preempts that window the moment it fires.

The 2.5 s grace window is a fallback, not the primary mechanism. Its only job is to stop a call hanging forever when the subscription event never arrives at all. I clamped this grace to whatever remained of the overall 45 s ringing and answer budget, so a late active signal could never turn a would-be success into a timeout. I removed both event handlers in a finally block so a retry could not stack listeners. If the handler registrations themselves failed, I failed open and greeted rather than hanging the call forever.

What I would check first

Verify whether the missing first turn happens on browser calls or only on SIP. A failure restricted to SIP rules out the model, isolating the issue to transport-specific echo or timing.
Check what event triggers your agent's greeting. Finding a greeting triggered by the call reporting active, rather than the track being subscribed, guarantees a missing first answer regardless of echo cancellation.
Look at your agent's interruption metrics. Finding an agent that interrupts itself during the greeting indicates either a warm-up value of zero or an unhandled carrier echo path.

Why the subscription gate is not enough

This is the second half of the outbound first-answer bug. The subscription gate decides when the greeting starts; the warm-up decides whether the reply to it is heard at all.

Teams that fix only the subscription gate observe that the caller's first word is still missing. They invariably conclude the gate did not work. Fixing one without the other leaves the symptom entirely in place.

The two obvious alternatives for the subscription gate are both wrong. A fixed sleep after the active signal is a guess at a distribution you have not measured. If the sleep is too short, the symptom stays. If it is too long, you have added dead air to every call and made the agent feel slow. Greeting on participant-connected is worse, because that fires earlier still.

Instrument the per-call gap before changing anything. The time between the two signals is the single number that tells you whether this is your bug, and it costs one timestamp subtraction per call to collect.

Shutting down cleanly

Ending a session exposes another set of timing failures that present as truncated audio or resource leaks. When a call ends, the closing line is often truncated mid-word. Separately, you might find idle workers attached to empty rooms long after the call finishes. This presents as idle workers and reads as a load problem rather than a leak.

A fixed sleep before hangup is a guess at the length of a variable utterance. Too short and it truncates the goodbye; too long and every call pays for dead air. The standard session close call does not drain by default. Calling it while speech is playing interrupts that speech by design.

I replace the sleep with an await on the current speech's playout, capped at 6 s. The cap ensures a wedged synthesiser cannot hold the call open indefinitely. End the turn by letting playout finish naturally.

Waiting for playout lengthens the window in which the far end can hang up. This makes a precedence rule on the call's end-reason necessary rather than optional.

For the leaked rooms, the mechanism is a cancelled teardown. A room delete issued inside a teardown that the outer timeout is about to cancel is cancelled along with it. The timeout fires, the coroutine is cancelled mid-await, and the room is left behind.

I shield the room delete from cancellation and give it its own 5 s timeout so it survives the enclosing teardown being cancelled. Finally, I treat a not-found response on delete as a success. Someone else having already deleted the room is the exact outcome you wanted.

Common questions

Why does the voice agent not hear the caller say yes?

The echo-cancellation warm-up window deletes the first few seconds of inbound audio. A short first answer falls entirely inside this window and never reaches transcription.

Why does the agent interrupt its own greeting on a phone call?

Carrier echo returns the agent's voice back to its own input. Without an adequate echo-cancellation warm-up, the agent hears itself and treats its own greeting as a user barge-in.

How do I stop the agent greeting before the caller is ready?

Gate the greeting on the audio track being actually subscribed, not just the call reporting active. The active signal flips a beat before the audio pipe is ready to receive input.

Why is the agent's last sentence cut off when the call ends?

Closing the session without awaiting speech playout interrupts the synthesiser. You must await the playout with a hard timeout cap before issuing the hangup command.

If the boundary this note describes is one you would rather own, a self-hosted deployment you control moves it inside your own infrastructure.

Canonical URL: https://www.dreamchasers.lt/blog/echo-cancellation-warm-up-eats-a-short-first-answer-on-phone-calls