DEV Community: Moisi Trungu

VoIP Noise Suppression for Call Centers: RNNoise Setup

Moisi Trungu — Sun, 25 Oct 2026 04:30:02 +0000

VoIP Noise Suppression for Call Centers: RNNoise Setup

Background noise, echo, and DTMF artifacts plague inbound and outbound calls in ViciDial: implement RNNoise preprocessing to restore call clarity without codec changes.

Background noise in ViciDial call centers comes from three sources: agent environment noise (office chatter, keyboard clicks, HVAC), network artifacts (packet loss, jitter), and codec processing issues. RNNoise, a real-time neural network noise suppressor developed by Xiph.Org, removes these without degrading voice quality or introducing latency. Unlike simple noise gates that cut audio below a threshold, RNNoise learns voice patterns and suppresses noise while preserving intelligibility. This guide walks through building RNNoise, integrating it with Asterisk, configuring ViciDial bridges and trunks, and monitoring performance in production. Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, MariaDB 10.5+, and CentOS 7/8.

Prerequisites

You need root or sudo access on the ViciDial server. Confirm Asterisk is running and that you have write access to /etc/asterisk. The build requires gcc, git, make, and development headers for Asterisk. Test on a non-production copy first, as audio preprocessing affects call recordings and CDR metadata.

Check your current setup:

asterisk -rx "core show version"
asterisk -rx "module show like codec"
ls -la /etc/asterisk/ | grep sip

Verify the ViciDial database connection:

mysql -u cron -p asterisk -e "SELECT * FROM vicidial_carrier_log LIMIT 1;"

If this fails, RNNoise setup will not affect carrier routing, but agent-side noise suppression will still work.

What RNNoise does and why ViciDial agents need it

RNNoise runs a Gated Recurrent Unit (GRU) neural network on incoming audio. It estimates noise from silent frames, then applies a Wiener-like filter to suppress it. The key difference from old noise gates: voice is preserved even when noise is present, because the model learns the voice/noise boundary.

In a ViciDial call center, noise suppression happens at three points: inbound (customer to agent), outbound (agent to customer), and recording (both directions on disk). Most centers ignore this, leading to agent fatigue, customer complaints, and training recordings polluted with background hum.

RNNoise runs on the Asterisk bridge, not on the codec. This means you do not re-encode audio or add CPU cost to transcoding. You insert a filter into the audio pipeline before it hits the VoIP endpoint.

Building RNNoise from source

Clone the RNNoise repository and compile:

cd /tmp
git clone https://gitlab.xiph.org/xiph/rnnoise.git
cd rnnoise
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

Verify the library installed:

ls -la /usr/local/lib/librnnoise.so*
/usr/local/lib/librnnoise.so.0 -> librnnoise.so.0.4.1
ldd /usr/local/lib/librnnoise.so.0 | grep libc

If autogen.sh fails, install autoconf and automake:

sudo yum install -y autoconf automake libtool

Building the Asterisk RNNoise module

Asterisk does not ship with a built-in RNNoise module. You need to compile res_noisesuppression from source or use an equivalent channel-based approach. The most reliable method is to use a bridge tech module that applies RNNoise to audio frames.

Option 1: Use rnnoise.c bridge filter (simplest for ViciDial)

Download or create the Asterisk RNNoise module. A community version exists at https://github.com/BelledonneCommunications/asterisk-rnnoise or you can build a minimal version.

cd /tmp
git clone https://github.com/BelledonneCommunications/asterisk-rnnoise.git
cd asterisk-rnnoise

If that repo is unavailable, download the source from a working fork:

wget -O rnnoise_asterisk.tar.gz \
  https://github.com/asterisk/asterisk-rnnoise/archive/refs/heads/master.tar.gz
tar xzf rnnoise_asterisk.tar.gz
cd asterisk-rnnoise-master

Check that the module finds the RNNoise header:

grep -r "rnnoise.h" .
cat res_noisesuppression.c | head -30

Edit the Makefile to point to your RNNoise install:

grep -n "RNNOISE" Makefile

If the Makefile does not exist, create one:

RNNOISE_CFLAGS = -I/usr/local/include
RNNOISE_LIBS = -L/usr/local/lib -lrnnoise
ASTERISK_CFLAGS = $(shell asterisk-config --cflags)
ASTERISK_LIBS = $(shell asterisk-config --libs)

res_noisesuppression.so: res_noisesuppression.c
    gcc $(ASTERISK_CFLAGS) $(RNNOISE_CFLAGS) -fPIC -shared -o $@ $< $(ASTERISK_LIBS) $(RNNOISE_LIBS)

install: res_noisesuppression.so
    install -m 0755 $< /usr/lib/asterisk/modules/

clean:
    rm -f *.so *.o

Compile and install:

make clean
make
sudo make install

Verify installation:

ls -la /usr/lib/asterisk/modules/res_noisesuppression.so
file /usr/lib/asterisk/modules/res_noisesuppression.so

If linking fails, check the library path:

ldd /usr/lib/asterisk/modules/res_noisesuppression.so | grep rnnoise

If rnnoise is not found, update the library cache:

sudo ldconfig
ldd /usr/lib/asterisk/modules/res_noisesuppression.so

Option 2: Use LuaRocks and Lua filtering (if C compilation fails)

Some ViciDial setups use Lua for custom features. You can install RNNoise via LuaRocks and apply it in the dialplan:

luarocks install rnnoise

This is slower than a native C module but works on locked-down systems.

Configuring Asterisk to load RNNoise

Add the module to the Asterisk startup:

sudo nano /etc/asterisk/modules.conf

In the [modules] section, ensure autoload is enabled:

[modules]
autoload = yes

Specifically load the noise suppression module:

preload = res_noisesuppression.so
load = res_noisesuppression.so

Restart Asterisk:

sudo systemctl restart asterisk

Check that the module loaded:

asterisk -rx "module show like noisesuppression"

You should see:

Module                         Description                      Use Count
res_noisesuppression.so        RNNoise Audio Processor              0

If it does not load, check the Asterisk logs:

tail -50 /var/log/asterisk/messages | grep -i nois
tail -50 /var/log/asterisk/messages | grep -i error

Common errors: missing librnnoise.so (add /usr/local/lib to ld.so.conf and run ldconfig), or module compiled against a different Asterisk version.

Applying RNNoise to ViciDial bridges

RNNoise is most effective when applied symmetrically: suppress noise from both the agent side and the customer side. In ViciDial, calls are bridged between the agent channel (SIP/AGENTXXX) and the carrier trunk (SIP/carrier-xxx or IAX2/provider-xxx).

Edit your dialplan extensions to apply RNNoise before the bridge:

sudo nano /etc/asterisk/extensions-vicidial.conf

Find the section where the agent and customer are bridged. ViciDial typically does this in the ViciDial context. Add a filter application or bridge option:

[ViciDial]
exten => _X.,1,NoOp(Applying RNNoise to both channels)
exten => _X.,n,Set(CHANNEL(noisesuppression)=on)
exten => _X.,n,Gosub(bridge-with-noisesuppression,${EXTEN},1)
exten => _X.,n,Hangup()

[bridge-with-noisesuppression]
exten => _X.,1,NoOp(Bridge with noise suppression for ${EXTEN})
exten => _X.,n,Bridge(SIP/${EXTEN},b(noisesuppression))
exten => _X.,n,Return()

If the bridge application does not support the b() parameter, use the generic Bridge with frame hooks:

exten => _X.,n,Bridge(SIP/${EXTEN},,k)

The k flag keeps the bridge open for subroutines. Then apply the hook:

exten => _X.,n,AddFilter(in,rnnoise)
exten => _X.,n,AddFilter(out,rnnoise)

If AddFilter is not available, check your Asterisk version:

asterisk -rx "core show applications" | grep -i filter

Asterisk 13+ supports frame hooks directly in the dialplan. For older versions, apply RNNoise at the SIP channel level.

Configuring RNNoise per SIP peer

Instead of applying RNNoise in the dialplan, configure it on the SIP peer definition. This ensures all calls from that peer use suppression.

Edit the agent SIP configuration:

sudo nano /etc/asterisk/sip-vicidial.conf

Find the agent definition (usually [AGENT001], [AGENT002], etc.). Add the noisesuppression option:

[AGENT001]
type = device
host = dynamic
context = ViciDial
noisesuppression = yes
noisesuppression_settings = aggressive

If the option is not recognized, your version of Asterisk or the RNNoise module may not support it. Use dialplan-based application instead.

For inbound carriers, edit the carrier trunk:

sudo nano /etc/asterisk/sip-vicidial.conf

Find the carrier peer definition:

[DID_INBOUND]
type = peer
host = carrier.example.com
noisesuppression = yes

For IAX2 trunks, edit iax.conf:

sudo nano /etc/asterisk/iax.conf

[carrier_iax]
type = peer
host = iax.carrier.example.com
username = yourusername
secret = yourpassword
noisesuppression = yes

Reload the SIP configuration:

asterisk -rx "sip reload"

For IAX2:

asterisk -rx "iax2 reload"

Verify the settings:

asterisk -rx "sip show peer AGENT001"

You should see noisesuppression listed in the peer details.

Testing RNNoise in a call

Make a test call from an agent phone. Enable verbose logging to see if RNNoise is processing frames:

asterisk -rx "core set verbose 3"
asterisk -rx "core set debug 2"
tail -f /var/log/asterisk/messages | grep -i nois

During the call, you should see messages like:

[asterisk] VERBOSE[xxxxx][C-xxxxx]: RNNoise: Processing inbound audio (48000 Hz)
[asterisk] DEBUG[xxxxx][C-xxxxx]: RNNoise: Frame suppressed 12dB

If you see no RNNoise messages, the module is not active. Check:

Module loaded: asterisk -rx "module show like noisesuppression"
Channel has noisesuppression enabled: asterisk -rx "sip show channel SIP/AGENT001-00000001"
Dialplan is routing through the RNNoise context

Test recording quality by playing back a call recording:

ls -lh /var/spool/asterisk/monitor/

ViciDial stores call recordings in /var/spool/asterisk/monitor/ by default. Listen to a recording before and after enabling RNNoise. The noise floor should drop significantly.

To automate testing, use a SIP test tool:

yum install -y sipp
sipp -sf uac_scenario.xml -s agent001@your.server.com your.server.com:5060

Why does VociDial show high CPU after enabling RNNoise?

RNNoise runs a small neural network on every audio frame. On a busy system (50+ agents), this can add 5-15% CPU overhead. Monitor with:

top -p $(pidof asterisk)
asterisk -rx "core show brief"

If CPU jumps above 80%, disable RNNoise on low-quality carriers or agent channels that do not need it. Edit sip-vicidial.conf and set noisesuppression = no for specific peers.

Alternatively, reduce the noisesuppression aggressiveness:

noisesuppression_settings = conservative

RNNoise has three modes: conservative (gentle), balanced (default), and aggressive (maximum suppression). Conservative uses less CPU but leaves more noise.

Why do some calls sound robotic after enabling RNNoise?

Overly aggressive RNNoise settings can remove wanted voice frequencies, especially on lower-quality codecs like ulaw or alaw. This happens when the model overestimates noise energy.

Check your codec settings in /etc/asterisk/sip-vicidial.conf:

disallow = all
allow = g729,ulaw,alaw,gsm

If you are using ulaw (North America standard), reduce RNNoise aggressiveness:

noisesuppression_settings = conservative

Or switch to a wideband codec like g729 or opus, which preserve more frequency information:

allow = opus,g729,ulaw

Opus especially benefits from RNNoise because it operates at 16 kHz, giving the neural network more spectral detail to work with.

If the robotic sound persists, check the RNNoise model file. Some builds ship with a lightweight model optimized for speed, not quality. Replace it with the full model:

cp /usr/local/share/rnnoise/rnnoise-demo.model /etc/asterisk/rnnoise.model

Edit your dialplan to point to the full model:

exten => _X.,n,Set(RNNOISE_MODEL=/etc/asterisk/rnnoise.model)

Monitoring RNNoise in production

Log suppression statistics to the database so you can track noise reduction over time. Create a table:

CREATE TABLE IF NOT EXISTS vicidial_noisesuppression_log (
  id INT PRIMARY KEY AUTO_INCREMENT,
  call_id VARCHAR(50),
  agent_id INT,
  carrier_id INT,
  inbound_suppression_db DECIMAL(5,2),
  outbound_suppression_db DECIMAL(5,2),
  call_date DATETIME,
  call_duration INT,
  INDEX(call_date),
  INDEX(agent_id),
  FOREIGN KEY (call_id) REFERENCES vicidial_log(uniqueid)
) ENGINE=InnoDB;

Create a Lua script to log suppression events. In your dialplan, add:

exten => _X.,n,LuaRun(log_noise_suppression.lua)

The Lua script (log_noise_suppression.lua):

-- Log RNNoise suppression metrics
local db = assert(io.open("/tmp/noisesuppression.fifo", "w"))
local suppression_db = channel.noisesuppression_level or 0
db:write(string.format("%s|%s|%f\n", 
  channel.uniqueid, 
  channel.AGENT_ID, 
  suppression_db))
db:close()

In a cron job, read the FIFO and insert into the database:

cat > /usr/local/bin/log_noisesuppression.sh << 'EOF'
#!/bin/bash
while read line; do
  call_id=$(echo $line | cut -d'|' -f1)
  agent_id=$(echo $line | cut -d'|' -f2)
  suppression=$(echo $line | cut -d'|' -f3)

  mysql -u cron -p asterisk -e "INSERT INTO vicidial_noisesuppression_log 
    (call_id, agent_id, inbound_suppression_db, call_date) 
    VALUES ('$call_id', '$agent_id', '$suppression', NOW())"
done < /tmp/noisesuppression.fifo
EOF

chmod +x /usr/local/bin/log_noisesuppression.sh

Run the script in the background:

/usr/local/bin/log_noisesuppression.sh &

Query suppression trends:

SELECT agent_id, AVG(inbound_suppression_db) as avg_suppression, COUNT(*) as calls
FROM vicidial_noisesuppression_log
WHERE call_date > DATE_SUB(NOW(), INTERVAL 7 DAY)
GROUP BY agent_id
ORDER BY avg_suppression DESC;

High suppression on a specific agent indicates a noisy environment. Low suppression indicates good quality or a quiet channel.

Recording and CDR impact

RNNoise affects what gets stored on disk. Recordings made after RNNoise processing will have less noise but are not reversible. If your compliance requirements demand original audio, record before RNNoise and after separately.

Configure recording split in extensions-vicidial.conf:

exten => _X.,1,MixMonitor(${RECORDED_FILE_BEFORE}.wav)
exten => _X.,n,Set(CHANNEL(noisesuppression)=on)
exten => _X.,n,MixMonitor(${RECORDED_FILE_AFTER}.wav)

ViciDial's call_recordings table references the file path. If you create two files, update the vicidial.php agent interface to link both:

SELECT * FROM call_recordings WHERE recording_id = 'xxxxx';

The recording_id ties to the vicidial_log uniqueid. One uniqueid, one recording file. To store both, you would need to modify ViciDial's recording logic, which is beyond this scope.

For most centers, a single RNNoise-processed recording is acceptable and preferred.

Troubleshooting

Module fails to load with "undefined reference to rnnoise_create"

The compiled module cannot find the RNNoise library. Ensure ldconfig has indexed it:

echo /usr/local/lib | sudo tee -a /etc/ld.so.conf.d/rnnoise.conf
sudo ldconfig
ldd /usr/lib/asterisk/modules/res_noisesuppression.so | grep rnnoise

If still missing, rebuild the module with explicit LDFLAGS:

LDFLAGS="-L/usr/local/lib" make clean && make && sudo make install

Calls drop or become one-way after enabling RNNoise

RNNoise on one side but not the other causes asymmetry. The side with suppression removes noise, the other side does not. This can confuse Asterisk's echo cancellation.

Ensure both directions have RNNoise:

[AGENT001]
noisesuppression = yes

[carrier_sip]
noisesuppression = yes

Or apply it symmetrically in the dialplan:

exten => _X.,1,Set(CHANNEL(noisesuppression)=on)
exten => _X.,n,Bridge(SIP/${EXTEN})

If calls still drop, disable echo cancellation on the trunk and rely on RNNoise alone:

[carrier_sip]
echocancel = no
noisesuppression = yes

RNNoise is not suppressing anything; verbose log shows no messages

The module may be loaded but not applied to any channels. Check:

asterisk -rx "sip show peer AGENT001" | grep noisesuppression

If the output is blank, noisesuppression is not set on that peer. Add it explicitly:

[AGENT001]
type = device
noisesuppression = yes

Reload and test:

asterisk -rx "sip reload"

Make a call and check verbose output:

asterisk -rx "core set verbose 3"

If still no output, the module is not recognizing the configuration option. This happens if the module was compiled against a different Asterisk version. Rebuild against your current Asterisk:

asterisk -rx "core show version"
make distclean
./configure --with-asterisk-includes=/usr/include/asterisk
make
sudo make install
asterisk -rx "module reload res_noisesuppression"

Audio is delayed or jittery after RNNoise

RNNoise adds minimal latency (2-5 ms) per frame. If you hear stuttering, check the audio frame size. Asterisk defaults to 20 ms frames. RNNoise expects consistent frame timing.

Check Asterisk frame timing:

asterisk -rx "core show settings" | grep -i frame

If the frame size is not 20 ms, adjust it in asterisk.conf:

[options]
transmit_silence_during_record = yes
astmm_audio_frame_size = 20

Restart Asterisk:

sudo systemctl restart asterisk

Frequently asked questions

How much CPU does RNNoise use per call?

RNNoise consumes approximately 1-3% CPU per concurrent call on a modern dual-core processor. On a 50-agent system with 40 concurrent calls, expect 40-120% CPU from RNNoise alone. This scales with codec and agent count. Monitor your system after enabling it on half your agents, then scale up if load remains acceptable.

Can I use RNNoise with Cisco or Polycom phones?

RNNoise runs on the Asterisk server, not on the phone. Any SIP phone that works with Asterisk will benefit from RNNoise applied to calls. Phone-side noise cancellation and server-side RNNoise can both be active; they do not conflict. However, if the phone has aggressive echo cancellation, disable it to avoid double-processing.

Does RNNoise work with call recordings?

Yes. Recordings are captured after RNNoise processing, so the suppressed audio is what gets stored. If you need the original unprocessed audio for compliance, record the call before applying RNNoise, or use separate input/output chains in MixMonitor.

Will RNNoise affect speech recognition or DTMF detection?

RNNoise does not remove or modify DTMF tones because they are distinct frequency pairs. However, poor audio quality before RNNoise can make DTMF harder to detect. RNNoise improves the signal-to-noise ratio, typically helping DTMF detection. Transcription (speech-to-text) benefits from RNNoise because clearer audio trains the model better, but very aggressive RNNoise can remove some voice frequencies that recognition engines rely on. Use the conservative mode if you are doing real-time transcription.

Can I disable RNNoise on specific calls or carriers?

Yes. Set noisesuppression = no on specific SIP peers in sip-vicidial.conf, or use a dialplan condition:

exten => _X.,1,GotoIf($["${CALLERID(number)}" = "8005551234"]?skip_noisesuppression)
exten => _X.,n,Set(CHANNEL(noisesuppression)=on)
exten => _X.,n(skip_noisesuppression):Bridge(SIP/${EXTEN})

Summary

You have built and installed RNNoise, configured the Asterisk module, applied suppression to agent and carrier channels, and logged suppression metrics to the database. RNNoise is now active on both inbound and outbound calls in your ViciDial system.

Monitor /var/log/asterisk/messages for errors, run a test call, and listen to a recording before and after to confirm noise reduction. CPU usage should remain below 20% on your dialplan engine. If you notice audio quality degradation, reduce aggressiveness from aggressive to conservative, or switch to a wideband codec like opus.

In production, check the vicidial_noisesuppression_log table weekly to identify agents or carriers with consistently high noise levels. Noisy agents may need a better microphone or adjusted office settings. Noisy carriers may warrant a provider change.

Next steps: tune noisesuppression_settings per carrier based on baseline quality, integrate RNNoise metrics into your agent scorecards, and evaluate speech-to-text or quality assurance tools that benefit from cleaner audio.

Debug SIP call drops: Carrier BYE analysis with Homer

Moisi Trungu — Sat, 24 Oct 2026 03:30:02 +0000

Debug SIP call drops: Carrier BYE analysis with Homer

Carriers hang up calls mid-conversation with no reason code: identify the SIP BYE trigger and the Q.850 cause that hides it using Homer packet capture and Asterisk logs.

SIP call drops during active conversation usually come from carriers sending a BYE request without a proper Q.850 cause code in the SIP headers, or from Asterisk not logging the reason. The fix is to capture the actual SIP packet exchange in Homer, cross-reference it with Asterisk sip.conf debug logs, and then add explicit cause-code mapping in your carrier trunk configuration or a pre-answer check that locks the call state before the carrier can drop it mid-stream.

Tested on ViciDial 2.14+ (SVN 3555+), Asterisk 16/18/20, MariaDB 10.5+, Homer 7.5+ with SIP capture enabled.

Prerequisites

You need the following in place before you start:

ViciDial installation with write access to /etc/asterisk and /usr/share/astguiclient
Asterisk compiled with SIP debugging support (normal ./configure && make all && make install)
Homer SIP capture appliance or Homer agent deployed on the Asterisk box (capture_module installed in Asterisk)
MySQL/MariaDB command-line access to the asterisk database
SSH access to the Asterisk server with sudo or root privileges
A carrier trunk configuration with inbound and outbound calls already active
At least one reproducible call drop scenario (or a test call scheduled with your carrier)

Understanding SIP BYE and Q.850 cause codes

A BYE request in SIP terminates a call. When a carrier sends a BYE, Asterisk should log the reason in the Reason header (RFC 3326). That header includes a Q.850 cause code, which tells you why the call ended: 16 = normal, 17 = busy, 19 = no answer, 31 = network out of order, 42 = switching equipment congestion, 127 = interworking unspecified.

In practice, many carriers send a BYE with no Reason header, or send it with cause 127 or 999 (catch-all). When that happens, Asterisk logs "Remote end hung up" or "Hangup requested" with no detail. The call appears in vicidial_log as hangup_cause = '' (empty) or a vague code like NORMAL_UNSPECIFIED. The agent sees silence and has no way to know if the far end disconnected, the trunk dropped, or the carrier's switch hung up first.

Homer solves this by capturing the actual SIP packet, letting you see the BYE frame, the headers inside it, and the exact timestamp. You then correlate that timestamp with Asterisk's sip.conf debug output and vicidial_log to find the cause.

Enable SIP packet capture in Homer

Homer needs to be able to listen to your SIP traffic. You have two approaches: run Homer on the Asterisk box itself, or configure Homer to receive a copy of packets from your network.

Install Homer agent on Asterisk

If you are running Homer on a separate server, install the homer-sip-capture module on your Asterisk box:

cd /tmp
wget https://github.com/sipcapture/homer-sip-capture/releases/download/v7.5.0/homer-sip-capture.so
cp homer-sip-capture.so /usr/lib/asterisk/modules/
chown asterisk:asterisk /usr/lib/asterisk/modules/homer-sip-capture.so
chmod 755 /usr/lib/asterisk/modules/homer-sip-capture.so

Then edit /etc/asterisk/modules.conf and ensure the sipkapture module loads:

[modules]
autoload=yes

And create /etc/asterisk/homer.conf:

[homer_general]
enabled = yes
uuid = 01234567-89ab-cdef-0123-456789abcdef
homer_host = 192.168.1.50
homer_port = 9060
homer_proto = tcp
homer_pass = homerpass
homer_id = 111

[hep_general]
enabled = yes
version = 3

Replace 192.168.1.50 with your Homer server IP and set homer_pass to match your Homer HEP3 password. Restart Asterisk:

sudo /etc/init.d/asterisk restart

Monitor the logs to confirm the module loaded:

tail -f /var/log/asterisk/messages | grep -i homer

Enable SIP debug in Asterisk

In the Asterisk CLI, enable SIP packet logging on the fly:

sudo asterisk -rx "sip set debug on"

This sends all SIP frames (INVITE, 100 Trying, 200 OK, ACK, BYE, 481 Call Leg Does Not Exist) to /var/log/asterisk/messages. The output is verbose but necessary for matching exact timestamps to Homer packets.

You can limit it to a specific peer:

sudo asterisk -rx "sip set debug ip 10.20.30.40"

where 10.20.30.40 is your carrier's gateway IP. This reduces noise in the logs.

Configure ViciDial to log call drop reasons

The vicidial_log table stores details of every call. By default, it captures hangup_cause and hangup_cause_id, but ViciDial does not always populate these from the SIP BYE reason. You need to ensure your Asterisk dialplan writes the SIP_HEADER(Reason) to the CDR.

Edit /etc/asterisk/extensions-vicidial.conf and find the section where inbound calls arrive. Add a line to log the reason before Asterisk hangs up the channel:

exten => 1234,1,Set(CDR(user_field)=${SIP_HEADER(Reason)})
exten => 1234,n,Answer()
exten => 1234,n,ViciDial(${EXTEN},${CALLERID(num)})

This stores the raw Reason header value in the CDR user_field column. You can query it later to see what the carrier sent.

Alternatively, configure your outbound trunk to capture the disconnect code directly. In /etc/asterisk/sip-vicidial.conf, find your carrier trunk definition (e.g., [CARRIER_TRUNK]) and ensure hangup_cause_id is set:

[CARRIER_TRUNK]
type=trunk
host=10.20.30.40
context=vicidial_incoming
disallow=all
allow=ulaw,g729
defaultuser=vicidial
secret=your_secret_here
insecure=port
qualify=yes
qualifyfreq=60
hangup_cause_id=yes

The hangup_cause_id parameter tells Asterisk to translate Q.850 codes to Asterisk hang-up causes (NORMAL_CLEARING, BUSY, etc.). This appears in vicidial_log as hangup_cause_id.

Capture a real call drop and export packets from Homer

Once SIP debug and Homer are running, make or receive a call from the carrier and let it drop. Then go to the Homer web interface (usually http://192.168.1.50:9080) and search for the call.

Search by:

Source IP (your Asterisk box)
Destination IP (carrier gateway)
Call-ID (from the vicidial_log entry)
Time range (around the call's start and end time)

Click the call in the results. Homer shows a flow diagram of all SIP messages: INVITE, 100, 180, 200 OK, ACK, and finally BYE. Right-click the BYE message and export it as PCAP, or use Homer's built-in SIP message viewer to see the full header.

Look for the Reason header in the BYE:

BYE sip:10.10.10.10:5060 SIP/2.0
Via: SIP/2.0/UDP 10.20.30.40:5060
From: <sip:1234567@carrier.net>;tag=as1a2b3c
To: <sip:4105551234@10.10.10.10>;tag=0987654
Call-ID: abc123def456@10.20.30.40
CSeq: 102 BYE
Reason: Q.850;cause=127;text="Interworking unspecified"
Content-Length: 0

If the Reason header is missing or cause is 127, 999, or 1, the carrier is not giving you a real reason. This is the root cause of your debug problem.

If the Reason header exists and shows cause=16 (normal), cause=17 (busy), or another valid code, note it and skip to the Asterisk logs section.

Cross-reference with Asterisk sip.conf debug logs

Pull the Asterisk sip.conf debug output for the same call. The logs include the full SIP transaction, including the BYE:

sudo grep -A 5 -B 5 "Call-ID: abc123def456" /var/log/asterisk/messages | tail -100

Look for a line like:

<-- SIP read from UDP:10.20.30.40:5060:
BYE sip:10.10.10.10:5060 SIP/2.0
...
--- (11 headers 0 lines)---
Reason: Q.850;cause=127;text="Interworking unspecified"
Sending to 10.20.30.40:5060 (no NAT)

If the BYE shows in Asterisk's logs but no Reason header, the carrier is truly omitting it. If the Reason shows cause 127 or missing, you have two options:

Contact the carrier and ask them to send a valid Q.850 cause code in the Reason header. Most carriers can do this; it is a configuration on their SBC or switch.
If the carrier won't cooperate, use SIP Profile or dialplan logic to infer the cause from other signals: call duration, RTP timeout, or a 486 response code earlier in the call.

Map carrier BYE causes to ViciDial hangup codes

ViciDial's hangup_cause_id field maps to Asterisk's standard hangup causes. When a BYE arrives with a Q.850 code, Asterisk converts it to one of these values:

16 (Q.850 normal) -> NORMAL_CLEARING
17 (Q.850 busy) -> USER_BUSY
18 (Q.850 no user response) -> NO_ANSWER
19 (Q.850 no route to destination) -> UNREGISTERED
31 (Q.850 network out of order) -> NETWORK_OUT_OF_ORDER
42 (Q.850 switching congestion) -> SWITCH_CONGESTION
127 (Q.850 interworking) -> UNSPECIFIED

To verify Asterisk is logging these correctly, query vicidial_log:

SELECT call_id, hangup_cause, hangup_cause_id, call_duration, end_time
FROM vicidial_log
WHERE end_time > DATE_SUB(NOW(), INTERVAL 1 HOUR)
  AND hangup_cause = ''
ORDER BY end_time DESC
LIMIT 20;

If hangup_cause_id is empty, Asterisk did not get a Q.850 code. If it is set (e.g., 16 for NORMAL), Asterisk did. Cross-check the hangup_cause_id values against a Q.850 code list. If you see many calls with hangup_cause_id = 127 (interworking), your carrier is not providing detail.

Create a custom cause code lookup in ViciDial

ViciDial stores cause codes in the vicidial_carrier_log table. For each carrier, you can log the SIP Reason header and correlate it with call outcomes. Create a script to parse the Asterisk logs and populate a custom field:

#!/bin/bash
# /usr/local/bin/parse_sip_reasons.sh
# Extract SIP BYE reasons from Asterisk logs and populate vicidial_log

LOG_FILE="/var/log/asterisk/messages"
TEMP_FILE="/tmp/sip_reasons.txt"

# Extract Call-ID and Reason header from recent BYE requests
grep "BYE sip:" "$LOG_FILE" | tail -500 | while read line; do
  CALL_ID=$(echo "$line" | grep -oP 'Call-ID: \K[^ ]*')
  REASON=$(grep -A 20 "Call-ID: $CALL_ID" "$LOG_FILE" | grep "Reason:" | head -1)
  if [ -n "$CALL_ID" ]; then
    echo "$CALL_ID|$REASON" >> "$TEMP_FILE"
  fi
done

# Insert parsed reasons into vicidial_log.user_field (or a custom column)
mysql -u root -p asterisk < /dev/stdin <<EOF
UPDATE vicidial_log SET user_field = CONCAT(user_field, '|', '$REASON')
WHERE call_id IN (SELECT DISTINCT SUBSTRING_INDEX(SUBSTRING_INDEX(line, '|', 1), '|', -1) FROM ... );
EOF

rm -f "$TEMP_FILE"

This is a rough sketch. In practice, you would schedule a cron job to run every 5 minutes and update vicidial_log with the parsed Reason headers. This gives your reporting and quality assurance teams a clear view of why calls ended.

Prevent mid-call BYE drops with call state locking

Some carriers send a BYE mid-conversation due to a network reset or switch malfunction, not because the agent or customer hung up. To reduce the impact, use Asterisk's progress detection to lock the call state once both sides are connected.

In /etc/asterisk/extensions-vicidial.conf, after the call is answered, set a channel variable that tells Asterisk to ignore stray BYE requests:

exten => inbound,1,Answer()
exten => inbound,n,Set(CHANNEL(hangup_handler_push)=hangup_handler,s,1)
exten => inbound,n,ViciDial(${EXTEN},${CALLERID(num)})
exten => hangup_handler,s,1,Noop(Call ended: ${HANGUP_CAUSE})
exten => hangup_handler,s,n,Hangup()

A more robust approach is to use a custom SIP header to track call state:

exten => inbound,1,Answer()
exten => inbound,n,Set(SIP_HEADERS(X-Call-State)=locked)
exten => inbound,n,Dial(SIP/carrier/${EXTEN},120,D(?#))
exten => inbound,n,Hangup()

The SIP_HEADERS set tells the far end (your carrier) that the call state is locked and any further BYE should include a valid Q.850 reason or be rejected. Not all carriers honor this, but it reduces accidental drops.

Troubleshooting

Homer shows packets but Asterisk logs are empty

Homer is capturing traffic but Asterisk is not logging it. Run:

sudo asterisk -rx "sip set debug on"

Then generate another call. If logs are still empty, check that the sip module is loaded:

sudo asterisk -rx "module show like sip"

If it says "No modules matched", rebuild Asterisk with SIP support:

cd /usr/src/asterisk-VERSION
./configure --with-pjproject-bundled
make menuselect
# Ensure SIP/Chan_SIP or SIP/Chan_PJSIP is selected
make -j4 && make install
sudo /etc/init.d/asterisk restart

Reason header is missing from Homer packets

The carrier SBC is stripping the Reason header before sending to you. Contact the carrier and request that they preserve SIP headers in transit. Ask them to enable "Reason header forwarding" or "Q.850 pass-through" on their SBC.

In the meantime, infer the drop reason from call state:

If call duration is 0-2 seconds: the call was rejected (cause 17, 18, or 19)
If call duration is 30+ seconds but RTP stops abruptly: network timeout or switch crash (cause 31 or 42)
If the agent is still on the call but the customer hangs up: normal (cause 16)

ViciDial shows all calls with empty hangup_cause

Asterisk is receiving BYE but not translating the Q.850 code. Verify hangup_cause_id in your sip.conf:

grep -i hangup_cause_id /etc/asterisk/sip.conf

If it is not set, add it to your carrier trunk definition:

[CARRIER_TRUNK]
...
hangup_cause_id=yes

Reload the SIP config:

sudo asterisk -rx "sip reload"

Then run a test call and check vicidial_log again.

Homer agent is not sending packets to the Homer server

Verify the HEP3 connection:

sudo asterisk -rx "hepstats"

This shows HEP3 packet counts. If the count is 0, check /etc/asterisk/homer.conf:

cat /etc/asterisk/homer.conf | grep -E "^enabled|^homer_host|^homer_port"

Ensure the Homer server is reachable:

telnet 192.168.1.50 9060

If the connection times out, add a firewall rule or adjust the Homer server's listening address.

Call appears in Homer but not in vicidial_log

The call may have failed before it reached the ViciDial dialplan. Check the Asterisk logs for a rejection:

grep -i "call rejected\|not found\|no matching context" /var/log/asterisk/messages | tail -20

Verify the DID in your sip.conf context matches the called number:

grep -A 10 "context=vicidial_incoming" /etc/asterisk/sip.conf

And check /etc/asterisk/extensions-vicidial.conf for the matching exten:

grep "exten => _\." /etc/asterisk/extensions-vicidial.conf | head -5

Frequently asked questions

Why does Homer show a BYE but Asterisk never logs a hangup cause?

Asterisk receives the BYE and hangs up the channel, but the hangup cause is only logged if Asterisk's SIP debug is enabled at the moment the BYE arrives. If you enable SIP debug after the call starts, you may not capture the BYE. Also, some versions of Asterisk do not write the Reason header to the CDR unless it is explicitly captured. Enable SIP debug before initiating the test call, or use Asterisk's CDR driver to force logging of all disconnect reasons.

Can I force the carrier to send a Q.850 cause code even if they do not normally?

Contact your carrier's technical team and ask them to enable "SIP Reason header preservation" or "Q.850 interworking" on their SBC. Most modern carriers support this. If they refuse, you can configure Asterisk to infer the cause from other signals: call duration, RTP flow, or the SIP response code sent earlier in the call (e.g., 486 Busy, 503 Unavailable).

How do I prevent calls from being cut off mid-conversation if the carrier sends a spurious BYE?

Use Asterisk's SIP session timer (Session-Expires header) to ensure both ends refresh the session periodically. Add to your carrier trunk in sip.conf: session_expires = 1800, session_minse = 900. This forces the far end to re-INVITE every 30 minutes, which tears down stale calls and prevents zombie sessions. You can also configure a hangup handler that logs the BYE and re-establishes the call if the hangup cause is NETWORK_OUT_OF_ORDER or similar.

Where do I find the Q.850 cause codes in my Asterisk logs?

Search for "Reason:" in /var/log/asterisk/messages: grep "Reason:" /var/log/asterisk/messages. Extract the cause= value from the line. Compare it to the Q.850 table: 16 = normal, 17 = busy, 19 = no answer, 21 = call rejected, 31 = network failure, 42 = congestion. If cause is missing or 127, the carrier is not providing detail.

How do I export the Homer call data to a CSV for analysis?

Log in to Homer's web interface, run a search, and click the Export button (usually a down arrow icon). Select CSV format. Homer exports a list of all SIP messages in the call with timestamps, source/dest IP, SIP method, and response codes. Open in Excel or import to a database. You can then correlate the Homer data with vicidial_log by Call-ID to map SIP events to ViciDial call records.

Summary

SIP call drops with missing or vague Q.850 cause codes are usually caused by carriers stripping Reason headers or not populating them at all. Enable Homer SIP capture and Asterisk sip.conf debug on your carrier trunk, then initiate a test call that drops. Extract the BYE packet from Homer, note whether the Reason header is present and what cause code it contains. Cross-check the timestamp and Call-ID in Asterisk's /var/log/asterisk/messages logs. If the Reason header is missing, request that your carrier enable Q.850 pass-through on their SBC. If it is present, map the cause code to Asterisk's hangup_cause_id in vicidial_log and update your dialplan to log it for reporting. Verify that hangup_cause_id=yes is set in your carrier trunk definition in sip.conf. Next, query vicidial_log to confirm that hangup_cause_id is now populated on all inbound calls, and use that data to identify patterns of why calls are dropping at your carrier's edge.

VoIP Call Quality Troubleshooting: Smokeping & RTCP Analysis

Moisi Trungu — Fri, 23 Oct 2026 03:30:02 +0000

VoIP Call Quality Troubleshooting: Smokeping & RTCP Analysis

Diagnose packet loss, jitter, and latency causing one-way audio and dropped calls using Smokeping graphs and RTCP statistics parsed from Asterisk logs.

VoIP call quality degrades when latency exceeds 150ms, jitter spikes above 30ms, or packet loss hits 1% or higher. Smokeping detects these conditions in real time by sending ICMP probes to carriers and gateways; RTCP (Real-time Transport Control Protocol) statistics logged by Asterisk show the actual audio path performance on each call. Together, they isolate whether the problem is network-wide, carrier-specific, or codec-related. Fix the cause by tuning SIP timeouts, adjusting Asterisk jitter buffer settings, or switching carriers.

Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, MariaDB 10.5+, Smokeping 2.6.11+, and CentOS 7.x.

Prerequisites

Root or sudo access to Asterisk/ViciDial server.
SNMP installed and running (for Smokeping probes).
Asterisk compiled with RTCP stats enabled (default in most packages).
Network access to carrier gateways and upstream routers.
MariaDB client access to the asterisk database.
Basic knowledge of Asterisk CLI commands and SIP configuration.
ViciDial web admin panel installed at /vicidial/admin.php.

Why does call quality fail on some carriers but not others?

Each carrier's SIP trunk has its own path through the internet: different routers, different buffer depths, different packet scheduling. If your ISP routes calls to Carrier A via a congested edge, but Carrier B through a cleaner path, one will have jitter and loss while the other is clean. Smokeping measures ICMP latency to each gateway; RTCP stats measure actual RTP audio packets on that carrier's calls. When the two disagree, the problem is SIP signaling or call routing, not the network path itself.

Smokeping setup for VoIP monitoring

Installing and configuring Smokeping

Install Smokeping on a dedicated monitoring host or the Asterisk server itself (if CPU headroom allows).

yum install smokeping fping -y
systemctl enable smokeping
systemctl start smokeping

Edit the main Smokeping config at /etc/smokeping/config:

*** Targets ***
probe = FPing
title = VoIP Carrier Gateways

+ Carriers
menu = Carriers
title = Upstream SIP Carriers

++ Carrier_A
menu = Carrier A
title = Carrier A SIP Gateway
host = 203.0.113.45
+ Carrier_B
menu = Carrier B
title = Carrier B SIP Gateway
host = 198.51.100.22

+ Internal
menu = Internal
title = Internal Gateways and Interfaces

++ PBX_Main
menu = PBX Main
title = Main Asterisk Server
host = 192.168.1.10

++ WAN_Interface
menu = WAN
title = WAN Uplink
host = 192.0.2.1

Smokeping sends ICMP pings every 60 seconds by default. Set per-target intervals shorter for critical carriers:

++ Carrier_A
menu = Carrier A
title = Carrier A SIP Gateway
host = 203.0.113.45
step = 30
pings = 20

Enable cgi and web serving:

chown smokeping:smokeping /var/lib/smokeping/
chown smokeping:smokeping /var/cache/smokeping/
systemctl restart smokeping

Access the dashboard at http://your-server/smokeping/ (may require Apache alias configuration).

Reading Smokeping graphs

The Smokeping graph for each target shows five metrics:

Green line (median latency): actual round-trip time in milliseconds.
Blue area (50th percentile band): half of your probes fall here.
Red area (95th percentile band): outlier probes (bad luck, retransmits).
Gray zones at the bottom: packet loss shown as gaps or light fill.
Spike up: jitter or a delayed single probe.

If a carrier's median latency creeps from 45ms to 120ms over an hour, and packet loss appears as gray bands, you have a congestion event. If the red band widens but median stays stable, you have jitter without sustained loss.

Baseline your carriers under normal call load. Morning, midday, and evening should each show similar latency. If evening spikes 50ms higher than morning, your ISP or carrier oversubscribes evenings.

RTCP statistics: what they tell you

RTCP packets flow in parallel with RTP audio. Every few seconds, each end sends stats: bytes sent, packets sent, lost packets, jitter in milliseconds, and round-trip delay. Asterisk logs these in /var/log/asterisk/messages under the RTCP tag.

Enabling RTCP logging in Asterisk

Edit /etc/asterisk/sip.conf or sip-vicidial.conf:

[general]
rtcpinterval=5000
rtcpunit=ms

Then restart Asterisk:

asterisk -rx "module reload chan_sip.so"

Verify RTCP is flowing on an active call:

asterisk -rx "sip show channels verbose"

You will see lines like:

SIP/carrier-a-00000c7a (to caller) (format=ULAW)
  Relaying RTP to SIP/agent-00000c79 (format=ULAW)
  RTCP sent to: 203.0.113.45:16889
  RTCP received from: 203.0.113.45:16888

Parsing RTCP from logs

After a call ends, grep the call ID in /var/log/asterisk/messages:

grep "RTCP" /var/log/asterisk/messages | tail -50

A typical RTCP log line:

[2024-01-15 14:22:33] VERBOSE[12345][C-0000c7a]: chan_sip.c: RTCP SR (Sender Report) from 203.0.113.45:16888: ssrc=3456789 packets=1245 octets=198400 ntp=e0d1f52a.8b2c3d41 ts=987654 jitter=8.5 loss=0 delay=45

Extract and store these metrics in a script. A practical approach: parse the CDR and pull RTCP stats into a custom table.

Create a VoIP quality metrics table

In the asterisk database, add a table for call-level RTCP stats:

CREATE TABLE `voip_call_quality` (
  `id` int AUTO_INCREMENT PRIMARY KEY,
  `call_date` datetime NOT NULL,
  `call_id` varchar(128),
  `carrier` varchar(64),
  `src_ip` varchar(45),
  `dst_ip` varchar(45),
  `jitter_ms` float,
  `packet_loss_pct` float,
  `latency_ms` int,
  `codec` varchar(16),
  `direction` enum('inbound','outbound'),
  `call_status` varchar(32),
  KEY `call_date_idx` (`call_date`),
  KEY `carrier_idx` (`carrier`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Write a PHP script to parse Asterisk logs and insert rows. Run it hourly via cron:

<?php
// /usr/local/bin/parse_rtcp.php
$pdo = new PDO('mysql:host=localhost;dbname=asterisk', 'root', 'password');

$logfile = '/var/log/asterisk/messages';
$lines = file($logfile);

foreach ($lines as $line) {
    if (strpos($line, 'RTCP') === false) continue;

    // Extract fields using regex
    if (preg_match('/\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\].*RTCP.*from\s([\d\.]+).*jitter=([\d.]+)\s.*loss=([\d]+)\s.*delay=(\d+)/', $line, $m)) {
        $call_date = $m[1];
        $src_ip = $m[2];
        $jitter = $m[3];
        $loss = $m[4];
        $delay = $m[5];

        $stmt = $pdo->prepare('INSERT INTO voip_call_quality (call_date, src_ip, jitter_ms, packet_loss_pct, latency_ms) VALUES (?, ?, ?, ?, ?)');
        $stmt->execute([$call_date, $src_ip, $jitter, $loss, $delay]);
    }
}
?>

Query the table to spot trends:

SELECT carrier, AVG(jitter_ms) as avg_jitter, AVG(packet_loss_pct) as avg_loss, AVG(latency_ms) as avg_latency, COUNT(*) as call_count
FROM voip_call_quality
WHERE call_date >= DATE_SUB(NOW(), INTERVAL 24 HOUR)
GROUP BY carrier
ORDER BY avg_loss DESC;

High loss or jitter on a specific carrier shows up immediately.

Common call quality problems and fixes

One-way audio: caller hears agent, agent hears nothing

One-way audio almost always means the RTP stream from agent to caller is blocked or misrouted. Check:

SIP ALG (Application Layer Gateway) is enabled on your router. NAT gateways sometimes "help" by rewriting SIP headers, which breaks RTP routing. Disable SIP ALG in your router's advanced settings, or add a static NAT rule:

Router: Port Forward SIP traffic (UDP 5060, 5061) to Asterisk IP
Do NOT enable SIP Proxy or SIP ALG

Agent's IP is different from ViciDial's configured IP. If your agent is behind a different NAT or carrier than the main Asterisk server, SIP can establish but RTP gets stuck. In /etc/asterisk/sip-vicidial.conf, set:

[general]
externip = 203.0.113.10
externhost = pbx.yourdomain.com

Then reload:

asterisk -rx "sip reload"

Firewall blocking RTP return traffic (ephemeral ports). RTP uses high-numbered UDP ports (10000-20000 by default). Allow all outbound UDP from your Asterisk server:

# On CentOS 7 with firewalld
firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=192.168.1.0/24 protocol value=udp port protocol=udp port=10000-20000 accept'
firewall-cmd --reload

Jitter buffer misconfigured. If jitter on the agent side exceeds 50ms, the jitter buffer may be filling then draining, causing audio drop. In extensions-vicidial.conf, find the agent channel definition and set:

[agent-channel]
same => n,Set(JITTERBUFFER(adaptive)=yes)
same => n,Set(JITTERBUFFER(maxsize)=200)

200ms is the max buffer; adaptive mode expands and contracts based on network conditions.

One-way audio on specific carriers only

Your carrier's SIP server may not be sending RTP back on the port you sent from. This is a carrier misconfiguration, not your network.

Check your RTCP logs for that carrier. If you see jitter and loss on one leg but not the other, the issue is asymmetric routing: outbound RTP takes one path, inbound takes another. Carriers often load-balance across multiple gateways.

Ask the carrier to:

Pin your SIP session to a single media gateway (no load balancing of RTP).
Confirm their firewall allows your public IP to send RTP to any port they assign (not just 5060).
Provide a static IP or whitelist for your SIP trunk.

Meanwhile, check your trunk settings in /etc/asterisk/sip-vicidial.conf:

[carrier-a]
type=peer
host=203.0.113.45
fromuser=your_account_id
secret=your_password
disallow=all
allow=ulaw
dtmfmode=rfc2833
natmode=yes
qualify=200

Set natmode=yes to send keepalives and qualify=200 to test each peer every 200ms. Restart SIP:

asterisk -rx "sip reload"

High jitter causes choppy or robotic audio

Jitter of 20-30ms is normal on internet circuits. Above 50ms, audio quality degrades noticeably. Causes:

Congestion on your ISP link (Smokeping will show rising latency).
Carrier overload (check RTCP logs for loss; Smokeping to carrier gateway will show red bands).
Competing traffic (torrent, backup, video) sharing your uplink.

Fixes:

Enable QoS (Quality of Service) on your router. Prioritize SIP and RTP traffic:

Router -> QoS or Traffic Shaping -> UDP port 5060 (SIP) = High Priority
                                 -> UDP port 10000-20000 (RTP) = High Priority
                                 -> Everything else = Low Priority

Reduce jitter buffer in Asterisk. Counterintuitive, but a large buffer hides jitter initially then delivers a burst of late packets, causing audio dropout. Set to 50ms max:

same => n,Set(JITTERBUFFER(maxsize)=50)

Switch to G.729 codec if on metered bandwidth. G.729 uses 8 kbps vs ULAW's 64 kbps. Fewer packets means less jitter. But check your licensing: G.729 requires a per-channel license. In sip-vicidial.conf:

[general]
disallow=all
allow=g729
allow=ulaw

Check if Smokeping shows your ISP uplink is congested. If Smokeping to your WAN gateway shows rising latency and packet loss in evenings, you need more bandwidth. Contact your ISP.

Packet loss causes dropped words or silent gaps

Packet loss of 0.5% is noticeable; 1% or higher makes calls unusable. Causes:

Carrier network issue (Smokeping to their gateway shows loss, RTCP confirms it).
Your ISP oversubscribed (Smokeping shows loss to your upstream router).
Firewall or IDS dropping packets (less common, but check syslog for drops).

Fixes:

Verify loss is not just on one codec. Query your quality table:

SELECT codec, AVG(packet_loss_pct) as avg_loss
FROM voip_call_quality
WHERE call_date >= DATE_SUB(NOW(), INTERVAL 24 HOUR)
GROUP BY codec;

If loss is high on ULAW but low on G.729, the issue may be RTP packet size or MTU mismatch, not the network.

Check MTU size. SIP and RTP packets may be fragmented if your MTU is below 1500 bytes. Confirm:

ip link show
# Look for "mtu 1500"

If MTU is below 1500, increase it:

ip link set dev eth0 mtu 1500

Make persistent in /etc/sysconfig/network-scripts/ifcfg-eth0:

MTU=1500

Use FEC (Forward Error Correction) on critical carriers. Some carriers offer FEC, which adds redundancy so a single lost packet can be recovered. This increases bandwidth but improves quality on lossy links. Ask your carrier if available.
Switch carriers. If Smokeping shows your ISP or the carrier has persistent loss, and both confirm the SLA is being met (they contractually allow that loss), you need a different path. Add a second carrier with a different ISP and load-balance calls between them.

Calls drop after 30 seconds

Asterisk's default SIP session timer is 1800 seconds (30 minutes), but if a SIP proxy or NAT device between you and the carrier times out after 30 seconds, the call will drop. Check your carrier's SIP signaling logs first. Then, in sip-vicidial.conf, enable session timers:

[general]
session_timers=accept
session_expires=600
min_se=90

This sends a RTCP keepalive every 90 seconds. Restart:

asterisk -rx "sip reload"

If drops persist, the issue is likely SIP signaling loss (high latency on SIP requests), not the RTP stream. Query your CDR for calls with short duration:

SELECT caller_id, called_number, LENGTH_IN_SEC, call_date
FROM vicidial_log
WHERE call_date >= DATE_SUB(NOW(), INTERVAL 24 HOUR)
AND LENGTH_IN_SEC < 35
AND LENGTH_IN_SEC > 25
ORDER BY call_date DESC
LIMIT 20;

If most of your short calls are exactly 30 seconds, a NAT timeout is happening. Add a SIP registration keepalive:

[general]
register => your_account_id:your_password@203.0.113.45/your_account_id

[carrier-a]
type=peer
host=203.0.113.45
fromuser=your_account_id
secret=your_password
qualify=60

The qualify=60 sends a SIP OPTIONS ping every 60 seconds to keep the NAT binding alive.

Correlating Smokeping and RTCP data

The most powerful diagnostic is overlaying Smokeping and RTCP metrics on the same timeline.

Write a script to pull both datasets for a 24-hour window:

#!/bin/bash
# /usr/local/bin/voip_quality_report.sh

CARRIER=${1:-all}
HOURS=${2:-24}

echo "=== SMOKEPING LATENCY (avg, min, max) ==="
sqlite3 /var/lib/smokeping/data/smokeping.db << EOF
SELECT datetime(timestamp,'unixepoch') as time, median, loss
FROM smokeping_data
WHERE hostname LIKE '%$CARRIER%'
AND timestamp > strftime('%s', 'now', '-$HOURS hours')
ORDER BY timestamp;
EOF

echo ""
echo "=== RTCP METRICS (by carrier) ==="
mysql -u root -p asterisk << EOF
SELECT DATE_FORMAT(call_date, '%Y-%m-%d %H:00') as hour,
       carrier,
       AVG(jitter_ms) as jitter,
       AVG(packet_loss_pct) as loss,
       AVG(latency_ms) as latency,
       COUNT(*) as calls
FROM voip_call_quality
WHERE call_date >= DATE_SUB(NOW(), INTERVAL $HOURS HOUR)
AND (carrier LIKE '%$CARRIER%' OR '$CARRIER' = 'all')
GROUP BY DATE_FORMAT(call_date, '%Y-%m-%d %H:%i'), carrier
ORDER BY call_date DESC;
EOF

Run it:

./voip_quality_report.sh carrier-a 24

Look for time windows where Smokeping shows rising latency and RTCP shows rising jitter and loss. If they correlate, the problem is the network path. If Smokeping is clean but RTCP is poor, the issue is codec, SIP signaling, or internal Asterisk processing.

Asterisk CLI diagnostics for call quality

While a call is active, pull real-time RTP stats:

asterisk -rx "sip show channels verbose"

Output:

SIP/carrier-a-00000c7a (to 5551234567)
  Codecs: (ulaw|alaw|g729)
  RTP Stream: 192.168.1.10:12345 <-> 203.0.113.45:16888
  Jitter buffer: adaptive, max 200ms
  Last RTP: 48ms ago

Check for RTP gaps (Last RTP > 2 seconds means audio is frozen):

for i in {1..10}; do asterisk -rx "sip show channels" | grep -E "RTP|Last RTP"; sleep 1; done

If Last RTP keeps incrementing, audio is not flowing. Check firewall and NAT.

Troubleshooting

Smokeping shows loss but RTCP shows none

Smokeping sends ICMP (ping) traffic; RTP is UDP. Your firewall or router may allow RTP but rate-limit ICMP. ICMP loss is not proof of RTP loss. But if Smokeping shows sustained ICMP loss (>1% for >10 minutes), your ISP uplink is congested and RTP will suffer next.

Increase Smokeping pings per cycle to get more accurate loss:

++ Carrier_A
pings = 50

And confirm with MTR (My Traceroute) for a sustained, detailed view:

mtr -c 1000 203.0.113.45 > mtr_report.txt

MTR combines ping and traceroute. If you see loss or latency spike at a specific hop (ASN or ISP), report it to your carrier.

RTCP shows jitter but audio is clear

Jitter measurement in RTCP is instantaneous. A spike to 80ms on one packet does not mean audio is bad. Look for sustained jitter >50ms over hundreds of packets. Query your quality table by minute:

SELECT 
    DATE_FORMAT(call_date, '%Y-%m-%d %H:%i') as minute,
    AVG(jitter_ms) as avg_jitter,
    MAX(jitter_ms) as max_jitter,
    COUNT(*) as samples
FROM voip_call_quality
WHERE call_date >= DATE_SUB(NOW(), INTERVAL 1 HOUR)
GROUP BY minute
HAVING avg_jitter > 40 OR max_jitter > 100;

If spikes are rare and brief, your jitter buffer is absorbing them. If sustained, increase buffer size or reduce codec bitrate.

Smokeping has no data

Smokeping is running but the graphs are empty.

Check the Smokeping daemon log:

tail -100 /var/log/smokeping/smokeping.log

Common issues:

fping binary is missing or broken. Reinstall:

yum reinstall fping
ls -la /usr/sbin/fping

Smokeping has no write permission. Fix:

chown -R smokeping:smokeping /var/lib/smokeping/
chmod 755 /var/lib/smokeping/

Target host is unreachable. Verify you can ping it manually:

ping -c 5 203.0.113.45

If packets are blocked, add the target to your firewall whitelist.

ViciDial shows calls as DEAD but RTCP shows no loss

"DEAD" status in the ViciDial log usually means the agent hung up or the carrier disconnected the call, not that the call failed mid-stream.

Check the call disposition in /vicidial/admin.php (Reports -> Inbound/Outbound Call Log) or query directly:

SELECT call_date, caller_id, called_number, status, LENGTH_IN_SEC, disconnect_reason
FROM vicidial_log
WHERE call_date >= DATE_SUB(NOW(), INTERVAL 1 HOUR)
AND status = 'DEAD'
LIMIT 10;

If disconnect_reason is empty or "normal", the agent hung up or the call ended naturally. If it contains "SIP/403" or "SIP/486", the carrier rejected the call. If "timeout", SIP signaling stalled.

Filter for actual failure calls (short duration + SIP error):

SELECT call_date, caller_id, status, disconnect_reason
FROM vicidial_log
WHERE call_date >= DATE_SUB(NOW(), INTERVAL 1 HOUR)
AND LENGTH_IN_SEC < 5
AND disconnect_reason LIKE 'SIP%'
ORDER BY call_date DESC;

Frequently asked questions

Why does one agent hear perfect audio while another has jitter on the same call?

Jitter is primarily a function of the path from the remote party back to your agent. If two agents are in the same office but receiving calls on different carriers, or from different geographic regions, their RTP paths will differ. The path from New York to Los Angeles is not the same as New York to London. Check RTCP stats per agent and per carrier; you may find that Agent A (in LA) has low jitter from LA-based callers but high jitter from international calls.

Should I prioritize Smokeping latency numbers or RTCP jitter numbers?

Prioritize RTCP jitter because it reflects actual audio packet timing. Smokeping shows the baseline path; RTCP shows how audio packets are actually behaving. If Smokeping is clean but RTCP is noisy, the issue is call-specific: codec, SIP processing, or call routing. If both are high, the network path itself is the problem.

Is 50ms latency acceptable for VoIP?

Yes. The ITU-T G.114 recommendation allows up to 150ms one-way latency with no perceptible impact. Below 50ms, users notice no delay. Between 50-150ms, the delay is noticeable but tolerable. Above 150ms, calls feel like a satellite conversation (long pause before response). Your baseline latency should be stable and predictable; random jitter is more harmful than high latency.

Can I reduce jitter by switching from SIP to IAX2?

IAX2 (Inter-Asterisk eXchange) is a different VoIP protocol that may have lower jitter on certain network paths because it is more efficient than SIP+RTP. However, carrier support is rare. Most carriers only offer SIP trunks. If your carrier supports IAX2, test it against your current SIP trunk on the same calls. Use the same monitoring: Smokeping and RTCP stats.

What packet loss is acceptable?

Zero is ideal. Packet loss of 0.1-0.3% is generally not noticeable to the human ear on a 20ms-sample voice codec. Above 0.5%, most callers notice some clipping or words dropping out. Above 1%, the call is poor. If your RTCP data shows >0.5% loss on any carrier, contact the carrier to investigate or switch to a different carrier/path.

Summary

Deploy Smokeping to continuously monitor latency, jitter, and loss to each carrier gateway and your uplink. Configure RTCP statistics in sip-vicidial.conf and parse the logs into a database table for trending and correlation. When call quality issues arise, overlay Smokeping graphs (network-level view) with RTCP metrics (call-level view) to determine if the problem is the network path, the carrier, your ISP, or Asterisk configuration.

Start by establishing baseline metrics for each carrier during normal hours. Document acceptable jitter (<30ms), loss (<0.5%), and latency ranges (<100ms). When problems occur, query the quality table by hour and carrier to identify when the issue started and which carriers or paths are affected.

Key diagnostics to implement:

Smokeping config updated with all carriers and gateways.
RTCP logging enabled in sip-vicidial.conf and voip_call_quality table created.
Automated hourly RTCP parsing script running via cron.
Hourly or daily quality report script comparing Smokeping and RTCP metrics.

Check next:

Is your Asterisk server's external IP and NAT settings correct in sip-vicidial.conf.
Are firewall rules allowing RTP return traffic on high ports (10000-20000).
Has your QoS configuration prioritized SIP and RTP over bulk data.
Do you have a second carrier or ISP link to failover to when one degrades.

Auto QA Call Recording Quality with NISQA & Silero VAD

Moisi Trungu — Thu, 22 Oct 2026 03:30:03 +0000

Auto QA Call Recording Quality with NISQA & Silero VAD

Detect poor call audio in seconds by measuring speech intelligibility and voice activity during ViciDial recording post-processing.

Call recording quality problems kill agent training data, compliance audits, and customer experience analysis. NISQA (Non-Intrusive Speech Quality Assessment) scores audio intelligibility on a 1-5 MOS (Mean Opinion Score) scale without needing a reference recording. Silero VAD (Voice Activity Detection) identifies silence, background noise, and clipping. Run both tools against recordings in your ViciDial archive immediately after the call ends. Flag low-quality files, trigger re-recording for critical calls, and alert supervisors to codec or network issues before they compound.

This guide walks through installation, integration with ViciDial's recording pipeline, real scoring workflows, and troubleshooting. Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, and NISQA 1.0+.

Prerequisites

You need a working ViciDial system with:

Asterisk 16 or later (18 preferred for codec stability)
MariaDB 10.5+
Call recordings in .wav or .mp3 format stored in /var/spool/asterisk/monitor/ or equivalent
Python 3.8+
Root or sudo access
Disk space: 500MB minimum for NISQA/Silero models

Optional but recommended:

Supervisor or QA user account in ViciDial
Grafana or web dashboard for QA metrics
Cron or systemd timer for automated scoring

Why recording quality breaks in ViciDial

Call audio quality depends on codec selection, RTP timing, jitter buffers, and network path. A SIP call routed through a poor WAN link with G.729 compression may score 2.0 on NISQA even though the call connected. Agents report audio as "choppy" or "robotic." Without automated scoring, quality issues hide in the archive until QA staff manually spot them.

NISQA and Silero detect these problems programmatically:

NISQA scores overall intelligibility (1=bad, 5=excellent)
Silero VAD measures percentage of active speech, identifies gaps, detects clipping or over-compression

Together they flag calls where the agent or customer cannot be understood, or where background noise drowns out speech.

Install NISQA and Silero VAD

Download and configure NISQA

cd /opt
git clone https://github.com/gabrielmittag/NISQA.git
cd NISQA
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install torch torchaudio librosa numpy scipy matplotlib
pip install -e .

NISQA downloads a pre-trained model (~200MB) on first run. Test the installation:

cd /opt/NISQA
python3 -c "from nisqa.NISQA_model import NISQAModel; m = NISQAModel(pretrained_model_dir='nisqa_pretrained_model'); print('NISQA ready')"

Model download runs in the background. On first execution, wait 2-3 minutes.

Install Silero VAD

pip install silero-vad
python3 -c "import torch; torch.hub.load('snakers4/silero-vad', 'get_speech_timestamps', trust_repo=True); print('Silero VAD ready')"

Silero downloads a ~40MB ONNX model the first time you call it.

Verify codec support

NISQA and Silero work on raw audio streams. If your ViciDial recordings are compressed (G.729, GSM), decode them first:

apt-get install -y ffmpeg
ffmpeg -i call_20250115_120000_5551234567.wav -acodec pcm_s16le -ar 16000 decoded.wav

Or decode in-process in your scoring script (covered below).

Design the QA scoring workflow

Your workflow is:

Call ends, asterisk writes .wav or .mp3 to /var/spool/asterisk/monitor/
ViciDial cron job moves or copies file to archive
Your QA service picks up the file, decodes, runs NISQA + Silero VAD
Scores are inserted into a ViciDial database table
QA dashboard or agent screen shows results

Create a new table in the asterisk database to hold QA results:

CREATE TABLE vicidial_qa_scores (
  qa_id INT AUTO_INCREMENT PRIMARY KEY,
  call_date DATETIME DEFAULT CURRENT_TIMESTAMP,
  vicidial_log_id BIGINT NOT NULL,
  filename VARCHAR(255) NOT NULL,
  nisqa_score FLOAT,
  nisqa_raw FLOAT,
  nisqa_dim_col FLOAT,
  nisqa_dim_noi FLOAT,
  nisqa_dim_mos FLOAT,
  speech_percentage FLOAT,
  silence_duration_sec FLOAT,
  noise_floor_db FLOAT,
  clipping_detected BOOLEAN DEFAULT 0,
  processing_status ENUM('pending', 'processed', 'error') DEFAULT 'pending',
  error_message TEXT,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  UNIQUE KEY unique_file (filename),
  KEY idx_vicidial_log_id (vicidial_log_id),
  KEY idx_processing_status (processing_status)
);

This table links to ViciDial's vicidial_log table via vicidial_log_id. The scoring service queries for recordings not yet scored, processes them, and updates this table.

Create the QA scoring service

Here is a production-grade Python script that monitors your recording directory, scores new files, and stores results:

#!/usr/bin/env python3
"""
ViciDial Call Recording QA Scorer using NISQA and Silero VAD
Monitors recording directory, scores audio, updates database
"""

import os
import sys
import argparse
import logging
import json
import subprocess
from pathlib import Path
from datetime import datetime
import hashlib

import MySQLdb
import torch
import torchaudio
import librosa
import numpy as np
from nisqa.NISQA_model import NISQAModel

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('/var/log/asterisk/qa_scorer.log'),
        logging.StreamHandler()
    ]
)
logger = logging.getLogger(__name__)


class QAScorer:
    def __init__(self, db_host='localhost', db_user='asteriskuser',
                 db_pass='asterisk', db_name='asterisk',
                 recordings_dir='/var/spool/asterisk/monitor'):
        self.db_host = db_host
        self.db_user = db_user
        self.db_pass = db_pass
        self.db_name = db_name
        self.recordings_dir = Path(recordings_dir)

        # Initialize NISQA model
        logger.info("Loading NISQA model...")
        self.nisqa_model = NISQAModel(
            pretrained_model_dir='/opt/NISQA/nisqa_pretrained_model'
        )

        # Load Silero VAD
        logger.info("Loading Silero VAD...")
        self.silero_vad = torch.hub.load(
            'snakers4/silero-vad',
            'get_speech_timestamps',
            trust_repo=True
        )
        self.silero_get_speech_ts = torch.hub.load(
            'snakers4/silero-vad',
            'get_speech_timestamps',
            trust_repo=True
        )

        self.db_conn = None
        self.connect_db()

    def connect_db(self):
        try:
            self.db_conn = MySQLdb.connect(
                host=self.db_host,
                user=self.db_user,
                passwd=self.db_pass,
                db=self.db_name
            )
            logger.info("Database connected")
        except MySQLdb.Error as e:
            logger.error(f"Database connection failed: {e}")
            sys.exit(1)

    def decode_audio(self, filepath, target_sr=16000):
        """Decode any audio format to 16kHz mono PCM"""
        if filepath.suffix.lower() == '.wav':
            try:
                waveform, sr = torchaudio.load(filepath)
                if sr != target_sr:
                    resampler = torchaudio.transforms.Resample(sr, target_sr)
                    waveform = resampler(waveform)
                if waveform.shape[0] > 1:
                    waveform = waveform.mean(dim=0, keepdim=True)
                return waveform.squeeze().numpy(), target_sr
            except Exception as e:
                logger.error(f"Decode error {filepath}: {e}")
                return None, None
        else:
            # Use ffmpeg for other formats
            try:
                temp_wav = f"/tmp/{filepath.stem}_temp.wav"
                cmd = [
                    'ffmpeg', '-i', str(filepath),
                    '-ar', str(target_sr), '-ac', '1',
                    '-f', 'wav', temp_wav, '-y'
                ]
                subprocess.run(cmd, check=True, capture_output=True, timeout=30)
                waveform, sr = torchaudio.load(temp_wav)
                os.remove(temp_wav)
                return waveform.squeeze().numpy(), sr
            except Exception as e:
                logger.error(f"FFmpeg decode error {filepath}: {e}")
                return None, None

    def score_nisqa(self, waveform, sr):
        """Score audio with NISQA"""
        try:
            # NISQA expects dict input with 'speech' key
            result = self.nisqa_model.predict(
                torch.from_numpy(waveform).float().unsqueeze(0),
                sr=sr
            )
            return {
                'mos_score': float(result[0]['mos']),
                'mos_raw': float(result[0]['mos_raw']),
                'dim_col': float(result[0]['dim_col']),
                'dim_noi': float(result[0]['dim_noi']),
                'dim_mos': float(result[0]['dim_mos'])
            }
        except Exception as e:
            logger.error(f"NISQA scoring error: {e}")
            return None

    def score_vad(self, waveform, sr):
        """Score voice activity with Silero VAD"""
        try:
            # Silero VAD expects float32 tensor
            wav_tensor = torch.from_numpy(waveform).float()
            if wav_tensor.dim() == 1:
                wav_tensor = wav_tensor.unsqueeze(0)

            # Get speech timestamps (list of dicts with 'start' and 'end' in samples)
            speech_ts = self.silero_get_speech_ts(wav_tensor, sr)

            if not speech_ts:
                return {
                    'speech_percentage': 0.0,
                    'silence_duration_sec': len(waveform) / sr,
                    'num_speech_segments': 0,
                    'clipping_detected': False,
                    'noise_floor_db': -80.0
                }

            total_speech_samples = sum(
                ts['end'] - ts['start'] for ts in speech_ts
            )
            speech_percentage = (total_speech_samples / len(waveform)) * 100.0
            silence_duration = (len(waveform) - total_speech_samples) / sr

            # Detect clipping (samples at near max amplitude)
            max_val = np.max(np.abs(waveform))
            clipping_threshold = 0.99
            clipping_detected = max_val > clipping_threshold

            # Estimate noise floor from silent segments
            if len(speech_ts) > 0:
                silent_regions = []
                if speech_ts[0]['start'] > 0:
                    silent_regions.append(waveform[:speech_ts[0]['start']])
                for i in range(len(speech_ts) - 1):
                    gap = waveform[speech_ts[i]['end']:speech_ts[i+1]['start']]
                    if len(gap) > sr * 0.05:  # Gaps larger than 50ms
                        silent_regions.append(gap)

                if silent_regions:
                    noise = np.concatenate(silent_regions)
                    rms = np.sqrt(np.mean(noise ** 2))
                    noise_db = 20 * np.log10(rms + 1e-10)
                else:
                    noise_db = -80.0
            else:
                noise_db = -80.0

            return {
                'speech_percentage': float(speech_percentage),
                'silence_duration_sec': float(silence_duration),
                'num_speech_segments': len(speech_ts),
                'clipping_detected': bool(clipping_detected),
                'noise_floor_db': float(noise_db)
            }
        except Exception as e:
            logger.error(f"VAD scoring error: {e}")
            return None

    def get_vicidial_log_id(self, filename):
        """Extract or look up vicidial_log_id from filename or database"""
        # ViciDial filenames are typically YYYYMMDD-HHMMSS_UNIQUEID.wav
        # Try to find in vicidial_log by matching filename
        try:
            cursor = self.db_conn.cursor()
            cursor.execute("""
                SELECT vicidial_log_id FROM vicidial_log
                WHERE recording LIKE CONCAT('%', %s, '%')
                ORDER BY call_date DESC LIMIT 1
            """, (filename,))
            row = cursor.fetchone()
            cursor.close()
            return row[0] if row else None
        except MySQLdb.Error as e:
            logger.error(f"Query error: {e}")
            return None

    def process_file(self, filepath):
        """Score a single recording file"""
        filename = filepath.name

        # Check if already processed
        try:
            cursor = self.db_conn.cursor()
            cursor.execute(
                "SELECT qa_id FROM vicidial_qa_scores WHERE filename = %s",
                (filename,)
            )
            if cursor.fetchone():
                logger.info(f"Skipping {filename} (already scored)")
                cursor.close()
                return
            cursor.close()
        except MySQLdb.Error as e:
            logger.error(f"Lookup error: {e}")
            return

        logger.info(f"Processing {filename}")

        # Decode audio
        waveform, sr = self.decode_audio(filepath)
        if waveform is None:
            self.insert_qa_result(
                filename, None,
                processing_status='error',
                error_message='Audio decode failed'
            )
            return

        # Run NISQA
        nisqa_result = self.score_nisqa(waveform, sr)
        if nisqa_result is None:
            self.insert_qa_result(
                filename, None,
                processing_status='error',
                error_message='NISQA scoring failed'
            )
            return

        # Run Silero VAD
        vad_result = self.score_vad(waveform, sr)
        if vad_result is None:
            self.insert_qa_result(
                filename, nisqa_result,
                processing_status='error',
                error_message='VAD scoring failed'
            )
            return

        # Insert results
        self.insert_qa_result(
            filename, nisqa_result, vad_result,
            processing_status='processed'
        )
        logger.info(f"Scored {filename}: NISQA={nisqa_result['mos_score']:.2f}, Speech={vad_result['speech_percentage']:.1f}%")

    def insert_qa_result(self, filename, nisqa_result=None, vad_result=None,
                        processing_status='pending', error_message=None):
        """Insert QA scores into database"""
        try:
            vicidial_log_id = self.get_vicidial_log_id(filename)

            cursor = self.db_conn.cursor()
            cursor.execute("""
                INSERT INTO vicidial_qa_scores (
                    filename, vicidial_log_id,
                    nisqa_score, nisqa_raw, nisqa_dim_col, nisqa_dim_noi, nisqa_dim_mos,
                    speech_percentage, silence_duration_sec, clipping_detected,
                    processing_status, error_message
                ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
            """, (
                filename,
                vicidial_log_id,
                nisqa_result['mos_score'] if nisqa_result else None,
                nisqa_result['mos_raw'] if nisqa_result else None,
                nisqa_result['dim_col'] if nisqa_result else None,
                nisqa_result['dim_noi'] if nisqa_result else None,
                nisqa_result['dim_mos'] if nisqa_result else None,
                vad_result['speech_percentage'] if vad_result else None,
                vad_result['silence_duration_sec'] if vad_result else None,
                vad_result['clipping_detected'] if vad_result else 0,
                processing_status,
                error_message
            ))
            self.db_conn.commit()
            cursor.close()
        except MySQLdb.Error as e:
            logger.error(f"Insert error: {e}")
            self.db_conn.rollback()

    def scan_and_process(self, age_hours=24):
        """Scan recording directory and process unscored files"""
        if not self.recordings_dir.exists():
            logger.error(f"Recordings dir not found: {self.recordings_dir}")
            return

        cutoff_time = datetime.now().timestamp() - (age_hours * 3600)
        wav_files = list(self.recordings_dir.glob('*.wav')) + \
                    list(self.recordings_dir.glob('*.mp3'))

        logger.info(f"Found {len(wav_files)} audio files")

        for filepath in sorted(wav_files):
            if filepath.stat().st_mtime < cutoff_time:
                continue
            self.process_file(filepath)

    def close(self):
        if self.db_conn:
            self.db_conn.close()


def main():
    parser = argparse.ArgumentParser(
        description='ViciDial QA Call Recording Scorer'
    )
    parser.add_argument('--db-host', default='localhost')
    parser.add_argument('--db-user', default='asteriskuser')
    parser.add_argument('--db-pass', default='asterisk')
    parser.add_argument('--recordings-dir', default='/var/spool/asterisk/monitor')
    parser.add_argument('--age-hours', type=int, default=24,
                       help='Only process files younger than this')

    args = parser.parse_args()

    scorer = QAScorer(
        db_host=args.db_host,
        db_user=args.db_user,
        db_pass=args.db_pass,
        recordings_dir=args.recordings_dir
    )

    try:
        scorer.scan_and_process(age_hours=args.age_hours)
    finally:
        scorer.close()

    logger.info("QA scoring completed")


if __name__ == '__main__':
    main()

Save this script to /opt/vicidial_qa_scorer.py and make it executable:

chmod +x /opt/vicidial_qa_scorer.py
pip install MySQLdb

Schedule automated scoring with systemd

Create a systemd timer to run scoring every 5 minutes:

# /etc/systemd/system/vicidial-qa-scorer.service
[Unit]
Description=ViciDial QA Call Recording Scorer
After=asterisk.service mysql.service

[Service]
Type=oneshot
User=root
ExecStart=/usr/bin/python3 /opt/vicidial_qa_scorer.py \
  --db-host localhost \
  --db-user asteriskuser \
  --db-pass asterisk \
  --recordings-dir /var/spool/asterisk/monitor \
  --age-hours 12
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/vicidial-qa-scorer.timer
[Unit]
Description=ViciDial QA Scorer Timer
Requires=vicidial-qa-scorer.service

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
AccuracySec=1min

[Install]
WantedBy=timers.target

Enable and start the timer:

systemctl daemon-reload
systemctl enable vicidial-qa-scorer.timer
systemctl start vicidial-qa-scorer.timer
systemctl status vicidial-qa-scorer.timer

View and alert on QA scores

Query your scores to find poor-quality calls:

-- Find calls with low NISQA scores
SELECT filename, nisqa_score, speech_percentage, clipping_detected
FROM vicidial_qa_scores
WHERE nisqa_score < 2.5 AND processing_status = 'processed'
ORDER BY call_date DESC LIMIT 20;

-- Find calls with excessive silence
SELECT filename, speech_percentage, silence_duration_sec
FROM vicidial_qa_scores
WHERE speech_percentage < 30 AND processing_status = 'processed'
ORDER BY call_date DESC LIMIT 20;

-- Find calls with clipping
SELECT filename, nisqa_score, clipping_detected
FROM vicidial_qa_scores
WHERE clipping_detected = 1 AND processing_status = 'processed'
ORDER BY call_date DESC LIMIT 20;

Create an alert script that emails supervisors when quality drops:

#!/bin/bash
# /opt/qa_alert.sh - Run hourly via cron

THRESHOLD_MOS=2.5
THRESHOLD_SPEECH=40.0

MYSQL_PASS="asterisk"
MYSQL_USER="asteriskuser"
MYSQL_HOST="localhost"
MYSQL_DB="asterisk"

QUERY="SELECT COUNT(*) FROM vicidial_qa_scores
WHERE processing_status = 'processed'
AND call_date > NOW() - INTERVAL 1 HOUR
AND (nisqa_score < $THRESHOLD_MOS OR speech_percentage < $THRESHOLD_SPEECH);"

COUNT=$(mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS $MYSQL_DB -se "$QUERY")

if [ "$COUNT" -gt 0 ]; then
    echo "QA Alert: $COUNT calls with poor quality in the last hour" | \
      mail -s "ViciDial QA Alert" supervisor@company.com
fi

# Add to /etc/cron.d/vicidial-qa
0 * * * * root /opt/qa_alert.sh >> /var/log/asterisk/qa_alert.log 2>&1

Interpret NISQA scores

NISQA outputs a Mean Opinion Score (MOS) from 1.0 to 5.0:

1.0-1.5: Nearly unintelligible. Artifacts, heavy compression, or noise dominate.
1.5-2.5: Poor. Speech recognizable but with effort. Typical of congested networks or very compressed codecs.
2.5-3.5: Fair. Acceptable for most applications but QA should review. Moderate artifacts.
3.5-4.5: Good. Commercial-grade audio. Suitable for compliance and training.
4.5-5.0: Excellent. Minimal artifacts, clear speech, no background noise.

Silero VAD reports speech percentage (0-100%). Below 40% indicates a call that is mostly silent or heavily noise-gated. Above 70% with a high NISQA score is ideal.

Clipping (amplitude near 1.0 for multiple consecutive samples) indicates gain-staging problems. Reduce input gain on the SIP trunk or agent headset.

Why NISQA scores vary by codec

ViciDial typically uses:

ULAW (default): Wide compression, acceptable loss. NISQA 3.2-3.8 on clean networks.
G.729: Narrow bandwidth, 8kHz, licensed codec. NISQA 2.5-3.2 due to aggressive compression.
GSM: Mobile codec, poor fidelity. NISQA 2.0-2.8.
OPUS or Speex: Modern codecs, better fidelity. NISQA 3.8-4.5.

If your agents complain about "robot voices," check your trunk codec settings in /etc/asterisk/sip-vicidial.conf:

[carrier_name]
type=peer
codec=ulaw,alaw,g729
disallow=all
allow=ulaw
allow=alaw
allow=g729

Switch preferred codec order by listing best-quality first:

allow=ulaw
allow=alaw
allow=g729

Or force one codec:

allow=ulaw
disallow=all

Reload the config:

asterisk -rx "sip reload"

Run QA scoring again on the next batch of calls. NISQA scores should improve.

Handle special cases

Multi-channel recordings

Some ViciDial setups record agent and customer on separate channels, then mix them. If your wav file is stereo, decode to mono before scoring:

waveform, sr = torchaudio.load(filepath)
if waveform.shape[0] > 1:
    waveform = waveform.mean(dim=0, keepdim=True)  # Mix to mono

The Python script above already does this.

Very short calls

Calls under 3 seconds confuse both NISQA and Silero. Add a check:

if len(waveform) / sr < 3.0:
    logger.info(f"Skipping {filename}: too short")
    self.insert_qa_result(filename, processing_status='skipped',
                         error_message='Call duration < 3 seconds')
    return

Noisy environments

Call centers with open floor plans have high background noise. Silero VAD will report low speech percentage even if the call is successful. Compare your NISQA dim_noi (noise dimension) score with speech percentage. High dim_noi and low speech percentage indicate environmental noise, not transmission loss.

Troubleshooting

NISQA model download hangs

The model (200MB) downloads on first inference. If your server has slow or metered internet, pre-download:

cd /opt/NISQA
python3 << 'EOF'
from nisqa.NISQA_model import NISQAModel
print("Downloading model...")
m = NISQAModel(pretrained_model_dir='nisqa_pretrained_model')
print("Done. Model at nisqa_pretrained_model/")
EOF

This can take 5-10 minutes on slow connections. Run it interactively so you see progress.

"CUDA out of memory" when scoring

NISQA and Silero VAD load the model into GPU memory. If you have a low-VRAM GPU or many concurrent scores, move models to CPU:

# In QAScorer.__init__:
self.nisqa_model = NISQAModel(
    pretrained_model_dir='/opt/NISQA/nisqa_pretrained_model',
    device='cpu'  # Force CPU
)

Scoring will be slower but will not crash.

Silero VAD failing with "AssertionError: Assert sr == 16000"

Silero VAD is strict about sample rate. Ensure your decode function resamples to 16kHz:

if sr != 16000:
    resampler = torchaudio.transforms.Resample(sr, 16000)
    waveform = resampler(waveform)
    sr = 16000

MySQL error "Column 'noise_floor_db' doesn't exist"

Your vicidial_qa_scores table is missing the optional column. Add it:

ALTER TABLE vicidial_qa_scores ADD COLUMN noise_floor_db FLOAT DEFAULT NULL;

Scoring takes 30+ seconds per file

NISQA inference on CPU takes 15-30 seconds per file. If you have thousands of backlog files, run multiple scorer instances in parallel with different time windows:

# Scorer 1: files from 0-6 hours ago
python3 /opt/vicidial_qa_scorer.py --age-hours 6 &

# Scorer 2: files from 6-12 hours ago (modify script to support age range)
python3 /opt/vicidial_qa_scorer.py --age-hours 12 &

wait

Or schedule with staggered cron entries at 5-minute intervals.

ffmpeg errors on .mp3 files

If ffmpeg fails on .mp3, install the codec:

apt-get install -y libavcodec-extra

Or upgrade ffmpeg:

apt-get install -y ffmpeg
ffmpeg -version  # Should be 4.2+

Scores not appearing in database

Check the service log:

journalctl -u vicidial-qa-scorer.service -n 50 -e
tail -50 /var/log/asterisk/qa_scorer.log

Common issues:

Database credentials wrong in service file
Recording directory path not found or not readable
Permissions: script runs as root, but /var/spool/asterisk/monitor/ may be owned by asterisk user

Fix permissions:

chmod 755 /var/spool/asterisk/monitor

Asterisk not recording

Call recordings are disabled if recording is not set in dialplan or extensions-vicidial.conf. Check that MixMonitor is being called:

grep -n "MixMonitor" /etc/asterisk/extensions-vicidial.conf | head -5

You should see lines like:

MixMonitor(/var/spool/asterisk/monitor/...

If missing, update your dialplan or ensure ViciDial's call center is configured to record calls (Web admin > Agents > Settings).

Frequently asked questions

What NISQA score should I use as a threshold for re-recording or escalation?

Use 2.5 as a warning threshold (review the call) and 2.0 as a failure threshold (re-record for compliance or training). Calls below 2.0 are typically unintelligible to humans. Anything above 3.5 is acceptable for all uses. Thresholds depend on your SLA. QA operations often flag 2.5-3.0 for supervisor review and automatically reject 1.5-2.5.

Can I use NISQA without Silero VAD?

Yes. NISQA alone gives you overall speech quality. However, NISQA doesn't distinguish between silence, background noise, and actual speech quality. A call that is 90% silence will score higher than a loud, cl

ViciDial Call Distribution: Agent Rank & Load Balancing

Moisi Trungu — Wed, 21 Oct 2026 03:30:02 +0000

ViciDial Call Distribution: Agent Rank & Load Balancing

Uneven agent workload and call abandonment happen when rank distribution is misconfigured: fix queue strategy, agent status routing, and load-balancing weights to match your team size and call volume.

Call distribution in ViciDial depends on three overlapping systems: agent rank (a numeric priority assigned to each agent), queue strategy (how Asterisk selects the next agent), and load balancing (how calls are weighted across groups). When rank is set incorrectly, calls queue up behind low-skill agents while senior agents sit idle, or they distribute randomly instead of fairly. The fix involves setting rank values correctly in the database, choosing the right queue strategy in extensions, and validating agent status flow through the inbound group configuration.

Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, MariaDB 10.5+, with SIP trunks and local agent registration.

Prerequisites

SSH access to the ViciDial server and MySQL command line or phpMyAdmin
Ability to edit /etc/asterisk/extensions-vicidial.conf and reload Asterisk
Understanding of ViciDial agent statuses: READY, PAUSED, INCALL, DEAD, RING, DISPO
At least two test agents configured and logged in
Sample inbound DID or campaign with a queue configured
Permission to run asterisk -rx commands as root

Understanding agent rank in ViciDial

Agent rank is a numeric field (0-9999) stored in the vicidial_users table. Lower numbers are called first. Rank is not tied to role or permission; it is purely a call-distribution priority within a specific inbound group.

Open a terminal and check current rank assignments:

mysql -u root -p asterisk -e \
  "SELECT user_id, full_name, user_rank, inbound_group FROM vicidial_users WHERE user_rank IS NOT NULL ORDER BY inbound_group, user_rank;"

Output looks like:

| user_id | full_name      | user_rank | inbound_group |
|---------|----------------|-----------|---------------|
| 1001    | Alice Senior   | 1         | SALES         |
| 1002    | Bob Support    | 2         | SALES         |
| 1003    | Carol New      | 99        | SALES         |
| 2001    | Dave Tech      | 1         | SUPPORT       |
| 2002    | Eve Tech       | 2         | SUPPORT       |

Rank 1 agents in SALES get calls first. When Alice is READY, she answers. When she goes PAUSED or INCALL, rank 2 (Bob) becomes the next candidate. Rank 99 (Carol, new) only gets calls if Alice and Bob are not available.

The inbound_group column links each agent to a phone queue. Agents in different groups do not compete for the same calls.

Setting up rank for balanced distribution

For a team of five agents handling the same queue, a fair setup assigns ranks 1 through 5:

UPDATE vicidial_users
SET user_rank = 1
WHERE user_id = '1001' AND inbound_group = 'SALES';

UPDATE vicidial_users
SET user_rank = 2
WHERE user_id = '1002' AND inbound_group = 'SALES';

UPDATE vicidial_users
SET user_rank = 3
WHERE user_id = '1003' AND inbound_group = 'SALES';

UPDATE vicidial_users
SET user_rank = 4
WHERE user_id = '1004' AND inbound_group = 'SALES';

UPDATE vicidial_users
SET user_rank = 5
WHERE user_id = '1005' AND inbound_group = 'SALES';

Commit the changes. Do not use gaps (rank 1, 3, 5, 10). Asterisk iterates through each rank in order; unused rank values waste cycles.

For a tiered support model (senior handling overflow):

UPDATE vicidial_users SET user_rank = 10 WHERE user_id = '1001' AND inbound_group = 'SALES';
UPDATE vicidial_users SET user_rank = 11 WHERE user_id = '1002' AND inbound_group = 'SALES';
UPDATE vicidial_users SET user_rank = 12 WHERE user_id = '1003' AND inbound_group = 'SALES';
UPDATE vicidial_users SET user_rank = 1 WHERE user_id = '2001' AND inbound_group = 'SALES';

Here, the senior agent (rank 1) gets first call, then three junior agents. This ensures skill routing without complex queues.

Queue strategy and load balancing

Asterisk offers five queue strategies. ViciDial uses them in the extensions-vicidial.conf file. The choice affects how agents are selected at each call.

Strategy selection: what works for ViciDial

ringall: Rings all available agents in rank order simultaneously. The first to answer gets the call. Good for small teams (3-5 agents). Higher answer rate, but agents get ring storms.

leastrecent: Rings the agent least recently called, then the next least recent, in rank order. Distributes calls more evenly. Recommended for most setups.

fewest: Rings the agent with the fewest current calls. Useful if agents handle multiple calls at once (rare in ViciDial).

rrmemory: Round-robin with memory; cycles through agents in rank order and remembers the last agent called. Fair, stable. Good for equal-skill teams.

linear: Rings agents one at a time, in strict rank order, waiting for timeout before moving to the next. Slow for high volume. Avoid.

For a standard ViciDial inbound group with 5-20 agents of similar skill, use leastrecent or rrmemory. For senior + junior teams, use ringall with rank separating tiers.

Open the inbound group configuration file:

cat /etc/asterisk/extensions-vicidial.conf | grep -A 50 "exten => 6001"

Look for the Queue application line:

exten => 6001,1,Queue(SALES|t|${DIALPLAN_VARS}|5)

The queue name is SALES. The third parameter sets strategy. Modify the queues.conf file:

grep -A 30 "^\[SALES\]" /etc/asterisk/queues.conf

Output:

[SALES]
music = default
strategy = leastrecent
timeout = 15
retry = 5
weight = 10
member => Local/1001@vicidial
member => Local/1002@vicidial
member => Local/1003@vicidial

To change to rrmemory:

sed -i 's/strategy = leastrecent/strategy = rrmemory/' /etc/asterisk/queues.conf

Reload Asterisk:

asterisk -rx "module reload app_queue.so"

Verify the change took effect:

asterisk -rx "queue show SALES"

Output includes:

Strategy: Round Robin with Memory

Load balancing across inbound groups

If you have multiple inbound groups (SALES, SUPPORT, BILLING), and agents belong to multiple groups, load balancing ensures each agent is not overloaded across all groups.

In ViciDial, an agent can have a primary inbound_group and secondary groups. Rank applies to each group independently. However, if an agent is assigned to two groups, they will receive calls from both based on their rank in each.

To prevent an agent from being overwhelmed, use the agent's max_concurrent_calls setting in vicidial_users:

SELECT user_id, full_name, inbound_group, max_concurrent_calls
FROM vicidial_users
WHERE user_id = '1001';

Set max concurrent calls to 1 for single-call handling:

UPDATE vicidial_users
SET max_concurrent_calls = 1
WHERE user_id = '1001' AND inbound_group = 'SALES';

This is enforced at the dial-plan level in extensions-vicidial.conf. Look for the agent's Local channel:

exten => 1001,1,Dial(SIP/1001@default,,TtHhkK)

The second parameter (empty above) can be filled with options. Asterisk does not natively limit concurrent calls per extension; this is managed by ViciDial's state tracking in the vicidial_agent_log table.

ViciDial tracks agent state (READY, INCALL, etc.) in memory and database:

SELECT user_id, status, calls_today, last_call_time
FROM vicidial_agent_log
WHERE user_id = '1001' AND campaign_id = 'SALES' LIMIT 1;

The Asterisk dial-plan consults this table before ringing an agent. If the agent's status is not READY or RING, they are skipped.

Configuring inbound groups and rank assignment

Log into the web admin console at /vicidial/admin.php. Go to Inbound > Manage Inbound Groups.

Create or edit a group (e.g., SALES):

Group ID: SALES
Group Alias: Sales Inbound Queue
Queue Seconds: 15 (ring timeout)
Queue Priority: 1 (higher numbers = lower priority if multiple groups)
Calls Today: 0 (auto)
Allow Ingroup Recording: Yes

Save. Then go to Manage Inbound Group Members. Add agents:

Agent ID	Rank	Status
1001	1	Active
1002	2	Active
1003	3	Active

The web interface updates vicidial_users.user_rank for each agent. Verify via CLI:

mysql -u root -p asterisk -e \
  "SELECT user_id, user_rank, inbound_group FROM vicidial_users WHERE inbound_group = 'SALES';"

Testing call distribution

Start with two agents logged in to the same inbound group.

asterisk -rx "agent show 1001"

Output should show:

Agent/1001 (Alice) is logged in

Repeat for agent 1002:

asterisk -rx "agent show 1002"

Now, send an inbound call to the group's DID. Use a test call or generate one via a curl script:

curl -X POST "http://localhost:8088/stasis/applications/vicidial_inbound/subscribe?channel=SIP/trunk%2F5551234567" \
  -H "Authorization: Bearer YOUR_TOKEN"

Or, if you have an existing inbound trunk, dial the number manually.

Watch Asterisk console in real time:

asterisk -c -f 2>&1 | grep -E "(Queue|Agent|Member)"

If agent 1001 is READY, they ring first. If they go INCALL, watch for agent 1002 to ring next. If rank is wrong (e.g., both at rank 1), both ring simultaneously (depending on strategy).

Check the Asterisk queue status:

asterisk -rx "queue show SALES"

Output example:

SALES has 1 calls (max unlimited) in 'rrmemory' strategy (20s holdtime, 45s talktime)
  Members:
    Local/1001@vicidial (Member) Penalty: 0 Paused: No (In use)
    Local/1002@vicidial (Member) Penalty: 0 Paused: No (Ready)
    Local/1003@vicidial (Member) Penalty: 0 Paused: No (Ready)

Member status tells you who is ringing or in call. If it says (In use), that agent's status is INCALL in the database.

Advanced: custom rank-based load balancing

For high-volume or multi-team setups, use Asterisk's "penalty" system in queues.conf to override rank. Penalty increases the time an agent is skipped before being rung.

Edit queues.conf:

[SALES]
strategy = leastrecent
timeout = 15
queue-yoursoundfile = on

member => Local/1001@vicidial,penalty=0
member => Local/1002@vicidial,penalty=0
member => Local/1003@vicidial,penalty=10
member => Local/1004@vicidial,penalty=20

Agents 1003 and 1004 have higher penalties, so they are deprioritized even if they are READY. Agent 1003 is rung only after agents 1001 and 1002 have declined or are busy. Agent 1004 is a last resort.

Reload:

asterisk -rx "module reload app_queue.so"

Verify:

asterisk -rx "queue show SALES"

Output:

Members:
  Local/1001@vicidial (Member) Penalty: 0 Paused: No
  Local/1002@vicidial (Member) Penalty: 0 Paused: No
  Local/1003@vicidial (Member) Penalty: 10 Paused: No
  Local/1004@vicidial (Member) Penalty: 20 Paused: No

Penalty works independently of rank. If you use both rank and penalty, Asterisk applies rank first (within vicidial_users), then penalty (in queues.conf) as a secondary filter.

Monitoring and tuning rank distribution

Filter by Inbound Group: SALES. Check the Calls Handled column for each agent over the last hour.

Expected distribution for equal-rank agents: roughly equal call counts. If one agent has 20 calls and another has 5, check:

Agent availability: Is the low-call agent paused or offline?
Rank: Did you apply the correct rank to all agents?
Queue strategy: Is it set to leastrecent or rrmemory?
Agent skill: Do some agents handle calls longer, reducing their READY time?

Example query to check agent call volume:

SELECT user_id, COUNT(*) as call_count
FROM vicidial_log
WHERE inbound_group_id = 'SALES' AND call_date > DATE_SUB(NOW(), INTERVAL 1 HOUR)
GROUP BY user_id
ORDER BY call_count DESC;

If agent 1001 has handled 15 calls and agent 1003 has handled 2, and both have equal rank, check agent 1003's inbound_group assignment:

SELECT user_id, inbound_group, user_rank FROM vicidial_users WHERE user_id = '1003';

If inbound_group is NULL or different, they are not in the same queue.

Why does ViciDial skip agents even when they are available?

Agents are skipped if their status in vicidial_agent_log is not READY. Check directly:

mysql -u root -p asterisk -e \
  "SELECT user_id, status FROM vicidial_agent_log WHERE user_id IN ('1001','1002','1003');"

Possible statuses:

READY: Available for calls
INCALL: On an active call
PAUSED: Manually paused by agent or admin
RING: Phone is ringing, awaiting answer
DISPO: Entering call disposition
DEAD: Disconnected or logged off

If an agent shows DEAD or PAUSED when you expect READY, they were disconnected or paused. In the agent screen at /agc/vicidial.php, the agent can click Ready, or an admin can force the status in the database:

UPDATE vicidial_agent_log
SET status = 'READY'
WHERE user_id = '1003' AND campaign_id = 'SALES';

Restart the agent's Asterisk session to propagate the change:

asterisk -rx "hangup request channel SIP/1003@*"

The agent must log back into the agent screen.

Handling agent rank changes without dropping calls

If you change rank while an agent is logged in, the change takes effect on the next call offered to that agent, not immediately. Existing INCALL agents are not affected.

To change rank safely during business hours:

Check current active calls:

asterisk -rx "queue show SALES"

If few calls are active, update rank:

UPDATE vicidial_users SET user_rank = 5 WHERE user_id = '1003' AND inbound_group = 'SALES';

Reload the queue in Asterisk:

asterisk -rx "module reload app_queue.so"

Verify change took effect:

asterisk -rx "queue show SALES"

If calls drop, it means the extension or agent was in the queue at the moment of reload. This is rare but possible in high-traffic systems. During production, make rank changes at low-traffic windows (nights, weekends) or disable the inbound group temporarily:

Go to Inbound > Manage Inbound Groups, set the group status to Inactive, save. Agents will go PAUSED. Update rank. Re-enable the group.

Troubleshooting rank and load-balancing issues

Calls queue instead of ringing agents

Problem: Calls sit in queue, hold time increases, agents have zero calls.

Cause 1: Agents are offline or in wrong group.

Check:

mysql -u root -p asterisk -e "SELECT user_id, inbound_group, status FROM vicidial_agent_log WHERE campaign_id = 'SALES';"

If no rows return, no agents are logged into SALES. Ensure agents log into the correct group at /agc/vicidial.php.

Cause 2: Queue members are not properly registered.

Check extensions-vicidial.conf for the queue context:

grep -A 5 "\[vicidial\]" /etc/asterisk/extensions-vicidial.conf | head -20

Look for:

exten => _1XXX,1,Dial(SIP/${EXTEN}@default)

If the agent extension is not properly registered in SIP, Dial fails silently. Check SIP peers:

asterisk -rx "sip show peers"

Agent 1001 should appear as SIP/1001 (if registered) or SIP/1001@default (if via trunk). If not listed, they are not registered.

Cause 3: Queue strategy does not match agent load.

If using linear strategy, calls ring agents one at a time, waiting for timeout. With 20 agents and a 15-second timeout, the queue waits 15 seconds per agent before moving to the next. For the 5th agent, the caller waits 75 seconds.

Change strategy to leastrecent or rrmemory in queues.conf.

Calls always route to the same agent

Problem: Agent 1001 handles all calls. Agents 1002, 1003 never ring.

Cause 1: Rank is not set for other agents.

Check:

SELECT user_id, user_rank FROM vicidial_users WHERE inbound_group = 'SALES';

If user_rank is NULL for agents 1002 and 1003, they have no rank and are not in the queue.

Fix: Assign rank to all agents:

UPDATE vicidial_users SET user_rank = 2 WHERE user_id = '1002' AND inbound_group = 'SALES';
UPDATE vicidial_users SET user_rank = 3 WHERE user_id = '1003' AND inbound_group = 'SALES';

Reload the queue:

asterisk -rx "module reload app_queue.so"

Cause 2: Agent 1001 is the only one in the queue.

Check queues.conf:

grep -A 10 "^\[SALES\]" /etc/asterisk/queues.conf

If only one member is listed, manually add others or regenerate the file via the ViciDial admin. Go to Inbound > Manage Inbound Groups, select SALES, and confirm all agents are listed. Save and check queues.conf again.

Cause 3: Other agents have a higher penalty or invalid penalty syntax.

Check:

asterisk -rx "queue show SALES"

If agents show Penalty: 999 or Penalty: infinity, they are effectively disabled. Edit queues.conf and set penalty to a numeric value (0-999):

member => Local/1001@vicidial,penalty=0
member => Local/1002@vicidial,penalty=0
member => Local/1003@vicidial,penalty=0

Agent is rung twice for one call

Problem: Agent 1001 gets two simultaneous rings for a single inbound call.

Cause: Agent is listed twice in queues.conf or the member list was not reloaded.

Check:

grep "1001@vicidial" /etc/asterisk/queues.conf

If the line appears twice, remove the duplicate:

sed -i '0,/member => Local\/1001@vicidial/!b; /member => Local\/1001@vicidial/{x;d;}' /etc/asterisk/queues.conf

(This removes the first occurrence; adjust if you need to remove a different one.)

Reload:

asterisk -rx "module reload app_queue.so"

One agent's calls have long hold times despite low call volume

Problem: Agent 1002 gets calls with 5-minute hold times. Agent 1001 has zero hold time.

Cause 1: Agent 1002 has a higher penalty and is being skipped.

Check:

asterisk -rx "queue show SALES" | grep 1002

If Penalty is high (>10), lower it.

Cause 2: Agent 1001's phone is ringing all calls but agent is not answering.

Check the agent's extension registration and call logs. Is agent 1001 repeatedly rejecting calls or letting them time out?

Look at recent calls for agent 1001:

SELECT call_id, start_time, duration, status FROM vicidial_log WHERE user_id = '1001' ORDER BY start_time DESC LIMIT 10;

If duration is 0 and status is ABANDONED, the call was not answered. Check if agent 1001's phone is registered and online in SIP:

asterisk -rx "sip show peers" | grep 1001

If unreachable or offline, agent will not answer calls. Ask agent to log in via SIP client or check their phone.

Frequently asked questions

Why does my inbound group show zero members after I assign rank to agents?

The queue members are populated from vicidial_users at module reload time. If rank is NULL or inbound_group does not match the queue name, the agent is not added. Check that inbound_group is set to the exact queue name (case-sensitive) and user_rank is a number between 0 and 9999. Then run asterisk -rx "module reload app_queue.so" and verify with queue show [groupname].

Can I assign the same rank to multiple agents?

Yes, but behavior depends on strategy. With ringall, both ring simultaneously. With leastrecent or rrmemory, the system tracks which agent was called least recently or last and picks that one. Assigning the same rank to two agents is equivalent to having no rank separation; use it only if agents have identical skill levels and you want simultaneous ring.

How do I prioritize VIP callers to a specific agent or rank?

ViciDial does not have built-in VIP queue routing by caller ID, but you can create a separate inbound group for VIP calls with a single dedicated agent, then route DID entries to that group. Alternatively, use a dialplan variable to set a higher priority in the queue by checking inbound_cid against a list and using asterisk's priorities or a custom AGI script to inject the call higher in the queue.

What happens if I delete or merge two inbound groups?

If you delete SALES inbound group in the web admin, all agents in SALES lose their inbound_group assignment (it becomes NULL), and calls to DIDs mapped to SALES will fail because the queue no longer exists. To merge SALES and SALES_BACKUP, update all agents in SALES_BACKUP to inbound_group = 'SALES' and delete SALES_BACKUP. Update the DID routing if necessary.

How do rank and penalty interact in Asterisk?

Rank is applied first by ViciDial's dial-plan; it determines which agents are in the queue and in which order they are listed. Penalty is then applied by Asterisk's queue engine to further reorder or deprioritize agents. If rank 1 has 0 penalty and rank 2 has 20 penalty, rank 1 is always called first; rank 2 is called only if rank 1 is not available. Penalty does not override rank; it supplements it.

Summary

Agent rank in ViciDial controls the order in which agents are offered calls within an inbound group. Setting rank correctly requires updating the vicidial_users table (user_rank field), choosing an appropriate queue strategy (leastrecent or rrmemory for most setups), and verifying agent inbound_group assignment matches the queue name. Load balancing is further refined using Asterisk's penalty system in queues.conf to deprioritize overflow agents or skill-based tiers. Testing is done by logging in two or more agents, sending test calls, and monitoring queue and agent status via Asterisk CLI. Common issues include NULL rank, incorrect inbound_group, offline agents, duplicate queue members, and mismatched strategies. Monitor call distribution via the agent performance report or direct SQL queries on vicidial_log to verify even load. Changes to rank take effect on the next call offered after a queue reload; high-traffic systems should make rank changes during off-peak hours or disable the group temporarily.

Query Millions of VoIP CDRs with DuckDB on Budget VPS

Moisi Trungu — Tue, 20 Oct 2026 03:30:03 +0000

Query Millions of VoIP CDRs with DuckDB on Budget VPS

Extract and analyze call records from Asterisk and ViciDial in seconds on 1GB RAM by switching from SQL to column-oriented format.

DuckDB queries raw CDR data 10 to 100 times faster than MySQL because it operates on compressed columnar storage instead of row-by-row lookups. On a budget VPS with 1GB RAM, you can load millions of call records from ViciDial's vicidial_log table, run complex aggregations on agent performance, call duration, and carrier routes, then export results in seconds. The setup requires only a single binary, basic file format conversion, and a cron job. This tutorial walks through importing real Asterisk CDRs into DuckDB, writing practical queries for call center operations, and troubleshooting the gaps between what ViciDial logs and what you actually need to see.

Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, MariaDB 10.5+, DuckDB 0.9.1+, on Debian 11/12 with 1GB RAM.

Prerequisites

Running ViciDial instance with MariaDB or MySQL backend
SSH access to the VPS where ViciDial runs (or a separate analytics host)
Asterisk 16+ with CDR logging enabled
500 MB free disk space for sample data
wget or curl to download DuckDB binary
Basic knowledge of SQL SELECT statements

No GUI tools needed. All work happens in the shell.

Why switch from MySQL to DuckDB for call records?

When you have 10 million call records in vicidial_log, a MySQL query like "SELECT COUNT(*), AVG(LENGTH_IN_SEC) FROM vicidial_log WHERE date BETWEEN..." can take 30 seconds or more. The database must scan every row, decompress fields, and filter. DuckDB pre-compresses data by column: all the LENGTH_IN_SEC values live together, indexed and packed, so the same query finishes in under a second on a single-core VPS.

The tradeoff: DuckDB is read-only for analytics. You do not replace ViciDial's live MariaDB. Instead, you export CDR snapshots (daily or hourly) into Parquet or CSV format, load them into DuckDB, and run reports offline. Write operations stay in MySQL. This pattern works because call records rarely change after the call ends.

Step 1: Download and install DuckDB

SSH into your VPS and install the DuckDB CLI binary.

cd /opt
wget https://github.com/duckdb/duckdb/releases/download/v0.9.1/duckdb_cli-linux-x86_64.zip
unzip duckdb_cli-linux-x86_64.zip
chmod +x duckdb
ln -s /opt/duckdb /usr/local/bin/duckdb
duckdb --version

Verify installation:

duckdb :memory: "SELECT 1 as test;"

Output should be:

┌────────┐
│ test   │
│ int64  │
├────────┤
│ 1      │
└────────┘

If you are on a system with restricted /opt permissions, extract to /home/vicidial/ or /tmp/ instead.

Step 2: Export ViciDial CDRs to CSV format

ViciDial stores call logs in the vicidial_log table. You will export a date range to a CSV file that DuckDB can ingest. Use the MariaDB client directly on the VPS.

mysql -u root -p asterisk -e "SELECT 
  call_date, 
  call_time, 
  caller_id_number, 
  called_party_number, 
  length_in_sec, 
  agent_id, 
  campaign_id, 
  call_status, 
  lead_id, 
  phone_number,
  user_id
FROM vicidial_log 
WHERE call_date >= '2024-01-15' AND call_date <= '2024-01-21'
INTO OUTFILE '/tmp/vicidial_cdr.csv' 
FIELDS TERMINATED BY ',' 
ENCLOSED BY '\"' 
LINES TERMINATED BY '\n';"

If you are on a managed host and do not have OUTFILE permission, use an alternative:

mysql -u root -p asterisk --batch --skip-column-names -e \
  "SELECT call_date, call_time, caller_id_number, called_party_number, \
  length_in_sec, agent_id, campaign_id, call_status, lead_id, phone_number, user_id \
  FROM vicidial_log WHERE call_date >= '2024-01-15'" | \
  sed 's/\t/,/g' > /tmp/vicidial_cdr.csv

Verify the file was created and contains data:

wc -l /tmp/vicidial_cdr.csv
head -3 /tmp/vicidial_cdr.csv

Sample output:

1247563 /tmp/vicidial_cdr.csv
2024-01-15,08:23:45,5551234567,5559876543,87,1023,SALES,LIVE,55412,5559876543,1023
2024-01-15,08:45:12,5551234560,5559876544,156,1024,SALES,LIVE,55413,5559876544,1024

Step 3: Create a DuckDB database and import the CSV

Initialize a DuckDB file and create a table from the CSV.

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
CREATE TABLE vicidial_log AS
SELECT 
  CAST(call_date AS DATE) as call_date,
  call_time,
  caller_id_number,
  called_party_number,
  CAST(length_in_sec AS INTEGER) as length_in_sec,
  CAST(agent_id AS VARCHAR) as agent_id,
  campaign_id,
  call_status,
  CAST(lead_id AS BIGINT) as lead_id,
  phone_number,
  CAST(user_id AS INTEGER) as user_id
FROM read_csv_auto('/tmp/vicidial_cdr.csv', sample_size=-1);

SELECT COUNT(*) as record_count, 
  MIN(call_date) as earliest_date,
  MAX(call_date) as latest_date
FROM vicidial_log;
EOF

DuckDB will auto-detect column types. Verify the import by querying row and date counts.

Expected output (adjust numbers to your data):

┌──────────────┬───────────────┬─────────────┐
│ record_count │ earliest_date │ latest_date │
│   1247562    │   2024-01-15  │  2024-01-21 │
└──────────────┴───────────────┴─────────────┘

Step 4: Automate daily CDR exports and DuckDB ingestion

Create a shell script that runs via cron each night to pull the previous day's records and append them to the DuckDB database.

cat > /home/vicidial/sync_cdr_to_duckdb.sh << 'SCRIPT'
#!/bin/bash

VICIDIAL_HOME="/home/vicidial"
DB_FILE="$VICIDIAL_HOME/cdr_analysis.duckdb"
CSV_TMP="/tmp/vicidial_cdr_daily.csv"
YESTERDAY=$(date -d "yesterday" +%Y-%m-%d)
YESTERDAY_END=$(date -d "yesterday" +%Y-%m-%d)
LOG_FILE="/var/log/duckdb_sync.log"

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting CDR sync for $YESTERDAY" >> $LOG_FILE

# Export yesterday's records from MariaDB
mysql -u root -p'password_here' asterisk --batch --skip-column-names -e \
  "SELECT call_date, call_time, caller_id_number, called_party_number, \
   length_in_sec, agent_id, campaign_id, call_status, lead_id, phone_number, user_id \
   FROM vicidial_log WHERE call_date = '$YESTERDAY'" | \
  sed 's/\t/,/g' > "$CSV_TMP"

if [ ! -s "$CSV_TMP" ]; then
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] No data found for $YESTERDAY, exiting" >> $LOG_FILE
  exit 0
fi

# Append to DuckDB
duckdb "$DB_FILE" << EOF
INSERT INTO vicidial_log
SELECT 
  CAST(call_date AS DATE) as call_date,
  call_time,
  caller_id_number,
  called_party_number,
  CAST(length_in_sec AS INTEGER) as length_in_sec,
  CAST(agent_id AS VARCHAR) as agent_id,
  campaign_id,
  call_status,
  CAST(lead_id AS BIGINT) as lead_id,
  phone_number,
  CAST(user_id AS INTEGER) as user_id
FROM read_csv_auto('$CSV_TMP', sample_size=-1);

SELECT COUNT(*) as rows_inserted FROM (
  SELECT * FROM vicidial_log WHERE call_date = CAST('$YESTERDAY' AS DATE)
);
EOF

RESULT=$?
if [ $RESULT -eq 0 ]; then
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] Sync completed successfully" >> $LOG_FILE
  rm -f "$CSV_TMP"
else
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] Sync failed with code $RESULT" >> $LOG_FILE
fi
SCRIPT

chmod +x /home/vicidial/sync_cdr_to_duckdb.sh

Edit the script to use your actual MariaDB password, then add to crontab:

crontab -e

Add the line:

0 1 * * * /home/vicidial/sync_cdr_to_duckdb.sh

This runs the sync script every day at 01:00 UTC (adjust the hour as needed).

Step 5: Query call data by agent performance

Now that CDRs are in DuckDB, write queries for common reporting needs. All examples assume your DuckDB file is at /home/vicidial/cdr_analysis.duckdb.

Daily call volume and average handling time by agent

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
SELECT 
  call_date,
  agent_id,
  COUNT(*) as calls_handled,
  ROUND(AVG(length_in_sec), 2) as avg_duration_sec,
  SUM(length_in_sec) as total_handle_time_sec,
  COUNT(CASE WHEN call_status = 'LIVE' THEN 1 END) as connected_calls,
  ROUND(100.0 * COUNT(CASE WHEN call_status = 'LIVE' THEN 1 END) / COUNT(*), 2) as connect_rate_pct
FROM vicidial_log
WHERE call_date >= '2024-01-15'
  AND call_date < '2024-01-22'
  AND agent_id IS NOT NULL
GROUP BY call_date, agent_id
ORDER BY call_date DESC, calls_handled DESC;
EOF

Output sample:

┌────────────┬──────────┬────────────────┬─────────────────────┬──────────────────────┬─────────────────┬──────────────────┐
│ call_date  │ agent_id │ calls_handled  │ avg_duration_sec    │ total_handle_time_sec│ connected_calls  │ connect_rate_pct │
│ 2024-01-21 │ 1023     │ 145            │ 187.43              │ 27176                │ 121              │ 83.45            │
│ 2024-01-21 │ 1024     │ 138            │ 201.12              │ 27754                │ 118              │ 85.51            │
│ 2024-01-20 │ 1023     │ 142            │ 183.22              │ 25996                │ 119              │ 83.80            │
└────────────┴──────────┴────────────────┴─────────────────────┴──────────────────────┴─────────────────┴──────────────────┘

Campaign performance: total cost per lead by carrier

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
SELECT 
  campaign_id,
  call_status,
  COUNT(*) as attempt_count,
  COUNT(DISTINCT lead_id) as unique_leads,
  ROUND(AVG(length_in_sec), 1) as avg_duration_sec,
  SUM(CASE WHEN length_in_sec >= 10 THEN 1 ELSE 0 END) as calls_over_10sec
FROM vicidial_log
WHERE call_date >= '2024-01-15'
GROUP BY campaign_id, call_status
ORDER BY campaign_id, attempt_count DESC;
EOF

Find long calls that may indicate transfer or hold

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
SELECT 
  call_date,
  call_time,
  agent_id,
  caller_id_number,
  called_party_number,
  length_in_sec,
  call_status
FROM vicidial_log
WHERE length_in_sec > 600
  AND call_date = '2024-01-21'
ORDER BY length_in_sec DESC
LIMIT 20;
EOF

Hourly call volume distribution

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
SELECT 
  call_date,
  SUBSTR(call_time, 1, 2) as hour,
  COUNT(*) as call_count,
  ROUND(AVG(length_in_sec), 1) as avg_duration_sec
FROM vicidial_log
WHERE call_date >= '2024-01-15'
GROUP BY call_date, SUBSTR(call_time, 1, 2)
ORDER BY call_date, hour;
EOF

Step 6: Export query results for reporting

DuckDB can output results in JSON, CSV, or Parquet formats. Use this to feed dashboards or spreadsheets.

duckdb /home/vicidial/cdr_analysis.duckdb -json << 'EOF' > /tmp/agent_report.json
SELECT 
  call_date,
  agent_id,
  COUNT(*) as calls_handled,
  ROUND(AVG(length_in_sec), 2) as avg_duration_sec
FROM vicidial_log
WHERE call_date >= '2024-01-15'
GROUP BY call_date, agent_id;
EOF

Or export to CSV:

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
COPY (
  SELECT 
    call_date,
    agent_id,
    COUNT(*) as calls_handled,
    ROUND(AVG(length_in_sec), 2) as avg_duration_sec
  FROM vicidial_log
  WHERE call_date >= '2024-01-15'
  GROUP BY call_date, agent_id
) TO '/tmp/agent_report.csv' (FORMAT CSV, HEADER TRUE);
EOF

Retrieve the file for import into your BI tool or Excel.

Step 7: Handle missing or malformed CDR data

Not all ViciDial configurations log every field consistently. You may encounter NULLs, empty strings, or misaligned columns.

When importing, check for data quality first:

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
SELECT 
  COUNT(*) as total_rows,
  COUNT(agent_id) as rows_with_agent,
  COUNT(call_date) as rows_with_date,
  MIN(length_in_sec) as min_duration,
  MAX(length_in_sec) as max_duration
FROM vicidial_log;
EOF

If you see a large number of NULL agent_ids, your ViciDial instance may not be logging agent interactions. Check that vicidial_log is configured to capture agent_id. In the ViciDial admin web interface, navigate to System Settings, then verify "Log Agent" is enabled.

If you see negative or extremely large values in length_in_sec, add a filter to the import query:

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
CREATE TABLE vicidial_log_clean AS
SELECT * FROM vicidial_log
WHERE length_in_sec >= 0 AND length_in_sec <= 36000;
EOF

This removes calls with duration over 10 hours (likely corrupted records).

Step 8: Set up a read-only DuckDB endpoint for other team members

If you have multiple analysts on the VPS, you can create a simple shell wrapper that allows running queries without exposing the database file directly.

cat > /usr/local/bin/query_cdr << 'SCRIPT'
#!/bin/bash
# Simple read-only CDR query wrapper

if [ -z "$1" ]; then
  echo "Usage: query_cdr 'SELECT ...' [format]"
  echo "Format: table, json, csv (default: table)"
  exit 1
fi

FORMAT=${2:-table}

case $FORMAT in
  json)
    duckdb /home/vicidial/cdr_analysis.duckdb -json "$1"
    ;;
  csv)
    duckdb /home/vicidial/cdr_analysis.duckdb << EOF
COPY ($1) TO '/dev/stdout' (FORMAT CSV, HEADER TRUE);
EOF
    ;;
  *)
    duckdb /home/vicidial/cdr_analysis.duckdb "$1"
    ;;
esac
SCRIPT

chmod +x /usr/local/bin/query_cdr

Team members can now run:

query_cdr "SELECT COUNT(*) FROM vicidial_log WHERE call_date = '2024-01-21'" csv

Note: This does not implement row-level access control. For stricter security, use a web API wrapper such as DuckDB's official Python HTTP server, which can enforce authentication.

Step 9: Optimize DuckDB for very large datasets

When your DuckDB file grows beyond 5GB, consider these optimizations.

Use Parquet format instead of CSV for imports

Parquet is compressed and column-oriented, much faster to load than CSV:

# Export from MariaDB to Parquet via Python
python3 << 'PYTHON'
import duckdb
import pandas as pd
import pymysql

conn = pymysql.connect(
  host="localhost",
  user="root",
  password="password",
  database="asterisk"
)

df = pd.read_sql(
  "SELECT * FROM vicidial_log WHERE call_date = '2024-01-21'",
  conn
)

duckdb_conn = duckdb.connect('/home/vicidial/cdr_analysis.duckdb')
duckdb_conn.register('df', df)
duckdb_conn.execute("""
  INSERT INTO vicidial_log SELECT * FROM df
""")
duckdb_conn.close()
conn.close()
PYTHON

Partition the database by date

Instead of one large table, create separate tables or Parquet files by month:

for month in 2024-01 2024-02 2024-03; do
  mysql -u root -p asterisk -e "SELECT * FROM vicidial_log WHERE call_date LIKE '$month%' INTO OUTFILE '/tmp/cdr_$month.csv' ..." &
done

Then load each month into a separate DuckDB instance or use DuckDB's partitioned table support.

Troubleshooting

DuckDB returns "No such table" error

Verify the table exists by connecting to the database file and listing tables:

duckdb /home/vicidial/cdr_analysis.duckdb ".tables"

If the table is missing, reimport the CSV. Confirm that the CSV file path in the read_csv_auto() function is absolute or relative to the working directory.

Cron job for sync_cdr_to_duckdb.sh did not run

Check the cron log:

grep "sync_cdr_to_duckdb" /var/log/syslog
# or on systemd-based systems:
journalctl -u cron

Verify the script has execute permissions:

ls -la /home/vicidial/sync_cdr_to_duckdb.sh

Test the script manually:

/home/vicidial/sync_cdr_to_duckdb.sh
cat /var/log/duckdb_sync.log

CSV import is very slow on large files

For files over 100MB, increase DuckDB's thread count:

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
SET threads = 4;
CREATE TABLE vicidial_log AS SELECT * FROM read_csv_auto('/tmp/vicidial_cdr.csv');
EOF

Or pre-compress the CSV using gzip and import directly:

gzip /tmp/vicidial_cdr.csv
duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
CREATE TABLE vicidial_log AS SELECT * FROM read_csv_auto('/tmp/vicidial_cdr.csv.gz');
EOF

DuckDB process runs out of memory on 1GB VPS

DuckDB is designed for in-memory performance, but on constrained hardware, you may hit swap. Reduce the number of concurrent queries or partition your data:

# Limit DuckDB to 512MB memory
duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
SET memory_limit = '512MB';
SELECT * FROM vicidial_log WHERE call_date = '2024-01-21' LIMIT 100;
EOF

Or reduce the date range per query:

# Query by single day instead of full week
for day in {15..21}; do
  duckdb /home/vicidial/cdr_analysis.duckdb << EOF
SELECT * FROM vicidial_log WHERE call_date = '2024-01-$day' ...
EOF
done

Query results include rows from calls without an agent_id

By design, DuckDB reads all rows. If your reporting requirement is "only calls handled by agents," filter explicitly:

SELECT * FROM vicidial_log
WHERE call_date = '2024-01-21'
  AND agent_id IS NOT NULL
  AND agent_id <> '';

Or create a materialized view:

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
CREATE VIEW agent_calls AS
SELECT * FROM vicidial_log
WHERE agent_id IS NOT NULL AND agent_id <> '';
EOF

Then query the view:

SELECT COUNT(*) FROM agent_calls WHERE call_date = '2024-01-21';

Frequently asked questions

Why does DuckDB require me to export CDRs first instead of querying MariaDB directly?

DuckDB is a columnar analytical database optimized for aggregations on static snapshots. MariaDB is a transactional row store optimized for updates. If you query MariaDB directly via a tool, it will still scan full tables row-by-row. DuckDB's advantage comes from copying the data into its compressed columnar format, which trades write speed for read speed. For live agent dashboards, keep querying MariaDB. For historical reports and bulk analysis, use DuckDB.

Can I delete old records from DuckDB to save disk space?

Yes. DuckDB tables support DELETE statements. To remove records older than 90 days:

duckdb /home/vicidial/cdr_analysis.duckdb << 'EOF'
DELETE FROM vicidial_log WHERE call_date < DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY);
VACUUM;
EOF

The VACUUM command reclaims disk space. Alternatively, do not import data older than your retention window in the first place.

How do I query DuckDB from a Python script without shelling out to the CLI?

Install the DuckDB Python package and use it as a library:

pip install duckdb

Then in your Python code:

import duckdb

conn = duckdb.connect('/home/vicidial/cdr_analysis.duckdb')
result = conn.execute("SELECT COUNT(*) FROM vicidial_log WHERE call_date = '2024-01-21'").fetchall()
print(result[0][0])
conn.close()

This avoids spawning a subprocess and is faster for multiple queries.

What if my ViciDial instance uses a different table schema, with custom columns?

Export only the columns that exist and are relevant to your analysis. Check your schema first:

mysql asterisk -e "DESC vicidial_log;" | head -20

Adapt the SELECT statement in the export script to include only those columns. DuckDB will infer types automatically. If a column is missing, add it to the export and then add it to the CREATE TABLE statement.

Can DuckDB work with live Asterisk CEL (Channel Event Log) data instead of ViciDial's vicidial_log?

Yes. Asterisk's CEL table in the asterisk database contains finer-grained events. Export it the same way:

mysql asterisk -e "SELECT * FROM cel WHERE eventtime > DATE_SUB(NOW(), INTERVAL 1 DAY) INTO OUTFILE '/tmp/asterisk_cel.csv' ..."

Then import to DuckDB. CEL data is more voluminous than CDR summaries, so expect larger files and longer query times. Filter by eventtype if you only need call setup, connect, and hangup events.

Summary

You have set up a production CDR analytics pipeline using DuckDB on a budget VPS. The workflow is:

Export daily call records from ViciDial's vicidial_log table to CSV format via MySQL.
Load the CSV into a DuckDB database file using read_csv_auto() and INSERT statements.
Run analytical queries on millions of records in milliseconds, grouping by agent, campaign, status, and time.
Export results to JSON or CSV for reporting.
Automate the daily import via a cron job.

Next steps: adjust the sync script to match your MariaDB credentials and backup schedule, add Parquet file exports for archival beyond 90 days, and consider setting up a scheduled report that emails query results to your management team each morning. Monitor the DuckDB file size and cron logs weekly to catch any import failures early. If queries slow down as data accumulates, partition by month or switch to quarterly analysis windows instead of appending to a single table indefinitely.

AI-Assisted VoIP Call Center Ops: Claude Hooks & Incident KB

Moisi Trungu — Mon, 19 Oct 2026 03:30:03 +0000

AI-Assisted VoIP Call Center Ops: Claude Hooks & Incident KB

Route inbound calls to the right agent queue in real time, auto-resolve carrier codec errors, and reduce mean time to recovery (MTTR) by 60% with Claude-powered incident detection in ViciDial.

Silence, one-way audio, and dropped calls in ViciDial usually trace to three causes: SIP codec mismatch between carrier and agent trunk, RTP socket timeout from firewall rule drift, or call state corruption in the vicidial_log table when agents go unavailable mid-call. Claude hooks hooked into your Asterisk dialplan can detect these patterns in near real time, query the ViciDial incident knowledge base, and auto-remediate by forcing G.711 transcoding, flushing stale RTP sessions, or rolling back agent call state without human intervention.

This guide covers building a production Claude integration for VoIP call center operations, including incident detection hooks in the Asterisk dialplan, a queryable incident knowledge base in ViciDial, and practical troubleshooting workflows for the most common failure modes.

Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, MariaDB 10.5+, and Claude API (claude-3-sonnet-20250229 or later).

Prerequisites

ViciDial 2.14 or later with Asterisk 16/18 running on CentOS 7 or Ubuntu 20.04 LTS.
Direct SSH access to the ViciDial server and permission to modify /etc/asterisk/ files.
MariaDB 10.5+ with root access to the asterisk database.
Python 3.10+ installed on the ViciDial box (or a separate worker box with network access to Asterisk and MySQL).
Claude API key with allowance for ~100-200 tokens per incident (cost ~$0.001 per incident).
Familiarity with Asterisk dialplan syntax (extensions-vicidial.conf, sip-vicidial.conf).
A text editor (nano, vim) and basic shell scripting.
curl or a Python HTTP library to post logs to Claude.

Why AI for call center ops at all?

Call centers generate logs faster than humans read them. A single SIP carrier misconfiguration can spawn 500 error lines in Asterisk console within seconds. The pattern is obvious to Claude: you see two consecutive CODEC MISMATCH warnings followed by RTCP TIMEOUT, and you know the inbound trunk is configured for uLaw but the agent endpoint reports aLaw. Claude can spot this correlation, query your carrier contract database, auto-suggest the fix, or execute it directly via AGI. This saves your on-call engineer 20 minutes of grep and packet sniffing per incident.

Architecture overview

The workflow is three parts:

Dialplan hooks in extensions-vicidial.conf that invoke a Python AGI script on call events (answer, hangup, transfer).
Incident detection logic written as a Python AGI that logs call state to MySQL and calls Claude with structured prompts.
Claude response parsed into remediation actions: restart a trunk, force codec, update call state, or page an oncall engineer via webhook.

The incident knowledge base lives in two tables:

vicidial_incident_log: raw call state at the moment of failure.
vicidial_incident_kb: a curated collection of known issues, root causes, and fixes that you feed to Claude as context.

Setting up the incident knowledge base tables

First, create the knowledge base tables in the asterisk database:

USE asterisk;

CREATE TABLE IF NOT EXISTS vicidial_incident_kb (
  incident_id INT AUTO_INCREMENT PRIMARY KEY,
  incident_name VARCHAR(255) NOT NULL UNIQUE,
  symptom VARCHAR(512),
  root_cause VARCHAR(512),
  detection_method VARCHAR(255),
  fix_command TEXT,
  fix_priority INT DEFAULT 50,
  created_date DATETIME DEFAULT CURRENT_TIMESTAMP,
  updated_date DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  enabled TINYINT DEFAULT 1
);

CREATE TABLE IF NOT EXISTS vicidial_incident_log (
  event_id INT AUTO_INCREMENT PRIMARY KEY,
  call_id VARCHAR(50),
  session_id VARCHAR(100),
  agent_id VARCHAR(20),
  carrier_id VARCHAR(50),
  trunk_name VARCHAR(100),
  event_type VARCHAR(50),
  event_timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
  event_detail TEXT,
  codec_in VARCHAR(20),
  codec_out VARCHAR(20),
  rtp_packets_lost INT,
  one_way_audio TINYINT DEFAULT 0,
  jitter_ms FLOAT,
  latency_ms FLOAT,
  resolved_by VARCHAR(100),
  resolution_time INT,
  claude_response TEXT,
  INDEX idx_call_id (call_id),
  INDEX idx_carrier (carrier_id),
  INDEX idx_trunk (trunk_name),
  INDEX idx_event_type (event_type),
  INDEX idx_timestamp (event_timestamp)
);

INSERT INTO vicidial_incident_kb (incident_name, symptom, root_cause, detection_method, fix_command, fix_priority)
VALUES
  ('codec_mismatch_ulaw_alaw', 'One-way audio or silence after answer', 'Inbound SIP carrier set to uLaw but agent endpoint is aLaw', 'CODEC MISMATCH in console AND RTCP timeout within 3 seconds', 'asterisk -rx "sip set debug on"; then review sip_vicidial.conf disallow/allow', 90),
  ('rtp_timeout_firewall', 'Audio drops after 30-45 seconds mid-call', 'Firewall rule for RTP socket timeout or NAT traversal failure', 'RTP socket timeout in Asterisk logs; traceroute shows packet loss on port range 10000-20000', 'Check iptables -L -n; verify udp 10000:20000 is ACCEPT for carrier IP; restart SIP channel', 85),
  ('stale_call_state_db', 'Agent marked unavailable but still in call queue', 'vicidial_log entry not updated when call ended; race condition in hangup handler', 'SELECT status FROM vicidial_log WHERE uniqueid matches call; status is still QUEUE or IN-CALL', 'UPDATE vicidial_log SET status=DONE WHERE uniqueid=X; restart vicidial agent screen', 75),
  ('dtmf_timeout_ivr', 'IVR does not register DTMF keypresses', 'SIP INFO frame setting incorrect; uLaw audio frame timing drift', 'No DTMF events in Asterisk verbose logs; SIP debug shows no INFO frames', 'asterisk -rx "sip set debug on"; verify dtmf_mode=auto in sip_vicidial.conf; check carrier SIP headers', 70),
  ('carrier_registration_failed', 'Inbound trunk shows UNREACHABLE', 'SIP REGISTER to carrier fails: bad auth or IP blacklist', 'Asterisk shows "Registration failed for X" on console; carrier portal shows failed logins', 'Verify SIP credentials; check IP whitelist at carrier; asterisk -rx "sip show registry"', 95);

Now insert some baseline context for Claude. This table will grow over time as you encounter and document new issues.

Building the AGI hook script

Create a Python AGI script at /usr/local/bin/vicidial_incident_agi.py:

#!/usr/bin/env python3
"""
ViciDial incident detection AGI hook.
Logs call state events to vicidial_incident_log and calls Claude for analysis.
Usage: Add to extensions-vicidial.conf under appropriate contexts.
"""

import sys
import os
import re
import json
import time
import MySQLdb
import requests
from datetime import datetime

# Configuration
MYSQL_HOST = "localhost"
MYSQL_USER = "root"
MYSQL_PASS = "your_db_password"
MYSQL_DB = "asterisk"
CLAUDE_API_KEY = os.environ.get("CLAUDE_API_KEY", "sk-...")
CLAUDE_MODEL = "claude-3-sonnet-20250229"
ASTERISK_LOG_PATH = "/var/log/asterisk/messages"

class VicidalIncidentAGI:
    def __init__(self):
        self.agi_data = {}
        self.db = None
        self.connect_db()

    def connect_db(self):
        try:
            self.db = MySQLdb.connect(
                host=MYSQL_HOST, user=MYSQL_USER, passwd=MYSQL_PASS, db=MYSQL_DB
            )
            self.db.set_character_set('utf8mb4')
        except MySQLdb.Error as e:
            self.log_error(f"DB connection failed: {e}")
            sys.exit(1)

    def read_agi_input(self):
        """Read AGI variables from Asterisk."""
        while True:
            line = sys.stdin.readline().strip()
            if not line:
                break
            if line.startswith("agi_"):
                key, val = line.split(":", 1)
                self.agi_data[key.strip()] = val.strip()

    def log_error(self, msg):
        """Write to Asterisk console and syslog."""
        print(f"VERBOSE \"{msg}\" 3", file=sys.stderr)
        sys.stderr.flush()

    def query_kb(self, symptom_keywords):
        """Fetch matching KB entries for Claude context."""
        try:
            cursor = self.db.cursor(MySQLdb.cursors.DictCursor)
            placeholders = ",".join(["%s"] * len(symptom_keywords))
            query = f"""
                SELECT incident_name, symptom, root_cause, fix_command, fix_priority
                FROM vicidial_incident_kb
                WHERE enabled = 1 AND (
                    {" OR ".join(f"symptom LIKE %s" for _ in symptom_keywords)}
                    OR
                    {" OR ".join(f"incident_name LIKE %s" for _ in symptom_keywords)}
                )
                ORDER BY fix_priority DESC LIMIT 10
            """
            params = symptom_keywords * 2
            cursor.execute(query, params)
            results = cursor.fetchall()
            cursor.close()
            return results
        except Exception as e:
            self.log_error(f"KB query failed: {e}")
            return []

    def tail_asterisk_log(self, lines=50):
        """Read last N lines of Asterisk log for context."""
        try:
            with open(ASTERISK_LOG_PATH, "r") as f:
                log_lines = f.readlines()[-lines:]
                return "".join(log_lines)
        except Exception as e:
            self.log_error(f"Log read failed: {e}")
            return ""

    def log_incident(self, call_id, event_type, detail):
        """Write event to vicidial_incident_log."""
        try:
            cursor = self.db.cursor()
            agent_id = self.agi_data.get("agi_extension", "unknown")
            carrier_id = self.agi_data.get("agi_arg_1", "unknown")
            trunk_name = self.agi_data.get("agi_arg_2", "unknown")

            sql = """
                INSERT INTO vicidial_incident_log
                (call_id, agent_id, carrier_id, trunk_name, event_type, event_detail, event_timestamp)
                VALUES (%s, %s, %s, %s, %s, %s, NOW())
            """
            cursor.execute(sql, (call_id, agent_id, carrier_id, trunk_name, event_type, detail))
            self.db.commit()
            cursor.close()
            return True
        except Exception as e:
            self.log_error(f"Incident log failed: {e}")
            return False

    def call_claude(self, prompt):
        """Call Claude API with incident context."""
        try:
            headers = {
                "Content-Type": "application/json",
                "x-api-key": CLAUDE_API_KEY,
            }
            payload = {
                "model": CLAUDE_MODEL,
                "max_tokens": 500,
                "messages": [
                    {"role": "user", "content": prompt}
                ]
            }
            response = requests.post(
                "https://api.anthropic.com/v1/messages",
                headers=headers,
                json=payload,
                timeout=10
            )
            response.raise_for_status()
            result = response.json()
            if result["stop_reason"] == "end_turn":
                return result["content"][0]["text"]
            return None
        except Exception as e:
            self.log_error(f"Claude API call failed: {e}")
            return None

    def handle_call_event(self):
        """Main event handler."""
        self.read_agi_input()
        call_id = self.agi_data.get("agi_uniqueid", "unknown")
        event_type = self.agi_data.get("agi_arg_0", "unknown")

        self.log_error(f"Incident AGI: call={call_id}, event={event_type}")

        # Determine if this is a problem event
        problem_keywords = ["CODEC", "MISMATCH", "TIMEOUT", "UNREACHABLE", "FAILED"]
        asterisk_log = self.tail_asterisk_log(100)
        has_problem = any(kw in asterisk_log for kw in problem_keywords)

        if not has_problem:
            return

        # Query KB for similar issues
        kb_results = self.query_kb(["codec", "timeout", "unreachable"])
        kb_context = "\n".join([
            f"- {r['incident_name']}: {r['symptom']} (fix: {r['fix_command'][:100]})"
            for r in kb_results
        ])

        # Build prompt for Claude
        prompt = f"""
You are a VoIP SIP call center operations expert. A call center incident was detected:

Call ID: {call_id}
Event type: {event_type}
Recent Asterisk logs (last 100 lines):
{asterisk_log}

Known similar incidents from our KB:
{kb_context}

Based on the logs and KB, identify:
1. Most likely root cause (one sentence).
2. Recommended immediate action (one sentence).
3. If safe to auto-remediate, suggest the exact CLI command.
4. Otherwise, suggest what to check first (one sentence).

Keep responses concise. Assume the operator has SSH access to the Asterisk box.
"""
        response = self.call_claude(prompt)
        if response:
            self.log_error(f"Claude analysis: {response[:200]}")
            self.log_incident(call_id, event_type, f"Claude: {response}")
        else:
            self.log_incident(call_id, event_type, "Claude call failed; check logs.")

    def close(self):
        if self.db:
            self.db.close()

if __name__ == "__main__":
    agi = VicidalIncidentAGI()
    try:
        agi.handle_call_event()
    finally:
        agi.close()

Make the script executable:

chmod +x /usr/local/bin/vicidial_incident_agi.py

Hooking the AGI into the dialplan

Edit /etc/asterisk/extensions-vicidial.conf and add a macro or context that invokes the AGI on inbound calls:

[macro-incident-hook]
exten => s,1,NoOp(Incident hook for ${UNIQUEID})
same => n,AGI(agi://localhost/vicidial_incident_agi.py,${ARG1},${ARG2},${ARG3})
same => n,Return()

[from-vicidial-inbound]
exten => _X.,1,NoOp(Inbound call from ${CALLERID(num)} to ${EXTEN})
same => n,Macro(incident-hook,ANSWER,${TRUNK_NAME},${CARRIER_ID})
same => n,ViciDial(${EXTEN},,,,,,,,)
same => n,Macro(incident-hook,HANGUP,${TRUNK_NAME},${CARRIER_ID})
same => n,Hangup()

On inbound call answer, the AGI is triggered and will check for any codec or timeout issues in the last 100 lines of the Asterisk log. If problems are detected, it queries the KB and calls Claude.

Diagnosing SIP codec mismatches

The most common incident is CODEC MISMATCH. To verify it manually:

asterisk -rx "sip show peers" | grep -E "UNREACHABLE|OK"
asterisk -rx "sip set debug on"

Then place a test call and watch the console. Look for lines like:

[2025-02-14 14:32:10] VERBOSE: channel.c: Channel SIP/carrier-00000001 answered SIP/agent-6001-00000002
[2025-02-14 14:32:11] WARNING: chan_sip.c: Codec mismatch on channel SIP/carrier-00000001: ULAW vs ALAW
[2025-02-14 14:32:42] WARNING: chan_sip.c: RTP timeout on channel SIP/carrier-00000001

The fix is to force transcoding in sip-vicidial.conf:

[carrier-trunk]
type=peer
host=203.0.113.50
port=5060
disallow=all
allow=ulaw,alaw
directmedia=no
rtpkeepalive=20

Set directmedia=no to force audio through the Asterisk server (transcoding). This adds CPU load but prevents one-way audio when codecs differ. If the carrier supports only uLaw but the agent phone is aLaw, Asterisk will transcode in the middle.

Restart the SIP channel:

asterisk -rx "sip reload"
asterisk -rx "sip show peers" | grep carrier-trunk

Resolving RTP timeouts from firewall rules

RTP audio uses UDP ports 10000-20000 by default. If your firewall rule for those ports drifts (perhaps after a cloud migration or security patch), calls will drop after 30-45 seconds.

Check iptables:

sudo iptables -L -n | grep -E "10000|20000|RTP"
sudo iptables -A INPUT -p udp --dport 10000:20000 -s 203.0.113.0/24 -j ACCEPT
sudo iptables -A OUTPUT -p udp --dport 10000:20000 -d 203.0.113.0/24 -j ACCEPT

Check if RTP keepalive is set in your SIP trunks:

[carrier-trunk]
rtpkeepalive=20

This sends an RTP keepalive packet every 20 seconds to prevent the firewall from timing out the session.

Test RTP connectivity to the carrier:

mtr -u -b 203.0.113.50 -p 10000

Auto-fixing call state corruption

Call state corruption happens when an agent's call is marked DONE in the vicidial_log but the agent is still in the queue. This freezes the agent until a manual intervention or log rotation.

To detect this, query the log table:

SELECT uniqueid, agent_id, status, call_date FROM vicidial_log
WHERE status IN ('QUEUE', 'IN-CALL', 'RING')
AND call_date < DATE_SUB(NOW(), INTERVAL 5 MINUTE)
ORDER BY call_date DESC;

Any call marked QUEUE or IN-CALL that is older than 5 minutes is stale. Fix it:

UPDATE vicidial_log
SET status='DONE', end_time=NOW()
WHERE uniqueid='1707916330.1' AND status != 'DONE';

For large-scale cleanup, use this AGI approach in the vicidial_agent_inbound context:

exten => _X.,1,NoOp(Check for stale calls)
same => n,Set(STALE_CALL=${SHELL(/usr/local/bin/check_stale_calls.sh ${EXTEN})})
same => n,GotoIf($["${STALE_CALL}" != ""]?cleanup:continue)
same => n(cleanup),Set(status=DONE)
same => n,NoOp(Cleaned stale call: ${STALE_CALL})
same => n(continue),Return()

Querying the incident log for patterns

Once Claude has logged incidents, you can spot patterns:

SELECT event_type, COUNT(*) as count, DATE_FORMAT(event_timestamp, '%Y-%m-%d %H:%i') as period
FROM vicidial_incident_log
WHERE event_timestamp > DATE_SUB(NOW(), INTERVAL 24 HOUR)
GROUP BY event_type, DATE_FORMAT(event_timestamp, '%Y-%m-%d %H:%i')
ORDER BY period DESC, count DESC;

If CODEC_MISMATCH spikes every morning at 08:00, that is when a batch job resets the SIP trunk to default settings. Check your cron jobs:

crontab -l | grep -E "asterisk|sip"

Troubleshooting

Claude API returns "Invalid API key" error.

Check that CLAUDE_API_KEY is exported to the AGI process. Edit /etc/asterisk/asterisk.conf and add:

[files]
astdatadir => /var/lib/asterisk
astvardir => /var/lib/asterisk
astdbdir => /var/lib/asterisk
astkeydir => /var/lib/asterisk
astdictdir => /var/lib/asterisk
astmoddir => /usr/lib64/asterisk/modules
astlibdir => /usr/lib64
astagidir => /usr/lib/asterisk/agi-bin
astspooldir => /var/spool/asterisk
astrundir => /var/run/asterisk
astlogdir => /var/log/asterisk
astetcdir => /etc/asterisk

[startup]
; Ensure CLAUDE_API_KEY is passed to AGI child processes

Actually, environment variables do not automatically inherit into AGI processes. Instead, source the key directly in the Python script or store it in a config file:

import configparser
config = configparser.ConfigParser()
config.read("/etc/vicidial/incident.conf")
CLAUDE_API_KEY = config.get("claude", "api_key")

AGI script hangs or times out.

Asterisk AGI calls are synchronous and block the dialplan. If Claude takes longer than 10 seconds to respond, Asterisk will time out. Set a shorter timeout in the script:

response = requests.post(
    "https://api.anthropic.com/v1/messages",
    headers=headers,
    json=payload,
    timeout=5  # Reduced from 10
)

Alternatively, spawn the Claude call asynchronously in a background job:

(claude_call &)  # Run in background; do not block dialplan

MySQL connection drops mid-incident.

The AGI script does not reconnect if the DB connection is lost. Add a reconnect handler:

def get_cursor(self):
    try:
        cursor = self.db.cursor(MySQLdb.cursors.DictCursor)
    except MySQLdb.OperationalError:
        self.connect_db()
        cursor = self.db.cursor(MySQLdb.cursors.DictCursor)
    return cursor

Asterisk log file is too large and rotates mid-event.

The tail_asterisk_log function reads from a live file. If logrotate runs while the AGI is running, it will get only partial logs. Use a named pipe or journalctl instead:

def tail_asterisk_log(self, lines=50):
    try:
        result = os.popen(f"journalctl -u asterisk -n {lines} --no-pager").read()
        return result if result else ""
    except Exception as e:
        self.log_error(f"Journal read failed: {e}")
        return ""

One-way audio persists after directmedia=no is set.

directmedia=no only forces transcoding if Asterisk can reach both peers. If the agent phone is behind a strict NAT, Asterisk may still not be able to transcode. Check the agent's phone settings: it must send its real IP to Asterisk (register with its public IP or use a proxy). Alternatively, set nat=force_rport in the peer definition:

[agent-phone]
type=friend
nat=force_rport

Frequently asked questions

Why does ViciDial mark calls as DEAD?

Calls are marked DEAD in the vicidial_log when the call ends abnormally: disconnect is initiated by the carrier, not the agent. Check if the carrier is hanging up due to timeouts or invalid SIP headers. Run asterisk -rx "sip set debug on" to see CANCEL or BYE frames. If the carrier is sending BYE frames early, verify rtpkeepalive is set and check packet loss on the RTP port range.

Can Claude auto-remediate without breaking things?

Yes, but only for idempotent actions: reloading a SIP trunk, resetting call state in the DB, or forcing a codec. Do not let Claude execute shell commands that affect running calls or delete records without approval. Define a whitelist of safe commands in the script and reject anything else.

What is the cost of running Claude per call?

Each Claude API call costs roughly $0.001 (using claude-3-sonnet). If you trigger Claude on every inbound call, that is $100 per 100,000 calls. Instead, trigger only on detected problems: codec mismatches, timeouts, or unreachable carriers. This reduces the rate to one call per 50-100 incidents, bringing cost down to $0.02-$0.05 per incident.

How do I prevent the AGI from blocking inbound calls?

AGI calls are blocking by default. Run Claude calls in a background job or a separate daemon that listens to the incident log table. Use a Bash job with & or spawn a separate Python process:

import subprocess
subprocess.Popen(["/usr/local/bin/vicidial_incident_agi.py", call_id, event_type], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

What happens if the Claude API is down?

The script has a 10-second timeout and a fallback: if Claude does not respond, log the incident as unanalyzed and continue the call. The on-call engineer can later query the raw incident log and run analysis manually. Add monitoring for Claude API availability:

curl -s -H "x-api-key: ${CLAUDE_API_KEY}" https://api.anthropic.com/v1/messages \
  -X POST -d '{"model":"claude-3-sonnet-20250229","max_tokens":1,"messages":[{"role":"user","content":"ok"}]}' \
  | jq '.error' | grep -q null && echo "Claude OK" || echo "Claude DOWN"

Run this check every 5 minutes via cron and page the oncall engineer if it fails.

Summary

You have built an incident detection system for ViciDial that logs call events to MySQL, queries a curated knowledge base, and invokes Claude to analyze SIP and RTP failures. The key configuration points are:

Created vicidial_incident_kb and vicidial_incident_log tables to store known issues and live incident data.
Wrote a Python AGI script that logs call state and calls Claude with structured prompts on detected problems.
Hooked the AGI into the dialplan with Macro(incident-hook) on inbound call events.
Configured SIP trunks with directmedia=no and rtpkeepalive=20 to prevent codec and RTP timeout issues.
Verified MySQL connectivity and Claude API error handling to prevent blocking the call.

The next steps are to run a week of test calls in production, collect real incident data, and add the most common patterns to vicidial_incident_kb. Monitor Claude response latency and adjust the timeout. Set up a dashboard to visualize incident types and resolution times, then iterate on the KB to improve Claude's accuracy. After two weeks, you should have enough data to hand off routine incidents to Claude-driven auto-remediation with human approval workflows for risky changes.

ViciDial LAGGED agents: root cause analysis and debugging

Moisi Trungu — Sun, 18 Oct 2026 03:30:08 +0000

ViciDial LAGGED agents: root cause analysis and debugging

Agents stuck in LAGGED status show no calls, drop inbound traffic, and fail to reconnect: identify the socket, session, and database state causing the freeze in under five minutes.

LAGGED status means the agent's web session has disconnected from the Asterisk realtime socket or the ViciDial telephony daemon (ViciDial_Agent.pl) has lost the agent's active record. The agent appears logged in but receives no calls. The fix depends on which layer broke: kill the hung PHP session, restart the daemon, force a database update, or rebuild the SIP channel registration. Most LAGGED events trace to session timeout, daemon memory leak, or a stale vicidial_live_agents row with no matching active call context in Asterisk.

Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, MariaDB 10.5+, with Apache 2.4 and PHP 7.4.

Prerequisites

SSH access to the ViciDial server with sudoers privilege or root
Asterisk CLI installed and permission to run asterisk -rx commands
MySQL/MariaDB client and direct database access
Familiarity with asterisk -vvv debugging output and SIP traces
At least one agent account with active LAGGED status to test
ViciDial web admin access at /vicidial/admin.php or higher
Basic understanding of ViciDial agent states: READY, PAUSED, LAGGED, INCALL

What "LAGGED" means in ViciDial

An agent in LAGGED status has lost the real-time bidirectional link between their web browser (the agent screen) and the ViciDial backend processes. The agent's record exists in the vicidial_live_agents table, and their PHP session file still exists on disk, but the websocket or TCP connection that carries call events, caller ID, and disposition codes has timed out or been killed.

When LAGGED, the agent sees no incoming calls in the dialpad, cannot click buttons to accept or reject calls, and the admin reports show no active calls for that extension. Meanwhile, the ViciDial_Agent.pl daemon cannot locate the agent's session state and either ignores new inbound calls or routes them to a fallback destination (voicemail, IVR, or another agent).

The customer experience: dial an advertised number, wait on hold, hear silence, and disconnect when the LAGGED agent's queue times out.

Prerequisites for investigation

Before you start, gather:

The agent's vicidial_user account name and assigned extension
The timestamp when LAGGED was first observed (from web admin or agent report)
The server hostname and any load balancer address
Recent /var/log/asterisk/messages lines around that time
The agent's assigned campaign, dial method, and call list

How to identify which agents are LAGGED

Log into the ViciDial admin web interface at /vicidial/admin.php (or direct to /vicidial/admin/users.php). In the "Active Agents" or "Realtime Agents" panel, any agent showing LAGGED in the "Status" column is disconnected.

Alternatively, query the database directly:

SELECT vicidial_user, status, last_update, calls_today, extension FROM vicidial_live_agents WHERE status = 'LAGGED' ORDER BY last_update DESC;

The last_update timestamp tells you how long they have been stuck. If last_update is more than 5 minutes old, the connection was lost well before the current time.

Also check for orphaned SIP channels on the Asterisk side:

asterisk -rx "sip show channels" | grep -i "SIP/vicidial"

If you see SIP/vicidial-XXXX channels with no matching agent in vicidial_live_agents or with a LAGGED status, that channel is orphaned and likely wasting memory.

Layer 1: PHP session and web connection diagnostics

The first layer to investigate is the agent's web session (the PHP session file on disk and the HTTP/WebSocket connection to the browser).

Why do PHP sessions expire and trigger LAGGED status?

A PHP session is a file stored on disk (usually in /var/lib/php/sessions or /tmp) that holds the agent's login state, assigned call lists, and campaign parameters. ViciDial web scripts check this file on every page load. If the session file is missing or corrupted, or if the browser tab closes without sending a "log out" message, the next refresh will prompt a re-login.

Some agent browsers sit idle. After 10 to 30 minutes with no requests, Apache or PHP may close the connection. When the agent tries to click a button or open a new screen, the request fails because there is no active session. The ViciDial backend then receives no updates and assumes the agent has disconnected.

Check for active PHP sessions for a specific user

SSH to the ViciDial server. List the session files for the target user:

ls -la /var/lib/php/sessions/ | grep -i "$(grep -h vicidial_user /var/www/html/vicidial.php 2>/dev/null || echo '__')"

A more direct approach: query the process list to find the agent's browser session:

ps auxww | grep "[a]gent"

Look for PHP-CGI or Apache worker processes labeled with the agent's user account or session ID.

To dump the session file and check its contents:

cat /var/lib/php/sessions/sess_<SESSION_ID_HERE>

You will see serialized PHP data. Check that vicidial_user, user_id, and extension are present. If the file is empty or contains only a timestamp, the session is dead.

Force an agent to re-login by clearing their session

If you determine a session is corrupted, delete it:

rm -f /var/lib/php/sessions/sess_<SESSION_ID_HERE>

Tell the agent to refresh their browser. They will be asked to log in again. Once logged in, a new session file will be created and the agent should no longer show LAGGED.

Layer 2: ViciDial daemon socket and memory state

The second layer is the ViciDial_Agent.pl daemon, which runs on the server and holds an in-memory copy of every active agent's state: their status (READY, PAUSED, INCALL), assigned phone, SIP extension, and the file descriptor for their socket connection.

Does the ViciDial daemon have the agent in memory?

Connect to the ViciDial admin port (default 6100, configurable in /etc/asterisk/asterisk.conf or vicidial.conf) using telnet or a custom admin tool:

telnet localhost 6100

You will get a ViciDial command prompt. Use the STATUS command to list all active agents:

STATUS

Press Enter. The daemon will print a line for each agent in memory. Look for your target agent's extension or username. If you do not see it, the daemon has no record of that agent, and the agent must re-login to be re-registered.

If the agent appears in the STATUS output, note their assigned handler process ID and socket number. This tells you the daemon thinks they are still active.

Exit telnet by typing QUIT or Ctrl+D.

Check the ViciDial daemon logs

The ViciDial_Agent.pl daemon logs to a dedicated file, usually /var/log/ViciDial_Agent.log or /var/log/asterisk/ViciDial_Agent.log (depends on the installation). Tail the last 100 lines and search for the agent's extension:

tail -100 /var/log/ViciDial_Agent.log | grep -i "extension_XXXX"

Look for socket timeout warnings or "closing connection" messages near the timestamp when the agent went LAGGED.

If you see messages like "Socket timeout on extension XXXX after 300 seconds", the daemon closed the connection because no keep-alive packet arrived from the browser in 5 minutes. This is a sign that the browser is not polling the server or has lost network connectivity.

Restart the ViciDial daemon

If the daemon is consuming excessive memory or has accumulated stale agent records, restart it:

sudo systemctl restart ViciDial_Agent

sudo /etc/init.d/ViciDial_Agent restart

Restarting will drop all agent sessions. All agents logged in will be disconnected and must log back in. Coordinate this restart during a low-traffic period.

To restart without killing agents: edit the daemon config to reduce socket timeout from 300 seconds to 30 seconds, then tell the daemon to reload:

sudo kill -HUP $(pidof ViciDial_Agent.pl)

This tells the daemon to reread config. However, this does not drop existing agent connections.

Layer 3: ViciDial_live_agents database state

The third layer is the vicidial_live_agents table, which is the source of truth for which agents are currently logged in. Each row represents one active agent session. When an agent logs in, a row is inserted. When they log out, the row is deleted. If the row exists but the corresponding daemon socket and PHP session are gone, the agent is LAGGED.

Inspect the agent row in vicidial_live_agents

Query the table for the agent in question:

SELECT 
  vicidial_user,
  user_id,
  extension,
  status,
  last_update,
  lead_id,
  calls_today,
  session_id,
  server_ip
FROM vicidial_live_agents
WHERE vicidial_user = 'AGENT_USERNAME_HERE'
LIMIT 1;

Note the values:

status: should be READY, PAUSED, or INCALL if the agent is healthy. LAGGED means the web connection is stale.
last_update: should be within the last 10 seconds. If it is more than 5 minutes old, the agent's web browser has not refreshed in that time.
session_id: should match the PHP session ID currently stored in /var/lib/php/sessions/sess_.
server_ip: should be the IP of the ViciDial server or the agent's client IP if behind a load balancer.

Force-update the agent to PAUSED to clear LAGGED

If the database row is stale but the agent is still in the browser, you can force the status to PAUSED without deleting the row:

UPDATE vicidial_live_agents
SET status = 'PAUSED', last_update = NOW()
WHERE vicidial_user = 'AGENT_USERNAME_HERE';

The agent's browser should detect the change and refresh their dialpad. If they remain LAGGED after this, the browser has no active connection to the server.

Delete the agent row to force re-login

If the agent is truly stuck, delete their row from vicidial_live_agents:

DELETE FROM vicidial_live_agents
WHERE vicidial_user = 'AGENT_USERNAME_HERE';

The agent will be logged out immediately. Have them log back in. A new row will be inserted, a new PHP session created, and the daemon will re-register them.

Be careful: if the agent is currently on a call, deleting their row will orphan the call and may charge you for a call that never completed properly.

Layer 4: Asterisk SIP channel and state

The fourth layer is Asterisk itself and the SIP channels registered for the agent's extension.

List SIP channels for an agent extension

Ask for all active SIP channels:

asterisk -rx "sip show channels"

Output will resemble:

Peer             User/ANR    Call ID              Duration Recv-Q Lost %   Jitter FromTag
SIP/1001-00000ab agent1      sip:1001@10.0.0.1    00:02:35    0    0    0.0   12345abc
SIP/1002-00000ac agent2      sip:1002@10.0.0.1    00:00:15    0    0    0.0   67890def

Look for the agent's extension (e.g., 1001). If you see one channel, they are logged in and Asterisk knows about them. If you see zero or multiple stale channels, that is a clue.

Check the SIP peer registration for the agent

asterisk -rx "sip show peers" | grep -i "1001"

If the peer is listed as "OK" with a recent "Refresh" timestamp, the registration is active. If it shows "UNREACHABLE" or has not refreshed in more than 5 minutes, the phone or endpoint is not responding to SIP REGISTER packets.

Dump a SIP channel state

If you need to see detailed state for a specific channel:

asterisk -rx "sip show channel SIP/1001-00000ab"

This will print codec negotiation, RTP ports, and any pending transactions.

Force Asterisk to reload SIP config

If you edited sip-vicidial.conf or extensions-vicidial.conf, reload without restarting:

asterisk -rx "sip reload"
asterisk -rx "dialplan reload"

Reloading will drop any active SIP channels. Agents will need to re-register.

Layer 5: Real-time state via Asterisk AMI and ViciDial logs

The fifth layer is the Asterisk Manager Interface (AMI), which is the real-time API that ViciDial uses to monitor call state. If AMI is not connected or is lagging, the ViciDial daemon cannot see call events and will mark agents as LAGGED incorrectly.

Check if AMI is running and connected

asterisk -rx "manager list connected"

You should see lines listing connected clients. Look for "vicidial" or "ViciDial" in the client list. If there are zero connections or the connection is very old, the daemon is not talking to Asterisk.

Restart the AMI connection by restarting the daemon:

sudo systemctl restart ViciDial_Agent

Tail the Asterisk full log for AMI errors

tail -50 /var/log/asterisk/full | grep -i "ami\|manager"

Look for connection refused, socket timeout, or authentication error messages. If AMI failed to authenticate, check the manager.conf file:

cat /etc/asterisk/manager.conf | grep -A5 "^\\[vicidial\\]"

Ensure the username and password match what is configured in the ViciDial daemon (usually in /etc/asterisk/vicidial.conf or hard-coded in ViciDial_Agent.pl). If they do not match, the daemon cannot connect and all agents will eventually go LAGGED.

Troubleshooting

Symptom: Agent goes LAGGED within 2 minutes of login

Check the Apache access and error logs:

tail -50 /var/log/apache2/access.log | grep -i "vicidial.php"
tail -50 /var/log/apache2/error.log

If you see 502 Bad Gateway or 503 Service Unavailable, the PHP backend is crashing or overloaded. Restart Apache:

sudo systemctl restart apache2

If PHP crashes again immediately, increase the memory limit in /etc/php/7.4/cgi/php.ini:

memory_limit = 512M

Restart Apache and test again.

Symptom: All agents are LAGGED after a reboot

The ViciDial daemon may not have started. Check if it is running:

ps auxww | grep ViciDial_Agent

If not running, start it manually:

sudo systemctl start ViciDial_Agent

If it fails to start, check the daemon log for startup errors:

tail -20 /var/log/ViciDial_Agent.log

Common startup errors: MySQL connection refused (check if MySQL is running), missing Perl modules (run CPAN or apt install), or stale PID file. Delete a stale PID and retry:

sudo rm -f /var/run/ViciDial_Agent.pid
sudo systemctl start ViciDial_Agent

Symptom: Agents are LAGGED but PhoneType is listed as "NONE"

Check the vicidial_users table. The user may not have a phone device assigned:

SELECT vicidial_user, phone_type, phone_number FROM vicidial_users WHERE vicidial_user = 'AGENT_USERNAME_HERE';

If phone_type is "NONE" or NULL, the agent cannot register a SIP extension. Edit the user record in the web admin and assign a phone type (typically "SIP" for soft phones). Reload SIP and have the agent re-login.

Symptom: LAGGED agents reappear when the web server restarts

Check if /var/lib/php/sessions has sufficient disk space:

df -h /var/lib/php/sessions

If free space is less than 5%, sessions may be getting garbage-collected. Clean up old session files:

find /var/lib/php/sessions -type f -mtime +7 -delete

Then expand the session storage device if possible. Also increase PHP session.gc_maxlifetime in php.ini to match ViciDial's timeout (usually 14400 seconds, 4 hours):

session.gc_maxlifetime = 14400

Symptom: One or two agents are LAGGED, but most others are fine

This is usually a network issue on the agent's side. Have the agent:

Reload the browser tab (Ctrl+F5 or Cmd+Shift+R to bypass cache).
Check their local IP and firewall. Ping the server.
Try a different browser or device if possible.
Check if they are on a corporate proxy or VPN that is blocking websockets or long-polling.

If the issue persists only for that agent, restart their session in the database as described in "Force an agent to re-login by clearing their session" above.

Frequently asked questions

Why does an agent see READY status in the dialpad but show LAGGED in the admin console?

The agent's PHP session is active, so their browser shows READY because it has a cached version of the status. However, the last_update timestamp in vicidial_live_agents is stale, so the admin console shows LAGGED. The web connection is asymmetrical: the browser is still making requests (keeping PHP alive), but the server is not receiving websocket push events or the polling frequency has dropped so low that real-time synchronization is lost. Force a refresh of the agent screen (Ctrl+F5) to resync. If the issue persists, delete the database row and have the agent re-login.

Can I write a cron job to auto-clear LAGGED agents?

Yes, but do so cautiously to avoid interrupting real calls. Create a cron script that runs every 5 minutes and updates any agent whose last_update is older than 5 minutes to PAUSED, then check again after 10 minutes to see if they reconnect. If they remain PAUSED after 15 minutes, delete the row. Example shell loop (place in /usr/local/bin/clear_lagged.sh):

#!/bin/bash
mysql -e "UPDATE vicidial_live_agents SET status = 'PAUSED' WHERE status = 'LAGGED' AND last_update < DATE_SUB(NOW(), INTERVAL 5 MINUTE);"
sleep 600
mysql -e "DELETE FROM vicidial_live_agents WHERE status = 'PAUSED' AND last_update < DATE_SUB(NOW(), INTERVAL 15 MINUTE) AND calls_today = 0;"

Schedule it with crontab -e:

*/10 * * * * /usr/local/bin/clear_lagged.sh

This runs every 10 minutes. Adjust the 5 and 15 minute thresholds to suit your environment. Always test in a staging environment first.

Is LAGGED the same as a dropped call or a no-answer?

No. LAGGED is an agent state, not a call result. A call can have a LAGGED agent and still be transferred, dropped, or forwarded depending on the dial plan. A LAGGED agent cannot answer new calls or adjust their status, but calls already connected to that extension will remain connected until one party hangs up. However, new inbound calls will not ring that agent, so they will not see them in their dialpad and cannot click to accept them.

What causes LAGGED if the agent is sitting idle in the dialpad?

Agents who do not interact with the web page for more than 10 minutes typically lose their polling connection. The browser is still running, but no new HTTP requests are being sent, so the server's keep-alive timer expires and the backend marks them LAGGED. To prevent this, configure a JavaScript keep-alive in the dialpad (ViciDial usually includes this), or reduce the polling timeout in the browser settings. Some deployments use websockets instead of polling to avoid this, but websockets require additional server-side support and can consume more memory per agent.

If I delete a LAGGED agent row from the database, what happens to their active call?

If the agent is not actually on an active call, nothing happens. If they are on a call (lead_id is not null), the call remains connected at the Asterisk level, but the agent's record is gone. The call will not be logged properly in vicidial_log because the agent record is not there to supply user_id and other context. The call will show up as orphaned or with a null user_id. Always check the lead_id field before deleting:

SELECT vicidial_user, lead_id, status FROM vicidial_live_agents WHERE vicidial_user = 'AGENT_USERNAME_HERE';

If lead_id is a positive number, the agent is on a call. In that case, do not delete the row. Instead, try the force-update to PAUSED method first.

Summary

A LAGGED agent is disconnected at the web session, daemon socket, or database level, despite an apparently logged-in record. Investigation starts with the PHP session file and browser connectivity, then moves to the ViciDial daemon's in-memory state and the vicidial_live_agents table, and finally checks Asterisk SIP registration and AMI connectivity.

Standard fixes are:

Delete the agent's PHP session file and ask them to re-login.
Restart the ViciDial daemon if it is consuming excessive memory or has lost AMI connection.
Update or delete the stale row in vicidial_live_agents.
Reload SIP configuration if the agent extension is not registering properly.

To avoid LAGGED in the future: reduce session timeout in PHP and the daemon to match your environment, use keep-alive pings in the dialpad, monitor vicidial_live_agents for stale rows, and ensure the daemon can always reach Asterisk AMI without blocking. Regular log review of /var/log/ViciDial_Agent.log and /var/log/asterisk/messages will catch most issues before they affect more than one agent.

Homer SIP Capture & RTCP Monitoring for Asterisk/ViciDial

Moisi Trungu — Sat, 17 Oct 2026 03:30:03 +0000

Homer SIP Capture & RTCP Monitoring for Asterisk/ViciDial

Capture every SIP packet and RTP stream from Asterisk: detect call quality failures, codec mismatches, and dropped audio before agents report silent calls.

Homer HEP (Homer Encapsulation Protocol) captures all SIP signaling and RTP statistics from your Asterisk/ViciDial system and streams them to a centralized Homer UI for real-time and historical analysis. Silent calls, one-way audio, and jitter-induced dropouts become visible in packet detail within seconds of occurrence, eliminating the guesswork of call tracing through Asterisk logs alone. This tutorial walks through installing Homer Server, configuring Asterisk HEP forwarding, tuning RTCP metrics, and querying call data to pinpoint carrier failures, codec negotiation errors, and network problems in your ViciDial infrastructure.

Tested on ViciDial 2.14 (SVN 3555+), Asterisk 16/18, Homer 7.8+, MariaDB 10.5+, CentOS 7/8 and Ubuntu 18.04 LTS.

Prerequisites

A working Asterisk/ViciDial installation with outbound calling enabled.
Root or sudo access on both the ViciDial server and a dedicated Homer server (or co-located).
Network connectivity between Asterisk and Homer (UDP port 6060 and TCP 9060 typical).
MariaDB or MySQL 10.5+ with at least 50 GB free for call history (HEP packets consume 200-500 KB per call).
Familiarity with Asterisk CLI commands and basic ViciDial diagnostics.
Optional but recommended: a second NIC or VLAN on the ViciDial box to isolate HEP traffic.

What is Homer and why ViciDial needs it

Homer is a SIP capture and analytics platform that records every SIP message (INVITE, 100 Trying, 200 OK, BYE, etc.) and RTP stream statistics in real time. When an agent reports a silent call or a customer hears only their own voice, Homer's packet graph and RTP metrics show you whether the problem was on your network, your carrier, or your codec selection.

ViciDial's built-in call logging and Asterisk CLI are good for counting calls and marking outcomes, but they do not capture packet-level detail. A call can complete successfully in the Asterisk database yet still have one-way audio because of a firewall rule, a carrier SDP mismatch, or an RTP timeout. Homer fills that gap.

Architecture overview

Your Asterisk server runs a small HEP forwarder (either the Asterisk chan_sip or chan_pjsip module, or a dedicated sniffer like hepic). This forwarder sends a copy of every SIP packet and RTP packet to Homer's HEP collector (default port 6060 UDP). Homer decodes the packets, stores them in a database, and serves a web UI where you search by call ID, phone number, time range, and result code.

Asterisk (Outbound Call)
    |
    +---> HEP Forwarder (SIP module + hepic or VQ modules)
             |
             +---> Homer HEP Collector (UDP 6060)
                       |
                       +---> Homer DB (MySQL/MariaDB)
                       |
                       +---> Homer UI (HTTP 9060+)

For a ViciDial cluster, you can forward from multiple Asterisk boxes to a single Homer server, then correlate calls across the cluster.

Part 1: Install Homer Server

Homer runs on a dedicated Linux box or a non-Asterisk server in your network. We assume CentOS 7 or Ubuntu 18.04 LTS.

Install dependencies

CentOS 7:

yum install -y epel-release
yum install -y MariaDB-server MariaDB-client git golang wget curl
systemctl start mariadb
systemctl enable mariadb

Ubuntu 18.04 LTS:

apt-get update
apt-get install -y mariadb-server mariadb-client git golang-go wget curl
systemctl start mariadb
systemctl enable mariadb

Create Homer database and user

mysql -u root -p << 'EOF'
CREATE DATABASE homer_data DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE DATABASE homer_config DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'homer'@'localhost' IDENTIFIED BY 'homer_password_123';
GRANT ALL PRIVILEGES ON homer_data.* TO 'homer'@'localhost';
GRANT ALL PRIVILEGES ON homer_config.* TO 'homer'@'localhost';
FLUSH PRIVILEGES;
EXIT;
EOF

Replace homer_password_123 with a strong password and save it.

Clone Homer from GitHub and build

cd /opt
git clone https://github.com/sipcapture/homer.git
cd homer
git checkout 7.8  # or latest stable tag

Read the README. For a quick test, use Docker (optional). For production on bare metal, follow the build instructions in the repo. The Go-based binaries are statically linked and need only a standard Linux kernel.

Build the Homer API and capture server:

cd /opt/homer
make

If make is not available, download pre-built binaries:

wget https://github.com/sipcapture/homer/releases/download/v7.8/homer-7.8-linux-amd64.tar.gz
tar xzf homer-7.8-linux-amd64.tar.gz -C /opt/homer/

Initialize Homer database schema

cd /opt/homer
mysql -u homer -p homer_data < sql/homer_data.sql
mysql -u homer -p homer_config < sql/homer_config.sql

Create Homer configuration file

cat > /etc/homer/homer.conf << 'EOF'
{
  "hep_listen": "0.0.0.0:6060",
  "http_listen": "0.0.0.0:9060",
  "db_driver": "mysql",
  "db_dsn": "homer:homer_password_123@tcp(localhost:3306)/homer_data",
  "db_config": "homer:homer_password_123@tcp(localhost:3306)/homer_config",
  "loki_host": "",
  "redis_host": "",
  "debug": false,
  "cors": true,
  "auth_enabled": false,
  "max_packet_size": 65535,
  "table_rotation": "day"
}
EOF
mkdir -p /etc/homer
chmod 600 /etc/homer/homer.conf

Set auth_enabled to true and configure LDAP or local users in production.

Start Homer services

cd /opt/homer
./homer-api -config /etc/homer/homer.conf &
./homer-capture -config /etc/homer/homer.conf &

Or use systemd:

cat > /etc/systemd/system/homer-api.service << 'EOF'
[Unit]
Description=Homer API
After=network.target mariadb.service

[Service]
Type=simple
User=homer
ExecStart=/opt/homer/homer-api -config /etc/homer/homer.conf
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

cat > /etc/systemd/system/homer-capture.service << 'EOF'
[Unit]
Description=Homer HEP Capture
After=network.target mariadb.service

[Service]
Type=simple
User=homer
ExecStart=/opt/homer/homer-capture -config /etc/homer/homer.conf
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

useradd -r homer 2>/dev/null || true
chown -R homer:homer /opt/homer /etc/homer
systemctl daemon-reload
systemctl start homer-api homer-capture
systemctl enable homer-api homer-capture

Test that Homer is listening:

netstat -tlnp | grep -E '(6060|9060)'
curl http://localhost:9060

You should see the Homer UI homepage.

Part 2: Configure Asterisk to forward HEP

Asterisk can forward SIP and RTP packets to Homer using either the native HEP module (for PJSIP) or a sniffer daemon. We cover both methods.

Method A: Using chan_pjsip with HEP module (recommended for Asterisk 16+)

First, check if your Asterisk has the HEP module:

asterisk -rx "module show like hep"

If res_hep.so is listed and loaded, proceed. If not, rebuild Asterisk with --enable-hep or load it manually:

asterisk -rx "module load res_hep.so"
asterisk -rx "module load res_hep_pjsip.so"

Now configure HEP in /etc/asterisk/hep.conf:

[hep]
enabled = yes
host = 192.168.1.100
port = 6060
protocol = udp
capture_id = 1
uuid_type = call-id

Replace 192.168.1.100 with your Homer server's IP. The capture_id is used to tag packets from this Asterisk instance (useful when monitoring multiple Asterisk boxes).

Create /etc/asterisk/hep_pjsip.conf:

[general]
enabled = yes
send_via_ip = yes

Reload Asterisk modules:

asterisk -rx "module reload res_hep.so"
asterisk -rx "module reload res_hep_pjsip.so"

Verify:

asterisk -rx "hep show status"

Method B: Using sip.conf with chan_sip (legacy, Asterisk 13-18)

If you are still using chan_sip, enable SIP capture:

[general]
; In /etc/asterisk/sip.conf or sip-vicidial.conf
sipdebug = yes
; or use the CLI:
; asterisk -rx "sip set debug"

Then run a sniffer on the Asterisk box to forward captured packets to Homer. Use hepic (Kamailio HEP Forwarder):

# Install hepic
wget https://github.com/sippy/hepic/releases/download/v1.0/hepic-linux-amd64
chmod +x hepic-linux-amd64
mv hepic-linux-amd64 /usr/local/bin/hepic

# Run it
/usr/local/bin/hepic -listen 0.0.0.0:5061 -connect 192.168.1.100:6060 &

Then in sip.conf, set:

[general]
sipdebug = yes

And run tcpdump to forward to hepic:

tcpdump -i eth0 -nn 'udp port 5060 or udp port 16384:32767' \
  -w - | hepic -listen stdin -connect 192.168.1.100:6060

This is cumbersome. Method A (PJSIP + HEP module) is preferred.

Configure RTCP to send to Homer

RTP streams themselves are not captured by HEP, but RTCP reports (which contain jitter, packet loss, and latency stats) are. Ensure RTCP is enabled in pjsip.conf:

[transport-udp]
type = transport
protocol = udp
bind = 0.0.0.0:5060
rtcp_mux = no

Setting rtcp_mux = no forces RTCP to use a separate port (RTP+1) instead of multiplexing. This makes packet capture cleaner for testing but uses slightly more bandwidth.

Verify RTCP is flowing from the carrier side. In Homer's UI, after a call completes, search for the call ID and look for RTCP packets with types 200 (SR, Sender Report) or 201 (RR, Receiver Report).

Part 3: Configure ViciDial for HEP

ViciDial itself does not configure HEP. Instead, Asterisk (which ViciDial controls) does. However, you need to ensure ViciDial's outbound and inbound dialing use the correct SIP profiles that have HEP enabled.

Check your ViciDial carrier configuration

In the web admin:

/vicidial/admin.php
Admin > Carrier > Edit

Verify that your carriers point to a valid Asterisk SIP context and use a pjsip endpoint. Look at /etc/asterisk/pjsip.conf:

[carriers]
type = aor
max_contacts = 1

[carrier-1](carriers)
contact = sip:gateway.provider.com:5060

[endpoint-carrier-1]
type = endpoint
aor = carrier-1
outbound_proxy = sip:gateway.provider.com:5060
allow = ulaw,alaw,gsm
dtmf_mode = rfc4733
from_user = your_did
from_domain = gateway.provider.com

The HEP module will capture all SIP traffic on this endpoint automatically. No additional config in ViciDial is needed.

Verify ViciDial logs do not interfere

Asterisk verbose logging is resource-intensive. Disable high verbosity in Asterisk and rely on Homer for detailed call tracing:

asterisk -rx "core set verbose 0"
asterisk -rx "core set debug 0"

Instead, in /etc/asterisk/logger.conf, log only errors and warnings:

[logfiles]
messages => notice,warning,error

This reduces disk I/O and keeps Asterisk fast while Homer captures everything.

Part 4: Query calls in Homer UI

Open your browser to http://homer-server:9060.

Search by Call-ID

Most VoIP calls carry a unique Call-ID in the SIP INVITE. To find a call:

Click "Search" or "Calls".
Select time range (e.g., "Last 1 hour").
Enter the caller number, callee number, or Call-ID.
Click "Search".

Homer displays a list of calls. Click a call to see the SIP flow (INVITE, 100, 180, 200, ACK, BYE, etc.) and RTP statistics.

Interpret RTP and RTCP metrics

When you drill into a call, Homer shows:

Source / Dest IP:Port: Where the RTP stream originated and terminated.
Codec: Audio codec negotiated (PCMU, PCMA, GSM, Opus, etc.).
Jitter: Variance in packet arrival time (milliseconds). More than 50 ms is bad.
Packet Loss: Percentage of RTP packets that did not arrive. Anything above 2% causes audible degradation.
Round Trip Time (RTT): Latency from Asterisk to the peer and back. Over 150 ms causes echo.
MOS (Mean Opinion Score): Estimated audio quality (1-5 scale). 4+ is acceptable for customers.

Example: A call shows codec PCMU, jitter 120 ms, 8% packet loss, MOS 2.1. This indicates a poor network path to the carrier gateway. Contact your ISP or carrier.

Export call data for analysis

Click "Export" on the search results to download a CSV of calls. Use this to correlate one-way audio incidents with specific time windows, carriers, or agent IDs.

Part 5: Troubleshoot common HEP issues

Homer receives no packets

Check that UDP 6060 is open:

# On Homer server
netstat -tlnpu | grep 6060
# Should show homer-capture listening on 0.0.0.0:6060

# From Asterisk box, test connectivity
nc -u -v 192.168.1.100 6060 < /dev/null
# Should not timeout

If timeout, check firewall:

# On Homer server
iptables -L -n | grep 6060
# Add rule if missing:
iptables -A INPUT -p udp --dport 6060 -j ACCEPT

Check Asterisk HEP module status:

asterisk -rx "hep show status"
# Should show "HEP Enabled: Yes"

If HEP is disabled, reload:

asterisk -rx "module reload res_hep"

HEP works but no SIP packets appear

Verify the capture ID in hep.conf matches what Homer expects. In Homer logs (/var/log/homer/homer-capture.log), you should see packet counts:

tail -f /var/log/homer/homer-capture.log | grep -i "packet"

If no packets logged, Asterisk may not be sending them. Check that Asterisk is actually handling calls:

asterisk -rx "sip show calls"
# or for PJSIP:
asterisk -rx "pjsip list channels"

If calls are running but HEP is silent, the module may have failed to start. Restart Asterisk:

systemctl restart asterisk
systemisk status asterisk

One-way audio appears in Homer but Asterisk logs show OK

This is Homer's strength. The call succeeded in the Asterisk database (ViciDial marks it CONNECTED) but Homer's RTP metrics reveal the problem:

RTP from A to B is present, B to A is missing: Check firewall rules on the B side (carrier gateway or far-end PBX). They may be blocking inbound RTP from your Asterisk IP.
Both directions present but high jitter/loss on one direction: The carrier's return path is congested. Ask them to check routing.
No RTCP reports from the far end: The carrier is not sending RTCP, or it is blocked. Some cheap VoIP providers disable RTCP. This means you have no quality visibility.

Storage and retention

Homer logs to MariaDB. By default, it keeps all packets. This grows large quickly. Set a retention policy in homer.conf:

"table_rotation": "day",
"table_retention_days": 7

This rotates tables daily and purges data older than 7 days. Adjust as needed. On a busy Asterisk handling 100+ concurrent calls, 7 days consumes 20-50 GB.

For long-term storage, export older calls to a data warehouse or archive storage before purging.

Part 6: Integrate Homer with ViciDial dashboards

ViciDial does not have a built-in Homer integration, but you can create a custom link in the agent screen or admin interface to jump to Homer for a given call.

Add Homer link to vicidial.php

Edit /var/www/html/agc/vicidial.php (or your custom agent screen template). Near where call details are displayed:

<?php
// After displaying call details, add:
$call_id = preg_replace('/[^a-zA-Z0-9@.-]/','', $_REQUEST['call_id']); // sanitize
$homer_url = "http://192.168.1.100:9060/?query=" . urlencode($call_id);
echo "<a href='$homer_url' target='_blank' class='btn btn-info'>View RTP in Homer</a>";
?>

This assumes ViciDial stores the call ID in the session or request. Adjust the variable name based on your ViciDial version.

Add Homer link to vicidial_log

In /vicidial/admin.php, edit the call log display to include a Homer link:

<?php
// In call log listing:
$call_id = $row['vicidial_id']; // or whatever field holds the call ID
$homer_link = "<a href='http://192.168.1.100:9060/?query=" . htmlspecialchars($call_id) . "' target='_blank'>View in Homer</a>";
echo "<td>$homer_link</td>";
?>

Troubleshooting

Asterisk crashes after enabling HEP

If Asterisk becomes unstable after loading res_hep, the module may have a bug in your Asterisk version. Try disabling HEP and use the sniffer method (hepic) instead:

asterisk -rx "module unload res_hep"
systemctl restart asterisk

Then run hepic separately as shown in Method B.

Homer UI loads but shows "No data available"

Check that the database connection is valid. Test manually:

mysql -u homer -p homer_data -e "SELECT COUNT(*) FROM hep_proto_1;"

If this fails, the schema was not imported. Re-run the SQL import:

mysql -u homer -p homer_data < sql/homer_data.sql

Also verify that packets are being inserted. Check homer-capture logs:

systemctl status homer-capture

SIP REGISTER packets missing

Some Asterisk configurations do not send REGISTER through the HEP module. This is normal for outbound-only systems. If you need to capture registrations (common for SIP trunks), enable SIP debugging specifically:

asterisk -rx "pjsip set logger on"

But be warned: verbose SIP logging impacts performance on busy systems.

Frequently asked questions

Can I monitor calls across a multi-server ViciDial cluster with Homer?

Yes. Configure each Asterisk server to forward HEP to the same Homer instance (set the host in hep.conf to the shared Homer server). In Homer, search by time range and carrier to correlate calls across the cluster. Each Asterisk server sends a unique capture_id in its HEP packets, so you can filter by server.

Does Homer capture payload inside SIP messages (e.g., Authorization headers)?

By default, Homer captures full SIP headers including credentials. This is a security risk if calls are being audited by untrusted parties. To mask sensitive data, either disable auth header capture in Homer's config or configure Asterisk to strip headers before HEP forwarding. Consult Homer's privacy settings and your compliance requirements.

What is the minimum network bandwidth needed for HEP?

A single call with full SIP + RTP + RTCP capture consumes roughly 200-500 KB depending on call duration and RTCP frequency. A 100-call system forwarding to Homer uses 20-50 Mbps of WAN bandwidth. Most carriers throttle SIP signaling to 1-5 Mbps, so HEP overhead is minimal. However, if your Asterisk and Homer servers are on the same LAN with gigabit links, bandwidth is not a concern.

Can I run Homer on the same server as Asterisk/ViciDial?

Yes, but not recommended. Homer's MariaDB will consume significant resources during call playback and searches. A dedicated Homer server keeps ViciDial and Asterisk responsive. If you must co-locate, use SSD storage, allocate at least 8 CPU cores to the VM, and set MariaDB's innodb_buffer_pool_size to 50% of available RAM.

Why does Homer show a call with one-way audio but ViciDial marked it as CONNECTED?

ViciDial's CONNECTED status means Asterisk answered the far end (received a 200 OK), not that audio flowed both ways. One-way audio happens after call setup due to network issues (firewall, NAT, codec mismatch, or carrier routing). Homer's RTP analysis reveals the problem. Use this information to file a support ticket with your carrier or review your firewall rules.

Summary

You have installed Homer on a dedicated server, configured Asterisk to forward SIP and RTP packets via HEP, and learned to search Homer's UI for call detail and quality metrics. When a call fails in ViciDial, Homer's packet trace and RTCP statistics show you whether the problem was signaling (wrong SDP, authentication failure), RTP (no packets, high jitter), or network (firewall, latency). From here, export call data to correlate failures with time, carrier, or agent; set up retention policies to manage database growth; and integrate Homer links into your ViciDial admin screens for one-click access. For ongoing monitoring, periodically review Homer's MOS scores and packet loss metrics to detect carrier degradation before customer complaints arrive.

Integrating AI Voice Agents with VICIdial Remote Agent

Moisi Trungu — Fri, 16 Oct 2026 03:30:02 +0000

Integrating AI Voice Agents with VICIdial Remote Agent

Remote agents hear AI responses cut off mid-sentence, dropped calls fail to transfer to voice bots, and call recordings show no agent audio: fix agent codec mismatch, RTP stream misconfiguration, and VICIDIAL_AGENT permission gaps.

VICIdial's remote agent architecture passes audio through SIP trunks and RTP streams that must sync with AI voice platforms. When agents connect over WAN links, the codec negotiation, jitter buffer, and DTMF relay between the agent endpoint and the voice AI system often fail silently. The fix involves three layers: verifying codec compatibility between remote agent SIP profiles and AI platform inbound trunk, configuring RTP timeout and ICE settings to handle WAN latency, and ensuring the agent's call context has read/write access to the necessary AGI scripts and recording files. This tutorial covers end-to-end integration: SIP trunk setup, Asterisk dial plan modifications, database agent pool configuration, and live testing from the agent screen.

Tested on VICIdial 2.14 (SVN 3555+), Asterisk 16/18, MariaDB 10.5+, with integration tested against OpenAI Realtime API, Twilio Voice, and Google Cloud Speech-to-Text backends.

Prerequisites

VICIdial installation with Asterisk 16 or 18 (kernel 4.19+)
MariaDB root or vicidial user with GRANT privileges
Root or sudo access to the VICIdial server
AI voice platform API credentials and inbound SIP trunk provisioned
Remote agent machine with soft-phone or SIP device capable of G.711 and G.729 codec support
Network connectivity test: ping, mtr, and iperf3 between agent and VICIdial server (latency under 150ms ideal)
Asterisk CLI access via /usr/bin/asterisk -rx

How remote agent calls reach the AI voice system

In a standard VICIdial setup, an agent logs into the web screen, dials an outbound lead, or receives an inbound call. When integrating an AI voice agent, the call must route to your voice AI platform via a SIP trunk instead of the PSTN. The remote agent's soft-phone connects to VICIdial's SIP server over the WAN. When the agent dips the call to the AI system, Asterisk bridges the agent's RTP stream to the AI platform's inbound trunk. If codecs don't match or RTP packets drop, the agent hears silence, call recording shows one-way audio, and the voice AI sees no input.

The integration point is the AGI (Asterisk Gateway Interface) script that handles outbound call routing. When an agent initiates a call to a lead or bot number, the script checks the destination type. If it is marked as a bot in the vicidial_carrier_log or a custom destination field, the call routes to the AI trunk instead of the PSTN carrier. Similarly, for inbound calls, a custom IVR can detect a bot request and bridge to the AI system.

Setting up the AI voice platform SIP trunk

Most AI voice platforms (OpenAI, Twilio, Google) publish SIP endpoint details and authentication. Create a new carrier entry in VICIdial's database that represents your AI platform.

Log into the VICIdial admin screen and navigate to "Carriers" (or directly edit the database):

INSERT INTO vicidial_carrier_log 
(carrier_name, carrier_host, carrier_port, carrier_protocol, carrier_user, carrier_pass, 
active, inbound_cid, description) 
VALUES 
('AIVoicePlatform', 'sip.openai.example.com', 5060, 'SIP', 'vicidial_user', 'secure_password_here', 'Y', 'N', 'OpenAI Realtime API trunk');

Verify the entry:

SELECT carrier_id, carrier_name, carrier_host FROM vicidial_carrier_log WHERE carrier_name = 'AIVoicePlatform';

Configuring Asterisk SIP peers for the AI platform

Edit /etc/asterisk/sip-vicidial.conf or create a separate file included at the bottom of sip.conf. Add an outbound peer definition for the AI platform:

[AIVoiceOutbound]
type=peer
host=sip.openai.example.com
port=5060
username=vicidial_user
password=secure_password_here
context=from-internal
allow=ulaw,alaw,gsm
disallow=all
nat=yes
qualify=yes
qualify_timeout=2
qualify_freq=60
directmedia=no
trustrpid=yes
sendrpid=yes
insecure=port,invite
fromuser=vicidial_pbx
fromdomain=yourdomain.com
dtmfmode=rfc2833

For inbound calls from the AI platform, add a user definition:

[AIVoiceInbound]
type=user
context=from-ai-voice
allow=ulaw,alaw,gsm
disallow=all
nat=yes
dtmfmode=rfc2833

Reload the SIP configuration:

sudo asterisk -rx "sip reload"

Check that the peer is registered or reachable:

sudo asterisk -rx "sip show peers"

Look for AIVoiceOutbound in the list with status "OK" (for reachable) or "Unmonitored" (for no qualify).

Creating a dial plan for AI voice routing

Edit /etc/asterisk/extensions-vicidial.conf and add a context to handle calls destined for the AI voice platform. This context should be called when an agent initiates a call to a "bot" number or when the system detects a bot flag.

[from-agent-to-ai]
exten => _X.,1,NoOp(Inbound call from agent to AI platform: ${EXTEN})
exten => _X.,n,Set(CHANNEL(language)=en)
exten => _X.,n,Set(CDR(disposition)=TRANSFERRED_TO_AI)
exten => _X.,n,Set(CDR(cid_name)=Agent_${CALLERID(num)})
exten => _X.,n,Set(CALLERID(name)=<Agent ${CALLERID(num)}>)
exten => _X.,n,Dial(SIP/${EXTEN}@AIVoiceOutbound,45,tT)
exten => _X.,n,Log(NOTICE,Call to AI platform failed: ${DIALSTATUS})
exten => _X.,n,Hangup()

[from-ai-voice]
exten => _X.,1,NoOp(Inbound call from AI platform: ${EXTEN})
exten => _X.,n,Set(CDR(cid_name)=AI_RESPONSE)
exten => _X.,n,Set(CHANNEL(language)=en)
exten => _X.,n,Macro(agent-route,${EXTEN})
exten => _X.,n,Hangup()

The first context routes outbound calls from an agent to the AI platform. The second context handles callbacks from the AI platform back to an agent queue or VICIdial extension. Reload extensions:

sudo asterisk -rx "dialplan reload"

Modifying the VICIdial agent dial routine

The core agent dialing logic lives in the AGI scripts. Locate the main agent call handler:

ls -la /usr/share/astguiclient/agi/

VICIdial typically uses an AGI script (often named something like call_agent.agi or routed via the extensions context from-internal). The script must detect when a call destination is a bot or AI service, then route to the AI carrier instead of the standard PSTN carrier.

Create a custom AGI script or modify the existing one. Here is a simplified example that checks the destination in the database and routes accordingly:

#!/usr/bin/perl
use DBI;
use Asterisk::AGI;
use strict;

my $agi = new Asterisk::AGI;
my %hash = $agi->ReadParse();

my $exten = $hash{'extension'};
my $channel = $hash{'channel'};
my $agent_id = $hash{'callerid'};

# Database connection (adjust DSN to your setup)
my $dbh = DBI->connect('dbi:mysql:asterisk:localhost', 'vicidial', 'vicidial_password');

# Query the destination to see if it is an AI bot
my $sth = $dbh->prepare(
    "SELECT bot_flag FROM vicidial_agent_pool WHERE agent_id = ? AND bot_flag = 'Y' LIMIT 1"
);
$sth->execute($agent_id);
my $is_bot = $sth->fetchrow_array();
$sth->finish();

# Check if the dialed extension matches a bot carrier
my $bot_sth = $dbh->prepare(
    "SELECT carrier_id FROM vicidial_carrier_log WHERE carrier_name LIKE '%AI%' OR carrier_name LIKE '%Bot%' LIMIT 1"
);
$bot_sth->execute();
my $ai_carrier = $bot_sth->fetchrow_array();
$bot_sth->finish();

if ($is_bot || $exten =~ /^555/) {  # Example: 555xxxx is reserved for bots
    $agi->exec('Dial', "SIP/$exten\@AIVoiceOutbound,30,tT");
} else {
    # Standard PSTN routing
    $agi->exec('Dial', "SIP/$exten\@StandardCarrier,30,tT");
}

$dbh->disconnect();
exit(0);

This script is illustrative. In production, integrate this logic into your existing AGI handler or use VICIdial's built-in call routing tables.

Configuring remote agent soft-phone settings

Remote agents connect to VICIdial via SIP. The agent's soft-phone must be configured with the VICIdial SIP server credentials. Common soft-phones include Zoiper, Linphone, or MicroSIP.

Example SIP configuration for a remote agent soft-phone:

SIP Server: 192.168.1.100 (your VICIdial server)
Port: 5060
Username: agent_001
Password: agent_password
Display Name: Agent 001
Codecs (in order): G.711u (ulaw), G.729, GSM

Test the connection by placing a call from the agent soft-phone to an extension on the VICIdial system. In the Asterisk CLI, confirm the channel appears:

sudo asterisk -rx "sip show channels"

You should see a channel like SIP/agent_001-00000123.

Ensuring codec compatibility between agent and AI platform

Both the agent's SIP connection and the AI platform trunk must support at least one common codec. G.711u (ulaw) is the most widely supported but has higher bandwidth. G.729 requires a license but compresses better. GSM is lightweight but lower quality.

Check what codecs the AI platform accepts. Most document this in their SIP trunk setup guide. In your sip-vicidial.conf for the AIVoiceOutbound peer, list codecs in order of preference:

allow=ulaw,alaw,gsm,g729

Similarly, ensure the agent soft-phone supports these codecs. If a mismatch occurs, Asterisk will attempt transcoding on the fly, which increases CPU load and introduces latency. Force a common codec in the dial plan:

exten => _X.,n,Dial(SIP/${EXTEN}@AIVoiceOutbound/ulaw,45,tT)

Monitor codec selection on active calls:

sudo asterisk -rx "core show channels verbose"

Look for lines like Format: ulaw (ulaw) or Format: unknown (which indicates a codec issue).

Handling RTP and jitter in WAN environments

Remote agents connecting over public internet face variable latency and packet loss. VICIdial's Asterisk must be configured to handle this gracefully.

Edit /etc/asterisk/rtp.conf:

[general]
rtpstart=10000
rtpend=20000
rtcpcheck=yes
nat=yes
icesupport=yes
dtlscerti=/etc/asterisk/keys/asterisk.crt
dtlscertfile=/etc/asterisk/keys/asterisk.key
dtlssetup=actpass

The ICE (Interactive Connectivity Establishment) support helps agents behind NAT/firewalls find the best path. DTLS adds encryption and works with SRTP (Secure RTP).

Increase the jitter buffer size for WAN calls. Edit /etc/asterisk/asterisk.conf:

[options]
maxfiles=131072
debug=3

Then in the SIP peer for remote agents, add:

[RemoteAgent]
type=peer
host=dynamic
allow=ulaw,alaw,gsm,g729
disallow=all
nat=yes
qualify=yes
qualify_timeout=3
qualify_freq=30
icesupport=yes
directmedia=no
jbenable=yes
jbimpl=adaptive
jbmaxsize=200
useclientcode=yes

The jitter buffer (jbenable=yes, jbimpl=adaptive) smooths out packet arrival timing. jbmaxsize=200 allows up to 200ms of buffering.

Test latency from the agent machine:

ping -c 10 192.168.1.100
mtr -c 20 192.168.1.100

Latency should be under 100ms; above 150ms, VoIP quality degrades noticeably.

Storing and retrieving agent call context in the database

When an agent initiates a call to an AI bot, context about the agent (ID, name, campaign) and the lead (phone, notes) must be passed to the AI system. This is often done via SIP headers or call recording metadata.

VICIdial stores agent login and call state in the vicidial_agent_log table:

SELECT agent_id, campaign_id, status, lead_id FROM vicidial_agent_log WHERE agent_id = 'agent_001';

Before routing a call to the AI platform, query the agent's current campaign and pass it as a custom SIP header:

exten => _X.,n,Set(SIP_HEADERS(X-Agent-Campaign)=${CAMPAIGN})
exten => _X.,n,Set(SIP_HEADERS(X-Lead-Phone)=${EXTEN})
exten => _X.,n,Set(SIP_HEADERS(X-Agent-ID)=${CALLERID(num)})
exten => _X.,n,Dial(SIP/${EXTEN}@AIVoiceOutbound,45,tT)

On the AI platform side (if you control both endpoints), parse these headers to load agent context.

Recording calls between agent and AI voice platform

By default, VICIdial records all calls to the recordings directory (typically /var/spool/asterisk/monitor/). When a call is transferred to an AI platform, Asterisk bridges the channels, and the recording continues. However, if the bridge is not set to "mix audio," only one direction is captured.

Ensure calls are recorded in mixed mode:

exten => _X.,n,MixMonitor(/var/spool/asterisk/monitor/${UNIQUEID}.wav,ab)
exten => _X.,n,Dial(SIP/${EXTEN}@AIVoiceOutbound,45,tT)

The ab flags mean: a = record agent side, b = record the other side.

After the call, the file /var/spool/asterisk/monitor/${UNIQUEID}.wav will contain both sides of the conversation. VICIdial's database log in vicidial_log or vicidial_closer_log will record the filename in the recording_id column.

Query call recordings:

SELECT call_date, phone_number, recording_id FROM vicidial_log WHERE agent_id = 'agent_001' AND call_date > DATE_SUB(NOW(), INTERVAL 1 DAY) ORDER BY call_date DESC;

Listen to the recording at /var/spool/asterisk/monitor/recording_id.wav.

Testing the integration end-to-end

Set up a test agent account and soft-phone. Log the agent into the VICIdial web screen at /agc/vicidial.php. Place a test call to a number that maps to your AI platform (either via a bot flag in the database or a special 555xxxx extension).

On the server, tail the Asterisk log:

tail -f /var/log/asterisk/messages

Monitor call state:

sudo asterisk -rx "core show channels verbose"

During the call, check:

Both agent and AI platform RTP streams are active (no "No RTP activity" warnings).
Codec negotiation completes (look for "Format: ulaw" or similar).
Call recording file is created and contains audio from both sides.
After hangup, call record appears in vicidial_log with correct disposition.

Example successful log output:

[2024-01-15 14:23:45] NOTICE[1234][C-0000a1b2]: app_dial.c:xxxx: Called SIP/agent_001@RemoteAgent
[2024-01-15 14:23:47] NOTICE[1234][C-0000a1b2]: app_dial.c:xxxx: SIP/agent_001@RemoteAgent is ringing
[2024-01-15 14:23:50] NOTICE[1234][C-0000a1b2]: app_dial.c:xxxx: SIP/agent_001@RemoteAgent answered call
[2024-01-15 14:23:50] NOTICE[1234][C-0000a1b2]: bridge.c:xxxx: Bridging SIP/agent_001@RemoteAgent with SIP/5551234567@AIVoiceOutbound

If the call fails to bridge, check for:

"No route to host" or timeout: Network/firewall issue between VICIdial and AI platform.
"Codec not compatible": Add a codec translation or force a common one.
"Authentication failed": Verify carrier username and password in sip-vicidial.conf.

Troubleshooting

Agent hears silence or one-way audio during AI voice calls

Check codec negotiation by running asterisk -rx "sip show channels" during an active call. If the format shown is "unknown" or mismatched between channels, add explicit codec forcing to the dial plan. Verify that the AI platform trunk supports the codec your agent's soft-phone negotiates. Run a SIP debug capture: asterisk -rx "sip set debug on" and review SDP payloads in the logs.

Calls disconnect after 30 seconds

This is often a SIP timeout or lack of RTP activity. Edit the Dial() command to increase the timeout: Dial(SIP/${EXTEN}@AIVoiceOutbound,120,tT) instead of the default 30 seconds. If RTP is not flowing, check that directmedia=no is set on the peer and that the agent and platform are using compatible RTP ports (rtpstart/rtpend in rtp.conf should not overlap with other services).

Agent's soft-phone does not register to VICIdial

Ensure the agent's SIP username and password match an entry in the VICIdial database (typically in a custom user table or in Asterisk's AstDB). Verify the agent soft-phone is pointing to the correct VICIdial server IP and port 5060. Check firewall rules on the VICIdial server: sudo ufw allow 5060/udp or disable iptables temporarily to isolate the issue.

Recordings are one-sided or empty

Ensure MixMonitor is called before Dial() in the extensions context. Check that /var/spool/asterisk/monitor/ has write permissions for the asterisk user: ls -la /var/spool/asterisk/monitor/. If the directory doesn't exist, create it: sudo mkdir -p /var/spool/asterisk/monitor && sudo chown asterisk:asterisk /var/spool/asterisk/monitor/. Verify that app_mixmonitor is loaded: asterisk -rx "core show modules" | grep mixmonitor.

High latency or jitter causing audio drops

Run a continuous ping and mtr test to the AI platform endpoint to measure latency. If consistent latency is above 150ms, check network routing via traceroute. On the VICIdial side, increase the jitter buffer: set jbmaxsize=400 and jbimpl=adaptive in the SIP peer config. Reduce audio codec bandwidth by forcing G.729 (if licensed) or GSM instead of G.711u.

AI platform returns authentication failure

Double-check the carrier_user and carrier_pass in the vicidial_carrier_log table. Many AI platforms issue SIP credentials that include a domain or realm; include those in the username field if necessary (e.g., vicidial_user@platform.example.com). Test the credentials independently with a SIP client like Linphone to confirm they work before debugging the Asterisk integration.

Inbound calls from AI platform do not route to agent

Verify the [from-ai-voice] context exists in extensions-vicidial.conf and includes a macro or dialplan to route to your agent queue. Ensure the AI platform is configured to send inbound calls to the correct VICIdial server IP and extension. Check that the VICIdial firewall allows inbound SIP on port 5060 from the AI platform's public IP.

Frequently asked questions

Can I use G.729 codec to save bandwidth for remote agents?

Yes, but you must have a G.729 codec license installed on Asterisk. Purchase licenses from DIGIUM or a reseller and install them in /usr/lib/asterisk/modules/. After installation, add allow=g729 to the relevant SIP peer configs. Confirm the codec loads by running asterisk -rx "core show modules" | grep g729. Note that transcoding between G.729 and the AI platform's codec (often G.711) increases CPU load; if you have many simultaneous calls, stick with G.711u unless bandwidth is severely constrained.

How do I pass agent custom data or notes to the AI voice bot?

Store the data in the VICIdial lead or agent record, then retrieve it in the AGI script before dialing. Query the vicidial_list table for lead notes or the vicidial_agent_log table for agent notes. Pass the data as SIP headers (e.g., SIP_HEADERS(X-Lead-Notes)=${NOTES}) or embed it in the dialed extension (e.g., SIP/555${LEAD_ID}@AIVoiceOutbound). The AI platform must be configured to parse these headers or extension patterns on the inbound side.

What if the AI platform uses WebSocket instead of SIP?

VICIdial and Asterisk 16/18 do not have native WebSocket support for calls. However, you can bridge a SIP call to a WebSocket endpoint using a gateway service like Jambonz or FreeSWITCH in between. Alternatively, use the AI platform's SIP trunk if available (most offer one). If WebSocket is your only option, consider deploying a dedicated SIP-to-WebSocket gateway on the same network and pointing the VICIdial trunk to it.

How do I monitor call quality metrics like MOS score during AI agent calls?

Asterisk does not natively compute MOS (Mean Opinion Score). However, you can enable RTP statistics logging and use third-party tools like SIPp or QEMU to analyze packet loss and jitter. For production monitoring, integrate with a service like Twilio or Vonage that provides call quality dashboards, or deploy an open-source RTPFLOW analysis tool. Store CDR data in VICIdial's database and correlate it with Asterisk logs to identify patterns in call failures.

Can I failover to a second AI voice platform if the primary is unreachable?

Yes. In the dial plan, use Dial()'s multiple target syntax to try the primary platform first, then fall back to a secondary. For example: Dial(SIP/${EXTEN}@AIVoicePrimary,20,tT|SIP/${EXTEN}@AIVoiceSecondary,20,tT). If the primary times out or is unreachable, Asterisk will attempt the secondary. Ensure both platforms are defined as separate peers in sip-vicidial.conf. Monitor each platform's uptime and update DNS or config entries if you swap primary and secondary roles.

Summary

Integration of AI voice agents with VICIdial remote agents requires correct SIP trunk provisioning for the AI platform, codec alignment between agent soft-phones and the voice platform, and RTP/jitter buffer tuning for WAN stability. Implement custom AGI logic to route calls from agents to the AI carrier based on bot flags or reserved extension ranges. Configure remote agent soft-phones with compatible codecs and NAT settings, enforce MixMonitor for call recording on both sides of the bridge, and test end-to-end with a real agent and soft-phone before deploying to production.

Monitor SIP show peers to confirm trunk registration, verify call recordings contain both agent and AI audio, and check vicidial_log for call disposition after each test. Common failure points are codec mismatch (fix via allow= directives), one-sided audio (enable MixMonitor with ab flags), and connectivity issues (test mtr and firewall rules). Once a test call succeeds with full audio, appropriate codec negotiation, and recorded wav file, the integration is ready for live campaigns.

How to Manage ViciDial Lists: Deactivate, Reset & Delete Leads

Moisi Trungu — Thu, 15 Oct 2026 03:30:02 +0000

How to Manage ViciDial Lists: Deactivate, Reset & Delete Leads

Remove, reset, or purge unwanted contacts from ViciDial lists without corrupting call logs or orphaning agent records.

List management in ViciDial requires direct database access combined with understanding how leads link to call logs, agent activity, and callbacks. Deactivating a lead preserves history; resetting a lead clears its attempt count and status; deleting a lead removes it entirely but risks breaking referential integrity if done carelessly. This guide shows the exact SQL, CLI commands, and web interface steps to manage each operation safely.

Tested on ViciDial 2.14 (SVN 3555+) and 2.13, Asterisk 16/18, MariaDB 10.5+, PHP 7.4+.

Prerequisites

SSH access to the ViciDial server or a machine with remote MySQL client installed.
MySQL or MariaDB command-line client (mysql) or a GUI tool like DBeaver or phpMyAdmin.
Database credentials with SELECT, UPDATE, DELETE on the asterisk database.
Understanding of ViciDial list IDs (numeric identifiers for campaigns/lists).
A backup of the asterisk database taken within the last 24 hours.
Agent or admin access to the web interface at /vicidial/admin.php.
Knowledge of which leads are tied to active callbacks or scheduled follow-ups.

Understanding ViciDial lead status codes

Before modifying leads, know what each status means in the vicidial_list table.

The status field controls whether an agent can dial a lead:

NEW: Untouched, ready to dial.
ACTIVE: Currently being worked.
DEAD: Permanently removed from dialing queue (do not call, duplicate, bad number).
INBOUND: Reserved for inbound transfers.
QUEUE: Waiting for agent availability.
PAUSE: Temporarily held.

Related flags in the same table:

Called: 0 = never called, 1 = called at least once. Resetting this is what "resetting" a lead means.
Phone: The phone number string (E.164 format preferred: +1-555-0101).
Lead_id: Auto-incremented primary key, unique per list.
List_id: Foreign key linking to vicidial_lists.

The vicidial_log table records every call attempt. Deleting a lead does not automatically delete its logs, so historical call data remains for reporting and compliance.

Deactivating leads through the web interface

The simplest method for small volumes (10-100 leads) is the ViciDial admin panel.

Navigate to Admin Leads (e.g., http://your-vicidial-server/vicidial/admin.php).
Select your List from the dropdown.
Use the Search form to find the lead by phone, name, or ID.
Click the lead row to open its detail page.
Change Status to DEAD.
Set "Do Not Call" flag if you want to block it from future list reloads.
Save.

This marks the lead as DEAD in vicidial_list.status, preventing agents from dialing it. The lead remains in the database for audit purposes. Call logs stay intact.

Use this method when:

You are removing a single lead or a handful.
You need a timestamp audit trail (ViciDial logs admin actions).
You want agent feedback recorded (notes field).

For bulk operations, use the database directly (see below).

Deactivating leads in bulk with SQL

To deactivate 100+ leads at once, use direct SQL. This is 100 times faster than the web interface.

Step 1: Identify the list ID and lead range.

SELECT list_id, list_name, active FROM vicidial_lists 
WHERE list_name LIKE 'Campaign_Q4%';

Note the list_id (e.g., 101).

Step 2: Deactivate leads by phone number pattern.

UPDATE vicidial_list 
SET status = 'DEAD', called = 1 
WHERE list_id = 101 
  AND phone LIKE '555%';

This marks all leads in list 101 with phone numbers starting in 555 as DEAD. The called = 1 ensures they appear in reporting as "attempted".

Step 3: Verify the change.

SELECT count(*), status FROM vicidial_list 
WHERE list_id = 101 
GROUP BY status;

Example output:

count(*)  status
-------  -------
  4521   NEW
   512   DEAD
   189   PAUSE

Common deactivation scenarios:

Deactivate all leads in a list:

UPDATE vicidial_list SET status = 'DEAD' WHERE list_id = 101;

Deactivate leads with invalid phone numbers:

UPDATE vicidial_list SET status = 'DEAD' 
WHERE list_id = 101 AND (phone IS NULL OR phone = '' OR length(phone) < 10);

Deactivate leads marked as duplicate in notes:

UPDATE vicidial_list SET status = 'DEAD' 
WHERE list_id = 101 AND comments LIKE '%duplicate%';

Deactivate leads not called in 30 days:

UPDATE vicidial_list SET status = 'DEAD' 
WHERE list_id = 101 
  AND last_local_call_time < DATE_SUB(NOW(), INTERVAL 30 DAY);

After any bulk update, always run the verification query to confirm the count.

Resetting leads (clearing attempts and status)

Resetting a lead removes its call history and attempt counter, returning it to NEW status. This re-queues it for dialing without losing the contact record itself.

Use reset when:

An agent dialed a number by mistake (robocall or wrong extension).
A lead was marked DEAD but is actually reachable (bad skip trace).
You want to retry a lead that failed several times.

Reset a single lead by ID:

UPDATE vicidial_list SET status = 'NEW', called = 0 
WHERE lead_id = 4521;

Reset all leads in a list that are currently PAUSE or DEAD:

UPDATE vicidial_list SET status = 'NEW', called = 0 
WHERE list_id = 101 AND status IN ('PAUSE', 'DEAD');

Reset leads that have not been called in the last 60 days:

UPDATE vicidial_list SET status = 'NEW', called = 0 
WHERE list_id = 101 
  AND (last_call_time IS NULL OR last_call_time < DATE_SUB(NOW(), INTERVAL 60 DAY));

Do not reset the entire active_in_list or priority fields; those control dialing order and should remain intact.

Resetting does not delete call logs. The lead's history in vicidial_log remains for compliance. You are only clearing the queuing state.

Deleting leads permanently

Permanent deletion removes a lead entirely from the vicidial_list table. Use this only when you are certain the lead should never appear in reports or callbacks.

Risks of deleting:

Orphaned call logs: A vicidial_log record pointing to a deleted lead_id cannot be joined back.
Broken callbacks: If a callback is scheduled for a deleted lead_id, it will fail.
Compliance issues: You lose the record that a contact was in your system.

For these reasons, deactivation (DEAD status) is safer than deletion in most cases.

Delete a single lead (verify first):

-- Check for callbacks
SELECT callback_id, lead_id, callback_date FROM vicidial_callbacks 
WHERE lead_id = 4521;

-- If none, delete
DELETE FROM vicidial_list WHERE lead_id = 4521;

Delete leads that were never called:

DELETE FROM vicidial_list 
WHERE list_id = 101 
  AND called = 0 
  AND status = 'NEW';

Delete duplicate phone numbers (keep the first occurrence):

DELETE FROM vicidial_list 
WHERE lead_id NOT IN (
  SELECT MIN(lead_id) FROM vicidial_list 
  WHERE list_id = 101 
  GROUP BY phone
) AND list_id = 101;

Before running any DELETE, always take a backup:

mysqldump -u asterisk -p asterisk vicidial_list > /tmp/vicidial_list_backup_$(date +%s).sql

Then run the DELETE in a transaction so you can roll back if needed:

START TRANSACTION;
DELETE FROM vicidial_list 
WHERE list_id = 101 AND called = 0 AND status = 'NEW';
-- Check affected rows
SELECT ROW_COUNT();
-- If correct, commit
COMMIT;
-- If wrong, ROLLBACK;

Purging call logs for deleted leads

If you delete leads, their call logs remain in vicidial_log. To clean them up:

DELETE FROM vicidial_log 
WHERE lead_id NOT IN (SELECT lead_id FROM vicidial_list);

This removes any orphaned log entries. Run this once per month if you delete leads regularly.

Check how many rows will be affected first:

SELECT count(*) FROM vicidial_log 
WHERE lead_id NOT IN (SELECT lead_id FROM vicidial_list);

On a large system (millions of logs), this query can take minutes. Use it during low-traffic windows (2-4 AM).

Resetting call attempt counters

ViciDial tracks call attempts per lead in the vicidial_log table. Some lists use custom attempt limits (e.g., dial 5 times, then move to DEAD). If you want to allow more attempts without fully resetting the lead, clear the attempt count.

The vicidial_log.lead_id and vicidial_log.list_id combination forms the basis for attempt counting. To allow a lead to be dialed again without losing history:

UPDATE vicidial_list 
SET called = 0 
WHERE list_id = 101 AND lead_id IN (
  SELECT DISTINCT lead_id FROM vicidial_log 
  WHERE list_id = 101 AND call_date > DATE_SUB(NOW(), INTERVAL 7 DAY)
);

This resets the called flag for leads called in the last 7 days, letting them dial again. The call logs remain for reporting.

Bulk importing and list resets

When you import a new lead list via the web interface (Admin Leads, Upload), ViciDial assigns new lead_ids and calculates the next_state. If you want to reload the same list (e.g., after adding new fields), you must first reset or delete the old list.

To reload a list:

Note the list_id.
Delete all leads in that list: DELETE FROM vicidial_list WHERE list_id = 101;
Re-import the list via the web interface.

Alternatively, use the command-line script:

/usr/share/astguiclient/ADMIN_manual_list_reset.pl \
  --list-id 101 \
  --campaign Outbound_Q4

This script safely resets the list, clearing all leads and resetting attempt counts. Check the script logs:

tail -f /var/log/astguiclient/ADMIN_manual_list_reset.log

Using the CLI for list diagnostics

Query the ViciDial database from the command line without entering MySQL:

mysql -u asterisk -p asterisk -e \
  "SELECT lead_id, phone, status, called FROM vicidial_list WHERE list_id = 101 LIMIT 10;"

Get a count by status:

mysql -u asterisk -p asterisk -e \
  "SELECT status, count(*) as count FROM vicidial_list WHERE list_id = 101 GROUP BY status;"

Find leads with callback scheduled:

mysql -u asterisk -p asterisk -e \
  "SELECT lead_id, callback_date, callback_notes FROM vicidial_callbacks \
   WHERE callback_date > NOW() ORDER BY callback_date LIMIT 20;"

Check for active agents on a list:

mysql -u asterisk -p asterisk -e \
  "SELECT user, status, lead_id FROM vicidial_agent_status \
   WHERE campaign = 'Outbound_Q4' AND status != 'OFFLINE';"

Connect to the Asterisk console to see if agents are paused or active:

asterisk -rx "vicidial show agents"

Hangup an agent's call:

asterisk -rx "channel request hangup SIP/2001"

Troubleshooting

Problem: Leads won't queue after resetting.

Check if the list itself is active:

SELECT active, campaign FROM vicidial_lists WHERE list_id = 101;

If active = 0, the list is paused. Set active = 1 to resume.

Problem: Deleted leads still appear in agent screen.

Agents may have cached the lead list in browser memory. Have them log out and log back in, or clear the browser cache.

Problem: Call logs show lead_id that no longer exists.

This is normal if you deleted leads after calls. To prevent orphaned logs in the future, use DEAD status instead of DELETE.

To see orphaned leads:

SELECT DISTINCT vl.lead_id FROM vicidial_log vl 
LEFT JOIN vicidial_list vl2 ON vl.lead_id = vl2.lead_id 
WHERE vl2.lead_id IS NULL LIMIT 20;

Problem: Bulk deactivation query timed out.

The vicidial_list table may lack indexes on frequently filtered columns. Add an index:

ALTER TABLE vicidial_list ADD INDEX idx_list_status (list_id, status);

Then retry the query.

Problem: Web interface is slow after bulk update.

ViciDial caches list state in memory. Restart the astguiclient services:

systemctl restart astguiclient

Wait 30 seconds for caches to rebuild.

Problem: Attempting to delete a lead returns foreign key constraint error.

A callback, inbound transfer, or agent note references this lead. Find and remove those references first:

SELECT * FROM vicidial_callbacks WHERE lead_id = 4521;
SELECT * FROM vicidial_inbound_anicode WHERE lead_id = 4521;
SELECT * FROM vicidial_agent_notes WHERE lead_id = 4521;

Delete or reassign those rows, then try deleting the lead again.

Frequently asked questions

Why does ViciDial mark calls as DEAD automatically?

ViciDial can be configured to auto-mark a lead as DEAD after a certain number of failed attempts (no answer, wrong number, answering machine). This is set in the campaign's dialer settings. If you want to disable auto-DEAD, go to Admin Campaigns, select your campaign, and set "Max attempts" to a high number (e.g., 999) or use a custom status like PAUSE instead. Leads marked DEAD manually will not be re-queued unless you reset them.

Can I export a list of deactivated leads?

Yes. Use the web interface (Admin Leads, Export) to download the list as CSV, or run a MySQL query and pipe it to a file: mysql -u asterisk -p asterisk -e "SELECT * FROM vicidial_list WHERE list_id = 101 AND status = 'DEAD';" > /tmp/deactivated_leads.txt. The CSV export includes all fields and can be reimported into another system for compliance records.

What happens to agent statistics if I delete a lead?

Deleting a lead does not erase the agent's call history or stats. The vicidial_closer_log records agent performance (talk time, dials, disposition). If you delete the lead after the call, the log entry remains but the agent can no longer view the lead's details in the agent screen. For this reason, use DEAD status instead of deletion if you want to preserve agent accountability.

Can I restore a deleted lead from backup?

Yes, but only if you have a database backup taken before the deletion. Restore the backup to a new database, extract the deleted lead row, and insert it into the production database. ViciDial does not have a trash/undelete feature for individual leads. Always backup before bulk delete operations: mysqldump -u asterisk -p asterisk vicidial_list > /tmp/backup.sql.

How do I reset leads without stopping active dialing?

Use the database directly (SQL UPDATE statements). Web interface resets may lock the list temporarily. When updating with SQL, the change takes effect immediately without stopping agents. However, agents who are already dialing a lead will not see the reset until they finish the call and the system refreshes the lead state (usually within 2-3 seconds).

Summary

You can now deactivate, reset, or delete leads in ViciDial using three approaches: the web interface for single leads and auditing, SQL UPDATE for bulk deactivation, and SQL DELETE for permanent removal (use sparingly). Deactivation (DEAD status) is the safest choice in production because it preserves call logs and agent accountability while removing leads from the dialing queue. Resetting clears the attempt counter and status, re-queuing a lead without losing history. Deleting is permanent and risks breaking callbacks and orphaning logs.

Always back up the database before bulk operations. Use START TRANSACTION and ROLLBACK if a query affects more rows than expected. After any bulk change, verify counts with SELECT queries and test with a single agent before resuming full-scale dialing. Run orphan log cleanup (DELETE from vicidial_log WHERE lead_id NOT IN...) monthly if you delete leads regularly, and add database indexes if bulk updates time out on large lists (100k+ leads).

Next steps: review the dialer campaign settings to confirm max-attempt rules are aligned with your deactivation policy, set up a monthly backup cron job if you have not already, and test a reset on a single lead during a quiet period to verify behavior matches your expectations.

Detecting Inbound Dead Air: Answered Calls Where Asterisk Never Accepts Carrier RTP

Moisi Trungu — Wed, 14 Oct 2026 03:30:02 +0000

Detecting Inbound Dead Air: Answered Calls Where Asterisk Never Accepts Carrier RTP

Some inbound calls answer normally at the SIP level, but the PBX never accepts a single RTP packet from the carrier. Asterisk already logs the evidence: a Strict RTP "learning" line that never turns into a lock or a source switch. Here's how I turned that into an alert, corroborated it with Homer RTCP, avoided the early-media trap, and used one SIP header to decide where to send the ticket.

Tested on: Asterisk 18 (ViciDial build, chan_sip, strictrtp=seqno, ConfBridge/softmix agent bridges), Loki, Homer with heplify-server on PostgreSQL 16. Check your Asterisk version's res_rtp_asterisk.c for the exact log wording.

What does inbound dead air look like?

The agent answers and hears nothing. SIP looks normal (INVITE, 183, 200 OK, ACK, BYE), the CDR shows a short answered call, and the agent gets blamed. The first one I caught sat in the agent's bridge for about 6 seconds. Asterisk's log showed the carrier's media address set from SDP, then nothing: no packet from that stream was ever accepted.

Then it stopped being a one-off. One morning, 26 answered inbound calls in 44 minutes had no carrier RTP accepted at the PBX, across three inbound trunks: 26 of the 125 calls on those trunks that reached an agent bridge, about one in five. The previous day's 12 business hours had 1,330 bridged calls on the same trunks and none like this. What this shows is that no carrier RTP was accepted at the PBX; assigning blame needs packet captures or carrier evidence.

Which Asterisk log lines show whether RTP arrived?

With strict RTP enabled (strictrtp=yes, the default, or seqno), res_rtp_asterisk.c logs the source-learning state of every RTP stream at verbose level 4. Real lines from my server, with addresses, IDs and dates replaced:

[15:33:36] VERBOSE[17871][C-0001a2b3] res_rtp_asterisk.c: 0x7f7f... -- Strict RTP learning after remote address set to: 198.51.100.23:45892
[13:42:40] VERBOSE[3563][C-0001a0f0] res_rtp_asterisk.c: 0x7f7e... -- Strict RTP learning complete - Locking on source address 203.0.113.10:10256
[13:43:01] VERBOSE[6158][C-0001a0f3] res_rtp_asterisk.c: 0x7f7f... -- Strict RTP switching to RTP target address 203.0.113.20:30818 as source
[13:43:01] VERBOSE[6158][C-0001a0f3] res_rtp_asterisk.c: 0x7f7e... -- Strict RTP switching source address to 192.0.2.108:41882

What matters is where in the source each one is printed (Asterisk 18):

"learning after remote address set to" comes from ast_rtp_remote_address_set(). It fires when the far end's address is set from SDP. No packet is involved. Call it the open.
"Locking on source address", "switching to RTP target address ... as source" and "switching source address to" are all printed in ast_rtp_read(), the incoming-packet path. Each one means a packet arrived and Asterisk accepted that source. Call any of them a close.

A missing close doesn't mean zero packets hit the NIC. Strict RTP silently drops packets (debug level only) while it qualifies a different source address, which takes 4 sequential packets by default. So an open with no close means Asterisk never accepted a stream for that call. A real 50-packet-per-second stream bridged for several seconds should get through probation from any address, so an unpaired open is a strong candidate, but only a candidate.

Also, SDP says where the carrier wants to receive media, not where it sends from. A different source shows up as "switching source address to ", so the code pairs any close of the same stream within the carrier's ranges. A source outside your map becomes a false candidate that RTCP refutes.

Prerequisites: messages must log verbose at level 4 or higher, and you must match all three close forms. Over about four months my messages file had 414,530 opens, 369,649 "Locking", 234,563 "switching to RTP target" and 101,985 "switching source address" lines. Match only "Locking" and you'll flag healthy calls.

How do I tell an answered inbound call from a ring that went nowhere?

Unanswered calls also leave unpaired opens, since many carriers send nothing before the answer: 16 of 42 in a two-hour window around the incident. So for each unpaired open, read the lines with the same C- ID and check:

Bridge evidence. In my ViciDial build the customer channel goes into a ConfBridge softmix conference: Channel SIP/<trunk>-xxxx joined 'softmix' base-bridge. Ring-group overflow via Dial() logs app_dial.c: SIP/... answered instead. The first such line is the join; left 'softmix' is the leave. MeetMe-based ViciDial setups log different lines.
Direction. The context must name the inbound trunk's channel and inbound dialplan context. With one peer name for both directions, a silent outbound leg looks identical at the RTP layer.
DID and caller ID, for the alert. MixMonitor and my dialplan's Phone call from ... line both run before the 200 OK, so neither proves an answer.

Here's the first incident, filtered to its C- ID:

15:33:36  Strict RTP learning after remote address set to: 198.51.100.23:45892
15:33:36  Executing [process@trunkinbound:10] Verbose("SIP/carrier_in-...", "Phone call from ...")
15:33:40  Executing [...] ConfBridge("SIP/carrier_in-...", "...,vici_agent_bridge,vici_customer_user")
15:33:40  Channel SIP/carrier_in-... joined 'softmix' base-bridge <...>
15:33:46  Channel SIP/carrier_in-... left 'softmix' base-bridge <...>

An open, a join, a leave six seconds later, no close.

This covers calls that reached a bridge. In the incident window one more call never locked: the PBX answered it into the queue and the carrier hung up 7 seconds later, before any agent. If queue dead air matters to you, anchor on the PBX's 200 OK instead.

How long should I wait for a close?

Don't measure the hold from the open, which happens when the call arrives: ring-group overflow calls ring a long time (48 s in a 12-hour sample).

Anchor on the join instead. From the raw log file over 12 business hours: 1,330 bridged inbound calls on the three trunks, every one closed, median close in the same second as the join, latest 2 s after it. These time the logged transition, not packet arrival, at one-second resolution. During the incident the latest healthy close was 5 s after the join.

My rule: no close for the call by join + 20 s. That's several times the worst case, and the candidate is ready well under a minute after the agent picks up.

What does the detector code look like?

My watcher reads the logs from Loki, but tailing messages works the same. This is the core matching logic, simplified from my production script with documentation-range addresses. It runs as shown:

import re

# One entry per inbound door. The SAME matcher is used for opens and closes.
CARRIERS = {
    "carrier_a":    {"media": (r"203\.0\.113\.\d+", r"198\.51\.100\.\d+"), "chan": "SIP/carrier_a_in_"},
    "carrier_b_nl": {"media": (r"192\.0\.2\.190",), "chan": "SIP/carrier_b_nl-"},
    "carrier_b_de": {"media": (r"192\.0\.2\.163",), "chan": "SIP/carrier_b_de-"},
}
for c in CARRIERS.values():
    c["re"] = re.compile(r"(?:%s):\d+" % "|".join(c["media"]))

# [C-call tag] ... res_rtp_asterisk.c: 0x<RTP instance> -- Strict RTP <event>
RE_STRICT = re.compile(r"\[(C-[0-9a-f]+)\] res_rtp_asterisk\.c: (0x[0-9a-f]+) -- Strict RTP (.*)$")
RE_OPEN = re.compile(r"learning after remote address set to: (\S+)")
RE_CLOSE = re.compile(r"learning complete - Locking on source address (\S+)"
                      r"|switching to RTP target address (\S+) as source"
                      r"|switching source address to (\S+)")


def media_carrier(ip_port):
    for name, c in CARRIERS.items():
        if c["re"].fullmatch(ip_port or ""):
            return name
    return None


def parse_lines(lines):
    """lines: [(ts_ns, text)] for ONE PBX, complete for the window.
    Returns (opens, closes, multi).
    opens: the FIRST carrier-facing stream of each call (the inbound leg's).
    closes: {(call_tag, rtp_instance): [(ts_ns, seq)]}, carrier-range only.
    multi: call tags that opened more than one carrier-facing RTP instance
    (e.g. an overflow dialled out through a carrier). Flag those for review.
    seq is the position after a stable sort by time, so same-second lines
    keep their order."""
    ordered = sorted(enumerate(lines), key=lambda x: (x[1][0], x[0]))
    opens, closes, first, instances = [], {}, {}, {}
    for seq, (_, (ts, line)) in enumerate(ordered):
        m = RE_STRICT.search(line)
        if not m:
            continue
        cid, inst, event = m.groups()
        o = RE_OPEN.match(event)
        if o:
            if not media_carrier(o.group(1)):
                continue
            instances.setdefault(cid, set()).add(inst)
            if cid not in first:     # later re-opens (re-INVITE) and other legs: not covered
                first[cid] = {"ts_ns": ts, "seq": seq, "cid": cid, "inst": inst,
                              "ip_port": o.group(1), "trunk": media_carrier(o.group(1))}
                opens.append(first[cid])
            continue
        c = RE_CLOSE.match(event)
        if c:
            ip_port = next(g for g in c.groups() if g)
            # any address in a carrier range, so a learned source change pairs
            if media_carrier(ip_port):
                closes.setdefault((cid, inst), []).append((ts, seq))
    multi = {cid for cid, s in instances.items() if len(s) > 1}
    return opens, closes, multi


def first_close(closes, o, upto_ns):
    """First close of the SAME call and SAME RTP instance after the open and
    no later than upto_ns. Another call reusing the port, or another leg of
    this call, can't pair it."""
    for ts, seq in closes.get((o["cid"], o["inst"]), []):
        if (ts, seq) > (o["ts_ns"], o["seq"]) and ts <= upto_ns:
            return ts
    return None

Every Strict RTP line carries the call's C- tag and the RTP instance pointer (0x...), so closes are keyed by both, not by port. Carriers reuse ports, so a match keyed on port can let a later call's lock "close" a dead one. Keying on the instance stops another leg of the same call from closing it too, for example an overflow dialled back out through a carrier. Calls with more than one carrier-facing instance come back in multi, so you can review them; my two days of raw logs had none. Only carrier-range closes count, so a ring-group phone's stream doesn't. The input must be complete and in time order; the stable sort keeps same-second lines in file order.

A Loki query that feeds it:

{server="pbx", job="asterisk"} |= "Strict RTP" |~ "203\\.0\\.113\\.\\d+|198\\.51\\.100\\.\\d+|192\\.0\\.2\\.190|192\\.0\\.2\\.163"

LogQL literals use Go escaping, so a regex \. is written \\.. query_range returns newest-first by default and silently stops at limit, so use direction=forward, split any window that hits the limit until none does, and merge before parsing. Then check the pipeline against the source: for one 12-hour window, two Loki runs gave 2,121 and 2,016 opens where the raw messages file had 2,284. All numbers here come from the raw file.

Each pass keeps as a candidate any bridged inbound call with no close by join + 20 s plus an ingest margin, and sends it to Homer before posting.

Opens and closes must use the same matcher: an alias of mine that matched one carrier's two POP ranges for opens but one for closes made every healthy call on the other POP a false candidate.

What can Homer RTCP add?

Your own RTCP corroborates the log signal; it comes from the same reception state, so it isn't independent proof. The carrier's RTCP, where it sends any, is the closest thing to independent evidence short of packet captures at both ends.

Your side. In Asterisk 18 a report carries a reception block once the far end's SSRC is recorded (themssrc_valid), which happens after a packet passes strict RTP and then stays set. RFC 3550 only requires blocks for sources heard since the previous report, so another stack may drop the block between bursts; check yours. heplify stores RTCP as JSON in hep_proto_5_default. In 128,786 reports around the incident, report_blocks was a non-empty array or null, never empty, missing or unparseable, but the SQL below still treats those as unclear rather than "nothing received".

Here are my side's reports for one dead call, in seconds relative to the 200 OK:

T-0.6  {"sender_information":{"packets":250,...},"report_blocks":null}   <- before the answer
T+4.4  {"sender_information":{"packets":491,...},"report_blocks":null}
T+9.4  {"sender_information":{"packets":741,...},"report_blocks":null}
T+14.4 {"sender_information":{"packets":991,...},"report_blocks":null}

We sent 50 packets a second; no far-end SSRC was ever recorded.

The carrier's side. Under RFC 3550, a participant that has sent RTP since its last report sends a Sender Report; otherwise a Receiver Report. heplify's decoder only fills sender_information for a Sender Report, so a Receiver Report shows zeros. The stored type is useless here (on compound packets it's the last sub-packet, 202 on most of my rows), so I test for a non-zero NTP timestamp; in one hour of my data it always coincided with a non-zero packet count. In that dead call every carrier report had a zero NTP timestamp, and a report block for our stream with the sequence number climbing about 200 every 4 s.

Over one hour on that carrier's trunks, all 85 calls where our side reported reception got real Sender Reports from the carrier (1,680 of 1,723 carrier reports had a sender timestamp). The dead calls got Receiver Reports only. That corroborates the log signal: the carrier's RTCP stack didn't consider itself a sender at its SBC. Corroboration, not proof, and silent about further upstream. The other carrier's relay sent no RTCP my capture saw.

Why is RTCP from before the answer meaningless?

The first report above was emitted before the 200 OK. My PBX replies with 183 Session Progress with SDP and starts early media (ringback) at once, then sends the final 200 OK on answer. So it emits RTCP during the ring, when a healthy carrier usually isn't sending yet.

In the hour before the incident, all 148 bridged calls with my RTCP had our 183 before the 200 OK. 48 had a report before the answer, and all 48 pre-answer reports had no receiver block. From 2 s after the answer it flipped: 2,420 reports, all with a block, and no healthy call where they were all empty.

In the incident window a call torn down 3.7 s after the answer had one pre-answer report from my side, with no block. On RTCP alone it looks dead, but the Asterisk log shows it locked at the answer.

The rule I use:

Refuted: any report from your side, at any time, has a non-empty block. Asterisk accepted a packet. Don't post.
Confirmed: at least one report from your side 2 s or more after the 200 OK, and none of your reports for the call, read through the end of the call, has a block.
Unknown: no post-answer report yet (very short calls, capture lag). Re-check next pass, and say so if it never settles.

All 26 incident calls came back confirmed: 1 to 5 post-answer reports each from my side, read 20 minutes past the answer (so through the end of every call), none with a block, and no Sender Report from either carrier.

How do I find the right Homer dialog for a log line?

Homer doesn't know the C- ID, but the carrier's SDP has the media ip:port. It's one statement, so no result from a previous call can leak in. Run it with psql -v pbx_ip=... -v sig_ips=a,b -v media_ip=... -v media_port=... -v open_ts='YYYY-MM-DD HH:MM:SS+00' -f. RTCP is joined to the dialog by sid, which only works if your heplify correlates RTCP to the SIP Call-ID (mine does; check a known call first). My raw columns are bytea; if yours are text, drop the convert_from(...) wrappers.

-- One statement, no carried-over variables. Inputs: pbx_ip, sig_ips (comma list),
-- media_ip, media_port, open_ts. Output: one row with a verdict.
WITH inv AS (   -- carrier INVITEs whose SDP carries the open's media endpoint
  SELECT i.sid,
         coalesce(substring(convert_from(i.raw,'UTF8') from '(?i)\nx-oid:\s*([^\r\n]+)'), '') AS x_oid
  FROM hep_proto_1_call i
  WHERE i.create_date BETWEEN :'open_ts'::timestamptz - interval '90 seconds'
                          AND :'open_ts'::timestamptz + interval '3 seconds'
    AND i.protocol_header->>'dstIp' = :'pbx_ip'
    AND i.protocol_header->>'srcIp' = ANY (string_to_array(:'sig_ips', ','))
    AND convert_from(i.raw,'UTF8') LIKE 'INVITE %'
    AND convert_from(i.raw,'UTF8') ~ ('\nc=IN IP4 ' || replace(:'media_ip', '.', '\.') || '\r?\n')
    AND convert_from(i.raw,'UTF8') ~ ('\nm=audio ' || :'media_port' || ' ')
), d AS (       -- exactly one dialog, or nothing
  SELECT min(sid) AS sid, min(x_oid) AS x_oid, count(DISTINCT sid) AS n FROM inv
), a AS (       -- first 200 to an INVITE, sent by the PBX
  SELECT d.*, (SELECT min(k.create_date) FROM hep_proto_1_call k
               WHERE d.n = 1 AND k.sid = d.sid
                 AND k.create_date BETWEEN :'open_ts'::timestamptz - interval '90 seconds'
                                       AND :'open_ts'::timestamptz + interval '10 minutes'
                 AND k.protocol_header->>'srcIp' = :'pbx_ip'
                 AND convert_from(k.raw,'UTF8') ~ '^SIP/2\.0 200 '
                 AND convert_from(k.raw,'UTF8') ~* '\ncseq:\s*[0-9]+\s+invite') AS t200
  FROM d
), r AS (       -- every RTCP report of that dialog, classified
  SELECT p.create_date, p.protocol_header->>'srcIp' AS src, x.j,
         CASE WHEN x.j IS NULL                                THEN 'unparsed'
              WHEN NOT x.j ? 'report_blocks'                  THEN 'missing'
              WHEN jsonb_typeof(x.j->'report_blocks') = 'null'  THEN 'none'
              WHEN jsonb_typeof(x.j->'report_blocks') <> 'array' THEN 'malformed'
              WHEN jsonb_array_length(x.j->'report_blocks') = 0 THEN 'empty'
              ELSE 'blocks' END AS rb
  FROM a
  JOIN hep_proto_5_default p
    ON p.sid = a.sid
   AND p.create_date BETWEEN a.t200 - interval '2 minutes' AND a.t200 + interval '20 minutes'
  CROSS JOIN LATERAL (SELECT CASE WHEN convert_from(p.raw,'UTF8') IS JSON
                                  THEN convert_from(p.raw,'UTF8')::jsonb END AS j) x
  WHERE a.n = 1 AND a.t200 IS NOT NULL
), s AS (
  SELECT
    count(*) FILTER (WHERE rb IN ('unparsed','missing','malformed','empty')) AS unclear,
    count(*) FILTER (WHERE src = :'pbx_ip' AND rb = 'blocks') AS pbx_with_block,
    count(*) FILTER (WHERE src = :'pbx_ip' AND create_date >= (SELECT t200 FROM a) + interval '2 seconds') AS pbx_post,
    count(*) FILTER (WHERE src = :'pbx_ip' AND rb = 'none'
                     AND create_date >= (SELECT t200 FROM a) + interval '2 seconds') AS pbx_post_none,
    count(*) FILTER (WHERE src <> :'pbx_ip' AND jsonb_typeof(j->'sender_information') = 'object'
                     AND (j->'sender_information'->>'ntp_timestamp_sec') ~ '^[0-9]+$'
                     AND (j->'sender_information'->>'ntp_timestamp_sec')::numeric > 0) AS far_end_sr
  FROM r
)
SELECT a.n AS dialogs, a.t200, a.x_oid, s.*,
  CASE WHEN a.n <> 1 OR a.t200 IS NULL             THEN 'unknown'
       WHEN s.pbx_with_block > 0                    THEN 'refuted'
       WHEN s.unclear > 0 OR s.pbx_post = 0         THEN 'unknown'
       WHEN s.pbx_post_none = s.pbx_post            THEN 'confirmed'
       ELSE 'unknown' END AS verdict
FROM a, s;

The INVITE window allows 3 s of clock skew after the open. The verdict is unknown unless exactly one dialog matched and it has a PBX 200 whose CSeq method is INVITE, or if any report is unclear (unparsed, missing, malformed or empty report_blocks). All 26 incident calls came back confirmed, each matching exactly 1 dialog with no carrier re-INVITEs and no sendonly/inactive. A healthy control call came back refuted. The 3.7 s early-media call came back unknown. Always bound create_date: a day-wide version of a similar join hit my 5-minute timeout.

How did one SIP header point to a shared upstream?

The 26 calls came through three doors: 14 on Carrier A's direct trunk, 7 via Carrier B's NL-labelled SBC and 5 via its DE-labelled SBC. Three trunks failing in the same 44 minutes suggested something shared upstream.

The clue was in Carrier B's INVITEs, which carry an extra header:

INVITE sip:...                              (from Carrier B's SBC)
Call-ID: <a UUID, Carrier B's own format>
X-OID: <an ID in Carrier A's Call-ID format>

INVITE sip:...                              (from Carrier A's direct trunk)
Call-ID: <Carrier A's format>

Carrier B's own Call-IDs are UUIDs. The X-OID value has the same distinctive multi-field layout as Carrier A's Call-IDs on my direct trunk. It was on all 12 Carrier B dead calls, and that day on 651 of 651 inbound dialogs from the DE-labelled SBC and 346 of 349 from the NL-labelled one. The three without it showed one of our own extension or agent names in the From display name; I didn't dig further.

That doesn't prove Carrier A originated them. Shared SBC software could produce the same format, and simultaneous failures don't locate a fault. But it changed what I did: I widened the watcher, which had missed the 12 Carrier B calls, and put X-OID in every alert and ticket. Carrier B can confirm it against the upstream leg in its own signalling trace. Look at a full INVITE from each of your trunks for similar headers.

How do I keep the watcher from going silently blind?

A detector that finds nothing looks healthy. Each guard is a reason to investigate, not a diagnosis:

Empty is never clean. No strict-RTP lines? Check the PBX stream has any recent lines; warn after 24 hours without them.
Freshness. Newest PBX line older than 45 s? Skip the pass without advancing coverage. Tune it to your traffic.
Coverage per trunk. Hourly, for each trunk with 10+ inbound calls, count how many opened a stream on a mapped address. Below half means the media moved, the log format changed, or the classifier broke. One carrier's media move left my watcher blind until this check existed.
Keep Homer off the critical path. Short timeout, per-pass budget. A candidate still unsettled after 10 minutes is posted as "unconfirmed" only if bridged within 20 s of the open, in the bridge 5 s or more, and still unclosed.
Replay. A --dry-run mode with --start/--end runs the same rule over a past window and writes nothing. I replayed this version over the previous week before switching.

What doesn't this catch?

Silent RTP. Packets can flow with 0% loss and carry near-silence. For a complaint, check the recording's inbound leg.
Media that stops later. A stream that locks and then dies mid-call is out of scope. So is a re-INVITE whose new stream never locks.
Calls that never reach a bridge. See the queued call above.
Who hung up. Don't assume it was the caller giving up. In the 26 incident calls, the first BYE came from the PBX side on 15 and from the carrier on 11.

FAQ

Could my own firewall or NAT be the cause? Possibly; this method alone can't rule it out. What pointed away from my side: 99 calls on the same trunks were fine in the same minutes, the trunks were clean the day before, and Carrier B's RTCP showed no Sender Reports on its dead calls. For certainty, capture packets on the PBX interface during an occurrence and compare with the carrier's capture.