DEV Community

Moisi Trungu
Moisi Trungu

Posted on • Originally published at flixzone.icu

Asterisk Core Dump & Crash Analysis — Debugging Segfaults

#devops #linux #sip #voip

Asterisk Core Dump & Crash Analysis — Debugging Segfaults

Learn how to capture, analyze, and fix segmentation faults in production Asterisk systems running ViciDial by examining core dumps, generating backtraces, and identifying memory corruption and module conflicts.

Prerequisites

Before you start debugging core dumps, ensure you have:

  • Root access to your Asterisk/ViciDial server
  • GDB (GNU Debugger) installed: apt-get install gdb or yum install gdb
  • Asterisk debug symbols (compile from source with debug flags or install -debuginfo package)
  • Basic familiarity with Asterisk CLI commands and ViciDial architecture
  • Backup of your running system before making debugging changes
  • At least 2GB free disk space in /var/spool/asterisk/ for core dumps
  • Knowledge of your Asterisk version (run asterisk -V)

Understanding Core Dumps

What is a Core Dump?

A core dump is a file containing the memory snapshot of your process at the moment it crashed. When Asterisk segfaults, the kernel writes this memory to disk (if enabled), allowing you to inspect the call stack, variable states, and identify what caused the crash.

Why Core Dumps Matter in Production

In ViciDial environments, unpredictable crashes disrupt:

  • Active agent calls
  • Inbound call routing
  • Database synchronization
  • Lead distribution
  • Campaign execution

Core dumps let you identify whether crashes are caused by:

  • Custom modules or dialplan issues
  • SIP peer misconfigurations
  • Memory leaks in third-party codecs
  • Race conditions in high-concurrency scenarios
  • Incompatible module versions

Enabling Core Dumps on Linux

Step 1: Configure Kernel to Write Core Dumps

Check current core dump limit:

ulimit -c
Enter fullscreen mode Exit fullscreen mode

If output is 0, core dumps are disabled. Enable unlimited core dumps:

ulimit -c unlimited
Enter fullscreen mode Exit fullscreen mode

Make this permanent by adding to /etc/security/limits.conf:

# Allow asterisk user to generate unlimited core dumps
asterisk soft core unlimited
asterisk hard core unlimited
asterisk soft nofile 65536
asterisk hard nofile 65536
Enter fullscreen mode Exit fullscreen mode

Verify:

su - asterisk -s /bin/bash -c "ulimit -c"
Enter fullscreen mode Exit fullscreen mode

Expected output: unlimited

Step 2: Set Core Dump Location

Define where core files are written. Edit /etc/sysctl.conf:

kernel.core_pattern = /var/spool/asterisk/core-%e-%p-%t
kernel.core_uses_pid = 1
Enter fullscreen mode Exit fullscreen mode

Apply immediately:

sysctl -p
Enter fullscreen mode Exit fullscreen mode

This creates files like /var/spool/asterisk/core-asterisk-12345-1703020800 when crashes occur.

Step 3: Ensure Directory Permissions 

mkdir -p /var/spool/asterisk
chown asterisk:asterisk /var/spool/asterisk
chmod 755 /var/spool/asterisk
Enter fullscreen mode Exit fullscreen mode

Configuring Asterisk for Debug Output

Enable Core Dumps in asterisk.conf

Edit /etc/asterisk/asterisk.conf:

[options]
; Enable core dumps even when run as daemon
dumpcore = yes

; Log all debug output
debug = 3

; Verbose logging for call flow
verbose = 3

; Log warnings about lock contention
lockingtest = yes
Enter fullscreen mode Exit fullscreen mode

Restart Asterisk (gracefully):

asterisk -rx "core restart graceful"
Enter fullscreen mode Exit fullscreen mode

Wait for all channels to clear before Asterisk terminates.

Enable Module-Specific Debug

For SIP debugging (ViciDial primary protocol):

asterisk -rx "sip set debug on"
asterisk -rx "sip set debug ip 192.168.1.100"  # Debug single peer
Enter fullscreen mode Exit fullscreen mode

For channel/bridge issues:

asterisk -rx "core set debug 3"
Enter fullscreen mode Exit fullscreen mode

View live logs:

tail -f /var/log/asterisk/messages
Enter fullscreen mode Exit fullscreen mode

Compiling Asterisk with Debug Symbols

If your Asterisk binary lacks debug symbols, backtraces will be useless. Recompile:

cd /usr/src/asterisk-16.x.x
./configure CFLAGS="-g -O0" CXXFLAGS="-g -O0"
make clean
make -j4
make install
Enter fullscreen mode Exit fullscreen mode

Key flags:

  • -g: Include debug symbols
  • -O0: Disable optimization (makes symbols more readable; impacts performance)

Production tip: Compile with -g -O2 to balance debugging capability with performance.

Verify symbols are present:

file /usr/sbin/asterisk
Enter fullscreen mode Exit fullscreen mode

Should show: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, with debug_info

nm -l /usr/sbin/asterisk | grep asterisk_log | head -5
Enter fullscreen mode Exit fullscreen mode

Should return symbol entries, not empty output.

Triggering and Capturing a Crash

Safe Testing: Use Test Dialplan

Create a crashing dialplan in /etc/asterisk/extensions-test.conf:

[test-crash]
exten => 999,1,NoOp(Testing crash scenario)
exten => 999,n,SIPNotify(reboot,${CHANNEL})
exten => 999,n,Wait(2)
exten => 999,n,Hangup()
Enter fullscreen mode Exit fullscreen mode

Or trigger via asterisk CLI:

asterisk -rx "core set debug 5"
asterisk -rx "dialplan eval ${SOME_BROKEN_VAR}"  # Intentional bad variable
Enter fullscreen mode Exit fullscreen mode

Identify Recent Crashes 

ls -lah /var/spool/asterisk/core-* 2>/dev/null | tail -10
Enter fullscreen mode Exit fullscreen mode 
ls -lah /tmp/core.* 2>/dev/null | tail -10
Enter fullscreen mode Exit fullscreen mode

Check system logs for segmentation fault:

journalctl -xe | grep -i segfault
dmesg | grep -i segfault
tail -50 /var/log/syslog | grep -i "segmentation\|core dump"
Enter fullscreen mode Exit fullscreen mode

Example journal output:

Nov 23 14:32:01 vicidial-01 kernel: [34567.891234] asterisk[12345]: segfault at 0x0 ip 00007ffff5a2c000 sp 00007fffabc12345 error 4
Enter fullscreen mode Exit fullscreen mode

Analyzing Core Dumps with GDB

Step 1: Locate the Core File 

ls -lah /var/spool/asterisk/core-asterisk-* | tail -1
Enter fullscreen mode Exit fullscreen mode

Output example:

-rw------- 1 asterisk asterisk 324M Nov 23 14:32 /var/spool/asterisk/core-asterisk-12345-1700731920
Enter fullscreen mode Exit fullscreen mode

Step 2: Start GDB Session 

gdb /usr/sbin/asterisk /var/spool/asterisk/core-asterisk-12345-1700731920
Enter fullscreen mode Exit fullscreen mode

You should see:

GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Reading symbols from /usr/sbin/asterisk...
Reading symbols from /usr/lib/debug/.build-id/...
[New LWP 12345]
[New LWP 12346]
...
Core was generated by `/usr/sbin/asterisk -f'
Program terminated with signal SIGSEGV, Segmentation fault.
Enter fullscreen mode Exit fullscreen mode

Step 3: Generate Full Backtrace 

(gdb) thread apply all bt full
Enter fullscreen mode Exit fullscreen mode

This shows the call stack for all threads. Example output:

Thread 47 (LWP 12389):
#0  0x00007ffff5a2c000 in chan_sip_pvt_lock () from /usr/lib/asterisk/modules/chan_sip.so
#1  0x00007ffff5a1f234 in sip_pvt_lock_full () from /usr/lib/asterisk/modules/chan_sip.so
#2  0x00007ffff5a3d123 in handle_incoming () from /usr/lib/asterisk/modules/chan_sip.so
#3  0x00007ffff5a2e456 in sip_io_read () from /usr/lib/asterisk/modules/chan_sip.so
#4  0x000055555555c789 in ast_channel_tech_read () at channel.c:5432
#5  0x000055555556b012 in __ast_read () at channel.c:4123
#6  0x000055555556c234 in ast_read () at channel.c:4234
#7  0x00007ffff7890000 in bridge_channel_wait_for_frame () from /usr/lib/asterisk/modules/app_bridging.so
#8  0x00007ffff7891200 in bridge_channel_run () from /usr/lib/asterisk/modules/app_bridging.so
#9  0x000055555558c345 in ast_spawn_extension () at pbx.c:3912
#10 0x000055555558c600 in pbx_exec () at pbx.c:4012
#11 0x00007ffff6234567 in handle_request_invite () from /usr/lib/asterisk/modules/chan_sip.so
Enter fullscreen mode Exit fullscreen mode

Step 4: Focus on the Crashing Thread

Find the thread that crashed (typically reported first):

(gdb) thread 1
(gdb) bt
Enter fullscreen mode Exit fullscreen mode

Limit output to top 20 frames:

(gdb) bt 20
Enter fullscreen mode Exit fullscreen mode

Step 5: Inspect Variables at Crash Point 

(gdb) frame 0
(gdb) info local
(gdb) print *variablename
(gdb) print sizeof(structname)
Enter fullscreen mode Exit fullscreen mode

Example: Check if a pointer is NULL:

(gdb) print pvt
$1 = (struct sip_pvt *) 0x0
Enter fullscreen mode Exit fullscreen mode

A NULL pointer dereference is your culprit.

Step 6: Examine SIP Channel State

For ViciDial crashes, inspect the SIP channel context:

(gdb) frame 2
(gdb) print pvt->callid
(gdb) print pvt->dialplan
(gdb) print pvt->thestate
(gdb) print pvt->lastinvite
Enter fullscreen mode Exit fullscreen mode

Common Crash Patterns in ViciDial

Pattern 1: Null Pointer Dereference in SIP Module

Symptom: Crash in chan_sip.so when handling INVITE

Backtrace indicator:

#0  0x... in chan_sip_pvt_lock
#1  0x... in handle_incoming
#2  0x... in sip_io_read
Enter fullscreen mode Exit fullscreen mode

Root cause: SIP peer object deallocated while handling incoming request.

Debugging:

(gdb) print pvt
$1 = (struct sip_pvt *) 0x0
(gdb) print dialog
$2 = (struct sip_pvt *) 0x0
Enter fullscreen mode Exit fullscreen mode

Fix: Add NULL check in custom dialplan or module. Check /etc/asterisk/sip-vicidial.conf for stale peer configs:

asterisk -rx "sip show peers" | grep -i "UNREACHABLE\|LAGGED"
Enter fullscreen mode Exit fullscreen mode

Remove outdated peers:

asterisk -rx "sip prune peers"
Enter fullscreen mode Exit fullscreen mode

Pattern 2: Memory Leak Leading to Crash

Symptom: Asterisk runs fine initially, crashes after hours/days of operation.

Indicator: Check memory usage:

ps aux | grep asterisk | grep -v grep
# Look at RSS column (resident set size)
Enter fullscreen mode Exit fullscreen mode

Monitor over time:

watch -n 5 'ps aux | grep asterisk | grep -v grep'
Enter fullscreen mode Exit fullscreen mode

If RSS grows continuously, memory leak is present. Backtrace will show:

#0  0x... in malloc (or realloc)
Enter fullscreen mode Exit fullscreen mode

Debugging: Enable MALLOC_DEBUG in Asterisk:

Edit /etc/asterisk/asterisk.conf:

[options]
allowmultiple = no
astdb = /var/lib/asterisk/astdb.sqlite3
debug = 3
verbose = 3
dumpcore = yes
Enter fullscreen mode Exit fullscreen mode

Recompile Asterisk with:

./configure --with-malloc-debug
make
make install
Enter fullscreen mode Exit fullscreen mode

Then run GDB and check memory allocator:

(gdb) handle SIG35 noprint nostop
(gdb) continue
# ... let it run and crash ...
(gdb) bt
(gdb) print heapmgr
Enter fullscreen mode Exit fullscreen mode

Pattern 3: Race Condition in Database Access

Symptom: Intermittent crashes when multiple channels access ViciDial database tables simultaneously.

Affected tables: vicidial_log, vicidial_closer_log, vicidial_carrier_log

Backtrace indicator:

#0  0x... in ast_db_get
#1  0x... in ast_app_parse_timezones
#2  0x... in custom_app_exec
Enter fullscreen mode Exit fullscreen mode

Debugging: Check for missing table locks in custom AGI scripts or dialplan:

grep -r "astdb\|ast_db_get" /etc/asterisk/extensions-vicidial.conf | head -20
Enter fullscreen mode Exit fullscreen mode

Fix: Use dialplan locks when accessing shared resources:

exten => 100,1,Lock(vicidial-db-lock)
exten => 100,n,NoOp(Safe database access)
exten => 100,n,Unlock(vicidial-db-lock)
Enter fullscreen mode Exit fullscreen mode

Or use database transactions in AGI scripts:

#!/bin/bash
# Safe database query in AGI
(
  flock -x 200
  mysql -N -e "SELECT * FROM vicidial_log WHERE lead_id = $LEAD_ID LIMIT 1" asterisk
) 200>/var/lock/vicidial-db.lock
Enter fullscreen mode Exit fullscreen mode

Pattern 4: Module Incompatibility

Symptom: Crash when loading custom module or third-party codec.

Backtrace indicator:

#0  0x... in module_symbol_not_found
#1  0x... in dlopen
Enter fullscreen mode Exit fullscreen mode

Debugging: Check loaded modules:

asterisk -rx "module show loaded" | grep -i "custom\|third"
Enter fullscreen mode Exit fullscreen mode

Check module dependencies:

ldd /usr/lib/asterisk/modules/codec_custom.so | grep -i "not found"
Enter fullscreen mode Exit fullscreen mode

Fix: Rebuild module with correct Asterisk version:

cd /usr/src/asterisk-16.x.x/addons
make clean
make
make install
Enter fullscreen mode Exit fullscreen mode

Verify module loads without errors:

asterisk -rx "module load codec_custom"
asterisk -rx "core show settings" | grep "Build Date"
Enter fullscreen mode Exit fullscreen mode

Creating a Core Dump Archive for Analysis

Preserve crashes for later investigation or vendor support:

#!/bin/bash
# save_core_dump.sh

CORE_FILE=$1
ARCHIVE_DIR="/var/spool/asterisk/crash-archives"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

mkdir -p $ARCHIVE_DIR

# Tar the core and associated binaries
tar -czf $ARCHIVE_DIR/crash-$TIMESTAMP.tar.gz \
  $CORE_FILE \
  /usr/sbin/asterisk \
  /usr/lib/asterisk/modules/*.so \
  /etc/asterisk/ \
  /var/log/asterisk/messages \
  2>/dev/null

# Generate backtrace for quick reference
gdb -batch -ex "bt" /usr/sbin/asterisk $CORE_FILE > \
  $ARCHIVE_DIR/crash-$TIMESTAMP.backtrace.txt 2>&1

echo "Core dump archived: $ARCHIVE_DIR/crash-$TIMESTAMP.tar.gz"
echo "Backtrace: $ARCHIVE_DIR/crash-$TIMESTAMP.backtrace.txt"
Enter fullscreen mode Exit fullscreen mode

Usage:

chmod +x /usr/local/bin/save_core_dump.sh
/usr/local/bin/save_core_dump.sh /var/spool/asterisk/core-asterisk-12345-1700731920
Enter fullscreen mode Exit fullscreen mode

Automated Crash Detection and Response

Create a Core Dump Watcher 

#!/bin/bash
# /usr/local/bin/asterisk-crash-monitor.sh

CORE_DIR="/var/spool/asterisk"
LOG_FILE="/var/log/asterisk/crash-monitor.log"
ALERT_EMAIL="admin@example.com"

check_for_new_cores() {
  find $CORE_DIR -name "core-*" -type f -mmin -5 2>/dev/null | while read core_file; do
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    # Extract backtrace
    backtrace=$(gdb -batch -ex "bt 30" /usr/sbin/asterisk "$core_file" 2>&1 | tail -50)

    # Log incident
    echo "[$timestamp] Crash detected: $core_file" >> $LOG_FILE
    echo "$backtrace" >> $LOG_FILE
    echo "---" >> $LOG_FILE

    # Send alert
    echo "$backtrace" | mail -s "Asterisk Crash: $(hostname)" $ALERT_EMAIL

    # Restart Asterisk if needed
    if ! pgrep -x asterisk > /dev/null; then
      systemctl start asterisk
      echo "[$timestamp] Asterisk restarted" >> $LOG_FILE
    fi
  done
}

# Run every 2 minutes
while true; do
  check_for_new_cores
  sleep 120
done
Enter fullscreen mode Exit fullscreen mode

Install as systemd service:

# /etc/systemd/system/asterisk-crash-monitor.service
[Unit]
Description=Asterisk Crash Monitor
After=network.target
Wants=asterisk.service

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/asterisk-crash-monitor.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Enable:

systemctl daemon-reload
systemctl enable asterisk-crash-monitor
systemctl start asterisk-crash-monitor
Enter fullscreen mode Exit fullscreen mode

GDB Automation Script

For unattended backtrace generation:

#!/bin/bash
# /usr/local/bin/extract_backtrace.sh

CORE_FILE=$1
BINARY="/usr/sbin/asterisk"
OUTPUT="${CORE_FILE}.backtrace.txt"

gdb -batch \
  -ex "thread apply all bt full" \
  -ex "quit" \
  $BINARY $CORE_FILE > $OUTPUT 2>&1

echo "Backtrace written to: $OUTPUT"
cat $OUTPUT
Enter fullscreen mode Exit fullscreen mode

Usage:

/usr/local/bin/extract_backtrace.sh /var/spool/asterisk/core-asterisk-12345-1700731920
Enter fullscreen mode Exit fullscreen mode

ViciDial-Specific Debugging Tips

Check for Lead Lock Deadlocks

ViciDial's lead distribution can deadlock when multiple processes compete for the same lead:

asterisk -rx "database show vicidial-lead-locks" | head -20
Enter fullscreen mode Exit fullscreen mode

If locks don't clear within seconds, investigate:

mysql> SELECT * FROM vicidial_log WHERE lead_id = <LEAD_ID> ORDER BY entry_date DESC LIMIT 1\G
Enter fullscreen mode Exit fullscreen mode

Clear stale locks manually (if safe):

asterisk -rx "database deltree vicidial-lead-locks"
Enter fullscreen mode Exit fullscreen mode

Monitor Call Bridge Issues

ViciDial bridges agent and carrier channels. Crashes often occur at bridge setup:

asterisk -rx "core set debug 3"
asterisk -rx "bridge show all"
Enter fullscreen mode Exit fullscreen mode

Check for stuck bridges:

asterisk -rx "channel show all" | wc -l
Enter fullscreen mode Exit fullscreen mode

If channel count grows unbounded, bridge leak exists. Restart may be necessary:

asterisk -rx "core restart graceful"
Enter fullscreen mode Exit fullscreen mode

Review Dialplan Syntax

Dialplan errors can trigger crashes in edge cases. Validate:

asterisk -rx "dialplan reload"
Enter fullscreen mode Exit fullscreen mode

Check for warnings in /var/log/asterisk/messages:

grep -i "warning\|error" /var/log/asterisk/messages | tail -50
Enter fullscreen mode Exit fullscreen mode

Inspect AGI Script Failures

Custom AGI scripts sometimes cause Asterisk to crash. Test scripts in isolation:

/var/lib/asterisk/agi-bin/vicidial_loader.agi test
Enter fullscreen mode Exit fullscreen mode

Add error handling to AGI scripts:

#!/bin/bash
trap 'echo "HANGUP" >&3' EXIT
trap 'exit' SIGTERM

while read -t 1 line; do
  case "$line" in
    agi_uniqueid:*) UNIQUEID="${line#*: }" ;;
    agi_calleridnum:*) CALLERID="${line#*: }" ;;
  esac
  [ -z "$line" ] && break
done

echo "EXEC Hangup" >&3
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

Issue: Core Dumps Not Being Generated

Check 1: ulimit

ulimit -c
Enter fullscreen mode Exit fullscreen mode

Should return unlimited. If not:

ulimit -c unlimited
echo "ulimit -c unlimited" >> /etc/profile
Enter fullscreen mode Exit fullscreen mode

Check 2: File System Space

df -h /var/spool/asterisk
Enter fullscreen mode Exit fullscreen mode

If full, clean old cores:

rm -f /var/spool/asterisk/core-*
Enter fullscreen mode Exit fullscreen mode

Check 3: SELinux

If enabled, SELinux may block core dumps:

getenforce
Enter fullscreen mode Exit fullscreen mode

If Enforcing, add policy:

semanage fcontext -a -t asterisk_var_spool_t "/var/spool/asterisk(/.*)?"
restorecon -Rv /var/spool/asterisk
Enter fullscreen mode Exit fullscreen mode

Issue: GDB Backtrace Shows "??" Instead of Function Names

Cause: Missing debug symbols.

Fix: Recompile Asterisk with -g flag:

./configure CFLAGS="-g -O2"
make -j4
make install
Enter fullscreen mode Exit fullscreen mode

Verify:

gdb /usr/sbin/asterisk -batch -ex "file /usr/sbin/asterisk" -ex "quit" 2>&1 | grep "debug_info"
Enter fullscreen mode Exit fullscreen mode

Issue: GDB Takes Too Long to Load Large Core Files

Workaround: Use gdb --nx to skip initialization:

gdb --nx /usr/sbin/asterisk /var/spool/asterisk/core-asterisk-12345-1700731920
Enter fullscreen mode Exit fullscreen mode

Or extract backtrace without full symbols:

gdb -batch -ex "bt" /usr/sbin/asterisk /var/spool/asterisk/core-asterisk-12345-1700731920 2>&1 | head -100
Enter fullscreen mode Exit fullscreen mode

Issue: Asterisk Keeps Crashing in Infinite Loop

Diagnosis: Check crash frequency:

ls -lh /var/spool/asterisk/core-* | wc -l
Enter fullscreen mode Exit fullscreen mode

Disable automatic restart temporarily:

systemctl stop asterisk
Enter fullscreen mode Exit fullscreen mode

Run Asterisk in foreground with debug:

asterisk -f -d -v -v -v
Enter fullscreen mode Exit fullscreen mode

Stop at first crash with Ctrl+C. Review console output for error messages.

Issue: Backtrace Points to Third-Party Module, But Module Isn't Loaded

Cause: Module was loaded in a previous process instance, shared library is corrupted.

Fix: Force reload all modules:

asterisk -rx "module reload FORCE"
Enter fullscreen mode Exit fullscreen mode

If crash persists:

asterisk -rx "module unload extra/codec_custom"
Enter fullscreen mode Exit fullscreen mode

Rebuild module:

cd /usr/src/asterisk-16.x.x/addons
make clean
./configure --with-asterisk=/usr
make
make install
Enter fullscreen mode Exit fullscreen mode

Summary

Debugging Asterisk core dumps in production ViciDial environments requires:

  1. Preparation: Enable core dumps at OS, kernel, and Asterisk levels. Compile with debug symbols.

  2. Capture: Monitor /var/spool/asterisk/ for new core files. Log crash timestamps and associated system state.

  3. Analysis: Use GDB to extract full backtraces with thread apply all bt full. Focus on the faulting thread. Inspect variable states at crash point.

  4. Pattern Recognition: Know common ViciDial crash patterns—SIP channel deallocation, database race conditions, memory leaks, module incompatibilities.

  5. ViciDial-Specific Checks: Verify lead locks aren't stuck, call bridges aren't leaking, dialplan is valid, AGI scripts have error handling.

  6. Response: Automate crash detection with systemd services. Generate backtrace archives for vendor support. Implement graceful restarts to minimize call loss.

  7. Remediation: Address root causes—patch Asterisk, rebuild incompatible modules, refactor dialplan, fix AGI scripts, update SIP peers.

Core dumps are your most powerful debugging tool in production. Invest in capturing and preserving them. A single backtrace can save weeks of troubleshooting and prevent recurring crashes that destroy agent productivity and customer experience.

Keep backtraces, logs, and configurations together when escalating to Asterisk support or ViciDial engineers. Always test fixes in staging before deploying to production.

Top comments (0)