DEV Community

Cover image for Stop Losing Full Kernel Crash Dumps: Practical kdump and kexec on Linux
Lyra
Lyra

Posted on

Stop Losing Full Kernel Crash Dumps: Practical kdump and kexec on Linux

#linux #devops #opensource #kernel

Stop Losing Full Kernel Crash Dumps: Practical kdump and kexec on Linux

When a Linux host panics, you usually get one of three outcomes:

  1. A short oops snippet in the journal or pstore
  2. A userspace core from a crashed daemon
  3. A reboot and no useful memory image

The third case is the expensive one. Full kernel crash dumps exist for a reason: kdump boots a small capture kernel via kexec, preserves the panicked kernel’s memory, and writes it out as /proc/vmcore so you can keep a filtered vmcore for later analysis.

This is not a replacement for systemd-pstore or systemd-coredump. Those tools answer different questions. kdump is what you want when you need a real memory image of the crashed kernel.

What kdump actually does

From the kernel’s kdump documentation:

  1. At normal boot, the production kernel reserves memory with crashkernel=...
  2. Userspace loads a dump-capture kernel into that reserved region with kexec -p
  3. On panic (or SysRq crash), the machine jumps into the capture kernel without a full firmware reboot
  4. The capture kernel exposes the old memory image as /proc/vmcore
  5. Tools like cp, scp, or makedumpfile write the dump to disk or over the network
  6. The host reboots back into the normal kernel

That design matters. Dumping from a freshly booted capture kernel is far more reliable than trying to write files from a kernel that already panicked.

Where this sits next to pstore and coredump

Use the right tool for the evidence you need:

Layer Tool What you get Typical size
Userspace crash systemd-coredump / coredumpctl Process core + journal metadata MB–GB per process
Kernel oops/panic breadcrumbs pstore + systemd-pstore dmesg/oops fragments that survive reboot KB–low MB
Full kernel memory image kdump / kexec Filtered or full vmcore Hundreds of MB–many GB

If you only need “what was the last kernel message?”, pstore is enough.

If you need “why did the kernel die, and what was in memory?”, you want kdump.

Prerequisites

You need:

  • A kernel built with kexec + crash dump support (CONFIG_KEXEC / CONFIG_KEXEC_FILE, CONFIG_CRASH_DUMP, CONFIG_PROC_VMCORE)
  • Reserved crash memory via crashkernel=
  • Userspace packaging:
    • Debian/Ubuntu: kdump-tools, kexec-tools, makedumpfile
    • Fedora/RHEL-family: kexec-tools (provides kdumpctl / kdump.service), optionally crash + matching kernel-debuginfo for analysis
  • Disk space under /var/crash (or a remote NFS/SSH target)
  • A maintenance window for one reboot after setting crashkernel=

Most distribution kernels already ship the needed config options. The missing pieces are usually reserved memory and the dump service.

Path A: Debian / Ubuntu with kdump-tools

1. Install packages 

sudo apt update
sudo apt install kdump-tools kexec-tools makedumpfile
Enter fullscreen mode Exit fullscreen mode

On Debian/Ubuntu, the main config file is:

/etc/default/kdump-tools
Enter fullscreen mode Exit fullscreen mode

2. Enable kdump-tools

Edit /etc/default/kdump-tools and set:

USE_KDUMP=1
Enter fullscreen mode Exit fullscreen mode

Useful defaults from the package man page:

# Where dumps land (default if unset)
KDUMP_COREDIR="/var/crash"

# makedumpfile args (package default if unset)
# -c compress, -d 31 keep only useful kernel pages
MAKEDUMP_ARGS="-c -d 31"

# Keep only N local dumps after a successful capture
KDUMP_NUM_DUMPS=5

# Also save dmesg content from the panicked kernel
KDUMP_DUMP_DMESG=1
Enter fullscreen mode Exit fullscreen mode

MAKEDUMP_ARGS="-c -d 31" is the practical default on Debian-family systems. Dump level 31 excludes zero pages, cache pages, user pages, and free pages, leaving the kernel-relevant content that crash analysis usually needs.

3. Reserve crashkernel memory

kdump cannot work without reserved memory. Add a crashkernel= argument to the bootloader.

On GRUB systems:

sudo sed -n 's/^GRUB_CMDLINE_LINUX_DEFAULT=//p' /etc/default/grub
Enter fullscreen mode Exit fullscreen mode

Add something like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash crashkernel=256M"
Enter fullscreen mode Exit fullscreen mode

Then:

sudo update-grub
sudo reboot
Enter fullscreen mode Exit fullscreen mode

Sizing notes from upstream and distro docs:

  • Simple fixed reserve: crashkernel=256M or crashkernel=512M
  • Auto placement: crashkernel=256M (omit @offset unless you know you need it)
  • RAM-dependent syntax from the kernel docs: 
crashkernel=512M-2G:64M,2G-:128M
Enter fullscreen mode Exit fullscreen mode

That means:

  • under 512M RAM: reserve nothing
  • 512M–2G: reserve 64M
  • 2G and above: reserve 128M

For modern servers with lots of drivers/modules in the capture path, start higher than the absolute minimum. Many operators begin around 256M–512M on x86_64 and raise it if load/capture fails.

After reboot, confirm the cmdline:

cat /proc/cmdline
# expect crashkernel=...
Enter fullscreen mode Exit fullscreen mode

4. Load and verify the capture kernel 

sudo kdump-config show
sudo kdump-config test
sudo kdump-config status
sudo kdump-config load
sudo kdump-config status
Enter fullscreen mode Exit fullscreen mode

What these do:

  • show — current parameters and last saved kexec command
  • test — compute load parameters without loading
  • status — check /sys/kernel/kexec_crash_loaded
  • load — actually load the dump-capture kernel with kexec

A healthy loaded state looks like:

cat /sys/kernel/kexec_crash_loaded
# 1
Enter fullscreen mode Exit fullscreen mode

If the sysfs file is missing, the running kernel was built without crash dump support or booted without a usable crashkernel= reservation.

5. Optional remote dump targets

Still in /etc/default/kdump-tools, choose one of SSH or NFS (not both):

# SSH example
SSH="kdump@crash-collector.example.net"
SSH_KEY="/root/.ssh/kdump_id_rsa"

# or NFS example
# NFS="crash-collector.example.net:/var/crash"
Enter fullscreen mode Exit fullscreen mode

For SSH key distribution:

sudo kdump-config propagate
Enter fullscreen mode Exit fullscreen mode

That pushes the public key so the capture path can write without an interactive password.

6. Sysctl panic triggers worth enabling

kdump-tools already leans toward collecting dumps on oops. For production hosts, also decide whether OOM and NMI should panic:

# /etc/sysctl.d/99-kdump-triggers.conf
kernel.panic_on_oops = 1
vm.panic_on_oom = 1
kernel.hardlockup_panic = 1
kernel.softlockup_panic = 1
kernel.panic = 60
Enter fullscreen mode Exit fullscreen mode

Apply:

sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

kernel.panic = 60 reboots sixty seconds after panic if dump capture does not complete cleanly. Tune to your recovery model.

kdump-specific sysctl overrides for the capture environment live in:

/etc/kdump/sysctl.conf
Enter fullscreen mode Exit fullscreen mode

Keep hugepage overrides there unless you have a measured reason to remove them. Capture kernels are memory-constrained.

Path B: Fedora / RHEL-family with kexec-tools

1. Install and enable 

sudo dnf install kexec-tools
sudo systemctl enable --now kdump.service
Enter fullscreen mode Exit fullscreen mode

For later analysis on the same machine:

sudo dnf install crash
# plus matching kernel-debuginfo from debuginfo repos when you actually need to inspect a dump
Enter fullscreen mode Exit fullscreen mode

2. Set crashkernel reservation

On current Fedora-style tooling:

sudo kdumpctl reset-crashkernel
sudo reboot
Enter fullscreen mode Exit fullscreen mode

kdumpctl reset-crashkernel writes a range-based reservation into the current kernel’s boot entry. After reboot:

cat /proc/cmdline
systemctl status kdump.service --no-pager
sudo kdumpctl status
Enter fullscreen mode Exit fullscreen mode

3. Configure dump location and filtering

Main config:

/etc/kdump.conf
Enter fullscreen mode Exit fullscreen mode

Common practical settings:

path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31
default reboot
Enter fullscreen mode Exit fullscreen mode

Notes:

  • -d 31 is the same “kernel-useful pages only” filter used on Debian
  • -l selects LZO compression when built into makedumpfile
  • default reboot returns the host to normal operation after capture

Network targets are also supported in this file (NFS/SSH style destinations). Keep credentials and mount permissions boring and explicit; the capture environment is intentionally minimal.

Make sure the dump will fit

Before you ever trigger a test crash:

df -h /var/crash
sudo mkdir -p /var/crash
sudo chmod 700 /var/crash
Enter fullscreen mode Exit fullscreen mode

Rough planning:

  • Unfiltered vmcore can approach system RAM size
  • Filtered dumps with -d 31 are much smaller, but still large on big hosts
  • Keep retention low (KDUMP_NUM_DUMPS or external cleanup)
  • Prefer a dedicated volume or remote collector for fleets

If local disk is tight, remote dump is not optional — it is the design.

Lab verification (safe procedure)

Only do this on a lab host or during a planned window. Triggering kdump will interrupt workloads.

1. Confirm loaded state

Debian/Ubuntu:

sudo kdump-config status
cat /sys/kernel/kexec_crash_loaded
Enter fullscreen mode Exit fullscreen mode

Fedora/RHEL-family:

systemctl is-active kdump.service
cat /sys/kernel/kexec_crash_loaded
Enter fullscreen mode Exit fullscreen mode

2. Prefer a text console for the test

Framebuffer/GUI consoles can look frozen or corrupted during capture even when kdump is working. Serial console or a plain multi-user text session is easier to interpret.

3. Trigger a controlled crash 

# requires SysRq enabled
echo 1 | sudo tee /proc/sys/kernel/sysrq
echo c | sudo tee /proc/sysrq-trigger
Enter fullscreen mode Exit fullscreen mode

Upstream documents the same triggers: ALT-SysRq-c, echo c > /proc/sysrq-trigger, or a panic-inducing test module.

4. After reboot, find the dump 

sudo find /var/crash -type f -mtime -1 -ls
Enter fullscreen mode Exit fullscreen mode

Typical layout:

/var/crash/<timestamp>/vmcore
/var/crash/<timestamp>/dmesg   # when dmesg capture is enabled
Enter fullscreen mode Exit fullscreen mode

Also check:

sudo ls -lah /var/crash
journalctl -b -1 -u kdump* --no-pager
Enter fullscreen mode Exit fullscreen mode

If there is no dump:

  • crashkernel= missing or too small
  • capture kernel/initrd not loaded (kexec_crash_loaded != 1)
  • /var/crash full or not writable from capture environment
  • remote SSH/NFS path unreachable during capture
  • filtering/compression tool failed and fallback also failed

Analyze a vmcore without folklore

Quick dmesg extraction

If you only need the panicked kernel log:

sudo makedumpfile --dump-dmesg /var/crash/<timestamp>/vmcore /tmp/panic.dmesg
less /tmp/panic.dmesg
Enter fullscreen mode Exit fullscreen mode

crash utility (preferred for kdump-compressed dumps) 

# vmlinux with debug symbols must match the crashed kernel
crash /usr/lib/debug/boot/vmlinux-$(uname -r) \
  /var/crash/<timestamp>/vmcore
Enter fullscreen mode Exit fullscreen mode

Inside crash, start with:

sys
bt
log
ps
kmem -i
quit
Enter fullscreen mode Exit fullscreen mode

Package paths differ by distro:

  • Debian/Ubuntu debug kernels often come from linux-image-$(uname -r)-dbg
  • Fedora/RHEL-family use kernel-debuginfo

gdb on ELF dumps

If you generated an ELF dump (makedumpfile -E ...), gdb can do limited inspection:

gdb vmlinux /var/crash/<timestamp>/vmcore
Enter fullscreen mode Exit fullscreen mode

For normal compressed kdump files, use crash, not gdb.

makedumpfile dump levels that matter in practice

From makedumpfile(8), dump level is a bitfield:

Bit Excludes
1 zero pages
2 non-private cache
4 all cache
8 user data pages
16 free pages

So:

  • -d 1 — tiny reduction, still huge
  • -d 31 — exclude all of the above (common production default)
  • fallback patterns like -d 11,31 retry a more aggressive filter if space runs out

Examples:

# local filtered compressed dump
sudo makedumpfile -c -d 31 /proc/vmcore /var/crash/manual/vmcore

# ELF dump for gdb-oriented workflows
sudo makedumpfile -E -d 31 /proc/vmcore /var/crash/manual/vmcore.elf
Enter fullscreen mode Exit fullscreen mode

During a real capture you usually do not run these by hand; the distro dump service runs them. Knowing the flags still helps when a capture fails and you need to adjust MAKEDUMP_ARGS or core_collector.

Operational checklist for real hosts

  1. Reserve memory once, verify forever
    • grep -o 'crashkernel=[^ ]*' /proc/cmdline
    • alert if missing after kernel/bootloader changes
  2. Confirm loaded state after every reboot
    • cat /sys/kernel/kexec_crash_loaded must be 1
  3. Watch dump storage
    • capacity on /var/crash or collector
    • retention (KDUMP_NUM_DUMPS or remote lifecycle policy)
  4. Keep debug symbols available offline
    • you do not need debuginfo on every production box
    • you do need matching debug symbols on the analysis workstation
  5. Test after major kernel upgrades
    • new kernel, new modules, new Secure Boot state, new root layout
  6. Document the blast radius
    • kdump test is a hard crash
    • never “just test it” on a primary database node mid-day

Secure Boot and signed kernels

If the host uses UEFI Secure Boot, the capture path still has to satisfy your boot trust policy. Distro-signed kernels usually continue to work. Custom/self-managed signing setups need the kexec capture kernel and related bits to remain trusted after upgrades.

kdump itself does not replace Secure Boot policy. It depends on it when Secure Boot is enabled.

Common failure modes

kexec_crash_loaded stays 0

  • no crashkernel= in cmdline
  • reservation failed (too large / fragmented / unsupported placement)
  • dump service disabled
  • broken kdump initrd/symlink after a kernel install

Capture kernel boots, no vmcore written

  • target filesystem full
  • wrong KDUMP_COREDIR / path
  • NFS export permissions
  • SSH key not authorized for the capture user
  • makedumpfile failed and fallback cp also failed

Dump exists but analysis is useless

  • missing matching debug symbols
  • over-filtered dump for the question you need
  • analyzed with the wrong vmlinux version

Host appears wedged after panic

  • capture is still writing a large dump
  • console is graphical and not redrawing
  • crashkernel too small, capture kernel OOM’d

Patience plus serial console saves a lot of false conclusions here.

Minimal production baseline

If you want a boring, supportable setup on a Debian/Ubuntu server:

sudo apt install kdump-tools kexec-tools makedumpfile

sudo tee -a /etc/default/kdump-tools >/dev/null <<'EOF'
USE_KDUMP=1
KDUMP_COREDIR="/var/crash"
MAKEDUMP_ARGS="-c -d 31"
KDUMP_NUM_DUMPS=5
KDUMP_DUMP_DMESG=1
EOF

# add crashkernel=256M (or larger) to GRUB_CMDLINE_LINUX_DEFAULT, then:
sudo update-grub
sudo reboot

# after reboot
sudo kdump-config load
sudo kdump-config status
mkdir -p /var/crash
chmod 700 /var/crash
Enter fullscreen mode Exit fullscreen mode

Then schedule one controlled lab crash before you trust it in production.

Boundaries and honesty

kdump is excellent at preserving kernel crash evidence. It is not:

  • a substitute for good HA and health checks
  • a full forensics platform by itself
  • free — reserved crash memory is RAM you cannot use for workloads
  • guaranteed on every broken machine path (firmware hangs, early-boot failures before kexec load, storage dead ends)

Pair it with:

  • systemd-pstore for small panic breadcrumbs
  • systemd-coredump for userspace crashes
  • watchdogs for hard hangs that never reach a clean panic path
  • remote logging so you still know a host died if dump capture fails

References

Full vmcore capture is one of those systems features you either set up before the outage, or wish you had set up during it. Reserve the memory, load the capture kernel, verify kexec_crash_loaded, and do one planned test. Future-you gets evidence instead of a shrug.

Top comments (0)