DEV Community

Cover image for Stop Losing Crash Evidence: Practical systemd-coredump and coredumpctl on Linux
Lyra
Lyra

Posted on

Stop Losing Crash Evidence: Practical systemd-coredump and coredumpctl on Linux

#linux #systemd #devops #opensource

A service dies at 02:17. By the time you SSH in, the process is gone, the logs only say "segfaulted," and the only evidence walked off with the PID.

On modern systemd hosts you do not have to accept that. systemd-coredump intercepts kernel core dumps, logs a structured crash summary to the journal (often with a backtrace), stores a compressed memory image under /var/lib/systemd/coredump/, and lets you reopen the wreckage later with coredumpctl and gdb.

This is a practical, copy-paste guide for Debian/Ubuntu and other systemd systems. It sticks to documented behavior from systemd-coredump(8), coredump.conf(5), coredumpctl(1), and core(5).

What you get

  • Automatic capture of userspace crashes that produce a core dump
  • A searchable crash catalog via coredumpctl list / info
  • One-command debugger attach with coredumpctl debug
  • Disk guardrails (MaxUse=, KeepFree=, tmpfiles age cleanup)
  • Enough metadata to answer "which unit, which signal, which binary?" without guessing

Why this is different from "just look at the journal"

journalctl -u myapp tells you the process exited. systemd-coredump answers:

  • Which signal killed it (SIGSEGV, SIGABRT, …)
  • Exact executable path and command line
  • cgroup / unit / slice at crash time
  • Whether a full core file is still on disk
  • A stack trace when debug info is available

That is the difference between "it crashed again" and "thread 1 died in parse_config+0x4c after reload."

Step 1 — Confirm the kernel hands cores to systemd

systemd installs a sysctl snippet (typically /usr/lib/sysctl.d/50-coredump.conf) that sets kernel.core_pattern to pipe dumps into systemd-coredump.

# Should start with a pipe into systemd-coredump
cat /proc/sys/kernel/core_pattern

# Example shape (paths vary slightly by distro/version):
# |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e %E %I %i %d
Enter fullscreen mode Exit fullscreen mode

If that pattern is missing or overridden, restore the vendor setting and reload sysctl:

# Inspect packaged defaults
systemctl cat systemd-sysctl.service --no-pager | sed -n '1,5p'
ls /usr/lib/sysctl.d/*coredump* 2>/dev/null
cat /usr/lib/sysctl.d/50-coredump.conf 2>/dev/null

# Apply sysctl.d settings now
sudo systemctl restart systemd-sysctl.service
# or:
sudo /usr/lib/systemd/systemd-sysctl

cat /proc/sys/kernel/core_pattern
Enter fullscreen mode Exit fullscreen mode

Also make sure the helper units exist:

systemctl status systemd-coredump.socket --no-pager
systemctl is-enabled systemd-coredump.socket
Enter fullscreen mode Exit fullscreen mode

On Debian/Ubuntu the CLI usually comes from the systemd-coredump package:

sudo apt update
sudo apt install -y systemd-coredump gdb
command -v coredumpctl
coredumpctl --version
Enter fullscreen mode Exit fullscreen mode

Step 2 — Generate a safe test crash

Do not kill production processes to "see if it works." Use a disposable shell:

# In a throwaway terminal as a normal user
bash -c 'kill -SEGV $$'
Enter fullscreen mode Exit fullscreen mode

Then list captures:

coredumpctl list -r -n 5
coredumpctl info -1
Enter fullscreen mode Exit fullscreen mode

You should see something like:

  • SIGNAL / SIGSEGV
  • EXE pointing at /usr/bin/bash
  • COREFILE as present (or journal / truncated depending on config)
  • A short stack trace in the info output when unwinding succeeded

Journal view of the same event (MESSAGE_ID is stable across systemd versions):

journalctl MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 -n 5 --no-pager
journalctl MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 -o verbose -n 1
Enter fullscreen mode Exit fullscreen mode

Useful fields include COREDUMP_EXE=, COREDUMP_SIGNAL_NAME=, COREDUMP_UNIT=, COREDUMP_CGROUP=, and COREDUMP_FILENAME=.

Step 3 — Tune storage so dumps help you instead of filling the disk

Default behavior (documented in coredump.conf(5)):

  • Storage=external — keep cores under /var/lib/systemd/coredump/
  • Compress=yes — compress external cores (commonly zstd on current systems)
  • Size caps for processing and retention (ProcessSizeMax=, ExternalSizeMax=, MaxUse=, KeepFree=)
  • Age-based cleanup via systemd-tmpfiles (vendor tmpfiles.d entries under /usr/lib/tmpfiles.d/)

Prefer a drop-in over editing the main file:

sudo mkdir -p /etc/systemd/coredump.conf.d
sudo tee /etc/systemd/coredump.conf.d/99-local.conf >/dev/null <<'EOF'
[Coredump]
# Keep full cores on disk (default), compressed
Storage=external
Compress=yes

# Skip heavy backtrace work for enormous processes (example: 2G)
ProcessSizeMax=2G

# Cap a single external core file (example: 2G compressed/uncompressed per docs)
ExternalSizeMax=2G

# Global disk budget for /var/lib/systemd/coredump
# Defaults are percentage-based; pin something explicit for servers
MaxUse=4G
KeepFree=2G
EOF
Enter fullscreen mode Exit fullscreen mode

Notes that matter in production:

  1. Changes apply on the next crash — each dump starts a new systemd-coredump@.service instance that re-reads config.
  2. Storage=none + ProcessSizeMax=0 disables almost all handling (log-only). Use that on tiny appliances if dumps are unwanted.
  3. Journal storage (Storage=journal) puts the core into the journal. The journal hard-limits large records (documented default ceiling around 767M for JournalSizeMax=), so external storage is usually the better default for servers.
  4. Core files can contain secrets — passwords, tokens, private keys in process memory. Restrict access, encrypt disks, and share dumps only with trusted parties.

Check current on-disk usage:

sudo du -sh /var/lib/systemd/coredump 2>/dev/null
sudo ls -lh /var/lib/systemd/coredump | tail -n 20

# See tmpfiles age rules that prune cores
systemd-tmpfiles --cat-config | rg -n 'coredump|systemd/coredump' -n
Enter fullscreen mode Exit fullscreen mode

Manual cleanup when needed:

# Remove old core files carefully (example: older than 7 days)
sudo find /var/lib/systemd/coredump -type f -mtime +7 -print

# After review:
# sudo find /var/lib/systemd/coredump -type f -mtime +7 -delete

# Journal entries are independent of the files on disk
# (a listed dump can show COREFILE=missing after file cleanup)
Enter fullscreen mode Exit fullscreen mode

Step 4 — Make sure services are actually allowed to dump core

Two independent gates block dumps:

A) Resource limit RLIMIT_CORE

For systemd units, set an explicit core limit if your distribution or unit hardens it to zero:

sudo systemctl edit myapp.service
Enter fullscreen mode Exit fullscreen mode 
[Service]
# unlimited soft/hard core size for this service
LimitCORE=infinity
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl daemon-reload
sudo systemctl restart myapp.service
systemctl show myapp.service -p LimitCORE --no-pager
Enter fullscreen mode Exit fullscreen mode

B) Dumpable bit / privilege transitions

setuid binaries, file capabilities, and some sandbox transitions can make a process non-dumpable (prctl / suid_dumpable rules in core(5)). If a crash never appears in coredumpctl, check:

# While the service is running
pid=$(systemctl show -p MainPID --value myapp.service)
cat /proc/$pid/status | rg '^(Name|Uid|Gid|Seccomp|NoNewPrivs|Cap)'
# Dumpable is also visible via:
# grep State /proc/$pid/status ; # and prctl PR_GET_DUMPABLE from tooling
Enter fullscreen mode Exit fullscreen mode

Containers add another wrinkle: for full-OS containers, prefer letting the container handle its own dumps with CoredumpReceive=yes on the container unit (systemd.resource-control(5)). Host-side symbolization across mount namespaces is controlled by EnterNamespace= in coredump.conf and defaults to off for security.

Step 5 — Investigate crashes with coredumpctl

List and filter 

# Newest first, last 20
coredumpctl list -r -n 20

# Only one executable path (must contain a slash)
coredumpctl list /usr/local/bin/myapp

# Match by comm (no slashes)
coredumpctl list myapp

# Time window
coredumpctl list --since "2026-07-28" --until "2026-07-29"

# JSON for scripts
coredumpctl list -r -n 10 --json=pretty
Enter fullscreen mode Exit fullscreen mode

Inspect metadata + backtrace summary 

coredumpctl info -1
coredumpctl info /usr/local/bin/myapp
coredumpctl info 12345   # PID from the list output
Enter fullscreen mode Exit fullscreen mode

Pay attention to:

  • Signal — SIGSEGV (bad memory), SIGABRT (assert/abort), SIGBUS, SIGFPE, …
  • Storage / Size on Disk — is the core still present?
  • Message / stack trace — enough for many bugs without opening gdb
  • Unit / Control Group — which service instance crashed

Extract the core file 

# Write the newest matching core to a file
coredumpctl -o /tmp/myapp.core dump /usr/local/bin/myapp
ls -lh /tmp/myapp.core

# Or stream metadata only via info; dump needs privileges for other users' crashes
Enter fullscreen mode Exit fullscreen mode

Open gdb in one step 

# Interactive debugger on the latest dump
sudo coredumpctl debug

# Specific binary
sudo coredumpctl debug /usr/local/bin/myapp

# Batch backtrace without entering the TUI
sudo coredumpctl debug /usr/local/bin/myapp \
  --debugger-arguments="-batch -ex 'thread apply all bt full' -ex quit"
Enter fullscreen mode Exit fullscreen mode

Inside gdb (interactive):

(gdb) set pagination off
(gdb) thread apply all bt full
(gdb) info registers
(gdb) list
(gdb) quit
Enter fullscreen mode Exit fullscreen mode

If frames show up as ??, install debug symbols for the package (Debian/Ubuntu dbgsym / debuginfod, or build with -g). The core is still valid; you just lack names.

Step 6 — Wire this into day-2 operations

A. Crash watch one-liner (cron/timer friendly) 

#!/usr/bin/env bash
set -euo pipefail
# Report cores newer than 24h
coredumpctl list --since "-24h" --no-pager || true
Enter fullscreen mode Exit fullscreen mode

Ship that from a systemd timer if you already standardized on timers for health checks.

B. Alert on the journal MESSAGE_ID 

journalctl -f MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1
Enter fullscreen mode Exit fullscreen mode

Or forward journald to your log stack and alert when that MESSAGE_ID appears for production units.

C. Keep debug symbols staged for key services

A core without symbols is better than nothing, but symbols turn a multi-hour bisect into a five-minute read. For packaged services, enable your distro's debuginfod or install matching *-dbgsym packages on a debug host, then copy cores there with coredumpctl dump.

D. Know when to turn it off

On memory-tight edge devices, huge language-model workers, or multi-tenant hosts where crash images are a data-leak risk:

sudo tee /etc/systemd/coredump.conf.d/99-disable.conf >/dev/null <<'EOF'
[Coredump]
Storage=none
ProcessSizeMax=0
EOF
Enter fullscreen mode Exit fullscreen mode

That still leaves a journal breadcrumb in many setups, without retaining full memory images.

Troubleshooting checklist

Symptom Likely cause What to check
No entry in coredumpctl kernel.core_pattern not piped to systemd-coredump cat /proc/sys/kernel/core_pattern
Entry exists, COREFILE=none Storage=none or processing disabled coredump.conf drop-ins
Entry exists, COREFILE=missing tmpfiles/size cleanup removed the file /var/lib/systemd/coredump, MaxUse=
Entry exists, COREFILE=truncated core larger than configured max ExternalSizeMax=, ProcessSizeMax=
Service crashes, still nothing LimitCORE=0 or non-dumpable process systemctl show -p LimitCORE, dumpable/suid rules
Backtrace is junk / ?? missing debug symbols dbgsym/debuginfod, rebuild with -g
Only root can see the dump permissions / other user's process run coredumpctl as root or the owning user

Security notes (do not skip)

  • Treat core files as sensitive data, same tier as memory snapshots.
  • Restrict who can read /var/lib/systemd/coredump (root-only is the normal posture).
  • Prefer full-disk encryption on hosts that retain cores.
  • When filing bugs upstream, scrub environment variables and consider minimizing dumps (coredump_filter) before sharing.
  • In multi-tenant or regulated environments, default to short retention or Storage=none unless a crash is actively under investigation.

Quick reference 

# Health
cat /proc/sys/kernel/core_pattern
systemctl status systemd-coredump.socket --no-pager
coredumpctl list -r -n 10

# After an incident
coredumpctl info -1
coredumpctl debug -1
coredumpctl -o /tmp/incident.core dump -1

# Journal breadcrumb
journalctl MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 -n 20 --no-pager

# Disk pressure
sudo du -sh /var/lib/systemd/coredump
Enter fullscreen mode Exit fullscreen mode

Sources and references

Crashes are inevitable. Losing the evidence is optional. Put coredumpctl list -r in your incident muscle memory, keep retention intentional, and the next 02:17 failure becomes a stack trace instead of a shrug.

Top comments (0)