DEV Community

Cover image for Stop Losing Kernel Panic Evidence: Practical systemd-pstore on Linux
Lyra
Lyra

Posted on

Stop Losing Kernel Panic Evidence: Practical systemd-pstore on Linux

#linux #systemd #devops #opensource

A userspace segfault leaves a process corpse. A kernel panic leaves a reboot — and often nothing useful in the journal.

If the box hard-resets, freezes, or trips a watchdog, the last useful lines of dmesg died with the previous boot. That is exactly what Linux pstore is for: a tiny persistent store (UEFI variables, ACPI ERST, or reserved RAM via ramoops) that can hold the tail of the kernel log across reboot. systemd-pstore then archives those records into /var/lib/systemd/pstore/ and the journal on the next boot, and clears the small backend so the next panic still has room.

This is not kdump. It will not give you a full vmcore. It will give you the panic stack and the last kernel messages after a bad reboot — with almost no infrastructure.

This guide sticks to documented behavior from systemd-pstore.service(8), pstore.conf(5), the vendor tmpfiles.d snippet, and the kernel ramoops docs.

What you get

  • Automatic archival of /sys/fs/pstore records after reboot
  • Files under /var/lib/systemd/pstore/ plus matching journal entries
  • A nearly empty pstore backend after each boot (Unlink=yes), so the next crash can write again
  • Optional kernel knobs so panics (and, if you want, clean shutdowns) actually dump kmsg into pstore
  • A path that still works when network-backed kdump cannot

How the pieces fit 

kernel panic / oops
        |
        v
 pstore backend (efi_pstore / ERST / ramoops / ...)
        |
        |  survives reboot
        v
 /sys/fs/pstore/   (dmesg-*, console-*, etc.)
        |
        |  early boot: systemd-pstore.service
        v
 /var/lib/systemd/pstore/  +  journal
        |
        v
 backend unlinked (freed) for the next event
Enter fullscreen mode Exit fullscreen mode

Important boundaries:

Tool Captures Needs
systemd-coredump / coredumpctl Userspace crashes Process dumpable, core pattern
pstore + systemd-pstore Kernel log tail / oops / panic notes Working pstore backend
kdump / kexec Full kernel crash dump (vmcore) Reserved memory, dump target, often network

Use pstore for "what did the kernel scream before it died?" Use kdump when you need a full memory image. They are complementary; the systemd man page explicitly calls them independent.

Step 1 — Confirm pstore is present 

# Filesystem should already be mounted on modern kernels
mount | grep pstore
# pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)

ls -la /sys/fs/pstore
# Empty on a healthy box that has already archived previous records

# Is the oneshot unit installed?
systemctl cat systemd-pstore.service --no-pager | sed -n '1,40p'
systemctl status systemd-pstore.service --no-pager || true
systemctl is-enabled systemd-pstore.service || true
Enter fullscreen mode Exit fullscreen mode

On current systemd (example: Debian trixie / systemd 257), the unit looks like this:

  • ConditionDirectoryNotEmpty=/sys/fs/pstore — runs only when there is something to archive
  • ConditionVirtualization=!container — skipped inside containers
  • After= / Wants=modprobe@efi_pstore.service — tries to load the EFI backend
  • Type=oneshot with ExecStart=/usr/lib/systemd/systemd-pstore
  • StateDirectory=systemd/pstore → /var/lib/systemd/pstore
  • WantedBy=sysinit.target — early boot archival

Enable it if it is not already:

sudo systemctl enable systemd-pstore.service
# It will start on the next boot when /sys/fs/pstore is non-empty
Enter fullscreen mode Exit fullscreen mode

Step 2 — Configure archival policy (drop-in)

Prefer a drop-in over editing /etc/systemd/pstore.conf directly.

sudo mkdir -p /etc/systemd/pstore.conf.d
sudo tee /etc/systemd/pstore.conf.d/99-local.conf >/dev/null <<'EOF'
[PStore]
# external (default): copy into /var/lib/systemd/pstore/ AND log to the journal
# journal: journal only
# none: do nothing (leave records sitting in the tiny backend)
Storage=external

# Remove records from /sys/fs/pstore after a successful archive.
# Default is yes — keep it yes so the backend has room for the next panic.
Unlink=yes
EOF

# Inspect the merged config
systemd-analyze cat-config systemd/pstore.conf
Enter fullscreen mode Exit fullscreen mode

From pstore.conf(5):

  • Storage=external (default) → archive under /var/lib/systemd/pstore/ and log to the journal
  • Storage=journal → journal only
  • Storage=none → exit without processing
  • Unlink=yes (default) → delete from pstore after archival so the small backend is free again

Vendor tmpfiles also keeps the on-disk archive for 14 days:

# From /usr/lib/tmpfiles.d/systemd-pstore.conf
# d /var/lib/systemd/pstore 0755 root root 14d
systemd-tmpfiles --cat-config | grep -n 'systemd/pstore' || true
sudo ls -la /var/lib/systemd/pstore
Enter fullscreen mode Exit fullscreen mode

Step 3 — Make the kernel actually write into pstore

Having systemd-pstore enabled is not enough if the kernel never dumps kmsg into the backend.

systemd ships commented knobs in /usr/lib/tmpfiles.d/systemd-pstore.conf:

Kernel parameter Effect (when set to Y)
/sys/module/kernel/parameters/crash_kexec_post_notifiers Store dmesg (including stack) into pstore on panic even when kdump is loaded
/sys/module/printk/parameters/always_kmsg_dump Also store dmesg on normal shutdown/reboot/halt

For panic forensics on servers, enable the first. Only enable the second if you deliberately want a kmsg snapshot on every clean reboot (it burns pstore capacity for non-failures).

# Do not edit the vendor file under /usr. Override with a higher-priority tmpfiles drop-in.
sudo tee /etc/tmpfiles.d/systemd-pstore.conf >/dev/null <<'EOF'
# Directory + retention (same as vendor default)
d /var/lib/systemd/pstore 0755 root root 14d

# Write panic/oops kmsg into pstore even if kdump is present.
# Note from vendor comments: this can slightly raise kdump failure risk.
w- /sys/module/kernel/parameters/crash_kexec_post_notifiers - - - - Y

# Optional — usually leave commented on production servers:
# w- /sys/module/printk/parameters/always_kmsg_dump - - - - Y
EOF

# Apply now (also applied automatically on future boots)
sudo systemd-tmpfiles --create /etc/tmpfiles.d/systemd-pstore.conf

# Verify
cat /sys/module/kernel/parameters/crash_kexec_post_notifiers
# expect: Y
Enter fullscreen mode Exit fullscreen mode

You can also set the panic-side behavior on the kernel command line where supported (distro/bootloader dependent), but the tmpfiles approach matches what systemd documents for these two module parameters.

Step 4 — Know which backend you have

pstore is a front-end. Backends differ by platform:

# Common on UEFI machines
lsmod | grep -E 'efi_pstore|erst|ramoops' || true
journalctl -k --no-pager | grep -iE 'pstore|efi_pstore|erst|ramoops' | tail -n 20

# Raw records appear here only until systemd-pstore archives + unlinks them
ls -la /sys/fs/pstore
Enter fullscreen mode Exit fullscreen mode

EFI / ACPI ERST (typical servers and laptops)

  • Often small (tens of KiB)
  • Survives reboot without reserved RAM tricks
  • systemd-pstore.service already Wants=modprobe@efi_pstore.service

ramoops (reserved RAM)

Useful on boards/VMs where you control memory layout. Kernel docs show patterns like:

# Example only — pick an address/size valid for YOUR platform
mem=128M ramoops.mem_address=0x8000000 ramoops.mem_size=0x100000 ramoops.ecc=1
Enter fullscreen mode Exit fullscreen mode

Or Device Tree compatible = "ramoops" reserved-memory nodes. Records show up as dmesg-ramoops-N under pstore. Do not copy an address from a blog into production without checking your firmware memory map — wrong reservations corrupt RAM or simply never persist.

Step 5 — Inspect archives after a bad reboot

After the machine comes back:

# On-disk archive tree (Storage=external)
sudo find /var/lib/systemd/pstore -type f | head -n 50
sudo ls -laR /var/lib/systemd/pstore | head -n 80

# Journal view of what systemd-pstore ingested
journalctl -u systemd-pstore.service --no-pager -b
journalctl --no-pager -b | grep -i pstore | tail -n 40

# If something is still sitting in the live backend (Unlink=no, or service did not run)
ls -la /sys/fs/pstore
sudo sh -c 'for f in /sys/fs/pstore/*; do echo "===== $f ====="; sed -n "1,80p" "$f"; done'
Enter fullscreen mode Exit fullscreen mode

What you usually care about inside a record:

  • Panic / oops banner
  • RIP / PC and tainted flags
  • Backtrace of the offending task
  • Last driver/filesystem messages before death

Treat these files like crash dumps: they can include paths, memory contents from logs, and environment details. Root-only permissions on /var/lib/systemd/pstore are the normal posture.

Step 6 — Safe functional checks (no production panic required)

A. Unit dry path 

# Confirm the binary and merged config
command -v /usr/lib/systemd/systemd-pstore || ls -l /usr/lib/systemd/systemd-pstore
systemd-analyze cat-config systemd/pstore.conf
systemctl cat systemd-pstore.service --no-pager

# Manual start is a no-op when pstore is empty (ConditionDirectoryNotEmpty)
sudo systemctl start systemd-pstore.service
systemctl status systemd-pstore.service --no-pager || true
Enter fullscreen mode Exit fullscreen mode

B. Synthetic pstore file (non-destructive archive path test)

Only do this on a lab host. This tests archival, not the kernel panic path:

# Lab only: drop a fake record so the oneshot condition becomes true
echo "lab pstore archival test $(date -Is)" | sudo tee /sys/fs/pstore/dmesg-lab-test >/dev/null 2>&1 || {
  echo "Backend rejected create (common on read-only or size-limited backends)."
  echo "Skip synthetic create; rely on a lab panic test below if needed."
}

ls -la /sys/fs/pstore
sudo systemctl start systemd-pstore.service
sudo ls -laR /var/lib/systemd/pstore | tail -n 40
ls -la /sys/fs/pstore   # should be empty again if Unlink=yes and archive succeeded
Enter fullscreen mode Exit fullscreen mode

Some backends only accept writes from the kernel. If create fails, that is expected — move on.

C. Lab panic (isolated VM only)

Never run this on a host you care about mid-flight:

# Isolated lab VM only
# 1) Ensure crash_kexec_post_notifiers=Y (Step 3)
# 2) Snapshot the VM
# 3) Trigger a panic:
# echo c | sudo tee /proc/sysrq-trigger
# 4) After reboot:
sudo ls -laR /var/lib/systemd/pstore
journalctl -u systemd-pstore.service -b --no-pager
Enter fullscreen mode Exit fullscreen mode

Enable SysRq first if your distro disables it (kernel.sysrq). Prefer a disposable VM over any shared hypervisor node.

Step 7 — Day-2 operations

Boot-time checklist script 

#!/usr/bin/env bash
# /usr/local/sbin/check-pstore.sh
set -euo pipefail

echo "== pstore mount =="
mount | grep ' type pstore' || echo "pstore not mounted"

echo "== live backend =="
ls -la /sys/fs/pstore 2>/dev/null || true

echo "== kernel dump knobs =="
echo -n "crash_kexec_post_notifiers="; cat /sys/module/kernel/parameters/crash_kexec_post_notifiers 2>/dev/null || echo missing
echo -n "always_kmsg_dump="; cat /sys/module/printk/parameters/always_kmsg_dump 2>/dev/null || echo missing

echo "== unit =="
systemctl is-enabled systemd-pstore.service 2>/dev/null || true
systemctl is-active systemd-pstore.service 2>/dev/null || true

echo "== on-disk archives (newest first) =="
if [[ -d /var/lib/systemd/pstore ]]; then
  sudo find /var/lib/systemd/pstore -type f -printf '%T@ %p\n' 2>/dev/null \
    | sort -nr | head -n 20 | while read -r _ts path; do ls -l "$path"; done
else
  echo "no /var/lib/systemd/pstore yet"
fi
Enter fullscreen mode Exit fullscreen mode

Wire it into a weekly timer if you already standardize health checks that way.

Alert when a new archive appears 

# Example: files newer than 24h under the archive tree
sudo find /var/lib/systemd/pstore -type f -mtime -1 -print
Enter fullscreen mode Exit fullscreen mode

Or alert on systemd-pstore.service start that actually processed files (combine with a count of new files). Empty boots where the condition skips the unit are normal.

Retention

Defaults already age out /var/lib/systemd/pstore at 14 days via tmpfiles. Tighten if hosts are disk-tight:

sudo tee /etc/tmpfiles.d/systemd-pstore-retention.conf >/dev/null <<'EOF'
# Keep kernel panic archives 7 days
d /var/lib/systemd/pstore 0755 root root 7d
EOF
sudo systemd-tmpfiles --create /etc/tmpfiles.d/systemd-pstore-retention.conf
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

Symptom Likely cause What to check
Unit never runs /sys/fs/pstore empty (good!) or condition failed systemctl status systemd-pstore.service, ls /sys/fs/pstore
Unit skipped in container ConditionVirtualization=!container Run on the host / VM, not inside a container
Panic happened, still no archive Kernel did not dump kmsg; backend missing crash_kexec_post_notifiers, lsmod, dmesg pstore lines
Records stuck in /sys/fs/pstore Service disabled or Storage=none / Unlink=false systemctl is-enabled, pstore.conf drop-ins
Backend always full Unlink=no or archival never ran Enable service, set Unlink=yes, free records
kdump + pstore interaction surprises Default notifier ordering Vendor note on crash_kexec_post_notifiers risk vs kdump
No EFI backend Old BIOS, locked-down firmware, missing module Try ramoops only if you can reserve RAM correctly

Debug the oneshot once:

sudo SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-pstore
echo exit:$?
Enter fullscreen mode Exit fullscreen mode

Security notes

  • Pstore and its archives can contain sensitive kernel log context. Keep /var/lib/systemd/pstore root-only.
  • Do not ship raw panic files to public issue trackers without scrubbing.
  • On multi-tenant hosts, short retention is reasonable.
  • pstore is local by design — good for air-gapped failure modes, still subject to full-disk encryption policy on the root filesystem where archives land.

Complete minimal playbook 

#!/usr/bin/env bash
set -euo pipefail

sudo systemctl enable systemd-pstore.service

sudo mkdir -p /etc/systemd/pstore.conf.d
sudo tee /etc/systemd/pstore.conf.d/99-local.conf >/dev/null <<'EOF'
[PStore]
Storage=external
Unlink=yes
EOF

sudo tee /etc/tmpfiles.d/systemd-pstore.conf >/dev/null <<'EOF'
d /var/lib/systemd/pstore 0755 root root 14d
w- /sys/module/kernel/parameters/crash_kexec_post_notifiers - - - - Y
EOF

sudo systemd-tmpfiles --create /etc/tmpfiles.d/systemd-pstore.conf

echo "=== unit ==="
systemctl is-enabled systemd-pstore.service
systemctl cat systemd-pstore.service --no-pager | sed -n '1,35p'

echo "=== config ==="
systemd-analyze cat-config systemd/pstore.conf

echo "=== knobs ==="
cat /sys/module/kernel/parameters/crash_kexec_post_notifiers

echo "=== backend ==="
mount | grep pstore || true
ls -la /sys/fs/pstore || true

echo "=== archives ==="
sudo ls -la /var/lib/systemd/pstore 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

What this does not replace

  • Not userspace core dumps — use systemd-coredump / coredumpctl for application SIGSEGV/SIGABRT.
  • Not a full crash dump — use kdump when you need vmcore and crash(8)/drgn analysis.
  • Not continuous observability — pstore is for the last words of a dying kernel, not metrics.
  • Not a substitute for firmware that has no persistent backend — verify EFI/ERST/ramoops actually exists on the platform.

Wrap-up

If you already fixed userspace crash capture and still shrug after a hard reset, wire pstore:

  1. Enable systemd-pstore.service
  2. Keep Storage=external and Unlink=yes
  3. Turn on crash_kexec_post_notifiers via tmpfiles
  4. After the next ugly reboot, look in /var/lib/systemd/pstore/ before you start guessing

Kernel panics will still ruin your morning. Losing the stack trace does not have to.

Sources and references

  • systemd-pstore.service(8) — archival service behavior, independence from kdump, journal + /var/lib/systemd/pstore/
  • pstore.conf(5) — Storage= (none / external / journal), Unlink=
  • Vendor unit /usr/lib/systemd/system/systemd-pstore.service — ConditionDirectoryNotEmpty=/sys/fs/pstore, WantedBy=sysinit.target, modprobe@efi_pstore.service
  • Vendor tmpfiles /usr/lib/tmpfiles.d/systemd-pstore.conf — 14-day archive dir, crash_kexec_post_notifiers, printk.always_kmsg_dump
  • Kernel docs: ramoops oops/panic logger — reserved RAM backend, record format, DT/module parameters
  • Linux pstore overview (kernel documentation tree) — platform persistent storage concepts used by EFI/ERST/ramoops backends

Top comments (1)

Collapse
 
topstar_ai profile image
Luis Cruz •

I appreciate how the article highlights the importance of preserving kernel panic evidence using systemd-pstore on Linux. The distinction between pstore and kdump is crucial, as they serve complementary purposes - pstore for capturing the kernel log tail and kdump for full kernel crash dumps. The step-by-step guide for confirming pstore presence and configuring archival policy is also very helpful. One aspect that caught my attention was the use of a drop-in configuration for /etc/systemd/pstore.conf.d/ to avoid editing the main configuration file directly, which is a good practice for maintaining configuration flexibility and ease of management. Have you considered exploring the integration of systemd-pstore with other logging and monitoring tools to enhance diagnostics and troubleshooting capabilities?