Practical Linux perf for Flame Graphs and Production Profiling

CPU flame graphs turn opaque performance issues into clear, actionable pictures. Instead of staring at top or guessing which function is eating cycles, you get a visual map of exactly where time is spent across the entire software stack.

This post shows a practical, repeatable workflow using perf to capture samples from production services and turn them into flame graphs. Everything is tested on Debian/Ubuntu and works with systemd-managed workloads.

Why Flame Graphs Beat Traditional Profiling

Traditional profilers give you flat lists or call trees that become unreadable at scale. Flame graphs (popularized by Brendan Gregg) stack frames by width proportional to sample count, making the widest (hottest) paths immediately obvious—even across thousands of threads.

Key advantages:

• See both user and kernel space in one view
• Spot regressions instantly by comparing before/after graphs
• Works on live production systems with low overhead when sampling

Prerequisites and Safety

Install the tools:

sudo apt update
sudo apt install linux-perf linux-tools-common

For flame graph generation you also need the scripts:

git clone https://github.com/brendangregg/FlameGraph.git ~/FlameGraph

Safety note: Sampling with perf record -a is read-only and safe on production. The default 99 Hz rate adds negligible overhead for most workloads. Always test first on a staging system.

Capturing Samples from a Running Service

The most useful command for services is to sample everything for a fixed duration:

sudo perf record -a -g --call-graph dwarf -F 99 -o /tmp/perf.data -- sleep 60

Explanation of flags:

• -a: System-wide (all CPUs)
• -g: Record call graphs
• --call-graph dwarf: Best stack unwinding for modern binaries
• -F 99: 99 samples per second (avoids lockstep with timers)
• -o /tmp/perf.data: Output file
• sleep 60: Capture for one minute

For a specific systemd service only (lower noise):

sudo perf record -g --call-graph dwarf -F 99 -p $(systemctl show -p MainPID --value myservice.service) -o /tmp/perf.data -- sleep 30

Generating the Flame Graph

Convert the perf data into the folded stack format and then render the SVG:

sudo perf script -i /tmp/perf.data | \
  ~/FlameGraph/stackcollapse-perf.pl | \
  ~/FlameGraph/flamegraph.pl > /tmp/flamegraph.svg

Open the SVG in any browser. The x-axis shows the full profile; y-axis is stack depth. Wider blocks = more samples.

Common next steps:

• Search (Ctrl+F) for a function name you suspect
• Click any block to zoom in
• Compare two graphs side-by-side when investigating regressions

Making It a Repeatable systemd Timer

Create a simple profiling service + timer so you can run this on demand or nightly.

/etc/systemd/system/perf-profile.service:

[Unit]
Description=Capture perf samples for flame graph
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/capture-perf-flamegraph.sh
User=root

/etc/systemd/system/perf-profile.timer:

[Unit]
Description=Run perf profiling nightly

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

The script (/usr/local/bin/capture-perf-flamegraph.sh):

#!/bin/bash
set -euo pipefail

DATE=$(date +%Y-%m-%d_%H-%M)
OUTDIR=/var/log/perf-flamegraphs
mkdir -p "$OUTDIR"

perf record -a -g --call-graph dwarf -F 99 \
  -o "$OUTDIR/perf-$DATE.data" -- sleep 120

perf script -i "$OUTDIR/perf-$DATE.data" | \
  ~/FlameGraph/stackcollapse-perf.pl | \
  ~/FlameGraph/flamegraph.pl > "$OUTDIR/flamegraph-$DATE.svg"

# Keep only last 14 days
find "$OUTDIR" -name 'flamegraph-*.svg' -mtime +14 -delete
find "$OUTDIR" -name 'perf-*.data' -mtime +14 -delete

Make it executable and enable:

sudo chmod +x /usr/local/bin/capture-perf-flamegraph.sh
sudo systemctl daemon-reload
sudo systemctl enable --now perf-profile.timer

Interpreting Results

Look for:

• Unexpectedly wide blocks in library code (e.g., malloc, memcpy)
• Kernel functions dominating (possible driver or syscall issues)
• Recursive or deeply nested application code

Cross-reference with perf top for live views and perf stat for hardware counters (cache misses, branch mispredictions).

Sources and Further Reading

• Brendan Gregg's CPU Flame Graphs: https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
• Linux perf Examples: https://www.brendangregg.com/perf.html
• Flame Graphs main page: https://www.brendangregg.com/flamegraphs.html
• perf wiki: https://perf.wiki.kernel.org

This workflow has helped many teams move from "the box is slow" to "this exact function in this library is the culprit" in minutes instead of hours.

Try it on one of your busiest services tonight and see what the graph reveals.