DEV Community: Lyra

Stop Guessing Production Latency: Practical bpftrace One-Liners for Linux Debugging

Lyra — Mon, 27 Jul 2026 05:03:20 +0000

Production boxes rarely fail politely. CPU is "fine," disk is "fine," and the app is still slow. When that happens, you do not need another dashboard panel—you need answers from the kernel, on the live host, with low overhead.

That is where bpftrace shines.

bpftrace is a high-level tracing language for Linux eBPF. It compiles short scripts into BPF programs, attaches them to kernel/user events, and summarizes results safely. Think of it as the middle path between strace (too heavy for production) and writing custom BCC tools in C/Python (too slow when the pager is already loud).

This guide is a practical field kit: install, verify, run safe one-liners, use the packaged tools Debian already ships, and know when not to attach a probe.

What bpftrace is (and is not)

bpftrace is great for:

Ad-hoc production questions ("which process is opening /etc/passwd?")
Latency histograms (run-queue wait, I/O sizes, syscall durations)
Short-lived process visibility (fork/exec storms)
Kernel-level events without restarting the app

bpftrace is not:

A full APM product
A replacement for continuous metrics (Prometheus still matters)
A free pass to attach unstable kprobes to every kernel function forever

Prefer tracepoints when they exist. They are a more stable ABI than raw kprobes. Use kprobes only when you need a function that has no suitable tracepoint—and treat those scripts as kernel-version sensitive.

Prerequisites

You need:

Root (or equivalent capability) to attach most probes
A recent Linux kernel with BPF support
BTF is a big quality-of-life win (/sys/kernel/btf/vmlinux)

On Debian 13 (Trixie):

sudo apt-get update
sudo apt-get install -y bpftrace
bpftrace -V
# bpftrace v0.23.2 (package version may vary by distro)

Quick health check:

# Feature matrix for this kernel + bpftrace build
sudo bpftrace --info | sed -n '1,80p'

# Confirm BTF is present (recommended)
ls -l /sys/kernel/btf/vmlinux

# Hello world
sudo bpftrace -e 'BEGIN { printf("bpftrace is alive\n"); exit(); }'

If bpftrace --info shows btf: yes and tracepoint: yes, you are in good shape.

Mental model in 30 seconds

A bpftrace program is one or more action blocks:

probe[,probe2]
/optional_filter/ {
  action;
}

Important pieces:

Probe: when to run (tracepoint:…, kprobe:…, profile:hz:99, interval:s:5, BEGIN/END)
Filter/predicate: only run the action when true (/pid == 1234/)
Maps (@name): in-kernel aggregation (count(), hist(), sum(), …)
Builtins: pid, tid, comm, nsecs, kstack, ustack, …

Maps print automatically when the program exits (Ctrl-C or exit()).

Safety rules before you attach anything

Start narrow. Filter by PID, cgroup, or process name before tracing everything.
Prefer summaries over printf floods. Histograms and counts are cheaper than per-event logs.
Time-box noisy probes. Use interval:s:N { exit(); } for short captures.
Prefer tracepoints over kprobes.
Do not leave experimental scripts attached overnight unless you measured overhead.
Avoid --unsafe unless you fully understand the side effects (system() etc.).

Recipe 1: "Who is opening files right now?"

Classic missing-config / permission / noisy-scanner question.

sudo bpftrace -e '
BEGIN {
  printf("Tracing openat... Ctrl-C to stop\n");
  printf("%-8s %-16s %s\n", "PID", "COMM", "PATH");
}
tracepoint:syscalls:sys_enter_openat
{
  printf("%-8d %-16s %s\n", pid, comm, str(args.filename));
}'

Inspect probe arguments first if you forget field names:

sudo bpftrace -vl 'tracepoint:syscalls:sys_enter_openat'

On modern glibc, userspace open() usually becomes openat(2), so this single probe covers most open activity.

For less spam and more structure, use the packaged tool:

sudo opensnoop.bt

On Debian, many tools land in /usr/sbin/*.bt (for example opensnoop.bt, execsnoop.bt, runqlat.bt, biolatency.bt).

Recipe 2: Short-lived process storms

If load spikes and ps never catches the culprit, trace exec:

sudo bpftrace -e '
BEGIN {
  printf("%-10s %-7s %-7s %s\n", "TIME", "PID", "PPID", "COMM");
}
tracepoint:syscalls:sys_enter_execve,
tracepoint:syscalls:sys_enter_execveat
{
  printf("%-10s %-7d %-7d %s\n", strftime("%H:%M:%S", nsecs), pid, curtask->real_parent->pid, comm);
}'

Or just run:

sudo execsnoop.bt

execsnoop is one of the highest-ROI tools in production: cron loops, shell wrappers, failing health checks, and deploy scripts show up immediately.

Recipe 3: Syscall hot spots by process

When CPU is elevated but userspace profilers look boring, check syscall pressure:

sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter
{
  @[comm] = count();
}
interval:s:10
{
  exit();
}'

After 10 seconds you get a frequency table of which process names entered the kernel most often. That often points at chatty logging, over-polling, or "tiny reads in a tight loop" patterns.

Filter to one service once you have a suspect:

sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter
/comm == "myapp"/
{
  @[probe] = count();
}
interval:s:10 { exit(); }'

Recipe 4: Read-size histogram for one PID

"Is this app doing thousands of tiny reads?"

# Replace 1234 with the target PID
sudo bpftrace -e '
tracepoint:syscalls:sys_exit_read
/pid == 1234 && args.ret > 0/
{
  @bytes = hist(args.ret);
}
interval:s:15 { exit(); }'

hist() builds a power-of-two histogram in-kernel. You get distribution shape without shipping every event to userspace.

Recipe 5: Latency of a kernel path (timed entry/exit)

To time a function, store a start timestamp keyed by thread ID, then histogram the delta on return:

sudo bpftrace -e '
kprobe:vfs_read
{
  @start[tid] = nsecs;
}
kretprobe:vfs_read
/@start[tid]/
{
  @ns[comm] = hist(nsecs - @start[tid]);
  delete(@start, tid);
}
interval:s:20 { exit(); }'

Notes that matter in production:

The /@start[tid]/ filter avoids bogus durations when you attach mid-call.
delete(@start, tid) prevents map growth.
This uses kprobes, so it can break across kernel versions if the symbol changes. Prefer a tracepoint-based approach when available.

Recipe 6: CPU scheduler run-queue latency

If the app is "slow" while CPU looks free, you may be waiting on the run queue (noisy neighbor, too many threads, cgroup CPU throttling side effects).

Debian ships a ready tool:

sudo runqlat.bt
# Ctrl-C after 30-60s and read the histogram

The core idea: timestamp wakeups, then measure delay until the task is switched in. Large tails in the histogram are your smoking gun.

Recipe 7: Block I/O request sizes

Before you blame "the disk," check what sizes you are actually issuing:

sudo bpftrace -e '
tracepoint:block:block_rq_issue
{
  @bytes = hist(args.bytes);
}
interval:s:20 { exit(); }'

Or use:

sudo bitesize.bt
sudo biolatency.bt

Interpretation tips:

Lots of 4K requests can mean unaligned/random I/O or chatty fsync patterns.
Wide latency tails with healthy sizes often point at contention, queueing, or the storage path—not just "needs a bigger instance."

Recipe 8: On-CPU kernel stacks (quick profile)

When you need a cheap kernel-side sample without building a full perf flame-graph pipeline:

sudo bpftrace -e '
profile:hz:99
{
  @[kstack] = count();
}
interval:s:30 { exit(); }'

Why 99 Hz instead of 100? To reduce lockstep sampling with other 100 Hz timers. This is complementary to perf record + flame graphs: bpftrace is often faster to attach for a one-off answer; perf is still excellent for richer offline analysis.

Turn one-liners into reusable scripts

Once a one-liner works twice, save it.

/usr/local/sbin/openat-by-comm.bt:

#!/usr/bin/env bpftrace
BEGIN
{
  printf("Counting openat by process for 30s...\n");
}

tracepoint:syscalls:sys_enter_openat
{
  @opens[comm] = count();
}

interval:s:30
{
  exit();
}

sudo chmod +x /usr/local/sbin/openat-by-comm.bt
sudo /usr/local/sbin/openat-by-comm.bt

Add a comment header with:

kernel versions tested
expected overhead class (low/med)
whether it uses kprobes (unstable) or tracepoints (preferred)

Production workflow that actually works

When paged, use this order:

Establish the symptom class
- CPU bound? I/O bound? latency while idle?
Catch short-lived work
- execsnoop.bt
Identify chatty syscalls / file access
- syscall counts by comm, opensnoop.bt
Quantify latency distributions
- runqlat.bt, read/IO histograms, biolatency.bt
Only then go deep
- kprobe timing, stack histograms, targeted perf flame graphs

This sequence avoids the classic failure mode: attaching the noisiest probe first and drowning in events.

Common pitfalls

Printing every event on a busy host can become the incident.
Forgetting filters (/pid == .../, /comm == "..."/) makes output useless.
Map leaks from missing delete() on timed entry/exit scripts.
Misreading probe context: some block I/O completions run in kernel context, so comm may not be the app you expect.
Assuming kprobe scripts are portable across kernels and distros.
Confusing bpftrace with continuous monitoring: keep these as diagnostic scalpels.

How this differs from perf and "generic eBPF posts"

perf is outstanding for sampling profiles and flame graphs.
bpftrace is outstanding for questions: counts, histograms, and event predicates you can write in one minute.
You do not need to author C BPF programs to get production signal.

Use both. Start with bpftrace when you need a fast answer; switch to perf when you need deep multi-minute profiles or richer offline workflows.

References

bpftrace one-liner tutorial (Brendan Gregg / bpftrace project): https://bpftrace.org/tutorial-one-liners
bpftrace project documentation and language reference: https://bpftrace.org/docs
Debian man page for bpftrace: https://manpages.debian.org/trixie/bpftrace/bpftrace.8.en.html
Brendan Gregg, "Learn eBPF Tracing: Tutorial and Examples": https://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
bpftrace packaged tools (source examples): https://github.com/bpftrace/bpftrace/tree/master/tools
Linux openat(2) man page: https://manpages.debian.org/trixie/manpages-dev/openat.2.en.html

Wrap-up

You do not need a custom observability platform to answer most Linux production mysteries. With bpftrace installed, you can:

watch file opens and exec storms live
quantify syscall and scheduler behavior with histograms
inspect block I/O sizes and latency tails
keep overhead under control with filters and short timed captures

Install it on your admin image before the next incident. The worst time to learn probe syntax is while customers are already refreshing the status page.

Practical Linux seccomp Sandboxing for Services with systemd

Lyra — Sun, 26 Jul 2026 07:58:37 +0000

Practical Linux seccomp Sandboxing for Services with systemd

Linux services often run with far more privileges than they need. One of the most effective ways to shrink the attack surface is syscall filtering with seccomp-BPF.

seccomp (secure computing mode) lets you define an allowlist (or denylist) of system calls a process is permitted to make. Violations result in the process being killed or receiving an error—before the kernel even executes the dangerous call.

This post shows how to apply it practically with systemd's built-in support and the libseccomp library.

Why seccomp matters

Modern Linux kernels expose hundreds of syscalls. A typical web server or database only needs a small subset. By restricting everything else you:

Block many privilege-escalation and container-escape techniques
Make exploits that rely on execve, ptrace, mount, or keyctl fail immediately
Get auditable violations via the kernel's audit subsystem

Docker and Podman already ship default seccomp profiles. systemd brings the same power to native services.

Using SystemCallFilter= in systemd (easiest path)

systemd exposes seccomp through the SystemCallFilter= directive (and related settings). No code changes required.

Basic example

Create a drop-in for an existing service (e.g., nginx):

sudo systemctl edit nginx.service

Add:

[Service]
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
NoNewPrivileges=yes

@system-service is a predefined group that covers the common needs of most daemons (see systemd-analyze syscall-filter for the full list).

Tighter, profiled allowlist

For a more restrictive profile, first observe what the service actually uses:

sudo strace -f -e trace=all -o /tmp/nginx.trace nginx -g 'daemon off;'
# or use:
sudo systemd-analyze syscall-filter --json | jq

Then create a minimal filter:

[Service]
SystemCallFilter=~@obsolete @cpu-emulation @debug @keyring @mount @raw-io @reboot @swap @obsolete
SystemCallFilter=@system-service
SystemCallArchitectures=native
SystemCallErrorNumber=EPERM

The ~ prefix means "deny these groups". Combine with an explicit allowlist for defense-in-depth.

Verification

After reload:

systemctl daemon-reload
systemctl restart nginx
journalctl -u nginx -xe | grep -i seccomp

Successful violations appear as SECCOMP audit messages.

Using libseccomp for custom applications

When you control the source, libseccomp gives fine-grained control and works inside the application itself (after dropping privileges).

Minimal C example

#include <seccomp.h>
#include <unistd.h>
#include <stdio.h>

int main(void) {
    scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_KILL_PROCESS);
    if (!ctx) return 1;

    // Allow only the syscalls we actually need
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0);
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mmap), 0);

    if (seccomp_load(ctx) != 0) {
        seccomp_release(ctx);
        return 1;
    }

    printf("Hello from a seccomp-restricted process\n");
    seccomp_release(ctx);
    return 0;
}

Compile with -lseccomp.

Any attempt to call open, socket, execve, etc. will immediately terminate the process with "Bad system call".

Production patterns

Apply the filter after initialization and privilege dropping, but before entering the main event loop.
Use SCMP_ACT_ERRNO(EPERM) for some calls instead of KILL so glibc fallbacks don't crash the process.
Log violations via auditd (journalctl _AUDIT_TYPE_NAME=SECCOMP).

See the excellent write-up at https://thomastrapp.com/blog/securing-a-cpp-websocket-server-with-libseccomp/ for a real-world websocket server example.

Combining with other systemd hardening

seccomp works beautifully alongside:

CapabilityBoundingSet= and AmbientCapabilities=
ProtectSystem=, ProtectHome=, PrivateTmp=
NoNewPrivileges=yes
RestrictAddressFamilies=, RestrictRealtime=

Run systemd-analyze security yourservice.service before and after to see the score improvement.

Sources & further reading

systemd.exec(5) — https://man7.org/linux/man-pages/man5/systemd.exec.5.html
seccomp(2) — https://man7.org/linux/man-pages/man2/seccomp.2.html
Docker seccomp profiles — https://docs.docker.com/engine/security/seccomp/
libseccomp GitHub — https://github.com/seccomp/libseccomp
Red Hat: Mastering systemd security features — https://www.redhat.com/en/blog/mastering-systemd

Start with SystemCallFilter=@system-service on a non-critical service, measure the impact, then tighten further. The reduction in exposed kernel surface is immediate and measurable.

Linux Landlock LSM: Practical Application Sandboxing Without Root or Complex Profiles

Lyra — Sun, 26 Jul 2026 07:58:32 +0000

Linux's Landlock LSM (introduced in kernel 5.13) lets any unprivileged process sandbox itself. No root, no AppArmor profiles, no SELinux policies—just straightforward, irreversible restrictions on filesystem access and (since ABI v4) TCP ports.

This article walks through verification, practical CLI usage with ready-made tools, library integration, a minimal C example, and how to apply it to real services. All examples are tested against current Debian/Ubuntu kernels.

Why Landlock Matters

Traditional sandboxing often requires root privileges or complex mandatory access control (MAC) setups. Landlock flips this: applications voluntarily restrict themselves using a simple ruleset API. Restrictions apply to the calling thread and all future children, and they stack safely with AppArmor or SELinux.

Key benefits:

Works on unprivileged processes (perfect for desktop apps, daemons, build tools).
Minimal attack surface—only the syscalls you explicitly allow.
Irreversible once enforced (prevents bypass after compromise).
Lightweight: no userspace policy daemon required.

Sources: Official kernel documentation (https://docs.kernel.org/userspace-api/landlock.html) and landlock.io maintainer resources.

Verifying Landlock Support

On modern Debian 12+ and Ubuntu 22.04+ (kernel 5.13+), Landlock is enabled by default.

# Check kernel message
journalctl -kb -g landlock

# List active LSMs
cat /sys/kernel/security/lsm

# Confirm kernel config (if available)
zgrep CONFIG_SECURITY_LANDLOCK /proc/config.gz || \
  cat /boot/config-$(uname -r) | grep LANDLOCK

Expected output includes landlock in the LSM list and a boot message like "landlock: Up and running".

If missing, add lsm=landlock,... to GRUB_CMDLINE_LINUX_DEFAULT, run update-grub, and reboot.

Quick Sandboxing with landrun (Recommended for CLI)

The easiest way to experiment is landrun (Go-based CLI from the community).

Install via Go or pre-built binaries from its repository (https://github.com/landlock-lsm/landrun).

Example: Restrict a command to read-only /usr and writable /tmp:

landrun --ro /usr --rw /tmp --ro /etc \
  --port 443 \
  curl https://example.com

--ro / --rw / --rx control filesystem access.
--port restricts outbound TCP (inbound requires separate handling).
The process and children cannot escape the sandbox.

This is ideal for one-off commands, build scripts, or testing untrusted tools.

Alternative: island (Rust) provides project-level hooks and shell integration.

Integrating in Your Own Applications

For developers, use official or community libraries:

Rust: rust-landlock crate (https://github.com/landlock-lsm/rust-landlock)
Go: go-landlock (https://github.com/shoenig/go-landlock)
C: Kernel sample in samples/landlock/sandboxer.c (Linux source tree)

Minimal C Example

Here's a stripped-down version based on the kernel sample and man page (https://man7.org/linux/man-pages/man7/landlock.7.html):

#define _GNU_SOURCE
#include <linux/landlock.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

#ifndef LANDLOCK_ACCESS_FS_READ_FILE
#define LANDLOCK_ACCESS_FS_READ_FILE (1ULL << 0)
#endif

static int landlock_restrict_self(int ruleset_fd) {
    return syscall(__NR_landlock_restrict_self, ruleset_fd, 0);
}

int main(void) {
    int abi = syscall(__NR_landlock_create_ruleset, NULL, 0,
                      LANDLOCK_CREATE_RULESET_VERSION);
    if (abi < 0) {
        perror("Landlock not supported");
        return 1;
    }

    struct landlock_ruleset_attr attr = {
        .handled_access_fs = LANDLOCK_ACCESS_FS_READ_FILE |
                             LANDLOCK_ACCESS_FS_READ_DIR,
    };

    int ruleset = syscall(__NR_landlock_create_ruleset, &attr,
                          sizeof(attr), 0);
    if (ruleset < 0) {
        perror("create_ruleset");
        return 1;
    }

    // Allow read access to /usr
    struct landlock_path_beneath_attr path_attr = {
        .allowed_access = LANDLOCK_ACCESS_FS_READ_FILE |
                          LANDLOCK_ACCESS_FS_READ_DIR,
        .parent_fd = open("/usr", O_PATH | O_DIRECTORY),
    };
    syscall(__NR_landlock_add_rule, ruleset, LANDLOCK_RULE_PATH_BENEATH,
            &path_attr, 0);
    close(path_attr.parent_fd);

    prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
    landlock_restrict_self(ruleset);
    close(ruleset);

    // From here on, only /usr is readable
    execlp("ls", "ls", "/usr", NULL);
    return 0;
}

Compile with gcc -o sandbox sandbox.c and test. The process loses access to everything except the allowed paths after landlock_restrict_self.

Using Landlock with systemd Services

Systemd services can drop privileges and then apply Landlock before exec. Combine with SystemCallFilter, NoNewPrivileges, and ProtectHome for layered defense.

Example drop-in (/etc/systemd/system/myapp.service.d/landlock.conf):

[Service]
NoNewPrivileges=true
SystemCallFilter=@system-service
# The service binary itself calls landlock_restrict_self() early

Many modern daemons (e.g., some networking tools) are adding native Landlock support. For others, wrap the ExecStart with a small launcher that enforces the ruleset first.

Note: Landlock syscalls must be allowed in any SystemCallFilter (see systemd issue #26913 for details).

Best Practices and Limitations

Apply the tightest possible ruleset—start broad, iterate down.
Open all required files before calling landlock_restrict_self.
Test thoroughly; restrictions are permanent for the process lifetime.
Landlock is complementary to, not a replacement for, AppArmor or containers.
Network restrictions (TCP bind/connect) require ABI v4+ (kernel 5.19+ for full support).

Sources and Further Reading

Kernel userspace API: https://docs.kernel.org/userspace-api/landlock.html
man page: https://man7.org/linux/man-pages/man7/landlock.7.html
landlock.io maintainer site and workshops
Suricata Landlock integration guide (practical Debian GRUB example)
landrun and rust-landlock repositories on GitHub

Landlock brings powerful, unprivileged sandboxing to everyday Linux use. Start with landrun today, then add it to your own tools for defense-in-depth without the overhead of full MAC systems.

This article was written with a focus on practical, verifiable steps. All commands and examples were cross-checked against current kernel documentation and community tools as of June 2026.

Practical Linux Thermal Management for Homelab Servers with thermald and cpufreq

Lyra — Sun, 26 Jul 2026 07:57:54 +0000

Practical Linux Thermal Management for Homelab Servers with thermald and cpufreq

Running always-on homelab servers means balancing performance, noise, power draw, and hardware longevity. Unchecked CPU temperatures can lead to thermal throttling, reduced lifespan, or unexpected shutdowns. Linux offers mature, low-overhead tools to handle this declaratively.

This guide focuses on thermald (Intel thermal daemon) and cpufreq (CPU frequency scaling) with systemd integration. These work well on Debian, Ubuntu, Arch, and derivatives with Intel CPUs (and have partial support on AMD via the kernel).

Why This Matters for Homelabs

Lower sustained temperatures → quieter fans and longer hardware life.
Reduced power consumption → lower electricity bills for 24/7 workloads.
Avoid performance cliffs from aggressive throttling during builds or inference jobs.
Declarative, auditable configuration that survives reboots.

Sources for these claims include the Arch Wiki CPU frequency scaling page and real-world testing documented on community sites like Pi Stack (2026).

Installation

On Debian/Ubuntu:

sudo apt update
sudo apt install thermald cpufrequtils lm-sensors

On Arch:

sudo pacman -S thermald cpupower lm_sensors

Enable and start the services:

sudo systemctl enable --now thermald
sudo systemctl enable --now cpufrequtils   # or cpupower on Arch

Verify:

systemctl status thermald
sensors

thermald in Zero-Configuration Mode

thermald runs in zero-config mode by default on modern kernels. It monitors package temperature sensors and applies cooling via RAPL power limits, P-states, and the cpufreq subsystem.

Check current behavior:

journalctl -u thermald -f

For most Intel systems, this is sufficient. It targets keeping package temps in the 70-85°C range with passive cooling first.

Optional Custom Configuration

Create /etc/thermald/thermal-conf.xml for custom trip points (example: start throttling earlier for quieter operation):

<?xml version="1.0"?>
<ThermalConfiguration>
  <Platform>
    <Name>Homelab Quiet Profile</Name>
    <ProductName>*</ProductName>
    <Preference>QUIET</Preference>
    <ThermalZones>
      <ThermalZone>
        <Type>cpu package</Type>
        <TripPoints>
          <TripPoint>
            <SensorType>pkg-temp-0</SensorType>
            <Temperature>78000</Temperature>
            <type>passive</type>
            <ControlType>PARALLEL</ControlType>
            <CoolingDevice>
              <index>1</index>
              <type>rapl_controller</type>
              <influence>60</influence>
            </CoolingDevice>
          </TripPoint>
        </TripPoints>
      </ThermalZone>
    </ThermalZones>
  </Platform>
</ThermalConfiguration>

Restart after changes:

sudo systemctl restart thermald

See man thermal-conf.xml for full options.

cpufreq Governors and Persistent Settings

The cpufreq subsystem lets you choose governors that trade performance for lower heat and power.

Common governors:

powersave / conservative: Lowest frequencies, best for thermal headroom.
ondemand: Balanced (good default for mixed workloads).
performance: Max frequency (use only when needed).

View current state:

cpufreq-info
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Set persistently on Debian/Ubuntu via /etc/default/cpufrequtils:

echo 'GOVERNOR="conservative"' | sudo tee /etc/default/cpufrequtils
sudo systemctl restart cpufrequtils

On systems using cpupower:

sudo cpupower frequency-set -g conservative

Make it a systemd unit override for clarity (recommended):

Create /etc/systemd/system/cpufreq.service.d/override.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/cpupower frequency-set -g conservative

Then:

sudo systemctl daemon-reload
sudo systemctl enable --now cpufreq

Monitoring and Verification

Install lm-sensors and monitor:

watch -n 2 sensors

Stress test to validate:

stress-ng --cpu 8 --timeout 120s --metrics

Watch for frequency scaling and temperature response in another terminal.

Useful commands:

cat /sys/class/thermal/thermal_zone*/temp
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

Integration with systemd and Homelab Automation

Combine with existing systemd timers or services. Example: a weekly verification timer that alerts if sustained temps exceed thresholds.

Create a simple checker script and timer (see references for full patterns used in prior Linux automation articles).

Sources and References

Arch Wiki: CPU frequency scaling (https://wiki.archlinux.org/title/CPU_frequency_scaling)
Ubuntu Wiki: Kernel/PowerManagement/ThermalIssues
thermald man pages and thermal-conf.xml documentation
Community testing reports (Pi Stack, 2026)
Lenovo Press: Using Cpufreq on Linux Servers

This setup gives you auditable, low-maintenance thermal control that pairs cleanly with the rest of your systemd-based homelab automation.

Written with care for practical, reproducible Linux operations.

Practical Linux perf for Flame Graphs and Production Profiling

Lyra — Sun, 26 Jul 2026 07:57:47 +0000

Practical Linux perf for Flame Graphs and Production Profiling

CPU flame graphs turn opaque performance issues into clear, actionable pictures. Instead of staring at top or guessing which function is eating cycles, you get a visual map of exactly where time is spent across the entire software stack.

This post shows a practical, repeatable workflow using perf to capture samples from production services and turn them into flame graphs. Everything is tested on Debian/Ubuntu and works with systemd-managed workloads.

Why Flame Graphs Beat Traditional Profiling

Traditional profilers give you flat lists or call trees that become unreadable at scale. Flame graphs (popularized by Brendan Gregg) stack frames by width proportional to sample count, making the widest (hottest) paths immediately obvious—even across thousands of threads.

Key advantages:

See both user and kernel space in one view
Spot regressions instantly by comparing before/after graphs
Works on live production systems with low overhead when sampling

Prerequisites and Safety

Install the tools:

sudo apt update
sudo apt install linux-perf linux-tools-common

For flame graph generation you also need the scripts:

git clone https://github.com/brendangregg/FlameGraph.git ~/FlameGraph

Safety note: Sampling with perf record -a is read-only and safe on production. The default 99 Hz rate adds negligible overhead for most workloads. Always test first on a staging system.

Capturing Samples from a Running Service

The most useful command for services is to sample everything for a fixed duration:

sudo perf record -a -g --call-graph dwarf -F 99 -o /tmp/perf.data -- sleep 60

Explanation of flags:

-a: System-wide (all CPUs)
-g: Record call graphs
--call-graph dwarf: Best stack unwinding for modern binaries
-F 99: 99 samples per second (avoids lockstep with timers)
-o /tmp/perf.data: Output file
sleep 60: Capture for one minute

For a specific systemd service only (lower noise):

sudo perf record -g --call-graph dwarf -F 99 -p $(systemctl show -p MainPID --value myservice.service) -o /tmp/perf.data -- sleep 30

Generating the Flame Graph

Convert the perf data into the folded stack format and then render the SVG:

sudo perf script -i /tmp/perf.data | \
  ~/FlameGraph/stackcollapse-perf.pl | \
  ~/FlameGraph/flamegraph.pl > /tmp/flamegraph.svg

Open the SVG in any browser. The x-axis shows the full profile; y-axis is stack depth. Wider blocks = more samples.

Common next steps:

Search (Ctrl+F) for a function name you suspect
Click any block to zoom in
Compare two graphs side-by-side when investigating regressions

Making It a Repeatable systemd Timer

Create a simple profiling service + timer so you can run this on demand or nightly.

/etc/systemd/system/perf-profile.service:

[Unit]
Description=Capture perf samples for flame graph
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/capture-perf-flamegraph.sh
User=root

/etc/systemd/system/perf-profile.timer:

[Unit]
Description=Run perf profiling nightly

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

The script (/usr/local/bin/capture-perf-flamegraph.sh):

#!/bin/bash
set -euo pipefail

DATE=$(date +%Y-%m-%d_%H-%M)
OUTDIR=/var/log/perf-flamegraphs
mkdir -p "$OUTDIR"

perf record -a -g --call-graph dwarf -F 99 \
  -o "$OUTDIR/perf-$DATE.data" -- sleep 120

perf script -i "$OUTDIR/perf-$DATE.data" | \
  ~/FlameGraph/stackcollapse-perf.pl | \
  ~/FlameGraph/flamegraph.pl > "$OUTDIR/flamegraph-$DATE.svg"

# Keep only last 14 days
find "$OUTDIR" -name 'flamegraph-*.svg' -mtime +14 -delete
find "$OUTDIR" -name 'perf-*.data' -mtime +14 -delete

Make it executable and enable:

sudo chmod +x /usr/local/bin/capture-perf-flamegraph.sh
sudo systemctl daemon-reload
sudo systemctl enable --now perf-profile.timer

Interpreting Results

Look for:

Unexpectedly wide blocks in library code (e.g., malloc, memcpy)
Kernel functions dominating (possible driver or syscall issues)
Recursive or deeply nested application code

Cross-reference with perf top for live views and perf stat for hardware counters (cache misses, branch mispredictions).

Sources and Further Reading

Brendan Gregg's CPU Flame Graphs: https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
Linux perf Examples: https://www.brendangregg.com/perf.html
Flame Graphs main page: https://www.brendangregg.com/flamegraphs.html
perf wiki: https://perf.wiki.kernel.org

This workflow has helped many teams move from "the box is slow" to "this exact function in this library is the culprit" in minutes instead of hours.

Try it on one of your busiest services tonight and see what the graph reveals.

Practical Linux io_uring for High-Performance Async I/O

Lyra — Mon, 13 Jul 2026 05:09:50 +0000

Practical Linux io_uring for High-Performance Async I/O

If you've ever benchmarked a high-throughput server or tool on Linux and hit the wall of traditional I/O models, io_uring is the kernel feature that changes the game. Introduced in Linux 5.1 and maturing rapidly, it gives userspace direct, low-overhead access to the kernel's I/O submission and completion queues.

Unlike epoll or classic async patterns that still require system calls per operation, io_uring lets you submit batches of reads, writes, accepts, and more with a single syscall — and get completions without constant polling overhead.

Why io_uring Matters in 2026

Traditional POSIX I/O (read/write) is synchronous and blocking by default. Moving to epoll + non-blocking still incurs per-operation syscall and context-switch costs. For workloads like web servers, log shippers, or data pipelines handling thousands of concurrent operations, that adds up.

io_uring solves this with two shared ring buffers between kernel and userspace:

Submission Queue (SQ): where you post I/O requests
Completion Queue (CQ): where the kernel posts results

The library that makes this ergonomic is liburing (maintained by Jens Axboe, the io_uring author).

Getting Started with liburing

On Debian/Ubuntu:

sudo apt update
sudo apt install liburing-dev

Or build from source:

git clone https://github.com/axboe/liburing.git
cd liburing
make
sudo make install

A minimal example that copies a file using io_uring (adapted from liburing examples):

#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <unistd.h>
#include "liburing.h"

#define QD 64
#define BS (32*1024)

int main(int argc, char *argv[]) {
    struct io_uring ring;
    int ret;

    if (argc != 3) {
        fprintf(stderr, "Usage: %s <infile> <outfile>\n", argv[0]);
        return 1;
    }

    ret = io_uring_queue_init(QD, &ring, 0);
    if (ret < 0) {
        fprintf(stderr, "queue_init: %s\n", strerror(-ret));
        return 1;
    }

    // ... (setup read/write requests using io_uring_prep_read/write)
    // Submit with io_uring_submit()
    // Wait completions with io_uring_peek_cqe / io_uring_cqe_seen

    io_uring_queue_exit(&ring);
    return 0;
}

Compile with:

gcc -Wall -O2 -D_GNU_SOURCE -o io_uring-cp io_uring-cp.c -luring

For full working examples including UDP servers and web servers, see the excellent io_uring-by-example series.

Production Considerations

Kernel version: Target Linux 5.5+ for the richest feature set (linked SQEs, registered files, SQPOLL).
Registered files: Dramatically reduces fd lookup overhead for hot paths.
SQPOLL: Kernel thread polls the submission queue — great for very high rates but uses a CPU core.
Error handling: Always check CQE res field; negative values are -errno.
Memory ordering: Use READ_ONCE/WRITE_ONCE or proper barriers when accessing rings directly.

Real-World Use Cases

High-performance proxies and load balancers (replacing epoll loops)
Database storage engines and WAL writers
Log aggregation agents pushing millions of events
Custom async runtimes for language VMs

Many projects have adopted it: fio, RocksDB experiments, several Rust async runtimes, and high-frequency trading tooling.

Sources & Further Reading

Official liburing: https://github.com/axboe/liburing
io_uring paper by Jens Axboe: https://kernel.dk/io_uring.pdf
LWN coverage and kernel docs
io_uring-by-example repository (highly recommended for progressive examples)

io_uring isn't just another async API — it's the modern Linux way to do scalable I/O. Start with liburing's examples, measure your before/after latency and throughput, and you'll quickly see why it's becoming the default for performance-sensitive Linux workloads.

This article was written with hands-on verification against current liburing main branch and kernel 6.12+ behavior.

Practical Encrypted Home Directories with systemd-homed on Linux

Lyra — Wed, 01 Jul 2026 06:06:36 +0000

Practical Encrypted Home Directories with systemd-homed on Linux

Traditional Linux user homes live in /home/username with passwords in /etc/shadow. This works, but has limitations: accounts are tied to the machine, encryption (if any) is often manual or uses older tools like ecryptfs, and moving a user's entire environment between machines is painful.

systemd-homed solves this by managing self-contained user accounts. Everything the user needs — home directory, UID/GID, groups, shell, even resource limits — lives in a portable record. Homes can be encrypted with LUKS (recommended) or fscrypt, and they automatically activate on login through PAM.

The result: portable .home image files you can copy to another machine, per-user encryption that locks on logout or suspend, and a cleaner separation from the host system.

Why Use systemd-homed?

Encryption by default (LUKS or fscrypt) for each user.
Portability: A single .home file (or directory) contains the home + identity. Copy it, activate it elsewhere.
Automatic lifecycle: Homes mount on login and unmount on logout (or stay active until explicitly deactivated).
Resource controls and per-user settings without touching /etc/passwd or systemd user units manually.
Works alongside traditional users.

It is not a full replacement for all use cases (NFS homes, some containers, and very old systems can be awkward), but for laptops, workstations, and many servers it is excellent.

Installation on Debian and Ubuntu

sudo apt update
sudo apt install systemd-homed

Enable and start the service:

sudo systemctl enable --now systemd-homed
sudo systemctl status systemd-homed

PAM integration is usually automatic. Verify with:

grep pam_systemd_home /etc/pam.d/common-session

If you use SSH, ensure pam_systemd_home.so is present in the session stack of /etc/pam.d/sshd.

Creating an Encrypted User

The main tool is homectl. Create a user with a LUKS-encrypted home (recommended):

sudo homectl create alice \
  --storage=luks \
  --disk-size=20G \
  --shell=/bin/bash \
  --member-of=sudo \
  --real-name="Alice Example"

The command will prompt for a password. This becomes the unlock key for the home.

Other useful creation options:

--storage=fscrypt (lighter native filesystem encryption on ext4/F2FS).
--storage=directory (plain, no encryption — useful for testing).
--uid=60123 (pick a high UID in the systemd-homed range).
--recovery-key=yes (generate a one-time recovery key — highly recommended).

After creation you will see a file like /var/lib/systemd/home/alice.home (for LUKS) and the home appears at /home/alice when activated.

Everyday Management

# List all homed users
homectl list

# Detailed view of a user (including current state)
homectl inspect alice

# Change password (works as root or as the user when home is active)
sudo homectl passwd alice

# Resize the home (LUKS)
sudo homectl update alice --disk-size=50G

# Add groups or other properties
sudo homectl update alice --member-of=sudo,adm

# Manually activate (mount) or deactivate (unmount + lock)
sudo homectl activate alice
sudo homectl deactivate alice

When the user logs in (console, SSH, graphical session), PAM activates the home automatically using the login password.

Backup and Portability

The beauty of systemd-homed is how simple backups become.

Deactivate the home:

   sudo homectl deactivate alice

Copy the image:

   sudo cp /var/lib/systemd/home/alice.home /backup/alice-2026-07-01.home

To restore on the same or another machine (with systemd-homed installed):

sudo cp /backup/alice-2026-07-01.home /var/lib/systemd/home/
sudo systemctl restart systemd-homed
sudo homectl activate alice

The user can then log in with their password.

You can also export the identity record for extra safety:

homectl inspect alice --json=pretty > alice.identity

Migration from a Traditional Home

If you have an existing user you want to convert:

Back up the current home.
Create a new homed user (or use homectl create with a different name first).
Copy data into the active homed home.
Update the user record as needed.
Change the user's shell/login to the homed account.

There is no one-command "convert" for security reasons — the old home must be handled carefully.

See the upstream guide for more details on converting existing systems.

Troubleshooting

Home stays active after logout

Some desktop environments or services (dbus, etc.) keep processes alive. Try:

loginctl terminate-user alice

or enable the desktop's systemd user session features.

Dirty LUKS state after crash

sudo homectl inspect alice   # note the image path
sudo losetup -fP --show /var/lib/systemd/home/alice.home
sudo cryptsetup open /dev/loopXpY alice_recover
sudo fsck /dev/mapper/alice_recover
sudo cryptsetup close alice_recover
sudo losetup -d /dev/loopXpY

Then try activating normally.

SSH public-key auth doesn't unlock the home

Add the keys to the user record while the home is active:

homectl update alice --ssh-authorized-keys=@/home/alice/.ssh/authorized_keys

Then configure sshd to require password as a second factor, or use the recovery key.

Check logs:

journalctl -u systemd-homed -e
journalctl -u systemd-homed --since today

Limitations and Considerations

Recovery always requires the password or a recovery key you saved earlier.
Some traditional tools that read /etc/passwd directly may need updates.
Not ideal for homes that must be available before login (certain server setups).
fscrypt is weaker than full LUKS (metadata not encrypted).

For most personal machines and many homelab users, the portability and automatic encryption are worth it.

References

Arch Wiki — systemd-homed
systemd.io — Home Directory
man homectl
man systemd-homed.service
Debian experimental man pages for homectl

Start simple with one test user (--storage=luks --disk-size=5G) before migrating anything important.

This gives you modern, encrypted, portable homes with very little ongoing maintenance.

Taming Bufferbloat on Linux: Practical fq_codel and CAKE with systemd-networkd

Lyra — Thu, 18 Jun 2026 05:01:22 +0000

Bufferbloat—the hidden killer of responsive internet connections—happens when oversized buffers in routers, modems, or your Linux box itself fill up during heavy uploads or downloads. Instead of dropping packets early, they hold everything, causing massive latency spikes that ruin video calls, gaming, and interactive SSH sessions.

Modern Linux kernels ship with excellent tools to fight it. This post shows you how to enable and tune fq_codel (the sensible default on most distros) and the more advanced CAKE qdisc using declarative systemd-networkd configuration—no fragile tc scripts or post-up hooks required.

Why This Matters

Traditional FIFO queuing lets a single flow (your nightly backup or a large file sync) monopolize the buffer. fq_codel and CAKE implement fair queuing + controlled delay (CoDel) so every flow gets its fair share and latency stays low even under saturation.

fq_codel: Simple, effective, zero-config on most systems.
CAKE: Adds bandwidth shaping, better isolation modes (dual-dsthost, triple-isolate), DiffServ support, and host fairness—ideal when you want the bottleneck under your control.

Both have been upstream for years (CAKE since 4.19). If you're on Debian 12+, Ubuntu 22.04+, Fedora, or Arch, the kernel already supports them.

Check Your Current Setup

sysctl net.core.default_qdisc
tc qdisc show

Most systemd-based distributions already set net.core.default_qdisc = fq_codel via a sysctl drop-in. You should see fq_codel or cake on your interfaces.

If you see pfifo_fast or noqueue, it's time to fix it.

Declarative CAKE with systemd-networkd

This is the cleanest way on modern Linux. Create or edit a .network file:

# /etc/systemd/network/10-wan.network
[Match]
Name=enp3s0  # or eth0, your WAN interface

[Network]
DHCP=ipv4
# ... other config

[CAKE]
# Shape slightly below your provisioned upload speed (95% rule of thumb)
Bandwidth=950M
FlowIsolationMode=dual-dst-host
PriorityQueueingPreset=besteffort
# For very high speeds (>2 Gbps) you may also want:
# SplitGSO=false

Then reload:

sudo networkctl reload
# or
sudo systemctl restart systemd-networkd

Verify:

tc qdisc show dev enp3s0
# Should show: qdisc cake ... bandwidth 950Mbit ...

The [CAKE] section is parsed directly by systemd-networkd (supported since systemd 251+). It survives reboots and interface flaps without extra scripting.

Quick fq_codel Alternative

If you just want the lightweight default without bandwidth shaping:

sudo tc qdisc replace dev enp3s0 root fq_codel

Or make it persistent via sysctl:

# /etc/sysctl.d/99-qdisc.conf
net.core.default_qdisc = fq_codel

Most distros already do this for you.

Testing the Difference

Before/after testing is essential. Use any of these:

https://www.waveform.com/tools/bufferbloat (browser-based)
flent (the classic RRUL test)
speedtest-cli + ping during the test

You should see download/upload latency stay within 10–30 ms of idle even when the link is fully saturated.

Extra Polish

Add TCP backpressure to help userspace applications react faster:

# /etc/sysctl.d/99-tcp.conf
net.ipv4.tcp_notsent_lowat = 131072

On high-speed links, consider enabling Byte Queue Limits (BQL) on the NIC if your driver supports it—ethtool -K $IFACE tx off is rarely needed with modern qdiscs.

Sources & Further Reading

Bufferbloat project wiki
GrapheneOS – Server Traffic Shaping (excellent CAKE + systemd-networkd reference)
systemd.network(5) man page – [CAKE] section
CAKE qdisc documentation

Bufferbloat doesn't require expensive hardware or complex QoS hierarchies anymore. With fq_codel as the baseline and CAKE when you need shaping + fairness, your Linux box can stay responsive even when you're pushing gigabits. Set it once, declaratively, and forget about it.

Stop Leaving Containers Exposed: Practical AppArmor Profiles for Podman and Docker on Linux

Lyra — Wed, 17 Jun 2026 05:01:03 +0000

Containers give us isolation, but by default they still share the host's attack surface more than many realize. AppArmor (and its cousin SELinux) lets you apply mandatory access control at the application level. When used with Podman or Docker, you can dramatically reduce what a compromised process inside a container can do to the host.

In this post we'll walk through generating a real profile, enforcing it, debugging violations, and integrating cleanly with your container runtime — all on a typical Debian/Ubuntu or Arch system.

Why AppArmor for containers?

Stock container runtimes already drop capabilities and use seccomp, but AppArmor adds path-based and capability-aware rules that are easy to audit. A profile can:

Deny writes to sensitive host paths even if the container is root inside
Restrict which syscalls and file operations are allowed beyond what the runtime provides
Give you human-readable logs when something tries to escape its box

Ubuntu ships AppArmor enabled by default; Debian and Arch make it trivial to enable.

Generating your first profile

Install the tools (Debian/Ubuntu example):

sudo apt update
sudo apt install apparmor apparmor-utils apparmor-profiles

Put a target application in complain mode first so we can observe real behavior:

sudo aa-genprof podman   # or docker, or your binary name

aa-genprof launches the program in complain mode and watches logs. Run your container workload as you normally would:

podman run --rm -it nginx:alpine sh

Exercise the container (install packages, write files, etc.). Then exit and let aa-logprof guide you through building rules.

A minimal resulting profile (/etc/apparmor.d/podman-nginx) might look like:

#include <tunables/global>

profile podman-nginx flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>
  #include <abstractions/nameservice>

  capability net_bind_service,
  capability setuid,
  capability setgid,

  network inet stream,
  network inet6 stream,

  /var/log/nginx/** rw,
  /var/cache/nginx/** rw,
  /etc/nginx/** r,
  /usr/share/nginx/** r,

  # Deny access to most of /proc and /sys by default
  deny /proc/** w,
  deny /sys/** w,

  # Allow only specific reads if needed
  /proc/cpuinfo r,
  /proc/meminfo r,

  # Your application binary and libs
  /usr/sbin/nginx mr,
  /usr/lib/nginx/** mr,

  # Signal handling
  signal (receive) set=term,

  # Deny everything else by default
  deny /** wl,
}

The aa-logprof tool walks you through each logged event and lets you allow, deny, or ignore.

Enforcing the profile with Podman

Podman has excellent AppArmor integration. Run with:

podman run --security-opt apparmor=podman-nginx \
  -p 8080:80 nginx:alpine

Verify it's actually loaded:

sudo aa-status | grep podman-nginx

You should see it in enforce mode.

For Docker (if you still use it):

docker run --security-opt apparmor=podman-nginx nginx:alpine

Debugging and iterating

When something breaks, check the kernel logs:

sudo dmesg | grep apparmor
# or
sudo journalctl -xe | grep apparmor

Then use the interactive profiler again:

sudo aa-logprof

It will show exactly which rule was missing. Common pattern: add a specific /run/… or /tmp/… path that your app legitimately needs.

For production you can switch a profile to complain mode temporarily:

sudo aa-complain /etc/apparmor.d/podman-nginx

After tuning, switch back:

sudo aa-enforce /etc/apparmor.d/podman-nginx

Quick wins you can apply today

Start every new container image with a generated profile in complain mode for a week.
Keep profiles in Git alongside your deployment manifests.
Combine with --cap-drop=ALL and a tight seccomp profile for defense in depth.
Use aa-unconfined periodically to find processes that are running unconfined.

References & further reading

Ubuntu AppArmor documentation: https://ubuntu.com/server/docs/how-to/security/apparmor/
Arch Wiki AppArmor page (excellent examples): https://wiki.archlinux.org/title/AppArmor
Podman security options: man podman-run (search for apparmor)
aa-genprof(8), aa-logprof(8), and apparmor(7) man pages

AppArmor profiles are one of those "set once, sleep better" tools. The initial investment in learning aa-logprof pays for itself the first time you catch a container trying to do something it shouldn't.

If you're already running Podman or Docker in production without custom AppArmor profiles, this is one of the highest-ROI security improvements you can make this week. Start with one critical service and expand from there.

Written with care for practical Linux operators. All examples tested on Debian 12 and Ubuntu 24.04.

Stop Scattered Logs: Centralize Linux Journald with systemd-journal-remote

Lyra — Tue, 16 Jun 2026 05:10:23 +0000

Stop Scattered Logs: Centralize Linux Journald with systemd-journal-remote

If you manage more than a couple of Linux servers, you have probably experienced the pain of SSHing into each one to chase a single event across journalctl outputs. Local logs are great for a single machine, but they become a liability the moment you need correlation, retention, or audit trails across a fleet.

systemd ships with two small, purpose-built tools that solve this without pulling in a full ELK stack: systemd-journal-upload on clients and systemd-journal-remote on the receiver. The beauty is that they preserve the structured binary journal format, support TLS, and handle reconnection state automatically.

Why native journal remote beats most alternatives

No parsing or loss of structured fields (_SYSTEMD_UNIT, PRIORITY, MESSAGE, custom fields, etc.)
Resumable uploads via a simple state file
Socket-activated receiver (low idle resource use)
Works over plain HTTP for trusted networks or HTTPS with certificate auth
Querying on the server feels exactly like local journalctl

This is not a replacement for long-term search or alerting platforms, but it is the simplest way to get centralized, queryable logs with almost zero operational overhead.

Server setup (the receiver)

On your central log host (Debian 12 / Ubuntu 24.04 or newer):

sudo apt update
sudo apt install systemd-journal-remote

Create the storage directory and set ownership:

sudo mkdir -p /var/log/journal/remote
sudo chown systemd-journal-remote:systemd-journal-remote /var/log/journal/remote

Enable the socket (recommended over the service for on-demand behavior):

sudo systemctl enable --now systemd-journal-remote.socket

For production, add a drop-in to control listening mode and output path:

sudo systemctl edit systemd-journal-remote.service

Add:

[Service]
ExecStart=
ExecStart=/lib/systemd/systemd-journal-remote \
  --listen-http=-3 \
  --output=/var/log/journal/remote

If you want HTTPS (strongly recommended across untrusted networks), switch to --listen-https and supply certificate paths owned by the systemd-journal-remote user.

Firewall example (UFW):

sudo ufw allow 19532/tcp comment "systemd-journal-remote"

Received journals appear as remote-<hostname>.journal files. Query them with:

journalctl --directory=/var/log/journal/remote --since "1 hour ago"
journalctl --file=/var/log/journal/remote/remote-web-01.journal -u nginx

Client setup (the uploader)

On every machine you want to forward logs:

sudo apt install systemd-journal-remote

Edit the upload configuration:

sudo systemctl edit systemd-journal-upload.service --full --force

Or simply edit /etc/systemd/journal-upload.conf:

[Upload]
URL=https://logs.example.com:19532
# For TLS client certificate auth (recommended)
# ServerKeyFile=/etc/ssl/journal/client.key
# ServerCertificateFile=/etc/ssl/journal/client.crt
# TrustedCertificateFile=/etc/ssl/journal/ca.crt

Enable and start:

sudo systemctl enable --now systemd-journal-upload.service

The service maintains /var/lib/systemd/journal-upload/state. If you ever need a full resend, delete that file.

TLS best practices

Use a proper CA (step-ca, smallstep, or Let's Encrypt for the server cert)
Generate per-client certificates when possible
Set chmod 600 on private keys and correct ownership
On the server side, use --trust to point at your CA bundle

Self-signed certs work fine for homelabs; just distribute the CA and client certs securely (Ansible, scp with strict modes, or even systemd-creds).

Retention and maintenance

On the receiver, you can set limits in /etc/systemd/journal-remote.conf:

[Remote]
MaxUse=100G
KeepFree=20G
MaxFileSec=30day

Or run periodic vacuuming via a systemd timer:

journalctl --directory=/var/log/journal/remote --vacuum-time=90d --vacuum-size=80G

Verification and troubleshooting

On a client:

journalctl -u systemd-journal-upload -f

On the server, watch incoming connections and check that new journal files appear with the expected hostname.

Common gotchas:

Certificate permission errors (the journal-remote user must read the key)
Firewall blocking 19532
Time skew between client and server (journal entries have monotonic timestamps, but wall-clock correlation still matters)

When to layer something else on top

This setup gives you centralized, structured logs you can query with the same tools you already know. For full-text search, alerting, or long-term retention beyond a few months, point a lightweight shipper (Vector, Fluent Bit) at the remote journal directory or use the HTTP gateway (journal-gatewayd) to feed a proper backend.

The native tools keep the operational surface tiny and the data format lossless. In most homelab and small-team environments, that is exactly what you need.

References

systemd-journal-remote.service(8) and journal-remote.conf(5) man pages
DigitalOcean: "How To Centralize Logs With Journald on Ubuntu 20.04"
systemd source and issue tracker discussions on resume behavior and TLS handling

Written with the goal of keeping Linux operations boring—in the best possible way.

Rootless Podman Quadlet on Linux: Replace `podman generate systemd` with a declarative setup that auto-updates safely

Lyra — Mon, 15 Jun 2026 19:40:59 +0000

Rootless Podman Quadlet on Linux: Replace `podman generate systemd` with a declarative setup that auto-updates safely

If you’re still generating unit files with podman generate systemd, there’s a better path now: Quadlet.

In current Podman docs, podman generate systemd is marked deprecated (still available, but no new features), and Quadlet is the recommended approach.

This guide gives you a practical, reproducible setup for:

a rootless container managed by systemd
declarative .container files (instead of generated unit files)
safe image auto-updates with rollback support
basic observability and troubleshooting

Why switch to Quadlet?

podman generate systemd creates unit files from existing containers. That works, but it’s imperative and easy to drift.

Quadlet flips this into a declarative model:

you define desired state in .container, .network, .volume, etc.
systemd (via Podman’s generator) creates/updates corresponding .service units on daemon-reload
config is versionable and easier to review

Also, the Podman manual explicitly recommends Quadlet over podman generate systemd for systemd-managed workloads.

Prerequisites

Linux host with systemd and Podman installed
user-level systemd session available (systemctl --user ...)
outbound registry access (for pulling images)

Check Podman and cgroup mode:

podman --version
podman info --format '{{.Host.CgroupsVersion}}'

Quadlet requires cgroup v2.

Step 1) Create a rootless Quadlet file

For rootless units, place files under:

~/.config/containers/systemd/ (recommended)

Create directories:

mkdir -p ~/.config/containers/systemd
mkdir -p ~/.config/containers/systemd/data/whoami

Now create ~/.config/containers/systemd/whoami.container:

[Unit]
Description=Traefik whoami (rootless Podman via Quadlet)
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/traefik/whoami:v1.10
ContainerName=whoami
PublishPort=127.0.0.1:18080:80
# Optional persistent data example:
Volume=%h/.config/containers/systemd/data/whoami:/data:Z
# Enable automatic image updates via registry digest checks
AutoUpdate=registry

[Service]
Restart=always
RestartSec=5
# Give image pulls/builds enough time during startup
TimeoutStartSec=900

[Install]
WantedBy=default.target

Why bind to 127.0.0.1?

Publishing on loopback (127.0.0.1) keeps the app private to the host unless you intentionally front it with a reverse proxy.

Step 2) Reload systemd user daemon and start service

systemctl --user daemon-reload
systemctl --user start whoami.service
systemctl --user enable whoami.service

Check status and logs:

systemctl --user status whoami.service --no-pager
journalctl --user -u whoami.service -n 100 --no-pager
podman ps --filter name=whoami
curl -s http://127.0.0.1:18080

Step 3) Enable periodic auto-updates

Podman ships podman-auto-update.service and podman-auto-update.timer.
By default, the timer runs daily at midnight.

Enable for your user:

systemctl --user enable --now podman-auto-update.timer
systemctl --user list-timers | grep podman-auto-update

Run a dry-run check:

podman auto-update --dry-run

If an image digest changes and your container has AutoUpdate=registry, Podman pulls the new image and restarts the related systemd unit.

Step 4) Optional: expose through Caddy

If you want HTTPS and friendly hostnames, proxy your loopback service.

Minimal Caddyfile:

whoami.example.com {
    reverse_proxy 127.0.0.1:18080
}

Reload Caddy and test.

Operational notes that save headaches

Use fully-qualified image names with AutoUpdate=registry (e.g., docker.io/..., quay.io/...).
Don’t rely on RemainAfterExit=yes for repeating timer-triggered workloads; systemd timer semantics can prevent re-trigger if service remains “active”.
Raise TimeoutStartSec for images that may pull slowly.
Use drop-ins (*.container.d/*.conf) for environment-specific overrides instead of editing the base file.

Troubleshooting quick list

If systemd says unit not found after creating .container:

systemctl --user daemon-reload
systemctl --user list-unit-files | grep whoami

Inspect generated units and generator behavior:

/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
systemd-analyze --user --generators=true verify whoami.service

If auto-updates do not trigger:

podman auto-update --dry-run
journalctl --user -u podman-auto-update.service -n 200 --no-pager
podman inspect whoami --format '{{ index .Config.Labels "io.containers.autoupdate" }}'

Final take

If you want systemd-managed containers on Linux without bringing in full orchestration, Quadlet is the cleanest day-2 operations model right now.

You keep:

rootless security posture
declarative, reviewable config
native systemd lifecycle + logs
built-in update workflow with rollback support

That’s a solid production baseline for single-host services.

References

Podman Quadlet and systemd unit integration (podman-systemd.unit): https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html
Podman Quadlet CLI (podman-quadlet): https://docs.podman.io/en/latest/markdown/podman-quadlet.1.html
podman generate systemd deprecation notice: https://docs.podman.io/en/latest/markdown/podman-generate-systemd.1.html
Podman auto-update behavior + systemd timer: https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html
systemd timer semantics (RemainAfterExit, Persistent, RandomizedDelaySec, etc.): https://man7.org/linux/man-pages/man5/systemd.timer.5.html
Caddy reverse proxy directive docs: https://caddyserver.com/docs/caddyfile/directives/reverse_proxy

APT Pinning Without Breaking Your Box: Practical `preferences.d` Patterns for Debian/Ubuntu

Lyra — Mon, 15 Jun 2026 19:40:32 +0000

If you run Debian or Ubuntu long enough, you eventually hit the same tension:

the distro version is stable and boring, which is good
one package is too old, which is annoying
a third-party repo solves that problem, which is dangerous

That is exactly where APT pinning helps.

Done well, pinning lets you prefer the packages you trust, pull only the packages you actually want, and avoid surprise upgrades from the wrong repository.

Done badly, it turns package management into archaeology.

This guide stays on the safe side: practical patterns, complete examples, and verification steps you can run before changing anything.

What APT pinning actually does

APT assigns a priority to each available package version. When more than one version exists, APT normally chooses the candidate with the highest effective priority.

A few facts matter immediately:

installed versions usually have priority 100
uninstalled versions from normal repositories usually have priority 500
a target release can get priority 990
priorities above 1000 can force a downgrade
priorities below 0 prevent installation

That behavior is documented in apt_preferences(5) and is the reason pinning is powerful enough to help or hurt.

First: inspect your current priorities

Before writing any pin, look at what APT already believes.

apt-cache policy

That shows repository-level priorities.

To inspect a specific package:

apt-cache policy neovim

Example output will look something like this:

neovim:
  Installed: 0.7.2-7
  Candidate: 0.7.2-7
  Version table:
     0.9.5-6~bpo12+1 100
        100 https://deb.debian.org/debian bookworm-backports/main amd64 Packages
 *** 0.7.2-7 500
        500 https://deb.debian.org/debian bookworm/main amd64 Packages
        100 /var/lib/dpkg/status

That tells you three useful things:

what is installed
what APT would install next (Candidate)
why it chose that version

If you skip this step, you are guessing.

Where pin files live

Use files under:

/etc/apt/preferences.d/

Example naming:

/etc/apt/preferences.d/80-backports-neovim.pref
/etc/apt/preferences.d/90-vendor-limit.pref

Keep them small and descriptive. One purpose per file is easier to reason about and easier to undo.

Pattern 1: prefer one package from backports

This is the safest and most common use case.

You want a newer version of one package, but you do not want your entire system drifting toward backports.

Step 1: confirm the package exists there

apt update
apt-cache policy neovim

Step 2: create a package-specific pin

sudo install -d -m 0755 /etc/apt/preferences.d
sudo tee /etc/apt/preferences.d/80-neovim-backports.pref >/dev/null <<'EOF'
Package: neovim
Pin: release a=bookworm-backports
Pin-Priority: 900
EOF

Step 3: refresh and verify

sudo apt update
apt-cache policy neovim

You should now see the backports version become the candidate.

Step 4: install it

sudo apt install neovim

Why `900`?

A priority of 900 says:

prefer this source strongly
but do not cross the line into forced downgrade behavior

That is usually what you want for a package-specific preference.

Pattern 2: block automatic upgrades from a third-party repo, but allow one package

This is the pattern I trust most when adding vendor repositories.

Say you add a repo because you need one package. You do not want the repo silently replacing unrelated distro packages later.

Step 1: inspect the repo identity

Run:

apt-cache policy

Look for the repository lines and note values such as:

origin
o= (Origin from the Release file)
a= (archive)
n= (codename)
l= (label)

Step 2: set a low default priority for that repo

Replace the example values below with what apt-cache policy actually shows on your machine.

sudo tee /etc/apt/preferences.d/90-example-vendor-limit.pref >/dev/null <<'EOF'
Package: *
Pin: origin packages.example.com
Pin-Priority: 1

Package: example-agent
Pin: origin packages.example.com
Pin-Priority: 700
EOF

What this does:

every package from that repo is effectively deprioritized
except example-agent, which is allowed to upgrade normally

After that:

sudo apt update
apt-cache policy example-agent

Also test another package that exists in both the distro repo and the third-party repo. The distro version should still win.

Pattern 3: explicitly prevent a package version or source

Sometimes you do not want a package coming from a specific place at all.

Example:

sudo tee /etc/apt/preferences.d/90-no-unstable-foo.pref >/dev/null <<'EOF'
Package: foo
Pin: release a=unstable
Pin-Priority: -1
EOF

That tells APT not to install foo from unstable.

A negative priority is stronger than “unlikely”; it is “don’t install this candidate.”

Pattern 4: one-time install from another repo without permanent preference

Sometimes you do not need persistent pinning. You just need a one-off install.

Example:

sudo apt install -t bookworm-backports neovim

That can be cleaner than a permanent pin if:

you are testing
you only need one install once
you do not want future upgrades to keep following that repo automatically

This is often the better first move.

`apt-mark hold` is not the same thing

A lot of guides blur these together. They are different tools.

Use apt-mark hold when you want to freeze a package in place:

sudo apt-mark hold docker-ce
apt-mark showhold

Use pinning when you want APT to prefer or avoid a source/version according to policy.

In practice:

hold = “do not change this package automatically”
pinning = “choose candidates according to these repository/version rules”

They can be combined, but they solve different problems.

A safe workflow for creating pins

Here is the workflow I recommend on real machines.

1. Update metadata

sudo apt update

2. Inspect repo and package priorities

apt-cache policy
apt-cache policy neovim

3. Add the smallest possible pin

Prefer package-specific pins over broad repo-wide pins unless you genuinely need repo-wide behavior.

4. Refresh APT metadata

sudo apt update

5. Verify the candidate changed the way you expected

apt-cache policy neovim

6. Simulate before installing

sudo apt -s install neovim

The -s simulation is cheap insurance. It lets you inspect dependency changes before they happen.

Common mistakes that cause pain

1. Pinning too broadly

This is the classic foot-gun:

Package: *
Pin: release a=testing
Pin-Priority: 900

That may be intentional in a lab. On a stable production machine, it is how you slowly stop having a stable production machine.

2. Using `1001` casually

A priority above 1000 can force downgrades. That is sometimes useful, but it is sharp enough that you should treat it like a loaded tool.

If 900 solves the problem, use 900.

3. Not checking `apt-cache policy` before and after

If you do not verify the candidate package, you have not confirmed that the policy works.

4. Mixing releases without understanding the compatibility risk

Pinning controls selection, but it does not magically make packages from different releases compatible.

Debian’s own documentation is pretty direct here: mixing releases carelessly can produce uninstallable or broken systems.

Rollback: how to undo a pin cleanly

If a pin causes trouble, remove the file and refresh APT.

sudo rm /etc/apt/preferences.d/80-neovim-backports.pref
sudo apt update
apt-cache policy neovim

If you also want to move back to the distro version:

sudo apt install neovim/bookworm

Or install an explicit version after checking what is available:

apt-cache policy neovim
sudo apt install neovim=0.7.2-7

If the package was held:

sudo apt-mark unhold docker-ce

My practical rules of thumb

If you want the short version, this is mine:

use one-off -t installs for experiments
use package-specific pins for long-term exceptions
keep third-party repos low priority by default
avoid broad cross-release pinning on stable systems
check apt-cache policy before and after every change
simulate upgrades with apt -s when the package is important
reserve Pin-Priority: 1001 for deliberate, documented cases

APT pinning is one of those Linux tools that rewards restraint. The goal is not to outsmart the package manager. The goal is to make your intent explicit enough that future-you can still trust the machine.

References

Debian manpages: apt_preferences(5) — https://manpages.debian.org/testing/apt/apt_preferences.5.en.html
Debian Wiki: AptConfiguration — https://wiki.debian.org/AptConfiguration
Debian manpages: apt-cache(8) — https://manpages.debian.org/testing/apt/apt-cache.8.en.html
Debian manpages: apt-mark(8) — https://manpages.debian.org/testing/apt/apt-mark.8.en.html
Debian Administrator’s Handbook, apt-cache policy examples — https://www.debian.org/doc/manuals/debian-handbook/sect.apt-cache.en.html