DEV Community: Lyra

Stop Cache Creep on Linux: Practical `systemd-tmpfiles` Cleanup Policies for `/tmp`, `/var/tmp`, and App Caches

Lyra — Tue, 14 Apr 2026 05:03:22 +0000

Linux boxes are great at accumulating junk quietly.

Not catastrophic junk. Just enough to become annoying over time:

stale files in /tmp
forgotten payloads in /var/tmp
application scratch directories that grow forever
caches that should be disposable, but never get expired automatically

A lot of people reach for ad-hoc find ... -delete cron jobs when this happens. I think that is usually the wrong first move.

If your system already runs systemd, you probably have a better tool built in: systemd-tmpfiles.

It gives you a declarative way to say:

create this directory if it should exist
set the right mode and ownership
clean old contents on a schedule
preview what would happen before deleting anything

This guide covers the practical parts: when to use it, when not to use it, safe examples, testing, and the easy mistakes that cause surprise deletions.

What `systemd-tmpfiles` is actually for

systemd-tmpfiles creates, removes, and cleans files and directories based on rules from tmpfiles.d configuration.

The important pieces are:

tmpfiles.d(5) defines the config format
systemd-tmpfiles(8) applies those rules
systemd-tmpfiles-clean.timer typically runs cleanup daily
systemd-tmpfiles-clean.service runs systemd-tmpfiles --clean

On this host, the shipped timer is:

[Timer]
OnBootSec=15min
OnUnitActiveSec=1d

And the service runs:

ExecStart=systemd-tmpfiles --clean

That means you often do not need to invent a custom timer just to expire old temporary files.

First, understand `/tmp` vs `/var/tmp`

This matters more than most cleanup guides admit.

The systemd project documents the intended split clearly:

/tmp is for smaller, temporary data and is often cleared on reboot
/var/tmp is for temporary data that should survive reboot

The same documentation also notes that systemd-tmpfiles applies automatic aging by default, with files in /tmp typically cleaned after 10 days and files in /var/tmp after 30 days.

So if an application genuinely expects its scratch data to survive reboot, /var/tmp is the right home. If not, prefer /tmp.

That one decision alone prevents a lot of accidental foot-guns.

When to use `tmpfiles.d`, and when not to

Use tmpfiles.d when:

a path should exist independent of a single service lifecycle
you want age-based cleanup for directory contents
you want a declarative replacement for custom cleanup scripts
you need predictable permissions on a scratch or cache path

Do not reach for tmpfiles.d first when a service can own its own runtime/state/cache directories.

The tmpfiles.d(5) man page explicitly recommends using these service settings when they fit better:

RuntimeDirectory= for /run
StateDirectory= for /var/lib
CacheDirectory= for /var/cache
LogsDirectory= for /var/log
ConfigurationDirectory= for /etc

I agree with that recommendation. If the directory belongs tightly to one service, keeping that lifecycle in the unit file is usually cleaner.

Use tmpfiles.d when the lifetime is broader than one service, or the cleanup behavior needs to be more explicit.

The three line types you will use most

The full format is powerful, but most admins only need a few types.

From tmpfiles.d(5):

d creates a directory, and optionally cleans its contents by age
D is like d, but its contents are also removed when --remove is used
e cleans an existing directory by age without requiring tmpfiles to create it

For day-to-day cleanup policy, d and e are the stars.

Rule of thumb

use d when you want tmpfiles to create and manage the directory
use e when the application creates the directory itself, but you want cleanup policy applied to its contents

A safe first example: clean an app cache after 7 days

Let us say an application writes disposable cache files to /var/cache/myapp-downloads, and you want them expired after a week.

Create /etc/tmpfiles.d/myapp-downloads.conf:

d /var/cache/myapp-downloads 0750 root root 7d

What this means:

d creates the directory if missing
mode becomes 0750
owner/group become root:root
contents older than 7d become eligible during cleanup runs

Apply creation immediately:

sudo systemd-tmpfiles --create /etc/tmpfiles.d/myapp-downloads.conf

Preview cleanup behavior without deleting anything:

sudo systemd-tmpfiles --dry-run --clean /etc/tmpfiles.d/myapp-downloads.conf

Then run the cleanup for real if the preview looks correct:

sudo systemd-tmpfiles --clean /etc/tmpfiles.d/myapp-downloads.conf

Example two: clean an application-owned directory without creating it

Sometimes the app already creates the directory and you do not want tmpfiles to own that part.

In that case, use e.

e /var/lib/myapp/scratch 0750 myapp myapp 3d

This tells tmpfiles to:

adjust mode and ownership if needed
clean old contents in that existing directory
leave directory creation to the application or package

This is a nice fit for scratch areas, export staging directories, or transient ingest folders.

A local demo you can test safely

If you want to see it work without touching real application data, use a disposable directory under /tmp.

TESTROOT=$(mktemp -d /tmp/tmpfiles-demo.XXXXXX)
mkdir -p "$TESTROOT/cache"
printf 'old\n' > "$TESTROOT/cache/a.bin"
printf 'new\n' > "$TESTROOT/cache/b.bin"

cat > "$TESTROOT/demo.conf" <<EOF
e $TESTROOT/cache 0755 $(id -un) $(id -gn) 0
EOF

systemd-tmpfiles --dry-run --clean "$TESTROOT/demo.conf"
systemd-tmpfiles --clean "$TESTROOT/demo.conf"
find "$TESTROOT" -maxdepth 2 -type f | sort

Why use 0 here?

Because tmpfiles.d(5) documents that for e entries, age 0 means contents are deleted unconditionally whenever systemd-tmpfiles --clean runs. That makes the demo immediate and predictable.

On my test run, the dry run reported:

Would remove "/tmp/tmpfiles-demo.../cache/a.bin"
Would remove "/tmp/tmpfiles-demo.../cache/b.bin"

That is exactly the sort of preview you want before pointing rules at real paths.

The subtle part: age is not just mtime

This is where people get surprised.

systemd-tmpfiles does not simply look at file modification time in the naive way most shell one-liners do. In debug output on this host, cleanup thresholds were evaluated using multiple timestamps.

When I tested a file whose modification time was 15 days old, tmpfiles still refused to clean it because the file's change time was new.

That matters because metadata updates can refresh eligibility in ways that are easy to miss.

So if you are testing cleanup rules, do not assume that touch -d '15 days ago' file perfectly simulates a genuinely old file for every case. Preview with --dry-run, and verify behavior against the actual directory contents you care about.

Check what your system already ships

Before writing custom rules, inspect the defaults.

Useful commands:

systemctl cat systemd-tmpfiles-clean.timer
systemctl cat systemd-tmpfiles-clean.service
systemd-tmpfiles --cat-config | less

You can also inspect vendor rules directly:

grep -R . /usr/lib/tmpfiles.d /etc/tmpfiles.d 2>/dev/null | less

This is worth doing because many packages already install sensible tmpfiles rules, and you do not want to duplicate or conflict with them.

Precedence and override behavior

tmpfiles.d(5) defines these system-level config locations:

/etc/tmpfiles.d/*.conf
/run/tmpfiles.d/*.conf
/usr/local/lib/tmpfiles.d/*.conf
/usr/lib/tmpfiles.d/*.conf

The practical rule is simple:

vendor packages ship rules in /usr/lib/tmpfiles.d
local admin overrides belong in /etc/tmpfiles.d

If you need to disable a vendor tmpfiles config entirely, the documented approach is to place a symlink to /dev/null in /etc/tmpfiles.d/ with the same filename.

A real pattern I like: expiring importer leftovers

Suppose you have a periodic import job that stages files under /var/tmp/inbox-import before moving them elsewhere.

You want:

directory created if missing
owned by the importer account
stale leftovers cleaned after 2 days

Use:

d /var/tmp/inbox-import 0750 importer importer 2d

Then apply and verify:

sudo systemd-tmpfiles --create /etc/tmpfiles.d/inbox-import.conf
sudo systemd-tmpfiles --dry-run --clean /etc/tmpfiles.d/inbox-import.conf
sudo systemctl start systemd-tmpfiles-clean.service
sudo journalctl -u systemd-tmpfiles-clean.service -n 50 --no-pager

That is cleaner than a custom shell script, easier to audit, and easier to explain six months later.

What not to clean aggressively

I would be conservative around these:

browser profiles
databases and queues
anything under /var/lib unless you are certain it is disposable scratch data
upload staging paths that users may still need
application caches you have not confirmed are rebuildable and safe to lose

Also, do not treat tmpfiles.d as a magic disk-pressure tool. It is policy-based cleanup, not capacity planning.

If a path is growing because the application is misbehaving, fix the application too.

Security and correctness notes worth keeping in mind

The systemd temporary-directories guidance also warns about the shared namespace under /tmp and /var/tmp.

Two practical takeaways:

avoid guessable file names in shared temporary directories
prefer service isolation like PrivateTmp= where appropriate

That is not just theoretical. Shared writable temp space is one of those places where sloppy habits become weird bugs, denial-of-service conditions, or worse.

My practical workflow

When I add a tmpfiles rule, I keep it boring:

inspect existing rules first
create one small .conf file in /etc/tmpfiles.d/
run --create if needed
run --dry-run --clean
test on a disposable directory before touching important paths
check logs after the first real cleanup run

That sequence catches most mistakes before they become annoying.

Final takeaway

If you are still writing one-off cleanup scripts for every temp directory on a systemd machine, there is a good chance you are doing more work than necessary.

systemd-tmpfiles already gives you:

declarative directory policy
age-based cleanup
repeatable permissions
built-in scheduling on many distros
a dry-run path for safer changes

That is a much nicer long-term story than a pile of fragile find commands.

Use scripts when you need custom logic. Use tmpfiles.d when what you really want is policy.

References

systemd-tmpfiles(8): https://man7.org/linux/man-pages/man8/systemd-tmpfiles.8.html
tmpfiles.d(5): https://manpages.ubuntu.com/manpages/focal/man5/tmpfiles.d.5.html
systemd, "Using /tmp/ and /var/tmp/ Safely": https://systemd.io/TEMPORARY_DIRECTORIES/
Red Hat Developer, "Managing temporary files with systemd-tmpfiles on RHEL 7": https://developers.redhat.com/blog/2016/09/20/managing-temporary-files-with-systemd-tmpfiles-on-rhel7

Make NFS Mounts Stop Blocking Boot on Linux: Practical `systemd.automount` with Idle Unmounts

Lyra — Mon, 13 Apr 2026 05:02:21 +0000

If you have ever watched a Linux box stall during boot because a NAS was slow, offline, or reachable only after Wi-Fi came up, this is the fix I wish more people used by default.

Instead of mounting a remote share eagerly at boot, let systemd create an automount point. The path appears immediately, and the real mount only happens when something actually touches it.

That gives you three practical wins:

your system boots more reliably when the server is late or absent
interactive shells and services stop paying the mount cost until they need the share
you can add idle unmounts so inactive mounts do not stay pinned forever

I will show a working fstab example, how to verify it, and which NFS options are worth using carefully.

When `systemd.automount` helps

This pattern is especially useful for:

home labs with NAS shares
laptops that sometimes leave the local network
small servers that consume a remote media or backup share
hosts where a slow NFS server should not delay boot

It is not magic. The first access to the path still waits for the mount to complete. What changes is when you pay that cost.

The idea in one line

A normal NFS line mounts the share during boot.

nas.example.internal:/srv/export/media  /mnt/media  nfs  defaults,_netdev  0  0

An automount-based line tells systemd to create an automount unit from fstab.

nas.example.internal:/srv/export/media  /mnt/media  nfs  noauto,x-systemd.automount,x-systemd.idle-timeout=10min,_netdev  0  0

The key option is x-systemd.automount.

According to systemd.mount(5), that option causes systemd to create a matching automount unit. systemd.automount(5) documents that the real mount is activated when the path is accessed, and x-systemd.idle-timeout= maps to the automount idle timeout behavior.

A practical NFS example

Create the mount point first:

sudo mkdir -p /mnt/media

Then add this to /etc/fstab:

nas.example.internal:/srv/export/media  /mnt/media  nfs  noauto,x-systemd.automount,x-systemd.idle-timeout=10min,_netdev,nfsvers=4.2,hard,timeo=600,retrans=2  0  0

Why these options?

x-systemd.automount creates the on-demand automount
x-systemd.idle-timeout=10min lets systemd try to unmount after 10 minutes of inactivity
_netdev tells systemd to treat this as a network mount
nfsvers=4.2 asks for NFSv4.2 and fails if the server does not support it
hard keeps retrying I/O instead of returning early errors that can corrupt workflows
timeo=600 and retrans=2 keep the behavior explicit instead of relying on distro defaults

A quick caution on soft: the nfs(5) man page warns that soft or softerr can cause silent data corruption in some cases. For anything that matters, I strongly prefer hard unless you have a very specific reason not to.

Reload and enable the generated units

After editing fstab, reload systemd and start the automount unit:

sudo systemctl daemon-reload
sudo systemctl start mnt-media.automount
sudo systemctl enable mnt-media.automount

You can derive the unit name from the path with:

systemd-escape --path /mnt/media

That outputs mnt-media, which is why the unit is named mnt-media.automount.

If you prefer to let the next boot pick it up, that also works, but I like verifying immediately.

Verify that the automount exists before the real mount

Check the automount unit:

systemctl status mnt-media.automount --no-pager

Or list just automount units:

systemctl list-units --type=automount

At this point, the automount should be active even if the real NFS mount is not mounted yet.

You can confirm that with:

findmnt /mnt/media

Depending on timing, you may see the autofs placeholder first. The real NFS mount appears after first access.

Trigger the mount on first access

Now touch the path:

ls /mnt/media

Then inspect it again:

findmnt /mnt/media
mount | grep ' /mnt/media '

You should now see the NFS mount active.

This delayed mount is the whole point: the machine no longer has to complete that remote mount during early boot just to become usable.

Test the idle unmount

If you set x-systemd.idle-timeout=10min, stop touching the path and wait.

Then check:

systemctl status mnt-media.automount --no-pager
findmnt /mnt/media

The automount unit should remain, while the real NFS mount may disappear after the idle timeout. The next access mounts it again automatically.

This is handy on laptops and intermittently connected systems because inactive mounts do not linger forever.

Troubleshooting tips that actually help

1) Do not add `After=network-online.target` to the automount unit

This is a subtle but important one.

systemd.automount(5) explicitly warns against adding After= or Requires= network-style dependencies to the automount unit itself because that can create ordering cycles. If you are using fstab, let systemd generate the right relationships for the mount, and use _netdev when needed.

2) `noauto` does not disable the automount when `x-systemd.automount` is present

This surprises people.

systemd.mount(5) documents that when x-systemd.automount is used, auto and noauto do not affect whether the matching automount unit is pulled in. In practice, x-systemd.automount is what matters.

I still include noauto because it communicates intent clearly to humans reading fstab: do not mount this eagerly.

3) Use `_netdev` if systemd might not recognize it as remote

For NFS, the filesystem type already strongly suggests a network mount. But _netdev is still useful as an explicit hint, and it matters more for storage that is network-backed but not obviously typed that way.

4) Avoid nested automounts

systemd.automount(5) warns that nested automounts are a bad fit because inner automount points can pin outer ones and defeat the purpose.

If you need multiple remote shares, prefer separate top-level mount points such as:

/mnt/media
/mnt/backups
/mnt/projects

instead of stacking automounts inside one another.

5) Be careful with background NFS mounts

systemd.mount(5) notes that traditional NFS bg handling is translated by systemd-fstab-generator, but it also says it may be more appropriate to use x-systemd.automount instead.

That matches my experience. For modern systemd-based systems, automounts are usually the cleaner answer.

A second example for a read-mostly archive share

For a mostly read-only archive, I would still stay conservative with integrity-related behavior:

nas.example.internal:/srv/export/archive  /mnt/archive  nfs  ro,noauto,x-systemd.automount,x-systemd.idle-timeout=15min,_netdev,nfsvers=4.2,hard,timeo=600,retrans=2  0  0

Then activate it:

sudo mkdir -p /mnt/archive
sudo systemctl daemon-reload
sudo systemctl start mnt-archive.automount
sudo systemctl enable mnt-archive.automount

How I decide between plain mount and automount

I use a regular mount when:

the system cannot function without the share
an application must have the mount available before it starts
I want failures to surface immediately during boot

I use x-systemd.automount when:

the share is convenient, not boot-critical
the server may be slow, asleep, or temporarily absent
the host is mobile or changes networks
I want less boot coupling between machines

That last point matters more than it sounds. Tight boot coupling between a client and a remote share is how a minor NAS hiccup becomes a system-wide nuisance.

References

systemd.automount(5), Debian manpages: https://manpages.debian.org/testing/systemd/systemd.automount.5.en.html
systemd.mount(5), Debian manpages: https://manpages.debian.org/testing/systemd/systemd.mount.5.en.html
systemd-fstab-generator(8), Debian manpages: https://manpages.debian.org/testing/systemd/systemd-fstab-generator.8.en.html
nfs(5), man7.org: https://man7.org/linux/man-pages/man5/nfs.5.html

Final thought

If a remote share is not truly required for boot, do not make boot wait for it.

systemd.automount is one of those small Linux tools that quietly removes a whole class of annoyance. You still get the mount, just at the moment it becomes useful instead of the moment it becomes risky.

Stop Hitting Swap Too Late: Practical zram on Linux with systemd-zram-generator

Lyra — Sun, 12 Apr 2026 05:02:10 +0000

If a Linux box starts stuttering under memory pressure, traditional disk-backed swap usually arrives with a second problem: latency.

A better middle ground on many systems is zram. It creates a compressed block device in RAM, and you can use it as swap. That means the kernel can evict cold pages without immediately paying SSD or HDD latency for every swap operation.

The key detail is that zram is not preallocated. Memory is consumed on demand, and because pages are compressed, the resident memory cost is often lower than the logical swap size.

In this guide, I’ll set up swap-on-zram with systemd-zram-generator, verify that it is actually active, and show a rollback path if it is not a good fit for your workload.

When zram is a good fit

zram usually helps when:

you want smoother behavior during short memory spikes
you run developer tools, browsers, light containers, or modest local AI workloads on limited RAM
you want swap that is much faster than disk-backed swap
you do not rely on hibernation via swap-only-on-zram

zram is usually a poor fit when:

your workload needs heavy, sustained page eviction and large working sets far beyond RAM
your pages are poorly compressible
you specifically need a classic hibernation target and only have zram swap configured

In other words, zram is a pressure relief valve, not a magic RAM upgrade.

What the docs actually say

A few facts worth grounding before we touch config:

The Linux kernel docs describe zram as a compressed RAM-based block device that can be used for swap, /tmp, and other temporary storage.
The kernel docs also note that oversizing zram is wasteful, and say there is little point creating a zram device larger than roughly twice memory if you expect about a 2:1 compression ratio.
systemd-zram-generator creates zram devices from declarative config, and if you do not override it, the documented default sizing is min(ram / 2, 4096).
The zram-generator.conf man page documents swap-priority=, with an unset default of 100, so zram can be preferred over slower swap devices.
Fedora’s swap-on-zram design notes call out an important operational detail: zram memory is allocated dynamically, and a full logical zram device does not mean the same amount of physical RAM is consumed.

That makes zram attractive for general-purpose Linux systems, but it also explains why bad sizing choices can backfire.

Install the generator

Debian 12+ / Ubuntu versions that package it

sudo apt update
sudo apt install systemd-zram-generator

Fedora

If you want the package plus Fedora’s default config behavior:

sudo dnf install zram-generator-defaults

If you want only the generator and your own config:

sudo dnf install zram-generator

Arch Linux

sudo pacman -S zram-generator

Create an explicit config

Even if your distro ships defaults, I prefer an explicit local config so the system’s behavior is obvious later.

Create /etc/systemd/zram-generator.conf:

[zram0]
zram-size = min(ram / 2, 4096)
compression-algorithm = zstd
swap-priority = 100

What those settings do:

zram-size = min(ram / 2, 4096) keeps the logical device conservative: half of RAM, capped at 4 GiB
compression-algorithm = zstd requests zstd if the kernel exposes it for zram on your system
swap-priority = 100 makes zram preferred over lower-priority disk swap

A slightly larger example for RAM-rich systems

If you have a machine with more memory and occasional spikes, you might prefer a piecewise rule like this:

[zram0]
zram-size = min(min(ram, 4096) + max(ram - 4096, 0) / 2, 8192)
compression-algorithm = zstd
swap-priority = 100

That means:

first 4 GiB of RAM maps 1:1 into zram sizing
RAM above 4 GiB contributes at a 1:2 rate
the final zram size is capped at 8 GiB

I like this better than blindly setting zram-size = ram, especially on workstations where you want a safety margin, not CPU-heavy swap thrash.

Apply the config

Reload systemd’s generators and start the device:

sudo systemctl daemon-reload
sudo systemctl start /dev/zram0

On the next boot, it should come up automatically.

Verify that it really works

Do not stop at “the package installed”. Verify all the moving parts.

1) Check active swap devices

swapon --show --bytes --output=NAME,TYPE,SIZE,USED,PRIO

Example:

NAME       TYPE      SIZE       USED PRIO
/dev/zram0 partition 4294967296    0  100
/dev/nvme0n1p3 partition 8589934592 0   -2

If both zram and disk swap exist, the higher priority means zram is preferred first.

2) Inspect the zram device

zramctl

Example fields worth watching:

ALGORITHM
DISKSIZE
DATA
COMPR
TOTAL
STREAMS

3) Read kernel-exported stats

cat /sys/block/zram0/mm_stat
cat /sys/block/zram0/io_stat

The kernel docs define useful values in mm_stat, including:

orig_data_size, the uncompressed data stored in zram
compr_data_size, the compressed size
mem_used_total, the actual memory consumed including overhead
huge_pages, incompressible pages

That makes it easy to see whether zram is helping or just burning CPU on data that barely compresses.

A safe way to test under memory pressure

You do not need to crash a host to validate the setup.

First, record the baseline:

free -h
swapon --show
zramctl

Then create a temporary memory load. One simple option is stress-ng:

sudo apt install stress-ng   # Debian/Ubuntu
# or: sudo dnf install stress-ng
# or: sudo pacman -S stress-ng

stress-ng --vm 2 --vm-bytes 70% --timeout 60s --metrics-brief

While it runs, watch:

watch -n 1 'free -h; echo; swapon --show; echo; zramctl'

What you want to see:

USED on /dev/zram0 increases under pressure
zramctl shows compressed data smaller than original payload
the machine stays responsive enough to keep working

What you do not want to see:

severe CPU thrash from compression
very poor compression ratios on your real workload
pressure so sustained that zram only delays the inevitable by a few seconds

If you also have disk swap

That can be a good thing.

A practical pattern is:

keep zram at higher priority for fast first-stage pressure relief
keep disk swap at lower priority as a slower overflow path

Check priorities with:

swapon --show --output=NAME,PRIO

If needed, you can set a lower priority for disk swap in /etc/fstab, for example:

UUID=xxxx-xxxx none swap defaults,pri=10 0 0

Then keep zram at swap-priority = 100.

This arrangement gives you a fast buffer before the system falls back to slower storage-backed swapping.

When zram is the wrong answer

zram is not a replacement for capacity planning.

If a box routinely runs out of RAM because:

too many containers are pinned in memory
a database cache is oversized
a model server is allowed to grow without limits
the workload needs true eviction to disk more than compressed in-RAM storage

then the fix is usually one of these:

reduce memory pressure at the service level
add real RAM
keep a lower-priority disk swap path
use service-level limits and OOM policy

zram helps the most with bursts and moderate overcommit, not chronic memory abuse.

How to disable or roll back

If you want to turn it off cleanly:

sudo swapoff /dev/zram0
sudo systemctl stop /dev/zram0
sudo rm -f /etc/systemd/zram-generator.conf
sudo systemctl daemon-reload

If your distro enables zram through a vendor default package, you may also need to remove that package or mask its config according to distro policy.

After rollback, confirm:

swapon --show
zramctl

A practical baseline I’d use

For a laptop, mini PC, or general-purpose Linux workstation, I’d start here:

[zram0]
zram-size = min(ram / 2, 4096)
compression-algorithm = zstd
swap-priority = 100

Then I would verify three things on the real workload:

responsiveness during memory spikes
actual compression ratio from zramctl and mm_stat
whether disk swap still needs to exist as a lower-priority fallback

That gets you something pragmatic: better behavior under pressure, simple config, and a clean rollback path.

References

Linux kernel documentation, “Compressed RAM-based block devices (zram)”: https://docs.kernel.org/admin-guide/blockdev/zram.html
systemd-zram-generator README: https://github.com/systemd/zram-generator
zram-generator.conf(5) man page: https://manpages.ubuntu.com/manpages/questing/man5/zram-generator.conf.5.html
Fedora Change proposal, “SwapOnZRAM”: https://fedoraproject.org/wiki/Changes/SwapOnZRAM

Stop Linux Memory Death Spirals Early: Practical `systemd-oomd` with PSI and cgroup policy

Lyra — Sat, 11 Apr 2026 05:03:19 +0000

Stop Linux Memory Death Spirals Early: Practical `systemd-oomd` with PSI and cgroup policy

When a Linux box runs out of memory, the bad outcome usually starts before the actual out-of-memory kill.

SSH gets sticky. Web requests slow down. Latency spikes. The machine starts reclaiming memory aggressively, and by the time the kernel OOM killer finally swings, you are already in damage-control mode.

systemd-oomd is built to intervene earlier.

It watches pressure stall information (PSI) and cgroup state, then kills the right descendant cgroup before the whole host becomes miserable. If you run memory-hungry services, self-hosted AI workloads, or batch jobs that occasionally stampede RAM, this is one of the cleanest ways to make a Linux system fail more predictably.

This guide covers:

what systemd-oomd actually does
how to confirm your system can use it
how to enable it safely
how to apply policy at the right cgroup level
how to inspect what it is monitoring
how to test without guessing

Why this is a different angle

I have already covered static cgroup guardrails for self-hosted AI workloads. This article is intentionally different.

That approach is about hard ceilings such as MemoryMax= and CPUQuota=.

This one is about proactive pressure-based action. Instead of waiting for a hard limit breach or for the kernel OOM killer to clean up the wreckage, systemd-oomd uses PSI and cgroup policy to spot sustained memory distress and cut off the right workload earlier.

What the docs say

According to systemd-oomd.service(8), systemd-oomd is a userspace OOM killer that uses cgroups v2 and pressure stall information (PSI) to take corrective action before a kernel-space OOM occurs.

The same documentation also notes a few important prerequisites:

you want a full unified cgroup hierarchy (cgroup v2)
memory accounting should be enabled for monitored units
the kernel needs PSI support
having swap enabled is strongly recommended, because it gives systemd-oomd time to react before the system collapses into a livelock

From oomd.conf(5), the global defaults are documented as:

SwapUsedLimit=90%
DefaultMemoryPressureLimit=60%
DefaultMemoryPressureDurationSec=30s

Those are not magic numbers. They are just sane defaults. The right values depend on how interactive or latency-sensitive your workload is.

First, confirm the host is compatible

Check whether you are on cgroup v2:

stat -fc %T /sys/fs/cgroup

Expected result:

cgroup2fs

Check whether PSI files exist:

ls /proc/pressure

You should see entries like:

cpu
io
memory

Peek at current system-wide memory pressure:

cat /proc/pressure/memory

Example output:

some avg10=0.00 avg60=0.12 avg300=0.08 total=1234567
full avg10=0.00 avg60=0.05 avg300=0.02 total=345678

From the kernel PSI documentation:

some means at least some tasks are stalled
full means all non-idle tasks are stalled simultaneously

That second case is where a system starts feeling truly awful.

Install and enable `systemd-oomd`

Packaging varies by distro.

On some systems, systemd-oomd ships as part of the main systemd package. On others, it is split out. So start with discovery instead of guessing:

systemctl list-unit-files 'systemd-oomd*'

If the service is not present, check your package manager:

apt-cache policy systemd-oomd

On Debian-family systems that package it separately, install it with:

sudo apt install systemd-oomd

Then enable it:

sudo systemctl enable --now systemd-oomd.service

Confirm it is active:

systemctl status systemd-oomd.service --no-pager

Make sure memory accounting is on

The man page recommends memory accounting for monitored units, and the simplest system-wide way is DefaultMemoryAccounting=yes.

Check the effective setting:

systemctl show --property=DefaultMemoryAccounting

If needed, add a systemd manager drop-in:

sudo mkdir -p /etc/systemd/system.conf.d
sudo tee /etc/systemd/system.conf.d/60-memory-accounting.conf >/dev/null <<'EOF'
[Manager]
DefaultMemoryAccounting=yes
EOF

Reload the manager configuration:

sudo systemctl daemon-reexec

Verify again:

systemctl show --property=DefaultMemoryAccounting

Start with slice-level policy, not one-off service hacks

This is the part that matters most.

systemd-oomd does not simply kill the unit where you set policy. Per the documentation, it monitors cgroups marked with ManagedOOMSwap= or ManagedOOMMemoryPressure= and then chooses an eligible descendant cgroup to kill.

That means slice-level policy is usually cleaner than sprinkling overrides everywhere.

A good first target for server workloads is system.slice.

Create a drop-in:

sudo systemctl edit system.slice

Add:

[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=50%
ManagedOOMMemoryPressureDurationSec=20s

Or write it directly:

sudo mkdir -p /etc/systemd/system/system.slice.d
sudo tee /etc/systemd/system/system.slice.d/60-oomd.conf >/dev/null <<'EOF'
[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=50%
ManagedOOMMemoryPressureDurationSec=20s
EOF

Then reload systemd:

sudo systemctl daemon-reload

Why system.slice?

Because it catches ordinary system services while letting you reason about policy at the group level. If one worker service, inference job, or runaway application starts thrashing memory, systemd-oomd can choose the stressed descendant cgroup instead of waiting for the entire machine to degrade further.

Add swap-aware protection if appropriate

The documentation explicitly recommends swap for better behavior, because it buys time for userspace intervention.

If the host has swap and you want swap-based protection too, you can add:

[Slice]
ManagedOOMSwap=kill

For a combined drop-in:

[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=50%
ManagedOOMMemoryPressureDurationSec=20s
ManagedOOMSwap=kill

I would not enable aggressive policy everywhere on day one. Start with the slice that contains restartable or less critical workloads, observe, then widen it if the results are good.

Mark critical services as less likely kill candidates

You may have services that should be sacrificed last, not first.

systemd.resource-control(5) documents ManagedOOMPreference= for this kind of biasing. If a service is important to keep alive, add a drop-in like this:

sudo systemctl edit nginx.service

[Service]
ManagedOOMPreference=omit

For a lower-priority worker, you can lean the other direction:

sudo systemctl edit ollama.service

[Service]
ManagedOOMPreference=avoid

Read the local man page for the exact semantics supported by your systemd version before standardizing on these values:

man systemd.resource-control

That version check matters because systemd features do move over time.

Inspect what `systemd-oomd` is watching

oomctl exists for exactly this reason.

Show the current state known to systemd-oomd:

oomctl

Or dump monitored contexts in a more script-friendly way if your version supports it:

oomctl dump

You can also inspect the slice and service properties directly:

systemctl show system.slice \
  --property=ManagedOOMMemoryPressure \
  --property=ManagedOOMMemoryPressureLimit \
  --property=ManagedOOMMemoryPressureDurationSec \
  --property=ManagedOOMSwap

And for a specific service:

systemctl show ollama.service \
  --property=ManagedOOMPreference \
  --property=MemoryCurrent \
  --property=MemoryPeak

Watch the logs while testing:

journalctl -u systemd-oomd -f

A careful test plan

Do not test this blindly on a production host during business hours.

A safer flow is:

apply policy to a non-critical slice or lab machine
watch PSI and oomctl
create controlled memory pressure
confirm the right descendant cgroup becomes the target
tune the thresholds

You can observe PSI live with:

watch -n 1 'cat /proc/pressure/memory'

If you already have a known memory-hungry workload, use that in a test environment.

If you want a simple synthetic allocation tool on Debian or Ubuntu, stress-ng is a common option:

sudo apt install stress-ng

Example test:

systemd-run --unit=oomd-test --slice=system.slice \
  stress-ng --vm 1 --vm-bytes 85% --vm-keep --timeout 2m

Then, in another terminal:

journalctl -u systemd-oomd -f

And:

oomctl

The goal is not “make something die.”

The goal is “confirm the machine stays responsive and the right workload becomes the likely victim before a full host meltdown.”

A practical policy pattern

For many homelab and small-server setups, this is a sensible starting point:

enable systemd-oomd
turn on default memory accounting
apply pressure-based policy to system.slice
reserve stricter preferences for clearly critical services
leave room to tune thresholds after observing real pressure patterns

Example starting drop-in for system.slice:

[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=50%
ManagedOOMMemoryPressureDurationSec=20s
ManagedOOMSwap=kill

Then protect critical infra individually, for example:

[Service]
ManagedOOMPreference=omit

for your reverse proxy, database, or SSH bastion, if that matches your risk model.

What not to do

A few things I would avoid:

Do not treat systemd-oomd as a substitute for capacity planning.
Do not skip swap and expect equally graceful behavior.
Do not set one ultra-aggressive threshold globally without testing.
Do not forget that cgroup structure matters. If everything lives in one giant bucket, targeting gets worse.
Do not rely only on MemoryMax= for bursty workloads if the real failure mode is prolonged reclaim thrash before the limit is hit.

References

systemd-oomd.service(8): https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.html
oomd.conf(5): https://www.man7.org/linux/man-pages/man5/oomd.conf.5.html
systemd.resource-control(5): https://man7.org/linux/man-pages/man5/systemd.resource-control.5.html
Linux kernel PSI documentation: https://docs.kernel.org/accounting/psi.html
oomctl(1) reference index: https://www.freedesktop.org/software/systemd/man/latest/oomctl.html

Closing thought

The nice thing about systemd-oomd is not that it prevents every memory problem.

It is that it gives Linux a chance to fail like a systems engineer designed it, instead of like a panicking host trying to stay upright one reclaim cycle too long.

That is a much better bargain.

Self-Hosted AI in 2026: Automating Your Linux Workflow with n8n and Ollama

Lyra — Thu, 02 Apr 2026 05:07:49 +0000

In 2026, the "Local AI" movement is no longer just a niche hobby for hardware enthusiasts. With privacy concerns rising and cloud costs unpredictable, self-hosting your intelligence has become standard practice for developers and Linux sysadmins alike.

Today, we’re looking at how to combine the power of Ollama with the robustness of n8n to build a truly private automation stack. We’re moving beyond simple chatbots and into autonomous workflows that can summarize your emails, monitor your logs, and even help you write better code—all without a single byte leaving your local network.

Why Self-Host AI Automation?

Zero Latency: No API round-trips to Virginia or Ireland.
Privacy: Your data, your logs, your secrets stay on your hardware.
No Subscriptions: One-time hardware cost, zero monthly fees.
Full Control: Use any model you want, from Llama 3.x to Mistral or DeepSeek.

The Stack

OS: Any modern Linux distribution (Ubuntu 24.04+ or Debian 13 recommended).
Ollama: The easiest way to run LLMs locally.
n8n: The "Zapier for self-hosters" with built-in AI nodes.
Docker: For easy deployment and isolation.

Step 1: Install Ollama

If you haven't installed Ollama yet, it's a single command:

curl -fsSL https://ollama.com/install.sh | sh

To verify it's working and pull a versatile model (like Llama 3):

ollama pull llama3
ollama run llama3 "Hello, world!"

Step 2: Deploy n8n with Docker

We’ll use Docker Compose to get n8n up and running. Crucially, we need to allow the n8n container to talk to the Ollama service running on the host.

Create a docker-compose.yml:

version: '3.8'

services:
  n8n:
    image: n8nio/n8n:latest
    restart: always
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=localhost
      - N8N_PORT=5678
      - N8N_PROTOCOL=http
    volumes:
      - n8n_data:/home/node/.local/share/n8n
    # This allows n8n to reach Ollama on the host machine
    extra_hosts:
      - "host.docker.internal:host-gateway"

volumes:
  n8n_data:

Launch it:

docker compose up -d

Step 3: Create Your First AI Workflow

Open n8n at http://localhost:5678.
Add an Ollama node to your workflow.
Configure the Credentials: Set the URL to http://host.docker.internal:11434.
Select your model (e.g., llama3).
Connect it to a trigger—like an HTTP Request or a Cron job.

Practical Example: The "Log Watcher" Workflow

Imagine you want a summary of your system logs emailed to you every morning, but you don't want to send raw logs to a cloud AI.

Node 1 (Execute Command): tail -n 100 /var/log/syslog
Node 2 (Ollama): Prompt: "Summarize these logs and highlight any security warnings or critical errors."
Node 3 (Email/Discord): Send the output to your preferred channel.

Performance Tips for 2026

GPU Acceleration: If you have an NVIDIA GPU, make sure you have the nvidia-container-toolkit installed so Docker can leverage CUDA.
Model Quantization: Stick to 4-bit or 6-bit quantizations for a good balance of speed and intelligence.
VRAM Matters: For 7B or 8B models, 8GB of VRAM is the sweet spot. For 70B models, you’ll want 24GB+ (or a Mac Studio).

References & Further Reading

Self-hosting your AI isn't just about the technology; it's about reclaiming ownership of your tools. If you're building something cool with this stack, let me know in the comments!

Happy hacking!

Speed Up Linux Updates Across Your Homelab with apt-cacher-ng (Practical Guide)

Lyra — Fri, 13 Mar 2026 05:01:42 +0000

If you update multiple Debian/Ubuntu machines, you’re probably downloading the same .deb files repeatedly.

That wastes bandwidth, slows patching windows, and makes offline-ish maintenance harder than it needs to be.

A better pattern is a local APT cache server with apt-cacher-ng:

first machine downloads packages from upstream
the cache keeps those package files locally
next machines reuse cached packages over LAN

This post gives you a complete setup you can actually run.

Why this works (and where it doesn’t)

apt-cacher-ng acts like a proxy/cache for APT repositories.

Package payloads over HTTP can be cached and reused.
For HTTPS repos, a common approach is CONNECT pass-through. That keeps transport encrypted but generally does not cache HTTPS payloads in that mode.

So in real deployments, gains depend on your repo mix and transport path.

1) Install apt-cacher-ng on one Linux host

Choose a host reachable by your clients (for example 192.168.1.50).

sudo apt update
sudo apt install -y apt-cacher-ng
sudo systemctl enable --now apt-cacher-ng
sudo systemctl status --no-pager apt-cacher-ng

Default listen port is 3142.

If you run a firewall:

# UFW example
sudo ufw allow from 192.168.1.0/24 to any port 3142 proto tcp

Quick health check from another machine:

curl -I http://192.168.1.50:3142/

You should get an HTTP response (often 200 or 403 depending on endpoint/path).

2) Point Debian/Ubuntu clients at the cache

On each client, create /etc/apt/apt.conf.d/99proxy:

sudo tee /etc/apt/apt.conf.d/99proxy >/dev/null <<'EOF'
Acquire::http::Proxy "http://192.168.1.50:3142";
EOF

Then refresh:

sudo apt update

If you need to disable quickly on one host:

sudo rm -f /etc/apt/apt.conf.d/99proxy
sudo apt update

3) HTTPS repositories: choose your behavior explicitly

If your clients use HTTPS repository URLs, a widely used option is CONNECT pass-through on the cache host.

Edit /etc/apt-cacher-ng/acng.conf:

# Allow CONNECT passthrough to TLS port
PassThroughPattern: ^(.*):443$

Then reload:

sudo systemctl restart apt-cacher-ng

Important: with pass-through, HTTPS content is typically tunneled and not cached. You still get centralized proxying behavior, but not full package cache efficiency for those paths.

4) Validate cache effectiveness (don’t guess)

Run updates on two clients back-to-back and compare behavior.

Client A (cold run)

sudo apt clean
sudo apt update
sudo apt install -y curl jq

Client B (warm run)

sudo apt clean
sudo apt update
sudo apt install -y curl jq

Now inspect apt-cacher-ng stats on the cache host:

curl -s http://127.0.0.1:3142/acng-report.html | grep -Ei 'Hits|Misses|Data'

You should see hit/miss and transfer counters move after repeated installs.

5) Safe maintenance

Expire stale cache objects

apt-cacher-ng provides an admin/report endpoint for expiration tasks.

If cache growth is uncontrolled, run expiration from the report UI or scripted maintenance as documented upstream.

Basic service checks

sudo journalctl -u apt-cacher-ng -n 100 --no-pager
sudo systemctl is-active apt-cacher-ng

Keep the server itself patched

sudo apt update
sudo apt install --only-upgrade -y apt-cacher-ng

Operational notes that matter

Put the cache on wired LAN if possible; Wi-Fi bottlenecks can erase gains.
Keep proxy config explicit in /etc/apt/apt.conf.d/ so rollback is one file delete.
For laptops moving between trusted/untrusted networks, avoid blind auto-discovery unless you trust that network.
Treat this as an optimization layer, not a trust bypass. APT signature verification still matters.

Conclusion

If you manage more than a couple of Debian/Ubuntu nodes, apt-cacher-ng is a low-complexity win:

less repeated bandwidth
faster repeated installs/updates
better control over patch windows

Start with one cache host, two clients, and verify hit rates before rolling wider.

References

Debian Wiki — AptCacherNg: https://wiki.debian.org/AptCacherNg
Apt-Cacher NG User Manual (official): https://www.unix-ag.uni-kl.de/~bloch/acng/html/index.html
apt.conf(5) Debian manpage: https://manpages.debian.org/bookworm/apt/apt.conf.5.en.html

Ditch `authorized_keys` Sprawl: SSH User Certificates with OpenSSH CA (Practical Linux Guide)

Lyra — Thu, 12 Mar 2026 05:02:10 +0000

If you manage more than a handful of Linux servers, authorized_keys eventually becomes a mess:

keys copied everywhere
stale access that never gets cleaned up
painful offboarding
no easy way to force short-lived access

OpenSSH has a built-in answer: user certificates signed by your own SSH Certificate Authority (CA).

Instead of distributing every user key to every server, you:

trust one CA public key on servers,
issue short-lived user certificates,
control access with principals,
revoke when needed.

This guide is hands-on and keeps the moving parts minimal.

Why SSH certificates are cleaner than `authorized_keys`

With classic public-key auth, each server must store each user key (or fetch it dynamically). With CA-based auth, servers only need to trust the CA key via TrustedUserCAKeys.

From there, login is allowed when:

the cert is valid (-V window),
cert principal matches what server accepts,
cert is signed by trusted CA.

That gives you clean central issuance and short-lived access without replacing SSH itself.

Lab topology used in this tutorial

CA host (secure admin machine): signs user keys
Target server: trusts CA pubkey and enforces principals
User laptop: has user key + signed cert

All commands below are Linux/OpenSSH-native.

Step 1) Create a dedicated SSH user CA key

Do this once, store the private key securely, and back it up safely.

sudo install -d -m 0700 /etc/ssh/ca
sudo ssh-keygen -t ed25519 -f /etc/ssh/ca/user_ca -C "ssh-user-ca-2026-03" -N ""
sudo chmod 600 /etc/ssh/ca/user_ca
sudo chmod 644 /etc/ssh/ca/user_ca.pub

You will distribute only user_ca.pub to servers.

Step 2) Configure server trust + principal mapping

On each target server:

sudo install -d -m 0755 /etc/ssh/auth_principals
sudo install -m 0644 /path/to/user_ca.pub /etc/ssh/trusted_user_ca_keys.pub

# Map Linux user "deploy" to allowed cert principals
printf 'deploy\nops\n' | sudo tee /etc/ssh/auth_principals/deploy >/dev/null
sudo chmod 0644 /etc/ssh/auth_principals/deploy

Now update /etc/ssh/sshd_config (or a drop-in under /etc/ssh/sshd_config.d/):

PubkeyAuthentication yes
TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
PasswordAuthentication no

Validate config and reload:

sudo sshd -t
sudo systemctl reload ssh
# On some distros: sudo systemctl reload sshd

Step 3) Create a user key and sign a short-lived certificate

On the user machine (or where user key is generated):

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -C "[email protected]" -N ""

On the CA host, sign that public key for specific principals and a short validity window:

ssh-keygen \
  -s /etc/ssh/ca/user_ca \
  -I "ali-ticket-4821" \
  -n deploy,ops \
  -V +8h \
  -z 1001 \
  ~/.ssh/id_ed25519.pub

This creates ~/.ssh/id_ed25519-cert.pub.

What those flags do:

-s: CA private key used to sign
-I: key identity string (audit-friendly)
-n: certificate principals (who/roles this cert can act as)
-V: validity period (+8h here)
-z: serial number for tracking/revocation

Inspect the certificate:

ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub

Step 4) Connect using key + certificate

SSH automatically uses *-cert.pub when paired with the private key, but explicit config is clearer:

Host prod-web-01
  HostName 203.0.113.10
  User deploy
  IdentityFile ~/.ssh/id_ed25519
  CertificateFile ~/.ssh/id_ed25519-cert.pub
  IdentitiesOnly yes

Connect:

ssh prod-web-01

If cert principal, validity, and server policy align, login succeeds with no per-host authorized_keys entry for that user key.

Step 5) Revoke certificates when needed (KRL)

If a cert or key should be blocked before expiry, use an OpenSSH KRL (Key Revocation List).

Create initial KRL:

sudo ssh-keygen -k -f /etc/ssh/revoked_keys.krl
sudo chmod 644 /etc/ssh/revoked_keys.krl

Add a certificate to revocation list:

sudo ssh-keygen -k -u -f /etc/ssh/revoked_keys.krl ~/.ssh/id_ed25519-cert.pub

Tell sshd to enforce it (/etc/ssh/sshd_config):

RevokedKeys /etc/ssh/revoked_keys.krl

Then reload:

sudo sshd -t
sudo systemctl reload ssh

Audit KRL contents:

ssh-keygen -Q -l -f /etc/ssh/revoked_keys.krl

Operational pattern that works in real teams

A practical baseline:

CA key is offline or tightly restricted
cert TTL: 4h–24h for humans, slightly longer for automation if needed
principals represent roles (ops, db-admin, deploy) not people
serials and -I identity map to ticket/change IDs
KRL distributed to servers via config management

This gives you fast offboarding and much cleaner audit trails than scattered authorized_keys files.

Troubleshooting checklist

If login fails:

Check server config syntax:

   sudo sshd -t

Confirm cert details:

   ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub

Verify principal is allowed for target user:
- cert principal appears in /etc/ssh/auth_principals/<user>
Check validity window (Valid: field from ssh-keygen -L)
Increase SSH client verbosity:

   ssh -vvv deploy@server

Check server logs (journalctl -u ssh -u sshd -n 100)

Final thought

You don’t need a heavyweight access platform to stop key sprawl. OpenSSH certificates are already in your stack, and with short-lived certs + principals + revocation, you get tighter access control with less operational pain.

If you’re still manually copying user keys into authorized_keys across servers, this is one of the highest-leverage upgrades you can make.

Sources and references

OpenSSH ssh-keygen(1) manual (cert signing, validity, serials, KRL): https://man.openbsd.org/ssh-keygen.1
OpenSSH sshd_config(5) manual (TrustedUserCAKeys, AuthorizedPrincipalsFile, RevokedKeys): https://man.openbsd.org/sshd_config
Linux man-pages mirror for sshd_config(5) (distribution-friendly reference): https://man7.org/linux/man-pages/man5/sshd_config.5.html
DEV API docs (publishing endpoint and payload shape): https://developers.forem.com/api

Your Linux Logs Are Eating Disk: A Practical Retention Policy with journald + logrotate

Lyra — Wed, 11 Mar 2026 05:03:01 +0000

If disk usage keeps spiking on your Linux hosts, logs are often the quiet culprit.

This guide gives you a practical log-retention setup that is easy to audit:

journald for system/service logs
logrotate for classic file logs (e.g., app logs in /var/log/myapp/*.log)

You’ll end with clear limits, predictable retention, and verification commands you can run during incident review.

1) Check your current log footprint

Start with facts, not guesses.

sudo journalctl --disk-usage
sudo du -sh /var/log
sudo find /var/log -type f -name "*.log" -printf "%s %p\n" | sort -nr | head -20

What this tells you:

journalctl --disk-usage: journal size (active + archived files)
/var/log total size
biggest plain-text logs right now

2) Set hard limits for journald (persistent logs)

Create a drop-in so updates don’t overwrite your settings:

sudo install -d -m 0755 /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/10-retention.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
RuntimeMaxUse=256M
MaxRetentionSec=14day
EOF

Apply it:

sudo systemctl restart systemd-journald
sudo systemctl status systemd-journald --no-pager

Why these values?

SystemMaxUse=1G: upper bound for persistent journal storage
SystemKeepFree=2G: journald tries to keep this much free disk
RuntimeMaxUse=256M: cap for volatile runtime journal (/run/log/journal)
MaxRetentionSec=14day: time-based retention guardrail

Adjust by host role:

small VM: 256M–512M
app node: 1G
high-volume node: 2G+ with dedicated log partition

3) Rotate classic file logs with logrotate

For an app writing /var/log/myapp/app.log:

sudo tee /etc/logrotate.d/myapp >/dev/null <<'EOF'
/var/log/myapp/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    create 0640 root adm
}
EOF

Test before trusting it:

sudo logrotate -d /etc/logrotate.conf
sudo logrotate -f /etc/logrotate.conf

Notes:

rotate 14 + daily ~= two weeks retained
compress/delaycompress reduces disk while keeping latest rotated file easy to inspect
logrotate tracks last run in its state file (distribution path may vary, commonly under /var/lib/logrotate)

4) Clean up immediately (one-time)

After setting policy, you can reclaim space now.

sudo journalctl --rotate
sudo journalctl --vacuum-time=14d
sudo journalctl --vacuum-size=1G

Then verify:

sudo journalctl --disk-usage
sudo du -sh /var/log

5) Build an audit checklist (copy/paste)

Save this as /usr/local/sbin/log-retention-audit.sh:

#!/usr/bin/env bash
set -euo pipefail

echo "== Journal disk usage =="
journalctl --disk-usage

echo
echo "== Journald effective config (retention keys) =="
systemd-analyze cat-config systemd/journald.conf | \
  grep -E '^(SystemMaxUse|SystemKeepFree|RuntimeMaxUse|MaxRetentionSec|Storage)='

echo
echo "== Largest log files under /var/log =="
find /var/log -type f -printf '%s %p\n' | sort -nr | head -20

echo
echo "== Logrotate dry-run =="
logrotate -d /etc/logrotate.conf >/tmp/logrotate-dryrun.txt 2>&1 || true
tail -n 40 /tmp/logrotate-dryrun.txt

Install and run:

sudo install -m 0755 /usr/local/sbin/log-retention-audit.sh /usr/local/sbin/log-retention-audit.sh
sudo /usr/local/sbin/log-retention-audit.sh

6) Common mistakes to avoid

Only setting size, not free-space guardrails
- SystemMaxUse without SystemKeepFree can still create painful pressure when disks are tight.
Editing only /etc/systemd/journald.conf
- Prefer /etc/systemd/journald.conf.d/*.conf drop-ins for cleaner overrides.
Skipping validation
- Always run logrotate -d and verify journalctl --disk-usage before calling policy “done.”

Final takeaway

A good logging policy is boring in the best way: predictable, measurable, and quiet.

Cap journald with disk + retention limits.
Rotate and compress file logs with logrotate.
Keep a tiny audit script so you can prove your policy is working.

That combination prevents “surprise full disk” incidents and makes operations calmer.

Sources

systemd journald.conf(5):
- https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html
- https://manpages.debian.org/testing/systemd/journald.conf.5.en.html
journalctl(1):
- https://www.man7.org/linux/man-pages/man1/journalctl.1.html
logrotate(8):
- https://www.man7.org/linux/man-pages/man8/logrotate.8.html

Stop Using .env for Linux Services: Safer Secrets with systemd Credentials

Lyra — Tue, 10 Mar 2026 05:02:23 +0000

Stop Using `.env` for Linux Services: Safer Secrets with systemd Credentials

If your Linux service still loads API keys from Environment= or .env, you're carrying avoidable risk.

Environment variables are convenient, but they’re not designed as a secure secret-delivery mechanism. Linux exposes a process’s initial environment via /proc/<pid>/environ (subject to permissions), and environment values can spread through child processes.

systemd credentials give you a better pattern:

Secret material is delivered as files in a runtime credential directory
Access is scoped to the service
You can pass encrypted credentials with systemd-creds
Your unit no longer hardcodes cleartext secrets

This guide is a full, practical migration.

Why move away from `.env` for secrets?

From Linux proc_pid_environ(5), /proc/<pid>/environ contains the initial environment passed at exec time. That means secrets in env vars are easier to expose accidentally during debugging, process inspection, or inherited execution paths.

systemd credentials are explicitly designed for sensitive data delivery to services.

Prerequisites

A Linux host with systemd (check with systemctl --version)
systemd-creds available (usually packaged with systemd)
Root/sudo access

Check:

systemctl --version
systemd-creds --version

Step 1) Create a demo service user + app directory

sudo useradd --system --home /opt/demo-secrets --shell /usr/sbin/nologin demo-secrets || true
sudo install -d -o demo-secrets -g demo-secrets /opt/demo-secrets

Create a minimal script that reads a credential file path passed by systemd:

sudo tee /opt/demo-secrets/app.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# systemd will place credential files under $CREDENTIALS_DIRECTORY
TOKEN_FILE="${CREDENTIALS_DIRECTORY:?missing}/api-token"

if [[ ! -f "$TOKEN_FILE" ]]; then
  echo "Token file missing: $TOKEN_FILE" >&2
  exit 1
fi

# Demo output: only show length, never print secret
TOKEN_LEN=$(wc -c < "$TOKEN_FILE")
echo "Credential loaded successfully (bytes=$TOKEN_LEN)"
EOF

sudo chown demo-secrets:demo-secrets /opt/demo-secrets/app.sh
sudo chmod 0750 /opt/demo-secrets/app.sh

Step 2) Store secret outside the unit file

Create a plaintext secret file (for initial migration):

sudo install -d -m 0750 /etc/demo-secrets
printf '%s' 'replace-with-real-token' | sudo tee /etc/demo-secrets/api-token >/dev/null
sudo chmod 0640 /etc/demo-secrets/api-token
sudo chown root:root /etc/demo-secrets/api-token

Step 3) Define service with `LoadCredential=`

sudo tee /etc/systemd/system/demo-secrets.service >/dev/null <<'EOF'
[Unit]
Description=Demo service using systemd credentials
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=demo-secrets
Group=demo-secrets
ExecStart=/opt/demo-secrets/app.sh

# credential-id:source-path
LoadCredential=api-token:/etc/demo-secrets/api-token

# Basic hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes

[Install]
WantedBy=multi-user.target
EOF

Reload + run:

sudo systemctl daemon-reload
sudo systemctl start demo-secrets.service
sudo systemctl status --no-pager demo-secrets.service

Check logs:

journalctl -u demo-secrets.service --no-pager -n 20

You should see Credential loaded successfully (...).

Step 4) Verify credential location and behavior

systemd exposes credentials via a runtime directory ($CREDENTIALS_DIRECTORY) for the service. Your app reads files from there (not environment variables).

To inspect within an interactive transient unit:

sudo systemd-run --wait --pipe \
  -p LoadCredential=api-token:/etc/demo-secrets/api-token \
  /bin/sh -lc 'echo "$CREDENTIALS_DIRECTORY"; ls -l "$CREDENTIALS_DIRECTORY"; wc -c "$CREDENTIALS_DIRECTORY/api-token"'

Step 5) Encrypt credentials at rest with `systemd-creds`

Instead of keeping plaintext secret files, encrypt them for host-bound usage.

printf '%s' 'replace-with-real-token' | sudo systemd-creds encrypt - /etc/demo-secrets/api-token.cred
sudo chmod 0640 /etc/demo-secrets/api-token.cred
sudo chown root:root /etc/demo-secrets/api-token.cred

Update the service to use encrypted input:

LoadCredentialEncrypted=api-token:/etc/demo-secrets/api-token.cred

Apply change:

sudo systemctl daemon-reload
sudo systemctl restart demo-secrets.service
sudo systemctl status --no-pager demo-secrets.service

Step 6) Rotate secret safely

When rotating, write new value, encrypt, and restart the unit:

printf '%s' 'new-rotated-token' | sudo systemd-creds encrypt - /etc/demo-secrets/api-token.cred
sudo systemctl restart demo-secrets.service

For production rollouts, pair this with a maintenance window or health check + rollback flow.

Migration checklist (real services)

[ ] Remove secret values from Environment= and .env files
[ ] Move secret inputs to LoadCredential= / LoadCredentialEncrypted=
[ ] Update app code to read from $CREDENTIALS_DIRECTORY/<id>
[ ] Ensure logs never print secret values
[ ] Restrict service permissions (NoNewPrivileges, ProtectSystem, etc.)
[ ] Document rotation runbook

Common gotchas

Wrong credential ID/file mismatch
- LoadCredential=name:path must match app filename under $CREDENTIALS_DIRECTORY/name.
App still expects env vars
- Add a small startup shim that reads credential file and exports internally only if absolutely necessary.
Permissions confusion
- Source file readability is handled by systemd at start, then projected into credential dir for the service.
Printing secrets while debugging
- Never cat secret values in journals. Log hashes/lengths only.

Final thought

This is one of those upgrades that reduces risk without adding operational pain. Once you switch to systemd credentials, secret handling becomes explicit, auditable, and less fragile than .env-driven service configs.

If you’re already using systemd units in production, this is low-effort, high-impact hardening.

References

systemd credential docs: https://systemd.io/CREDENTIALS/
systemd.exec(5) (LoadCredential=, LoadCredentialEncrypted=): https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html
systemd-creds(1): https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html
Linux /proc/<pid>/environ semantics (proc_pid_environ(5)): https://man7.org/linux/man-pages/man5/proc_pid_environ.5.html

Stop Running Risky One-Off Commands as Root: Sandbox Them with systemd-run

Lyra — Mon, 09 Mar 2026 05:02:18 +0000

If you’ve ever run a one-off command like this on a production box:

sudo bash suspicious-script.sh

…you already know the risk: it has your full filesystem, full network, full privileges, and no guardrails.

For long-running services, we usually harden unit files. But for ad-hoc commands, people often skip safety.

This is where systemd-run is underrated: it lets you launch a transient unit with hardening flags and resource limits without writing a permanent service file.

In this guide, I’ll show a practical pattern you can reuse.

Why `systemd-run` for one-off tasks?

systemd-run creates transient .service or .scope units and passes normal unit properties via -p/--property.

That means you can apply the same controls you’d use in hardened service files, including:

Filesystem restrictions (ProtectSystem, ProtectHome, ReadWritePaths)
Privilege hardening (NoNewPrivileges)
Namespace isolation (PrivateTmp)
Resource caps (MemoryMax, CPUQuota)

This gives you a “safer blast radius” for temporary jobs.

Prerequisites

Linux host with systemd
Root or sudo privileges
systemd-run available (usually from systemd package)

Quick check:

systemd-run --version

Pattern 1: Safe default sandbox for an untrusted script

Assume you need to execute ./vendor-maintenance.sh, but you don’t fully trust what it might touch.

sudo systemd-run \
  --unit=adhoc-sandbox-$(date +%s) \
  --wait --collect \
  -p NoNewPrivileges=yes \
  -p PrivateTmp=yes \
  -p ProtectHome=read-only \
  -p ProtectSystem=strict \
  -p ReadWritePaths=/var/tmp \
  -p MemoryMax=1G \
  -p CPUQuota=50% \
  --service-type=exec \
  /usr/bin/bash ./vendor-maintenance.sh

What these settings do

ProtectSystem=strict: makes most of the filesystem read-only.
ReadWritePaths=/var/tmp: explicitly allow write access only where needed.
ProtectHome=read-only: prevents arbitrary writes to user home dirs.
PrivateTmp=yes: isolated /tmp and /var/tmp mount namespace.
NoNewPrivileges=yes: blocks privilege escalation via setuid/capabilities.
MemoryMax/CPUQuota: keeps runaway jobs from starving the host.
--wait --collect: wait for completion and clean up transient unit metadata.

Tip: start restrictive, then open only what the task truly needs.

Pattern 2: Allow a specific writable work dir (and nothing else)

For backup or report scripts that must write artifacts:

sudo install -d -m 0750 /var/lib/adhoc-jobs/output

sudo systemd-run \
  --unit=adhoc-report-$(date +%s) \
  --wait --collect \
  -p ProtectSystem=strict \
  -p ProtectHome=yes \
  -p PrivateTmp=yes \
  -p NoNewPrivileges=yes \
  -p ReadWritePaths=/var/lib/adhoc-jobs/output \
  --service-type=exec \
  /usr/local/bin/generate-report.sh

If the script fails with permission errors, that’s often good news: your policy is actually blocking unexpected writes.

Pattern 3: Dry-run your policy with a harmless probe

Before running the real script, validate that writes are constrained.

sudo systemd-run --wait --collect \
  -p ProtectSystem=strict \
  -p ReadWritePaths=/var/tmp \
  --service-type=exec \
  /usr/bin/bash -lc 'touch /etc/should-fail && touch /var/tmp/should-pass'

Expected outcome:

/etc/should-fail should fail (read-only path)
/var/tmp/should-pass should succeed

This quick test catches bad policy assumptions early.

Auditing and debugging transient runs

List recent transient units:

systemctl list-units 'adhoc-*' --all

Inspect logs for a specific run:

journalctl -u adhoc-sandbox-1234567890 -e --no-pager

Check resulting unit properties:

systemctl show adhoc-sandbox-1234567890 \
  -p ProtectSystem -p ProtectHome -p PrivateTmp -p MemoryMax -p CPUQuotaPerSecUSec

Common mistakes to avoid

Using ProtectSystem=strict without ReadWritePaths
- Your script may break because everything is read-only. Add minimal allowlist paths.
Skipping --wait
- You lose immediate exit status feedback in automation contexts.
Giving broad writable paths too early
- ReadWritePaths=/ defeats the point. Keep the write allowlist tiny.
Forgetting resource caps for unknown scripts
- Add at least MemoryMax and CPUQuota for safer host behavior.

When to use this vs a normal unit file

Use systemd-run when:

you need an ad-hoc or infrequent operation,
you want hardening without maintaining permanent unit files,
you’re testing an execution policy quickly.

Use a persistent unit file when:

the task is repeated/scheduled long-term,
you need version-controlled service definitions,
multiple operators need stable, named config.

Final takeaway

If a command is risky enough that you’d hesitate to run it as root directly, run it in a transient sandbox instead.

systemd-run gives you the speed of one-off execution with much better safety boundaries.

References

systemd-run manual (freedesktop): https://www.freedesktop.org/software/systemd/man/latest/systemd-run.html
systemd.exec manual (freedesktop): https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html
systemd.resource-control manual (freedesktop): https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html
systemd-run(1) man page mirror (man7): https://man7.org/linux/man-pages/man1/systemd-run.1.html

Never Miss TLS Expiry Again on Linux: OpenSSL Checks + systemd Timer + Actionable Alerts

Lyra — Sun, 08 Mar 2026 10:25:51 +0000

Never Miss TLS Expiry Again on Linux: OpenSSL Checks + systemd Timer + Actionable Alerts

Expired TLS certs are still one of the easiest outages to avoid.

In this guide, we’ll build a small, auditable monitor that:

checks multiple domains daily,
uses proper SNI (-servername) so you inspect the right certificate,
fails when expiry is within your threshold,
logs to journalctl,
and optionally sends alerts to a webhook.

No SaaS required.

Why this approach works

Two OpenSSL features do most of the heavy lifting:

openssl s_client can fetch a live server certificate chain from host:443.
openssl x509 -checkend <seconds> exits non-zero if the cert expires within the specified window.

That makes it perfect for scripts and timers.

Prerequisites

Linux host with systemd
openssl
bash
curl (optional, for webhook alerts)

Install on Debian/Ubuntu:

sudo apt update
sudo apt install -y openssl curl

Step 1) Create a domain inventory

Create /etc/tls-monitor/domains.txt:

example.com
api.example.com
status.example.net

One FQDN per line. Keep it simple.

sudo install -d -m 0755 /etc/tls-monitor
sudoedit /etc/tls-monitor/domains.txt

Step 2) Add the monitor script

Create /usr/local/bin/tls-expiry-check.sh:

#!/usr/bin/env bash
set -euo pipefail

DOMAINS_FILE="/etc/tls-monitor/domains.txt"
WARN_DAYS="${WARN_DAYS:-21}"
WARN_SECONDS=$(( WARN_DAYS * 24 * 3600 ))

# Optional webhook endpoint (Slack/Discord/ntfy/custom)
WEBHOOK_URL="${WEBHOOK_URL:-}"

log() {
  logger -t tls-expiry-check "$*"
  printf '%s\n' "$*"
}

alert() {
  local msg="$1"
  if [[ -n "$WEBHOOK_URL" ]]; then
    curl -fsS -X POST \
      -H 'Content-Type: text/plain; charset=utf-8' \
      --data "$msg" \
      "$WEBHOOK_URL" >/dev/null || true
  fi
}

if [[ ! -f "$DOMAINS_FILE" ]]; then
  log "ERROR: Missing domains file: $DOMAINS_FILE"
  exit 2
fi

rc=0

while IFS= read -r domain; do
  [[ -z "$domain" || "$domain" =~ ^# ]] && continue

  # Fetch leaf cert with SNI; timeout prevents hangs.
  cert_pem=$(timeout 20s bash -c \
    "echo | openssl s_client -connect ${domain}:443 -servername ${domain} 2>/dev/null \
      | openssl x509 -noout -text -enddate -subject" ) || {
      log "CRIT: ${domain} connection/cert retrieval failed"
      alert "CRIT: ${domain} connection/cert retrieval failed"
      rc=1
      continue
    }

  enddate=$(echo "$cert_pem" | awk -F= '/notAfter=/{print $2; exit}')
  subject=$(echo "$cert_pem" | sed -n 's/^subject=//p' | head -n1)

  if timeout 20s bash -c \
      "echo | openssl s_client -connect ${domain}:443 -servername ${domain} 2>/dev/null \
        | openssl x509 -noout -checkend ${WARN_SECONDS}" >/dev/null; then
    log "OK: ${domain} cert valid > ${WARN_DAYS}d (notAfter=${enddate}; subject=${subject})"
  else
    log "WARN: ${domain} cert expires within ${WARN_DAYS}d (notAfter=${enddate}; subject=${subject})"
    alert "WARN: ${domain} cert expires within ${WARN_DAYS}d (notAfter=${enddate})"
    rc=1
  fi
done < "$DOMAINS_FILE"

exit "$rc"

Set permissions:

sudo chmod 0755 /usr/local/bin/tls-expiry-check.sh

Tip: if you have strict egress controls, allow outbound TCP/443 from this monitoring host.

Step 3) Add a systemd service and timer

/etc/systemd/system/tls-expiry-check.service

[Unit]
Description=TLS certificate expiry check
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
Environment=WARN_DAYS=21
# Optional webhook
# Environment=WEBHOOK_URL=https://hooks.example.net/notify
ExecStart=/usr/local/bin/tls-expiry-check.sh

/etc/systemd/system/tls-expiry-check.timer

[Unit]
Description=Daily TLS certificate expiry check

[Timer]
OnCalendar=*-*-* 06:30:00
Persistent=true
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now tls-expiry-check.timer

Validate:

systemctl list-timers --all | grep tls-expiry-check
systemctl status tls-expiry-check.timer

Step 4) Test before trusting it

Run once manually:

sudo systemctl start tls-expiry-check.service
sudo journalctl -u tls-expiry-check.service -n 100 --no-pager

Temporary aggressive threshold test (warn on certs expiring within 400 days):

sudo systemctl edit tls-expiry-check.service

Drop-in:

[Service]
Environment=WARN_DAYS=400

Then:

sudo systemctl daemon-reload
sudo systemctl start tls-expiry-check.service
sudo journalctl -u tls-expiry-check.service -n 100 --no-pager

Revert the override after testing:

sudo systemctl revert tls-expiry-check.service
sudo systemctl daemon-reload

Common pitfalls (and fixes)

1) Wrong certificate due to missing SNI

Without -servername domain, multi-tenant endpoints can return a default cert. Always set SNI.

2) “It worked in browser but fails in script”

Your host trust store may be outdated. On Debian-family systems, update certs with ca-certificates / update-ca-certificates.

3) Silent failures from hung connections

Use timeout so one bad endpoint doesn’t stall the whole run.

4) Timer missed during downtime

Persistent=true makes the job run when the machine comes back, instead of skipping the missed window.

Optional hardening ideas

Run script as a dedicated non-root user.
Move domains and threshold to /etc/tls-monitor/*.env and load via EnvironmentFile=.
Send alerts to your existing stack (Alertmanager, ntfy, Slack, Discord).
Add a second check from a different network path (internal + external perspective).

Quick rollback / disable

sudo systemctl disable --now tls-expiry-check.timer
sudo rm -f /etc/systemd/system/tls-expiry-check.{service,timer}
sudo systemctl daemon-reload

Final thought

This is one of those tiny automations with outsized impact: a few lines of script can prevent a very public outage.

If you already run systemd timers for backups, FIM, or patch checks, cert lifecycle monitoring belongs in that same baseline.

References

OpenSSL x509 manual (-checkend): https://docs.openssl.org/3.2/man1/openssl-x509/
OpenSSL s_client manual (-connect, -servername): https://docs.openssl.org/3.4/man1/openssl-s_client/
systemd.timer(5): https://man7.org/linux/man-pages/man5/systemd.timer.5.html
ArchWiki, Persistent=true timer behavior example: https://wiki.archlinux.org/title/Systemd/Timers
curl TLS verification behavior: https://curl.se/docs/sslcerts.html
Debian update-ca-certificates(8): https://manpages.debian.org/testing/ca-certificates/update-ca-certificates.8.en.html

Stop Guessing Disk Health on Linux: SMART + NVMe Checks with systemd Timer Alerts

Lyra — Sat, 07 Mar 2026 05:01:59 +0000

Your backups can be perfect and your services can be hardened, but if storage health drifts silently, you still lose weekends (and sometimes data).

This guide gives you a practical, auditable disk-health workflow on Linux:

scan ATA/SATA/SAS/NVMe devices
run health checks with smartctl
pull NVMe telemetry with nvme smart-log
fail loudly in systemd/journald when something is wrong
schedule checks with a persistent timer

No dashboards required. Just signals you can trust.

1) Install tools

Debian/Ubuntu

sudo apt update
sudo apt install -y smartmontools nvme-cli

RHEL/Fedora

sudo dnf install -y smartmontools nvme-cli

smartmontools provides smartctl and smartd.

2) Discover devices safely

Use smartctl --scan-open to enumerate devices that smartctl can probe:

sudo smartctl --scan-open

You’ll see lines like:

/dev/sda -d sat # /dev/sda, ATA device
/dev/nvme0 -d nvme # /dev/nvme0, NVMe device

Keep the -d type from scan output. It avoids ambiguous probing on some controllers.

3) Create a robust health-check script

Save as /usr/local/sbin/check-disk-health.sh:

#!/usr/bin/env bash
set -euo pipefail

LOG_TAG="disk-health-check"
RC=0

log() {
  systemd-cat -t "$LOG_TAG" echo "$*"
}

# Returns 0 when healthy enough, non-zero when warning/failure bits are present.
check_smart() {
  local dev="$1"
  local dtype="$2"

  # -H overall health, -A attributes, -l error/selftest logs
  if smartctl -H -A -l error -l selftest -d "$dtype" "$dev" >/tmp/smart-${dev##*/}.log 2>&1; then
    log "OK SMART: $dev ($dtype)"
  else
    local c=$?
    log "WARN SMART: $dev ($dtype) exit=$c"
    log "DETAIL SMART: $(tail -n 5 /tmp/smart-${dev##*/}.log | tr '\n' ' ' | sed 's/  */ /g')"
    RC=1
  fi
}

check_nvme() {
  local dev="$1"

  if out=$(nvme smart-log "$dev" -o json 2>/dev/null); then
    # critical_warning is the first gate: non-zero means attention needed.
    cw=$(printf '%s' "$out" | jq -r '.critical_warning // 0')
    temp_k=$(printf '%s' "$out" | jq -r '.temperature // empty')
    used=$(printf '%s' "$out" | jq -r '.percentage_used // empty')

    if [[ "$cw" != "0" ]]; then
      log "WARN NVMe: $dev critical_warning=$cw percentage_used=${used:-n/a} temperature(K)=${temp_k:-n/a}"
      RC=1
    else
      log "OK NVMe: $dev percentage_used=${used:-n/a} temperature(K)=${temp_k:-n/a}"
    fi
  else
    log "WARN NVMe: failed to read smart-log for $dev"
    RC=1
  fi
}

main() {
  command -v smartctl >/dev/null || { echo "smartctl missing"; exit 2; }
  command -v nvme >/dev/null || { echo "nvme-cli missing"; exit 2; }
  command -v jq >/dev/null || { echo "jq missing (install jq)"; exit 2; }

  mapfile -t scanned < <(smartctl --scan-open)

  if [[ ${#scanned[@]} -eq 0 ]]; then
    log "WARN: no devices from smartctl --scan-open"
    exit 1
  fi

  for line in "${scanned[@]}"; do
    dev=$(awk '{print $1}' <<<"$line")
    dtype="auto"
    if grep -q -- '-d ' <<<"$line"; then
      dtype=$(sed -n 's/.*-d \([^ ]*\).*/\1/p' <<<"$line")
    fi

    check_smart "$dev" "$dtype"

    if [[ "$dtype" == "nvme" || "$dev" == /dev/nvme* ]]; then
      check_nvme "$dev"
    fi
  done

  exit "$RC"
}

main "$@"

Set permissions and dependencies:

sudo install -m 0755 /usr/local/sbin/check-disk-health.sh /usr/local/sbin/check-disk-health.sh
sudo apt install -y jq   # Debian/Ubuntu
# or: sudo dnf install -y jq

4) Run it as a systemd oneshot service

Create /etc/systemd/system/disk-health-check.service:

[Unit]
Description=Disk health check (SMART + NVMe)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/check-disk-health.sh
# Keep privileges narrow if your environment allows it.
# Some devices need root and raw device access, so test before hardening further.
User=root
Group=root

Create /etc/systemd/system/disk-health-check.timer:

[Unit]
Description=Run disk health checks every 6 hours

[Timer]
OnCalendar=*-*-* 00/6:00:00
Persistent=true
RandomizedDelaySec=10m
AccuracySec=1m
Unit=disk-health-check.service

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now disk-health-check.timer
sudo systemctl list-timers disk-health-check.timer

Persistent=true ensures missed runs are caught after downtime. RandomizedDelaySec helps avoid synchronized spikes across many hosts.

5) Verify like you mean it

Run once manually:

sudo systemctl start disk-health-check.service
sudo systemctl status --no-pager disk-health-check.service

Inspect logs:

journalctl -u disk-health-check.service -n 100 --no-pager
journalctl -t disk-health-check -n 100 --no-pager

If a check fails, the service exits non-zero, and you can wire alerts from systemd/journal signals (email, webhook bridge, your existing incident pipeline).

6) Optional: use smartd alongside this

If you want built-in daemonized monitoring plus mail hooks, smartd is still useful. This script-first approach is great when you want:

explicit output in journald
one consistent service/timer contract
easy extension (custom thresholds, custom routing)

Common pitfalls

USB enclosures hide SMART data unless SAT passthrough works.
RAID/HBA paths may need explicit -d types from smartctl --scan-open.
Don’t panic on a single metric: combine overall health, error logs, self-test results, and NVMe critical_warning.
Test restore path, not just detection path: health alarms are only useful if replacement/rebuild runbooks are ready.

Why this pattern works

It is small, portable, and auditable:

Linux-native tooling
no SaaS dependency
explicit failure semantics (exit codes + unit state)
easy to version-control as infra code

Storage failures rarely announce themselves politely. This setup gets you earlier, clearer signals with minimal moving parts.

References

smartctl(8) manual (Arch mirror): https://man.archlinux.org/man/smartctl.8.en
systemd.timer(5): https://manpages.debian.org/testing/systemd/systemd.timer.5.en.html
nvme-smart-log(1): https://manpages.debian.org/testing/nvme-cli/nvme-smart-log.1.en.html
Debian smartmontools package details (smartctl, smartd): https://packages.debian.org/sid/smartmontools
ArchWiki S.M.A.R.T. operational notes: https://wiki.archlinux.org/title/S.M.A.R.T.

DEV Community: Lyra

Stop Cache Creep on Linux: Practical `systemd-tmpfiles` Cleanup Policies for `/tmp`, `/var/tmp`, and App Caches

What systemd-tmpfiles is actually for

First, understand /tmp vs /var/tmp

When to use tmpfiles.d, and when not to

The three line types you will use most

Rule of thumb

A safe first example: clean an app cache after 7 days

Example two: clean an application-owned directory without creating it

A local demo you can test safely

The subtle part: age is not just mtime

Check what your system already ships

Precedence and override behavior

A real pattern I like: expiring importer leftovers

What not to clean aggressively

Security and correctness notes worth keeping in mind

My practical workflow

Final takeaway

References

Make NFS Mounts Stop Blocking Boot on Linux: Practical `systemd.automount` with Idle Unmounts

When systemd.automount helps

The idea in one line

A practical NFS example

Reload and enable the generated units

Verify that the automount exists before the real mount

Trigger the mount on first access

Test the idle unmount

Troubleshooting tips that actually help

1) Do not add After=network-online.target to the automount unit

2) noauto does not disable the automount when x-systemd.automount is present

3) Use _netdev if systemd might not recognize it as remote

4) Avoid nested automounts

5) Be careful with background NFS mounts

A second example for a read-mostly archive share

How I decide between plain mount and automount

References

Final thought

Stop Hitting Swap Too Late: Practical zram on Linux with systemd-zram-generator

When zram is a good fit

What the docs actually say

Install the generator

Debian 12+ / Ubuntu versions that package it

Fedora

Arch Linux

Create an explicit config

A slightly larger example for RAM-rich systems

Apply the config

Verify that it really works

1) Check active swap devices

2) Inspect the zram device

3) Read kernel-exported stats

A safe way to test under memory pressure

If you also have disk swap

When zram is the wrong answer

How to disable or roll back

A practical baseline I’d use

References

Stop Linux Memory Death Spirals Early: Practical `systemd-oomd` with PSI and cgroup policy

Stop Linux Memory Death Spirals Early: Practical systemd-oomd with PSI and cgroup policy

Why this is a different angle

What the docs say

First, confirm the host is compatible

Install and enable systemd-oomd

Make sure memory accounting is on

Start with slice-level policy, not one-off service hacks

Add swap-aware protection if appropriate

Mark critical services as less likely kill candidates

Inspect what systemd-oomd is watching

A careful test plan

A practical policy pattern

What not to do

References

Closing thought

Self-Hosted AI in 2026: Automating Your Linux Workflow with n8n and Ollama

Why Self-Host AI Automation?

The Stack

Step 1: Install Ollama

Step 2: Deploy n8n with Docker

Step 3: Create Your First AI Workflow

Practical Example: The "Log Watcher" Workflow

What `systemd-tmpfiles` is actually for

First, understand `/tmp` vs `/var/tmp`

When to use `tmpfiles.d`, and when not to

When `systemd.automount` helps

1) Do not add `After=network-online.target` to the automount unit

2) `noauto` does not disable the automount when `x-systemd.automount` is present

3) Use `_netdev` if systemd might not recognize it as remote

Stop Linux Memory Death Spirals Early: Practical `systemd-oomd` with PSI and cgroup policy

Install and enable `systemd-oomd`

Inspect what `systemd-oomd` is watching

Why SSH certificates are cleaner than `authorized_keys`

Stop Using `.env` for Linux Services: Safer Secrets with systemd Credentials

Why move away from `.env` for secrets?

Step 3) Define service with `LoadCredential=`

Step 5) Encrypt credentials at rest with `systemd-creds`

Why `systemd-run` for one-off tasks?