DEV Community: Mito

Raspberry Pi OS Trixie throws away your logs on reboot, and /var/log/journal exists anyway to reassure you it doesn’t

Mito — Tue, 18 Aug 2026 00:00:00 +0000

TL;DR : Raspberry Pi OS Trixie ships /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf with Storage=volatile. Your journal lives in RAM and is gone after a reboot — exactly when you need it. /var/log/journal/ exists but stays empty, which makes it look like persistence is already working. Fixing it takes a drop-in and a flush; the drop-in alone silently does nothing until the next boot.

The symptom

Your Pi does something bad — crash-loops, drops off the network, wedges. You reboot it to get back in, then go looking for what happened:

journalctl -b -1


Specifying boot ID or boot offset has no effect, no persistent journal was found.

The logs from the boot you actually care about do not exist. They never made it to disk.

Why it’s easy to convince yourself this is already fine

Two things conspire here.

First, the directory exists. /var/log/journal/ is present on a stock Trixie install:

$ ls -la /var/log/journal/
total 8
drwxr-sr-x+ 2 root systemd-journal 4096 Jun 18 09:19 .
drwxr-xr-x 6 root root 4096 Aug 16 18:31 ..

For years, the canonical way to enable persistent journald has been “create /var/log/journal and it starts persisting”. The directory’s presence is normally the signal that persistence is on. Here it’s present and empty, and the emptiness reads as “nothing has been logged yet” rather than “nothing will ever be written here”.

Second, the default config file agrees with you. /etc/systemd/journald.conf — the file you’d naturally open to check — says:

#Storage=auto

Commented out, default auto. And auto means “persist if /var/log/journal exists”, which it does. So the config file you inspected and the directory you found both say persistence should be working.

Neither of them is where the decision is being made.

Where the decision actually is

$ ls /usr/lib/systemd/journald.conf.d/
40-rpi-volatile-storage.conf
syslog.conf

$ cat /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf
[Journal]
Storage=volatile

A vendor drop-in, shipped by the distro, overriding the default you read in the main config file. Drop-ins in /usr/lib/systemd/journald.conf.d/ take precedence over /etc/systemd/journald.conf, so volatile wins and the journal is written to /run/log/journal/ — a tmpfs — instead.

The reasoning behind the default is defensible: it protects SD cards from log write wear, which is a real failure mode on Pis. The problem isn’t the choice, it’s that the choice is invisible from every place you’d normally look.

Don’t guess at precedence. Ask systemd what it actually resolved:

systemd-analyze cat-config systemd/journald.conf

That prints every file in load order with its contents, so you can see exactly which line won.

The fix everyone posts, and why it isn’t enough

The advice you’ll find in forum threads is: add a higher-priority drop-in under /etc. That part is right:

sudo mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nStorage=persistent\n' | sudo tee /etc/systemd/journald.conf.d/99-persistent-storage.conf
sudo systemctl restart systemd-journald

Then check:

$ journalctl --header | grep 'File path'
File path: /run/log/journal/<machine-id>/system.journal

Still /run. The setting is correct — systemd-analyze cat-config confirms Storage=persistent is winning — and the journal is still in RAM. Restarting the service was not enough.

What’s missing is the flush. Moving the journal from the runtime location to the persistent one is a distinct operation, normally performed at boot by systemd-journal-flush.service. Change the setting mid-session and nothing triggers it:

sudo journalctl --flush


$ journalctl --header | grep 'File path'
File path: /var/log/journal/<machine-id>/system.journal
File path: /var/log/journal/<machine-id>/user-1000.journal

Now it’s real.

This is the part that makes the bug expensive. Without the flush, the drop-in does take effect — at the next reboot. So if you apply the incomplete fix, verify it by checking journalctl --header, and see /run, you’ll reasonably conclude your drop-in didn’t work and start debugging precedence rules that are already correct.

On the filename everyone tells you to use

Forum threads specify the override must be named 99-something.conf, on the grounds that it has to sort after the vendor file to win.

The sorting rule is real. The number isn’t. On the image checked here (2026-06-18 Trixie arm64 Lite) the vendor file is 40- , not the 70- those threads describe. Anything above 40- wins, so 50- would do.

99- is still the right choice — not because 40- demands it, but because it survives the vendor renaming the file, which apparently already happened once. Just don’t take the specific number in a forum post as fact about your system. Check:

ls /usr/lib/systemd/journald.conf.d/

The trade-off you’re accepting

Turning this on means writing logs to your boot media continuously. The vendor default exists for a reason: on a Pi running from an SD card, that’s real wear on a device that’s already the most common hardware failure point.

Worth turning on if you boot from an SSD or NVMe, or if you’re debugging something that survives reboots. Worth thinking twice about on a plain SD card — and if you do enable it there, cap the size so it can’t grow without bound:

[Journal]
Storage=persistent
SystemMaxUse=200M

The generalisable habit

Three separate things pointed at “persistence is on”: the directory existed, the main config said auto, and after the first fix attempt the config resolution confirmed persistent. All three were true. None of them described where log bytes were actually being written.

journalctl --header was the only thing that reported the live state — the actual open file — and it’s the check worth building the habit around. Config tells you intent. Headers tell you reality. When they disagree, something in between hasn’t run yet.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.\n\n*See the toolkit \xe2\x86\x92*\n

Stripe retries a failed webhook for three days, and I found out by getting the same email three times

Mito — Tue, 18 Aug 2026 00:00:00 +0000

TL;DR : Stripe webhook delivery is at-least-once, not exactly-once. If a webhook fails for any reason and you fix the cause later, every queued retry succeeds afterward — and if your handler isn’t idempotent, that means re-running your whole side effect (sending an email, granting access, provisioning something) once per retry. Keep a marker per event/session and check it before acting, not just on send.

The setup

A small toolkit sold as a one-time purchase: buy via a Stripe Payment Link, a webhook fires on checkout.session.completed, a Cloudflare Worker verifies the signature and emails a download link. Straightforward, and it had already been tested end-to-end with a real charge before going live.

Shortly after that real charge, three separate copies of the exact same download email arrived, roughly thirty minutes apart.

What actually happened

Nothing was resending on my end. Nothing in the Worker was looping. The explanation was upstream: the very first delivery attempt for that charge had failed — a webhook secret had been misconfigured minutes earlier — so Stripe’s signature check on my side rejected it with a 400.

Stripe’s webhook delivery isn’t “try once and give up.” A non-2xx response is treated as a delivery failure, and Stripe retries on a backoff schedule for up to three days. The first few retries land roughly every few minutes to half an hour depending on how long the endpoint’s been failing.

The secret got fixed a few minutes after the first failure. From Stripe’s side, nothing about that mattered — it just kept retrying the same event on its normal schedule. Once the secret was correct, the next retry succeeded. So did the one after that. Each one was a completely valid, correctly signed, legitimate-looking webhook for a real completed checkout — because it was. Nothing distinguished retry three from delivery one except that my endpoint had already acted on delivery one’s twin.

Why this is easy to miss until it costs you

The natural mental model when building a webhook handler is “an event happens once, so my handler runs once.” That model is wrong in a way that doesn’t show up in testing, because a clean test run typically doesn’t have a failed first attempt — you write the handler, it works, you ship it. The retry behavior only becomes visible the first time something fails at exactly the wrong moment: mid-deploy, during a secret rotation, during any transient error on either side.

And the failure mode isn’t a crash or an error someone will report. It’s a customer quietly getting the same email two or three times, mildly annoyed, possibly assuming something is wrong with the business rather than realizing the delivery layer is precisely doing what it’s specified to do.

The fix

Stripe’s Checkout Session ID is a stable identifier for the same real-world event across every retry. That’s the natural idempotency key: before acting, check whether that session has already been handled; if it has, acknowledge and stop.

const marker = `delivered/${session.id}`;
if (await env.TOOLKIT_BUCKET.head(marker)) {
  console.log(`Session ${session.id} already delivered; skipping duplicate send.`);
  return new Response('ok (already delivered)', { status: 200 });
}

try {
  const downloadUrl = await createDownloadUrl(url.origin, env.DOWNLOAD_SIGNING_SECRET);
  await sendDownloadEmail(env, email, downloadUrl);
  // Written only after a confirmed send, so a failed attempt stays retryable.
  await env.TOOLKIT_BUCKET.put(
    marker,
    JSON.stringify({ email, deliveredAt: new Date().toISOString(), eventId: event.id }),
  );
  return new Response('ok', { status: 200 });
} catch (err) {
  // 500 so Stripe retries — a transient send failure shouldn't lose a sale.
  return new Response('delivery failed', { status: 500 });
}

The ordering matters as much as the check. The marker is written after the email send succeeds, not before it and not unconditionally. Writing it earlier would mean a transient failure in the email provider permanently marks the session as handled, and the customer never gets their download at all — trading a duplicate-email bug for a much worse silent-failure bug. The retry mechanism that caused this problem is also the thing you want protecting you against provider outages; the fix has to keep both properties.

How to check it without waiting for another real charge

You don’t need to make another purchase to verify this. Forge a correctly signed request against your own secret and send it directly:

SESSION="cs_idempotency_test_$(date +%s)"
# ...compute t=<timestamp>,v1=<hmac-sha256 of "<timestamp>.<body>" with the webhook secret>...

curl -X POST "$WEBHOOK_URL" -H "stripe-signature: $SIG" --data "$BODY" # expect: ok
curl -X POST "$WEBHOOK_URL" -H "stripe-signature: $SIG" --data "$BODY" # expect: ok (already delivered)
curl -X POST "$WEBHOOK_URL" -H "stripe-signature: $SIG" --data "$BODY" # expect: ok (already delivered)

Same technique as forging a signature to test rejection — construct the real thing yourself instead of guessing at what production will do.

The generalisable habit

“At-least-once” is the standard delivery guarantee for webhooks, queues, and most event systems, precisely because “exactly-once” is expensive or impossible to guarantee end-to-end. Any time you’re consuming events from a system that documents at-least-once delivery, the assumption to design against isn’t “this fires once” — it’s “this will eventually fire more than once, probably at the worst possible time, and my handler needs to be safe either way.”

The cost of skipping that isn’t a crash you’ll notice in a log. It’s a side effect running twice, silently, on exactly the request path most likely to be running for a real customer at the exact moment you were still shaking out the rest of the system.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.\n\n*See the toolkit \xe2\x86\x92*\n

Your Discord bot is ignoring you, and the status output is lying about why

Mito — Mon, 17 Aug 2026 00:00:00 +0000

TL;DR : If a self-hosted bot connects successfully, can post to a channel, but never responds when you @mention it, check whether your framework has the bot’s own application ID configured. Without it, the bot receives your message, fails to recognise its own mention, and drops it. Any “message content intent” warning you see at the same time is probably a red herring.

The symptom

The bot was up. channels status --probe said connected, works. Sending from the bot into a Discord channel worked on the first try — the test message landed instantly.

Then I mentioned the bot in that same channel and got nothing. No error, no reply, no visible reaction. Just silence.

The misleading clue

The status line included this:

intents:content=limited

That looks like an answer. “Content is limited” reads exactly like the well-known Discord gotcha where a bot can’t read message text because Message Content Intent isn’t enabled in the Discord Developer Portal. That intent genuinely is required for bots to see the text of ordinary guild messages, it genuinely is off by default, and forgetting it genuinely is one of the most common Discord bot mistakes.

So I chased it. Enabled the intent in the portal — it was already on. Restarted the gateway to force a fresh connection — still content=limited. Went looking for a config key to force it on, found documentation describing intents.messageContent, tried to set it, and got:

Config validation failed: channels.discord.intents:
must not have additional properties: "messageContent"

That error is the moment the investigation should have turned, and it’s worth dwelling on. The documented key didn’t exist in the schema. Two readings were available: either the docs were for a different version, or I was in the wrong place entirely. I checked the actual schema:

openclaw config schema | python3 -c "
import json,sys
d=json.load(sys.stdin)
print(json.dumps(d['properties']['channels']['properties']['discord']['properties']['intents'], indent=2))
"

The whole set of supported keys came back as presence, guildMembers, voiceStates. There was no message-content key at all — not disabled, not misspelled, not present. Which meant content=limited could not be something I was failing to configure.

What the logs actually said

Reading the raw channel log instead of the status summary took about ten seconds and settled it:

info {"module":"discord-auto-reply"}
{"channelId":"...","reason":"no-mention"}
discord: skipping guild message

Two things are true in that one line, and they point in opposite directions from where I’d been looking:

The bot received the message. It could not log a decision about a message it never got. Message content was never the problem.
It decided the message contained no mention — and dropped it on that basis.

And critically, the log line was identical for a plain message and for a message where I had explicitly @mentioned the bot. Same reason: "no-mention", both times. The bot wasn’t failing to read my message. It was failing to recognise itself in it.

The cause

A Discord mention arrives in the message payload as <@BOT_USER_ID>. To decide “was I mentioned?”, the receiving code has to compare that ID against its own. If the framework doesn’t know its own application/client ID, that comparison can’t succeed, and every mention looks like a non-mention.

The config had no applicationId set:

$ openclaw config get channels.discord.applicationId
Config path not found: channels.discord.applicationId

The bot’s own ID had been sitting in the startup log the whole time:

discord client initialized as 1538471699438370957; awaiting gateway readiness

Setting it fixed the bot immediately:

openclaw config set --json channels.discord.applicationId '"1538471699438370957"'

Note the --json flag and the nested quotes. Without them the value is parsed as a number and rejected — Discord snowflake IDs are numeric strings, and they must stay strings.

Restart the gateway, mention the bot, get a reply.

Why this is worth writing down

Not because the fix is hard — it’s one config key. It’s worth writing down because of how the failure presents itself.

Everything visible pointed at the intent. The status output named it. The symptom (bot can’t act on my messages) matches the intent problem exactly. The intent problem is far more commonly discussed online, so every search reinforces it. And the fix for the intent problem — toggling a switch in the Developer Portal — is easy enough that you’ll happily do it, see nothing change, and conclude you need to dig deeper into intents rather than sideways into something else.

Meanwhile the actual cause is invisible from the status output. A missing applicationId doesn’t warn you. It doesn’t degrade a status line. It just quietly makes one boolean always false.

The generalisable habit

When a status summary and a raw log disagree about what’s happening, the log is describing events and the summary is describing an interpretation. The summary is a convenience layer written by someone who guessed which distinctions would matter to you; it flattens away detail by design. The log records what the code actually decided, and it tells you why in the code’s own terms.

reason: "no-mention" was a precise, correct, complete description of the failure. It was available before I made my first change. I reached for the status line because it was already on screen, and spent the next several steps debugging an interpretation instead of an event.

So: when a component “connects fine but doesn’t react”, check the receive path’s decision log before you touch a single config value. And when a documented config key doesn’t exist in the running schema, treat that as evidence you’re in the wrong subsystem — not as an invitation to find a different way to set it.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.\n\n*See the toolkit \xe2\x86\x92*\n

Undervoltage doesn’t look like a power problem. It looks like bad Wi-Fi.

Mito — Mon, 17 Aug 2026 00:00:00 +0000

TL;DR : On a Raspberry Pi, an inadequate power supply rarely announces itself as “the power is bad”. It shows up as flaky Wi-Fi, hung logins, and mysterious slowness. Check vcgencmd get_throttled (or /sys/class/hwmon/hwmon*/in0_lcrit_alarm) before you debug the symptom you can see.

The symptom

A freshly imaged Pi 4B, running headless-ish off a monitor and keyboard for setup. Two things were wrong, and neither of them sounded electrical:

Wi-Fi wouldn’t come up. hostname -I returned 127.0.1.1 and nothing else — no DHCP lease, no address, on a network every other device was using without complaint.
The console was unusable. Typing a username at the login prompt appeared to do nothing; after a delay the prompt would simply return. It read like a broken install or a corrupt filesystem.

The natural reading of those two symptoms together is “bad image” or “bad SD card”. That is where I would have gone next.

The line that changed the diagnosis

Buried in the console spam was this:

[899.552015] hwmon hwmon2: Undervoltage detected!

One line, printed once, ~15 minutes into uptime, surrounded by unrelated boot chatter. Easy to scroll past — and in fact it had scrolled past several times before it got read.

What undervoltage actually does

The Pi’s firmware monitors the 5V rail. When it drops below roughly 4.63V, it flags undervoltage and starts protecting itself: the CPU is throttled, clocks are capped. That much is documented and expected.

What’s less obvious — and what makes this so hard to diagnose from symptoms — is the second-order effects:

The wireless chip is voltage-sensitive. A rail that sags under load will produce exactly what I was seeing: a radio that scans but won’t associate, or associates and drops. It presents as a networking fault, not a power fault.
Everything gets slower, unevenly. A throttled CPU plus a stressed rail makes interactive work feel like the machine is hanging rather than running slowly.
It’s the single most common cause of SD card corruption on Pis. Losing the rail mid-write is how filesystems get damaged. So the “bad SD card” hypothesis isn’t wrong, exactly — it’s downstream. Undervoltage creates bad SD cards.

That last point is why this matters more than a performance footnote. If you chase the symptom and reflash the card, you will produce a machine that works briefly and then breaks again, and you will blame the card a second time.

How to check it directly

Don’t infer it from behaviour. Ask the firmware:

vcgencmd get_throttled

throttled=0x0 means clean. Anything else is a bitfield:

Bit	Meaning
0 (`0x1`)	Under-voltage right now
1 (`0x2`)	ARM frequency currently capped
2 (`0x4`)	Currently throttled
16 (`0x10000`)	Under-voltage has occurred since boot
17 (`0x20000`)	Frequency capping has occurred
18 (`0x40000`)	Throttling has occurred

The split matters. Low bits mean it’s happening as you look. High bits mean it happened earlier and has since recovered — which is exactly the fingerprint of an intermittent supply, and exactly what you’d otherwise dismiss as “it’s fine now, must have been a fluke”.

If vcgencmd isn’t installed (it isn’t always, on minimal Lite images), the kernel exposes the same alarm through hwmon and needs no packages:

cat /sys/class/hwmon/hwmon*/in0_lcrit_alarm

0 is healthy, 1 means undervoltage is currently flagged.

Note that the “has occurred” bits only clear on reboot. So the correct test loop after changing anything is: reboot, use the machine, then re-check — not check, change, check again on the same boot.

The actual fix

A Pi 4B wants 5.1V at 3A. The usual culprits, in the order they actually bite:

The cable, not the adapter. This is the one people skip. A thin or long USB-C cable drops meaningful voltage under load even when the supply is genuinely rated 3A. Short and thick beats nominally-correct-at-the-plug.
Phone chargers that don’t negotiate. A charger can be rated well above 15W and still hand a Pi 5V at low current, because the Pi doesn’t perform the USB-PD negotiation the charger is waiting for.
Whatever else is drawing from the Pi’s rails. USB peripherals, HATs, SPI displays and GPIO-powered fans all come out of the same budget. A setup that’s stable bare can go unstable the moment you add a display.

Swapping to a supply that could actually hold the rail cleared both symptoms at once — Wi-Fi associated immediately, and the console stopped stalling. throttled=0x0 from then on.

The generalisable habit

The reason this costs people hours is that the failure surfaces far from its cause. Power is the substrate everything else runs on, so when it’s marginal, the visible breakage appears in whichever subsystem is least tolerant — usually the radio. You end up debugging NetworkManager while the actual fault is a cable.

So it’s worth making the power check reflexive rather than diagnostic: on any Pi that’s misbehaving in more than one way at once, run get_throttled first. It takes two seconds, and it either eliminates an entire category of cause or hands you the answer.

Two unrelated subsystems failing simultaneously is rarely two bugs. It’s usually one thing underneath both of them.

A ready-to-run version of this check — which works with or without vcgencmd, decodes the bitfield into plain English, and tells you which of the three causes above to look at — is in the toolkit.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.\n\n*See the toolkit \xe2\x86\x92*\n

The rpi-clone PARTUUID trap: why your Pi 4B silently refuses to boot from a cloned SSD

Mito — Sun, 16 Aug 2026 00:00:00 +0000

TL;DR : rpi-clone copies your SD card to a USB SSD beautifully, but it does not update the root=PARTUUID=... reference in the destination’s cmdline.txt. Flip the boot order to USB before catching this, and your Pi either boots the wrong disk or fails to boot at all. Five-minute fix once you know to look for it.

The setup

Moving a Raspberry Pi’s root filesystem off the SD card and onto a USB-attached SSD is one of the highest-value changes you can make to a Pi that’s meant to run unattended for months: SD cards wear out under sustained write load, SSDs don’t (practically speaking), and boot/IO latency drops noticeably as a bonus.

The standard playbook is:

Update the bootloader EEPROM (sudo rpi-eeprom-update -a), reboot.
Clone the running SD card to the SSD with rpi-clone.
Flip raspi-config → Advanced Options → Boot Order to prefer USB.
Reboot and confirm findmnt / shows the SSD.

Steps 1, 3, and 4 are exactly as advertised. Step 2 has a landmine in it.

What actually happens

rpi-clone does a genuinely good job: it partitions the destination to match the source, rsyncs the filesystem across, and rewrites the destination’s /etc/fstab with the destination’s own PARTUUIDs. If you check fstab after cloning, it’s correct.

What it does not do is touch /boot/firmware/cmdline.txt on the destination. That file still contains the source disk’s root=PARTUUID=..., copied verbatim.

This is easy to miss because nothing in the clone process complains. The clone finishes, reports success, and looks done.

Why it bites you specifically at the worst moment

You won’t notice during the clone. You won’t notice right after, either — you’re still running from the SD card at that point, so everything looks fine. The mismatch only matters the moment the firmware actually tries to boot the kernel from the SSD, using the SSD’s own cmdline.txt — which is exactly what happens right after you flip the boot order and reboot.

At that point the kernel is told “find your root filesystem at PARTUUID 10052bbd-02” (the SD card’s identifier), while the disk it’s actually booting from carries an entirely different, freshly-generated PARTUUID (say, b11c96d4-02). Depending on whether the SD card is still physically present, you get either a boot that silently falls back to the SD card’s root filesystem (confusing — your changes “don’t stick”), or a kernel panic waiting for a root device that isn’t there (worse).

How to catch it before rebooting

Right after the clone finishes — before you touch the boot order — compare the source and destination PARTUUIDs directly:

# What the SD card's root partition is actually called
sudo blkid /dev/mmcblk0p2

# What the SSD's root partition is actually called
sudo blkid /dev/sda2

# What the SSD's own boot config *thinks* the root partition is called
sudo mount /dev/sda1 /mnt
grep -o 'root=PARTUUID=[a-f0-9-]*' /mnt/cmdline.txt

If the third value matches the first (the SD card’s) instead of the second (the SSD’s own), you have the bug.

The fix

sudo sed -i "s/root=PARTUUID=<OLD_PARTUUID>/root=PARTUUID=<NEW_PARTUUID>/" /mnt/cmdline.txt

Then do one more sweep to make sure nothing else on the destination still references the stale identifier — it’s a one-liner and costs nothing:

sudo grep -rl "<OLD_PARTUUID>" /mnt 2>/dev/null

If that returns nothing, unmount and reboot into the boot-order change with confidence.

A ready-to-run version of this check-and-fix — with a dry-run mode and an automatic backup of cmdline.txt before it touches anything — is in the toolkit that comes with this post.

The general lesson

Any tool that clones a running system to a new disk has to solve the “how does the new disk know it’s the new disk” problem somewhere. Some tools solve it by rewriting every reference at clone time; rpi-clone solves most of it that way (fstab) but leaves one file (cmdline.txt) untouched. That’s not a bug exactly — it’s a scope boundary the tool doesn’t advertise loudly. The general habit worth keeping: after any disk clone, before changing what your firmware boots from, grep the new disk for the old disk’s identifiers. If anything comes back, you’ve found your landmine before it found you.

Toolkit

This post's fix is available as a tested, ready-to-run script in the toolkit.\n\n*See the toolkit \xe2\x86\x92*\n