DEV Community

Cover image for I Could Not Turn On Journal Sealing; Only the Debug Log Said Why
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

I Could Not Turn On Journal Sealing; Only the Debug Log Said Why

#systemd #journald #linux #security

Logs are the most trusted and least questioned files on a machine. After an incident you open them and believe what they say. systemd has an answer to that trust: Forward Secure Sealing. journald seals the file cryptographically at regular intervals, and if somebody tampers with it afterwards, journalctl --verify tells you.

The journald.conf documentation puts it this way for Seal=: "If enabled (the default), and a sealing key is available (as created by journalctl's --setup-keys command), Forward Secure Sealing (FSS) for all persistent journal files is enabled." So what ships enabled by default is the mechanism, not the key. The key is yours to create. Creating it is what I sat down to do, and the first command hit a wall at the first step:

$ journalctl --setup-keys --interval=1m
Generating seed...
Generating key pair...
Failed to generate key pair: Operation not supported
Enter fullscreen mode Exit fullscreen mode

The error message names nothing. Finding the cause took half an hour; fixing it took one package. Then sealing actually engaged, and the real half of this article began: what sealing catches, what it does not, and what that PASS line really promises.

The lab

Every number below comes from two privileged containers with systemd genuinely running as PID 1:

$ journalctl --version | head -1
systemd 257 (257.13-1~deb13u1)
$ . /etc/os-release; echo "$PRETTY_NAME"
Debian GNU/Linux 13 (trixie)
$ uname -r
6.10.14-linuxkit
Enter fullscreen mode Exit fullscreen mode

On the comparison side sits Debian 12: systemd 252 (252.39-1~deb12u2). One disclosure up front: that linuxkit in uname -r is the kernel of a Docker Desktop virtual machine on a macOS host. Both Debians share the same non-Debian kernel. What gets measured here is Debian userspace: journald, journalctl, packaging. No claim in this article rests on the kernel.

Sealing needs a persistent journal. Without /var/log/journal, journald keeps everything under /run/log/journal in memory and there is no file to seal. The directory exists on both.

What "Operation not supported" does not say

The setting is right where it should be:

$ systemd-analyze cat-config systemd/journald.conf | grep -i seal
#Seal=yes
Enter fullscreen mode Exit fullscreen mode

The man page still documents --setup-keys. The file header shows no sign of a gap either, only the sealing flag is missing:

$ journalctl --header | grep -E "File path|Compatible flags"
File path: /var/log/journal/a452a00b.../system.journal
Compatible flags: TAIL_ENTRY_BOOT_ID
Enter fullscreen mode Exit fullscreen mode

Raising the log level gave up the answer:

$ SYSTEMD_LOG_LEVEL=debug journalctl --setup-keys --force 2>&1 | head -3
libgcrypt.so.20 is not installed: libgcrypt.so.20: cannot open shared object
file: No such file or directory
Failed to generate key pair: Operation not supported
Enter fullscreen mode Exit fullscreen mode

That line is printed at LOG_DEBUG, so it never shows up in a normal run. Sealing depends on libgcrypt, and since systemd v256 the library is no longer an ordinary shared-library dependency; it is loaded at runtime through dlopen(). The upstream NEWS file announced the change along with its warning: those libraries might no longer be pulled in automatically when ELF dependencies are resolved. On the Debian side the result looks like this:

$ apt-cache show systemd | grep -iE "^(Depends|Recommends|Suggests)" \
    | tr ',' '\n' | grep -ci gcrypt
0
Enter fullscreen mode Exit fullscreen mode

The systemd package does not ask for libgcrypt20, neither as a dependency nor as a recommendation. The fix is one line and it works instantly:

$ apt-get install -y libgcrypt20
$ journalctl --setup-keys --force --interval=1m >/dev/null; echo "rc=$?"
rc=0
Enter fullscreen mode Exit fullscreen mode

The scope of the trap deserves stating. On a desktop or a full server install, libgcrypt20 is usually already there because something else pulled it in; apt-cache rdepends libgcrypt20 lists libxslt1.1, the webkit libraries, strongSwan plugins. This silent loss mostly hits minimal images, containers and thin servers built with debootstrap. So the answer to "can sealing be turned on here" belongs to the machine, not to the distribution.

Once sealing really engages

With the key pair in place, a /var/log/journal/<machine-id>/fss file appears: the sealing key, 482 bytes, owned by root:systemd-journal. Watch the mode, because on two separate installs it came out as 0640. That means accounts in the systemd-journal group can read the sealing key. The verification key is printed to the screen exactly once; keeping it off the machine is your job, because you will not see it a second time.

Second point: journald cannot add a sealing flag to the header of a file it already has open. Generating the key and restarting the service was not enough, a new file was needed:

$ journalctl --rotate
$ journalctl --header | grep "Compatible flags"
Compatible flags: SEALED SEALED_CONTINUOUS TAIL_ENTRY_BOOT_ID
Enter fullscreen mode Exit fullscreen mode

Now verification says something meaningful:

$ journalctl --file=.../system.journal --verify --verify-key=$VK
PASS: /var/log/journal/a452a00b.../system.journal
=> Validated from Sun 2026-10-04 11:51:50 UTC to Sun 2026-10-04 11:53:00 UTC,
   final 2.758424s entries not sealed.
Enter fullscreen mode Exit fullscreen mode

That last line is my favourite output in this article. The tool announces its own blind spot: the final 2.76 seconds of entries are unsealed. Sealing reaches as far as the next seal, and the window in between stays open. The man page says the same thing while explaining --interval=: a shorter interval raises CPU consumption and shortens the time range of undetectable alterations. The default is 15 minutes. In the lab it was pulled down to one minute, which is why the window here is a matter of seconds.

Verification without a key, on the other hand, did not behave the way I assumed. Expecting at least a structural check on a sealed file, this came back:

$ journalctl --file=.../system.journal --verify
Journal file ... has sealing enabled but verification key has not been passed
using --verify-key=.
FAIL: ... (Required key not available)
Enter fullscreen mode Exit fullscreen mode

The source is unambiguous about it: on a sealed file with no key, journal_file_verify() returns -ENOKEY before it even opens a temporary directory. Not a single structural check runs. The operational translation is just as blunt: lose the verification key and you cannot form even a structural opinion about your journal's integrity.

How wide does the tail get

For five minutes nothing was written to the machine by hand. The container's own services kept writing, which matters, because a seal only lands while an entry is being appended. The result:

=> Validated from Sun 2026-10-04 12:18:58 UTC to Sun 2026-10-04 12:23:00 UTC,
   final 59.603160s entries not sealed.
Enter fullscreen mode Exit fullscreen mode

Six seals had accumulated in the file, at epochs 0, 0, 1, 2, 3, 4. With a one-minute interval the tail reaches a full minute. At the 15-minute default that same window is a quarter of an hour.

The source holds a more uncomfortable detail here. In v257 there are only two places that write a seal: journal_file_append_first_tag() when the file is created, and journal_file_maybe_append_tag() called from the entry-append path. No entry, no seal. On a quiet machine the tail is therefore not bounded by the interval at all; it stays unsealed indefinitely.

My own assumption had been that running journalctl --rotate after a critical event would seal the tail. In the v257 source there is no path that appends a seal while closing a file, and no run of mine showed rotation sealing the old file's tail either. Do not lean on it. What you can lean on is that --verify prints the window on every run.

What sealing looks like inside the file

Seeing where the seal lands is the shortest path to understanding its reach. The file-format document gives the TAG object's layout: an object header, a sequence number, an epoch number and a 32-byte SHA-256 HMAC, 64 bytes in total. I walked my own file raw, stepping through the objects from start to end and counting their types:

object types: ENTRY 73, DATA 258, FIELD 40, ENTRY_ARRAY 90, TAG 3
TAG (seqnum, epoch): (1, 0)  (2, 0)  (3, 1)
Enter fullscreen mode Exit fullscreen mode

Three seals in a file spanning roughly 70 seconds, with the interval set to one minute. The first seal lands as the file is created: journal_file_append_first_tag(), called from journal_file_open() for newly created files only, folding the header plus the two hash-table objects into the HMAC. The second sits in the same epoch. Hold on to that detail, I will come back to it.

Each seal is computed over the objects that arrived since the previous one. The document adds an important footnote: while computing the HMAC, volatile header fields of objects are skipped, such as linked-list pointers added later. What validates the consistency of those fields is not the seal but the structural checks in --verify.

I did not take the cost side, but it can be derived from the structure that was measured: a 64-byte TAG at the default 15-minute interval means 96 seals a day, roughly 6 KB. Pull it down to one minute and you get 1,440 seals, about 92 KB. The disk side is irrelevant. The cost the man page warns about is on the CPU, because the key is derived forward at every seal. That figure is not one I took, so I will not quote one; if you plan to shorten the interval, run it on your own hardware and look.

What --verify checks independently of sealing

Sealing is half the story. The journal_file_verify() source holds a long checklist that runs even without a key: hashes of data and field objects, object sizes and alignment, consistency of compression flags, strictly increasing entry sequence numbers, timestamps ordered monotonically within the same boot, hash-table chains free of cycles, entry-array chains that never jump backwards, and header counters that reconcile with the actual content.

That list catches corruption well. What it does not catch is a deliberate edit that leaves all of those fields consistent. The hash on data objects is keyed, and my file's header carries the KEYED-HASH flag, but the key sits inside the file itself. It is protection against hash flooding, not authentication. Whoever holds the file can recompute the hash. Authentication needs a secret the attacker does not have, and sealing is exactly what brings one.

There is one more wrinkle: a sealed file will not even open on a machine without sealing support. Copying the same sealed file to a Debian 13 host with no libgcrypt20 and trying to verify it:

$ journalctl --file=/root/t/muhurlu.journal --verify
Failed to open files: Operation not supported
Enter fullscreen mode Exit fullscreen mode

Verification does not fail; the file cannot be read at all. If you plan to open sealed archives on a thin recovery box later, libgcrypt20 has to be there too.

What sealing catches

Three kinds of tampering, each on a copy of the file, at single-byte or single-bit granularity.

A message inside the sealed region. I changed one character inside a log line:

392be0: Invalid object contents: Bad message
File corruption detected at .../t13-mesaj.journal:3746784
  (of 8388608 bytes, 44%).
FAIL: ... (Bad message)
Enter fullscreen mode Exit fullscreen mode

Caught, but sealing contributed nothing here. Every data object carries its own hash and changing the text breaks it. The same edit would be caught in an unsealed file. A corruption check and authentication are not the same thing, and the first one is what this test demonstrates.

The seal itself. To isolate sealing, I flipped a single bit in the HMAC field of a TAG object:

3955d8: Tag failed verification
FAIL: ... (Bad message)
Enter fullscreen mode Exit fullscreen mode

This is the proof that sealing works. According to the file-format document, every TAG carries a SHA-256 HMAC over the objects before it, with the key derived forward at each epoch.

Dropping the flag. Can an attacker pretend the file was never sealed? I zeroed the compatibility flags in the header:

38f980: Tag object in file without sealing
FAIL: ... (Bad message)
Enter fullscreen mode Exit fullscreen mode

No. With TAG objects still in the file, clearing the flag does not silence verification, it makes more noise.

Then I deleted the logs

A real attacker does not flip bytes, they delete. The sealing key lives on the very machine it protects, so anybody with root can read it. I saved the harshest test for last: stop journald, delete every .journal file, leave the fss key alone, start the service, write a clean history.

$ journalctl --no-pager -q | wc -l
1384
$ systemctl stop systemd-journald
$ rm -f /var/log/journal/*/*.journal
$ systemctl start systemd-journald
$ for i in $(seq 1 6); do logger -t audit "FORGE-$i"; done
$ journalctl --no-pager -q | wc -l
11
Enter fullscreen mode Exit fullscreen mode

One thousand three hundred and eighty-four lines gone, eleven left. The new file opened with SEALED SEALED_CONTINUOUS flags, so on paper I have a sealed history. Here is what verification said:

38f980: Tag failed verification
FAIL: ... (Bad message)
Enter fullscreen mode Exit fullscreen mode

The wipe left a mark. The same result came back three times across three separate containers, and there is a control run to go with it: without any deletion, a new file produced by a plain rotation passes the same verification. So this is not a blanket false alarm.

On Debian 12 the same move gives a different answer:

PASS: /var/log/journal/41a11bc4.../system.journal
=> No sealing yet, 2.020730s of entries not sealed.
Enter fullscreen mode Exit fullscreen mode

There, 1,543 lines dropped to 19 and verification was not bothered in the slightest.

This is where I owe you honesty. I did not trace the Debian 13 FAIL down to which byte broke and why, so I am not pinning the mechanism on a patch or a flag. What I hold is a behaviour reproduced three times and a control run that does not refute it. I claim the behaviour, not the mechanism.

The patch that seals empty epochs

Debian 13's files carry an extra flag, SEALED_CONTINUOUS, that Debian 12's do not. The flag arrived with Felix Dörre's patch, merged on 8 November 2023 in the v255 cycle, and the patch states what it closes: "Currently empty epochs are not sealed. This allows an attacker to truncate a sealed log and continue it without any problems showing when verifying the log."

The same gap is on record as CVE-2023-31438. The NVD description: a sealed log file can be truncated and log sealing resumed such that the integrity check shows no error, despite modifications. The record also notes the vendor replying that none of the findings was a security vulnerability, and the entry is tagged disputed.

One sentence in the patch text sits where most summaries skip: "This partially addresses CVE-2023-31438." It also says what closing it completely would take: verifying that there is exactly one seal per epoch, and not sealing before the epoch has ended. The author left the premature sealing in place, having found it deliberate but not understood its purpose.

That open door is sitting in my own measurement. Look at the TAG dump above: (1, 0) and (2, 0), two seals in epoch 0. The premature sealing the patch says it did not remove is exactly this. The verifier's continuity check accepts it too; in the source, a file's first tag and a second tag in the same epoch are explicitly exempt.

The flag has one more practical side effect. Hand a sealed file produced by Debian 12 to Debian 13's journalctl and you get this:

000000: This log file was sealed with an old journald version where the
sequence of seals might not be continuous. We cannot guarantee completeness.
PASS: /root/t/bw.journal
=> Validated from Sun 2026-10-04 11:45:41 UTC to Sun 2026-10-04 11:48:00 UTC,
   final 4.528312s entries not sealed.
Enter fullscreen mode Exit fullscreen mode

A PASS, but not a complete one. If you keep archives produced by older releases, make a habit of looking for that line in your verification report.

Delete the key too, and verification goes quiet

So what if the attacker deletes the key file as well? Same scenario, one extra step:

$ systemctl stop systemd-journald
$ rm -f /var/log/journal/*/*.journal /var/log/journal/*/fss
$ systemctl start systemd-journald
$ journalctl --header | grep "Compatible flags"
Compatible flags: TAIL_ENTRY_BOOT_ID
$ journalctl --verify --verify-key=$VK
PASS: /var/log/journal/a452a00b.../system.journal
Enter fullscreen mode Exit fullscreen mode

PASS, even with the verification key handed over, and not a word of complaint. The logic is consistent: the file is not sealed, and an unsealed file has no seal to verify. From a security standpoint the consequence is this: journalctl --verify never tells you "this file should have been sealed". You have to hold that expectation somewhere the machine cannot reach.

Diagram

The bottom-left corner of that diagram is the real lesson here. PASS has two separate meanings and the output says both with the same word.

How v262 changes this picture

systemd v262 shipped on 22 September 2026 and sealing moved from libgcrypt to OpenSSL. Debian 13's systemd package already carries a libssl3t64 dependency, so the missing-package trap in this article does not change shape when 262 lands; it disappears.

What replaces it is sneakier. In the new source, verifying a sealed file on a machine without sealing support no longer returns -ENOKEY:

if (journal_auth_supported())
        return -ENOKEY;
else
        log_notice("Journal file is sealed, but journal sealing support is "
                   "disabled. Skipping seal verification.");
Enter fullscreen mode Exit fullscreen mode

So the situation that produces a hard FAIL on 257 will, from 262 onward, leave a notice line and carry on toward PASS. It is another variant of the quiet PASS this whole article is about, this time inside upstream itself. When your distribution moves to 262, that notice line is what your verification reports will need you to look for.

What I do in practice

  • Check the flag, do not trust the default. If journalctl --header | grep "Compatible flags" shows no SEALED, there is no sealing. On minimal images this is the first place to look.
  • Do not leave the check to a human. Advice is not a control. A small timer that greps journalctl --header for SEALED and alerts when it is absent is the "hold the expectation off the machine" argument turned into something operational.
  • Put libgcrypt20 on the install list. When it is missing the error message hides the cause, and the diagnostic path is SYSTEMD_LOG_LEVEL=debug. Recovery boxes that will open sealed archives need it too.
  • Get the verification key off the machine. A key kept on the same disk is a convenience for whoever takes that disk, not evidence. It is printed once.
  • Know the price of rotating keys. --setup-keys --force mints a new pair. The verification key belongs to a key pair, not to a machine, so the moment you rotate, your existing sealed files can no longer be verified with the key in your hand. Do not rotate without archiving the old one.
  • Remember that early boot sits outside the seal. Only persistent files are sealed. Everything journald writes under /run/log/journal before systemd-journal-flush.service is unsealed by design, so the first seconds of boot are structurally out of scope.
  • Pick the interval deliberately. The default is 15 minutes, and so is the length of the undetectable window. The "entries not sealed" line in --verify reports it on every run.
  • Do not confuse sealing with staying put. Sealing shows that existing bytes have not changed, not that the records still exist. Detecting a missing message is a different problem: signed syslog (RFC 5848) numbers messages so the receiver can see which ones are absent. A copy shipped to a central collector does the same job, and the setup in central logging with systemd-journal-remote exists for exactly that. What gets deleted locally survives remotely.
  • Remember what rotating logs costs. How copytruncate silently eats records when it truncates a file in place is something I took the measurements for earlier; in a sealed world such truncations cost you not only data but evidence.
  • Detection and repair are separate things. The distinction from the dm-integrity article holds here too: sealing tells you about damage, it does not undo it. What undoes it is a backup or a remote copy.

Conclusion

Sealing works wherever you can turn it on. What tripped me up was not its quality but the silent assumption that it existed. The Seal=yes line sits in the documentation as a default, anyone skipping the condition next to it can believe their sealing is on, and nothing announces that it is not: no service log, no startup error, and certainly not the PASS from --verify.

Journal integrity is not a setting but a chain. The library has to be installed, the key generated, the file rotated, the verification key kept outside, and somebody has to hold the knowledge that "this file should have been sealed" off the machine. Break any link in that chain and journalctl --verify still prints PASS. A quiet PASS is no different from an alarm that stays silent.

Official Sources

Top comments (0)