Printing coredumps automatically with systemd and ReceiptIO

I had a fun idea one day after seeing some people online playing with receipt printers. I frequently run into app crashes on my Linux system, all of which generate core dumps for review later, and those are logged in the system log. Wouldn't it be funny if they were logged not only in software, but also in physical form? It took me a while to finally go through with it, but I now have a receipt printer set up to automatically print out a crash log for every crash on my system. Here's how I did it.

systemd-coredump

On Linux systems using systemd (which is the vast majority of them), a component called systemd-coredump is installed to automatically take in coredump files that the kernel generates, and processes the files before outputting them on disk. Coredump files are stored in /var/lib/systemd/coredump by default, and the coredumpctl command can be used to output various information about the coredumps. (This will be useful later.)

My first task was to get the system to run a command whenever a coredump occurred. I spent some time digging through man pages on systemd-coredump and coredumpctl, but I unfortunately came up with nothing - it doesn't seem to have any functionality to e.g. run a service on crash. I tried hacking with the inner functionality of the dump service, trying to hook into its trigger, but this didn't result in anything useful. I also thought about replacing systemd-coredump entirely, but then I'd be lacking the digest that it creates.

Then I came up with a solution...

Watching directories with systemd

I realized that I could simply have a command be triggered whenever the coredump directory gets written to, which only happens when a program crashes. systemd supports special unit files with the extension .path to watch a specific file or directory. When the path is updated, systemd triggers a .service unit with the same name, which can be a shell script with the commands I need. Perfect!

I put together a simple path unit /etc/systemd/system/print-coredump.path:

[Path]
PathChanged=/var/lib/systemd/coredump

[Install]
WantedBy=paths.target

And I made an accompanying service unit to run a bash script:

[Unit]
Description=Automatically print core dump

[Service]
Type=oneshot
ExecStart=/usr/local/bin/print-coredump

To enable watching, I simply had to run sudo systemctl enable --now print-coredump.path.

ReceiptIO

Next I needed a way to send text to the printer. I went googling around for a bit, and eventually stumbled upon ReceiptIO. ReceiptIO is a NodeJS library and program that automatically formats and prints out text (and images) using its own formatting language inspired by Markdown. It also has a web viewer to preview printouts, which was useful before I had the printer in my hands, and it also helped me understand what changes I'd need to make it pretty.

I installed it globally with npm: sudo npm install -g receiptio

To start with, I made a very simple bash script to print out the most recent coredump generated to an SVG (the default output if no printer is attached):

#!/bin/bash
sleep 1 # wait for the file to be fully written
coredumpctl info -1 | receiptio -i -o /home/jack/printtest.svg

As I expected, the output it generated was nowhere near formatted in a way that was readable. So I then spent some time writing some commands to process the coredump digest into a nicer format.

Formatting the output

I started by plugging in the output of coredumpctl info straight into the web viewer. My first challenge was that all of the text was centered horizontally - this looked really ugly. At first, I simply made all of the text align left, but the header info at the top didn't quite look right.

One thing that coredumpctl does in its output is that it formats the metadata block at the top in two columns using spaces. However, this doesn't translate well to the receipt printer, so my output was overflowing the lines, and it just didn't look good. To fix that, I used the column formatting commands that ReceiptIO provides. I used sed to replace every a: b with a:|b - | tells ReceiptIO to separate the sides into columns, and having no spaces on either side of it means that the left side is right aligned, and the right side is left aligned (i.e. the text is effectively centered). One caveat: I did decide to undo this change on the final line, which I wanted to keep centered - this involved another sed expression to change it back if the left side was architecture.

After that, I needed to do some formatting on the stack trace entries. I wanted to keep the "introduction" to each thread centered, but each frame would be left-aligned. This simply involved adding a | at the start of each frame line, which conveniently always starts with #. A simple sed expression fixed that. I also made it replace the empty lines with horizontal rules (- alone on a line) for further prettification.

My last gripe was that the underscores in function names in the traces were being interpreted as control characters - like in Markdown, _a_ and __a__ apply special formatting. This required one last sed expression to escape every _ in the output.

Building the command

Next I combined the changes into a single command line. Watch out, this is a pretty big command!

coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/-/g' | receiptio -d /dev/usb/lp0 -c 42

Let's break that really long sed line down into each part:

• If you aren't aware of how sed works: basically, it's a simple text processing language that transforms each line of text passed into it via a pipe. The most basic command, and the only one used here, is s for substitution, which takes a regular expression pattern and replaces it with other text. It takes the syntax s/find/replace/flags, where flags may be empty to replace only once, or it can be g to replace all.
• Each -e option is a single replacement command. This makes it so I don't have to run sed a bunch of different times - it can be done all in one command.
• The first -e command gets rid of any spaces at the start of a line, which basically removes the column formatting that coredumpctl applies.
• The second -e command replaces all :s with :|, placing metadata lines into two columns as described above.
• The third -e command left-aligns all lines starting with # (stack frame lines).
• The fourth -e command escapes all _s with \_ (double backslash so sed doesn't see it as an escape itself).
• The fifth -e command removes the column formatting from the final line (with architecture: in it).
• The last -e command finds all empty lines and replaces them with a single -, which indicates a horizontal rule.

After that behemoth, the output is sent to receiptio, which prints to the printer over USB, with 42 columns of characters (the spec of my printer).

And sure enough it worked!

Further tweaking

One of the main things that bothered me about this print was that there was so much whitespace on the left side that wasn't being used, which could've been used for more text on the right. After looking through the docs, I found that I could use a special command, {w:}, to set the widths of each column to a specific value.

I played around a bit in the web viewer, and settled on the command {w:15 *}, which makes the left side 15 characters wide, and makes the right side the rest of the width (27 in my case). But I had one problem - the stack traces were now limited to being 15 characters wide, so I needed to reset the columns with {w:*} after the header. This looked like a tough problem at first, since sed only sees the file as independent lines. However, I was able to fix it by making the command piggyback off of the horizontal rule replacement - in other words, now it would replace empty lines with both {w:*} and -.

Prepending the {w:15 *} command appeared tough at first, but with some internet digging, I managed to come up with this command:

{ echo "{w:15 *}" ; coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/{w:*}\n-/g' ; } | receiptio -d /dev/usb/lp0 -c 42

This uses piping on two combined commands: first an echo, and then the coredump command, now with the {w:*} commands filled in. This worked fine, but I soon had to swap it out for a better solution.

Next I added a little logo to the top - it was kinda boring with just text. I took the monochrome Linux logo from the kernel, scaled it up, and added a ! after it to indicate a crash. Then I saved it as a monochrome PNG. ReceiptIO can display an image using the {i:} command, which takes base64 of a PNG file as an argument. I didn't want to shove all this into the same command line, so I split it off into a separate file /usr/local/share/coredump-header.txt:

{i:iVBORw0KGgoAAAANSUhEUgAAAWgAAADwCAAAAAAIzKGTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAEWElEQVR42u3d23KjMBCE4WjL7//Ksxdbu8XGhgh0QBJf38VxsOtPq5lBEqT4oh76BQHQQBPQQANNQANNp/Xq+WFp5/XgaAJ6MqU+wzZlvCc4moAWHSdzY/kA4WigRUe73Fg4PTgaaNFRPTfi7c3B0QT00HoN8j3S24/B0QS06Kjd1HA0Ab1qdCR0ORpo0XFHKRIcTes4eryiPji6phrlFdAyGmgCGmjlXbfzeDo8v6fVexaOBlp0XKv/43DlRqx+WZWjgRYdZ5X+hkN8Wr/xNUtpERwtOqhfdKSMX6WdYOFoAvrJVcc2HziagJ6qXr4/Or5VIE/bhs/RQKs6zibewQWNs2HC0QT0Q6Jj26ekzURt/J8nJmcJaNGx13TEfidicpaA7qvC0/JAO2fNsBDQqo7MpmN7GWTtnoWjgRYdVdJjpsVgHA30k1uTWxuWnA0s24ZlveaFo4FWdZwNvIOmY6/2SJv0WKNn4WigRcfZMuNjemhYCGjRkVl7/Nh+xf8bW74VIRxNQNcck+NFRyr4prHoVAtHA63q6BdsHE1APyY62s2MTF14cDTQ8zYnHaKjyhPcDmZm500PjgZa1dG/3lggPTgaaNHRp8CyJIyAfkJ0pJ1X3rfVR8afx3KFB0d36sqB7qQ6T7rPnI1NebmRWXg0TI/E0aoOuqdhiTOD/fiGPxxNQC9ZdVQ/RecHSKvCQ9UxS3sC9EJVR1Q6gsukBPTC0dFugL8HyMcnQAVHE9DzVx3VB/Iys7QcDbTo6FBy7H3cvIUHR3c69wA9YdVhsxtHAy06Ris5OJqAXrXqaFfDpvl7Fo7uVKkCzdFAE9DLVR1n10ylx18J4WiggSaggQYagj4dOdAcDTQBDfTjNcEMy/sTW2bc5MLRHD175Twj6Ni5MDtReogOoNfSlU33p27/VVhs3JiqHC06qGLV8bF9aPRw2JxbAM3SvHA00E+Ojr0HxVZPj8zjBEcT0MNHx/EDpvs/mD44moAeODryn8q0TY+6QztO3jKdo0UH3Rgdhdc/U+306Ky6l3/NGf58Zko1UAP9E+ZKqF91B2x890K/IV/xs9KnlwoP7mSYxfmr+JI30LWJ3pXRfdLjT2RVaZR2OZfdPJ6jO1kd6C7B0eiO6LdMlf6bpR1zly5Hd3I70J30ajSK71poEYXp0ex7c7SMFh2DV1hjLlDnaKBFR/74vbzKnaMJ6IqN/FxVx4X0SOv+FzhadKg6WqRHxdAoO1SkNvnN0aJjtDNeWUFy87WO1OZoRUwicfSdnraApgvpKO1kXh2+dKplqozcqNPZvaVH+WEtcvzZHpbtDttvLwq60Q1DVB0aFrocHeM/BKH6VjuOFh1UGB3b1d0DDs8pig2OBnpR5W7MuLznPZ38q8xAmKXY4GigHx4dl2Pk8sKwWCg3OBroRzUsB43MdhQfDOQW05wz5gZHA63qyKlDrrU2+ZkTHE1ALxEdxNFAA01AA01AAw00NdBvS6PYP/o0f6cAAAAASUVORK5CYII=}
{w:15 *}

Then I updated the command to concatenate that file and stdin from the coredumpctl/sed chain:

#!/bin/bash
sleep 1
coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/{w:*}\n-/g' | cat /usr/local/share/coredump-header.txt - | receiptio -d /dev/usb/lp0 -c 42

This produced my final output:

Oops...

While writing this blog, I had a little accident. plasmashell, the KDE desktop manager, crashed, and successfully demonstrated that the script worked! But...

Yeah, uh, plasmashell has a LOT of threads. 220" (555cm) of threads, in fact. It was hilarious to watch it keep printing for a solid 20 seconds straight (this printer is fast!), but I don't need a useless piece of paper that's longer than the room I live in, especially when I get these sorts of crashes every day. To safeguard, I made my final edit: stopping the output after 100 lines (should be reasonable).

#!/bin/bash
sleep 1
coredumpctl info -1 | sed -e 's/^ *//g' -e 's/: /:|/g' -e 's/^#/|#/g' -e 's/_/\\_/g' -e 's/architecture:|/architecture: /' -e 's/^$/{w:*}\n-/g' | cat /usr/local/share/coredump-header.txt - | head -n 100 | receiptio -d /dev/usb/lp0 -c 42

Conclusion

This was probably a complete waste of my time and nearly $100 in acquiring the receipt printer and paper. But it was a really fun time to put it together, especially when I cried laughing at the sheer length of that plasmashell crash. It also taught me some things about receipt printers and coredumps. Maybe someone will decide it's worth it to set this up, which is why I'm putting this post out there. Anyway, remember folks: if you're gonna accidentally print over 6 yards of useless receipts, it had better not be coming out of your pocket.