Breakpad is a library and tool suite that allows you to distribute an application to users with compiler-provided debugging information removed, record crashes in compact "minidump" files, send them back to your server, and produce C and C++ stack traces from these minidumps. Breakpad can also write minidumps on request for programs that have not crashed.
It works by stripping the debug information from the executable and saving it into "symbol files." When a crash occurs or upon request, the Breakpad client library generates the crash information in these "minidumps." The Breakpad minidump processor combines these files with the symbol files and generates a human-readable stack trace. The following picture, also from Breakpad's documentation, describes this process:
In WPE, Breakpad support was added initially for the downstream 2.28 branch by Vivek Arumugam and backported upstream. It'll be available in the soon-to-be-released 2.38 version, and the WebKit Flatpak SDK bundles the Breakpad client library since late May 2022.
As a developer feature, Breakpad support is disabled by default but can be enabled by passing
-DENABLE_BREAKPAD=1 to cmake when building WebKit. Optionally, you can also set
-DBREAKPAD_MINIDUMP_DIR=<some path> to hardcode the path used by Breakpad to save the minidumps. If not set during build time,
BREAKPAD_MINIDUMP_DIR must be set as an environment variable pointing to a valid directory when running the application. If defined, this variable also overrides the path defined during the build.
To generate the symbol files, Breakpad provides the
dump_syms tool. It takes a path to the executable/library and dumps to stdout the symbol information.
Once generated, the symbol files must be laid out in a specific tree structure so
minidump_stackwalk can find them when merging with the crash information. The folder containing the symbol files must match a hash code generated for that specific binary. For example, in the case of the
$ dump_syms WebKitBuild/WPE/Release/lib/libWPEWebKit-1.1.so > libWPEWebKit-1.1.so.0.sym $ head -n 1 libWPEWebKit-1.1.so.0.sym MODULE Linux x86_64 A2DA230C159B97DC00000000000000000 libWPEWebKit-1.1.so.0 $ mkdir -p ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000 $ cp libWPEWebKit-1.1.so.0.sym ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000/
Besides the symbol files, we need a minidump file with the stack information, which can be generated in two ways. First, by asking Breakpad to create it. The other way is, well, when the application crashes :)
To generate a minidump manually, you can either call
google_breakpad::WriteMiniDump(path, callback, context) or send one of the crashing signals Breakpad recognizes. The former is helpful to generate the dumps programmatically at specific points, while the signal approach might be helpful to inspect hanging processes. These are the signals Breakpad handles as crashing ones:
SIGILL(Note: this is for illegal instruction, not the ordinary
Now, first we must run Cog:
$ BREAKPAD_MINIDUMP_DIR=/home/lauro/minidumps ./Tools/Scripts/run-minibrowser --wpe --release https://www.wpewebkit.org
$ ps aux | grep WebProcess <SOME-PID> ... /app/webkit/.../WebProcess $ kill -TRAP <SOME-PID> $ ls /home/lauro/minidumps 5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp $ file ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp /home/lauro/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp: Mini DuMP crash report, 13 streams, Thu Aug 25 20:29:11 2022, 0 type
Note: In the current form, WebKit supports Breakpad dumps in the WebProcess and NetworkProcess, which WebKit spawns itself. The developer must manually add support for it in the UIProcess (the browser/application using WebKit). The exception handler should be installed as early as possible, and many programs might do some initialization before initializing WebKit itself.
Once we have a
.dmp crash log and the symbol files, we can use
minidump_stackwalk to show the human-readable crash log:
$ minidump_stackwalk ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp ./symbols/ <snip long header> Operating system: Linux 0.0.0 Linux 5.15.0-46-generic #49-Ubuntu SMP Thu Aug 4 18:03:25 UTC 2022 x86_64 CPU: amd64 family 23 model 113 stepping 0 1 CPU GPU: UNKNOWN Crash reason: SIGTRAP Crash address: 0x3e800000000 Process uptime: not available Thread 0 (crashed) 0 libc.so.6 + 0xf71fd rax = 0xfffffffffffffffc rdx = 0x0000000000000090 rcx = 0x00007f6ae47d61fd rbx = 0x00007f6ae4f0c2e0 rsi = 0x0000000000000001 rdi = 0x000056187adbf6d0 rbp = 0x00007fffa9268f10 rsp = 0x00007fffa9268ef0 r8 = 0x0000000000000000 r9 = 0x00007f6ae4fdc2c0 r10 = 0x00007fffa9333080 r11 = 0x0000000000000293 r12 = 0x0000000000000001 r13 = 0x00007fffa9268f34 r14 = 0x0000000000000090 r15 = 0x000056187ade7aa0 rip = 0x00007f6ae47d61fd Found by: given as instruction pointer in context 1 libglib-2.0.so.0 + 0x585ce rsp = 0x00007fffa9268f20 rip = 0x00007f6ae4efc5ce Found by: stack scanning 2 libglib-2.0.so.0 + 0x58943 rsp = 0x00007fffa9268f80 rip = 0x00007f6ae4efc943 Found by: stack scanning 3 libWPEWebKit-1.1.so.0!WTF::RunLoop::run() + 0x120 rsp = 0x00007fffa9268fa0 rip = 0x00007f6ae8923180 Found by: stack scanning 4 libWPEWebKit-1.1.so.0!WebKit::WebProcessMain(int, char**) + 0x11e rbx = 0x000056187a5428d0 rbp = 0x0000000000000003 rsp = 0x00007fffa9268fd0 r12 = 0x00007fffa9269148 rip = 0x00007f6ae74719fe Found by: call frame info <snip remaining trace>
This article briefly overviews enabling and using Breakpad to generate crash dumps. In a future article, we'll cover using Breakpad to get crashdumps while running WPE on embedded boards like RaspberryPis.