DEV Community: Lauro Moura

Using Breakpad to generate crash dumps with WPE WebKit

Lauro Moura — Mon, 29 Aug 2022 03:57:36 +0000

Introduction and BreakPad overview

Breakpad is a tool from Google that helps generate crash reports. From its description:

Breakpad is a library and tool suite that allows you to distribute an application to users with compiler-provided debugging information removed, record crashes in compact "minidump" files, send them back to your server, and produce C and C++ stack traces from these minidumps. Breakpad can also write minidumps on request for programs that have not crashed.

It works by stripping the debug information from the executable and saving it into "symbol files." When a crash occurs or upon request, the Breakpad client library generates the crash information in these "minidumps." The Breakpad minidump processor combines these files with the symbol files and generates a human-readable stack trace. The following picture, also from Breakpad's documentation, describes this process:

In WPE, Breakpad support was added initially for the downstream 2.28 branch by Vivek Arumugam and backported upstream. It'll be available in the soon-to-be-released 2.38 version, and the WebKit Flatpak SDK bundles the Breakpad client library since late May 2022.

Enabling Breakpad in WebKit

As a developer feature, Breakpad support is disabled by default but can be enabled by passing -DENABLE_BREAKPAD=1 to cmake when building WebKit. Optionally, you can also set -DBREAKPAD_MINIDUMP_DIR=<some path> to hardcode the path used by Breakpad to save the minidumps. If not set during build time, BREAKPAD_MINIDUMP_DIR must be set as an environment variable pointing to a valid directory when running the application. If defined, this variable also overrides the path defined during the build.

Generating the symbols

To generate the symbol files, Breakpad provides the dump_syms tool. It takes a path to the executable/library and dumps to stdout the symbol information.

Once generated, the symbol files must be laid out in a specific tree structure so minidump_stackwalk can find them when merging with the crash information. The folder containing the symbol files must match a hash code generated for that specific binary. For example, in the case of the libWPEWebKit:

$ dump_syms WebKitBuild/WPE/Release/lib/libWPEWebKit-1.1.so > libWPEWebKit-1.1.so.0.sym
$ head -n 1 libWPEWebKit-1.1.so.0.sym
MODULE Linux x86_64 A2DA230C159B97DC00000000000000000 libWPEWebKit-1.1.so.0
$ mkdir -p ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000
$ cp libWPEWebKit-1.1.so.0.sym ./symbols/libWPEWebKit-1.1.so.0/A2DA230C159B97DC00000000000000000/

Generating the crash log

Besides the symbol files, we need a minidump file with the stack information, which can be generated in two ways. First, by asking Breakpad to create it. The other way is, well, when the application crashes :)

To generate a minidump manually, you can either call google_breakpad::WriteMiniDump(path, callback, context) or send one of the crashing signals Breakpad recognizes. The former is helpful to generate the dumps programmatically at specific points, while the signal approach might be helpful to inspect hanging processes. These are the signals Breakpad handles as crashing ones:

SIGSEGV
SIGABRT
SIGFPE
SIGILL (Note: this is for illegal instruction, not the ordinary SIGKILL)
SIGBUS
SIGTRAP

Now, first we must run Cog:

$ BREAKPAD_MINIDUMP_DIR=/home/lauro/minidumps ./Tools/Scripts/run-minibrowser --wpe --release https://www.wpewebkit.org

Crashing the WebProcess using SIGTRAP:

$ ps aux | grep WebProcess
<SOME-PID> ... /app/webkit/.../WebProcess
$ kill -TRAP <SOME-PID>
$ ls /home/lauro/minidumps
5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp
$ file ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp
/home/lauro/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp: Mini DuMP crash report, 13 streams, Thu Aug 25 20:29:11 2022, 0 type

Note: In the current form, WebKit supports Breakpad dumps in the WebProcess and NetworkProcess, which WebKit spawns itself. The developer must manually add support for it in the UIProcess (the browser/application using WebKit). The exception handler should be installed as early as possible, and many programs might do some initialization before initializing WebKit itself.

Translating the crash log

Once we have a .dmp crash log and the symbol files, we can use minidump_stackwalk to show the human-readable crash log:

$ minidump_stackwalk ~/minidumps/5c2d93f2-6e9f-48cf-6f3972ac-b3619fa9.dmp ./symbols/
<snip long header>
Operating system: Linux
                  0.0.0 Linux 5.15.0-46-generic #49-Ubuntu SMP Thu Aug 4 18:03:25 UTC 2022 x86_64
CPU: amd64
     family 23 model 113 stepping 0
     1 CPU

GPU: UNKNOWN

Crash reason:  SIGTRAP
Crash address: 0x3e800000000
Process uptime: not available

Thread 0 (crashed)
 0  libc.so.6 + 0xf71fd
    rax = 0xfffffffffffffffc   rdx = 0x0000000000000090
    rcx = 0x00007f6ae47d61fd   rbx = 0x00007f6ae4f0c2e0
    rsi = 0x0000000000000001   rdi = 0x000056187adbf6d0
    rbp = 0x00007fffa9268f10   rsp = 0x00007fffa9268ef0
     r8 = 0x0000000000000000    r9 = 0x00007f6ae4fdc2c0
    r10 = 0x00007fffa9333080   r11 = 0x0000000000000293
    r12 = 0x0000000000000001   r13 = 0x00007fffa9268f34
    r14 = 0x0000000000000090   r15 = 0x000056187ade7aa0
    rip = 0x00007f6ae47d61fd
    Found by: given as instruction pointer in context
 1  libglib-2.0.so.0 + 0x585ce
    rsp = 0x00007fffa9268f20   rip = 0x00007f6ae4efc5ce
    Found by: stack scanning
 2  libglib-2.0.so.0 + 0x58943
    rsp = 0x00007fffa9268f80   rip = 0x00007f6ae4efc943
    Found by: stack scanning
 3  libWPEWebKit-1.1.so.0!WTF::RunLoop::run() + 0x120
    rsp = 0x00007fffa9268fa0   rip = 0x00007f6ae8923180
    Found by: stack scanning
 4  libWPEWebKit-1.1.so.0!WebKit::WebProcessMain(int, char**) + 0x11e
    rbx = 0x000056187a5428d0   rbp = 0x0000000000000003
    rsp = 0x00007fffa9268fd0   r12 = 0x00007fffa9269148
    rip = 0x00007f6ae74719fe
    Found by: call frame info
<snip remaining trace>

Final words

This article briefly overviews enabling and using Breakpad to generate crash dumps. In a future article, we'll cover using Breakpad to get crashdumps while running WPE on embedded boards like RaspberryPis.

WebAssembly Runtimes - WASI and Wasmtime

Lauro Moura — Fri, 18 Oct 2019 02:54:32 +0000

After a while (and a release at work), here we are again. To continue the series, we will take a quick look at wasmtime and WASI.

Wasmtime

Wasmtime is a "standalone wasm-only runtime for WebAssembly using the Cranelift JIT". It is part of CraneStation, an "umbrella" project for compiler tools where Mozilla is one (if not the main) parties involved.

The Cranelift JIT is kinda similar to LLVM. Other runtimes like Wasmer - which we will talk about in a future post can switch between Cranelift and LLVM as the backend to generate the code.

But before we go into more wasmtime details, it is important to talk about how we will be interacting with these runtimes.

WASI

As I said in the first post, portability is an important target of WebAssembly. Not only about running a given .wasm file in any place, but also about WebAssembly providing a - as much as possible - portable interface to build programs on top of it. This led to the development of the WebAssembly System Interface - WASI (here is a great post by the great Lin Clark about what is a system interface and why, what, how and who is developing one for WebAssembly).

While system interfaces like POSIX served us well for decades, the challenges of a truly universal system interface in the context of WASM are more strict. This happens because, like a webpage in the browser, you could be running WASM programs from any source and this demands a new approach to security.

WASI thus is being developed with a capability-oriented approach. Instead of the POSIX way of a program inheriting all access rights of the user, in WASI it behaves kinda like installing an app on your phone. Each time a WASM program tries to access a restricted resource, the runtime checks if the user granted access to that resource. We will see more details later in a WASI example.

To begin, we need to install the wasi-sdk that will make it easier for us to create WASI-ready wasm files. It is a clang-based toolchain that is also part of the CraneStation initiative. There are packages available in Github. In my Ubuntu install, the executables were installed to /opt/wasi-sdk/bin. The tutorial for wasi-sdk is here.

Getting wasmtime

The easiest way to get wasmtime is by running their installer with the command below. It will install the runtime to $HOME/.wasmtime/bin and automatically set up the profile scripts to add it to the $PATH when you open a new terminal. Don't forget to re-add the wasi-sdk bin folder to PATH in this new terminal.

$ curl https://wasmtime.dev/install.sh -sSf | bash

As expected a hello world in wasm/wasi is the same as in the desktop:

#include <stdio.h>

int main(void)
{
    printf("Hello, world!\n");
    return 0;
}

To compile, you use the wasi-sdk clang just like you would with a regular C compiler:

$ clang hello.c -o hello.wasm

And to run, just call the wasmtime executable:

$ wasmtime hello.wasm
Hello, World

That's it. You should have the message printed in your terminal. WASI allows the standard output and error descriptors (stdout and stderr) to be freely accessed.

Now, what if we tried to access files? For example, let's implement a cat program. It just reads a file and prints it to the terminal. Save the program below to a file named cat.c.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>

int main(int argc, char* argv[])
{
    if (argc != 2) {
        fprintf(stderr, "usage: %s <input file>\n", argv[0]);
        exit(1);
    }

    int n = 0;
    char buf[BUFSIZ];

    int f = open(argv[1], O_RDONLY);
    if (f < 0) {
        fprintf(stderr, "error opening file %s: %s\n", argv[1], strerror(errno));
        exit(0);
    }

    while((n = read(f, buf, BUFSIZ)) > 0) {
        printf("%.*s", n, buf);
    }

    if (n < 0) {
        fprintf(stderr, "read error: %s\n", strerror(errno));
        exit(0);
    }
    return 0;
}

Like the previous example, compile...

$ clang cat.c -o cat.wasm

And run...

$ wasmtime cat.wasm hello.c
error opening file hello.c: Capabilities insufficient

Ooops. Looks like reading files is a restricted operation. Which makes sense, given files could hold sensitive information. So, how do we grant access to them? With wasmtime, just use the option --dir=<path>:

$ wasmtime --dir=. cat.wasm hello.c
#include <stdio.h>...

Conclusion

This was just a gentle introduction to using wasi-sdk and wasmtime to play with WASI-based Webassembly programs. In the next post, we will see how to embed wasmtime into other programs and do more complex interactions.

WebAssembly runtimes overview - intro

Lauro Moura — Sat, 28 Sep 2019 03:30:14 +0000

Write once, run anywhere - the Holy Grail of Portability

Since long time ago, one of the holy grails of computing is portability. In the 90's Java promised, "write once, run anywhere (WORA)". But while you could do that with simple programs, you probably would quickly start fighting the details of the different OS's, devices, versions, you had to run. WORA then quickly becomes WODE (Write Once, Debug Everywhere). Another issue is that you would end up depending on the JVM being available on the target and having to program mainly with just Java or other JVM languages like - at the time - Groovy, JRuby, etc.

Fast forward to the 2010s, and another "Java" - I know, I know - is what had got us the closest to the WORA world so far. Javascript slowly overcame some hurdles from being something like a joke to become a - slightly more - respectable language and the universal programming language of the Web.

But we are still kinda tied to a single language. Transpilers try to mitigate this issue but you could need to develop systems for the Web in other programming languages completely different from JavaScript, to reuse existing components, for example.

Enter WebAssembly

All this led to the creation of WebAssembly. It is a compact binary format aimed at fast execution and small binaries sizes. Like Javascript, it runs sandboxed, providing security features that allow running code from the Web without - mostly - putting your system at risk. It already allows running C, C++, Rust and many other languages (thanks, LVVM) "anywhere".

The quotes are because, unfortunately, there is still the issue of dealing with the details of where you run. To try to reduce this task, there is an ongoing work to standardize the basic operations in a system interface (WASI).

So, what now?

Sometimes you need portability without all the bloat of HTML/CSS/JS parsing/rendering/etc. This led to the creation of platforms like Docker, where you could reuse the software from practically any language and run anywhere you had the docker engine running.

While the main showcase of WebAssembly initially was the web browser, there is new exciting work on using it outside it too. Like Solomon Hykes (of docker itself) said:

Solomon Hykes

@solomonstre

If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task! twitter.com/linclark/statu…

20:39 PM - 27 Mar 2019

Lin Clark @linclark
WebAssembly running outside the web has a huge future. And that future gets one giant leap closer today with... 📢 Announcing WASI: A system interface for running WebAssembly outside the web (and inside it too) https://t.co/HdEAZAyqYu

694 1691

This series of posts is intended to be a quick overview of some of the main tools that allow running WebAssembly outside the browser.

The initial list of runtimes we'll be checking:

We'll cover the scenarios each runtime tries to address. From running standalone wasm binaries to embedding the runtime into an actual application.

A more extensive list of WebAssembly runtimes can be found on Github.

So, on to the runtimes.

Generic extension methods ambiguity in C#

Lauro Moura — Thu, 15 Aug 2019 19:33:59 +0000

As part of our work in the C# bindings for EFL, recently I stumbled upon a kinda non-obvious issue using C#'s extension methods.

While implementing support for the MVVM factories, extension methods on the factories were generated so you could bind properties from the model to properties of the objects that would be created. For example

var factory = new Efl.ItemFactory<ListItem>();
factory.Text().bind("modelProperty");

The factory will create instances of ListItem and upon instantiating them, will bind the modelProperty of the associated model element to the Text property of the newly created ListItem so whenever the model changes, the created widget shows the updated value.

To make these properties available, the following extension methods were generated for each property in each "wrappable" class (i.e. class that could go inside a factory):

/// Extensions for ListItem
static Bindable<string> Text<T>(this ItemFactory<T> fac) where T : ListItem
{
  ...
}
/// Extensions for GridItem
static Bindable<string> Text<T>(this ItemFactory<T> fac) where T : GridItem
{
  ...
}

So far, so good. But the code above has an important limitation. The where constraint is not part of the signature. And if we tried to call one of these extension methods that have the same name but two different Ts, ambiguity arises and the compilation will fail with CS0121 The call is ambiguous between the following methods or properties:....

The solution we used (thank god Stack Overflow) was to add an extra default parameter dependent on the T and the type of the constraint. This would allow selecting an unique method based on the generic type parameter.

Using said method, we could do something like:

/// Magic tag to select the unique method
class MagicTag<TBase, TInherited> : where TInherited : TBase
{ }

/// Extensions for ListItem
static Bindable<string> Text<T>(this ItemFactory<T> fac, MagicTag<ListItem, T> x=null) where T : ListItem
{
  ...
}
/// Extensions for GridItem
static Bindable<string> Text<T>(this ItemFactory<T> fac, MagicTag<GridItem, T> x=null) where T : GridItem
{
  ...
}

This successfully gives enough info to the compiler select the single correct method without ambiguity in most cases. The exception is where we would have two methods with the same name and TInherited is actually a derived class from TBase. But in our case the code generator avoids generating repeated methods for inherited classes, bypassing this issue.