Howto: Building Chromium on Ozone-GBM for RPi4

Introduction

I recently spent several days on building Chromium on the Ozone-GBM platform for Raspberry Pi 4 Model B (RPi4).

This attempt is to gain smaller footprint and better performance for my embedded system projects, but unfortunately no satisfactory result for now. In this article I'll share steps and tips I got so that you can join the effort as a tester or a developer.

My former attempt with Ozone-Dispmanx for RPi3 has been shared in GitHub:

yaegashi / ozone-dispmanx

Chromium Ozone port to Raspberry Pi dispmanx

Chromium Ozone port to Raspberry Pi dispmanx

Introduction

This is a Chromium Ozone port to Raspberry Pi using dispmanx. Dispmanx is a native display management API supporting OpenGL ES/EGL surfaces on VideoCore IV GPU equipped by Raspberry Pi. It can achieve siginificant performance improvement compared to the traditional X11 platform.

Current status:

• Only fullscreen content_shell works (no navigation, no address bar, no bookmarks)
• No GPU process separation, run with --in-process-gpu.
• Keyboard/mouse input support using evdev driver, but no VT console handling.
• It's forced to support WebGL - renderings are severely broken / no output for most pages.

Building

To build Chromium for Raspberry Pi, set up the build environment and check out src repository by following the official guide. Then follow the instruction below in src directory.

Set up ARM sysroot with dispmanx library:

./build/linux/sysroot_scripts/install-sysroot.py --arch=arm
git clone --depth 1 https://github.com/raspberrypi/firmware
cp -R firmware/hardfp/opt build/linux/debian_wheezy_arm-sysroot
cp firmware/hardfp/opt/vc/lib/pkgconfig/bcm_host.pc build/linux/debian_wheezy_arm-sysroot/usr/lib/pkgconfig

…

View on GitHub

Cross building environment

To build Chromium for RPi4, you'll first follow the generic instruction on the official developer site. Read Checking out and building Chromium on Linux.

System requirements:

• Linux build machine
  • Denoted as (build) in this article
  • OS: Ubuntu 18.04 or 20.04
  • CPU: AMD64, more than 4 cores / 8 threads recommended
  • Memory: more than 16GB recommended
  • Storage: more than 100GB free space
• Raspberry Pi 4 Model B
  • Denoted as (pi) in this article
  • OS: Raspberry Pi OS (32-bit) with desktop
  • Storage: 1GB free space

In this article, we're going to build Chromium executable for 32-bit ARM (Debian armhf).

Clone depot_tools.git anywhere you want:

(build) $ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

Add its path to PATH. Put the following line in ~/.bashrc and restart your shell:

export PATH=$PATH:/path/to/depot_tools

Run fetch to clone the Chromium sources anywhere you want, then run install-build-deps.sh to set up build dependencies:

(build) $ mkdir ~/chromium
(build) $ cd ~/chromium
(build) $ fetch --nohooks chromium
(build) $ ./src/build/install-build-deps.sh

To run Chromium on Ozone-GBM, you have to set up the cross-build environment targeting ChromiumOS on ARM. Details are explained in Linux Chromium Arm Recipes.

Edit .gclient file generated by previous fetch. Append 2 lines as follows:

target_cpu = ["arm"]
target_os = ["chromeos"]

Run gclient sync and runhooks:

(build) $ gclient sync
(build) $ gclient runhooks

This should acquire additional dependencies including ChromiumOS sources and install Debian armhf sysroot in src/build/linux/debian_sid_arm-sysroot.

Using ccache is strongly recommended, so set it up now:

(build) $ sudo apt install ccache
(build) $ ccache -M 20GB

First ninja build (ozone_demo)

Run gn in the source tree to prepare a directory for the production build:

(build) $ cd ~/chromium/src
(build) $ gn args out/p1

This creates a build directory out/p1 and launch a text editor on args.gn in it for you. You should put the following lines and save it.

# Set build arguments here. See `gn help buildargs`.
cc_wrapper = "ccache"
dcheck_always_on = true
enable_nacl = false
has_native_accessibility = false
is_chrome_branded = false
is_debug = false
is_official_build = false
ozone_auto_platforms = false
ozone_platform = "gbm"
ozone_platform_gbm = true
target_cpu = "arm"
target_os = "chromeos"
target_sysroot = "//build/linux/debian_sid_arm-sysroot"
use_evdev_gestures = false
use_glib = false
use_gtk = false
use_ozone = true
use_pulseaudio = false
use_system_minigbm = false
use_vc4_minigbm = true
use_xkbcommon = false

You can manually edit out/p1/args.gn. Make sure to run gn gen out/p1 after making any changes.

Now you can run ninja to build chromium executables. For the first attempt, I recommend to build ozone_demo for smoke testing:

(build) $ ninja -C out/p1 -j16 ozone_demo

-jN specifies the number of parallel build jobs. The recommended N setting is the number of CPU threads of the build machine. ozone_demo build contains less than 10k targets and it can take several hours depending on the spec of your build machine.

Each time it builds, you need to bring executable files to your RPi4. Because there're so many temporary files in the build directory, I use the following shell script pack.sh to make an archive of files needed on runtime:

#!/bin/bash

OUTDIR=${1-out/production}
DESTDIR=chromium-${OUTDIR##*/}
ARCHIVE=$DESTDIR.tar.gz

if ! test -d "$OUTDIR"; then
        echo "Not a directory: $OUTDIR"
        exit 1
fi

cd $OUTDIR
rm -rf $DESTDIR $ARCHIVE
mkdir -p $DESTDIR

tar     --exclude $DESTDIR \
        --exclude $ARCHIVE \
        --exclude gen \
        --exclude obj \
        --exclude pyproto \
        --exclude resources \
        --exclude Packages \
        --exclude 'newlib_*' \
        --exclude 'nacl_bootstrap*' \
        --exclude 'glibc_*' \
        --exclude 'clang_*' \
        --exclude 'irt_*' \
        --exclude '*.ninja*' \
        --exclude '*.runtime_deps' \
        --exclude '*.info' \
        --exclude '*.TOC' \
        -cf - . | tar -C $DESTDIR -xf -

tar -czvf $ARCHIVE $DESTDIR

pack.sh creates a tar+gz archive in the build directory. You can transfer it to your RPi4 host.

(build) $ ./pack.sh out/p1
chromium-p1/
chromium-p1/libEGL.so
chromium-p1/ozone_demo
chromium-p1/libminigbm.so
chromium-p1/gn_logs.txt
chromium-p1/android_clang_arm64/
chromium-p1/icudtl.dat
chromium-p1/libGLESv2.so
chromium-p1/android_clang_arm/
chromium-p1/swiftshader/
chromium-p1/swiftshader/libEGL.so
chromium-p1/swiftshader/libGLESv2.so
chromium-p1/args.gn
(build) $ ls -lh out/p1/chromium-p1.tar.gz 
-rw-rw-r-- 1 yaegashi yaegashi 13M Aug 16 13:49 out/p1/chromium-p1.tar.gz
(build) $ scp out/p1/chromium-p1.tar.gz [e-mail]:

RPi4 settings

This section is assuming Raspberry Pi 4B running the latest Raspbian 10 as of 2020-08-16. Tested with RPi4s of RAM 4GB/8GB.

First, check /boot/config.txt for dtoverlay=vc4-fkms-v3d and gpu_mem with sufficient memory (>256MB) exists.

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2

[all]
#dtoverlay=vc4-fkms-v3d
gpu_mem=256

With vc4-fkms-v3d enabled, you should be able to see device files in /dev/dri:

(pi) $ ls -l /dev/dri
total 0
drwxr-xr-x  2 root root        100 Aug 16 15:15 by-path
crw-rw----+ 1 root video  226,   0 Aug 16 15:15 card0
crw-rw----+ 1 root video  226,   1 Aug 16 15:15 card1
crw-rw----+ 1 root render 226, 128 Aug 16 15:15 renderD128

You have to be given permissions to access those devices. Use adduser to join video and render group, id to check your current membership:

(pi) $ sudo adduser pi video
(pi) $ sudo adduser pi render
(pi) $ exit
... logging in another session
(pi) $ id
uid=1000(pi) gid=1000(pi) groups=1000(pi),4(adm),20(dialout),24(cdrom),27(sudo),29(audio),44(video),46(plugdev),60(games),100(users),105(input),107(render),109(netdev),997(gpio),998(i2c),999(spi)

Unfortunately, the latest Raspbian kernel is known to have a problem in the DRM driver with Ozone-GBM as of the time I'm writing this article. If your Raspberry Pi is already running kernel version 5.4.x or above, you have to downgrade it to version 4.19.118 in the previous release.

Before downgrading, upgrade every package to the latest version, including Linux kernel:

(pi) $ sudo apt update
(pi) $ sudo apt dist-upgrade
(pi) $ sudo reboot

After reboot, manually downgrade raspberrypi-kernel package to version 1.20200512-2 acquired from the package repository:

(pi) $ wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel_1.20200512-2_armhf.deb
(pi) $ sudo apt install ./raspberrypi-kernel_1.20200512-2_armhf.deb 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'raspberrypi-kernel' instead of './raspberrypi-kernel_1.20200512-2_armhf.deb'
The following packages will be DOWNGRADED:
  raspberrypi-kernel
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 0 B/71.7 MB of archives.
After this operation, 1,245 kB disk space will be freed.
Do you want to continue? [Y/n] Y
...
Unpacking raspberrypi-kernel (1.20200512-2) over (1.20200811-1) ...
...
(pi) $ sudo reboot

After reboot, you should check the running kernel version by uname -a:

(pi) $ uname -a
Linux pi4b2 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux

You cannot run Ozone-GBM executables and Raspbian GUI desktop at the same time. To suspend the desktop, press Ctrl + Alt + F1 to switch to the VT console. To return to the desktop, press Alt + ← or → several times to get to VT7.

Now you're ready to run your first build. Extract copied archive and run ozone_demo executable with the following environment variable and command line switches:

(pi) $ tar -xzvf chromium-p1.tar.gz
(pi) $ cd chromium-p1
(pi) $ EGL_PLATFORM=surfaceless ./ozone_demo --disable-explicit-dma-fences

If everything goes well, you will be able to see your RPi4 cycling colors in the whole screen. Press Ctrl + C to stop it.

When it craches or renders unexpected output, you should collect debug log messages and examine what's going on:

(pi) $ EGL_PLATFORM=surfaceless EGL_LOG_LEVEL=debug ./ozone_demo --enable-logging --v=1 --disable-explicit-dma-fences 2>&1 | tee debug.log

Pro Tip: you can run Ozone-GBM executables in the SSH remote sessions, and use chvt to switch the VT console. Allocating a new VT (8th or later) is the safer and recommended option. Switch to 7th VT to return to the desktop:

(pi) $ sudo chvt 8
(pi) $ EGL_PLATFORM=surfaceless ./ozone_demo --disable-explicit-dma-fences
(pi) $ sudo chvt 7

Building and running Chromium

After succesfull smoke testing with ozone_demo, let's continue bulding the next executable: chrome, the chromium browser.

(build) $ ninja -C out/p1 -j16 chrome

This will display to you approx. 50k build targets and it can take 10+ hours to finish.

When it builds, run again pack.sh to make a new archive of the build directory and transfer to the RPi4 host:

(build) $ ./pack.sh out/p1
(build) $ scp out/p1/chromium-p1.tar.gz [e-mail]:

Extract it on RPi4 host and run chrome as follows:

(pi) $ tar -xzvf chromium-p1.tar.gz
(pi) $ cd chromium-p1
(pi) $ EGL_PLATFORM=surfaceless ./chrome --force-system-compositor-mode

This finally brings you the Chromium browser on Ozone-GBM on RPi4, along with the mocked ChromiumOS desktop.

Current status and known problems

Before you actually test it, I have to say we shouldn't expect too much on it for now. You'll soon realize it's quite buggy, unstable compared to the Raspbian desktop (Xorg) version, and has too many issues to use in production.

• Always too many flickers on frame updates. It often gets quite disordered and shows previous frames again. Compositor problem?
• WebGL is working, not perfect though.
  • Many of three.js examples work, but the performance is terribly low with badly noticeable flickers. Broken rendering seen in Ozone-Dispmanx on RPi3 gets fixed.
  • You can play AngryBots2 (Unity's demo), but it takes so long time to load the scene.
• For YouTube, it often suffers intermittent interrupts of playbacks in the beginning of movies before it gets stably smooth.
• The mouse wheel input is not working.
• The keyboard input is also passed to the shell in the active VT console. The current workaround would be to run on an unused VT console (chvt 8) from the remote shell.
• content_shell on Ozone-GBM cannot display anything, while the browser instance is running offscreen. Apparently it's missing event handlers that should set up screens connected.

Prebuilt binaries for testing

I've uploaded the archive of my build on OneDrive:

• https://1drv.ms/u/s!ArStVFUJcWgSjHOsjfgt21fAB_pD?e=KBP7Vg

Download chromium-p1.tar.gz and test it on your RPi4 if you have interest with Chromium on Ozone-GBM development. Beware that it won't run on the latest kernel from Raspbian and the special settings is needed. Read the instruction in this article.

Conclusions

Current Chromium on Ozone-GBM for RPi4 is experimental; it's full of issues and at limited performance, far from production use. But you can say it's just immature and in the early stage of development, so there's a great margin to improve.

After deprecation of dispmanx, Ozone-GBM with drm/v3d driver would be in the right direction for Chromium of embedded use for RPi4, worth for us to make effort.

I've shared the first step guide for developing and testing Ozone-GBM for RPi4, along with prebuilt binaries. I'll update this article as needed in future. Any comments and suggestions to improve Ozone-GBM and this article are much appreciated!

Acknowledgements

I thank people in Raspberry Pi's graphics programming forum for sharing valuable information.

Comments

[Ewan Roycroft]

Great article, thanks for sharing! Some comments/questions below:

I have not found it necessary to add target_cpu = ["arm"] to .gclient, I'm not sure if this has any particular benefits.

Another good way of sharing files between the (build) and (pi) machines is to use sshfs. In this way, you do not need to transfer the files every time you build. You simply mount the build location when you start your Pi and the files are kept in sync.

sshfs -o allow_other build-machine:/path/to/chromium chromium

Is there any reason you run ozone_demo with --disable-explicit-dma-fences? I have not come across this flag and have successfully run it without. Is there any particular benefit or was it just to test something?

Another useful thing to note is that the --kiosk flag works in Ozone-GBM. This removes most of the Chromium OS and browser UIs and leaves you with a clean webpage.

I have posted a crbug for the flickering/stuttering issue: crbug.com/1104165

Another bug I've found is that any keystrokes made in Chromium will also be entered into the terminal behind the scenes (see bbc/chromium #2).

[Takeshi Yaegashi]

Hi @ewanroycroft , thanks again for the valuable comment.

target_cpu = ["arm"] is mentioned in Linux Chromium Arm Recipes.

--disable-explicit-dma-fences is needed to avoid DCHECK() you are also facing in bbc/chromium #5. I suspect this function CreateForGpuFence() is almost meaningless for platforms other than Android...

I'll add bbc/chromium #2 as a known problem in the article. Probably we need implement correct VT console mode handling for Ozone-GBM. A possible workaround would be to switch to an unused VT by running chvt 8 in the remote shell.

I appreciate other suggestions and comments from you, and it's great that you've already filed an issue in crbug! It would be nice if I could provide concise reproducible steps and testcases to attract more attention from upstream developers.

[Ewan Roycroft]

Thanks for your reply, some interesting stuff here.

So target_cpu = ["arm"] simply pre-installs the sysroot, makes sense.

I will have to give --disable-explicit-dma-fences a go with issue #5, thanks.

I'll let you know if I find a definitive solution to issue #2, I think your suggestion sounds like a good idea.

I am discussing the crbug with one of the minigbm developers, who suggests it may be a VC4 KMS bug. I am going to try building/running the drm-tests in hopes of determining the cause, to pass on to Chromium/RPi devs. I haven't got any 100% reproducible test cases yet, but I have posted a video with some examples: youtu.be/HAYFYRwHQE8

[Takeshi Yaegashi]

You might want to see FydeOS people's repository:

FydeOS / chromium_os-raspberry_pi

Build your Chromium OS for Raspberry Pi 3B/3B+/4B

They have patches against kernel/mesa/minigbm fixing the flickering issue.

I've tried it out with their prebuilt image. The performance looks better than chrome I built, but the flickering issue seems to still remain, especially with the omnibox. And it won't start the desktop on RPi4 with 8GB.

[Conrad Kreyling]

I rebuilt both vc4.ko and v3d.ko using this patch from that repo, and ozone sprang to life. I should note I'm also building for 64-bit userspace via the new Ubuntu image, which provides a healthy uplift in performance. (And ostensibly fixes the 8GB limitation?)

[Ewan Roycroft]

@yaegashi
Have you ever tried running chrome with --disable-explicit-dma-fences? Or just ozone_demo?

I have just run Chrome with that flag set and it no longer suffers from the stuttering bug :)

[Steffen Deusch]

I've successfully built chromium ozone-gbm for an Intel NUC thanks to this tutorial. I did not expect this to work out of the box (although VA-API video decoding is totally garbled), but it's still a nice proof of concept. Thank you very much!
I'd love to see this without the need of targeting for ChromeOS and having a proper way of programatically configuring multiple displays (something like Joone describes at the end of this article medium.com/joone/implement-a-singl...).

Off topic: All things considered the state of accelerated Chromium on Linux feels still pretty immature (I'm coming from a hobbyist kiosk background).
Over the last months I've tried classic X11 with bad tearing issues, ozone/wayland with no current VA-API support (and, after figuring out tearing on X11 is fixed by using the EGL backend, worse performance than classic X11), and now ozone/gbm on a variety of different devices. For every device another combination is needed to get things working okay-ish. I think ozone/gbm could be a nice solution for this, but I'm not sure if there are any plans to support this besides of ChromeOS...

[Ewan Roycroft]

I am also working on using VAAPI with Ozone-GBM. You can fix the garbled output by ensuring minigbm uses Y-tiled buffers. See this thread:
crbug.com/1161729

I have also got V4L2 video acceleration working on the RPi:
github.com/bbc/chromium/commits/px...

Albeit with a bug re: hardware overlays:
crbug.com/1164957

[praveents]

I am trying to build the chromium-browser for a custom linux board using DRM/KMS. Below are the arg details used to build chromium.

While running chromium, I am getting DRM check failed.

FATAL:ozone_platform_drm.cc(90)] Check failed: false.

I have drm, kms library on target, and test code (non-ozone) works fine with DRM.

Is there anything that I am missing in build arguments and any thoughts why would ozone drm check failed would be of great help.

target_cpu = "arm"
arm_version = 7
target_os = "linux"
arm_arch = "armv7-a"
arm_float_abi = "hard"
arm_fpu = "neon"
arm_tune = "cortex-a9"
arm_use_neon = true
arm_use_thumb = true
dcheck_always_on = true
enable_nacl = false
has_native_accessibility = false
is_chrome_branded = false
is_debug = false
is_official_build = false
ozone_auto_platforms = false
use_ozone = true
ozone_platform_headless = true
ozone_platform = "drm"
ozone_platform_gbm = true
use_glib = false
use_gtk = false
use_pulseaudio = false
use_xkbcommon = false
use_cups = false
use_gio = false
use_kerberos = false
use_libpci = false
use_udev = false
rtc_use_pipewire = false
v8_enable_lazy_source_positions = false
use_pangocairo = false
enable_basic_printing = true
use_x11 = false
ozone_platform_drm = true

Thanks