Yocto: roll your own embedded Linux distribution

What is Linux?

Linux is an impressive operating system that can run on a wide range of devices, from supercomputers to cell phones, manufacturing devices, network switches, and even cow milking machines. What's even more amazing is that this software is maintained by thousands of skilled software engineers and is available for free.

Initially, Linux was not developed as an embedded operating system. Instead, it was created by Linus Torvalds, a Finnish university student, who made his work available to everyone, accepted feedback from others, and delegated tasks to other skilled engineers. As the project grew, it attracted more talented engineers who contributed to Linux, increasing its value and visibility and thus creating a virtuous cycle that continues to this day.

Linux was first designed to run on the Intel IA-32 architecture and later ported to a Motorola processor. The porting process was difficult, so Linus Torvalds decided to redesign the architecture so that it could be easily ported, creating a clean interface between the processor-dependent parts of the software and those that are architecture independent. This decision paved the way for Linux to be ported to other processors.

Linux is a kernel that manages communication between software and hardware, but it is not very useful on its own. Linux distributions combine software from various projects to create a complete operating system. Popular distributions include Debian, Ubuntu, Fedora, Red Hat, and CentOS, which cater to different use cases such as desktops, servers, and embedded devices. Although these distributions share the same major Linux kernel version, each one curates a specific set of system software, libraries, and functionalities to meet different user needs and hardware architectures. The portability of Linux software, such as the GNU toolchain, has contributed to its success in embedded systems. Much of this software was designed to be adaptable across platforms, making it a perfect fit for resource-constrained embedded devices. This has extended the reach of Linux far beyond traditional desktops.

Why Use Embedded Linux?

Embedded Linux is similar to the Linux distributions that run on millions of desktops and servers worldwide, but it is tailored to a specific use case. Unlike desktop and server machines, embedded devices have limited resources such as memory, processor cycles, power consumption, and storage space. These resources matter a lot in the embedded field because they determine the unit cost of a device that may be produced in the millions. For instance, even a few extra MB or GB of storage can significantly impact the cost of a device. Similarly, extra memory may require additional batteries, which can add weight to the device. Moreover, a processor with a high clock speed produces heat, and some environments have very tight heat budgets, which means only limited cooling is available. Therefore, in embedded programming, if you're using Linux or some other operating system, most of the efforts focus on optimizing limited resources to make the most out of them.

When it comes to embedded operating systems, Linux is not the most lightweight option available, as compared to others like VxWorks, Integrity, and Symbian. Some embedded applications make use of frameworks like ThreadX for application support, which runs directly on the hardware, without the need for an operating system. Another option is to skip the framework and write code that runs directly on the device's processor.

The main difference between using a traditional embedded operating system and Linux is the separation between the kernel and the applications. In Linux, applications run in an execution context that is entirely separate from the kernel. This means that the application cannot access memory or resources other than what the kernel allocates. This high level of process protection ensures that if a program is defective, it is isolated from the kernel and other programs, resulting in a more secure and robust system. However, this protection comes at a cost.

The adoption of Linux is on an increasing trend despite its high resource overhead when compared to other options. This suggests that engineers working on projects find the added cost of Linux to be worth it. Although the cost and power demands of system-on-chip (SOC) processors have decreased in recent years, to the extent that they are no more expensive than a low-power 8-bit microcontroller from the past, more sophisticated processors can still be used where necessary. Many design solutions utilize off-the-shelf SOC processors and don't run the leads from the chip for the Ethernet, video, or other unused components.

Linux has become popular due to its ability to provide unique capabilities and features that aren't available in other embedded solutions. These capabilities are crucial to implementing sophisticated designs that distinguish devices in today's market. The open-source nature of Linux allows embedded engineers to take advantage of the continuous development happening in the open-source environment. This development occurs at a pace that no single software vendor can match.

Here's where Embedded Linux shines, offering a compelling combination of technical advantages:

• Modular Kernel and Open-Source Power: The core strength of Embedded Linux lies in the modularity of the Linux kernel. This allows developers to include only the necessary components, minimizing resource usage and optimizing performance for the specific hardware. Additionally, the open-source nature of Linux grants access to a vast developer community and a wealth of existing code, accelerating development and reducing costs.
• Robust Network Stack: Embedded Linux boasts a mature and reliable network stack, a crucial element for devices that need to communicate with other systems. This stack provides functionalities like TCP/IP (Transmission Control Protocol/Internet Protocol), enabling functionalities like file transfer, remote access, and internet connectivity. Protocols like Wi-Fi, Bluetooth, and cellular networking can also be integrated, allowing embedded devices to seamlessly connect to various networks.
• SSH (Secure Shell) for Secure Remote Access: Embedded Linux often includes SSH, a secure communication protocol that allows developers and administrators to remotely access and manage devices. This eliminates the need for physical interaction with potentially hard-to-reach devices, streamlining maintenance and troubleshooting processes.
• Web Server Capabilities: For embedded devices that need to serve web content, lightweight web servers like Lighttpd or uhttpd can be incorporated within the Embedded Linux system. These servers are ideal for situations where a full-fledged web server might be overkill, offering a resource-efficient way to provide a web interface for configuration or data visualization.
• C Library ( libc ): The C library, or libc, is a fundamental component of Embedded Linux. It provides a collection of essential functions for C programming, simplifying common tasks like memory management, input/output operations, and string manipulation. This rich library empowers developers to create efficient and portable C applications for embedded systems.

What is Yocto?

The Yocto Project is an open-source collaboration project that provides templates, tools, and methods to help us create customized Linux-based systems for embedded products, regardless of the hardware architecture. It can generate tailored Linux distributions based on the glibc and musl C standard libraries, as well as Real-Time Operating System (RTOS) toolchains for bare-metal development.
The Yocto Project aggregates several companies, communities, projects, and tools with the same purpose – to build Linux-based embedded products. These stakeholders are in the same boat, driven by their community needs to work together.
The Yocto Project provides various tools that prepare its build environment, utilities, and toolchains, which minimize the host software dependency. These tools are independent of the host Linux distribution, which reduces the number of host utilities required to produce the same result. This has an essential benefit of considerably increasing determinism, reducing build host dependencies, and increasing first-time builds.

Background

The OpenEmbedded project was initiated in January 2003 by core developers from the OpenZaurus project. They began working with a new build system, which led to the development of the OpenEmbedded build system. This build system is inspired and based on the Gentoo Portage package system called BitBake. As a result, the project has grown its software collection and the list of supported machines has expanded rapidly.
Due to the chaotic and uncoordinated development of OpenEmbedded, it was quite challenging to use it in products that require a more stable and polished code base. This led to the birth of the Poky distribution, which started as a subset of the OpenEmbedded build system. Poky had a more polished and stable code base across a limited set of architectures. Moreover, its reduced size allowed Poky to develop highlighting technologies, such as IDE plugins and Quick Emulator (QEMU) integration, which are still in use today.
In an effort to streamline their work, the Yocto Project and OpenEmbedded project joined forces to create a core build system known as OpenEmbedded Core. This system combines the best elements of both Poky and OpenEmbedded, with a focus on incorporating additional components, metadata, and subsets. In November of 2010, the Linux Foundation announced that the Yocto Project would take over this work as a sponsored project.

Poky

Poky is a reference distribution of the Yocto Project that utilizes the OpenEmbedded build system technology. It consists of a set of tools, configuration files, and recipe data (called metadata), that is platform-agnostic and supports cross-compiling through the BitBake tool, OpenEmbedded Core, and a default set of metadata. Additionally, it offers the capability to merge and build numerous distributed open-source projects to create a fully customizable, comprehensive, and coherent Linux software stack.

BitBake

BitBake is a powerful task scheduling and execution system that can parse Python and Shell Script code. The code parsed by BitBake generates and runs tasks, which are a set of steps ordered as per the code's dependencies.

With BitBake, you can evaluate all available metadata, manage dynamic variable expansion, dependencies, and code generation. It also keeps track of all tasks to ensure their completion, maximizing the use of processing resources to reduce build time and improve predictability.

OpenEmbedded Core

The OpenEmbedded Core metadata collection provides the engine of the Poky build system. It provides the core features and aims to be generic and as lean as possible. It supports seven different processor architectures (ARM, ARM64, x86, x86-64, PowerPC, PowerPC 64, MIPS, MIPS64, RISC-V32, and RISC-V 64), only supporting platforms to be emulated by QEMU.

Metadata

The metadata comprises both recipes and configuration files, which are composed of a combination of Python and Shell Script text files, making it an incredibly flexible tool. Poky utilizes this metadata to expand OpenEmbedded Core and includes two different layers, each of which is a subset of metadata. These layers are:

• meta-poky: This layer provides the default and supported distribution policies, visual branding, and metadata tracking information such as maintainers, upstream status, etc. This layer is designed to act as a curated template that could be utilized by distribution builders to create their own custom distribution.
• meta-yocto-bsp: This layer provides the Board Support Package (BSP) used as the reference hardware for the Yocto Project development and Quality Assurance (QA) process.

The Yocto Project releases

The Yocto Project has a biannual release cycle in April and October. This ensures continuous development while focusing on stability and providing points of increased testing. A release is considered Stable or Long-Term Support (LTS) depending on its readiness.

The support period for each release differs significantly. The stable release is supported for 7 months, with 1 month of overlapped support for every stable release. The LTS release, on the other hand, has a minimum support period of 2 years, which can be optionally extended.

After the official support period ends, a release moves to Community support and eventually reaches End of Life (EOL). A release becomes Community supported if a community member becomes the community maintainer. If there is no change in the source code for 2 months or the community maintainer is no longer active, the release reaches EOL.

You can access the release information of the Yocto Project by using the release page.

Preparing Build System

The process needed to set up our host system depends on the Linux distribution we use. Poky has a set of supported Linux distributions. Let’s suppose we are new to embedded Linux development. In that case, it is advisable to use one of the supported Linux distributions to avoid wasting time debugging issues related to the host system support.
If you use the current release of one of the following distributions, you should be good to start using the Yocto Project on your machine:

• Ubuntu
• Fedora
• CentOS
• AlmaLinux
• Debian
• OpenSUSE Leap

If your preferred distribution is not in the preceding list, it doesn’t mean it is not possible to use Poky on it. Your host development system must meet some specific versions for Git, tar, Python, and GCC. Your Linux distributions should provide compatible versions of those base tools.

Debian-based distribution

To install the necessary packages for a headless host system, run the following command:

sudo apt install gawk wget git diffstat unzip texinfo gcc
build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev pylint3 xterm python3-subunit mesa-common-dev zstd liblz4-tool

If you have Ubuntu, your default shell is dash. Some scripts might have a problem with it, so you should change dash to bash with the command below:

dpkg-reconfigure dash

and select "NO" on the prompt.

Fedora

To install the needed packages for a headless host system, run the following command:

sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat python3-pexpect findutils which file cpio python python3-pip xz python3-GitPython python3-jinja2 SDL-devel xterm rpcgen mesa-libGL-devel perl-FindBin perl-File-Compare perl-File-Copy perl-locale zstd lz4

Downloading the Poky source code

After we have installed the required packages on our development host system, we can download the current LTS version (at the time of writing) of Poky source code using Git, with the following command:

git clone https://git.yoctoproject.org/poky -b Scarthgap

Preparing the build environment

Inside the poky directory exists a script named oe-init-build-env, which sets up the building environment. But first, the script must be run-sourced (not executed) as follows:

source oe-init-build-env [build-directory]

Here, [build-directory] is an optional parameter for the name of the directory where the environment is configured. If it is empty, it defaults to build. The [build-directory] parameter is the place where we perform the builds.
The output from source oe-init-build-env build displays some important configurations such as the file location, some project URLs, and some common targets, such as available images.

Knowing the local.conf file

When we initialize a build environment, it creates a file called build/conf/local.conf. This config file is powerful, since it can configure almost every aspect of the build process. We can set the target machine and the toolchain host architecture to be used for a custom cross-toolchain, optimize options for maximum build time reduction, and so on. The comments inside the build/conf/local.conf file are excellent documentation and a reference of the possible variables and their defaults. The minimal set of variables that we probably want to change from the default is the following:

MACHINE ??= "qemux86-64"

The MACHINE variable is where we determine the target machine we wish to build. At the time of writing, Poky supports the following machines in its reference BSP:

• beaglebone-yocto: This is BeagleBone, which is the reference platform for 32-bit ARM
• genericx86: This is generic support for 32-bit x86-based machines
• genericx86-64: This is generic support for 64-bit x86-based machines
• edgerouter: This is EdgeRouter Lite, which is the reference platform for 64-bit MIPS The machines are made available by a layer called meta-yocto-bsp. Besides these machines, OpenEmbedded Core, inside the meta directory, also provides support for the following Quick Emulation (QEMU) machines:
• qemuarm: This is the QEMU ARMv7 emulation
• qemuarmv5: This is the QEMU ARMv5 emulation
• qemuarm64: This is the QEMU ARMv8 emulation
• qemumips: This is the QEMU MIPS emulation
• qemumips64: This is the QEMU MIPS64 emulation
• qemuppc: This is the QEMU PowerPC emulation
• qemuppc64: This is the QEMU PowerPC 64 emulation
• qemux86-64: This is the QEMU x86-64 emulation
• qemux86: This is the QEMU x86 emulation
• qemuriscv32: This is the QEMU RISC-V 32 emulation
• qemuriscv64: This is the QEMU RISC-V 64 emulation Extra BSP layers available from several vendors provide support for other machines.

Building a target image

Poky provides several predesigned image recipes we can use to build our binary image. We can check the list of available images by running the following command from the poky directory:

ls meta*/recipes*/*images/*.bb

All the recipes provide images that are a set of unpacked and configured packages, generating a filesystem that we can use with hardware or one of the supported QEMU machines.
Next, we can see the list of most commonly used images:

• core-image-minimal: This is a small image allowing a device to boot. It is handy for kernel and bootloader tests and development.
• core-image-base: This console-only image provides basic hardware support for the target device.
• core-image-weston: This image provides the Wayland protocol libraries and the reference Weston compositor.
• core-image-x11: This is a basic X11 image with a terminal.
• core-image-sato: This is an image with Sato support and a mobile environment for mobile devices that use X11. It provides applications such as a terminal, editor, file manager, media player, and so on.
• core-image-full-cmdline: A console-only image with more full-featured Linux system functionality installed.

For example, to build core-image-full-cmdline, run the following command:

bitbake core-image-full-cmdline

Running images in QEMU

We can use hardware emulation to speed up the development process, as it enables a test run without involving any actual hardware. Fortunately, most projects have only a tiny portion that is hardware-dependent.
QEMU is a free, open source software package that performs hardware virtualization. QEMU-based machines allow testing and development without real hardware. ARMv5, ARMv7, ARMv8, MIPS, MIPS64, PowerPC, PowerPC 64, RISC-V 32, RISC-V 64, x86, and x86-64 emulations are currently supported. We will go into more detail about QEMU usage in sw, Speeding Up Product Development through Emulation – QEMU.
OpenEmbedded Core provides the runqemu script tool, which is a wrapper to make use of QEMU easier. The way to run the script tool is as follows:

runqemu [machine] [zimage] [filesystem]

Here, [machine] is the machine/architecture to be used as qemux86-64, or any other supported machine. Also, [zimage] is the path to a kernel (for example, bzImage-qemux86-64.bin).
Finally, [filesystem] is the path to an ext4 image (for example, filesystem-qemux86-64.ext4) or an NFS directory. All parameters in the preceding call to runqemu [zimage] and [filesystem] are optional. Just running runqemu is sufficient to launch the image in the shell where the build environment is set, as it will automatically pick up the default settings from building the environment. For example:

runqemu qemux86-64 core-image-full-cmdline

References

• Salvador, O., & Angolini, D. (2017). Embedded Linux Development using Yocto Projects: Learn to leverage the power of Yocto Project to build efficient Linux-based products. Packt Publishing Ltd.

• Simmonds, C. (2017). Mastering Embedded Linux Programming. Packt Publishing Ltd.

• Sally, G. (2010). Pro Linux embedded systems. Apress.