Raghu Bharadwaj

Posted on Sep 16 Originally published at techveda.live

Containers vs Packages: Delivering Apps on an Embedded Device

#embedded #linux #yocto #docker

Keep the application inside the base image until the platform team and the application team release on different schedules. At that point deliver it as a separate signed artifact, replaced as a whole, not as packages installed on the running device. In the containers vs packages comparison, containers suit several independently released applications; a runtime package manager is the option to avoid in production, because it lets every unit in the field reach a different state.

Assume the platform decision is settled: the device boots from an A/B root filesystem built by Yocto or Buildroot, and the update client installs a signed bundle and rolls back after a failed boot. One question is usually still open, and is decided by default rather than by design: where does the application live, and how does a new version reach it? The containers vs packages framing covers the two answers teams argue about most, but four are worth weighing.

The context

The decision arises the first time two groups ship on different calendars. The platform group updates the kernel, the BSP and the base userspace every few months; the application group wants to ship a fix on a Thursday. With only a full base image to carry it, that fix needs a full platform release.

Release cadence. One image means one cadence for everything inside it.
Flash budget. In a symmetric A/B layout the root filesystem is stored twice: a 400 MB model file costs 800 MB.
Integrity. A read-only root verified with dm-verity cannot be written to after the build.
Rollback. A base rollback restores the base, not an application from another channel.
Test surface. Every independent channel multiplies the version combinations in the field.

If the A/B layout is not settled, decide that first; we covered it in Choosing an A/B Update Layout for Your Product.

Containers vs packages: the four options

Option 1: the application lives in the base image

The application is a recipe in your build system, lands in the root filesystem, and is updated by shipping a new base image. One artifact, one version number, one test matrix. Buildroot's manual states the position directly: binary packages allow partial upgrades, creating a large number of version combinations that would have to be tested, whereas replacing the whole image guarantees the system in the field is the one that was tested.

For: simplest integrity story, works with a read-only verified root, one version to support.
Against: one cadence for all software, a full download for a one-line fix, a reboot for every application change, duplicated flash.

Option 2: a runtime package manager on the target

Yocto can produce this. Adding package-management to IMAGE_FEATURES ships the package databases and the target-side package tools in the image, so the device can install and upgrade from a feed at runtime. PACKAGE_CLASSES selects the backend: package_rpm, package_deb or package_ipk. Buildroot does not offer this at all, because it does not track which package installed which file.

For: small incremental updates, and useful on development images where you want strace or gdb without a rebuild.
Against: incompatible with a read-only or dm-verity-protected root; dependency resolution runs on the device, so units drift apart; a compromised feed or signing key lets the device install arbitrary software.

Option 3: OCI containers

The application ships as a container image and a runtime such as Podman or Docker starts it. On Yocto this comes from the meta-virtualization layer, which carries the runtime recipes and expects virtualization in DISTRO_FEATURES. The base image stays read-only; the container store lives on the writable data partition.

For: the application and its dependencies travel together, so it does not depend on the libraries the base image carries; applications from different teams are versioned separately; namespaces and resource limits give isolation.
Against: the runtime, its storage layers and its network configuration are more software to harden and track for CVEs; the image store needs a writable filesystem; and the container layer gives no rollback of the base.

Option 4: a separate artifact, without a container runtime

Independent versioning does not require containers. Three mechanisms are worth knowing.

RAUC artifact repositories. RAUC installs artifacts into named repositories rather than into slots; these were added in RAUC 1.13. The cases its manual lists are container and VM images, large data files such as maps or machine learning models, firmware for other microcontrollers, and optional add-on services. An artifact is replaced under the same name, independently of the base and usually without a reboot.
systemd system extension images (sysext). A read-only image that extends /usr/ and /opt/ at runtime by overmounting them with an overlayfs. Images go in /var/lib/extensions/, and each carries an extension-release file whose ID= and SYSEXT_LEVEL= or VERSION_ID= must match the host, so compatibility is checked at merge time, before basic.target. In systemd since version 248, with no dependency resolution.
OSTree. A content-addressed store of complete filesystem trees. Deployments are hardlinks into /ostree/repo, so an upgrade costs space proportional to the new files plus a constant overhead.

Across all three:

For: independent versioning, single storage rather than one copy per slot, no container runtime.
Against: less common in most teams; no isolation; sysext binaries link against host libraries unless statically linked.

The decision

For the common case — one application, one team owning it, the same cadence as the platform — keep the application in the base image. A second delivery channel that nobody needs is cost without benefit, and it weakens the guarantee that the software in the field is the software you tested.

Once cadences separate, deliver the application as a separate signed artifact. Which mechanism depends on your needs.

Choose containers when you have more than one independently released application, when they come from different teams, or when you need isolation between them.
Choose a RAUC artifact repository or a sysext image when you have one or two applications, no isolation requirement, a tight flash budget, and a team that maintains the build.
Choose a runtime package manager only on development and bench images, or on an edge server whose operations team accepts fleet divergence.

Whichever you choose, write the compatibility contract down before the first field release: which application versions run on which platform versions, and what the device does outside that range. A sysext image enforces this through the extension-release match; otherwise you enforce it. Installing a system extension is:

raghu@techveda.org:~$ sudo cp app-2.4.raw /var/lib/extensions/
raghu@techveda.org:~$ sudo systemd-sysext refresh
raghu@techveda.org:~$ systemd-sysext list

We cover these decisions on real hardware in our Embedded Linux and Yocto training.

Consequences

You are testing a matrix, not a build. With two channels, the platform-and-application combinations grow with every release on either side. Decide which combinations you support and test those.

Rollback stops being one operation. The A/B mechanism rolls the base back after a failed boot, but not an application from another channel, because it is not in the slot being switched. A bad application version needs its own recovery path.

Storage moves rather than disappears. Taking a large payload out of the A/B root filesystem removes the duplicate copy, which is the reason RAUC gives for artifact repositories. But the store needs a writable partition, formatted for your flash.

Attack surface follows the choice. A container runtime is a large body of code you now track and patch. A package manager is smaller, but has a larger consequence if its key or feed is compromised. Baking the application in adds neither.

Read-only roots and package managers do not combine. If you have committed to dm-verity, option 2 is closed. Decide delivery together with the integrity design.

Key takeaways

Application delivery is driven by release cadence, not technology preference.
In the containers vs packages comparison, containers give independent versioning and isolation at the cost of a runtime you maintain; a package manager gives small updates at the cost of fleet divergence.
RAUC artifact repositories, systemd-sysext and OSTree give independent versioning without a container runtime.
Any separate channel needs an explicit compatibility contract and its own rollback path.
dm-verity and an on-device package manager cannot coexist, so decide delivery and integrity together.

Frequently asked questions

Can I put a package manager on a read-only root filesystem?
No. A package manager writes into the root filesystem, and a read-only root has no writable location for it. If that root is protected with dm-verity, any modification breaks hash verification. Use a container image, a RAUC artifact or a sysext image instead.

Does a container update replace the need for A/B base updates?
No. A container image update changes the application only. The kernel, bootloader, drivers and base userspace are still delivered by the base image mechanism, which is the only thing that can roll them back after a failed boot.

How does a separate artifact save flash compared with baking it in?
In a symmetric A/B layout the root filesystem exists in both slots, so a large payload is stored twice. RAUC gives this as the reason for artifact repositories: container images, map files and machine learning models go into a repository rather than a slot, so they are stored once, on a writable partition you then manage.

When does a sysext image fit better than a container?
When you have one or two applications, a read-only base image and no isolation requirement, a system extension gives independent versioning without a container runtime. It fits poorly when you need several mutually isolated applications, because system extensions have no dependency mechanism.

DEV Community