DEV Community: Bruno Verachten

Above this line: a lift ride under your hypervisor

Bruno Verachten — Thu, 05 Nov 2026 11:14:32 +0000

Originally published on November 5, 2026 at https://bruno.verachten.fr/2026/11/05/above-this-line-a-lift-ride-under-your-hypervisor/

▶ 0. The lift ride (you are here)
-1. One setting changed, two lines moved (26 November)
-2. The roommate in the closet (12 November)
Monday morning. No :latest for firmware (19 November)
RISC-V. Down to the last binary (3 December)

Photo: Orphan Girl Mine, Butte, Montana, by James St. John, CC BY 2.0, resized.

In October 2025, Ruter, the public transport operator in Oslo, asked itself a very simple question: who actually controls our electric buses? So they took two of their own, a brand-new Yutong built in China and a three-year-old VDL from the Netherlands, and drove them into an isolated environment inside a mountain. No network. Just two buses, some probes, and a lot of rock.

What did they find? The new one was built for over-the-air updates: its manufacturer could reach each bus directly, battery and power-supply control system included, over the mobile network, through a Romanian SIM card. (A Chinese bus, in Oslo, talking through Romania. I’ve stopped trying to make the geography work.) The old one had no OTA at all.

Ruter’s conclusion, in its own publication, is that the manufacturer could, in theory, stop the bus or render it unusable. In theory. Nobody switched off a bus. But keep the shape of it in mind: closed firmware that somebody else can update.

Hold on to the bus. It comes back.

Ruter wondered who controls its buses. I asked the same thing about servers, mine and yours, to open my talk at Cloud Nord 2026 in Lille. This is the written version (the slides are online, in French, sources in the appendix).

Hands up, hands down

First question to the room: who has ever opened a BMC’s web interface? (The BMC is the little board that lets you power a server on remotely, and a lot more.) A handful of hands went up. Then: who has ever read the code running inside it?

Every hand dropped.

That’s the whole talk, in two questions. (If you only run cloud VMs and have never touched a BMC: lucky you, there’s a paragraph for you further down.)

I’m Bruno, community manager at Vates (we make XCP-ng and Xen Orchestra). At Cloud Nord 2025 I was turning Android phones into Jenkins agents; this year I came back to teach people to distrust their real, perfectly legit servers. « Je progresse dans la paranoïa, c’est l’âge. » I’m getting more paranoid. It’s my age.

I’m no firmware expert, by the way: nearly everything below, I discovered from summer 2026 onwards, so this series is me doing my homework in public.

The method: the "descenseur"

We take the lift from the VM all the way down to the silicon. On stage I called it « le descenseur » (French says ascenseur, and we were only ever going down). Think strace, except we don’t stop at the kernel. On each floor, we call an imaginary function:

audit(floor) ==  0;  // readable: you read it, you patch it      -> green
audit(floor) ==  ?;  // partly: an open door exists, few use it  -> orange
audit(floor) == -1;  // EACCES: you can't read it or remove it   -> red

EACCES is what Linux hands you when it refuses you a file. At Cloud Nord the room voted with raised hands before each verdict. You can vote in your head. I won’t count (promise).

Somewhere on the way down there’s a line. Above it, you read the code. Below it, you take the vendor’s word for it. That’s the line in the title, and on every floor we’ll check which side of it we’re standing on (spoiler: it doesn’t improve).

One more map, because other people’s writing uses it. x86 counts rings from 3 (apps) down to 0 (kernel), and the hypervisor gets "ring -1", which is a figure of speech, not a piece of hardware. Libreboot calls the ME and the PSP (floor -3, patience) "ring -2" or "ring -3". Not a ring at all: another processor, reaching the centre. Rings aren’t floors, but they point the same way. Down. Audit your hypervisor all you like, you still haven’t audited the bottom.

Floor 0: everything’s fine

Ground floor: your VMs, and the hypervisor underneath. Xen, KVM: GPL, public code. You read it, you patch it, you send the patch upstream. audit("hypervisor") == 0. Green.

Everything’s fine… until you power the machine on.

Even at home. On stage I’d promised to put my own product in orange, so here goes: the hypervisor’s code is green, but booting it is another story. Secure Boot is the bouncer at the firmware’s door ("Sneakers? You’re not getting in."). Turn it on, and an XCP-ng host won’t boot, installer or installed. I hit it in my lab, on an OptiPlex. First I blamed the USB stick. Then the ISO. Then the IP KVM. Best part? The ISO really was corrupt as well. Two problems, one symptom. With a clean image and Secure Boot on:

All bootable devices failed Secure Boot verification.

Not a bug.

So what is it, then? Partly signatures: XCP-ng’s boot chain isn’t signed with a key your firmware trusts out of the box. The 8.1 release notes say Secure Boot relies on proprietary packages, the forum says it would need a valid Microsoft certificate, and the request has been open since 2019. Guests have had Secure Boot since 8.2.1 (same name, different thing, just to keep you on your toes). And enrolling your own keys? Then you sign the whole chain, every update included.

« Le Secure Boot ne vérifie pas que le code est ouvert. Il vérifie qui l’a signé. » Secure Boot doesn’t check that the code is open. It checks who signed it.

Who holds the keys. Sounds a bit like the bus, doesn’t it?

Floor -1: open upstream, closed at your place

Knock knock. On floor -1, the firmware that boots the machine answers. And your vendor’s UEFI? A closed binary. On 22 October 2026, TianoCore published GHSA-99cj-7hg9-mp4m, a critical pre-boot flaw in the PXE driver of EDK2, the reference UEFI implementation. The fix had been on master since 15 October. In your server, though, it lands when your vendor ships a new binary. Not before. (Dan Goodin at Ars Technica explains why that reaches so many servers: most UEFI suppliers build on EDK2.)

« Ouvert en amont, fermé chez vous. » Open upstream, closed at your place.

Why orange, then, and not red? Because doors exist. coreboot does the bare minimum of hardware bring-up, then hands over to a payload such as TianoCore, which is EDK2, compiled by you this time. Libreboot is to coreboot what Debian is to Linux: a distribution of it, the purist one, mostly on older machines (hello, ThinkPad X200 from 2008). On recent Intel, memory init goes through Intel’s FSP blob, and the Libreboot FAQ says replacing it would take the world’s best developer years. (Not me, then.) Dasharo, also coreboot-based, ships on laptops and on a handful of server boards with paid support.

Dasharo can even stop the Intel ME after boot, with the HAP bit, a flag that tells the ME to halt once the hardware is up. Stopped. Not removed. We’ll meet it again two floors down.

Dasharo’s setup menu offers three Intel ME modes. Screenshot: 3mdeb, from the Dasharo documentation, Apache-2.0.

Floor -2: the roommate in the closet

Who has ever updated their BMC’s firmware? Who knows which OS runs inside it? (Before this talk, not me. Not a clue.)

Remember the Romanian SIM? Your server has one too: the BMC. Power, console, ISOs, its own OS and its own network, even while the server is off. A roommate living in your closet, with the house keys. The keys come with the server, though not always all of them: on one machine I was lent, mounting an ISO needed a paid licence. That same machine, with 1.5 TB of RAM, spent more than four minutes in POST (the power-on self-test, the machine checking itself) before even looking for its devices. Four minutes of code you can’t read, and your OS isn’t even born yet. Long enough to wonder what it’s doing in there.

Orange anyway, thanks to OpenBMC, an open source Linux distribution for BMCs. Facebook started it in 2014, and Facebook’s own 2015 engineering post says Facebook runs it in production. My servers? None of them. So I booted the public image for Bletchley, a Meta board, in QEMU on my laptop. That one gets its own post.

Floor -3: knock knock, who’s there?

Guessing game. A reasonable one-socket server: how many processors inside? The invoice says one. The invoice is an optimist. There’s the CPU you paid for, the BMC, your SSD’s controller, and the one that answers when you knock on floor -3.

Yes, a fourth one. And it was there before you.

On Intel, it’s the Management Engine: a separate little computer in the chipset since 2006. It has handled part of hardware init since Nehalem, its firmware is signed by Intel, and on recent chips it runs a 486-derived core running MINIX (my favourite piece of trivia in the whole talk). Without valid signed firmware, the machine shuts down after 30 minutes, according to people who took it apart.

Half an hour of freedom.

And AMD? No favourites here. The PSP is an Arm core on the same die as your x86 cores, and per the coreboot and Libreboot docs its firmware must be AMD-signed, or your x86 cores never start.

Not thirty minutes: zero.

Your processor has two brains, and you only hold the keys to one.

This isn’t Intel bashing or AMD bashing (I just did both, to be fair), it’s architecture. And these locks exist to protect you: anti-rollback, for one, stops anyone dragging you back to an older, vulnerable firmware. They just don’t hand you the key. (AMD friends reading this: « cœur sur vous », much love. I meant it on stage too.)

So I looked at a real one: an AMD EPYC Genoa belonging to a partner (not the loaner from floor -2), booted from a live key. The kernel reported its PSP fused, debug locked and anti-rollback on, all of it protective, and Secure Boot had no keys enrolled at all.

No TPM fitted on that box, so fwupd’s HSI scale (built with PCs in mind) stopped it at HSI:0, on step one. Ouch. Count BIOS, BMC, network cards, disk controller and SSDs, and that’s at least ten closed firmwares. « Je ne jette la pierre à personne. » I’m not throwing stones at anyone. Yours probably looks much the same.

To be fair, I ran the same checks on my own Intel NUC: ME version 18, manufacturing mode locked, and Boot Guard fused to the board vendor’s key, so only that vendor’s firmware boots (on my NUC, yes, that one stung). It has a TPM (the small chip that keeps track of what booted), and fwupd gives it HSI:3, a better score earned partly because of those vendor locks.

« Plus sûr, oui. Plus ouvert, non. » Safer, yes. More open, no.

audit("ME/PSP") == -1. Red. Can’t read it, can’t remove it.

Two minutes, and it tells you a lot: fwupdmgr security. The HSI scale was designed for laptops and desktops, so read a server’s score with that in mind.

"But Bruno, there’s me_cleaner!" (Nobody said it. I felt it.) me_cleaner strips much of the ME firmware, and the HAP bit stops the ME once the hardware is up, but by then it has already done its job. Never run it on a machine you need. For the PSP, says Libreboot, there’s no equivalent.

In the cloud, this floor is still under your cluster (told you there was a paragraph for you). On AMD, a confidential VM’s SEV-SNP attestation report comes from firmware running in, you guessed it, the PSP. You trust the host less and the chip maker more.

And here’s the part I genuinely like: with openSIL, AMD publishes its silicon initialisation code as open source, and coreboot boots recent EPYC chips with it on a few boards. A real step, and AMD is the one taking it. The PSP itself still runs AMD-signed firmware, so openSIL opens the initialisation, not the PSP. But a door that wasn’t there is now ajar: not orange yet, but the first crack in the red.

Floor -4: the bottom of the cave

Knock knock… nothing. On x86 the instruction set is proprietary and core designs are secret. Nobody expected to read a Xeon’s RTL (the hardware description the chip is built from) this morning. (Good. I didn’t bring one.) audit("ISA") == -1. Red.

Not a law of physics, though: OpenPOWER is one open ISA, and on Raptor’s machines the platform firmware sources are published and auditable. There’s another open ISA, and I’m keeping it warm for later.

Express ride back up

Here’s the map you voted on: green at the top, orange in the middle, red at the bottom. So is open source pointless? (Deliberately provocative.)

No.

« L’open source, c’est une direction. Pas une case à cocher. » Open source is a direction. Not a checkbox.

I skipped a few floors on the way down (my talk, my rules). Your SSD has its own processor and closed firmware, and in 2019 Radboud University researchers showed at IEEE S&P that several self-encrypting SSDs gave up their data without the user’s password.

In October 2026, an intermediate certificate of the certificate authority eMudhra turned out to have a private key equal to 4. Yes, four. Mozilla’s bug 2083449 tracks eight weak intermediates; eMudhra attributes it to a script error. It was spotted because the public half was public (I checked it myself, I couldn’t resist). A 4 inside SSD firmware? Nobody would ever see it.

The line moves both ways. After 2015 FCC rules on 5 GHz radios, TP-Link locked out third-party firmware entirely. The 2016 consent decree brought a \$200,000 penalty (for its own firmware exceeding power limits, not for open source) and a commitment to work with the open source community and chip makers on reopening the door. On an Archer A7 sold in the US, OpenWrt installs from TP-Link’s own update page. Are the two connected? I can’t prove it.

Monday morning: make the red visible

I won’t ask you to flash coreboot onto production. Your CISO would find me, and rightly so.

« On ne rend pas le rouge vert, on le rend visible. » We don’t turn red into green. We make it visible.

Five verbs. Inventory: list every firmware you have, and "no TPM" is a line in that list too. Measure: Secure Boot, our bouncer from floor 0, checks who signed each step; measured boot is the CCTV camera, recording each step in the TPM. Attest: have something else compare those recordings with what you expect, because a camera nobody watches is decoration. Pin: friends don’t let friends use latest, firmware included. Isolate: the BMC gets its own network, and nobody else’s. Red stays red, but you go from "I trust it" to "I know when it changes". That’s not nothing. Want to watch one setting move two lines in the TPM log? That’s the floor -1 post. The rest of Monday morning gets its own.

The line moves

What if, somewhere, floor -3 opened when you knocked?

Same lift, different building: RISC-V. On floor -3, where x86 keeps its ME, RISC-V puts OpenSBI, the open source reference firmware running at the machine’s most privileged level: audit("OpenSBI") == 0. On floor -4, the ISA is an open specification. The core? It depends. SiFive sells the P550 as licensed IP; CVA6 publishes its RTL, though trying it on an FPGA needs a closed vendor tool. An open ISA isn’t an open core (a confusion I used to make myself, so no judgement here).

On my first Banana Pi BPI-F3, the bootloader is GPL source, at the exact commit my distribution says it built. OpenSBI is open source too, though built from a contributor’s tree. Lovely. But a second, 32-bit core runs esos.elf, a prebuilt binary my distribution copies in, even though SpacemiT publishes its source. My most open board, and still a binary nobody here rebuilt. It gets its own post.

The SpacemiT K3 announces everything a hypervisor needs: H for the CPU, AIA for interrupts, an IOMMU for devices. 16 cores!

Well… 8.

Only eight of the sixteen have the H extension. Julien Aubé ran the commands on his board for me, and on that one board AIA is there while Linux sees no IOMMU.

Still, RISC-V servers are starting to look like servers.

A rack server with its lid off, on a stand at the RVI Summit China 2026. Photo: Ludovic Henry, used with his permission.

No board? Scaleway rents bare-metal RISC-V, the Elastic Metal EM-RV1, launched in February 2024. According to Scaleway, it boots U-Boot, OpenSBI, then Linux, on top of "minimal proprietary code". RISE’s GitHub runners run on these machines by default, and most of my riscv64 porting goes through them.

The bus comes back

Told you. After Oslo, Denmark’s Movia had EY audit electric buses from several manufacturers, and found no basis to consider Chinese buses less cybersecure than the others. And the manufacturer that could least demonstrate its cybersecurity management? The Dutch one, VDL: the maker of Oslo’s bus without OTA.

EY gives both halves: without OTA, nobody takes over the fleet remotely, and nobody patches, monitors or audits it either. Not having OTA doesn’t make you auditable.

« Injoignable, ce n’est pas vérifiable. » Unreachable isn’t verifiable.

The risk was never the bus’s passport. It’s closed firmware that someone else writes and signs, at the heart of a critical system. And you have several in every server.

Takeaways

Know which floor you’re on. For each layer, ask whether you can read it, patch it, or only believe it.
Spend two minutes on fwupdmgr security. It opens nothing, but it tells you where you stand.
Treat firmware like container images. Inventory it, pin it, never :latest it.
Put the BMC in its own cave. Separate management network, restricted access, default accounts changed.
Watch the bottom of the stack. openSIL and RISC-V show the line can move down.

Thanks to the Cloud Nord crew for having me back, to Julien Aubé for running commands on his K3 for my slides, and to Ludovic Henry for the photo. The video (in French) and the slides are online. Next in the series: floor -2, where I boot a BMC on my laptop.

audit("this post") == 0;  // sources in the slides' appendix

A RISC-V hypervisor in slow motion: XCP-ng and Xen Orchestra, from inside QEMU

Bruno Verachten — Fri, 23 Oct 2026 20:16:57 +0000

Originally published on October 23, 2026 at https://bruno.verachten.fr/2026/10/23/a-risc-v-hypervisor-in-slow-motion-xcp-ng-and-xen-orchestra-from-inside-qemu/

Photo by John Thomas on Unsplash.

In November 2025, at KubeCon in Atlanta, we showed XCP-ng on Ampere: an Ampere Altra-based System76 Thelio Astra running XCP-ng, managed by Xen Orchestra. A real Arm machine, doing real hypervisor things, in front of real people.

So in October we set ourselves the same goal on RISC-V. (One new architecture a year is clearly not enough for some of us, and by "some of us" I mostly mean me.) The target was a host that Xen Orchestra can manage, and a guest on it running containers: Docker first, then Kubernetes with K3s.

Did we get there?

In QEMU, yes. Xen Orchestra lists the RISC-V host and its VMs, opens their consoles, shows their stats, starts and stops them. A Debian guest on that host runs Docker, and K3s reports Ready about a minute after starting. And as a bonus, the same guest shows an XFCE desktop in Xen Orchestra’s console tab.

Now, before anyone gets excited and starts shopping for boards: unlike the Arm demo, all of this runs in QEMU, emulated on an x86 machine. Since early October, that x86 machine has been pretending to be a RISC-V machine, with a lot of dedication and very little speed. We have not run this full stack on a RISC-V board yet. This is a prototype, research and development in the wild, and what follows is a progress report rather than an announcement (I’m repeating that to myself as much as to you, because my enthusiasm has a habit of slipping its leash).

The people who did the hard part first

None of what follows exists without these people, so they come first.

Baptiste Le Duc and Oleksii Kurochko have been porting the Xen hypervisor itself to RISC-V for a while now, and the xcp-ng.org blog has followed them: Xen booted a guest on real RISC-V hardware in September. Then Zheng Zhang (ISCAS, the Institute of Software, Chinese Academy of Sciences) took those patches and booted Xen on his own SpacemiT K3, the first RVA23 board Xen runs on.

The original version of this story on xcp-ng.org was written together with Baptiste, Yann Sionneau and Julian Vetter, so their names belong up here too.

So where does my October fit in?

A hypervisor is not a virtualization platform. XCP-ng is Xen plus a dom0 with its whole package set, plus XAPI, the toolstack that Xen Orchestra talks to. Baptiste’s trees already carry the RISC-V Xen, the RISC-V toolstack (XAPI, xenopsd, xenguest), the dom0 kernel, and the dom0 image build. Since early October, we have been working on the layer around them: package that stack for riscv64, boot it, fix what breaks along the way, and get Xen Orchestra to manage the result. The unglamorous plumbing, in other words, which happens to be my favourite kind.

A word on method, because I’d rather say it now than have someone find out later: I built much of this with heavy help from AI coding assistants, so expect parts of it to change as the patches go through upstream review.

What works today

The stack runs in QEMU with 3 harts and 5 GiB of RAM. The dom0 is AlmaLinux Kitten running the riscv64 RPMs of our latest build round, round 19, Xen and the dom0 kernel included. Xen Orchestra 6.9.1 is a riscv64 build of its own (gounthar/xen-orchestra-riscv64). On October 20th, on that round, it gave:

the host and its VMs in the inventory
host and VM consoles
host and VM stats (the graphs are in the XO 5 interface, which XO 6 links to)
start, clean shutdown, force shutdown, and start again
a Debian 13 (trixie) guest with a disk on a storage repository and a network that reaches the internet
an XFCE desktop in the VM’s console tab

That fourth line looks boring. It wasn’t, and I’ll come back to it.

Figure 1. A Debian 13 RISC-V guest booting to a login prompt, in the Xen Orchestra 6 console.

Figure 2. CPU, memory, network, and disk statistics for the same guest, in Xen Orchestra 5.

Inside that guest, on a kernel built with the container options, Docker runs riscv64 containers and K3s reports Ready.

Is it polished? Not quite. On one start, Xen Orchestra showed no IP address for the guest for more than five minutes, although the guest had its lease after 24 seconds. Xen Orchestra reports the guest’s disk as 1 GiB when the guest sees 8 GiB (a rounding error of a mere factor of eight), and Docker pulls sometimes need a retry.

The packages are public, as experimental and unsigned releases in gounthar/xcpng-riscv64-rpms: 19 build rounds so far, published from round 10. They come from Yann Dirson's meta-xcpng BitBake layer, with a riscv64 port of ours on top that is not part of his layer. (Yann Dirson, not to be confused with Yann Sionneau above. Yes, there are two Yanns in this story. No, it doesn’t get easier.)

In a riscv64 AlmaLinux Kitten container, with the Kitten repositories and the latest round, this resolves and installs:

dnf install xen-hypervisor xen-tools kernel-xcpng-riscv64 xcp-networkd xcp-ng-release-config

Let me be careful about what that means, because it’s easy to read too much into a green dnf run. It proves the packaging, not that the stack runs. And like the Xen Orchestra build, these packages are a personal, unsupported, and not production-ready project. Please don’t put them anywhere near anything you care about.

Starting a VM twice in one boot

Remember that boring-looking line about "start again"?

For the first few days, we could start exactly one VM per boot. Shut it down, and the next start never ran. One VM per boot is not so much a virtualization platform as a very elaborate bootloader.

The dead guest was never freed, and it took a fix on each side. In Xen, the RISC-V functions that release a dead domain’s resources were still stubs, and foreign mappings never dropped the page reference they take. In the dom0 kernel, the function that should unmap a guest’s pages returned success without telling Xen anything, so about 28 MB of every dead guest stayed pinned. (A function that reports success without doing the work: we’ve all worked with one, and some of us have written one.)

With both fixes, we ran three start, shutdown, start cycles in a row, and free memory came back to its baseline each time. Both fixes are merge requests on Baptiste’s trees, waiting for his review.

Getting the guest on the network

No network, no container images, so this one stood squarely between me and Docker. The first networked guest gave up right away:

grant table add_to_physmap failed, err=-38

-38 is ENOSYS (the kernel’s polite way of saying "never heard of it"). Xen on RISC-V did not yet implement the call a guest uses to grow its grant table, and a network frontend needs 512 grants to start. The Arm port has that case; RISC-V did not.

We expected one patch and needed three: the missing case in Xen, a scratch page for grant seeding in libxenguest, and an init function in the RISC-V Linux port that returned an error and stopped grant table setup halfway. (Expect one, get three. If you’ve done any porting work, you’re nodding right now.) After those, the guest was able to ping the dom0, and later reached the internet. These are also merge requests on Baptiste’s trees, waiting for review.

A desktop in the console tab

Both consoles in Xen Orchestra (XO) were text-only at first. Perfectly respectable, and perfectly black and white.

Julian Vetter had already solved this for the Arm port, so we borrowed his approach: QEMU runs in dom0 only as a Xen PV backend, the guest uses Xen’s PV framebuffer, and Xen Orchestra connects to it like any other console.

On RISC-V that meant building a riscv64 QEMU from his XCP-ng QEMU branch plus two RISC-V patches, teaching xenopsd’s RISC-V builder to start it, and enabling the keyboard and mouse frontend in the guest kernel. Then one setting on the VM:

platform:vfb=true

And XFCE shows up in the console tab.

Figure 3. An XFCE desktop on a RISC-V guest, in the Xen Orchestra 6 console tab. uname -a reports riscv64.

I know a desktop is not what anyone runs a hypervisor for. It does make a better demo than a login prompt.

And once there’s a desktop, you know what has to run on it. On October 6th, Chocolate Doom 3.1.0 ran in that same console tab, with the Freedoom Phase 1 game data (not id’s WAD), both straight from Debian trixie’s riscv64 repositories. Installing the two packages took 6 minutes 39 seconds under emulation.

The console’s first trick was to sit on its loading logo forever. Xen Orchestra reaches a VM’s console through XAPI’s websocket proxy, and that proxy only accepted sockets named vnc-N, while the PV framebuffer’s QEMU names its socket pv-vnc-N. Once XAPI accepted that name too (build round 13), the logo gave way to the desktop. Then the mouse pointer kept drifting to the right, because QEMU doesn’t tell the guest’s virtual keyboard driver how big the screen is. Telling the guest kernel directly fixed it:

xen_kbdfront.ptr_size=1280,800

Figure 4. Freedoom on a RISC-V guest, in the Xen Orchestra 6 console tab, on October 6th, on an earlier build round.

The same run on video, recorded on 2026-10-06.

Is it playable? I honestly don’t know: nobody measured the frame rate, and under emulation I wouldn’t bet on it. But it runs, and some traditions have to be respected.

One fix that came from upstream

For a while, Xen Orchestra would lose the RISC-V pool every time XAPI was restarted, and only a restart of xo-server would bring it back.

That one, we didn’t have to fix ourselves: Xen Orchestra 6.9.0 fixed it. Since we updated, xo-server reconnects on its own, about 20 seconds after a XAPI restart. (I’d love to take credit for that one. I can’t. All I did was update.)

What emulation costs

QEMU emulating RISC-V on x86 is slow, and it shows.

That minute for K3s is emulation time. A guest reads its disk at about 800 KB/s. Yes, kilobytes, per second, and I’ll let that sink in slowly, at roughly the speed of the disk. A clean shutdown takes about 20 seconds. The packages are built under emulation too: the OCaml build alone takes 72 minutes (plenty of time to reconsider a few life choices).

None of those figures say anything about how RISC-V hardware performs. They only explain why we want to switch to using a real hardware board.

What’s next

Three directions, none of them dated (on purpose: a date in a blog post reads like a promise, and this is a prototype):

Real RISC-V hardware. Our boot path still assumes QEMU’s virtual machine layout. Booting the full stack on a board, as our Arm demo did on Ampere, is the next milestone.
Upstreaming. Our patches go to Baptiste’s Xen, Linux, and toolstack trees first, for review, then upstream from there. Some are merged in his trees; most are still waiting.
Reproducible package builds. Native builds on a RISC-V board instead of emulation, and a riscv64 port that fits meta-xcpng cleanly.

How to help

Don’t have a RISC-V board? This stack hasn’t met one yet either, so you’re in good company.

You don’t need one to contribute. Start with the packages: in a riscv64 AlmaLinux Kitten container, download the latest round and install it as a local repository, as the repository’s README shows, and run the install command above. That only proves they install, but a dependency that does not resolve or a file in the wrong place is a bug, and telling us about it on the forum is already a contribution.

If you know RISC-V, virtualization, or packaging, you are welcome to review Baptiste’s Xen tree, Linux tree, and xen-api fork. (Yes, this is me asking for reviewers again. I’m told persistence is a feature.)

What I’m taking away from October

An emulator number is an emulator number: 800 KB/s disk reads say something about QEMU on x86 and nothing about RISC-V silicon. When RISC-V is missing something, look at the Arm port first, because the grant table case was already there, and so was Julian’s console approach. Budget for three patches when you expect one. Update before you debug, since the XAPI reconnect problem had been fixed in Xen Orchestra 6.9.0 all along, waiting for me to notice. And a package that installs is not a stack that runs.

This is my personal retelling of a post published on the XCP-ng blog, written with a bit more mischief than the house voice allows. The original, co-authored with Baptiste Le Duc, Yann Sionneau and Julian Vetter, lives on xcp-ng.org: XCP-ng and Xen Orchestra on RISC-V: a progress report from the emulator.

Xen Summit 2026 in Munich: RISC-V takes the graveyard slot and wakes the room

Bruno Verachten — Mon, 28 Sep 2026 18:39:42 +0000

Originally published on September 28, 2026 at https://bruno.verachten.fr/2026/09/28/xen-summit-2026-in-munich-risc-v-takes-the-graveyard-slot-and-wakes-the-room/

Every September, part of the Xen community does something slightly radical: it stops talking through mailing lists and Matrix handles and spends three days in the same room. (Faces! Attached to the handles! It never stops being a little strange.) This year the room was in Munich, hosted by Renesas, from 15 to 17 September. Fourteen of us from Vates made the trip, more colleagues followed remotely, and we gave three talks.

Here’s what we said, what I heard, and why I came back more excited about RISC-V than I left. Which, if you know me, is quite a feat. The needle was already pretty far into the red.

Why leave the comfort of the mailing list?

Most of our work with the wider Xen community happens in writing, and that works. Honestly, it does. But some topics would take weeks to settle in a chat thread, and some would never come up at all, because nobody would think to raise them in the middle of a normal working day, somewhere between two rebases. Put the same people in one place for three days, and those topics surface on their own, over coffee, in a corridor, or at dinner. No git send-email required.

Vates was one of the sponsors, but the Summit belongs to the Xen community, not to us. (Sponsoring an event doesn’t buy you the room, and that’s precisely what makes the room worth being in.) Silicon vendors, consultancies, universities, and embedded companies were all there, which made the conversations much wider than the ones we have among ourselves. Our own echo chamber has lovely acoustics, but it’s still an echo chamber.

Three talks walk into a summit

XCP-ng State of the Union (me, Tuesday 10:00). A round of everything that happened since the Spring Meetup in April (five months, roughly three geological eras in our line of work): the market around us, XCP-ng 8.3 LTS and QCOW2, Xen Orchestra 6, the DevOps tooling, then Arm and RISC-V.

One figure I was very happy to put on screen: Vates authored 104 commits to Xen 4.22, 10.9% of the release, making us its third-largest contributor, and more than half of those commits were RISC-V work. You can probably guess which part of that sentence made me happiest.

The RISC-V slide ended with a direct ask to the room: come and review the series, because more reviewers move it faster. Yes, I stood in front of a room full of reviewers and asked for reviewers. Subtle, I am not. Several people came to find me afterwards (more on that below, and it’s the good kind of afterwards).

XCP-ng+Xen status update (Teddy Astie, Tuesday 13:30). Teddy walked through the Xen work behind XCP-ng, and his list is long enough that I’ll give it to you as a list rather than as one heroic sentence (the first draft of this post tried the heroic sentence; the sentence won):

general x86 improvements, including work in progress on guests with more than 128 vCPUs;
Arm for cloud, where XAPI support is partly working and some of it is already upstream;
confidential guests with AMD SEV, which work experimentally, with patches on the mailing list where help is welcome;
PV-IOMMU;
and GPU virtualisation, through both virtio-gpu and SR-IOV vGPUs on AMD and Intel cards.

That’s a lot of plates spinning. His slides are on the Summit schedule.

Xen on RISC-V: From Dom0less to xl (Baptiste Le Duc and Oleksii Kurochko, Wednesday 17:00). This is the talk I keep telling people about (whether they asked or not), so it gets its own section.

The graveyard slot, resurrected

Anyone who has spoken at a conference knows the last slot of day two. The shared dinner is next, the room is tired, and half the audience is already thinking about beer. In Munich. You can hardly blame them.

Baptiste and Oleksii got the room back anyway.

Baptiste has a sense of showmanship, and the enthusiasm, the stubbornness, and the plain joy the two of them put into this work came through in every slide. The audience applauded in the middle of the talk, more than once. Mid-talk applause, at 17:00 on day two, from hypervisor people. Let that sink in for a second.

And that audience matters. These were committers and maintainers, people who already follow the RISC-V port on the mailing list and know every patch series by name (and, I suspect, by version number). For them, the talk turned Xen on RISC-V from something ethereal into something concrete. From vaporware to hardware, if you’ll forgive me.

The title says the plan, and the plan has two lanes. Upstream, the port goes dom0less first: Xen builds its guests itself at boot, from a static configuration. Downstream, Baptiste and Oleksii already run guests with several vCPUs, on QEMU and on real hardware: a HiFive Premier P550, and a SpacemiT K3 that a community contributor booted independently.

Then comes a full dom0 with the xl toolstack, and that part works too: xl info, xl list, xl create and xl console are all there. Four commands. Doesn’t sound like much? If you’ve typed them on x86 without a second thought, try picturing the work behind each one on a young architecture.

They ended with a demo, pre-recorded in case the conference network let them down, creating a guest from dom0 and talking to it through its console. Is pre-recording cheating? No. That’s experience. Anyone who has ever bet a live demo on conference Wi-Fi knows exactly why, and the knowledge is rarely acquired cheaply.

Baptiste wrote up the P550 story in Xen on RISC-V just ran on real hardware!, including why a core designed before the hypervisor extension was ratified made it harder than it sounds (and "harder than it sounds" is doing a lot of heavy lifting in that sentence). Their slides are on the Summit schedule.

RVA23: a profile, and not the LinkedIn kind

For years, RISC-V meant small boards for hobbyists and embedded work. That is changing, and the reason isn’t a chip. It’s a profile.

RVA23 is a ratified set of mandatory extensions, and the hypervisor extension is one of them. An RVA23 chip is not automatically fast enough to be a server, but it has what virtualisation needs, in a form software can rely on. Once that baseline exists, hardware built for servers can follow, and a server that can’t run a hypervisor is not much use in a datacenter (a very expensive space heater, at best).

So the real question for Xen is whether a type-1 hypervisor will be ready when that hardware arrives. Showing up to the party after the cake is gone is never a great look. That is what Baptiste and Oleksii are working on: making sure we’re there before the candles are lit.

The hallway track: no slides, all signal

The corridor conversations were at least as exciting as the talks. No slides, no timer, nobody waving a "five minutes left" sign at you.

Several people came over after my talk because, as one of them put it, they had noticed my "unbridled enthusiasm" for RISC-V. Fair enough. Guilty as charged. I’m not sure anyone has ever managed to put a bridle on it, least of all me. From those conversations:

a university lab that runs real RISC-V hardware in the loop for its CI, which is exactly what a port like ours needs;
an embedded engineer who would love to see Xen on RISC-V in safety-critical systems one day;
and, my favourite detail, people swapping links to order RISC-V boards on the spot, because everyone is waiting for hardware.

Nothing says "this is getting real" quite like hypervisor people doing their shopping in the corridor.

The atmosphere was friendly and generous, with the kind of good-natured competition where everyone leaves wanting to push a bit harder. Friendly rivalry: the renewable energy of open source.

Why I care (and why the bridle stays off)

I’m not a visionary. But I do have a decent memory, and I’ve seen this pattern before: about fifteen years ago, with Arm, a new architecture went from "interesting" to "real" in a few years, and the people who had their software ready early were the ones who mattered. I think RISC-V can go further than Arm did, because the ISA itself is open. Open ISA, open hypervisor: I do like the symmetry.

Is getting ready early a gamble? Sure.

Call it a calculated RISC.

There are enough signals now to say: let’s go. Or, since we were in Munich, los geht’s.

What’s next

The talk recordings aren’t out yet; they’ll be linked here once the Xen Project publishes them. (Refresh responsibly.)
Baptiste and Oleksii’s work deserves more than a section in a recap, and a dedicated post is on the way.
If you want to follow the RISC-V port, or answer that call for reviewers (yes, I’m asking again; persistence is a feature, not a bug), the patch series are on the xen-devel mailing list.

Thanks to the Xen Project team and to Renesas for organising the Summit, and to everyone who came to talk to us, enthusiasm bridled or not.

yum said 301, the machine said 2024

Bruno Verachten — Wed, 29 Jul 2026 16:18:12 +0000

Originally published on July 29, 2026 at https://bruno.verachten.fr/2026/07/29/yum-said-301-the-machine-said-2024/

I sat down to put the July batch of updates on the XCP-ng 8.3 host in my home lab, which is normally a thirty second job with a coffee in the other hand, and got this instead:

http://mirrors.xcp-ng.org/8/8.3/base/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 301 - Moved Permanently
Trying other mirror.
...
failure: repodata/repomd.xml from xcp-ng-base: [Errno 256] No more mirrors to try.

Read that as an operator and there is exactly one conclusion on offer. The mirrors moved, the repo files are stale, someone upstream forgot to update something. It is the only story that message tells, and it tells it with real confidence.

So I did what the message told me to do. I opened /etc/yum.repos.d/xcp-ng.repo and stared at the URLs. I ran yum clean all. I ran rm -rf /var/cache/yum for good measure. I was one text editor away from pinning a mirror by hand and bypassing whatever was broken instead of understanding it.

None of it helped, and it cost me the better part of an afternoon, because every single word of that error message is about the wrong thing.

The host did not know what day it was.

The redirect is real, and it is completely fine

The first thing worth doing is proving the mirrors innocent, since that is where the message points. Following the chain by hand with curl, from my laptop rather than from the host that was failing, which matters more than it sounds like it does and which I will come back to:

curl -sIL http://mirrors.xcp-ng.org/8/8.3/base/x86_64/repodata/repomd.xml

http://mirrors.xcp-ng.org/...          301 -> https://mirrors.xcp-ng.org/...
https://mirrors.xcp-ng.org/...         302 -> https://xcpng-mirror.as208069.net/...
https://xcpng-mirror.as208069.net/...  200, 3117 bytes, application/xml

Two hops and then the file. The first hop is the plain 301 that pushes you from HTTP to HTTPS, which every sane public mirror does now. The second is MirrorBits picking a mirror near you, and you can watch it think out loud in the headers it emits on the way past:

link: <https://xcpng.reloumirrors.net/...>; rel=duplicate; pri=1; geo=fr
link: <https://rg2-xcpng-mirror.reptigo.fr/...>; rel=duplicate; pri=2; geo=fr
link: <https://xcpng-mirror.as208069.net/...>; rel=duplicate; pri=3; geo=fr

That is a mirror network doing precisely what it is designed to do, ranking candidates by priority and by my geography, and handing me one. curl -L walks the whole thing and comes out the other side with a 200 and a valid XML file. Nothing moved. Nothing is down.

Which means the 301 in that error message is not a fault report at all. It is a description of normal, healthy infrastructure, printed at the exact moment something completely different fell over.

Two boxes, one command, one variable

There is a much cheaper version of that check, and if you have a second machine you should run it before you touch anything else. I have another host on the same LAN, same switch, same egress, same repo files. I ran the same update there.

Upgrade  28 Packages
Total download size: 103 M

xcp-ng-release 8.3.0-38, xen-hypervisor 4.17.6-9.3.1.xcpng8.3, the whole 26.1.11-1.3 XAPI stack, straight down from xcp-ng-updates without a murmur.

One command, and in one go I had eliminated a mirror outage, a bad repo file shipped upstream, a broken batch, DNS, my router, and my ISP. All of it, gone from the list. Whatever was wrong lived on the one host, and I had wasted my afternoon looking outward when the control test was sitting there in the next room.

I keep relearning that lesson and I keep being surprised by it, so I am writing it down this time.

The certificate had a start date

Once you know the fault is local, the next question is what the failing host does differently from the working one, and the answer turned out to be about TLS.

The mirrors use Let’s Encrypt certificates on a normal ninety day cycle. mirrors.xcp-ng.org and updates.xcp-ng.org were both valid from 2026-05-31 to 2026-08-29, and the mirror I was actually landing on, xcpng-mirror.as208069.net, from 2026-07-01 to 2026-09-29. All of them comfortably current.

Here is the bit that I think trips everybody, including me. An X.509 certificate does not just have an expiry date. It has a notBefore as well as a notAfter, and a host that believes it is living before that notBefore will refuse the handshake with certificate is not yet valid. That is a sentence most of us have simply never had to read, because in fifteen years of operations the failure has always run the other way round. Certificates expire. They do not un-expire.

Then I ran the least imaginative command available on the failing host:

date

Sun Sep  8 10:49:04 CEST 2024

September 2024. Twenty two months before the certificate it was being asked to trust had even been issued. The other host, checked in the same minute, was on the correct date with chrony synced to within 46 microseconds of its upstream.

One variable, and it was the one nobody thinks to look at.

This is also why that clean curl chain earlier came off my laptop. Run the same curl -sIL on the host with the 2024 clock and you do not get 301, 302, 200 and a tidy XML file. curl follows the 301 to HTTPS exactly as before, hits the same handshake yum was hitting, and stops there with SSL certificate problem: certificate is not yet valid. Which is, in fairness, the truth, and rather better than what yum told me. If you are following along on the broken box and wondering why your output looks nothing like mine, that is why, and you can take it as a free confirmation of the diagnosis.

And a machine you have recently installed is exactly the machine most likely to be confidently wrong about the year. A dead RTC battery, a board that came back from a power cut on its factory default date, an isolated segment where NTP never resolved. Nothing stops you, and everything else works.

So why does yum lie about it?

Because of two pieces of code in python-urlgrabber that were never really meant to meet each other.

yum on dom0 fetches through urlgrabber sitting on top of pycurl. The header callback updates the object’s idea of what it is downloading the instant it sees a Location: header, which is before anything at all has been attempted at the new address:

if buf.lower().find(b'location') != -1:
    location = b':'.join(buf.split(b':')[1:])
    location = location.strip()
    self.scheme = urlparse.urlsplit(location)[0]
    self.url = location

So after that first 301, self.scheme is https. Perfectly reasonable so far.

Then the transfer fails, and the exception handler goes to compose a message for me:

code = self.http_code          # getinfo(RESPONSE_CODE), still 301
errstr = str(e.args[1]) or pyerr2str.get(errcode, '<Unknown>')
if code and not 200 <= code <= 299:
    scheme = _bytes_repr(self.scheme)
    msg = '%s Error %d - %s' % (scheme.upper(), code,
                                scheme in ('http', 'https')
                                and responses.get(code) or errstr)
else:
    msg = 'curl#%s - "%s"' % (errcode, errstr)

A TLS handshake that fails never gets as far as producing an HTTP response code, so RESPONSE_CODE is still sitting at 301 from the hop that succeeded. scheme is now https because the header callback moved it. The condition holds, the first branch fires, and errstr is thrown away without ever being printed.

errstr was SSL certificate problem: certificate is not yet valid.

What comes out instead is HTTPS Error 301 - Moved Permanently, which is the scheme of the hop that failed, the status code of the hop that worked, and the human readable text of neither of them.

That also explains a piece of folklore you will find on the forum, which is that changing the repo URL to https:// makes yum tell you the truth. It does, and it is worth understanding why, because it is not a fix. Point the repo straight at https:// and there is no redirect at all, so RESPONSE_CODE is 0, the else branch runs, and libcurl’s own honest message survives intact. Same failure, same host, same second. Different sentence.

The general form of this is worth carrying around in your head, because it is not specific to XCP-ng or to these mirrors at all: in yum, any transport failure that happens after a redirect is reported as the redirect. TLS, DNS, connection refused, all of it gets the same treatment. If a yum error names a 3xx, the 3xx is the last thing that went right.

The best part: the time daemon was working perfectly

I could have stopped at "set NTP", fixed the clock, taken the updates and gone to bed. I am glad I did not, because the interesting part was still underneath, and on this host NTP was already on.

systemctl is-active chronyd
systemctl is-enabled chronyd

active
enabled

Service running. Service enabled, so it survives reboots. Network fine, DNS fine, egress fine. Every check an operator would actually think to run comes back green, and on that evidence you would confidently conclude that time is not your problem and go and look somewhere else. Which is what I had spent the afternoon doing.

Then:

chronyc sources

210 Number of sources = 0

Zero. Here is /etc/chrony.conf in its entirety, with the comments stripped out:

driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony

There is no server line. There is no pool line. chronyd had been up for a day and a half, running flawlessly, doing exactly and only what it had been told to do, which was nothing whatsoever.

That is the failure mode worth naming out loud, because it is invisible from every angle except one. A time daemon with no sources is indistinguishable from a healthy time daemon unless you run chronyc sources, and nobody runs chronyc sources until they already suspect the clock. So the host serenely holds whatever date it was installed with, for as long as you care to leave it, and the first thing that ever tells you about it is a yum error about a redirect.

My best guess at the origin is the installer’s time step, where you can either hand it NTP servers or just set the date yourself. Choosing the second produces precisely this file. It is a reasonable thing for an installer to offer and a reasonable thing to pick when you are in a hurry, and the consequence turns up months later wearing a completely unrelated costume.

So the check is not "is NTP running". Running is not the question. The question is chronyc sources, and if that says zero, you have found it.

The fix, which takes a minute

The whole diagnostic is one command, and I want that on the record because of how long it took me to type it:

date

If the date is wrong, the next command decides which of two quite different problems you have:

chronyc sources

If that lists sources, the daemon knows where to get the time and has simply not applied it yet, and you can step the clock straight away:

chronyc makestep
date

makestep is the step people skip, and it matters. chrony corrects offsets by slewing the clock gently rather than yanking it, which is the right behaviour when you are drifting by milliseconds and completely useless when you are out by two years. You can sit and watch a correctly configured host stay broken indefinitely without it.

If chronyc sources came back with Number of sources = 0, which is the case this whole post is about, then makestep on its own will do exactly nothing and tell you it succeeded. There is no measured offset for it to step to, because there is nothing measuring. You have to give the daemon somewhere to look first. Put the servers into /etc/chrony.conf:

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

On an isolated network, point those at whatever local time source you have. Then restart the daemon and check that the sources actually came up before you trust anything downstream of them:

systemctl restart chronyd
chronyc sources
chronyc makestep
date

The stock config carries makestep 1.0 3, so once there are real sources chronyd will usually step the clock itself within its first few updates and the explicit makestep is belt to that braces. Running it costs nothing and saves you staring at date wondering how long to wait.

Editing the file rather than just stepping the clock is the part worth not skipping. Fix the time without fixing the config and you have fixed it precisely until the next reboot, at which point the host quietly goes back to being wrong and you get to have this afternoon all over again.

Or set NTP from xsconsole, which does the same thing with fewer chances to fat finger it.

Then:

yum clean all
yum check-update
yum update

On my host check-update came straight back with the full 28 package July batch, and yum update took the lot and exited 0, with the repository file exactly as it shipped. Total elapsed time once I actually knew the cause: about a minute.

Small thing worth knowing if you are scripting any of this, and pleasingly on theme for a post about misleading exit status: yum check-update returns 100 when there are updates waiting, not 0. Zero means there is nothing to do. Treat 100 as an error and you will have built yourself a second thing that reports the wrong state confidently.

Which brings me to the thing I nearly did, and you probably will too. If you pinned a mirror by hand while you were flailing around, undo it and put http://mirrors.xcp-ng.org back. Hand pinning opts you out of the geo selection you saw in those duplicate link headers earlier, and much more importantly it opts you out of failover on the day that one mirror has a bad afternoon. It is exactly the sort of "temporary" edit that is still sitting there two years later, on the afternoon it finally costs you something.

One worry you can put straight back down, because it stopped me for a few minutes before I did the fix. Dragging a host’s clock forward by two years does not break the host’s own certificate. XAPI issues it with a ten year validity, so a machine installed while it believed it was 2024 is holding one valid until 2034, and correcting the date lands comfortably inside that window. You do not need xe host-refresh-server-certificate for this. There is a real pool join certificate failure that looks alarming in a vaguely similar way, but it is a different animal and this is not it.

Somebody else got there first, in an hour

The best evidence for all of this is not mine, and I want to be clear about that. There is a forum thread from last November where the person who hit this worked the whole thing out themselves in under an hour, from the same host inside the same session, and left behind the three observations that make it airtight: the 301 with http:// in the repo file, the honest certificate is not yet valid with https://, and then setting NTP and reverting to the original URL and watching it just work. I spent longer than they did and arrived at the same place. There is also topic 9560, from 2024, which is the same symptom on 8.2 and simply ends, with nobody ever having named the cause.

Somebody suggested on that November thread that this deserved a page in the troubleshooting docs. They were right, and it did not happen at the time, which is more or less what the second thread looks like from the outside. There is a documentation change in review now that adds it under the common problems, next to the existing clock entry, plus a line in the update prerequisites saying the plain thing out loud: getting to the repositories needs a correct clock, and the error you get when you do not have one does not mention clocks. It is a small change and I have no idea yet whether it lands in the shape it is currently in, but it should at least mean that three minutes with a search engine gets you the answer rather than two forum threads and a shrug.

The one line takeaway, because it is the line I wish I had read first: if yum blames a redirect, run date before you touch the repo file, and if the clock is wrong run chronyc sources, because a time daemon can be running, enabled, and pointed at absolutely nothing.

The fastest Windows on XCP-ng is the one you don't install

Bruno Verachten — Tue, 07 Jul 2026 20:19:28 +0000

Originally published on June 29, 2026 at https://bruno.verachten.fr/2026/06/29/the-fastest-windows-on-xcp-ng-is-the-one-you-dont-install/

Photo by Jens Lelie on Unsplash.

I needed a Windows box on XCP-ng for a side project, and I did the obvious thing first. I tried to install it.

That was a mistake, and it took me the better part of two days to admit it.

If you have ever fed a Windows ISO to an XCP-ng VM, you know the wall. Windows Setup boots, you click through the language screen, and then it stops dead on "A media driver your computer needs is missing." XCP-ng hands the VM its disk over the Xen paravirtual block bus, which shows up inside the guest as xvda, and the WinPE environment that drives Setup has never heard of it. It is the same situation as installing Windows on Proxmox or plain KVM without the VirtIO drivers loaded. The disk is right there, the installer just cannot see it, so the "where do you want to install Windows" screen is empty and "Next" stays greyed out.

On Proxmox you click "Load driver", point at the VirtIO floppy, and move on. On XCP-ng the equivalent storage driver (xenvbd, plus its bus parent xenbus) only ships inside the guest tools MSI, so getting it in front of WinPE is more work. Here is the short version of what I went through, because it is the reason the rest of this post exists.

I extracted the PV drivers out of XenTools-x64.msi with msitools on a Linux box, dropped them into a $WinPEDriver$ folder so WinPE would auto-load them at boot, and built a remastered install ISO. Then I learned that Windows ISOs store their payload in UDF, so the usual xorriso "add a file" trick copies almost nothing forward. So I did a full extract with 7z, injected the drivers, and rebuilt a dual BIOS and UEFI bootable ISO. Then I learned that WinPE’s ISO9660 driver cannot read a file larger than 4 GB, and the install.wim is 5.8 GB, which is exactly why Microsoft uses UDF in the first place. So I split the WIM with wimsplit into chunks under 4 GB. And because the VM is headless, I had to catch the "Press any key to boot from CD" prompt by spamming keystrokes at the firmware over QMP, since there is no keyboard at the firmware screen to press.

I got it working. The disk finally appeared, Setup ran, and then the fresh install bricked into Automatic Repair on first boot. At that point I stopped and asked myself the question I should have asked at the start.

Figure 1. How the from-scratch install ended: a fresh Windows 11 stuck in Automatic Repair on first boot.

Why am I installing Windows at all?

Microsoft hands you a finished disk

The Microsoft Evaluation Center publishes Windows Server as a ready-made VHD, sitting right there next to the ISO. It is a sysprepped, fully installed copy of Windows on a virtual disk. No Setup, no driver injection, no keystroke roulette at a firmware prompt. You attach the disk to a VM, power it on, answer a couple of out-of-box questions, set a password, and you are looking at a desktop.

Two caveats before you reach for it, because they matter and I would rather you hear them now than after you have built something on top.

It is Windows Server, not a client edition like Windows 11. For a lab, a proof of concept, or anything where you just need a Windows userland to run something, that is fine, and Server even comes with the Desktop Experience so it looks and feels like a normal Windows. If you specifically need client Windows, this is not your image.

It is an evaluation image. The licence runs 180 days, and you are expected to activate it online within the first 10 or it starts shutting itself down on a timer. For a throwaway lab box, that is a feature: it cleans up after itself. For anything you mean to keep, plan to activate it or treat it as disposable.

Mine was the Datacenter evaluation, build 26100. Here is the whole recipe.

Step 1: grab the VHD, not the ISO

From the Evaluation Center, pick Windows Server 2025 and look for the VHD download rather than the ISO. Fill in the short form, and you get a roughly 11 GB .vhdx file. That file is the entire point: it is Windows already installed, captured as a disk image, waiting to be dropped onto a hypervisor.

Step 2: convert it, but not on dom0

xe vdi-import understands raw and VHD, but not the newer VHDX container, so the file needs converting first. qemu-img does that in one line. My instinct was to run it on the XCP-ng host itself, since that is where the disk is going anyway.

Do not do that. The way it fails is nasty, because it fails quietly. The qemu-img that ships in dom0 is a stripped-down build that only knows raw and qcow2. Point it at a VHDX and, in my case, it did not error out and tell me it could not read the format. It mis-detected the file as raw, reported the virtual size as equal to the file size (which is nonsense for a dynamic VHDX), and any conversion on top of that produces garbage. You will not find out until much later, when the VM refuses to boot, and you will not know why.

So convert on a normal machine that has a full qemu-img. Any modern Linux distribution has one; my WSL2 box was fine. A current qemu-img reads the VHDX correctly and reports the real geometry: a 64 GiB virtual disk with about 11 GB of actual data inside it.

qemu-img convert -f vhdx -O vpc -o subformat=dynamic ws2025.vhdx ws2025.vhd

The -O vpc is the VHD output format (the name is a leftover from Virtual PC), and subformat=dynamic keeps it thin, so you end up with a compact dynamic VHD that is about 11 GB on disk despite presenting a 64 GiB drive to the VM.

Step 3: import the disk

Copy the converted VHD over to the host. It does not need to live on dom0’s root partition, which is usually tiny: put it on the storage repository mount, which has the room. Then create a VDI sized to match the disk’s virtual size exactly, and import the file into it.

That virtual size is not the round 64 GiB you would expect, and this bit me. When qemu-img writes a dynamic VHD it rounds the virtual size up to the nearest CHS (cylinder, head, sector) geometry boundary, so a nominal 64 GiB disk comes out as 68720517120 bytes, not the arithmetic 68719476736. The VDI has to match the VHD exactly, or be larger, or vdi-import rejects it for being too small. So give vdi-create the rounded number that qemu-img actually produced:

xe vdi-create sr-uuid=<SR> name-label=ws2025 virtual-size=68720517120 type=user
xe vdi-import uuid=<VDI> filename=/run/sr-mount/<SR>/ws2025.vhd format=vhd

The import returns 0 and looks, frankly, like it did nothing. Here is the trap: do not trust the physical-utilisation field to confirm the data landed. On a thin (ext) storage repository it lied to me, reading about 135 KiB while 11 GB of Windows sat on the disk. It is the same unreliable metric that had me staring at "0 MiB" mid-install during the from-scratch saga.

The ground truth is the backing file on disk. Run an sr-scan, then look at the actual .vhd the SR created for your VDI:

du -h /run/sr-mount/<SR>/<VDI>.vhd   # should report about 11 G

If that reads roughly 11 G, the import worked, whatever the xe metric tells you.

Step 4: make the VM UEFI, not BIOS

This is the gotcha that cost me the most time, and it cost me time because the internet lied to me with great confidence. Page after page describes the Windows Server evaluation "VHD" as a Generation 1 image, which in Hyper-V terms means BIOS firmware and an MBR partition table. So I dutifully built a BIOS VM, attached the disk, set it to boot, and started it.

The firmware found the disk. It listed it as "XS Virtual IDE Controller, Hard Drive 64 GB", printed "Boot device: Hard Disk, success", and then sat there. Forever. No error, no spinner, just a cursor and silence.

Figure 2. The BIOS VM finds the disk and reports "Boot device: Hard Disk, success", then hangs. There is no MBR boot code on a GPT disk for it to run.

The image is not Generation 1. For build 26100 it is GPT and UEFI, and you can prove it to yourself in about thirty seconds by looking at the first sectors of the converted VHD. You will find a protective MBR (a single partition entry of type 0xEE, which is GPT’s way of telling old tools "this whole disk is taken, hands off"), and the literal string EFI PART sitting at LBA 1, which is the GPT header signature. A BIOS VM finds the disk, looks at the MBR for boot code to run, finds only the protective stub, and has nothing to execute. Hence the polite hang.

The fix is two lines. Build the VM from the "Other install media" template (so XCP-ng does not impose an OS-specific default), then flip it to UEFI and turn off Secure Boot. Secure Boot needs Microsoft’s UEFI certificates enrolled in the pool (that is what secureboot-certs install sets up), and a fresh XCP-ng 8.3 pool does not ship them by default, so a Secure Boot guest has nothing to validate its signed bootloader against until you enroll them. Turning it off is the no-friction path to a booted desktop:

xe vm-param-set uuid=<VM> HVM-boot-params:firmware=uefi
xe vm-param-set uuid=<VM> platform:secureboot=false

Or, if you would rather keep Secure Boot on, you can, as long as the pool has the certificates. You enroll them once per pool with a single command, and from then on every UEFI guest can use Secure Boot:

secureboot-certs install

That fetches the standard Microsoft PK, KEK, db and dbx certificates and stores them at the pool level. Check what a pool already holds with secureboot-certs report; on the pool I tested, all four were already in place, so there was nothing to install. Then build the VM exactly as above, but flip the switch the other way:

xe vm-param-set uuid=<VM> HVM-boot-params:firmware=uefi
xe vm-param-set uuid=<VM> platform:secureboot=true

The evaluation disk boots with Secure Boot enforcing, because its bootloader is signed by Microsoft and the pool now holds the certificate that vouches for it. To prove it landed rather than take my word for it, open a shell in the guest and ask Windows directly:

Confirm-SecureBootUEFI

True means Secure Boot is on and enforcing. On my Windows Server 2025 build 26100 it came back True on the first boot, with the UEFI SetupMode variable reading 0, which is the firmware’s way of saying the keys are enrolled and it is checking signatures for real.

One version caveat, because this leans on the host package behaving. Confirm your dom0 is not running the withdrawn varstored 1.2.0-3.1, which mishandled certificate enrollment; check with rpm -q varstored. The pool I tested was on varstored 1.3.2-2.1 (XCP-ng 8.3), where it works cleanly.

Attach the imported VDI as the bootable disk, give the VM a network interface, set the boot order to disk, and start it.

First boot runs the usual sysprep "Getting ready" dance and reboots itself once partway through, while Windows specializes the image for this particular VM. Be patient here. The console is painfully slow while that happens, and that is a consequence of how we built this VM rather than something XCP-ng does to every guest. A VM created from the "Other install media" template has no vga key in its platform map, and with that key unset qemu hands the HVM guest an emulated Cirrus adapter with 4 MB of video memory. "Getting ready" can take close to ten minutes on it, and it looks identical to a hang. The Windows templates set vga: std and videoram: 8 instead. Hold that thought, because the same template choice is about to cost us something bigger. If you want reassurance that it is actually working and not stuck, check xentop on the host: a VM that is busy specializing is burning CPU, a hung one is not. After the reboot you land at the out-of-box screens. The US region and keyboard defaults are fine to accept, then you set the built-in Administrator password, and you are at a Windows desktop.

Figure 3. With firmware set to UEFI, the imported disk boots: Windows Server 2025 specializing on first run.

Figure 4. Out-of-box setup, region and keyboard. No installer ran to get here, only an imported disk.

One pleasant surprise: the network just works, before you have installed a single paravirtual driver. Windows Server 2025 has an inbox driver for the emulated NIC that XCP-ng presents, so the VM picked up a DHCP lease (and even a public IPv6 address) the moment it reached the desktop. RDP and SSH were reachable immediately. I enabled the in-box OpenSSH server so I could drive the machine from a real shell instead of squinting at a slow VNC console:

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Start-Service sshd
Set-Service -Name sshd -StartupType Automatic

After the from-scratch ordeal, where every keystroke went through a flaky console bridge, having a plain SSH prompt into Windows felt like a small miracle.

Figure 5. The emulated network card has an inbox driver, so the VM pulls a DHCP lease on first boot, before any paravirtual drivers are installed.

The flags that template did not set

Added after publication, because a colleague’s support question sent me back to look at my own VM.

Building from "Other install media" is the right call for getting this disk to boot, and it has a cost I did not notice at the time. That template is deliberately generic, so it does not apply the Viridian settings that the Windows templates do. Viridian is the platform’s name for the Hyper-V enlightenments Windows expects to find, and the documentation is blunt about it: all six parameters should be true for proper operation of Windows.

Put the two templates side by side and the gap is obvious:

xe template-param-get uuid=<Windows Server 2025 template> param-name=platform
xe template-param-get uuid=<Other install media template> param-name=platform

The Windows Server 2025 template sets all six. "Other install media" sets viridian: true and stops there. My imported VM inherited the short version, so it had been running on one flag out of six the whole time.

It is not only Viridian. Put the two platform maps side by side and the Windows template also sets vga: std and videoram: 8, which is the slow console from earlier in this post. One template choice, three defaults you do not get.

Check yours:

xe vm-param-get uuid=<VM> param-name=platform

If any of viridian, viridian_time_ref_count, viridian_reference_tsc, viridian_apic_assist, viridian_crash_ctl or viridian_stimer is missing, or present and set to false, set them with the VM shut down:

xe vm-param-set \
  uuid=<VM> \
  platform:viridian=true \
  platform:viridian_time_ref_count=true \
  platform:viridian_reference_tsc=true \
  platform:viridian_apic_assist=true \
  platform:viridian_crash_ctl=true \
  platform:viridian_stimer=true

I did measure it in the end, because a claim like this is worth more than a shrug. Same VM, same host, the other four VMs on that host shut down. diskspd 2.2, five iterations of each of these:

diskspd.exe -c2G -d25 -W10 -b4K -r -o1 -t1 -Sh -L -w0 C:\bench\test.dat
diskspd.exe -c2G -d20 -W5 -b4K -r -o8 -t4 -Sh -L -w0 C:\bench\test.dat

-Sh turns off Windows-side caching for that file. It does nothing about the layers below the guest, and neither did I, which matters further down. I measured at one flag, flipped to six, went back to one, then back to six, so that a reboot could not quietly take the credit. The first iteration of every run is discarded, because it reads off a warm cache and flatters whichever configuration happens to go first. That is not a theoretical worry: on the queue-depth-1 test the discarded iteration came back at 28.8 MiB/s, against 10.4 to 13.4 for the rest of the run.

4K random read, medians of eight measurements per group:

4K random read	1 of 6 flags	6 of 6 flags
Queue depth 1, average latency	0.312 ms	0.195 ms
Queue depth 1, IOPS	3,143	5,127
Queue depth 8 across 4 threads, IOPS	45,147	71,674

The two groups did not overlap, and the run-to-run spread went from roughly a quarter to a few percent.

Then the honest part, because I would rather nobody changed a production VM on the strength of my afternoon. I flipped five flags at once, so I cannot tell you which one matters, or whether it is one of them or all five together. I cannot explain the mechanism either. I have no good story for why timer and APIC enlightenments would move 4K random read throughput this far, and I can invent one about reducing VM exits, but inventing one is precisely what I am trying to avoid here. It is also possible I measured something that merely correlates with the config change rather than the flags themselves.

Add to that one VM, one host, and local disks rather than iSCSI or NFS. One of the four runs drifted upward partway through in a way that looks like host caching, which is exactly the layer I never controlled, and which is why the table gives medians and no percentages at all. Please do not quote these as a general figure, and if you know why the difference is this large, I would rather hear it than keep guessing.

What I am confident about is the first half of this section, because it is only reading template parameters back. A VM built this way gets one flag instead of six, and putting the other five back is cheap.

The documentation attributes a harder failure to the same missing synthetic timer: a Windows Server 2025 guest hangs randomly at 0% CPU. The flags are the first thing that page tells you to check, and they only cover the host side of it. If you have ever set Windows boot parameters to disable the synthetic timer, those have to come back out too:

bcdedit /deletevalue "{current}" useplatformclock
bcdedit /deletevalue "{current}" useplatformtick

That wants an elevated prompt, and it wants a reboot: the BCD store is read at boot and nowhere else, so nothing changes until the guest comes back up.

The same gap catches anything that did not come from a Windows template, such as a VM imported from a VHD like this one. Migrating in from VMware is less clear-cut than I first assumed: the documentation says Viridian "is also enabled on V2V-migrated and Packer-created VMs if using the correct templates", so the outcome depends on what the migration landed on. I have not run a V2V import myself, so read the platform map rather than assuming it either way.

Step 5: the guest tools, and the one real trap

The emulated NIC works, but you still want the XCP-ng guest tools. They bring the paravirtual network and storage drivers (xennet and xenvbd), which are faster than the emulated devices, and the management agent that reports the guest’s IP back to the host so XAPI and Xen Orchestra can show you where the VM actually is on the network. Installing them is where the one true footgun of this whole exercise lives, so slow down for this part.

There is a VM setting called has-vendor-device. The name sounds like exactly what you would want on: something for the PV drivers to bind to. That intuition is precisely wrong. I turned it on, rebooted with the network up, and Windows Update quietly reached out and installed Citrix’s PV drivers underneath me, without asking. Because has-vendor-device=true is not "let the PV drivers bind". It is the "Manage Citrix PV drivers via Windows Update" feature. Flip it on, give Windows a network, and Update treats that as permission to manage your Xen drivers for you.

In my case they installed broken. The bus driver came up, but its children sat in an error state, so the NIC never moved off the emulated device anyway. Worse, the XCP-ng XenTools-x64.msi then refused to install at all. It checks for exactly this situation, found the Citrix drivers already present, threw "incompatible devices found", and exited 1603. So the helpful Windows Update behaviour had locked me out of the supported tools.

The supported path, straight from the XCP-ng docs, is the opposite of what the setting’s name suggests:

Leave has-vendor-device set to false. That is the expected, documented state for XCP-ng’s own tools. The PV drivers bind to the always-present base Xen platform device regardless; the vendor device is only the Windows Update trigger, and you do not want that trigger pulled.
If Windows Update already grabbed the Citrix drivers (as mine had), run XenClean.exe from the guest tools CD as administrator. It rips out the Citrix drivers and reboots itself, no Safe Mode gymnastics required.
Then run XenTools-x64.msi.

After XenClean, the tools install, and one reboot later every xen* device came up clean in Device Manager, the NIC switched over to the "XCP-ng PV Network Device", and the host started reporting the guest’s IP. The PV driver version was 9.1.200.0, the current release when I tested. Done.

One component in that package is worth calling out separately if your VM is UEFI, which this one is: the XCP-ng Standard VGA Display Driver. The documentation says a UEFI VM needs it before Windows screen resolution can be changed from inside the OS at all, which is a fair description of the console I have grumbled about twice in this post. Whether it also shortens the "Getting ready" wait I have not measured, so treat that as the docs' claim about resolution and nothing more.

The one-line takeaway, because it is the line I wish I had read first: leave has-vendor-device off, and let XCP-ng’s MSI install the drivers. Turning it on hands your driver management to Windows Update and Citrix, and breaks the supported path.

The payoff

From a downloaded VHD to a running, networked, paravirtualised Windows Server, with the disk saved for reuse, took me well under an hour the second time around. Compare that with the day and a half I burned teaching the installer to find its own disk, only to watch the result brick on first boot. When the platform already hands you a finished disk image, importing it beats installing.

Figure 6. The payoff: a full Windows Server desktop on XCP-ng, and the installer never ran once.

I keep the converted VHD in a small image library now, with a plain text file next to it listing these exact steps. Next time I need a throwaway Windows on XCP-ng, it is a copy and an import away, and the hard part is already done.

As for why I wanted a Windows VM on XCP-ng in the first place, that involves an Intel iGPU, GPU passthrough, and a hardware video encoder that only exists on the Windows side of the fence. But that is a story for the next post.

Twelve Years, One Bet on PowerPC: The Power Progress Community Story

Bruno Verachten — Tue, 12 May 2026 20:44:24 +0000

Originally published on May 12, 2026 at https://bruno.verachten.fr/2026/05/12/twelve-years-one-bet-on-powerpc-the-power-progress-community-story/

In 2014, a small group of Italian volunteers decided to build an open hardware PowerPC laptop from scratch. No salary. No investor deck. No deadline. Just a shared conviction that the world needed an open computer built on an open architecture, running free software, designed by and for the community.

Twelve years later, they’re still at it. I recently spoke with Roberto Innocenti, president of the Power Progress Community, and the conversation stayed with me for days.

Photo by Сергей on Pexels

Who they are

Power Progress Community is a non-profit registered in Italy. Every member is a volunteer. Nobody gets paid except the electronic engineers they hire for specific funded tasks. Roberto describes their "business plan" as deliberately flexible: no crowdfunding countdown clock, no tight time-to-market pressure. Just an open donation campaign they can sustain indefinitely.

Their mission reads like something from the early free software movement. Solidarity-based knowledge. Freedom of choice. Technology designed to resist surveillance capitalism. Open hardware accessible to people far from the world of computing. They run coding workshops in Italian schools using repurposed old hardware. They partner with universities so students can design open hardware as thesis projects.

This is not a startup. It’s something older and stranger: a community running on passion.

The technology

The project centers on a PowerPC laptop and now a desktop, built around the NXP T2080. That’s a 64-bit Power Architecture chip with four dual-threaded e6500 cores and AltiVec vector support. The desktop version, called the Powerboard Tyche, comes in Micro ATX form factor with two PCIe x16 video card slots, two NVMe slots, and dual gigabit Ethernet.

Why PowerPC? Roberto puts it bluntly: "The PowerPC architecture design is newer than the other successful CPU architectures: x86 (1978), MIPS (1981), ARM (1983) — PowerPC (1991)." More to the point, since August 2019 the Power ISA became fully open. Developers can now design chips based on the royalty-free specification. The architecture that once ran inside every Mac, every PlayStation 3, and every Nintendo Wii is now genuinely free silicon.

The Powerboard Tyche schematics are published under the CERN Open Hardware Licence 1.2. PCB source files are on GitLab. They’re pursuing OSHWA certification. NXP itself agreed to let them publish the T2080 reference design as open hardware. That last part is not a given with any chip manufacturer.

Twelve years in the making

The timeline is a story of persistence against everything that can go wrong in open hardware.

The project started in 2014 with a laptop idea. The formal association launched in 2016. By 2017 they were running their first crowdfunding campaign, partnering with ACube Systems for electrical engineering and with Slimbook (a Spanish Linux laptop maker) for the chassis. By 2022 they had physical prototypes.

Then came the hard part.

The prototype booted fine on NXP’s reference development kit. It refused to boot on their custom board. For eighteen months, from November 2023 to April 2025, the team tried every fix they could think of. CPLD updates. Firmware comparisons. Desoldering chips to isolate the fault. The board still wouldn’t boot.

In May 2025, after consulting their donor community, they made a call: pivot to a desktop first. Simpler enclosure. Lower production cost. Merged design based on the updated NXP T2080 reference board. In June 2025, NXP provided the latest T2080 RDB design. In October 2025, schematics were done. By March 2026, the PCB layout was complete.

That pivot isn’t a failure. It’s twelve years of knowing when to adapt without dropping the goal.

Why this matters

At Vates, the company behind the XCP-ng open source hypervisor, there’s an ongoing effort to port Xen to RISC-V, another open instruction set architecture that barely existed a decade ago. XCP-ng itself isn’t there yet. But the direction is clear: open silicon, community governance, computing that isn’t owned by a handful of companies. Watching Power Progress Community work through a decade of non-x86 hardware development feels like the same conversation.

There’s also an audience overlap worth naming. The open source sysadmin and virtualization world is full of people who came from Amiga, PowerPC Macs, and older non-x86 systems. People who remember when computing was more architecturally diverse, and think it should be again. If that’s you, or if you simply believe computing is healthier with more than two or three viable architectures, you probably want to know this project exists.

Where they are now and how to help

Update (2026-11-02): the prototype campaign reached its goal: €9,395 donated against a €9,100 target when I checked, enough to pay the manufacturer for three Powerboard Tyche desktop prototypes (not the five mentioned below). The firmware and hardware engineers are now testing them: the CPLD and uPic are programmed and the first SPD messages show on the console. Their post doesn’t report a boot yet. The project also says it has been awarded a grant from the NGI Zero Commons Fund through NLnet, with the first funds expected in December. Details in their post, 48 hours to balance the prototypes: the final effort (30 October 2026). The paragraph below is the state of things in May.

The Powerboard Tyche desktop PCB design is done. The KiCad schematics are published. What’s left is funding, in two steps. First, roughly €3,500 to balance the outstanding payment on the PCB design itself. Once that’s settled, they can commit to the manufacturer and open the campaign to build the first five physical prototypes, which needs around €9,000 in total. Modest sums relative to what’s already been invested in engineering and community time.

If you want to follow the project:

Project updates and timeline: powerpc-notebook.org
Donation campaign: fundraising.powerpc-notebook.org
Community forum: forum.powerprogress.org
GitLab (schematics, PCB sources): gitlab.com/power-progress-community

Roberto has presented at SFScon (the South Tyrol Free Software Conference in Bolzano) every year since 2018. His 2025 talk covers the full story from laptop prototype struggles to the desktop pivot. The slides are available here.

Open hardware is slow work. Slower than software, slower than cloud services, slower than anything the venture capital cycle rewards. The Power Progress Community has been doing it anyway. Twelve years. Volunteer time. Donor funding. University partnerships. One prototype that wouldn’t boot, and the stubbornness to redesign rather than quit.

The PCB is done. I hope the prototype isn’t far behind.

Compile FFmpeg on the Odroid XU4 to get hardware acceleration

Bruno Verachten — Wed, 22 Apr 2026 07:44:26 +0000

Originally published on November 28, 2019 at https://bruno.verachten.fr/2019/11/28/compile-ffmpeg-on-the-odroid-xu4-to-get-hardware-acceleration/

sudo apt-get build-dep ffmpeg

sudo apt install debhelper libchromaprint-dev frei0r-plugins-dev libgnutls28-dev libx265-dev ladspa-sdk libaom-dev libserd-dev libsord-dev libsratom-dev libsndfile1-dev libiec61883-dev libavc1394-dev libavc1394-0 libass-dev libbluray-dev libbs2b-dev libcaca-dev

 mkdir ffmpeg_sources
   36  cd ffmpeg_sources
   37  wget http://deb.debian.org/debian/pool/main/f/ffmpeg/ffmpeg_4.1.4.orig.tar.xz
   38  xz -d ffmpeg_4.1.4.orig.tar.xz
   39  tar -xf ffmpeg_4.1.4.orig.tar
   40  wget http://deb.debian.org/debian/pool/main/f/ffmpeg/ffmpeg_4.1.4.orig.tar.xz
   41  ll
   42  cd ffmpeg-4.1.4/
   43  wget http://deb.debian.org/debian/pool/main/f/ffmpeg/ffmpeg_4.1.4-1~deb10u1.debian.tar.xz
   44  xz -d ffmpeg_4.1.4-1~deb10u1.debian.tar.xz
   45  ll
   46  tar -xf ffmpeg_4.1.4-1~deb10u1.debian.tar
   47  ll
   48  rm ffmpeg_4.1.4-1~deb10u1.debian.tar

sudo -E dpkg-buildpackage -us -uc -d

ffmpeg -i bbb_sunflower_1080p_30fps_stereo_abl.mp4 -an -vcodec h264_v4l2m2m -b:v 2M -pix_fmt nv21 output.mp4
ffmpeg version 4.1.4-1~deb10u1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: --prefix=/usr --extra-version='1~deb10u1' --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --arch=arm --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_30fps_stereo_abl.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2013-12-16T17:46:34.000000Z
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
    composer        : Sacha Goedegebure
  Duration: 00:10:34.53, start: 0.000000, bitrate: 5482 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x2160 [SAR 1:1 DAR 8:9], 4999 kb/s, 30 fps, 30 tbr, 30k tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2013-12-16T17:46:34.000000Z
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
    Metadata:
      creation_time   : 2013-12-16T17:46:41.000000Z
      handler_name    : GPAC ISO Audio Handler
    Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
    Metadata:
      creation_time   : 2013-12-16T17:46:41.000000Z
      handler_name    : GPAC ISO Audio Handler
    Side data:
      audio service type: main
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0x575a90] driver 'exynos-gsc' on card 'exynos-gsc gscaler'
    Last message repeated 1 times
[h264_v4l2m2m @ 0x575a90] driver 's5p-mfc' on card 's5p-mfc-enc'
[h264_v4l2m2m @ 0x575a90] Using device /dev/video11
[h264_v4l2m2m @ 0x575a90] driver 's5p-mfc' on card 's5p-mfc-enc'
[h264_v4l2m2m @ 0x575a90] h264 profile not found
[h264_v4l2m2m @ 0x575a90] Encoder adjusted: qmin (0), qmax (51)
Output #0, mp4, to 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    composer        : Sacha Goedegebure
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: h264 (h264_v4l2m2m) (avc1 / 0x31637661), nv21, 1920x2160 [SAR 1:1 DAR 8:9], q=2-31, 2000 kb/s, 30 fps, 15360 tbn, 30 tbc (default)
    Metadata:
      creation_time   : 2013-12-16T17:46:34.000000Z
      handler_name    : GPAC ISO Video Handler
      encoder         : Lavc58.35.100 h264_v4l2m2m
[mp4 @ 0x5695f0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[mp4 @ 0x5695f0] Non-monotonous DTS in output stream 0:0; previous: 832000, current: 0; changing to 832001. This may result in incorrect timestamps in the output file.

Terminal recording on the OrangePi Zero

Bruno Verachten — Wed, 22 Apr 2026 07:44:23 +0000

Originally published on January 3, 2020 at https://bruno.verachten.fr/2020/01/03/terminal-recording-on-the-orangepi-zero/

Terminal recording on the OrangePi Zero

Lately, I started writing a few articles on the OrangePi Zero, as I am trying to use it as part of a conference recording tool and to reference what I know about this board on the OrangePi Community docs website.

Of course, I have to show my command lines, and their results.

I used to post my commands on gist, then use link:https://carbon.now.sh/?bg=rgba(171%2C 184%2C 195%2C 1[Carbon]&t=seti&wt=none&l=auto&ds=true&dsyoff=20px&dsblur=68px&wc=true&wa=true&pv=56px&ph=56px&ln=false&fm=Hack&fs=14px&lh=133%&si=false&es=2x&wm=false) to generate an image with the commands and their outputs… But that’s quite a lot of boring work…

I then discovered asciinema and the recording that can be transformed into a GIF file thanks to asciicast2gif.

It worked pretty well under Ubuntu for my previous projects, but it was not really practical to launch it, issue an ssh command, and then exit twice before having the recording… and modifying it to get rid of the SSH command.

Furthermore, GIF are not usable for visually impaired people, so using something like SVG where they could copy and paste would be better in my opinion…

Unfortunately, SVG is not handled by medium, so it will have to be used on other media, like my Ghost instance (not available yet).

Enters termtosvg. That’s a wonderful tool aimed at producing animated SVG after recording terminal sessions.

It compiled like a breeze (well, a strong, smelly and cold breeze) on the Zero, so I used it quite intensively (before discovering it could not be used on Medium).

I then searched for tools able to transform an animated SVG to GIF, so that I would not have to redo all my terminal recordings… and was not able to do so.

There is one library called librsvg that could be used by ffmpeg, but I have not been able to compile it on the Zero, and it’s just an hypothesis, there is no guarantee at all that it could be used to produce GIF files from animated SVG files.

termtosvg can produce files than can be used by asciinema and then asciicast2gif to produce gif files, but the process is cumbersome and does not work on the Zero. I had to use a X86 Linux machine to do it.

Anyway, with this article, you won’t be able to produce GIF images starting from a terminal recording if you’re using exclusively a Zero, but beautiful animated SVG that you will be able to use on your blog, or anywhere else where animated SVG is supported.

If you’re brave enough to also use an X86 machine, go to the end of this article, and take time to get married, get children before getting your first GIF file.

TermToSVG installation

TL;DR

git config --global http.proxy http://proxy-machine:proxy-port
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo nano /etc/fstab
git clone https://github.com/nbedos/termtosvg.git
sudo apt-get install python3-setuptools libxml2-dev libxslt-dev python3-dev python3-pip
pip3 install wheel
cd termtosvg
echo "Let's modify the Makefile so that we use python3 and pip3 as I can't get rid of Python2.7.x on this machine..."
echo "See at https://gist.github.com/gounthar/b1870915710a0437b4e4eb8690e93553#file-makefile-patch"
make build
pip3 install dist/termtosvg-1.1.0-py3-none-any.whl

The real installation process

Let’s try to install termtosvg on the Zero. First of all, let’s configure the proxy if needed and then get the source code:

poddingue@orangepizero:~$ git clone https://github.com/nbedos/termtosvg.git
Cloning into 'termtosvg'...
remote: Enumerating objects: 99, done.
remote: Counting objects: 100% (99/99), done.
remote: Compressing objects: 100% (59/59), done.
remote: Total 1592 (delta 59), reused 68 (delta 40), pack-reused 1493
Receiving objects: 100% (1592/1592), 1.37 MiB | 2.08 MiB/s, done.
Resolving deltas: 100% (930/930), done.
poddingue@orangepizero:~$ cd termtosvg/
poddingue@orangepizero:~/termtosvg$ make build
rm -rf dist && \
python setup.py sdist bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'
make: *** [Makefile:31: build] Error 1

Of course, the setuptools module can’t be found. I have not installed/configured Python2 setup tools. But as I had already done if for Python3, I modified the Makefile so that python is now python3, and pip is now pip3:

PIP=pip3
PYTHON=python3

If I had not done it yet, it would have looked that way:

poddingue@orangepizero:~/termtosvg$ sudo apt-get install python3-setuptools
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  python-setuptools-doc
The following NEW packages will be installed:
  python3-setuptools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 306 kB of archives.
After this operation, 1,353 kB of additional disk space will be used.
Get:1 http://httpredir.debian.org/debian buster/main armhf python3-setuptools all 40.8.0-1 [306 kB]
Fetched 306 kB in 0s (1,225 kB/s)
Selecting previously unselected package python3-setuptools.
(Reading database ... 66711 files and directories currently installed.)
Preparing to unpack .../python3-setuptools_40.8.0-1_all.deb ...
Unpacking python3-setuptools (40.8.0-1) ...
Setting up python3-setuptools (40.8.0-1) ...

Now that Python setup tools are installed, let’s try it again:

poddingue@orangepizero:~/termtosvg$ make build
rm -rf dist && \
python3 setup.py sdist bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'
make: *** [Makefile:31: build] Error 1

Ok, fair enough, I’ll install wheel thanks to pip.

pip3 install wheel
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: wheel
  The script wheel is installed in '/home/poddingue/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed wheel-0.33.6

And then, as gently suggested by pip3:

export PATH=/home/poddingue/.local/bin:$PATH

Ok, let’s try it again:

poddingue@orangepizero:~/termtosvg$ make build
rm -rf dist && \
python3 setup.py sdist bdist_wheel
running sdist
running egg_info
creating termtosvg.egg-info
writing termtosvg.egg-info/PKG-INFO
writing dependency_links to termtosvg.egg-info/dependency_links.txt
writing requirements to termtosvg.egg-info/requires.txt
writing top-level names to termtosvg.egg-info/top_level.txt
writing manifest file 'termtosvg.egg-info/SOURCES.txt'
reading manifest file 'termtosvg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no files found matching 'man/*.man.?'
writing manifest file 'termtosvg.egg-info/SOURCES.txt'
running check
[...]
Creating tar archive
removing 'termtosvg-1.0.0' (and everything under it)
[...]
running build_scripts
[...]
installing to build/bdist.linux-armv7l/wheel
running install
running install_lib
[...]
Copying termtosvg.egg-info to build/bdist.linux-armv7l/wheel/termtosvg-1.0.0-py3.7.egg-info
[...]
adding 'termtosvg-1.0.0.dist-info/RECORD'
removing build/bdist.linux-armv7l/wheel

Happy? Not really…

poddingue@orangepizero:~/termtosvg$ pip3 install dist/termtosvg-1.0.0-py3-none-any.whl
Processing ./dist/termtosvg-1.0.0-py3-none-any.whl
Collecting pyte (from termtosvg==1.0.0)
  Downloading https://files.pythonhosted.org/packages/66/37/6fed89b484c8012a0343117f085c92df8447a18af4966d25599861cd5aa0/pyte-0.8.0.tar.gz (50kB)
    100% |████████████████████████████████| 51kB 768kB/s
Collecting lxml (from termtosvg==1.0.0)
  Downloading https://files.pythonhosted.org/packages/e4/19/8dfeef50623892577dc05245093e090bb2bab4c8aed5cad5b03208959563/lxml-4.4.2.tar.gz (2.6MB)
    100% |████████████████████████████████| 2.6MB 74kB/s
Collecting wcwidth (from termtosvg==1.0.0)
  Downloading https://files.pythonhosted.org/packages/58/b4/4850a0ccc6f567cc0ebe7060d20ffd4258b8210efadc259da62dc6ed9c65/wcwidth-0.1.8-py2.py3-none-any.whl
Building wheels for collected packages: pyte, lxml
  Running setup.py bdist_wheel for pyte ... done
  Stored in directory: /home/poddingue/.cache/pip/wheels/c0/dd/4a/d0ec26b9d07a3b48e25ba3456dc9bcab875686af6da9e23fcd
  Running setup.py bdist_wheel for lxml ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-4whala87/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-w27wrikj --python-tag cp37:
  Building lxml version 4.4.2.
  Building without Cython.
  ERROR: b'/bin/sh: 1: xslt-config: not found\n'
  _ make sure the development packages of libxml2 and libxslt are installed _

  Using build configuration of libxslt
  running bdist_wheel
  running build
  running build_py
  [...]
  arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/usr/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-armv7l-3.7/src/lxml/etree.o -w
  In file included from src/lxml/etree.c:692:
  src/lxml/includes/etree_defs.h:14:10: fatal error: libxml/xmlversion.h: No such file or directory
   #include "libxml/xmlversion.h"
            ^~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  Compile failed: command 'arm-linux-gnueabihf-gcc' failed with exit status 1
  creating tmp
  cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitp1a2wq_x.c -o tmp/xmlXPathInitp1a2wq_x.o
  /tmp/xmlXPathInitp1a2wq_x.c:1:10: fatal error: libxml/xpath.h: No such file or directory
   #include "libxml/xpath.h"
            ^~~~~~~~~~~~~~~~
  compilation terminated.
  *********************************************************************************
  Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
  *********************************************************************************
  error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1
  [...]
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-4whala87/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-5dypr5tk/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-install-4whala87/lxml/

Damn!

poddingue@orangepizero:~/termtosvg$ sudo apt install libxml2-dev libxslt-dev python3-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
python3-dev is already the newest version (3.7.3-1).
The following additional packages will be installed:
  icu-devtools libicu-dev libxslt1.1
Suggested packages:
  icu-doc
The following NEW packages will be installed:
  icu-devtools libicu-dev libxml2-dev libxslt1-dev libxslt1.1
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.5 MB of archives.
After this operation, 44.6 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://httpredir.debian.org/debian buster/main armhf icu-devtools armhf 63.1-6 [168 kB]
Get:2 http://httpredir.debian.org/debian buster/main armhf libicu-dev armhf 63.1-6 [8,934 kB]
Get:3 http://httpredir.debian.org/debian buster/main armhf libxml2-dev armhf 2.9.4+dfsg1-7+b3 [718 kB]
Get:4 http://httpredir.debian.org/debian buster/main armhf libxslt1.1 armhf 1.1.32-2.2~deb10u1 [217 kB]
Get:5 http://httpredir.debian.org/debian buster/main armhf libxslt1-dev armhf 1.1.32-2.2~deb10u1 [507 kB]
Fetched 10.5 MB in 5s (2,318 kB/s)
Selecting previously unselected package icu-devtools.
(Reading database ... 66802 files and directories currently installed.)
Preparing to unpack .../icu-devtools_63.1-6_armhf.deb ...
Unpacking icu-devtools (63.1-6) ...
Selecting previously unselected package libicu-dev:armhf.
Preparing to unpack .../libicu-dev_63.1-6_armhf.deb ...
Unpacking libicu-dev:armhf (63.1-6) ...
Selecting previously unselected package libxml2-dev:armhf.
Preparing to unpack .../libxml2-dev_2.9.4+dfsg1-7+b3_armhf.deb ...
Unpacking libxml2-dev:armhf (2.9.4+dfsg1-7+b3) ...
Selecting previously unselected package libxslt1.1:armhf.
Preparing to unpack .../libxslt1.1_1.1.32-2.2~deb10u1_armhf.deb ...
Unpacking libxslt1.1:armhf (1.1.32-2.2~deb10u1) ...
Selecting previously unselected package libxslt1-dev:armhf.
Preparing to unpack .../libxslt1-dev_1.1.32-2.2~deb10u1_armhf.deb ...
Unpacking libxslt1-dev:armhf (1.1.32-2.2~deb10u1) ...
Setting up icu-devtools (63.1-6) ...
Setting up libxslt1.1:armhf (1.1.32-2.2~deb10u1) ...
Setting up libicu-dev:armhf (63.1-6) ...
Setting up libxml2-dev:armhf (2.9.4+dfsg1-7+b3) ...
Setting up libxslt1-dev:armhf (1.1.32-2.2~deb10u1) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...

All set? Not so sure…

Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-9lgddbtt/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ghyj96g8/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-install-9lgddbtt/lxml/

The error may have happened because of low memory. Let’s this machine breathe a little better by giving it more swap:

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo nano /etc/fstab

Now, let’s add this line in /etc/fstab.

/swapfile swap swap defaults 0 0

Let’s try to install it once again. Go and have a coffee, or go for a walk… You could maybe even take time to marry and start a family before the build finishes. Anyway, after some time, you’ll have your beautiful termtosvg package ready to rock!

pip3 install dist/termtosvg-1.0.0-py3-none-any.whl
Processing ./dist/termtosvg-1.0.0-py3-none-any.whl
Collecting lxml (from termtosvg==1.0.0)
  Using cached https://files.pythonhosted.org/packages/e4/19/8dfeef50623892577dc05245093e090bb2bab4c8aed5cad5b03208959563/lxml
Requirement already satisfied: pyte in /home/poddingue/.local/lib/python3.7/site-packages (from termtosvg==1.0.0) (0.8.0)
Requirement already satisfied: wcwidth in /home/poddingue/.local/lib/python3.7/site-packages (from termtosvg==1.0.0) (0.1.8)
Building wheels for collected packages: lxml
  Running setup.py bdist_wheel for lxml ... -


done
  Stored in directory: /home/poddingue/.cache/pip/wheels/73/52/7c/5cd696851d3e5e31a05023dd402b04659a6ec695ecc566c9d3
Successfully built lxml
Installing collected packages: lxml, termtosvg
Successfully installed lxml-4.4.2 termtosvg-1.0.0

This time, it looks like our installation has been successful. Let’s try it for real now.

Testing

Just enter termtosvg, do your stuff, and then do exit. You will have a message that will tell you where your newly created file lies.

That’s cool, but can we do any better? The man page tells us we can record a terminal session with a specific screen geometry:

termtosvg -g 80x24 animation.svg
Recording started, enter "exit" command or Control-D to end
poddingue@orangepizero:~$ echo "Hello, World!"
Hello, World!
poddingue@orangepizero:~$ exit
Rendering ended, SVG animation is animation.svg

Pretty handy!

I am recording my casts with a 200x60 geometry, exporting them in the SVG format. The first step is to record them:

termtosvg record -g 200x60 whatever-name-I-choose.cast

I then ask termtosvg to render the cast as SVG, so that I can use them in websites that can handle them:

termtosvg render whatever-name-I-choose.cast whatever-name-I-choose.svg
Rendering started
Rendering ended, SVG animation is whatever-name-I-choose.svg

Want more? You’re GIF hungry? Ok I resign myself to give you my process, which is not pretty, and requires an X86 machine. As we have seen before, we can record a terminal session in asciicast v2 format. The cast file we have produced during the previous step to generate SVG, can then be used by asciinema (using phantomjs in the background) to produce a GIF file.

docker pull asciinema/asciicast2gif
alias asciicast2gif='docker run --rm -v $PWD:/data asciinema/asciicast2gif'
asciicast2gif -s 2 -t solarized-dark whatever-name-I-choose.cast demo.gif

And here is the final result:

And in GIF for the CMS that can’t display properly SVG.

Sphinx: Installing sphinxbase on the OrangePi 4B

Bruno Verachten — Wed, 22 Apr 2026 07:44:21 +0000

Originally published on June 5, 2020 at https://bruno.verachten.fr/2020/06/05/sphinx-installing-sphinxbase-on-the-orangepi-4b/

This post will be part of a series regarding generating subtitles for videos with the OrangePi 4B running Armbian. We’re not yet sure of what we’ll use to generate those subtitles, so for the time being, we’ll use Sphinx, and in the following months, we’ll try to setup something that will take advantage of the NPU.

Today, we’ll install SphinxBase.

git clone https://github.com/cmusphinx/sphinxbase.git
cd sphinxbase/
./autogen.sh

Unfortunately we get:

_Error_: You must have `libtool' installed.
Get ftp://ftp.gnu.org/pub/gnu/libtool/libtool-2.2.6b.tar.gz
(or a newer version if it is available)

Ok, my bad:

sudo apt install libtool

Let’s give it another try:

./autogen.sh
[...]
configure: error: in `/home/poddingue/sphinxbase':
configure: error:
  Could not link test program to Python. Maybe the main Python library has been
  installed in some non-standard library path. If so, pass it to configure,
  via the LDFLAGS environment variable.
  Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
  ============================================================================
   ERROR!
   You probably have to install the development version of the Python package
   for your distribution.  The exact name of this package varies among them.
  ============================================================================

See `config.log' for more details

I see. I may have forgotten to install Python-dev…

sudo apt install python3-dev
[...]
./autogen.sh
[...]
configure: error: in `/home/poddingue/sphinxbase':
configure: error:
  Could not link test program to Python. Maybe the main Python library has been
  installed in some non-standard library path. If so, pass it to configure,
  via the LDFLAGS environment variable.
  Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
  ============================================================================
   ERROR!
   You probably have to install the development version of the Python package
   for your distribution.  The exact name of this package varies among them.
  ============================================================================

How come? Are you still working with Python 2.x? Naaaah… Ok, you never know, let’s try it this way:

sudo apt install python-dev

Yes, that was it. Now, let’s tackle this error:

checking python extra linking flags... -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
checking consistency of all components of python development environment... yes
checking for swig... no
checking for swig2.0... no
configure: error: swig not found

This should be solved by

sudo apt install swig
./autogen.sh
[...]
config.status: executing libtool commands
Now type `make' to compile the package.

Bingo! Time for a good old

make -j6
make check
[...]
Testsuite summary for sphinxbase 5prealpha
============================================================================
== TOTAL: 10
== PASS:  10
== SKIP:  0
== XFAIL: 0
== FAIL:  0
== XPASS: 0
== ERROR: 0
============================================================================

All went fine. Let’s install it then:

 sudo -E make install

Done. Next step, install Pocket Sphinx.

Installing TensorFlow on the Orange Pi 4B

Bruno Verachten — Wed, 22 Apr 2026 07:44:19 +0000

Originally published on June 19, 2020 at https://bruno.verachten.fr/2020/06/19/installing-tensorflow-on-the-orange-pi-4b/

This quick howto is heavily based on Orange Pi’s documentation. To this day, Orange Pi recommended versions of TensorFlow and Python are quite old:

TensorFlow：1.14.0 Python environment：python2.7

Installation of Python environment and dependencies

sudo apt-get install python-pip python-dev libatlas-base-dev
[sudo] password for poddingue:
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-dev is already the newest version (2.7.16-1).
The following additional packages will be installed:
  libatlas3-base python-pip-whl
Suggested packages:
  libatlas-doc liblapack-doc
Recommended packages:
  python-all-dev python-setuptools python-wheel
The following NEW packages will be installed:
  libatlas-base-dev libatlas3-base python-pip python-pip-whl
Need to get 7,168 kB of archives.
After this operation, 40.1 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://debian.proxad.net/debian buster/main arm64 libatlas3-base arm64 3.10.3-8 [2,409 kB]
Get:2 http://debian.proxad.net/debian buster/main arm64 libatlas-base-dev arm64 3.10.3-8 [2,954 kB]
Get:3 http://debian.proxad.net/debian buster/main arm64 python-pip-whl all 18.1-5 [1,591 kB]
Get:4 http://debian.proxad.net/debian buster/main arm64 python-pip all 18.1-5 [215 kB]
Fetched 7,168 kB in 1s (8,178 kB/s)
Selecting previously unselected package libatlas3-base:arm64.
(Reading database ... 112671 files and directories currently installed.)
Preparing to unpack .../libatlas3-base_3.10.3-8_arm64.deb ...
Unpacking libatlas3-base:arm64 (3.10.3-8) ...
Selecting previously unselected package libatlas-base-dev:arm64.
Preparing to unpack .../libatlas-base-dev_3.10.3-8_arm64.deb ...
Unpacking libatlas-base-dev:arm64 (3.10.3-8) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../python-pip-whl_18.1-5_all.deb ...
Unpacking python-pip-whl (18.1-5) ...
Selecting previously unselected package python-pip.
Preparing to unpack .../python-pip_18.1-5_all.deb ...
Unpacking python-pip (18.1-5) ...
Setting up libatlas3-base:arm64 (3.10.3-8) ...
update-alternatives: using /usr/lib/aarch64-linux-gnu/atlas/libblas.so.3 to provide /usr/lib/aarch64-linux-gnu/libblas.so.3 (libblas.so.3-aarch64-linux-gnu) in auto mode
update-alternatives: using /usr/lib/aarch64-linux-gnu/atlas/liblapack.so.3 to provide /usr/lib/aarch64-linux-gnu/liblapack.so.3 (liblapack.so.3-aarch64-linux-gnu) in auto mode
Setting up libatlas-base-dev:arm64 (3.10.3-8) ...
update-alternatives: using /usr/lib/aarch64-linux-gnu/atlas/libblas.so to provide /usr/lib/aarch64-linux-gnu/libblas.so (libblas.so-aarch64-linux-gnu) in auto mode
update-alternatives: using /usr/lib/aarch64-linux-gnu/atlas/liblapack.so to provide /usr/lib/aarch64-linux-gnu/liblapack.so (liblapack.so-aarch64-linux-gnu) in auto mode
Setting up python-pip-whl (18.1-5) ...
Setting up python-pip (18.1-5) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...

Install TensorFlow

There are several ways to install TensorFlow, but here we will install if from a downloaded binary. Download the PIP installation package for TensorFlow from the following page: https://github.com/lhelontra/tensorflow-on-arm/releases/tag/v1.14.0 .

This should give something like:

curl "https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.14.0/tensorflow-1.14.0-cp27-none-linux_aarch64.whl" --output tensorflow-1.14.0-cp27-none-linux_aarch64.whl

Now, let’s use PIP to install TensorFlow.

pip install tensorflow-1.14.0-cp27-none-linux_aarch64.whl
Processing ./tensorflow-1.14.0-cp27-none-linux_aarch64.whl
Collecting keras-preprocessing>=1.0.5 (from tensorflow==1.14.0)
  Downloading https://files.pythonhosted.org/packages/79/4c/7c3275a01e12ef9368a892926ab932b33bb13d55794881e3573482b378a7/Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42kB)
    100% |████████████████████████████████| 51kB 2.5MB/s
Collecting enum34>=1.1.6 (from tensorflow==1.14.0)
  Downloading https://files.pythonhosted.org/packages/6f/2c/a9386903ece2ea85e9807e0e062174dc26fdce8b05f216d00491be29fad5/enum34-1.1.10-py2-none-any.whl
Collecting wheel (from tensorflow==1.14.0)
  Downloading https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Collecting wrapt>=1.11.1 (from tensorflow==1.14.0)
  Downloading https://files.pythonhosted.org/packages/82/f7/e43cefbe88c5fd371f4cf0cf5eb3feccd07515af9fd6cf7dbf1d1793a797/wrapt-1.12.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named setuptools
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Dd0045/wrapt/

Of course…

sudo apt-get install -y python-setuptools
[...]
  error: invalid command 'bdist_wheel'

Damn!

pip install wheel

And

export PATH=$PATH:/home/poddingue/.local/bin

in `~/.bashrc`, because of

pip install wheel
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Installing collected packages: wheel
  The script wheel is installed in '/home/poddingue/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed wheel-0.34.2

Let’s issue once more `

` bash
pip install tensorflow-1.14.0-cp27-none-linux_aarch64.whl

error: libhdf5.so: cannot open shared object file: No such file or directory
`

Bad luck.

bash Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-irkqLB/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-N9oIr_/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-install-irkqLB/h5py/

Something must be missing…

Not better… I think the machine is trying to tell me something… A missing library perhaps?

bash sudo apt install python-h5py pip install tensorflow-1.14.0-cp27-none-linux_aarch64.whl [...] Successfully installed astor-0.8.1 backports.weakref-1.0.post1 funcsigs-1.0.2 gast-0.3.3 google-pasta-0.2.0 keras-applications-1.0.8 mock-3.0.5 tensorflow-1.14.0 tensorflow-estimator-1.14.0 termcolor-1.1.0 rm tensorflow-1.14.0-cp27-none-linux_aarch64.whl

It worked!

Now, let’s edit a test.py file:

` python
import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')

sess = tf.Session()

sess = tf.compat.v1.Session()
print(sess.run(hello))
`

bash python test.py Hello, TensorFlow!

We all have to start somewhere, don’t we?

TL;DR

bash sudo apt-get install -y python-setuptools python-h5py python-pip python-dev libatlas-base-dev curl -L "https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.14.0/tensorflow-1.14.0-cp27-none-linux_aarch64.whl" --output tensorflow-1.14.0-cp27-none-linux_aarch64.whll pip install wheel pip install tensorflow-1.14.0-cp27-none-linux_aarch64.whl cat >> ~/.bashrc export PATH=$PATH:/home/poddingue/.local/bin CTRL+D source ~/.bashrc

Installing Caffe on the Orange Pi 4B

Bruno Verachten — Wed, 22 Apr 2026 07:44:17 +0000

Originally published on June 22, 2020 at https://bruno.verachten.fr/2020/06/22/installing-caffe-on-the-orange-pi-4b/

Install dependencies

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev
libopencv-dev libhdf5-serial-dev protobuf-compiler cmake
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libopenblas-dev liblapack-dev libatlas-base-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

Download Caffe and modify configuration parameters

git clone https://github.com/BVLC/caffe
cd caffe
cp Makefile.config.example Makefile.config

Modify the parameters of Makefile.config Change #CPU_ONLY := 1 into CPU_ONLY := 1 Change ＃OPENCV_VERSION := 3 into OPENCV_VERSION := 3

Install Caffe

mkdir build
cd build
cmake ..
  Protobuf compiler version doesn't match library version 3.6.1

Okay… Let’s try a sudo apt install protobuf-compiler.

protoc --version
libprotoc 3.6.1
cmake ..
  Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS

My bad, let’s install it.

sudo apt install libhdf5-dev
cmake ..
  Could not find a package configuration file provided by "OpenCV" with any

Let’s install openCV, then…

sudo apt install libopencv-dev
cmake ..
[...]
-- Configuring done
-- Generating done
-- Build files have been written to: /home/poddingue/caffe/build

Next step, let’s build it for real. By the way, kiddoes, don’t try this -j6 at home unless you have a good heatsink… Or the machine will stop/hang up/reboot without prior alert.

make all -j6
make pycaffe -j6
make test -j6
make runtest -j6

Run a simple example

Run the caffe example — mnist instance: Mnist is a handwritten digital library. Originally used for handwritten numeral recognition on checks, and it is now the DL’s starting exercise library. The special model for mnist identification is Lenet, which is the earliest CNN model. The training samples of mnist data are 60,000 units, and the test samples are 10,000 units. Each sample is a black and white picture of size 28*28, and the handwritten number is 0-9, so it is divided into 10 categories.

Download the mnist data first

sh data/mnist/get_mnist.sh
cd data/mnist

There are four files in the directory after running the previous command:

ll
total 53676
-rwxr-xr-x 1 poddingue poddingue      408 Jun 22 16:55 get_mnist.sh
-rw-r--r-- 1 poddingue poddingue  7840016 Jul 21  2000 t10k-images-idx3-ubyte # Training set samples
-rw-r--r-- 1 poddingue poddingue    10008 Jul 21  2000 t10k-labels-idx1-ubyte # Corresponding annotation of training set
-rw-r--r-- 1 poddingue poddingue 47040016 Jul 21  2000 train-images-idx3-ubyte # Testing set sample
-rw-r--r-- 1 poddingue poddingue    60008 Jul 21  2000 train-labels-idx1-ubyte # Corresponding annotation of testing set

Converting the data

These data cannot be used directly in Caffe, and need to be converted into LMDB data:

cd ../..
sh examples/mnist/create_mnist.sh
Creating lmdb...
I0622 17:37:06.109530 12691 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_train_lmdb
I0622 17:37:06.114603 12691 convert_mnist_data.cpp:88] A total of 60000 items.
I0622 17:37:06.114666 12691 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I0622 17:37:11.407480 12691 convert_mnist_data.cpp:108] Processed 60000 files.
I0622 17:37:12.192065 12698 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_test_lmdb
I0622 17:37:12.194784 12698 convert_mnist_data.cpp:88] A total of 10000 items.
I0622 17:37:12.198009 12698 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I0622 17:37:12.566424 12698 convert_mnist_data.cpp:108] Processed 10000 files.
Done.
ll examples/mnist/
total 136
-rw-r--r-- 1 poddingue poddingue  4520 Jun 22 16:55 convert_mnist_data.cpp
-rwxr-xr-x 1 poddingue poddingue   634 Jun 22 16:55 create_mnist.sh
-rw-r--r-- 1 poddingue poddingue   777 Jun 22 16:55 lenet_adadelta_solver.prototxt
-rw-r--r-- 1 poddingue poddingue   778 Jun 22 16:55 lenet_auto_solver.prototxt
-rw-r--r-- 1 poddingue poddingue  6003 Jun 22 16:55 lenet_consolidated_solver.prototxt
-rw-r--r-- 1 poddingue poddingue   871 Jun 22 16:55 lenet_multistep_solver.prototxt
-rw-r--r-- 1 poddingue poddingue  1738 Jun 22 16:55 lenet.prototxt
-rw-r--r-- 1 poddingue poddingue   886 Jun 22 16:55 lenet_solver_adam.prototxt
-rw-r--r-- 1 poddingue poddingue   790 Jun 22 16:55 lenet_solver.prototxt
-rw-r--r-- 1 poddingue poddingue   830 Jun 22 16:55 lenet_solver_rmsprop.prototxt
-rw-r--r-- 1 poddingue poddingue  2282 Jun 22 16:55 lenet_train_test.prototxt
-rw-r--r-- 1 poddingue poddingue  4814 Jun 22 16:55 mnist_autoencoder.prototxt
-rw-r--r-- 1 poddingue poddingue   451 Jun 22 16:55 mnist_autoencoder_solver_adadelta.prototxt
-rw-r--r-- 1 poddingue poddingue   423 Jun 22 16:55 mnist_autoencoder_solver_adagrad.prototxt
-rw-r--r-- 1 poddingue poddingue   466 Jun 22 16:55 mnist_autoencoder_solver_nesterov.prototxt
-rw-r--r-- 1 poddingue poddingue   433 Jun 22 16:55 mnist_autoencoder_solver.prototxt
drwxr--r-- 2 poddingue poddingue  4096 Jun 22 17:37 mnist_test_lmdb
drwxr--r-- 2 poddingue poddingue  4096 Jun 22 17:37 mnist_train_lmdb
-rw-r--r-- 1 poddingue poddingue 11948 Jun 22 16:55 readme.md
-rwxr-xr-x 1 poddingue poddingue   106 Jun 22 16:55 train_lenet_adam.sh
-rwxr-xr-x 1 poddingue poddingue   118 Jun 22 16:55 train_lenet_consolidated.sh
-rwxr-xr-x 1 poddingue poddingue  4517 Jun 22 16:55 train_lenet_docker.sh
-rwxr-xr-x 1 poddingue poddingue   115 Jun 22 16:55 train_lenet_rmsprop.sh
-rwxr-xr-x 1 poddingue poddingue   101 Jun 22 16:55 train_lenet.sh
-rwxr-xr-x 1 poddingue poddingue   120 Jun 22 16:55 train_mnist_autoencoder_adadelta.sh
-rwxr-xr-x 1 poddingue poddingue   119 Jun 22 16:55 train_mnist_autoencoder_adagrad.sh
-rwxr-xr-x 1 poddingue poddingue   120 Jun 22 16:55 train_mnist_autoencoder_nesterov.sh
-rwxr-xr-x 1 poddingue poddingue   117 Jun 22 16:55 train_mnist_autoencoder.sh

The examples/mnist/create_mnist.sh has generated two folders in the examples/mnist/ directory, namely mnist_train_lmdb and mnist_test_lmdb after the LMDB data conversion succeeded. The data.mdb and lock.mdb stored in the folder are the running data we need.

LevelDB Data

If you want to run leveldb data, run the program under examples/siamese/ folder.

 ./examples/siamese/create_mnist_siamese.sh
Creating leveldb...
Done.
ll examples/siamese/
total 204
-rw-r--r-- 1 poddingue poddingue   4368 Jun 22 16:55 convert_mnist_siamese_data.cpp
-rwxr-xr-x 1 poddingue poddingue    617 Jun 22 16:55 create_mnist_siamese.sh
-rw-r--r-- 1 poddingue poddingue 158921 Jun 22 16:55 mnist_siamese.ipynb
-rw-r--r-- 1 poddingue poddingue   1488 Jun 22 16:55 mnist_siamese.prototxt
-rw-r--r-- 1 poddingue poddingue    810 Jun 22 16:55 mnist_siamese_solver.prototxt
drwxr-xr-x 2 poddingue poddingue   4096 Jun 22 17:39 mnist_siamese_test_leveldb
drwxr-xr-x 2 poddingue poddingue   4096 Jun 22 17:39 mnist_siamese_train_leveldb
-rw-r--r-- 1 poddingue poddingue   4907 Jun 22 16:55 mnist_siamese_train_test.prototxt
-rw-r--r-- 1 poddingue poddingue   5949 Jun 22 16:55 readme.md
-rwxr-xr-x 1 poddingue poddingue    125 Jun 22 16:55 train_mnist_siamese.sh

Next modify the configuration file lenet_solver.prototxt

vi ./examples/mnist/lenet_solver.prototxt

As needed, set the maximum number of iterations at 100 and the last line, solver_mode, to CPU:

== The train/test net protocol buffer definition
net: "examples/mnist/lenet_train_test.prototxt"
== test_iter specifies how many forward passes the test should carry out.
== In the case of MNIST, we have test batch size 100 and 100 test iterations,
== covering the full 10,000 testing images.
test_iter: 100
== Carry out testing every 500 training iterations.
test_interval: 500
== The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
== The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75
== Display every 100 iterations
display: 100
== The maximum number of iterations
max_iter: 10000
== snapshot intermediate results
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
== solver mode: CPU or GPU
solver_mode: CPU

Finally, run this example:

root@OrangePi:~/caffe# time sh examples/mnist/train_lenet.sh
I0110 06:28:06.117972 25078 caffe.cpp:197] Use CPU.
I0110 06:28:06.118988 25078 solver.cpp:45] Initializing solver from parameters:
test_iter: 100
test_interval: 500
base_lr: 0.01
display: 100
XUNLONG Software Co,. Ltd All rights reserved
36 www.orangepi.com
max_iter: 10000
lr_policy: "inv"
gamma: 0.0001
power: 0.75
momentum: 0.9
weight_decay: 0.0005
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
solver_mode: CPU
net: "examples/mnist/lenet_train_test.prototxt"
......
......
......
I0110 07:25:37.016746 25078 sgd_solver.cpp:284] Snapshotting solver state to binary proto file
examples/mnist/lenet_iter_10000.solverstate
I0110 07:25:37.146054 25078 solver.cpp:327] Iteration 10000, loss = 0.00200602
I0110 07:25:37.146195 25078 solver.cpp:347] Iteration 10000, Testing net (#0)
I0110 07:25:54.299634 25080 data_layer.cpp:73] Restarting data prefetching from start.
I0110 07:25:55.010598 25078 solver.cpp:414] Test net output #0: accuracy = 0.9913
I0110 07:25:55.010757 25078 solver.cpp:414] Test net output #1: loss = 0.0273203 (* 1 =
0.0273203 loss)
I0110 07:25:55.010777 25078 solver.cpp:332] Optimization Done.
I0110 07:25:55.010793 25078 caffe.cpp:250] Optimization Done.
real 57m48.984s
user 58m6.479s
sys 0m1.977s

⑦Operation conclusion According to the above results, the running time of the CPU is about 58 minutes, with an accuracy of about 99%.

🍊 ARM your continuous integration system with fruits! 🍌

Bruno Verachten — Wed, 22 Apr 2026 07:44:14 +0000

Originally published on January 11, 2021 at https://bruno.verachten.fr/2021/01/11/arm-your-continuous-integration-system-with-fruits/

The goal of this howto is to show you how to add your own ARM Gitlab runner to your Gitlab instance Runners herd.

Whaaaaaaaaaat?

Ok, let me introduce quickly the concepts.

Continuous integration allows the developers on a project to check that each change they made did not introduce any regression. Each commit will be followed by a build of their application, and by a run of their test suite. If every test runs still the same way after the commit, then all is fine. Gitlab is a web-based Git repository manager (with tons of features). You can use it directly online, install your own community-edition copy, or pay for top level support.

Today, we’ll be only interested in their continuous integration feature.

Starting Continuous Integration (i.e. CI) is really straightforward with Gitlab. Just create a .gitlab-ci.yml file in your repository hosted in Gitlab, populate it with the commands needed to build your application, and you’re good to go! 🎉 But wait… Where does that build and test execute? That’s where the runners enter. Runners are machines linked to Gitlab that will execute the builds. These can be real machines, virtual machines or containers. By default, you have access to shared runners on gitlab.com , or even on your company Gitlab instance.

These runners are tagged (🏷️) so that you can also tag your builds accordingly. If you need a runner that will build a Docker image, it could be tagged docker for example, and so would be your build step in your .gitlab.yml\file. Your builds can now easily be customized without ruining your own machine configuration… and even use Docker.

Why would you add a 🏃?

If you are using the main Gitlab shared instance, chances are that you will have to be patient to get the results of your build. The Shared Runner concept is really cool, as it allows to share the load, but … you’re not alone waiting in that queue… And the same could be true for your company’s Gitlab instance, as it will soon become popular. So you could add your own machine to the Gitlab Runners herd, just for your project. Pretty neat if you’re tired to wait for other people builds to finish. The problem is a new machine is not cheap. That’s way cool to have your own Gitlab Runner running in your office, but have a quick look at your pockets contents. Do you have enough cash to buy a new machine? Maybe not… But wait… In the introduction, I was talking about ARM Runners. Why would you need to add an ARM processor machine as a Gitlab Runner?

The reason why I chose to install an ARM Gitlab Runner

Well, if your code does not depend on a specific family of processors, there is no other cheaper alternative to build a new Gitlab Runner.

\$15 for an OrangePi Zero machine able to run Docker and Gitlab Runner can’t be beaten. And… if you want to develop for that kind of architecture (ARMV7, aarch64…), you will need that kind of runner. I am trying to port software to the ARM processor, and these ARM Gitlab Runners have proven to be really useful. I can develop on my favorite IDE (on a X86 machine for example), commit, and automagically, my modified code will be compiled and tested on an ARM machine. I have to admit all of this could also be done with X86 Runners, as QEMU is able to emulate ARM processors. And even the build of ARM Docker image can be done on X86 Gitlab Runners, thanks to the amazing work of guys at Balena.

But where’s the fun? And furthermore… We all have a Raspberry Pi taking layer after layer of dust. Who has not been bored after having his first led blink? Why not put it back to work?

Let’s go then for a cheap and fun solution.

Choose your preferred fruit

In this article, we’re not going to install Gitlab Runner on a 🍊 Pi, but on a Raspberry Pi (because we had one laying around), but this should work on any fruit (aka ARM Docker capable device), as 🍌 or 🍊.

image::http://www.banana-pi.org/\[\[🍌\] Pi M1, courtesy of Armbian](../assets/images/2021/02/04/bananapi.png)

This version of Gitlab Runner we’ll use needs Docker, that’s why we need a Linux distribution which supports Docker. We’re using an Hypriot distro because of the Raspberry Pi. We could have used Raspbian, but HypriotOS is the best Docker-oriented distribution for the Raspberry Pi. From downloading the image to the first docker \command will take only five minutes.

There is also a 64bits fork (as lots of Raspberry Pi distributions are not available for 64bits, that’s a nice plus). To me, that’s the first choice if you want to use Docker on a Raspberry Pi out of the box.

For all other devices, prefer Armbian if supported (like the Orange Pi Zero that can run as a Gitlab Runner too). It’s really easy to install Docker on this distribution.

First of all, let’s download the latest Hypriot distro.

Downloading and installing the Hypriot Distro

Go to the Hypriot download page and choose the latest distro.

Current image available on 02/04/2021 is Version 1.12.3 : https://github.com/hypriot/image-builder-rpi/releases/download/v1.12.3/hypriotos-rpi-v1.12.3.img.zip

On Windows

Next go to your Download folder and extract the downloaded zip file by right-clicking on the file and then clicking on 7-zip → extract here…. if you use 7-zip. Whatever your unzipping tool may be, right-click and choose the right option to uncompress the file. After 7-zip is finished with extracting you will have a file with a .img extension.

On OsX

It’s almost the same as for Windows, except that the tool could be Keka.

On Linux

Just use 7z.

Writing the image

In the next step we will flash this file onto your SD card. We have to use the finest piece of sofware able to write CORRECTLY and CHECK the written image (i.e. Etcher) on any platform. Current version is 1.4.4.

Too bad it does not exist for armv7. I do insist on the use of Etcher, because lots of supposed Linux bugs are because of a badly written image, or to a faulty SDCard. Etcher will take care of these issues. If possible, do not use any USB Dongle or gizmo to host the SDCard, prefer your laptop own SDCard reader. And for the SDCard, prefer a REAL and GENUINE SDCard.

How can you differentiate a real one from a bad copy? Well, you can’t, or not easily… Anyway, unzip the image, and write it with Etcher to your SDCard. In the latests versions of Etcher, you don’t even need to unzip the image file beforehand.

Upon completion, close Etcher and eject the SD card before removing it from its slot. In order to eject the SD card, use the secure eject function in the menu next to your system clock. This is usually in the very bottom right corner of your screen.

Now it’s time to start your Raspberry Pi with the SD card

Boot your Raspberry Pi

Connect the Raspberry Pi to your local network via an Ethernet cable
put the SD card into the designated slot
plug in the power adapter

After finishing the last step your Raspberry Pi will boot and the LEDs should start blinking. The very first boot will take one to three minutes as the file system will be resized.

Find its IP on the network

After booting, you can find the Raspberry Pi at your network with a simple ping black-pearl.local \— no more searching for IP addresses required thanks to the integrated Avahi service discovery.

https://gist.githubusercontent.com/gounthar/754f2988dc9ee643ea1d8be9a9c42223/raw/5437d39cdb07ab6bfb536ae468b483d8c197a29a/ssh%2520pirate.txt

As you can see in the following ifconfig command, the MAC Address of the Raspberry always starts with B8. That could prove useful if ever this first method does not work.

https://gist.githubusercontent.com/gounthar/ceb2680f38e613289f4e55b9e8c4f819/raw/0e7888a483783a92eba88f5d1f9996a47b54cea3/ifconfig.txt

So, what shall we do if the very first ping/ssh `does not work with `black-pearl? Well, as the Raspberry Pis MAC Address always start with B8 we can use nmap and keep only the IPs whose attached MAC Address sport a B8(yes, it’s gross). If your network starts at 10.XX.30.XX `and ends at `10.XX.50.XX, ip_start would be 30 and ip_end would be 50.

https://gist.githubusercontent.com/gounthar/2b4372abd7a623f7cf670575a76026d1/raw/e27c56e792fdf73652b4b53d07fa6bb91df1b7c7/nmap.txt

This will give you a list of Raspberry IPs on which you will then be able to try to connect with

Machine name and user password

The default user “pirate” (password “hypriot”) can be changed or removed before the first boot. Everything happens in /boot/user-data. You can add your public SSH key, disable password logins and specify a different user account before booting your Raspberry Pi. WiFi can be customized and enabled to have Docker up and running through the air without attaching a keyboard and monitor. How cool is that? You can also change the name of the machine later on with sudo raspi-config. The changes will have to be made in Network Options/hostname.

Once it’s done, reboot and enjoy your new machine name:

https://gist.githubusercontent.com/gounthar/596bc88f23d98de76bb70b11eb66e47b/raw/13f2210c233b6e1dc4d8da31f72e9063cfd18271/ssh%2520pirate%2520arm%2520runner.txt

Proxy settings

If you’re like me, stuck behind a corporate proxy, you will have to set a few things before being able to update your machine and run Docker.

Package manager

Edit /etc/apt/apt.conf.d/10proxy and add your proxy (create it if it doesn’t exist yet):

https://gist.githubusercontent.com/gounthar/e3288a613845b18aee3fe1098422d59f/raw/2eb3c68c8f2b3473902763e2f2b22be79918eea6/acquire.txt

A sudo apt-get update should prove that the configuration is working

https://gist.githubusercontent.com/gounthar/dc7829b564944c460e16a68870c13409/raw/afb35d6fe8731476befdf5f62f5513d4372f3f6a/apt%2520update.txt

While you’re there, update the machine with sudo apt-get upgrade to have the freshest Docker environment available.

Docker

If you try to download a Docker image with docker pull … it will fail because Docker doesn’t use the same proxy settings as apt.

https://gist.githubusercontent.com/gounthar/8384bb56d7ef8a9380c1f0d43013162e/raw/4f1f2aa8837966fe040e88c9e5c1aefc48d9f17a/docker%2520pull.txt

So you will have to create a few directories and files to let Docker know it should use your proxy:

https://gist.githubusercontent.com/gounthar/67dfa1f13e3d3064a02d17cb3ed9a51d/raw/3bb338cd496e0032f2472178f23fe96416eee0a2/docker%2520http%2520proxy.txt

/etc/systemd/system/docker.service.d/https-proxy.conf will contain:

https://gist.githubusercontent.com/gounthar/d04bb35e54b87f1a97e7ef6493f746a3/raw/2af846f014e46c76df96f22f0cbdcbe70fd35926/proxy%2520conf.txt

The http `version of this file will have the same proxy settings (just remove the `S of HTTPS). Once you’re done with the modifications, you will have to tell Docker to restart to take this into account:

https://gist.githubusercontent.com/gounthar/107de82adde721e3f7803778b3e08ffb/raw/c0d8670a60efe2585880aaf07a11e398ee17d230/restart%2520docker

Then you’ll be able to check that your modifications are correct:

https://gist.githubusercontent.com/gounthar/2b1df86210ee515b1c074e7140d29b35/raw/bad834d6d2c23387ed8c5097fec0a52d29805056/properties.txt

Date

Check that the date on your machine is correct by issuing date in the terminal. If it’s not correct, you will have some troubles with getting a Docker image. So, if the date/time are not right, you can set them up with:

https://gist.githubusercontent.com/gounthar/ecc647cb01fe16bcd2f439d492b1e663/raw/15c97e75b7f8164a92f19dfb834fc9a6381fea9c/date.txt

Gitlab Runner

You already know Gitlab Runner, I’m sure you are already using some shared runners with your CI on your company’s gitlab. Most of the time, you don’t need to setup your own, but as there aren’t that many ARM gitlab runners for the time being… Gitlab Runner has been ported to ARM in 2017 by Mauricio Ugaz as it wasn’t available, and everyone needs gitlab-runner on Docker running on ARM, am I right?

Note: there are two versions available. Alpine and Ubuntu. As Alpine is the default image, when you run docker pull klud/gitlab-runner Alpine version will be downloaded.

About the image

This image is built for ARM devices, based on the official repo of the GitLab Runner. That’s a fantastic work that made my daily work really more enjoyable.

Overview

Runner container setup

You need to mount a volume into the gitlab-runner container in order to share the configuration. This could be done this way:

https://gist.githubusercontent.com/gounthar/925bd7f20136b68b7a1bf337b31ff9ac/raw/4a48f4353261d247e0696ef581cd829bc44242ad/docker%2520run%2520txt

As we want to use the Docker executor, it is necessary to mount the Docker socket this way. Don’t forget to create /etc/gitlab-runner \directory ahead of time:

https://gist.githubusercontent.com/gounthar/9c76e1523d04a29d490703fe68c32793/raw/79db3c2dd7f67d1083c77a2ac1109fc6d9b53e50/docker%2520run%2520II.txt

Register runner

Once the container is up and running, you can then register the runner on your GitLab server. Go to your project on your Gitlab instance, click on Settings, then on CI/CD, and click on Expand next to Runner settings. Scroll down to Setup a specific Runner manually, and note the information regarding the URL and the token.

Specify the following URL during the Runner setup: https://gitlab.com/.

Use the following registration token during setup: the_token_that_is_yours_and_yours_alone

So once you have the information you need (URL and token), you can then do this:

https://gist.githubusercontent.com/gounthar/968c840f9a5ef8ee98edfbd6d6154fff/raw/3ea52f940b92282d7fa97ebc5e8ab0ff3fb6219e/register.txt

Unfortunately, with the latest versions of Docker from 19.03, you may get this message.

With Docker 19.03, TLS is enabled by default, to use TLS you need to update the GitLab Runner configuration so that the certificates are shared between the service and build container. To do this, update your config.toml so that it looks like that.

Let’s check if this new runner can now be seen. Refresh your CI/CI settings, expand Runner settings, and go down. You should now be able to see your brand new Gitlab-ci runner proudly sporting its ARM tag! 🤩

<!--Image for post-→

Congrats!

You could also do it this way:

https://gist.githubusercontent.com/gounthar/d6ce6108e3f25a11a8c70bc74ab65e97/raw/aaa4af34343124e9d3dbed731cffd1dde6c33243/register%2520II.txt

So now you have your own ARM runner, ready to work.

Tip

For my case, I am using the standard image to build my Docker images with this Runner, and it works just fine. Dockerfiles and info about Docker in Docker images for ARM are here.

Troubleshooting

In case you’re using docker in docker in the runner, you may experience some issues

https://gist.githubusercontent.com/gounthar/7fd4ce41ebdd8b79bf7dfe2472af7119/raw/a3d0b0ff8891609078695ea60de0fc69e7d76fa0/daemon.txt

So in order to address this issue you need to look within the config folder you mounted in the runner container. There is a config file which is/etc/gitlab-runner/config.toml\. You need to add two lines (regarding the privileged`and the `volumes)`and then restart the runner with `docker restart arm-runner. If you can’t find the file, it is maybe in ~/.runner/config.toml/; copy it in /etc/gitlab-runner after editing it and restart docker.

https://gist.githubusercontent.com/gounthar/31d41be5afb4b2c3ce0c66dfc1f00f33/raw/b08565f575646763ca3f71321a625e47f3c3619b/config.toml

So now you can enjoy your first ARM Gitlab build!

We now have an ARM machine sporting a recent kernel, running a recent version of gitlab-runner on top of a recent version of Docker.

We have reached the first level of achievement, having Gitlab runner running on the Pi thanks to Docker. Can we go any further? Could we build Docker images thanks to Gitlab runner on the Pi?

Building Docker Images

If you’re going to build images on this runner you can use the standard Docker image, as it is multi-arch now. Just type docker at “Please enter the Docker image” in the first method or in --docker- image "image:tag" with the second method. Sometimes, you may experience some troubles when using Docker in Docker. So in order to address this issue you need to look within the config folder you mounted in the runner container. There is a config file which is /etc/gitlab-runner/config.toml. You need to add two lines (regarding the privileged and the volumes)and then restart the runner with docker restart arm-runner.

If you can’t find the file, it is maybe in ~/.runner/config.toml/ copy it in /etc/gitlab-runner after editing it and restart docker.

It looks like we’ve reached the second level of achievement, being able to build Docker images through Gitlab runner.

What about another challenge?

Super tiny Runner

Let’s try it on another SBC, much, much cheaper, with half the amount of memory. This time, we’ll change fruit, let’s leave the raspberry aside and try with an Orange… Do you think it could work?

The Orange Pi Zero is one of my preferred SBCs, and almost the cheapest to this date.

You’ll get if for about 9€, and it can run Docker with a very recent kernel thanks to the Armbian distro. This distro is a fantastic community project which allows lots of SBCs to run recent Linux kernels with optimizations.

Armbian is a lightweight Debian based distribution specialized for ARM developing boards. It’s Compiled from scratch, It has powerful build and software development tools, and a vibrant community to boot. It supports lots of chips like the Allwinner H2+, H3, H5, Amlogic S905, Rockchip RK3288 and RK3399, Samsung Exynos 5422… If we go to the download section you can see there are tons of different cards that can run Armbian.

<!-- INSERT HERE DOWNLOAD SECTION VIDEO -→

It you choose OrangePi zero, you can see the specifications. image::https://www.youtube.com/watch?v=dX9I6xDn5AE\[\]

One of them is Docker, which means you can run Docker on this board with Armbian.

<!-- INSERT HERE ORANGE PI ZERO ARMBIAN VIDEO -→

Yes, you can run Docker on a 512MB ARM machine. You can even run gitlab runner on Docker on it. And even Docker on gitlab runner on Docker on it… Il you click on Docker, you will then be able to see the list of SBCs able to run Docker with Armbian.

<!-- INSERT HERE DOCKER MACHINES ARMBIAN VIDEO -→

Impressive, isn’t it?

Docker does not come preinstalled with Armbian, so you have to install it like that.

image::../assets/images/2021/02/04/docker install on armbian.png[docker install on armbian]Note that you could get Docker preinstalled, as you can build this distro by yourself as long as you have an Ubuntu machine available. The rest of the commands to run gitlab runner are strictly the same than for Raspberry Pi, so no need to repeat them. Let’s just have a look at the result, once the runner has been installed, registered and started. Feel like taking a look ?

Here is a build of a Docker image on an Orange Pi Zero. Isn’t it fantastic? We reached the third level of Achievement…

64 bits Runner

Some of the boards handled by Armbian or HypriotOS are ARM64 also called aarch64, or ARMV8. To tell you the truth, even the Raspberry Pi 3B is aarch64… It’s just that Raspbian does not handle it for compatibility reasons with their older boards. You can find 64 bits distros for the Raspberry Pi, like ArchLinux, DietPi, Hypriot… For other cards, Armbian does exist for ARM64, so does Docker. Until 10 months ago, we were kind of stuck, as gitlab-runner for aarch64 did not exist image::https://www.youtube.com/watch?v=dfP3wjcpveY\[\] <!-- INSERT HERE MERGE REQUEST VIDEO -→

The merge request had been running for almost 3 years until it got accepted. We are now able to use gitlab-runner on our 64 bits ARM machine, and without using Mauricio Ugaz fork, just the standard Docker image directly supplied by gitlab.

To sum up, we managed to setup a gitlab-ci Runner on a Raspberry Pi, and then on an OrangePi Zero . You now know how to transform your dusty Raspi into a gitlab- runner, even able to build Docker images. In the company I work for, we have also installed it on other SBCs (Odroid XU4, Nano Pi Fire3 ), and on a ThunderX2 just for fun. We are using them everyday, as they are cheap, disposable, fun to use, and have a very small footprint, so that we install them just about anywhere. I hope to have tickled your curiosity regarding these little beasts called SBCs, and that some of you will join the peaceful ARM rebellion.