DEV Community: ChienPang Lee

Copying fail

ChienPang Lee — Tue, 12 May 2026 14:30:03 +0000

Copy Fail (CVE-2026-31431) has started a fierce fire that's gone rampant in the Linux woods. Are we exploitable? The answer is almost certain because pretty much every actively maintained enterprise distribution has it. While folks are anxiously looking for a way to put it out, Dirty Frag and Copy Fail 2: Electric Boogaloo have caught up to the game, spilling oil on the flame. The correct way to address these is to upgrade your kernel to new versions that have the fixes merged. That's, however, easier said than done. Unless you're dealing with your own laptop where you're already on a relatively modern distro version, it's more complicated than "apt update && apt install && reboot". You have a service running on a certain vendor's Linux. The vendor needs weeks, likely months, to get you a release. You have your own company policy and concerns to schedule a widespread kernel rollout that would likely incur service interruptions. The reasons for procrastination go on and on, except that the risk stays high, as well as client inquiry. If the ideal solution is not going to happen soon, what're the mitigations we can do now? Be warned! The commonly-known kernel module suppression may not work, even if your overall services and OS operations could not be guaranteed. Here I'm proposing an addition to the short-term mitigation actions - monitor if your system is exploited by privilege escalations.

Take Copy Fail for example. It has to do with the Linux kernel's internal crypto API, managing functions like kTLS and IPsec. kernel’s internal cryptographic subsystem and may be impacted if the algif_aead module is disabled or restricted as a mitigation. Common features are KTLS (Knernel TLS), IPsec (Internet Protocol Security), Disk Encryption (dm-crypt/LUKS), User-space Crypto Offloading and Zero-Copy Networking: Functions like splice() and sendmsg().
One of the concrete recommendation to mitigate this CVE is to disable algif_aead module and restrict it from being loaded.

echo "install algif_aead /bin/false" > /etc/modprobe.d/disable-algif.conf
rmmod algif_aead 2>/dev/null || true

The fake module installation works on my Ubuntu 24.04 LTS (kernel 6.18-7) but not one another Linux platform of Centos9 (kernel 6.12.74-1).

Here is a working example with a happy ending on my laptop.

chien-pang@pop-os:~/Downloads$ cat /etc/modprobe.d/disable-copyfail.conf
install algif_aead /bin/true
chien-pang@pop-os:~/Downloads$ lsmod | egrep -i "algif|aead"
algif_hash             16384  1
algif_skcipher         12288  1
af_alg                 32768  6 algif_hash,algif_skcipher
chien-pang@pop-os:~/Downloads$ lsmod | grep algif_aead
chien-pang@pop-os:~/Downloads$ modprobe algif_aead ; echo $?
0
chien-pang@pop-os:~/Downloads$ lsmod | grep algif_aead
chien-pang@pop-os:~/Downloads$ ./copyFail30.py
Traceback (most recent call last):
  File "/home/chien-pang/Downloads/./copyFail30.py", line 36, in <module>
    while i<len(e):c(f,i,e[i:i+4]);i+=4
                   ^^^^^^^^^^^^^^^
  File "/home/chien-pang/Downloads/./copyFail30.py", line 30, in c
    a=s.socket(38,5,0);a.bind(("aead","authencesn(hmac(sha256),cbc(aes))"));h=279;v=a.setsockopt;v(h,1,d('0800010000000010'+\
'0'*64));v(h,5,None,4);u,_=a.accept();o=t+4;i=d('00');u.sendmsg([b"A"*4+c],[(h,3,i*4),(h,2,b'\x10'+i*19),(h,4,b'\x08'+i*3),]\
,32768);r,w=g.pipe();splice(f, w, o, offset_src=0);splice(r, u.fileno(), o)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
chien-pang@pop-os:~/Downloads$ id
uid=1000(chien-pang) gid=1000(chien-pang) groups=1000(chien-pang),4(adm),27(sudo),107(lpadmin)
chien-pang@pop-os:~/Downloads$ ls /root
ls: cannot open directory '/root': Permission denied

On the contrary, the same mitigation doesn't work on another system. Blacklisting it from grub menu did not help.

[cc2 ~]$ cat /proc/cmdline
BOOT_IMAGE=/Part2/bzImage root=UUID=571ee1af-1421-479d-845d-ea6b4f97292f ro net.ifnames=0 acpi=force intel_iommu=on amd_iommu=on iommu=pt console=ttyS0 console=tty0 initcall_blacklist=algif_aead_init
cc2 ~]# su - nova
Last login: Tue May 12 12:15:09 CST 2026 on pts/2
[nova@cc2 ~]$ id
uid=116(nova) gid=124(nova) groups=124(nova),123(libvirt),64055(qemu)
[nova@cc2 ~]$ ls /root ; echo $?
ls: cannot open directory '/root': Permission denied
2
[nova@cc2 ~]$ lsmod | egrep -i "algif|aead"
[nova@cc2 ~]$ /tmp/copyFail30.py
[cc2 /var/lib/nova]# id
uid=0(root) gid=124(nova) groups=124(nova),123(libvirt),64055(qemu)
[cc2 /var/lib/nova]# 
[cc2 /var/lib/nova]# ls /root ; echo $?
0

The exploitation works like a charm. This is also what makes "Copy Fail" notorious as the exploit script would work as is without further dependencies or complex preparations. Note the empty output of lsmod | egrep -i "algif|aead". The subsystem is still up though it's never loaded. It turns out the kernel has it compiled as built-in.

[cc2 /var/lib/nova]# grep  af_alg /proc/kallsyms | tail
ffffffff98d52d20 r __ksymtab_af_alg_release_parent
ffffffff98d52d2c r __ksymtab_af_alg_sendmsg
ffffffff98d52d38 r __ksymtab_af_alg_unregister_type
ffffffff98d52d44 r __ksymtab_af_alg_wait_for_data
ffffffff98d52d50 r __ksymtab_af_alg_wmem_wakeup
ffffffff99f33ff0 t __pfx_af_alg_init
ffffffff99f34000 t af_alg_init
ffffffff9a185a10 d __initcall__kmod_af_alg__884_1325_af_alg_init6
ffffffff9a35b420 t __pfx_af_alg_exit
ffffffff9a35b430 t af_alg_exit

AF_ALG symbols exist in kernel memory which makes the likely-hood high that kernel was compiled with CONFIG_AF_ALG=y.

In any way, you still have to worry about breaking existing features or functionalities even if the path of blacklisting modules works. And then what about "Dirty Frag" and "Copy Fail 2"?

Remediations takes long
Rolling-out new kernel is a painful operation
Mitigation recommendations may not work
Mitigation recommendations could break features

Given the challenges, what else can we do to reduce risks? My other piece of advice is to reduce attack surface but that'd be another story because they usually involve modifications of existing configurations or account setups.
What I find helpful is implementing a scanning tool (script) that can tell me if my system has obviously been compromised? Combining it with an event/alert notification system would immediately boost our confidence level while pending on remediations.

#!/usr/bin/bash                                                                                                                                                                                                                                                                                                              

Fmt() {
    printf "%-8s %-8s %-8s %-8s %-16s %-12s %s\n" \
           "${1:-PID}" "${2:-PPID}" "${3:-UID}" "${4:-EUID}" "${5:-CapEff}" "${6:-ParentUID}" "${7:-CMD}"
}

FmtHeader() {
    echo "------------------------------------------------------------------------------------------"
    Fmt
}
FmtContent() {o
    Fmt "$pid" "$ppid" "$uid" "$euid" "$capeff" "$parent_uid" "$cmd"
}

FmtHeader
for pid in /proc/[0-9]*; do
    pid=${pid#/proc/}

    status="/proc/$pid/status"
    cmdline="/proc/$pid/cmdline"

    [[ -r "$status" ]] || continue

    uid=$(awk '/^Uid:/ {print $2}' "$status")
    euid=$(awk '/^Uid:/ {print $3}' "$status")
    capeff=$(awk '/^CapEff:/ {print $2}' "$status")
    ppid=$(awk '/^PPid:/ {print $2}' "$status")

    # read parent UID safely                                                                                                                                                                                                                                                                                                 
    parent_uid="NA"
    if [[ -r "/proc/$ppid/status" ]]; then
        parent_uid=$(awk '/^Uid:/ {print $2}' "/proc/$ppid/status")
    fi

    # command name                                                                                                                                                                                                                                                                                                           
    if [[ -r "$cmdline" ]]; then
        cmd=$(tr '\0' ' ' < "$cmdline")
        cmd=${cmd:0:80}
    else
        cmd="?"
    fi

    if [[ "$capeff" != "000001ffffffffff" || "$parent_uid" == "NA" ]] ; then
        continue
    else
        parent_cmdline="/proc/$ppid/cmdline"
        pcmd=$(tr '\0' ' ' < "$parent_cmdline")
        pcmd=${pcmd:0:80}
    fi

    if [[ "$uid" == "0" && "$parent_uid" != "0" ]]; then
        FmtContent
        echo "[!] ALERT: process ($pid[$(id -nu $uid)]) spawned from non-root parent ($ppid[$(id -nu $parent_uid)] $pcmd $(ps -o etime $ppid | tail -1 | xargs))"
    elif [[ "$parent_uid" != "0" ]]; then
        FmtContent
        echo "[!] WARN: capabilities detected in non-root lineage (PID $pid[$(id -nu $uid)])"
    elif [[ "$uid" != "$parent_uid" && "$uid" == "0" ]]; then
        FmtContent
        echo "[!] INFO: UID escalation detected (PID $pid[$(id -nu $uid)])"
    fi
done

Saving it as /tmp/copying-fail-detect.sh that attempts to detect "Copy Fail" in action. This is an effective detector that looks into all running privileged processes that have their user IDs different from those of their parents. On the same system, the execution looks like:

[cc2 ~]# bash /tmp/copying-fail-detect.sh
------------------------------------------------------------------------------------------
PID      PPID     UID      EUID     CapEff           ParentUID    CMD
2988938  2988937  0        0        000001ffffffffff 116
[!] ALERT: process (2988938[root]) spawned from non-root parent (2988937[nova] python3 /tmp/copyFail30.py  36:16)
...(truncated)...

The alert provides us with a vital message and details in good depth. There is a process with pid 2988938, running root privilege. This process was invoked by its parent 2988937 whose user was nova. The actual command of the parent process was "python3 /tmp/copyFail30.py" and till now it has run this much time 36:16.

The detection has its short-coming of only checking the overall system process status at that second it's executed. On the flip side, it's a short and simple script. What else do you expect? Making it a scheduled job together with your notification or monitoring systems would serve you well when asked about your risk management and confidence level in the perspective of process privilege escalations.

Often times, I found the results of executing this "copying-fail-detect.sh" hilarious in that the very first wave of attackers are usually not actual hackers from the other end of the earth but the internal employees who found the exploit program and gave it a shot on the internal systems. Some of them, me included, are from security IT sector doing risk assessments and evaluating how much harm the CVE can do. Whatever conclusion they have reached, they were either too tired (rushing back home) or too excited (presenting to the team) and never came back to wipe their ass clean. As soon as the audit took place, the red records needed to be justified with embarrassments.

That'd better be my first thing in the morning tomorrow.

Jenkins nodes are offline

ChienPang Lee — Mon, 11 May 2026 08:58:43 +0000

Lab had a re-arrangement. We had decided to add a few new machines, sort the inventory and consolidate resources such as NIC cards, hard drives, memory dimms, etc. This also triggered a conscious decision of moving Jenkins service (the Jenkins master) to a new server which was new on pretty much every aspect, including OS. When all power was resumed and CI jobs kicked off, they hung.

I have a stage “hex” that was supposed to run some tests, estimated to be completed within 15 min. After the lab updates, a job was kicked off; it took around 7 hours without ending. As a result of that I blindly aborted it and gave it another shot, hoping it was only a glitch of CI flows. Another hour passed with the latest job while there’s still no good news. It’s a trouble-shooting task then.

Scrolling down to the bottom of job log, I saw:

Still waiting to schedule task
All nodes of label ‘bldsrv_private_200.13_centos9’ are offline

As I looked into the status of the mentioned node of that label, I ran into numerous unsuccessful executions:

Clicking one of the failed executions gave me further information:

The error of executing agent indicated something wrong with the docker container.

Docker Agent [localhost:5000/centos9-jail:latest on tcp://10.32.200.13:4243 ID 7076e0254ff86c26e7d9c6a8b3762fff5d6ae11401d96349e358570234942068]

Indeed we do use Docker Clouds. The configurations were double-checked as mentioned in previous post Jenkins CI the unseen building bricks of parallelization.

[root@bldsrv-200-13 jenkins]# systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
     Active: active (running) since Mon 2026-05-04 19:50:42 CST; 2h 2min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 2170586 (dockerd)
      Tasks: 179
     Memory: 837.3M
     CGroup: /system.slice/docker.service
             ├─2170586 /usr/bin/dockerd -H tcp://10.32.200.13:4243 -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock
             ├─2171095 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5000 -container-ip 172.17.0.2 -container-port 5000 -use-listen-fd
             ├─2171103 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5000 -container-ip 172.17.0.2 -container-port 5000 -use-listen-fd
             ├─2207093 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 3128 -container-ip 172.17.0.3 -container-port 3128 -use-listen-fd
             ├─2207101 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 3128 -container-ip 172.17.0.3 -container-port 3128 -use-listen-fd
             ├─2207395 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.4 -container-port 22 -use-listen-fd
             ├─2207403 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32768 -container-ip 172.17.0.4 -container-port 22 -use-listen-fd
             ├─2207411 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32769 -container-ip 172.17.0.4 -container-port 443 -use-listen-fd
             ├─2207420 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32769 -container-ip 172.17.0.4 -container-port 443 -use-listen-fd
             ├─2207428 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32770 -container-ip 172.17.0.4 -container-port 6900 -use-listen-fd
             └─2207437 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32770 -container-ip 172.17.0.4 -container-port 6900 -use-listen-fd

May 04 21:52:56 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:56.996976118+08:00" level=info msg="Container failed to exit within 10s of signal 37 - using the force" container=c3dfb867eb11eebeecc524654357fbdaa9e7ffef1192c603a9e5ee2e45da887e
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.010293297+08:00" level=info msg="ignoring event" container=deba996363cb8b8e3fb6fa29d6a5fb6546bd4cd69f38ffdca4ea25db8cf14467 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.020578385+08:00" level=info msg="Container failed to exit within 10s of signal 37 - using the force" container=efe16ccd188d15c25840ec693db96fdd09e519fb3ee1795b0f96e364f5df29c6
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.029078650+08:00" level=info msg="ignoring event" container=a569e6c24e8d3080a8205583587ea84137a1988126181029e659e4cb263b48c2 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.062594392+08:00" level=info msg="ignoring event" container=c3dfb867eb11eebeecc524654357fbdaa9e7ffef1192c603a9e5ee2e45da887e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:57 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:57.071636945+08:00" level=info msg="ignoring event" container=efe16ccd188d15c25840ec693db96fdd09e519fb3ee1795b0f96e364f5df29c6 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 04 21:52:59 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:59.609183494+08:00" level=warning msg="Error getting v2 registry: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:52:59 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:52:59.609542011+08:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:53:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:53:09.611275126+08:00" level=warning msg="Error getting v2 registry: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:53:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:53:09.611553781+08:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"

Docker daemon was running, listening on correct port which could be connected by Jenkins Clouds config. Note there were errors in the status:

May 04 21:56:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:56:09.617075720+08:00" level=warning msg="Error getting v2 registry: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
May 04 21:56:09 bldsrv-200-13 dockerd[2170586]: time="2026-05-04T21:56:09.617843533+08:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"

For Jenkins Clouds to work, it relies on a docker image that has to be pulled and run. In our case, we organized our local image and push it into a local registry.

[root@bldsrv-200-13 jenkins]# docker images
REPOSITORY                            TAG       IMAGE ID       CREATED        SIZE
localhost:5000/centos9-jail           latest    3b5e86b7aa54   9 hours ago    5.84GB
registry                              2         26b2eb03618e   2 years ago    25.4MB

[root@bldsrv-200-13 jenkins]# docker ps | grep registry
8fccbf02c57e   registry:2                           "/entrypoint.sh /etc…"   10 days ago          Up 2 hours          0.0.0.0:5000->5000/tcp, [::]:5000->5000/tcp

We did have our jail image, tagged ‘localhost:5000/centos9-jail’ Meanwhile, the “registry” service was running. Let’s see if our jail image was actually pushed into ‘registry’.

[root@bldsrv-200-13 jenkins]# curl http://localhost:5000/v2/_catalog
{"repositories":["app-jail","centos9-jail"]}

Right on! It’s there, sitting the the belly of registry service. Next question. Can the image be used?

[root@bldsrv-200-13 ~]# docker run --rm -ti localhost:5000/centos9-jail cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Looked all good. As a matter of fact, Jenkins could also pull and run the image and it’s been attempting to do so, as shown below

root@bldsrv-200-13 jenkins]# docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED              STATUS              PORTS                                                                                                                                   NAMES
f771b48b1fb6   localhost:5000/centos9-jail:latest   "supervisord"            6 seconds ago        Up 5 seconds        22/tcp, 443/tcp, 6900/tcp                                                                                                               inspiring_noether
ed50f88e0e1e   localhost:5000/centos9-jail:latest   "supervisord"            16 seconds ago       Up 15 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               jovial_hawking
3c6cce9c417e   localhost:5000/centos9-jail:latest   "supervisord"            26 seconds ago       Up 25 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               boring_liskov
21ceedb00793   localhost:5000/centos9-jail:latest   "supervisord"            36 seconds ago       Up 35 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               musing_rubin
0d60e410ad0c   localhost:5000/centos9-jail:latest   "supervisord"            46 seconds ago       Up 45 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               vigilant_ptolemy
083a0201dc5b   localhost:5000/centos9-jail:latest   "supervisord"            56 seconds ago       Up 55 seconds       22/tcp, 443/tcp, 6900/tcp                                                                                                               flamboyant_benz
3271582c18d4   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               heuristic_elion
72372f1a4869   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               optimistic_chebyshev
0f87f9209861   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               fervent_mayer
2542392eacd1   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               admiring_satoshi
0eef81092693   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               nice_hellman
bed3885c4c43   localhost:5000/centos9-jail:latest   "supervisord"            About a minute ago   Up About a minute   22/tcp, 443/tcp, 6900/tcp                                                                                                               vigorous_haibt
bf755d1e458d   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               stupefied_hawking
99df541b4c70   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               musing_jackson
0a0737b47f4c   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               eager_wescoff
19a58b56547a   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               stoic_mayer
7ac55893caaa   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               great_jang
1a507b140b72   localhost:5000/centos9-jail:latest   "supervisord"            2 minutes ago        Up 2 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               eloquent_bardeen
51276557fda2   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               busy_perlman
44847b167803   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               recursing_saha
979b9a80d68a   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               mystifying_hodgkin
5707acd16c39   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               intelligent_shtern
0ee9f3464321   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               exciting_mcclintock
7121a36795c5   localhost:5000/centos9-jail:latest   "supervisord"            3 minutes ago        Up 3 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               optimistic_diffie
b5a6bd81a5fe   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               upbeat_wilson
718a423f9a48   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               unruffled_boyd
bbddb26e1f10   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               angry_cohen
85ce490a3c81   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               adoring_chandrasekhar
961a1450dd79   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               sleepy_wing
dfc296d708f6   localhost:5000/centos9-jail:latest   "supervisord"            4 minutes ago        Up 4 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               distracted_pascal
ce7ed1b4ae53   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               kind_mclaren
241bb3ffa64f   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               gallant_ramanujan
d62e31687600   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               adoring_greider
1da6532c8f20   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               sleepy_black
cbf98eb3fcf4   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               goofy_sanderson
4e65eebe1e50   localhost:5000/centos9-jail:latest   "supervisord"            5 minutes ago        Up 5 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               serene_turing
36d67da7a5f2   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               zealous_bose
9a286435dc64   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               inspiring_franklin
2e91567b689e   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               clever_swanson
f42d26c35459   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               jolly_wozniak
aae69071cf05   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               sweet_swirles
66d94d544711   localhost:5000/centos9-jail:latest   "supervisord"            6 minutes ago        Up 6 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               stoic_mclaren
2b84df260fd8   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               confident_agnesi
d93c712cc0db   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               flamboyant_goodall
3c82531ce7c1   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               mystifying_newton
ba44f12b9a35   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               eager_gauss
8b8518ed60da   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               nervous_pike
4e99a80bb0f0   localhost:5000/centos9-jail:latest   "supervisord"            7 minutes ago        Up 7 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               cool_hypatia
d76c8e79f357   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               kind_curie
569d813a33e3   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               relaxed_darwin
f8e51f13c4ce   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               lucid_visvesvaraya4
33aee7bf5b58   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               nice_lovelace
8d93a012dbf5   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               magical_curie
90c14aebb7ee   localhost:5000/centos9-jail:latest   "supervisord"            8 minutes ago        Up 8 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               cool_mirzakhani
f083bbad20f3   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               thirsty_faraday
087195cae4ad   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               suspicious_boyd
f7bdc9eb5218   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               lucid_germain
24a2e44056e0   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               gracious_vaughan
65f0a0f24e30   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               wizardly_proskuriakova
a23734fa562c   localhost:5000/centos9-jail:latest   "supervisord"            9 minutes ago        Up 9 minutes        22/tcp, 443/tcp, 6900/tcp                                                                                                               musing_cannon
a249f9cd2e1c   localhost:5000/centos9-jail:latest   "supervisord"            10 minutes ago       Up 10 minutes       22/tcp, 443/tcp, 6900/tcp                                                                                                               xenodochial_greider
81ab4f2c90f7   localhost:5000/centos9-jail:latest   "supervisord"            10 minutes ago       Up 10 minutes       22/tcp, 443/tcp, 6900/tcp                                                                                                               agitated_poitras
a100b4aef1bf   localhost:5000/centos9-jail          "/sbin/init"             2 hours ago          Up 2 hours          0.0.0.0:32768->22/tcp, [::]:32768->22/tcp, 0.0.0.0:32769->443/tcp, [::]:32769->443/tcp, 0.0.0.0:32770->6900/tcp, [::]:32770->6900/tcp   centos9_cubecos_chienpang_200.13
f9683335cda9   ubuntu/squid:latest                  "entrypoint.sh -f /e…"   2 hours ago          Up 2 hours          0.0.0.0:3128->3128/tcp, [::]:3128->3128/tcp                                                                                             squid
8fccbf02c57e   registry:2                           "/entrypoint.sh /etc…"   10 days ago          Up 2 hours          0.0.0.0:5000->5000/tcp, [::]:5000->5000/tcp                                                                                             registry
[root@bldsrv-200-13 jenkins]#

The numerous attempts of container processes matched what we have seen in the above snapshot of Jenkins agents. So far we had a direction that Jenkins Clouds did not work. The backend services, images, registry and Jenkins configurations all seemed to work. Moreover, Jenkins started the containers (many of them) but none could make him happy.

Without further clues, I turned my attention to Jenkins logs.

At this point, we finally knew what Jenkins did not like - the Java versions

Caused by: java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 65.0), this version of the Java Runtime only recognizes class file versions up to 61.0

We had a new server, with newly-installed OS (Ubuntu 24.04.4 LTS). Jenkins server itself is also a container service. I have organized it as one of our Makefile target that it’d pull latest jenkins image from Docker hub upon execution.

root@bldsrv-200-10:~# docker exec -ti jenkins ls -lt /usr/share/jenkins/ref/init.groovy.d/
total 0
root@bldsrv-200-10:~# docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED        STATUS        PORTS                                                                                          NAMES
d3bfdf795397   danieldent/nginx-ssl-proxy   "/init"                  11 hours ago   Up 11 hours   80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp                                                nginx-proxy
33b482e7e628   jenkins/jenkins:lts          "/usr/bin/tini -- /u…"   11 hours ago   Up 11 hours   0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp, 0.0.0.0:50000->50000/tcp, [::]:50000->50000/tcp   jenkins
dd02cbd4bf72   aheimsbakk/munin-alpine      "/usr/bin/dumb-init …"   5 weeks ago    Up 11 hours   0.0.0.0:80->80/tcp, [::]:80->80/tcp                                                            munin-server
d9734f81f8d5   registry:2                   "/entrypoint.sh /etc…"   5 weeks ago    Up 3 days     0.0.0.0:5000->5000/tcp, [::]:5000->5000/tcp                                                    registry
root@bldsrv-200-10:~# docker inspect --format ''  jenkins
2.555.1

root@bldsrv-200-10:~# docker exec -ti jenkins java --version
openjdk 21.0.10 2026-01-20 LTS
OpenJDK Runtime Environment Temurin-21.0.10+7 (build 21.0.10+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.10+7 (build 21.0.10+7-LTS, mixed mode)

And here is the culprit, the Java version of latest Jenkins image on https://hub.docker.com/r/jenkins/jenkins is 2.555.1, exactly what we have currently. However, the Java version 21 of Jenkins master does not match that of our agent (jail image Java version 17). See below in our worker nodes.

[root@bldsrv-200-13 jenkins]# docker run --rm -ti localhost:5000/centos9-jail:latest java --version
openjdk 17.0.18 2026-01-20 LTS
OpenJDK Runtime Environment (Red_Hat-17.0.18.0.8-2) (build 17.0.18+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.18.0.8-2) (build 17.0.18+8-LTS, mixed mode, sharing)

As the cause becomes clear so is the fix. Either keeping lowering the Java version in jail or bumping it up in the Docker image launched by Jenkins Clouds. For the develop wheel to move forward, the later is preferred.

RUN dnf install -y java-21-openjdk java-21-openjdk-devel
RUN alternatives --set java java-21-openjdk.x86_64

The final dockerfile was updated with an additional line to adhere it to version 21 because other packages, such as maven, in my relatively old Centos8 base image requres Java 17. We’d end up having multiple versions of Java SDK, not obvious which would be the default one.

Four days later, we had a green pipeline again. The impairment was cleared such that dev work could move forward. Agile enough? Not in my eyes. Tech enough? I’d say so.

Read the full technical article here: Jenkins nodes are offline

Jenkins CI the unseen building bricks of parallelization

ChienPang Lee — Mon, 11 May 2026 08:40:25 +0000

One of my Jenkins pipelines was set up and, without further scrutiny, it just worked. It ran multiple stages that took several hours and eventually ended in green — the kind of result that contributes to a bright, sunny day. Though it was just past lunchtime, I cracked open a bottle of Gösser and called it a day.

These small rewards in the middle of routine work feel deserved, especially when you’re the one making the calls. There’s a subtle but important connection between ownership and responsibility — when things succeed, you feel it; when they fail, you own that too.

But celebration often are followed by reflection. The next step. What could be improved?

Given our quick observation, one obvious area is reducing total pipeline execution time. That naturally leads to parallelization. Jenkins, our most senior and tireless employee ,“Jenkins”, has a firm grip on this as well. In a Jenkins Pipeline, parallel execution is achieved through the parallel block (in Groovy), allowing independent stages and steps (like system tests and end-to-end tests) to run simultaneously.

However, digging deeper into parallelization reveals that it’s not just a switch you flip and expect light. For parallel execution to be reliable and scalable, certain foundational bricks have to be in place. Two key bricks are:

Environment Reusability
Environment Isolation

In my pipelines are the followings stages: source → jail → build → unit → system → e2e → publish

Each stage depends on the outputs of previous one. This falls intuitively into reusability category. For example, source (code) is fetched once and reused multiple times later. Same thing goes to jail and build. The stages take place in sequence.

def BLDSRV = "bldsrv_" + BUILD_TYPE
lock("${BLDSRV}")
    node("${BLDSRV}") {
        stage('source') {
            echo "checkout source"
        }
        stage('jail') {
            echo "prepare build env. jail"
        }
        stage('build') {
            echo "build, compile and generate firmware"
        }
        stage('unit') {
            echo "run unit tests"
        }
        stage('system') {
            echo "run system tests"
        }
        stage('e2e') {
            echo "run e2e tests"
        }
        stage('publish') {
            echo "publish and deploy artifacts"
        }
    }
}

The stages enclosed within the node block would execute on the build server defined by the BLDSRV variable, which maps to nodes configured in Jenkins. Additionally, the lock block ensures that only one job at a time can occupy a given node (resource label), preventing interference from other jobs running concurrently on the same resource.

Besides, inside each stage we’d like steps to run in parallel. build, unit, system and e2e are good fits for implementing parallelization. Take e2e test as an example. It needs to use disk which is installed with previouly built firmware from system stage which in turn relies on the firmware from build stage and as well as green results of unit test. The goal of e2e is to boot from the installed firmware and perform tasks such as configuration and executing test suites against it.

This is where another key Jenkins feature steps up. The parallel block.

def BLDSRV = "bldsrv_" + BUILD_TYPE
lock("${BLDSRV}")
    node("${BLDSRV}") {
        stage('source') {
            echo "checkout source"
        }
        stage('jail') {
            echo "prepare fake root build env."
        }
        stage('build') {
            echo "build, compile and generate firmware"
        }
        stage('unit') {
            echo "run unit tests"
        }
        stage('system') {
            echo "run system tests"
        }
        stage('e2e') {
            def e2etest=[:]
            e2etest["test_suite1"] = {
                echo "run e2e test suite 1"
            }
            e2etest["test_suite2"] = {
                echo "run e2e test suite 2"
            }
            parallel e2etest
        }
        stage('publish') {
            echo "publish and deploy artifacts"
        }
    }
}

This basic pipeline would work in a quick demo but it’d fall short with our end to end testing goal — validating a fully-deployed, networked, configured cloud platform, consisting of multiple nodes, running the firmware(Operating System) produced in our build stage.

To address this, I leveraged another powerful capability, Jenkins Clouds. Underneath the surface it is backed by Docker containerization technology.

[root@localhost workspace]# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target nss-lookup.target docker.socket firewalld.service containerd.service time-set.target
Wants=network-online.target containerd.service
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
# ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecStart=/usr/bin/dockerd -H tcp://10.32.200.13:4243 -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutStartSec=0
RestartSec=2
Restart=always

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process
OOMScoreAdjust=-500

[Install]
WantedBy=multi-user.target
[root@localhost workspace]# cat /etc/docker/daemon.json
{
  "storage-driver": "overlay2",
  "data-root": "/var/lib/docker",
  "features": {
    "buildkit": true
  },
  "exec-opts": ["native.cgroupdriver=cgroupfs"]
}

[root@localhost workspace]# systemctl status docker -l
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
     Active: active (running) since Fri 2026-04-24 15:04:23 CST; 5 days ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 987664 (dockerd)
      Tasks: 887
     Memory: 10.9G
     CGroup: /system.slice/docker.service
             ├─ 987664 /usr/bin/dockerd -H tcp://10.32.200.13:4243 -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock

The above are the docker settings on worker node. Jenkins Clouds needs to be able to communicate and control it. Make sure Jenkins sees and can talk to your docker daemon of worker node by having a successful Test Connection.

With these in place, our pipeline evolves into:

def BLDSRV = "bldsrv_" + BUILD_TYPE
def CLOUD_NODE = BLDSRV + "_" + PLATFORM
lock("${BLDSRV}")
    node("${BLDSRV}") {
        stage('source') {
            echo "checkout source"
        }
        stage('jail') {
            echo "prepare fake root build env."
        }
        stage('build') {
            echo "build, compile and generate firmware"
        }
        stage('unit') {
            echo "run unit tests"
        }
        stage('system') {
            echo "run system tests"
        }
        stage('e2e') {
            def e2etest=[:]
            e2etest["test_suite1"] = {
                node("${CLOUD_NODE}") {
                    echo "run e2e test suite 1"
                }
            }
            e2etest["test_suite2"] = {
                node("${CLOUD_NODE}") {
                    echo "run e2e test suite 2"
                }
            }
            parallel e2etest
        }
        stage('publish') {
            echo "publish and deploy artifacts"
        }
    }
}

Now, each e2e test suite runs in parallel within its own containerized environment. These environments are fully isolated, with separate networking and filesystems. At the same time, shared data (such as build artifacts) is made available through volume mapping, ensuring reusability. All we’ve gone through this far is, nonetheless, still not the full story of the challenges I had. My e2e test construct is huge. The bootable drive and required data drives themselves take 100+ GB. The initial configuration alone took long time, which is another piece I had to optimize and made it reusable. Some long running steps can be started and put into background while letting other executiions take off. Not to mention the management of locally-hosted Docker registry where localhost:5000/centos9-jail resides. The fun never really ends. But anyhow, it’s another time to have a toast.

Read the full technical article here: Jenkins CI the unseen building bricks of parallelization

Small footprint of a power editor in initramfs

ChienPang Lee — Mon, 04 May 2026 11:15:17 +0000

Have you ever found yourself trapped in a restricted root filesystem where every megabyte has to be weighed? For most people, this is a rarity, but when working with initramfs—the transient filesystem used by the Linux kernel during the boot process—space is everything.

A text mode editor is a vital tool for navigating filesystem, debugging problems and changing configurations. However, as a dedicated Emacs user, the standard lightweight alternatives like vi, nano, or pico is not in the favored list. In such an extreme environment where I from time to time have to handle critical boot issues, a working, light, and powerful editor is to me not just a vital tool but a luxury.

The Challenge: Why Not `emacs-nox`?

Navigating a filesystem without a proper editor is like living in a read-only world. While echo redirection and sed can handle simple tasks, complex configuration changes become a painful exercise in precision.

Normally, I would reach for emacs-nox (the full Emacs experience without the X11 overhead). But in an initramfs environment, adding emacs-nox and its dependencies can balloon the image size by over 100MB. For a system that needs to be lean and fast, that’s simply not an option.

The Solution: `mg` (Micro GNU Emacs)

mg is a microscopic editor that maintains Emacs keybindings while remaining incredibly lightweight. On a standard Ubuntu system, it’s a tiny footprint:

# apt show mg
Package: mg
Version: 20230501-1
Priority: optional
Section: universe/editors
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Harald Dunkel <harri@afaics.de>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 273 kB
Provides: editor
Depends: libbsd0 (>= 0.5.0), libc6 (>= 2.38), libtinfo6 (>= 6)
Homepage: https://homepage.boetes.org/software/mg/
Download-Size: 121 kB
APT-Manual-Installed: yes
APT-Sources: http://mirror.math.princeton.edu/pub/ubuntu noble/universe amd64 Packages
Description: microscopic GNU Emacs-style editor

On a standard Linux operating system like Ubuntu, it is as simple as

root@frad:~/Workspace/# # apt-get install mg

Nonetheless, we are dealing with a mimimized distroless rootfilesystem that doen't come with package managers of any sort. To get mg to work inside mini rootfs, assuming you're on a Ubuntu, install mg with the above. After installation, the binary can be located with

All that is still on host system. To get mg working in initramfs, we have to perform a manual port.

Step 1: Binary Placement

First, we locate the binary on the host and copy it into our target initramfs directory:

root@frad:~/Workspace/# # which mg
/usr/bin/mg
root@frad:~/Workspace/# install -p -D -m 0755 /usr/bin/mg ~/Workspace/img/mini_initramfs

Let take a peek how it'd work inside the rootfs

root@frad:~/Workspace/# chroot ~/Workspace/img/mini_initramfs
/ # echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/ # mg
/bin/sh: mg: not found
/ # ls -lt /bin/mg
-rwxr-xr-x    1 0        0           231080 Apr 20 06:48 /bin/mg
/ #

Invoking md command inside the mini initramfs came back with an error that "mg" was not found. Further commands confirmed that the binary was there and path was also correct. The installed mg turned out to be a dynamically-linked binary so the shared libraries also need to be copied.

Step 2: Resolving Dependencies

To fix this, we use ldd to identify the shared library dependencies:

root@frad:~/Workspace/# ldd /usr/bin/mg
    libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6
    libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    /lib64/ld-linux-x86-64.so.2

We must replicate this structure inside our initramfs:

Create directory structure and symlinks

root@frad:~/Workspace/# mkdir -p ~/Workspace/img/mini_initramfs/usr/lib/x86_64-linux-gnu
root@frad:~/Workspace/# ln -sf usr/lib ~/Workspace/img/mini_initramfs/lib
root@frad:~/Workspace/# ln -sf ../lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 ~/Workspace/img/mini_initramfs/lib64/
root@frad:~/Workspace/# ldd /usr/bin/mg | grep "=>" | awk '{print $3}' | xargs -i cp {} ~/Workspace/img/mini_initramfs{}

Step 3: Solving the "Terminal Panic"

After this, give it another try!

root@frad:~/Workspace/# chroot ~/Workspace/img/mini_initramfs
/ # mg
panic: Terminal setup failed
/ # echo $TERM
xterm

mg was launched with previous errors gone. But there appeared to be a new error about terminal panic. The terminal is "xterm". With some more digging it pointed to missing data/db of xterm capabilities. As such, I copied xterm pertinent data into rootfs

/bin/bash ~/Workspace/scripts/install ~/Workspace/img/mini_initramfs /usr/share/terminfo/x/xterm usr/share/terminfo/x

root@frad:~/Workspace/# chroot ~/Workspace/img/mini_initramfs
/ # ls -lt /usr/share/terminfo/x/
total 4
-rw-r--r--    1 0        0             3991 Apr  8  2024 xterm
/ # mg
/ #
/ # ls -lt /lib/x86_64-linux-gnu/ /lib64
lrwxrwxrwx    1 0        0                9 Apr 20 08:19 /lib64 -> usr/lib64

/lib/x86_64-linux-gnu/:
total 2652
-rwxr-xr-x    1 0        0           236616 Apr 20 08:19 ld-linux-x86-64.so.2
-rwxr-xr-x    1 0        0             3632 Apr 20 08:19 ld-linux-x86-64.so.2.debug
-rw-r--r--    1 0        0            80888 Apr 20 08:19 libbsd.so.0
-rwxr-xr-x    1 0        0          2125328 Apr 20 08:19 libc.so.6
-rw-r--r--    1 0        0            55536 Apr 20 08:19 libmd.so.0
-rw-r--r--    1 0        0           208328 Apr 20 08:19 libtinfo.so.6
/ # du -hsc /lib/x86_64-linux-gnu/ /lib64 /bin/mg
2.6M    /lib/x86_64-linux-gnu/
0   /lib64
228.0K  /bin/mg
2.8M    total
/ # mg /lib/x86_64-linux-gnu/
/ #

The Result

And there we go: a functional, Emacs-style editor with small footprint (2.8M in total) in a distroless, customized and size-critical filesystem. I'd play a mission-critical role when it comes to future trouble-shooting during bootstrapping, expecially when kernel panics.