DEV Community: Mathieu Kerjouan

rdomain, wireguard, packet filter and OpenBSD

Mathieu Kerjouan — Tue, 03 Nov 2026 09:00:00 +0000

rdomain is an interesting feature present on OpenBSD to create isolated routing domain. I rarely used this one - except for testing - but it was never deployed in production. In fact, it was pretty rare to have an OpenBSD router/firewall running outside of my own lab. A previous post was showing how to isolate a pool of workers and forward all their traffic through using onion routing using proxies. In this post, the workers traffic will be routed to a remote server using a wireguard tunnel. Both the firewall and the remote server are running on OpenBSD.

A picture is worse thousand of words, then, the infrastructure schema below should give you an idea of the network structure and the connections between the different devices. Not everything is presented though, because my network is a bit more complex than that, but this is a big picture.

Before doing anything, we should probably start to learn what is an rdomain and how to use it properly.

It's All About Isolation and Routing

rdomain appeared for the first time in OpenBSD 4.9. This goal was to offer a way to have isolated routing tables. By default, all routes are put in the rdomain 0, and it can lead to some issues when dealing with complex network. With an rdomain, an operator is able to create dynamically a new routing table isolated from the rest of the system, even if it is limited to 255 rdomain, it shold be enough for most of the use case.

Processes can be started in their own rdomain with the help of the route command and its exec subcommand without forgetting the -T flag. A way to check in which domain a process is in, is to use ps command with the rtable option. Let first checks the output of the ps command.

$ ps -o rtable,pid,ppid,%cpu,%mem,command
RTABLE   PID  PPID %CPU %MEM COMMAND
     0 59603 14917  0.0  0.0 -ksh (ksh)
     0 13501 59603  0.0  0.2 tmux: client (/tmp/tmux-1000/default) (tmux)
     0 68852 66599  0.0  0.0 -ksh (ksh)
     0 73792 66599  0.0  0.0 -ksh (ksh)
     0 75757 66599  0.0  0.0 -ksh (ksh)
     0 85484 75757  0.0  0.1 ps -o rtable
     0 47120 66599  0.0  0.0 -ksh (ksh)
     0 86889 66599  0.0  0.0 -ksh (ksh)

AS you can see, all processes are currently running in the default rdomain 0. To use another rdomain, the route command can be used, like it has been seen previously. Let check that by spawning a sleep in the background for 10 seconds in the rdomain 1.

$ route -T1 exec sleep 10 &
[1] 96416

$ ps -o rtable,pid,ppid,%cpu,%mem,command
RTABLE   PID  PPID %CPU %MEM COMMAND
     0 59603 14917  0.0  0.0 -ksh (ksh)
     0 13501 59603  0.0  0.2 tmux: client (/tmp/tmux-1000/default) (tmux)
     0 68852 66599  0.0  0.0 -ksh (ksh)
     0 73792 66599  0.0  0.0 -ksh (ksh)
     0 46155 73792  0.0  0.0 sleep 1
     0 75757 66599  0.0  0.0 -ksh (ksh)
     1 96416 75757  0.0  0.0 sleep 10
     0 26428 75757  0.0  0.1 ps -o rtable
     0 47120 66599  0.0  0.0 -ksh (ksh)
     0 86889 66599  0.0  0.0 -ksh (ksh)

The sleep process is correctly using the rtable 1, but what kind of differences are existing between all of those routing tables? Well, the default routing table is initialized during the startup of the system, but all other rdomain should be empty. Again, the route command can be used to show that or the netstat command can be used instead, it is offering more filtering options.

$ netstat -rnT0 -finet
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            192.168.123.254      UGS        9 10038979     -     8 re0
224/4              127.0.0.1          URS        0        0 32768     8 lo0
10.255.255.240/28  10.255.255.253     UCn        1        0     -     
...

An existing rdomain 1 is currently present on my test example. As you can see, the output returned by netstat is totally different than the one from the rdomain 1.

$ netstat -rnT1 -f inet
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            172.17.0.1         UGS        0    17202     -     8 wg0
10.255.255.240/28  10.255.255.254     UCn        1        0     -     4 pair1
10.255.255.253     fe:e1:ba:d5:62:48  UHLch      1        6     -     3 pair1
10.255.255.254     fe:e1:ba:d6:4a:44  UHLl       0        4     -     1 pair1
10.255.255.255     10.255.255.254     UHb        0        0     -     1 pair1
127.0.0.1          127.0.0.1          UHl        0      315 32768     1 lo1
...

The rdomain 2 does not exist though, and when trying to show its content, an error is returned by netstat. The same error is returned to any unitialized rdomains.

$ netstat -rnT2
netstat: routing table 2: No such file or directory

$ netstat -rnT3
netstat: routing table 3: No such file or directory

rdomains can be created when dealing with network interfaces and pseudo devices. For example, let created rdomain 2 and rdomain 3 by defining lo2 and lo3 loopback interfaces.

$ doas ifconfig lo2 create rdomain 2 alias 127.0.0.1 up

$ doas ifconfig lo2
lo2: flags=2008049<UP,LOOPBACK,RUNNING,MULTICAST,LRO> rdomain 2 mtu 32768
        index 26 priority 0 llprio 3
        groups: lo
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo2 prefixlen 64 scopeid 0x1a
        inet 127.0.0.1 netmask 0xff000000

$ doas ifconfig lo3 create rdomain 3 alias 127.0.0.1 up

$ doas ifconfig lo3
lo3: flags=2008049<UP,LOOPBACK,RUNNING,MULTICAST,LRO> rdomain 3 mtu 32768
        index 27 priority 0 llprio 3
        groups: lo
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo3 prefixlen 64 scopeid 0x1b
        inet 127.0.0.1 netmask 0xff000000

Both interfaces are totally isolated in their own address space, with its own routing table, this is why the same IP address (here 127.0.0.1) can be set. To extend the example, 2 vether interfaces can be created as well.

$ doas ifconfig vether2 create alias 10.0.0.1/24 rdomain 2 up

$ ifconfig vether2
vether2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 2 mtu 1500
        lladdr fe:e1:ba:d9:1b:6c
        index 30 priority 0 llprio 3
        groups: vether
        media: Ethernet autoselect
        status: active
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255

$ doas ifconfig vether3 create alias 10.0.1.1/24 rdomain 3 up

$ ifconfig vether3
vether3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 3 mtu 1500
        lladdr fe:e1:ba:da:b5:73
        index 31 priority 0 llprio 3
        groups: vether
        media: Ethernet autoselect
        status: active
        inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255

To interconnect those networks 2 pair interfaces can be created. A small subnet will be create to route the packets between both networks.

$ doas ifconfig pair2 create alias 172.31.255.253/30 rdomain 2 up

$ ifconfig pair2
pair2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 2 mtu 1500
        lladdr fe:e1:ba:de:a4:28
        index 35 priority 0 llprio 3
        groups: pair
        media: Ethernet autoselect
        inet 172.31.255.253 netmask 0xfffffffc broadcast 172.31.255.255

$ doas ifconfig pair3 create alias 172.31.255.254/30 rdomain 3 up 

$ ifconfig pair3
pair3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 3 mtu 1500
        lladdr fe:e1:ba:df:34:e5
        index 36 priority 0 llprio 3
        groups: pair
        media: Ethernet autoselect
        inet 172.31.255.254 netmask 0xfffffffc broadcast 172.31.255.255

The pair interfaces must be patched together.

$ doas ifconfig pair2 patch pair3

$ ifconfig pair2
pair2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 2 mtu 1500
        lladdr fe:e1:ba:de:a4:28
        index 35 priority 0 llprio 3
        patch: pair3
        groups: pair
        media: Ethernet autoselect
        status: active
        inet 172.31.255.253 netmask 0xfffffffc broadcast 172.31.255.255

$ ifconfig pair3
pair3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> rdomain 3 mtu 1500
        lladdr fe:e1:ba:df:34:e5
        index 36 priority 0 llprio 3
        patch: pair2
        groups: pair
        media: Ethernet autoselect
        status: active
        inet 172.31.255.254 netmask 0xfffffffc broadcast 172.31.255.25

It should now be possible to reach pair2 network from pair3 and vice versa. The best way to check that is to use the route exec command.

$ route -nT2 exec ping -c 3 172.31.255.254
PING 172.31.255.254 (172.31.255.254): 56 data bytes
64 bytes from 172.31.255.254: icmp_seq=0 ttl=255 time=0.615 ms
64 bytes from 172.31.255.254: icmp_seq=1 ttl=255 time=0.369 ms
64 bytes from 172.31.255.254: icmp_seq=2 ttl=255 time=0.382 ms

--- 172.31.255.254 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.369/0.455/0.615/0.113 ms

$ route -nT3 exec ping -c3 172.31.255.253
PING 172.31.255.253 (172.31.255.253): 56 data bytes
64 bytes from 172.31.255.253: icmp_seq=0 ttl=255 time=0.352 ms
64 bytes from 172.31.255.253: icmp_seq=1 ttl=255 time=0.364 ms
64 bytes from 172.31.255.253: icmp_seq=2 ttl=255 time=0.378 ms

--- 172.31.255.253 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.352/0.364/0.378/0.011 ms

$ route -nT3 exec ping -c1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
ping: sendmsg: Can't assign requested address
ping: wrote 10.0.0.1 64 chars, ret=-1

The 10.0.0.1/24 and 10.0.1.0/24 networks are not reachable yet, because no routes have been configured. Let fix that.

$ doas route -nT2 add 10.0.1.0/24 172.31.255.254
add net 10.0.1.0/24: gateway 172.31.255.254

$ netstat -rnfinet -T2
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
10.0.0/24          10.0.0.1           UCn        0        0     -     4 vether2
10.0.0.1           fe:e1:ba:d9:1b:6c  UHLl       0        8     -     1 vether2
10.0.0.255         10.0.0.1           UHb        0        0     -     1 vether2
10.0.1/24          172.31.255.254     UGS        0        0     -     8 pair2
127.0.0.1          127.0.0.1          UHl        0        0 32768     1 lo2
172.31.255.252/30  172.31.255.253     UCn        1        0     -     4 pair2
172.31.255.253     fe:e1:ba:de:a4:28  UHLl       0       10     -     1 pair2
172.31.255.254     fe:e1:ba:df:34:e5  UHLch      1       13     -     3 pair2
172.31.255.255     172.31.255.253     UHb        0        0     -     1 pair2

$ doas route -nT3 add 10.0.0.0/24 172.31.255.253
add net 10.0.0.0/24: gateway 172.31.255.253

$ netstat -rnfinet -T3
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
10.0.0/24          172.31.255.253     UGS        0        2     -     8 pair3
10.0.1/24          10.0.1.1           UCn        0        0     -     4 vether3
10.0.1.1           fe:e1:ba:da:b5:73  UHLl       0        0     -     1 vether3
10.0.1.255         10.0.1.1           UHb        0        0     -     1 vether3
127.0.0.1          127.0.0.1          UHl        0        0 32768     1 lo3
172.31.255.252/30  172.31.255.254     UCn        1        0     -     4 pair3
172.31.255.253     fe:e1:ba:de:a4:28  UHLch      1        7     -     3 pair3
172.31.255.254     fe:e1:ba:df:34:e5  UHLl       0       14     -     1 pair3
172.31.255.255     172.31.255.254     UHb        0        0     -     1 pair3

After having added these routes, both network are available.

$ route -nT3 exec ping -c3 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=255 time=0.476 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=255 time=0.371 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=255 time=0.272 ms

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.272/0.373/0.476/0.083 ms

$ route -nT2 exec ping -c3 10.0.1.1
PING 10.0.1.1 (10.0.1.1): 56 data bytes
64 bytes from 10.0.1.1: icmp_seq=0 ttl=255 time=0.477 ms
64 bytes from 10.0.1.1: icmp_seq=1 ttl=255 time=0.355 ms
64 bytes from 10.0.1.1: icmp_seq=2 ttl=255 time=0.271 ms

--- 10.0.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.271/0.368/0.477/0.085 ms

Managing one or two rdomain can easily be done with packet-filter, but if you have more than that, it could become rapidly a complex task. While looking for more examples and use cases from the *BSD users, many authors were recommending using packet-filter anchors to make the configuration easier to understand and to control. Let check that, if you don't know, packet-filter configuration file is located in /etc/pf.conf, but before editing this file, two new pflog interfaces will be required to log the packets from these domains.

$ doas ifconfig pflog2 create up

$ doas ifconfig pflog3 create up

The rdomain 2 ruleset will be managed by the rd2 anchor and the rdomain 3 will be managed by the rd3 anchor. Both of them will only have one rule for now, to log all packets in transit respectively to pflog2 and pflog3 pseudo interfaces.

anchor "rd2" on rdomain 2 {
  match log (all, to pflog2)
}

anchor "rd3" on rdomain 3 {
  match log (all, to pflog3)
}

$ doas pfctl -f /etc/pf.conf

This is another great feature available by default with packet-filter, instead of having text logs output, the typical logs messages when using iptables or nftables on Linux systems, the packets are directly duplicated to a dedicated interfaces, compatible with tcpdump and giving us the possibility to export them in PCAP format.

$ doas tcpdump -i pflog2
tcpdump: listening on pflog2, link-type PFLOG
13:49:03.419153 172.31.255.253 > 10.0.1.1: icmp: echo request
13:49:03.419380 10.0.1.1 > 172.31.255.253: icmp: echo reply

$ doas tcpdump -i pflog3
tcpdump: listening on pflog3, link-type PFLOG
13:49:03.419267 172.31.255.253 > 10.0.1.1: icmp: echo request
13:49:03.419330 10.0.1.1 > 172.31.255.253: icmp: echo reply

The packets displayed in the tcpdump output have been generated by a simple ping command. As you can see, we have know a way to understand more deeply what can happen on each rdomain previously created.

$ route -nT2 exec ping -c1 10.0.1.1
PING 10.0.1.1 (10.0.1.1): 56 data bytes
64 bytes from 10.0.1.1: icmp_seq=0 ttl=255 time=0.479 ms

--- 10.0.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.479/0.479/0.479/0.000 ms

One question to ask is if it is possible to forward packets from one rdomain to another via packet-filter. Indeed, the configuration support on rdomain NUMBER and rtable NUMBER parameters, so, it should be possible to achieve. To test this the pair interfaces previously created must be destroyed.

$ doas ifconfig pair2 destroy

$ doas ifconfig pair3 destroy

Be sure the loopback interfaces (lo2 and lo3 in this example) are not skipped, the firewall must have the control on all traffic on these rdomains. When an ICMP packet is sent on rdomain 2, it will be redirected to rdomain 3.

anchor "rd2" on rdomain 2 {
  match log (all, to pflog2)
  pass out inet proto icmp \
    to 10.0.0.1 \
    rtable 3 \
    rdr-to 10.0.1.1
}

anchor "rd3" on rdomain 3 {
  match log (all, to pflog3)
}

$ doas pfctl -f /etc/pf.conf

$ route -nT2 exec ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
...

$ doas tcpdump -i pflog2
14:17:19.787826 10.0.0.1 > 10.0.0.1: icmp: echo request

$ doas tcpdump -i pflog3
14:17:19.788009 10.0.0.1 > 10.0.1.1: icmp: echo request

Without a route to 10.0.1.1/24, previously removed alongside the pair interfaces, it fails. Packet-filter will do the redirection anyway because it can switch from different routing tables and domains. It means the packet from 10.0.0.1 is correctly forwarded to the 10.0.1.1 address. Unfortunately, 10.0.1.1 can't reply to 10.0.0.1 because it does not have it in its own table. ICMP is perhaps not the best protocol to do those kind of redirection because it is stateless, and it can become very complex to manage the redirection. It is possible to generate an ICMP reply quite easily though, even without a correct routing table.

anchor "rd2" on rdomain 2 {
  match log (all, to pflog2)
  pass out inet proto icmp \
    to 10.0.0.1 \
    icmp-type echoreq \
    rtable 3 \
    rdr-to 10.0.1.1
}

anchor "rd3" on rdomain 3 {
  match log (all, to pflog3)
  pass in inet proto icmp \
    from 10.0.0.1 to 10.0.1.1 \
    icmp-type echoreq \
    rtable 2 \
    rdr-to 10.0.0.1
}

$ route -nT2 exec ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes

$ doas tcpdump -i pflog2
14:55:06.725461 10.0.0.1 > 10.0.0.1: icmp: echo request

$ doas tcpdump -i pflog3
14:55:06.725640 10.0.0.1 > 10.0.1.1: icmp: echo request
14:55:06.725780 10.0.1.1 > 10.0.0.1: icmp: echo reply

With this last example, it is possible to redirect a packet from one rdomain to another, but it is probably not the best idea to that in this way, because the rules will become quickly hard to understand. Regarding the anchors, one can also use another alternative syntax by passing the rdomain to each rules, like the code snippet below.

match log (all, to pflog2) on rdomain 2
pass out on rdomain 2 inet proto icmp \
  to 10.0.0.1 \
  icmp-type echoreq \
  rtable 3 \
  rdr-to 10.0.1.1

VPN Server Configuration

The VPN server will act as the default router for the workers present behind the firewall they are connected to. This server is already quite busy, many services like OpenVPN, Tinc, and few others are already present and running. The private network attributed to all those services were taken randomly, and it's a small mess. To avoid confusion, only the rules for the VPN using Wireguard will be presented, in their own rdomain.

$ doas ifconfig lo1 create rdomain 1 alias 127.0.0.1 up

$ cat | doas tee /etc/hostname.lo1 < EOF
rdomain 1
alias 127.0.0.1
up
EOF

Because rdomain 1 is totally isolated from rdomain 0 where the vio0 interface is present, it is necessary to create an interconnection network between them. The best way is to create 2 pair interfaces. The subnet will be set to 10.255.255.250/30. pair0 will stay in rdomain 0, and will give knowledge to the routes available on rdomain 1.

$ doas ifconfig pair0 create rdomain 0 up

$ cat | doas tee /etc/hostname.pair0 < EOF
inet 10.255.255.253/30
patch pair1
up
!route add 172.17.0.0/24 10.255.255.254
!route add 172.16.0.0/24 10.255.255.254
EOF

pair1 will be pushed in rdomain 1 and like pair0, it will announce the route available on its side.

$ doas ifconfig pair1 create rdomain 1 up

$ cat | doas tee /etc/hostname.pair1 < EOF
rdomain 1
inet 10.255.255.254/30
up
!route -T1 add default 10.255.255.253
EOF

With this pseudo interface configured, it is now possible to reach the network(s) present in rdomain 1. Let configures the VPN with a wireguard pseudo interface. The following snippet hides the private information to avoid leaks, but the idea is quite simple.

generate a private key using openssl from a random integer encoded in base64;
use a new random port (let say it will be 12345 for the rest of the configuration) using openssl as well;
add the public key from the firewall (it will be shown in the next part)
configure the end-point, in our case, it will be the public address of my firewall, let say 1.2.3.4 for now;
creates the wg0 interface using ifconfig containing the previously created parameters. Set also the wgaip to explicitly allow the IP address from the firewall;
finally, create the /etc/hostname.wg0 file containing the very same information to recreate automatically the VPN after a reboot.

$ server_wgkey=$(openssl rand -base64 32)

$ server_wgport=$(printf "%d\n" "0x$(openssl rand -hex 2)")

$ server_wgpeer="${firewall_public_key}"

$ server_wgendpoint="${firewall_ip_address}"

$ doas ifconfig wg0 create wgport ${server_wgport} \
  wgkey ${server_wgkey} \
  wgpeer ${server_wgpeer} \
  wgaip 172.17.0.2/32 \
  wgaip 172.16.0.0/24 \
  wgendpoint ${server_wgendpoint} \
  172.17.0.1/24 \
  rdomain 1 \
  up

$ cat | doas tee /etc/hostname.wg0 < EOF
wgport ${server_wgport} wgkey ${server_wgkey}
wgpeer ${server_wgpeer} wgaip 172.17.0.2/32 wgaip 172.16.0.0/24
wgendpoint ${server_wgendpoint}
172.17.0.1/24
rdomain 1
up
!route -T1 add 172.16.0.0/24 172.17.0.2
EOF

Having a dedicated pflog interface for this rdomain can also be helpful, so, one can create the pflog1 interface.

$ doas ifconfig pflog1 create up

$ cat | doas tee /etc/hostname.pflog1 < EOF
up
EOF

The configuration is practically over, packet-filter configuration will be the last step. The first part of this configuration will ensure the packets from the VPN are correctly translated to the public address set on the vio0 interface. In short, addresses from the worker network will use the public address of this server.

match out on rdomain 0 \
  from 10.255.255.253/28 \
  nat-to (vio0) static-port

match out on rdomain 1 \
  from 172.17.0.0/24 \
  rtable 0 \
  nat-to (vio0) static-port

match out on rdomain 1 \
  from 172.16.0.0/24 \
  rtable 0 \
  nat-to (vio0) static-port

DNS resolution can be a problem, and to avoid DNS leaks, all packets from the worker network will be redirected to a local unbound instance, in charge of doing the domain resolution. It will also be easier to see what kind of queries are sent to domain servers and to filter them in one place.

match in log (to pflog1) on rdomain 1 \
  proto {tcp, udp} \
  to port 53 \
  rtable 0 \
  rdr-to 127.0.0.1

Packet-filter is blocking all traffic by default. Wireguard needs to have its port accessible to other nodes though, then, the port previously create will be opened.

pass in on vio0 proto udp \
  to port 12345 \
  label "wireguard vpn"

Finally, the last rules are only for rdomain 1. All packets are blocked and logged by default in this rdomain. ICMP and few TCP ports are allowed though, for SSH, HTTP and HTTPS.

anchor "workers" on rdomain 1 {
  block log (to pflog1)
  pass inet proto icmp
  pass inet proto tcp to port {22, 80, 443} keep state
}

Packet-filter configuration can be reloaded or restarted.

$ doas pfctl -nf /etc/pf.conf

$ doas pfctl -f /etc/pf.conf

With this configuration, any nodes connected to wireguard will use the IP address of the server instead of their own (or the IP address of their routers). Especially useful if your workers are running in private place and you don't want to share your real IP address to Gitlab (or any other provider).

Fixing the right access to the hostname.if files can also be a good idea.

$ doas chmod 600 /etc/hostname.*

Firewall Configuration

The firewall will be way easier to configure, the process will be similar than the one for the server. At first, creating a loopback interface for rdomain 1 can help.

$ cat | doas tee /etc/hostname.lo1 < EOF
rdomain 1
alias 127.0.0.1
up
EOF

The main physical interface called re1 will be put in rdomain 1, and will not be configured. The IP address will be set on a vether pseudo interface created during the next step.

$ cat | doas tee /etc/hostname.re1 < EOF
rdomain 1
up
EOF

vether0 is the main interface for rdomain 1. It is the gateway for the workers' network and will be put in rdomain 1.

$ cat | doas tee /etc/hostname.vether0 < EOF
rdomain 1
inet 172.16.0.1/24
up
EOF

A bridge called bridge0 will be used to interconnect vether0 and re1, and will be put also in rdomain 1.

$ cat | doas tee /etc/hostname.bridge0 < EOF
add vether0
add re1
rdomain 1
up
EOF

My private network should have access to the workers' network, for monitoring and maintenance purpose. The workers will not be allowed to pass there though, this rule will be enforced in packet filter. Anyway, a small interconnection network will be created and set on pair0.

$ cat | doas tee /etc/hostname.pair0 < EOF
inet 10.255.255.253/30
rdomain 0
patch pair1
up
!route add 172.16.0/24 10.255.255.254
EOF

pair1 will also be created and connected to pair0. This time, pair1 will be put in rdomain 1.

$ cat | doas tee /etc/hostname.pair1 < EOF
inet 10.255.255.254/30
rdomain 1
up
!route -T1 add ${local_private_network} 10.255.255.253
EOF

A wireguard wg0 speudo interface is also required to connect the firewall to the VPN. The procedure is the very same than before, except the information are created for the firewall this time, and not for the server. wgaip parameter is also set to 0.0.0.0/0 to allow any network to pass through the VPN, very useful if the VPN will be set as default gateway.

$ fw_wgkey=$(openssl rand -base64 32)

$ fw_wgport=$(printf "%d\n" "0x$(openssl rand -hex 2)")

$ fw_wgpeer="${server_public_key}"

$ fw_wgendpoint="${server_ip_address}"

$ cat | doas tee /etc/hostname.wg0 < EOF
wgport ${fw_wgport} wgkey ${server_wgkey}
wgpeer ${fw_wgpeer} wgaip 0.0.0.0/0
wgendpoint ${server_wgendpoint}
172.17.0.2/24
rdomain 1
up
!route -T1 add default 172.17.0.1
EOF

Finally, the packet-filter configuration present in /etc/pf.conf. The first part defines the macros. Classic.

worker_if = "{ re1, bridge0, vether0 }"
worker_network = "172.16.0.0/24"
worker_gateway = "172.16.0.1

To avoid leaks, all packets from the workers' network will be dropped and not allowed to pass through the outgoing interface re0. This is probably not necessary here, because in the full configuration a block log is also present, and should block these connections.

block drop log \
  on $worker_if \
  from $worker_network \
  to (re0)

The connections from the private network (here set to ! $worker_network but it could be set with a network) are allowed to reach the workers' network with a small subset of protocols, including ICMP, SSH, HTTP and HTTPS.

pass on pair inet proto icmp \
  from ! $worker_network \
  to $worker_network

pass on pair inet proto tcp \
  from ! $worker_network \
  to $worker_network port {22, 80, 443}

Finally, the core configuration for rdomain 1 contains the rules filter the connections for the workers:

an explicit block is redefined, so, by default, everything is blocked;
the private network is allowed to use SSH, HTTP and HTTPS in the workers' network;
the workers are allowed to use ICMP
the workers are allowed to use SSH, HTTP and HTTPS;
only worker nodes are allowed to use bridge0;
only worker nodes are allowed to use wg0;
workers nodes have the right to connect to the gateway via HTTP;
all DNS queries are forwarded automatically to the VPN via the VPN gateway (172.17.0.1);
NTP connections are locally forwarded to an instance of ntpd running on the firewall.

anchor "workers" on rdomain 1 {
  block log (to pflog1)

  pass inet proto tcp \
    from ! $worker_network \
    to $worker_network port {22, 80, 443}

  pass inet proto icmp

  pass inet proto tcp \
    from $worker_network \
    to port {22, 80, 443} \
    keep state

  pass on bridge0 \
    from $worker_network

  pass on wg0 \
    from $worker_network

  pass inet proto tcp \
    from $worker_network \
    to $worker_gateway port 80

  pass in inet proto {tcp, udp} \
    from 172.16.0.0/24 \
    to port domain rdr-to 172.17.0.1

  pass in inet proto udp \
    from 172.16.0.0/24 \
    to port ntp \
    rtable 0 \
    rdr-to 127.0.0.1
}

Let fix the access rights on the files previously created.

$ doas chmod 600 /etc/hostname.* /etc/pf.conf

The network can be started or restarted. If other interfaces are configured, it is also possible to start them one by one or by giving a list of them as parameters for the netstart script.

$ doas sh /etc/netstart

If everything looks good... Packet filter can be restarted.

$ doas pfctl -nf /etc/pf.conf

$ doas pfctl -f /etc/pf.conf

The firewall should now be connected to the server using wireguard. Let check that with a simple ping to openbsd.org. Unfortunately, because lo1 has been created and set with 127.0.0.1, it is not possible to resolve a domain name locally on this firewall, but, it is still possible to use any other ip addresses. Indeed, a redirection is done on the VPN side, and any connections to DNS are forwarded on a local unbound instance. Let's try that.

$ route -nT1 exec dig @1.2.3.4 google.fr +short
142.251.142.195

$ route -nT1 exec ping -c3 142.251.142.195
PING 142.251.142.195 (142.251.142.195): 56 data bytes
64 bytes from 142.251.142.195: icmp_seq=0 ttl=118 time=24.746 ms
64 bytes from 142.251.142.195: icmp_seq=1 ttl=118 time=25.723 ms
64 bytes from 142.251.142.195: icmp_seq=2 ttl=118 time=25.193 ms

--- 142.251.142.195 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 24.746/25.221/25.723/0.399 ms

$ route -nT1 exec nc -zv 142.251.142.195 80
Connection to 142.251.142.195 80 port [tcp/www] succeeded!

$ route -nT1 exec nc -zv 142.251.142.195 443
Connection to 142.251.142.195 443 port [tcp/https] succeeded!

It seems the firewall is correctly using the VPN and the packets are correctly routed to some public IP addresses. To be sure, one can also use tcpdump to see what's happening on wg0.

$ tcpdump -ni wg0
tcpdump: listening on wg0, link-type LOOP
14:51:40.154762 172.17.0.2.31538 > 142.251.142.195.443: SWE 4156193580:4156193580(0) win 16384 <mss 1452,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 2865047363 0> (DF)
14:51:40.180844 142.251.142.195.443 > 172.17.0.2.31538: S 1448631416:1448631416(0) ack 4156193581 win 65535 <mss 1412,sackOK,timestamp 728224899 2865047363,nop,wscale 8> (DF) [tos 0x80]
14:51:40.180995 172.17.0.2.31538 > 142.251.142.195.443: . ack 1 win 256 <nop,nop,timestamp 2865047383 728224899> (DF)
14:51:40.182406 172.17.0.2.31538 > 142.251.142.195.443: F 1:1(0) ack 1 win 256 <nop,nop,timestamp 2865047383 728224899> (DF)
14:51:40.206851 142.251.142.195.443 > 172.17.0.2.31538: F 1:1(0) ack 2 win 1050 <nop,nop,timestamp 728224925 2865047383> [tos 0x80]
14:51:40.207024 172.17.0.2.31538 > 142.251.142.195.443: . ack 2 win 256 <nop,nop,timestamp 2865047413 728224925> (DF)
14:51:40.987301 172.16.0.2.48148 > 172.65.251.78.443: . ack 2499027552 win 78 <nop,nop,timestamp 674758452 2705939126>
14:51:41.013134 172.65.251.78.443 > 172.16.0.2.48148: . ack 1 win 19 <nop,nop,timestamp 2705969587 674743439> (DF)
14:51:41.333030 172.65.251.78.443 > 172.16.0.2.48148: . ack 1 win 19 <nop,nop,timestamp 2705969908 674743439> (DF)
14:51:41.334165 172.16.0.2.48148 > 172.65.251.78.443: . ack 1 win 78 <nop,nop,timestamp 674758799 2705969587>
14:51:42.840018 172.65.251.78.443 > 172.16.0.3.40668: . ack 1758166881 win 16 <nop,nop,timestamp 2453064192 1968171036> (DF)
14:51:42.840607 172.16.0.3.40668 > 172.65.251.78.443: . ack 1 win 78 <nop,nop,timestamp 1968186512 2453048737>

It is now confirmed, the traffic is going into the VPN!

Workers Configuration

The last part of this infrastructure is directly related to the "workers". Those workers will host gitlab-runners and probably other services like podman or even k3s to build, test and release my own private application locally. Those servers are old laptops, and other small devices connected in their dedicated switch, behind the firewall. Most of them will use Alpine Linux. So, first, the network configuration.

$ cat | doas tee /etc/network/interfaces < EOF
auto lo
iface lo inet loopback

auto eth0
        address 172.16.0.2/24
        network 172.16.0.0
        broadcast 172.16.0.255
        gateway 172.16.0.1
EOF

To apply the configuration, simple invoke the service command.

$ doas service networking restart

Then the dnsmasq configuration, in the previous article, it was used to worked the domain name requests to the Tor DNS service. In this case, it will be used to forward the requests to the gateway, then the gateway will forward them to the VPN.

$ cat | doas tee /etc/dnsmasq.conf < EOF
no-resolv
server=172.16.0.1
local-service
listen-address=127.0.0.1
conf-dir=/etc/dnsmasq.d/,*.conf
EOF

Be sure dnsmasq is started during boot and start the service if it's not already done.

$ doas rc-update add dnsmasq default

$ doas service dnsmasq start

Ensure /etc/resolv.conf is correctly configured to ask dnsmasq.

$ cat | doas tee /etc/resolv.conf < EOF
nameserver 127.0.0.1
EOF

Voilà. A worker is configured! The idea was to have something really simple on this side to deploy many workers when needed. The dnsmasq is a bonus, because simply configuring the /etc/resolv.conf would have been way enough.

Testing

One of the most important section of this article... Testing the configuration from the workers point of view. First, the DNS resolution is mandatory and must work. Checked.

$ dig +short openbsd.org
199.185.178.80

All packets must be forwarded through the VPN and not locally. As one can see, the packets are passing to the gateway, then, to the VPN and go out from the server (the IP addresse(s) have been replaced by x.x.x.x). Checked.

$ traceroute openbsd.org -nI
traceroute to openbsd.org (199.185.178.80), 30 hops max, 46 byte packets
 1  172.16.0.1  0.591 ms  0.179 ms  0.434 ms
 2  172.17.0.1  24.231 ms  25.197 ms  24.946 ms
 3  x.x.x.x  25.732 ms  26.328 ms  28.007 ms
 4  x.x.x.y  24.522 ms  26.557 ms  25.837 ms
 5  x.x.y.z  25.012 ms  24.913 ms  25.359 ms
 6  80.249.213.240  26.202 ms  26.065 ms  31.208 ms
 7  64.125.20.28  145.869 ms  *  146.492 ms
 8  64.125.29.184  145.833 ms  145.111 ms  146.056 ms
 9  *  64.125.31.110  146.041 ms  *
10  *  *  *
11  *  *  *
12  64.125.30.199  147.083 ms  145.227 ms  145.968 ms
13  64.125.31.158  145.922 ms  146.089 ms  146.342 ms
14  64.125.18.179  145.466 ms  145.984 ms  145.082 ms
15  64.125.18.180  148.951 ms  147.816 ms  146.111 ms
16  64.125.18.210  145.243 ms  146.458 ms  146.015 ms
17  208.185.75.98  146.226 ms  146.872 ms  146.485 ms
18  204.209.252.4  146.952 ms  148.495 ms  147.707 ms
19  199.185.230.2  147.111 ms  147.041 ms  147.363 ms
20  199.185.178.80  145.313 ms  144.660 ms  145.852 m

HTTP connections should work. It can be verified with nc. Checked.

$ nc -zv openbsd.org 80
Connection to openbsd.org (199.185.178.80) 80 port [tcp/http] succeeded!

HTTPS connections must work. This one is important because some repository will be cloned using HTTPS. Checked.

$ nc -zv openbsd.org 443
Connection to openbsd.org (199.185.178.80) 443 port [tcp/https] succeeded!

Another important service to check is SSH. Most of the repositories are using Git over SSH, usually listening on port TCP/22 by default. The workers must have access to that as well. Checked!

$ nc -zv ftp.eu.openbsd.org 22
Connection to ftp.eu.openbsd.org (193.156.13.178) 22 port [tcp/ssh] succeeded!

Finally, let check if Alpine Linux can be updated and upgraded... Checked!

$ doas apk update
v3.23.7-30-g7a9ffafb21a [https://mirror.leaseweb.com/alpine/v3.23/main]
v3.23.7-29-gecf474a05c5 [https://mirror.leaseweb.com/alpine/v3.23/community]
OK: 27586 distinct packages available

$ doas apk upgrade
OK: 1146.9 MiB in 302 packages

My own local workers/builders infrastructure is now running behind a firewall and a VPN. The IP address seen by gitlab is not the one from my place, but from another server.

Troubleshooting and Improvements

Managing rdomain can become very complex, and one must be disciplined to manage packet-filter rules but also the pseudo devices created. tcpdump will be the best tool to investigate a routing issue and will quickly give an idea of the root cause. route and netstat commands will also be really useful to inspect the state of the network stack. Finally, creating pflog interfaces to log and follow the packet flow will also be of a great help.

Another point to improve in this design is about trusting the traffic. The workers will use gitlab-runners, and based on the news, supply-chain attacks or 0days are kinda dangerous. The previous configuration using Tor was interesting because a proxy was required. In the current situation an HTTP proxy would have been a good idea to deploy, to filter and log all outgoing connections. A bastion for ssh (or an ssh proxy) would have been great as well. Indeed, to avoid an attack, restricting the access to a small subset of allowed IP addresses or domains is the bare minimum.

Conclusion

Coming back to my roots was pretty great. Configuring VPN, routes and other kind of network stuff on OpenBSD is always a fun experience. rdomain is a furtive technology mostly known by some OpenBSD wizards and other hardcore system/network administrators. The number of publications on this topic is pretty low, but the OpenBSD documentation is here to save our life.

Anyway, rdomain is a great tool to isolate networks and avoid leaks. Configuring it correctly can be challenging though, and testing it is also another layer of complexity, but the result is - as usual with OpenBSD - rock solid. This article is already huge, and having a list of resources/references can also probably help you to deal with all of that.

The rdomain(4) man page, explaining how rdomain is working with few examples;
The pfctl(8) command man page, where you will find how to use pfctl, mainly to reload the firewall rules;
The pf.conf(5) configuration page man page, where you will find more about rdomain and rtable;
The route(8) command mange page, where you will learn how to use the route command to switch between rdomains;
The netstart(8) script man page, to restart/reload interfaces;
The hostname.if(5) configuration file man page, where the configuration of the interfaces will be stored;
The wg(4) pseudo device man page, where you will find more information about how to create, configure and maintain a wireguard interface;
The pair(4) pseudo device man page, where you will understand how to use pair to connect different rdomain together;
The vether(4) pseudo device man page, where you will learn how to use the virtual ethernet interface;
The bridge(4) pseudo device man page, useful to link many interfaces together;
The ifconfig(8) command man page, to deal with the pseudo devices and network interfaces;
The pflog(4) pseudo interface man page to see what kind of data one can collect and how to use it with tcpdump;
Implementing Routing Domains on an OpenBSD workstation for use with WireGuard talk presented in 2024, with the recording available on youtube;
Introduction to global anycast using OpenBSD (on a budget) live streaming talk presented in 2024, with the recording available on youtube;
Practical use of OpenBSD routing domains with redundant
firewalls talk presented in 2023, with the recorded session available on youtube;
Installation de pare-feux redondants avec OpenBSD talk slides presented in 2021 (but in French);
Mininet on OpenBSD Using rdomains for Interactive SDN Testing and Development talk presented in 2018, with the youtube video and its paper;
pf+rdomains create splendid multi-tenancy firewalls talk presented in 2017, with the youtube video;
Branch VPN Solution with OpenBSD talk slides presented in 2017;
using routing domains/routing tables in a production network talk slides presented in 2015;
OpenBSD Routing Tables and Routing Domains by the Unfriendly Grinch, offering a great rdomain usage example;
Secure torrenting with OpenBSD rdomains, a post showing us how to isolate/secure torrenting using rdomain;
OpenVPN on OpenBSD in its own rdomain to prevent data leak, another post where rdomain is used with OpenVPN;
Full WireGuard setup with OpenBSD, probably one of the first post available on the web about wireguard and rdomain on OpenBSD;
Configuring PF to route between routing domains on ServerFault;
OpenBSD PF rdr-to across rdomains on Linux Questions;

Happy hacking and have fun!

EDIT(20261103): adding a missing route in /etc/hostname.wg0

Cover Image by Rafael Peier on Unsplash

WeeklyReview#202644

Mathieu Kerjouan — Mon, 02 Nov 2026 10:00:00 +0000

This week was mostly dedicated to clean up and fix my CI/CD local infrastructure. The first version of this one was undocumented and using Tor as relay to hide my private IP address. Unfortunately, most of the tor exit nodes were blocked by Google, and following the recent news regarding the 333 rogue exit nodes, forwarding this traffic through my own servers is probably a better idea. Both configuration have been configured - even if it's not yet finished but it works.

In the end, it was great to do some old-school sysadmin/netadmin tasks on OpenBSD. This system is still my favorite after all these years. Yes, it is not fast as a Linux-like distribution, it does not have all fancy hardware support from other operating system, but damn... In few hours, one is able to deploy a crazy stable and secure infrastructure.

On my Dart/Flutter application side, the database management was - and still is - a problem. sqflite and other interface to it are unpleasant to use. One of my target is to use ribs_sqlite over sqlite or sqflite, but designing it this way requires to change some of my first design.

Another thing, the front-end part was my main target because it was my weak point. By doing that, a big part of my time was dedicated to learn more about UI/UX and mobile design. A post is planned to be released - one day - on that.

Coding

🏗️ nanofil: A C compiler written in Prolog that emits x86-64 assembly in the Fil-C ABI. Its object files link with clang-built Fil-C code, and they keep Fil-C's guarantees

🌐 Learn Erlang Step-By-Step: a new initiative to learn Erlang based on a real-world project! I like the concept and how it has been done.

📝 A faster way to compute signed 64-bit floor division by constants: a way to optimize floor division when the divisor is known during compile time.

📝 Writing a Linux Debugger Part 1: Setup: a serie of article on developing its own linux debugger in C++.

System

🏗️ Beszel: a lightweight server monitoring platform that includes Docker statistics, historical data, and alert functions.

Database

🏗️ pgCluu: a Perl program used to perform a full audit of a PostgreSQL Cluster and System performance.

🌐 SafeGrd: SafeGrd backs up your databases, files and mail, encrypts and locks each copy, and restores it on a schedule to prove it works. An agent with your production keys can drop the database, but not the backups.

🏗️ pgrecon: an open-source Oracle to PostgreSQL migration assessment and schema conversion tool. It inventories an Oracle database from an offline dump and runs a deterministic rule engine over it.

🏗️ PostgreSQL Anonymizer: an extension to mask or replace personally identifiable information (PII) or commercially sensitive data from a Postgres database.

🏗️ sqlean: There are a lot of SQLite extensions out there, but they are incomplete, inconsistent and scattered across the internet. Sqlean brings them together, neatly packaged into domain modules, documented, tested, and built for Linux, Windows and macOS.

Security

🌐 Tailworm, we have zero-trust in our infrastructure: a report audit of the tailscale project, containing a lot of security issues.

🐦 Keystone: Statement Regarding the Security Incident on Our Official Website and Customer Guidance

📑 Don’t Look Up: There Are Sensitive Internal Links in the Clear on
GEO Satellites

📝 Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

🌐 tpbreak: TP-Link firmware decryption at scale — 52,609 images. The verification public key is also the decryption key: the AES key/IV fall out of the RSA-PSS salt. No private key required.

🌐 SMS of Death: an unchecked cast on an array can cause a denial of service for a long period of time on Android. It can lead to SQLite database corruption and also impact the phone to boot. It can brick the phone. If I understand correctly, it will not be fixed... Crazy.

📝 Alpine Linux GitLab security incident: attempted exploitation of CVE-2026-97229: a gitlab upgrade where the main fix was not applied because of Ruby. It confirms me Alpine Linux team is great, instead of ignoring this attack, they are redeploying/rebuilding everything. Great project.

📝 SPECTER: Leaking Data from Air-Gapped Computers by Toggling USB Power States

📝 Thermal Noise Communication: Sending Data Without a Carrier Signal, with an RTL-SDR as the Receiver

📝 Official Claude Code VS Code Extension Compromised to Install Persistent SSH Backdoor

📝 When intermediate CAs fail, language stacks discover they own the blast radius

📑 Bypassing Secure Boot using Fault Injection

Security/cryptography

📝 The case of the 9 bit AES key: encrypt your document yourself... PDF encryption from open-source world is just broken. The keys/seeds are not coming from cryptographic random sources, then, it is easy to break.

📝 QSYRUPWD, Part 2: Correcting the Rijndael Key Builder

📝 Lean for Cryptanalysts: A Practical Introduction to Formal Proofs: a lot of people are talking about the lean language, it is on my long list of thing to learn. When someone is writing a post about Lean and cryptanalysis, for sure it will be a banger. So, this is a great example of lean usage, and it motives me even more to learn/use it.

Security/RedTeam

🌐 Exploit for php2shell-audit: a "working" (yet non-tested on my side) exploit against php2-shell.

🌐 Exploit for CVE-2026-111497 (brotli): a "working" (yet non-tested on my side) exploit on nodejs brotli vulnerability.

📝 Bluetooth Low Energy Security Testing, Consolidated: Introducing Caeruleus: Bluetooth fascinate me since High School, at this time, I unsuccessfully tried to create an audio/video streaming application on Nokia SymbianOS in C++. It was fun, and bluetooth is a really interesting piece of technology. Having a list of tools to identify and attack this protocol is always useful, especially if you are developing Bluetooth application.

Embedded

📝 Digikey Reference Design Library

Update/Upgrade

🔄 LibreSSL 4.4.0: including eMudhra security fix.

🔄 Sel4 17.0.0: many bug and security fixes. Documentation was also improved. Few platforms have been removed (Arndale, Odroid-X, HiKey, Inforce IFC6410, exynos5). The full changelog is available on Github.

🔄 Elixir 1.20.5: Security and bugs fixes release, including 2 CVEs.

🔄 Flutter 3.47.9: bugs fixes release.

Theory

📑 Evidence for the representation of non-hierarchical structures in language

Leaks of the Week

🐦 Sony: more than 2.5 millions customers leaked, including sensitive information like mails and phone number.

📝 Solnet: Switzerland network provider impacted by a data leak impacting more than 5000 customers.

📝 Rassemblement National: more than 432000 subscribers data leaked from this French political party.

🐦 iCloudITS: Saudi IT Provider leaked more than 15GB of data from their database.

📝 Bangkok Post: more than 820000 members impacted by a data leak

📝 Grand Orient de France: 6.72GB of data leaked from one of the main French freemasonry.

🐦 French Unknown IT Company: if true, an IT company was impacted by a huge data leaks, including their private keys API for Stripe and Vercel.

Misc

📝 Anthropic announces bankruptcy by...?

📝 Who’s Aligned to Whom? How AI Agents Shape Experts’ Research Judgment Calls: The conclusion contains all you need to know.

experts may defer underdetermined decisions to an AI agent, and that this deferral can be hard to identify, even for the experts themselves [...] Judgment also depends on skill, and people who stop making these decisions may lose the ability to make them [...] institutions will need an explicit strategy for developing human skills as models become more capable [...] Institutions may want to protect these skills and human oversight even with extremely capable AI, for safety, resilience, or their own sake.

📝 Claude Code burns ~15–20% of your tokens on a 3-word status line. Here's how to turn it off

📝 Manifestos, Fear, and ‘Loser Thinking’: Inside Proton’s AI Reckoning: What a great leader and innovator... Calling his engineers and technicians loser is really an incredible act. So, time to move to another solution. By the way, tutanota is perhaps not a good alternative.

Cover Image by Nikolett Emmert on Unsplash

devto is a really unstable platform to publish article. every weeks something is broken; today, I can't update one very long post... The PUT request got stuck on the server side. Crazy.

Mathieu Kerjouan — Fri, 30 Oct 2026 15:52:19 +0000

Isolated Network for Tor-like Services

Mathieu Kerjouan — Wed, 28 Oct 2026 10:00:00 +0000

The first part of my career in IT was to be network technician/administrator, configuring firewalls, routers and switches for companies. Since this time, few devices are always ready to do some interesting stuff related to network, one of them is a pretty old pcengine apu2 firewall using the latest OpenBSD release. It was mostly a canary-like computer to check if everything was fine during an OpenBSD update/upgrade. Since one of my project is being entirely coded in Dart/Flutter, and because this language is not yet supported by OpenBSD, it would have been good to have a dedicated CI/CD gitlab runner running somewhere... Why not using an old laptop and isolating it behind this firewall?

Configuring OpenBSD

My APU2 is having 3 Gigabit Ethernet port called re0, re1, and re2. All of them are using the re driver. re0 is the main interface with a direct connect to another router.

$ doas cat /etc/hostname.re0
dhcp
inet6 autoconf
inet alias ${gateway_address}
up

re1 is the interface where the server(s) hosting the worker(s) will be connected. Even if this interface will be used, it will not be set with an IP address but put in a bridge. All interfaces can be started using the netstart script.

$ doas cat /etc/hostname.re1
up

$ doas ksh /etc/netstart re1

A virtual interface vether will be created to deal with the private network dedicated for the workers.

$ doas cat /etc/hostname.vether0
inet 172.16.0.1/24
up

$ doas ksh /etc/netstart vether0

bridge0 is also created to connect re1 and vether. Based on the current configuration, a veb interface would have probably be also another correct choice.

$ doas cat /etc/hostname.bridge0
destroy
add vether0
add re1
up

$ doas ksh /etc/netstart bridge0

The router configuration is ready. All workers will use 172.16.0.1 as default route, and packet filter will be in charge to filter which part of the traffic is right or not.

Tor Bootstrapping

Tor is easy to install, but can be challenging to configure. Let install it first via pkg_add.

$ doas pkg_add tor

The main Tor configuration is located in /etc/tor/torrc and also in /etc/torrc.d/ directory. The first thing to configure is the DNS resolution to avoid leaks.

$ doas touch /etc/torrc.d/dns.conf

$ echo "AutomapHostsOnResolve 1" \
  | doas tee /etc/torrc.d/dns.conf

$ echo "AutomapHostsSuffixes .exit,.onion" \
  | doas tee /etc/torrc.d/dns.conf

Then, because our workers will mostly be connected to servers in North America or Europe, it can be nice to define the geolocation of the exit nodes. In the following case, the workers will use exit nodes from USA, UK, Europe, and Japan.

$ doas touch /etc/torrc.d/exit_nodes.conf

$ echo 'ExitNodes {us},{uk},{eu},{ez},{jp} StrictNodes 1' \
  | doas tee /etc/torrc.d/exit_nodes.conf

The last step is to allow Tor to listen on the correct interface, in this case, re1 with the IP address 172.16.0.1. A first instance will listen on TCP/9080 and allow only onion routing to the standard internet. A second instance will be dedicated for onion traffic only and will listen on TCP/9081. Both of them will be used as HTTP Proxies.

$ cat | doas tee /etc/torrc.d/workers.conf << EOF
HTTPTunnelPort 172.16.0.1:9080 IsolateDestAddr IsolateDestPort NoOnionTraffic ClientUseIPv4
HTTPTunnelPort 172.16.0.1:9081 IsolateDestAddr IsolateDestPort OnionTrafficOnly ClientUseIPv4
EOF

Another instance will use a SOCK Proxy this time, and will listen on TCP/9050.

$ cat | doas tee -a /etc/torrc.d/workers.conf << EOF
SOCKSPort 127.0.0.1:9050 IsolateDestAddr IsolateDestPort ClientUseIPv4 SafeSocks 1
EOF

Finally, all the DNS requests must be resolved via Tor and not any other kind of DNS to avoid DNS requests leaks. To do that, Tor will listen on TCP/9053 and will use its own DNS resolver.

$ cat | doas tee -a /etc/torrc.d/workers.conf << EOF
DNSPort 172.16.0.1:9053
EOF

The configuration looks good. Let enable Tor via rcctl...

$ doas rcctl enable tor

... And start it.

$ doas rcctl start tor

Regarding this configuration, one could also configure packet filter to forward the packet directly to a Tor instance on the loopback interface, but it can make the firewall configuration a bit more complex. Another idea is to use another proxy server like Privoxy to do the relay.

Packet Filter Configuration

Tor is ready, it should listen on the correct interface, but the firewall must be configure to avoid leaks. Let defines the macros first, to help us deal with the different parameters. Packet filter configuration is located in /etc/pf.conf.

tor_if = "{ re1, bridge0, vether0 }"
tor_network = "172.16.0.0/24"
tor_gateway = "172.16.0.1"
tor_proxy_ports = "{ 9050, 9080, 9081 }"
tor_dns = 9053

Using Tor can already be painful due to the speed of the connection, so, packet filter can be optimized a bit to win few milliseconds (sometimes). The loopback interface is also ignored, at this time, it's not necessary to filter packet from it.

set optimization aggressive
set skip on lo

Let configure the anti spoofing features and scrub all packets coming into the firewall.

antispoof for re0
antispoof for $tor_if
match in all scrub (no-df random-id)

The default rule is to drop everything. The next part of the configuration will allow only which packets are authorized to pass through the firewall.

block drop log on re0 
block drop log on re0 from $tor_network
block drop log on $tor_if

Nobody wants to lose access to its firewall, then, the TCP/22 port (ssh) is explicitly authorized from the main interface (re0). All packets going out on this interface are also allowed. For debugging and maintenance purpose, ICMP and ICMPv6 protocols are allowed as well.

pass in quick on re0 proto tcp to port 22 
pass out quick on re0 keep state
pass on re0 proto icmp 
pass on re0 proto icmp6

Here the first part of the Tor firewall configuration. Firstly, to help see if the workers are up or down, ICMP are allowed only in one way, the workers are not allowed to send ICMP, but other servers and equipments from other part of the network are.

pass in proto icmp \
  from ! $tor_network \
  to $tor_network tag \
  tor_icmp
pass on $tor_if proto icmp \
  from $tor_network \
  to $tor_gateway \
  tag tor_icmp
pass on $tor_if proto icmp \
  from $tor_gateway \
  to $tor_network \
  tag tor_icmp
pass out proto icmp tagged tor_icmp

The workers are allowed to communicate between them. Indeed, it can be useful especially if the workers can share some data, or if a server is installed to cache the artifacts. A fine grained list of host can eventually be created here. The second rules explicitly block all packets coming from the Tor network to any other network different than this network. It means workers' packets will not be able to traverse the router.

pass on $tor_if inet proto tcp \
  from $tor_gateway \
  to $tor_network \
  keep state

pass on $tor_if inet proto tcp \
  from ! $tor_network \
  to $tor_network \
  keep state

When configuring a CI/CD, it always nice to have a correct time from all the workers, if possible returning an UTC timestamp. Unfortunately, bare-metal servers can slowly be inaccurate regarding their local time, and having access to an NTP server can be helpful. Instead of allowing all workers to connect directly to a remote NTP server, the firewall - in this case - will act as a relay. Indeed, an ntpd daemon will also listen on re1 interface and will offer the correct time.

pass in on $tor_if inet proto udp \
  from $tor_network \
  to $tor_gateway port ntp \
  keep state

DNS is also another part of a network, but workers can't be allowed to request unknown server from internet. So, Tor has been previously configured to listen on port UDP/9053 for its DNS resolver. The workers must be able to reach this port and do the DNS resolution.

pass on $tor_if inet proto udp \
  from $tor_network \
  to $tor_gateway port $tor_dns \
  keep state

Finally, the Tor proxies are also allowed to receive the connections from the workers.

pass on $tor_if inet proto tcp \
  from $tor_network \
  to $tor_gateway port $tor_proxy_ports \
  keep state

A bonus step, because my configuration for the workers required some extra-data, the workers can also reach out the firewall on the port TCP/80 (HTTP) to fetch some kind of information. The firewall here will act as a simple HTTP server.

pass in on $tor_if inet proto tcp \
  from $tor_network \
  to $tor_gateway port http \
  keep state

Let check packet filter configuration to see if everything is correct with the help of pfctl.

$ doas pfctl -nf /etc/pf.conf

If all is good, the firewall can be started.

$ doas pfctl -ef /etc/pf.conf

The configuration presents on my firewall is a bit more complex, but this one should work correctly.

From the Workers

With the previous configuration, the workers are easily reachable by adding a static route from any other part of my network to the firewall.

$ doas route add 172.16.0.0/24 ${gateway_ip}
add net 172.16.0.0/24: gateway ${gateway_ip}

From my laptop, the firewall interface for the Tor network is answering to the pings, that's a good sign.

$ ping -c 3 172.16.0.1
PING 172.16.0.1 (172.16.0.1): 56 data bytes
64 bytes from 172.16.0.1: icmp_seq=0 ttl=255 time=2.865 ms
64 bytes from 172.16.0.1: icmp_seq=1 ttl=255 time=2.782 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=255 time=1.974 ms

--- 172.16.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.974/2.540/2.865/0.402 ms

It seems the worker present on the Tor network (running on alpine linux) is also answering to the pings.

$ ping -c 3 172.16.0.2 
PING 172.16.0.2 (172.16.0.2): 56 data bytes
64 bytes from 172.16.0.2: icmp_seq=0 ttl=63 time=4.281 ms
64 bytes from 172.16.0.2: icmp_seq=1 ttl=63 time=3.336 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=63 time=4.176 ms

--- 172.16.0.2 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 3.336/3.931/4.281/0.423 ms

The TCP/22 port is working as well.

$ nc -zv 172.16.0.2 22
Connection to 172.16.0.2 22 port [tcp/ssh] succeeded

This is the time to connect to this worker and check the firewall configuration.

$ ssh 172.16.0.2
...

From the worker, it is possible to ping the default route 172.16.0.1 without problems.

$ ping -c 3 172.16.0.1
PING 172.16.0.1 (172.16.0.1): 56 data bytes
64 bytes from 172.16.0.1: seq=0 ttl=42 time=0.617 ms
64 bytes from 172.16.0.1: seq=1 ttl=42 time=0.669 ms
64 bytes from 172.16.0.1: seq=2 ttl=42 time=0.758 ms

--- 172.16.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.617/0.681/0.758 ms

Public IP addresses, like 8.8.8.8 are not reachable, the firewall is blocking these packets.

$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

DNS resolution is important though, and all workers must have a local DNS server to use the Tor resolver. In this scenario, dnsmasq is used here.

$ cat /etc/dnsmasq.conf
no-resolv
server=172.16.0.1#9053
local-service
listen-address=127.0.0.1
conf-dir=/etc/dnsmasq.d/,*.conf

Then, /etc/resolv.conf must be updated.

$ cat /etc/resolv.conf 
nameserver 127.0.0.1

With this new configuration, workers are allowed to resolve domain names via the Tor network.

$ host google.com
google.com has address 172.253.63.139
google.com has IPv6 address 2607:f8b0:4006:819::200e
Host google.com not found: 4(NOTIMP)
Host google.com not found: 4(NOTIMP)

All workers should also configure their environment with the correct proxies, usually HTTP_PROXY and HTTPS_PROXY variables.

$ set | grep -i proxy
HTTPS_PROXY='http://172.16.0.1:9080'
HTTP_PROXY='http://172.16.0.1:9080'
http_proxy='http://172.16.0.1:9080'
https_proxy='http://172.16.0.1:9080'

curl is looking for those variables by default, and should use them. It can be checked by checking the public IP address used via api.ipify.org.

$ curl https://api.ipify.org; echo
204.8.96.80

Indeed, it returns an IP address way different than the one configured on my main router.

What About Privoxy?

Most of the time, Privoxy is configured alongside Tor to use it as relay. Indeed, privoxy is way more flexible than Tor itself and can offer a better control. In our case, it will not really be necessary to use it, because nothing will be filtered from the network where the workers are present. Only the firewall needs to be correctly configured to avoid data leaks.

Conclusion

It was an interesting project, but not a really great idea in the end. Most of the tor exit-nodes are blocked by Google, even more, with the recent certificate issues, using a worker always connected to Gitlab to fetch the latest version through an untrusted highly-risky network is perhaps not the best thing to do right now. This idea can be improved though, and instead of using tor, one can create a VPN and force the server to use it instead of using your public IP address.

The same kind of configuration can also be used for the i2p services, it will perhaps be a bit different because i2p does not have an onion routing service... but it should work if one wants to force the workers or the servers connected to the network to only use i2p. Another cool stuff would be to create a server relay between tor and i2p.

A bigger article on Tor is planned, but it is a really huge one and take time to write it. This one is just a quick note regarding the configuration of the workers from my CI/CD infrastructure. Below a schema to help you understand how it works.

As usual, have fun and happy hacking!

Cover image by Georgi Kalaydzhiev on Unsplash

WeeklyReview#202643

Mathieu Kerjouan — Mon, 26 Oct 2026 10:00:00 +0000

Black October: the day people discovered Internet is a dangerous place this week. And what a week... It was absurdly hard days for the security world, and it's probably not done yet (because this note has been updated a lot). A huge amount of security issues and other crazy discoveries were announced in few hours/days. Good luck to all my sysadmin friends working to make their infrastructure safer.

One of the very first big issue was due to a certificate authority problem. Many certificates generated by eMudhra (an Indian company) were totally unsafe. Indeed, their private keys were all an unique integer between 0 and 255 (uint8), instead of a long integer encoded on 384 bits. All started on 17th October, when "Wayne" (a security guy playing with new tools), discovered 8 odd certificates. A ticket was opened on Mozilla's Bugzilla. The thread is still very active. The consequences are concerning. Python is impacted and many packages must be updated. The Erlang Ecosystem Foundation did also a post about this issue explaining which modules must be upgraded as soon as possible. All Operating Systems are also impacted by this issue, and the certificates must be upgraded locally as well. The mitmproxy project already includes on its last version those certificates. More information below.

eMudhra emSign PKI Services: CP/CPS documents not updated for current requirements from Mozilla's Bugzilla thread;
eMudhra: Private key compromise of Intermediate Certificates from Mozilla's Bugzilla thread;
Information for Python users regarding eMudhra / emSign CA certificate security incident from Python's Discuss thread;
eMudhra CA Intermediate Certificates have been compromised, CA updates needed on all TLS components from the Erlang Ecosystem Foundation;
Alpine Linux 3.24.3 from Alpine Linux website;
OpenBSD 8.0 Errata 005 from the OpenBSD website;
FreeBSD -- TLS certificate store update due to eMudhra compromise from FreeBSD VuXML;
eMudhra TLS/SSL CA compromise from the NetBSD blog;
Debian ca-certificates security update, from the Debian mailing-list;
Eight eMudhra CA Keys Compromised, Triggering Emergency Fixes Across Browsers and Software from The Hacker News;
India's eMudhra exits website security business after flaws trigger global alarm from Reuters;
Oracle Security Alert Advisory - CVE-2026-91403 from Oracle/Java website.

PHP is well known for Wikimedia, Drupal, Wordpress, Symphony and many other frameworks/projects widely used around the world. Unfortunately, PHP design is not the best, and during the past decades, many critical issues have been impacted it. This week, 2 huge security issues have been released with a CVE score greater than 9 and allowing remote code executions. What's crazy here is those vulnerabilities have been discovered in 2024 (!) during an audit.

PHP was my first "real" language, like probably many of the developers and sysadmin of my generation. This language was - and still is - a big joke regarding design and consistency. During the week, PHP was also impacted by 2 big security issues with a score greater than 9 on the CVE scale. The whole story is just a pure mess. Called php2shell, the bug can execute arbitrary code on a remote server. Those bugs have been discovered in 2024 (!) during an audit. The PHP Foundation published a post on their blog explaining the whole process. Another crazy thing is, someone published an exploit few days before the official release, without any consideration of the companies being impacted by this news.

PHP Issues Emergency Patches for Two Critical PHP2Shell RCE Flaws as Exploit Attempts Emerge on The Hacker News;
PHP2Shell: What Actually Protects You Before the 14:00 UTC PHP RCE Patch, and What Doesn't on the Hacker News;
PHP2Shell: Two Unpatched Critical PHP RCE Flaws (CVSS 9.2) Trigger Emergency Security Alert on The Hacker News;
Two Critical PHP Flaws Leaked Ahead of the October 21 Security Release on WebHosting.today;
Today's PHP Security Release, and How We're Strengthening PHP Security on the Official PHP Foundation blog;
PHP2Shell: CVE-2026-109618 and CVE-2026-109619 exploits on Abraxas Labs;
php2shell/php2dos - php-stale-global-uaf exploit on Github;
php2shell - php-oom-longjmp exploits on Github;
Jesus Christ patch your shit now on xeiaso.net.

A second breach in PHP has also been announced, not related to PHP2Shell, linked to the Session process.

Informational advisory: Session upload progress from PHP official website;
CVE-2026-115560 — PHP session upload progress mitigation

At the same time, QUIC/HTTP/3 was also impacted by another 0day vulnerability called quiche, causing a DoS and eventually an OOM on many implementation of this protocol.

Header Accordions: QPACK amplification attacks against HTTP/3 servers on Foxio.io;
headeraccordions exploit/poc on Github.

It's not over... Gitlab also released emergency patches for Gitlab Community and Enterprise Edition containing 12 fixes, including one critical vulnerability allowing an unauthenticated user to execute arbitrary code remotely.

GitLab Critical Patch Release: 19.5.1, 19.4.3, 19.3.6, 19.2.9, 19.1.9, 19.0.10, 18.11.13 on Official Gitlab Documentation:
Critical GitLab Flaw (CVSS 10.0) Could Allow Unauthenticated RCE Across 12 Years of Releases on The Hacker News.

Signal released also a series of patch to fix a critical security issue, allowing someone to keep the micro and the camera active after a call.

Signal bug could keep your mic - and camera - on after hangup
Signal v2.72.0 on Gitlab;
CWE-754: Improper Check for Unusual or Exceptional Conditions on Common Weakness Enumeration website.

NodeJs, as usual, is always there to offer us more vulnerabilities, even when we don't need them. Nodejs team released a long series of patches fixing dangerous security issues, 4 of them with high level risks.

Wednesday, October 21, 2026 Security Releases on NodeJS official blog;
Node.js security advisory: Brotli decoder RCE (CVE-2026-111497) on Vercel;
"No way to prevent this" say users of only language where this regularly happens on Xeiaso.net.

Do you think it's over? Not at all! Apache traffic HTTP server released a new released to fix 15 vulnerabilities, including remote code execution and many other crazy holes.

"No way to prevent this" say users of only language where this regularly happens on xeiaso.net;
[SECURITY] Multiple vulnerabilities fixed in Apache Traffic Server 9.2.16, 10.1.5 and 10.2.1 on apache traffic mailing-list.

A dangerous memory corruption bug on Bind9 can lead an attacker to execute remote code. Again, high severity issue, needs to be patched as soon as possible.

CVE-2026-94300: Concurrent GSS-TSIG messages corrupt memory on ISC.org;
"No way to prevent this" say users of only language where this regularly happens on Xeiaso.net.

Tor - which is becoming less and less trusted due to many design flaws and unfixed security issues - has also been impacted by a serious attack. 333 exit nodes on the tor network have been compromised and were used to do man in the middle attack on SSH connections. In case of success, the attackers were able to watch the commands and install backdoors.

Attackers Hijacked Over 300 Tor Exit Relays to Intercept SSH Connections from the Hacker News;
[tor-relays] Re: Ebury 1.8.3 on two Quetzalcoatl relay servers: timeline, lessons, open questions from the Tor project mailing list.

The French provider OVH has been victim of a supply chain attack, replacing the binaries of many versions of their CLI tool.

OVH supply chain compromise on Twitter.

The LiteLLM Python package has been compromised with a supply chain attack as well, injecting malwares via Pypi and Docker images.

PHP, NodeJS, and... Ruby. This language requires to be patched due to remote code execution vulnerability related to the Regexp module (integer overflows and double free issue). All versions of Ruby are impacted.

Regexp compilation fixes related to GitLab's CVE-2026-89078, CVE-2026-93577, CVE-2026-97229 and CVE-2026-110264 from the official Ruby news.

Kubernetes will also have a bad time, with 4 critical CVEs released very soon, including an issue on Azure Kubernetes, allowing remote code execution.

CVE-2026-83938 - Azure Kubernetes Configuration Service Elevation of Privilege Vulnerability

I don't even think it's done... And to be honest, I am not the only one to do a summary of all those security issues. Here few of them from Twitter:

While writing my article on the libsodium library port on Dart using FFI, I also checked the Erlang driver called erlang-libsodium. Sadly this package was kinda old (version 1.0.18+ released in 2019). Few issues and PRs have been opened to fix the library, but the project seems to be dead. Anyway, I wanted to use is to test the interoperability with the Dart implementation, and decided to fork it. libsodium has been updated to version 1.0.22 (the latest), the PRs have been merged, and I'm working on doing a lot of test cleanup. After that, I will - perhaps - ports the missing features from the latest release. The project can be found here: niamtokik/erlang-libsodium on Github.

What's happening in the economy right now is interesting -- and also terrifying. During the week, most of the big players just started to fall, as well as the Asian market.

The Semiconductor (SOX) Index was hammered 11.4% [...] Supporting the bursting Bubble thesis, the semi equipment stocks were under notable pressure. Applied Materials lost 19.3%, Lam Research 17.2%, and KLA 15.3%. Coherent dropped 25.5%, Lattice Semiconductor 17.8%, and Marvell Technology 17.2%. Nvidia slumped 6.1%, down a quick 12% in eight sessions. Broadcom fell 9.1% to a six-month low. Micron sank 14.6%, Intel 14.2%, and AMD 13.1%. South Korea’s KOSPI Index was hammered 14.5% this week, with a seven-session drop of 18.3%. [...] Japan’s Nikkei Index fell 5.4%.
-- Credit Bulletin Report

In fact, this situation is a direct result of something I saw many time in the past. Most of the current companies are not really companies anymore, but they look like sects. They are selling dreams, and faith to people without having an ounce of the knowledge or the experiences required to understand the bare minimum. You don't even need to give big proofs to make them pay, you know the aphorism "fake it until you make it". You just need one small miracle to make your company grows exponentially. The result? Some "expert" will also fall in the trap, and will start to preach. It will attract more people, blindly believing in them and in fine, in the technology sold by the company. The reality is harsh, and every actions produce reactions.

Investors are losing confidence that revenues from the AI boom will be sufficient to support the vast debt piles financing data centres, pushing bonds backing some of the largest US infrastructure projects into increasingly stressed territory.
-- Credit Bulletin Report

The recent message(s) from Linus Torvalds are hilarious. Torlvads has a strong reputation to be against a lot of new/modern technologies, but for LLMs, that's okay, that's just another tool. It's fascinating to see someone like him using LLMs and trashing other people opinions on this subject when you know the Open Secure AI Alliance joined the Linux Foundation in September. What kind of companies are present in this alliance? NVIDIA, Microsoft, Cisco, CrowdStrike, Hugging Face, IBM, Palantir, Red Hat and many others.

It's not done yet, nvidia is also giving money to Linux Foundations in many other ways. They are LVFS sponcors and they are supporting many other open-source projects (including the Linux Foundation). In fact, we don't even know the amount of money nvidia is giving to the Linux Foundation because it is not public. In doubt, follow the money, even a monk can be corrupted with gold.

Anyway, the LLM buzz is like the blockchain/crypto-currency one. They have both been created to do what fraudsters do the best: steal the money from your pocket by selling you a dream. The sects are doing the same with their victims. The result? What will happen when investors will discover all of these companies are selling snakeoil? They will simply ask their investment back. With $71 billions debt for Anthropic, and the small revenue generated by the company, it will become hard to explain that to investors.

In short: this is a bubble, it will pop one day, probably very soon (if it's not already the case). To my brainwashed IT colleagues, please stop be part of these frauds...

Coding

🏗️ DOOM on the Aspire PIXO — natively

System

📝 Introducing uncm(4): finally! a replacement for urndis driver working with recent version of Android! Say hello to uncm, providing support for the USB Network Control Model Ethernet devices.

📝 The hidden design compromises of Docker layers

📝 dbupdate(8) added to -current

🌐 exe.dev: Computers for developers and agents (Durable Sandboxes that are fast, secure, and sharable).

Database

🏗️ credcheck: provides few general credential checks, which will be evaluated during the user creation, during the password change and user renaming.

🏗️ pg_hint_plan: makes it possible to tweak PostgreSQL execution plans using so-called "hints" in SQL comments.

Security

📝 PwnEscape: Guest-to-Host KVM Escape (CVE-2026-99999)

📑 FloatDoor: Platform-Triggered Backdoors in LLMs

🌐 Private key compromise of eMudhra Intermediate Certificates: the thread from mozilla regarding the corrupted certificate.

🌐 eMudhra Intermediate CA Compromise: a longer post with solutions to apply after the CA corrupted certificate.

📝 How I got over 300 Tor exit nodes removed: a huge attack on tor network, where 333 nodes have been deployed to do MITM attack on SSH connections.

📝 Dumping Encrypted Viltrox Lens Firmware - two times

📝 One Parser, Two Bugs, Many Routers: Exploiting Broadcom WiFi Across eCos and Linux

🌐 ExploitGrid: Get notified when a CVE hits the software you run.

📝 Several Linux Kernel Vulnerabilities Accidentally Posted To Public Kernel Mailing List

📝 Scoop: Unvetted AI agents are accessing U.S. bank accounts

Security/cryptography

📑 Claude reveals ECDSA curve origins:

📝 How were the NIST ECDSA curve parameters generated?

📝 Can You Slop Out High-Assurance Cryptography?

Update/Upgrade

🔄 OpenBSD 8.0: another release of OpenBSD.

🔄 RabbitMQ 4.4.0

🔄 Phoenix 1.8.6

Misc

📝 Theo de Raadt: The Man Who Built the Internet's Locks

🐦 Meta just got hit with the largest penalty ever imposed on the company.: Meta will pay $25 bn for misleading Facebook users about how their data was protected.

Leaks of the week

French Communist Party: 183000 users impacted.

La France Insoumise: leaks about campaigns and organization.

ADMR: 4.5 millions people impacted, the leak include data about health and daily habits.

AMSAM: 922GB of data stolen including invoices, bills, many kind of internal documents about 400 people.

// Detect dark theme var iframe = document.getElementById('tweet-2113335587038167169-955'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2113335587038167169&theme=dark" }

Cover image by Matthew Hamilton on Unsplash

A review of the libsodium library in Dart

Mathieu Kerjouan — Thu, 22 Oct 2026 08:00:00 +0000

This article is a big sandbox based on the notes I took during my test of the sodium Dart package, an implementation of the libsodium (a well known C cryptographic library), itself coming from a fork of NaCl. libsodium was ported on many languages like javascript, rust, go, ocaml, haskell, erlang, elixir and many others.

sodium implementation tries to be a 1:1 interface to libsodium but with the object oriented paradigm in mind and the specificity of the Dart language. In fact, the few examples present in the source code and the documentation looks limited to me, and did not answer my questions. The test suite is huge, and a big part of the following code will be inherited from it, to avoid you, dear readers, to do the same annoying task. sodium can simply be installed using the dart pub add sodium command in your current project.

$ dart pub add sodium
Resolving dependencies... 
Downloading packages... 
+ archive 4.5.2
+ code_assets 2.1.0
+ csslib 1.0.2
+ freezed_annotation 3.1.0
+ hooks 2.2.0
+ html 0.15.7
+ json_annotation 4.12.0
+ posix 6.5.2
+ record_use 1.1.1
+ sodium 4.1.2
Changed 10 dependencies!

The following code can easily be copy/pasted, but you should probably not do it. Keep in mind those are simply examples for some use case. Cryptography is an hard topic requiring years of experiences to do something secure. So, be really careful with my code.

Bootstrapping

The sodium package offers 2 APIs, the "normal" one called Sodium and the "sumo" one, called SodiumSumo. In fact, it seems to matter ONLY if you plan to create a JavaScript/WEB application. In this case, you will need to chose the correct API depending of your needs and based on the API status table. In most of the case, if you plan to create a 100% native application, the SodiumSumo API should be enough. One can read more on the APIs differences from the README. To import the SodiumSumo API:

import 'package:sodium/sodium_sumo.dart';

If the normal API is required for some reason, simply use this header, and remove the Sumo part of the next command.

import 'package:sodium/sodium.dart';

Now, the sodium library needs to be initialized/loaded. Indeed, a global instance of this object is required, and can be created using the SodiumSumoInit class, with the help of the init() constructor. It should return an instance of SodiumSumoFFI object (an instance of SodiumSumo class).

var sodium = await SodiumSumoInit.init();
print("sodium: $sodium");
print("version: ${sodium.version}");
print("randombytes: ${sodium.randombytes}");
print("runtimeType: ${sodium.runtimeType}");

Let check that.

$ dart run
...
sodium: Instance of 'SodiumSumoFFI'
version: 1.0.22
randombytes: Instance of 'RandombytesFFI'
runtimeType: SodiumSumoFFI

Neat. It seems our main object is ready to do a bit of encryption/decryption.

PRNG

libsodium has been created with portability and security in mind. It embeds its own PRNG, but will the one exposed by the operating system if any. For example, on most of the Linux/Unix operating system, it will read the /dev/urandom file. If you are using OpenBSD (like me), it will probably use arc4random instead. The sodium Dart implementation requires an instance of the Randombytes object.

The first method to test will be the random() one, and it will return a random integer. In the code below, 8 random integers are generated.

final List<int> random = List
  .generate(8, (_) => sodium.randombytes.random());
print("random: $random");

random: [1516676843, 4096214122, 3177965578, 2291181405, 4245144458, 3289700635, 106475476, 1321601073]

The next method that can be used is to return an uniform random integer (inside a range) and is called uniform(). In the following code, 8 uniform random numbers are generated. The output should contain integer between 0 and 1023.

final List<int> randomUniform = List<int>
  .generate(8, (_) => sodium.randombytes.uniform(1024));
print("randomUniform: $randomUniform");

randomUniform: [322, 681, 185, 1000, 297, 457, 376, 872]

It is also possible to generate list of random integers using the buf() method. The code snippet below will generate 8 random numbers.

final List<int> randomBuf = sodium.randombytes.buf(8);
print("randomBuf: $randomBuf");

randomBuf: [241, 28, 11, 77, 198, 128, 177, 157]

libsodium is also offering a way to generate random numbers from a custom seed. It can be useful for many scenario, including testing. To do that the bufDeterministic method can be used, where the first argument is the length of the list returned, and the second is the seed.

final Uint8List seed1 = Uint8List.fromList(
  List<int>.generate(32, (_) => 0)
);
final Uint8List deterministicBuf1 =
  sodium.randombytes.bufDeterministic(8, seed1);
print("seed1: $seed1");
print("deterministicBuf: $deterministicBuf1");

final Uint8List seed2 = Uint8List.fromList(
  List<int>.generate(32, (_) => 1)
);
final Uint8List deterministicBuf2 =
  sodium.randombytes.bufDeterministic(8, seed2);
print("seed2: $seed2");
print("deterministicBuf: $deterministicBuf2");

deterministicBuf1: [161, 31, 143, 18, 208, 135, 111, 115]
deterministicBuf2: [61, 161, 61, 31, 87, 207, 88, 221]

The same seed should always output the same random number. This MUST BE set to 32 bytes (a list of 32 integers), if it's not the case, it will throw a RangError.

RangeError (seed): Only allowed value is 32, but was: 4

If you want to create test with reproducible output from sodium, the seedBytes attribute can be used.

Finally the stir() method is more or less related to any kind of custom PRNG that could have been created by the developer. It will reseed the generator, like defined in the libsodium documentation.

Generic Hash Function (BLAKE2b)

Hashes are another important tools to have when dealing with encryption, and authentication. The official crypto package created by the Dart team already offers a great support for the most used algorithms, including MD5, SHA1, SHA2 and HMAC (only for a subset of the algorithms previously listed). On the other hand, libsodium selected the BLAKE2b hash function to be its generic one.

BLAKE2 is a modern hash function specified in RFC7693. BLAKE2b is an optimized version of BLAKE2 for 64 bits platform and can create output from 1 to 64 bytes. The reference implementation in pure-C is available on the official BLAKE2 Github repository.

An hash can be created in two ways with sodium. The first one is by using the instance present in the sodium.crypto.genericHash as a function (see the call() method). The code snippet bellow will return the BLAKE2b hash of the string "test", with a 32 bytes (256 bits) output encoded in Base64.

Uint8List message1 = utf8.encode("test");
final hash1 = sodium.crypto.genericHash(
  message: message1,
  outLen: 32
);
print("hash1: ${base64.encode(hash1)}");

hash1: kosgNmlD4q/RHrwOri5TqTvxd6T881vMZNUDcE5l4gI=

The same can also be done using a Stream object via the stream() method.

final message2 = () async => utf8.encode("test");
final hash2 = await sodium.crypto.genericHash.stream(
  messages: Stream.fromFuture(message2()),
  outLen: 32
);
print("hash2: ${base64.encode(hash2)}");

hash2: kosgNmlD4q/RHrwOri5TqTvxd6T881vMZNUDcE5l4gI=

Both results are identical, like expected from a hash function using the same input. libsodium is also giving access to SHA-2 and SHA-3 hash functions. Unfortunately, it seems the Dart implementation is not offering an interface to them. If you need them, the crypto package will be required.

Short Input Hash Function (SipHash-2-4)

libsodium can use more specific hash functions. For example, when it comes to small input, a classic hash function can be a problem. It is useful for internal data-structure and algorithm. SipHash-2-4 is the default algorithm used here.

To create a digest, one can use the ShortHash object instance from the main sodium object, called shortHash. This object is implementing the call() and then, can be used as a function.

final key = SecureKey.random(sodium, 16);
final smallInput = Uint8List.fromList([0,0,0,0]);
final smallHash = sodium.crypto.shortHash(
  key: key,
  message: smallInput,
);

print("key: ${base64.encode(key.extractBytes())}");
print("smallInput: ${base64.encode(smallInput)}");
print("smallHash: ${base64.encode(smallHash)}");

key: tbaaEL+9ztfX000L54FlDw==
smallInput: AAAAAA==
smallHash: baRxJQlLnXg=

More information about this hash (and its usage) can be found on the official libsodium documentation.

Extendable Output Functions (Shake and TurboShake)

libsodium also offers Extendatable Output Functions (XOF), these functions can be configured to return a flexible output size. A Xof object instance of each of these algorithms can be found on the main sodium object instance created during the initialization part:

xofShake128 for Shake128 algorithm;
xofShake256 for the Shake256 algorithm
xofTurboshake128 for the TurboShake128 algorithm;
xofTurboshake256 for the TurboShake256 algorithm.

Let have a look at the Shake128 function first. An instance of a can be found in the initialized sodium object created at the beginning of this post.

final Uint8List xofShake128Input = Uint8List.fromList([0,0,0,0]);
final Uint8List xofShake128Output8 = sodium.crypto.xofShake128(
  message: xofShake128Input,
  outLen: 8,
);  
final Uint8List xofShake128Output32 = sodium.crypto.xofShake128(
  message: xofShake128Input,
  outLen: 32, 
);  
final Uint8List xofShake128Output64 = sodium.crypto.xofShake128(
  message: xofShake128Input,
  outLen: 64, 
);  
print("xofShake128Input: ${base64.encode(xofShake128Input)}");
print("xofShake128Output8: ${base64.encode(xofShake128Output8)}");
print("xofShake128Output32: ${base64.encode(xofShake128Output32)}");
print("xofShake128Output64: ${base64.encode(xofShake128Output64)}");

xofShake128Input: AAAAAA==
xofShake128Output8: j5cMpCyEnKA=
xofShake128Output32: j5cMpCyEnKAMSPiNiDgMV0j0Lc9SkIgAHcUmt12xu74=
xofShake128Output64: j5cMpCyEnKAMSPiNiDgMV0j0Lc9SkIgAHcUmt12xu74TNSTXBo+lx6AI6GKH1Mc7+qGNds7/V7XPqcJ1ArqOvA==

As you can see, the output is flexible, and the first part will always be the same (a propriety of XOF). Let check that with the TurboShake256 algorithm.

  final Uint8List xofTurboshake256Input = Uint8List.fromList([0,0,0,0]);
  final Uint8List xofTurboshake256Output8 = sodium.crypto.xofTurboshake256(
    message: xofTurboshake256Input,
    outLen: 8,
  );  
  final Uint8List xofTurboshake256Output32 = sodium.crypto.xofTurboshake256(
    message: xofTurboshake256Input,
    outLen: 32, 
  );  
  final Uint8List xofTurboshake256Output64 = sodium.crypto.xofTurboshake256(
    message: xofTurboshake256Input,
    outLen: 64, 
  );  
  print("xofTurboshake256Input: ${base64.encode(xofTurboshake256Input)}");
  print("xofTurboshake256Output8: ${base64.encode(xofTurboshake256Output8)}");
  print("xofTurboshake256Output32: ${base64.encode(xofTurboshake256Output32)}");
  print("xofTurboshake256Output64: ${base64.encode(xofTurboshake256Output64)}");

xofTurboshake256Input: AAAAAA==
xofTurboshake256Output8: /kSBXrzATNM=
xofTurboshake256Output32: /kSBXrzATNNdtXhOHkidRkpZ9IOK3jeHTWo+Zr1/pfg=
xofTurboshake256Output64: /kSBXrzATNNdtXhOHkidRkpZ9IOK3jeHTWo+Zr1/pfj20vK3er1FEbuD12Chmf11GTTi8VVe+OPfW3Tom3cI4w==

More information regarding these algorithms can be seen on the libsodium xof documentation.

Password Hashing (Argon2)

libsodium offers an implementation of the Argon2 algorithm, it can be an alternative to the argon2 package. Argon2 is a key derivativation function designed to be protected against brute-force; At this time, this is the recommended algorithm to store hashed password. libsodium gives us 2 versions of the algorithm, Argon2i and Argon2id, this last one is recommended by the OWASP team, but the default one used by sodium is Argon2i. Anyway, let do some test here.

To create a new hash, the already created instance of the Pwhash object in sodium.crypto.pwhash can be used and the str() can be invoked. It will return a formatted string containing the functions parameters and the random salt generated automatically. The opsLimit represents the amount of computations to use (more CPU usage) and the memLimit represents the amount of memory in bytes to allocate.

final String password = "secret_password";
final hashedPassword = sodium.crypto.pwhash.str(
  password: password,
  opsLimit: 1024,
  memLimit: 8192,
);  
final resultValid = sodium.crypto.pwhash.strVerify(
  passwordHash: hashedPassword,
  password: password,
);  
final resultInvalid = sodium.crypto.pwhash.strVerify(
  passwordHash: hashedPassword,
  password: "test",
);  
print("hashedPassword: $hashedPassword");
print("resultValid: $resultValid");
print("resultInvalid: $resultInvalid");

hashedPassword: $argon2id$v=19$m=8,t=1024,p=1$PfWonIyaTiDh0mhyUT55UA$Y63dvkUjVtKYnvBSdzstDN0QIhyi6xcba83efmMaa7I
resultValid: true
resultInvalid: false

A Pwhash object exposes properties to know the limitations of these values, here a list of some of them.

If you want other flavor like Argon2id13, the callStr() can be used instead but it will require new parameters, like the length of the output, a random salt and the name of the algorithm to use.

final Uint8List salt = Uint8List.fromList(sodium.randombytes.buf(16));
final hashPasswordBis = sodium.crypto.pwhash.callStr(
  outLen: 1024,
  password: password,
  salt: salt,
  opsLimit: 1024,
  memLimit: 8192,
  alg: CryptoPwhashAlgorithm.argon2id13,
);  
print("salt: ${base64.encode(salt)}");
print("hashPasswordBis: ${base64.encode(hashPasswordBis.extractBytes())}");

salt: NAf4uZwlge7mm1kwT5pTZw==
hashPasswordBis: e47DIsAic0Tx9S1g/gIItsrCcf0WpLEaiIj8Jbzn1cneVfPz8w7PH50WmEbj2+kPUZFxgtZdyfRVsjP/N8l521BMxwxAA/hPWFz9a5qM4+BhaKFStMVl5JHQnq+P48ucYTTH7dNwoXe6KuyvR4/gI2kYYdzYQOnymeOzdZxl9Ipz85CfFSJD4S40zuGGLqzL/He2Edabnny1W2PFfBHQoBP1k7egv/b2ZoHVayAWyxOaRHbQ5wrsVIRzliKmG9x8bgDyq/sqP/M+MV2cdD8h5spPq6PFq88+KKAuXD4XORbnSocmCw84oqaOcscubsEcPOnMVhJYEAJIRgytDTlpHjgWNykNahQAkiLBp+UkLQx+pmVAwDG47zyTsEQmyjeAgVL+yVqRFPfT6m5hKJLVXC7wrttR/flDqh+TUgVrZ3FS6rQBOaJA6E4yc9IC/JSVO95UrsBRGABYd5K9tYJccOrEpbb3Rx69dPKw66RWUn8WIxeRB0yf+BiqD1xx6Mqp4axRmvSFTZmLV3Pz9fkXZqYDjGpucytLevUDmumtoHbP3dobuRk0lxhIIFDLfqx2cvymumnByePpwmnTC+1AXw83EX+NBMOzKcXPGVepGONNt0HnT3riW99+kBmRDDPmeHINT7I3k4VsAcWexRxdRHuGCbg9CjxQ4kzLD5DWdibS+FtivhR8D2UWgaH3QZGmNxYpslyhhzCNVcNGO+vgNVH3omyUh7gAbC/oMCL0j1u9u3he4sXBKfCERs5YACIHB7YHVEFTvgZHFfk78SNq4zyNQWfPeMlTGVLrshsOM5BjcTuUg8EJ6AnwiUVkygdSYih1wIX48jwprPW+K53SEig2exm4S6jbl4L65W/+9Ciaue9gobGQVN7zlL2+H9FopLdU+ieMXQJhIGemVZqL3+Geq1KoOm2H4ufJ9olyNbzry8cxZQpgFSdIAfqicgIYeJpBmH8Rdn4C0J9L/I6khDE+Ew63oVuw/Uk6xahYUnQDtveQG86A13F+p459DiSxQR7sJjNvUuSlKHkFWLF9qTU4wGmgzoiBIHCro9KZGegKA2H92kdQP1j+EhPxBeeeiQu/X59Miq+hDiMm0sqt4bIOra+lKBJJ8/T/Ly6VHU6NYvt0JozqazXkmFSh6rj0tfOWuZqmlRpn+OQPoWnLvPurm3iIVo+q3OS9QnZcpCKgrxQr3aJ3NBI5PnoYxslnWJUjwC9FCRTesYqmrMXjk09XCFmF+p96VqhYTXJ1hB3yu70fgFcARvY4jJocnqTfN9UBzGvWRWjK+lZDc/4imQRrbMlCzKxSVgnQKHZRjbG/xpMramlZxqn9WJ8SC3vgI2nZN7RpmCOrm9B7AAboNw==

As you can see, the output is an Uint8List and not a formatted String. Unfortunately, in this mode, it is not possible to generate the Unix-like format. The libsodium official documentation contains more information on this feature.

Symmetric Encryption (chacha20-Poly1305)

sodium is offering 3 algorithms, all of them based on the Aead class

A message can be encrypted using the encrypt() method.

final Uint8List message = utf8.encode("my message");
final Uint8List nonce = Uint8List.fromList(sodium.randombytes.buf(8));
final SecureKey keyAead = SecureKey.random(sodium, 32);
final encrypted = sodium.crypto.aeadChaCha20Poly1305.encrypt(
  message: message,
  nonce: nonce,
  key: keyAead,
);
print("message: $message");
print("key: ${base64.encode(key.extractBytes())}");
print("encrypted: ${base64.encode(encrypted)}");

message: [109, 121, 32, 109, 101, 115, 115, 97, 103, 101]
key: GkjKYnsKNITUsROWYOiujw==
encrypted: SsldZX2XkQqcRTgiqyP0KH97Yz/YkQ1BIE8=

The ciphertext produced by the previous method can be decrypted using the decrypt() method.

Uint8List decrypted = sodium.crypto.aeadChaCha20Poly1305.decrypt(
  cipherText: encrypted,
  nonce: nonce,
  key: keyAead,
);
print("decrypted: ${utf8.decode(decrypted)}");

decrypted: my message

Nothing really fancy here, the interface is quite easy to use. Be careful regarding the nonce and the key to use though, both of them are important and must be configured accordingly.

Public Key Generation

This post was - at first - created to understand how public key generation was done with sodium. Indeed, the Dart community does not really offer a lot of choice when it comes to encryption, and instead of using yet another library created from scratch (this is usually not a good idea in cryptography), the port to libsodium looked way more safer and interesting.

Anyway, libsodium is only supporting Ed25519 and X25519 algorithms. The first one is for authentication (signatured), the second is for key exchange. In both case, the method is the same to generate a pair of keys and then, return a KeyPair object.

var seedBytesPK = sodium.crypto.box.seedBytes;
var secureRandomSeedPK = sodium.secureRandom(seedBytesPK);
var keyPair = sodium.crypto.box.seedKeyPair(secureRandomSeedPK);
print("seedBytesPK: $seedBytesPK");
print("secureRandomSeedPK: $secureRandomSeedPK");
print("keyPair: $keyPair");
print("publicKey: ${base64.encode(keyPair.publicKey)}");
print("secretKey: ${base64.encode(keyPair.secretKey.extractBytes())}");

seedBytesPK: 32
secureRandomSeedPK: Instance of 'SecureKeyFFI'
keyPair: KeyPair(publicKey: [29, 132, 192, 107, 208, 25, 162, 211, 228, 70, 109, 169, 238, 124, 161, 248, 143, 24, 82, 64, 86, 39, 93, 254, 93, 54, 113, 62, 30, 145, 73, 50], secretKey: Instance of 'SecureKeyFFI')
publicKey: HYTAa9AZotPkRm2p7nyh+I8YUkBWJ13+XTZxPh6RSTI=
secretKey: /8SV7xv0ReOpTgau97cmmGZM5PdKSA13hbfxYYs0F1Q=

A keypair can also created using a seed via the seedKeyPair method. More information can be found in the Box class documentation.

Public Key Signature (Ed25519)

libsodium is using Ed25519 to sign messages with a private key. 3 modes of signatures have been defined by this library, detached mode, combined mode and multi-part messages mode. All of them have been ported to sodium via the Sign class.

Before using these modes, it is also necessary to understand the key generation process. One can sign a message by generating a new random key when calling the keyPair() method, or derive a new key from an existing one using the seedKeyPair() method. For example, to derive a key from the keypair generated in the previous section:

var newSeedBytes = sodium.crypto.box.seedBytes;
var newSecureRandomSeed = sodium.secureRandom(newSeedBytes);
var newKeyPair = sodium.crypto.box.seedKeyPair(newSecureRandomSeed);
var derivedKeyPair = sodium.crypto.sign.seedKeyPair(newKeyPair.secretKey);
print("newSeedBytes: $newSeedBytes");
print("newSecureRandomSeed: $newSecureRandomSeed");
print("newKeyPair: $newKeyPair");
print("derivedKeyPair: $derivedKeyPair");
print("derivedKeyPair.publicKey: ${base64.encode(derivedKeyPair.publicKey)}");
print("derivedKeyPair.secretKey: ${base64.encode(derivedKeyPair.secretKey.extractBytes())}");

newSeedBytes: 32
newSecureRandomSeed: Instance of 'SecureKeyFFI'
newKeyPair: KeyPair(publicKey: [17, 113, 139, 202, 227, 188, 47, 55, 65, 155, 129, 183, 29, 22, 170, 28, 186, 137, 114, 114, 10, 130, 235, 100, 124, 126, 64, 156, 63, 99, 192, 108], secretKey: Instance of 'SecureKeyFFI')
derivedKeyPair: KeyPair(publicKey: [12, 10, 30, 205, 0, 239, 115, 167, 34, 206, 106, 197, 126, 152, 34, 58, 162, 79, 75, 239, 76, 73, 249, 136, 211, 66, 86, 31, 187, 16, 201, 200], secretKey: Instance of 'SecureKeyFFI')
derivedKeyPair.publicKey: DAoezQDvc6cizmrFfpgiOqJPS+9MSfmI00JWH7sQycg=
derivedKeyPair.secretKey: S1dqHYqlbEO39PGZM1rY6smO/34FgEGKw7cq2G/HNy4MCh7NAO9zpyLOasV+mCI6ok9L70xJ+YjTQlYfuxDJyA==

The combined mode is prepending the signature to the message. To sign a message with this mode, the call() method is used, and to validate the message, the open() method is required.

var messageCombined = utf8.encode("test");
var signatureCombined = sodium.crypto.sign(
  message: messageCombined,
  secretKey: randomKeyPair.secretKey,
);
print("signatureCombined: $signatureCombined");
print("signatureCombined: ${base64.encode(signatureCombined)}");

var verifyCombined = sodium.crypto.sign.open(
  signedMessage: signatureCombined,
  publicKey: randomKeyPair.publicKey,
);
print("verifyCombined: $verifyCombined");
print("verifyCombined: ${utf8.decode(verifyCombined)}");

signatureCombined: [175, 80, 179, 20, 23, 55, 168, 83, 169, 53, 225, 232, 239, 10, 9, 101, 53, 236, 192, 245, 177, 194, 17, 42, 195, 100, 161, 72, 62, 172, 251, 39, 160, 26, 90, 165, 49, 158, 220, 169, 183, 149, 174, 209, 44, 85, 214, 189, 62, 246, 93, 249, 215, 40, 54, 32, 240, 150, 56, 78, 93, 23, 101, 7, 116, 101, 115, 116]
signatureCombined: r1CzFBc3qFOpNeHo7woJZTXswPWxwhEqw2ShSD6s+yegGlqlMZ7cqbeVrtEsVda9PvZd+dcoNiDwljhOXRdlB3Rlc3Q=
verifyCombined: [116, 101, 115, 116]
verifyCombined: test

The detached mode produces a signature without altering the message. To sign the message, the detached() method is invoked, and to verify it, the verifyDetached() is called.

var messageDetached = utf8.encode("test");
print("message: ${base64.encode(messageDetached)}");
var signatureDetached = sodium.crypto.sign.detached(
  message: messageDetached,
  secretKey: randomKeyPair.secretKey,
);
print("signatureDetached: ${base64.encode(signatureDetached)}");

var verifyDetached = sodium.crypto.sign.verifyDetached(
  message: messageDetached,
  publicKey: randomKeyPair.publicKey,
  signature: signatureDetached,
);
print("verifyDetached: $verifyDetached");

message: dGVzdA==
signatureDetached: r1CzFBc3qFOpNeHo7woJZTXswPWxwhEqw2ShSD6s+yegGlqlMZ7cqbeVrtEsVda9PvZd+dcoNiDwljhOXRdlBw==
verifyDetached: true

Finally, the multi-part messages mode is used to sign/verify huge messages and is using a Stream in the Dart implement. To sign a message in multi-part messages mode, the

var messageMulti = () async {
  return utf8.encode("test");
};
var signatureMulti = await sodium.crypto.sign.stream(
  messages: Stream.fromFuture(messageMulti()),
  secretKey: randomKeyPair.secretKey,
);
print("signatureMulti: ${base64.encode(signatureMulti)}");

var verifyMulti = await sodium.crypto.sign.verifyStream(
  messages: Stream.fromFuture(messageMulti()),
  signature: signatureMulti,
  publicKey: randomKeyPair.publicKey,
);
print("verifyMulti: $verifyMulti");

signatureMulti: KqgvmypphuUC0aLzZCQ56rQQ2CwidxdQFoBBlVDqLhvqiyQKZSbBvq76UsChFy9GLm+rM6ncBocevJHTthMNBg==
verifyMulti: true

Don't hesitate to check the API documentation, especially the SignSumo class and the call() method.

Public Key Encryption

libsodium offers an easy interface to encrypt data with via a public key. In fact, it will use the public/private key to authenticate the message while a secret a shared between both part of the communication. The messages are then encrypted using the XSalsa20 stream cipher. Key pairs can be generated using the keyPair() method from aBox object instance.

var kpAlice = sodium.crypto.box.keyPair();
var kpBob = sodium.crypto.box.keyPair();
print("kpAlice: ${base64.encode(kpAlice.publicKey)}");
print("kpBob: ${base64.encode(kpBob.publicKey)}");

kpAlice: UfVCLUDk7+W/8hdnGvGWXUAazrzhDmGJvx1b14ZTtRc=
kpBob: U9zxXl5KfUO3XcKNoLeYg/hZawnzjMIaLQ8kf01pNF8=

Then, the easy() method can be invoked to create an encrypted payload containing all required information for the exchange.

var msg1 = utf8.encode("hello Bob!");
var nonce1 = sodium.randombytes.buf(24);
var cipher1 = sodium.crypto.box.easy(
  message: msg1,
  nonce: nonce1,
  publicKey: kpBob.publicKey,
  secretKey: kpAlice.secretKey,
);
print("msg1: ${base64.encode(msg1)}");
print("nonce1: ${base64.encode(nonce1)}");
print("cipher1: ${base64.encode(cipher1)}");

msg1: aGVsbG8gQm9iIQ==
nonce1: ACpqcCbUEGfeUtqF4qhjJhKLZ/56mJHh
cipher1: t0dIKRIlLxTX3Cp656DN/XgRy8rka9GApsw=

After the reception of the message by the receiver, the clear text can be extracted using the openEasy() method.

var recv1 = sodium.crypto.box.openEasy(
  cipherText: cipher1,
  nonce: nonce1,
  publicKey: kpAlice.publicKey,
  secretKey: kpBob.secretKey,
);
print("recv1: ${utf8.decode(recv1)}");

recv1: hello Bob!

Another exchange can be initiated using the same principle (for example to send a response to the previous message).

var msg2 = utf8.encode("Hey alice");
var nonce2 = sodium.randombytes.buf(24);
var cipher2 = sodium.crypto.box.easy(
  message: msg2,
  nonce: nonce2,
  publicKey: kpAlice.publicKey,
  secretKey: kpBob.secretKey,
);

msg2: SGV5IGFsaWNl
nonce2: XqvB6EwaF9GxPvSSM/TMuSkaXLmUfnDF
cipher2: q0fyatMRoWChE17eKUouJinJTqahCa8P3g==

var recv2 = sodium.crypto.box.openEasy(
  cipherText: cipher2,
  nonce: nonce2,
  publicKey: kpBob.publicKey,
  secretKey: kpAlice.secretKey,
);
print("recv2: ${utf8.decode(recv2)}");

recv2: Hey alice

libsodium is also offering a detached mode, it will give more flexibility when sending the information, instead of using a complete payload, the required information can be put in some other formats. To use this mode, the detached() and openDetached() methods can be used.

It is also possible to use a Sealed Box with the help of the seal() and sealOpen() methods. The idea is to make the exchange even more secure by always creating an ephemeral key.

More information can be found on the official libsodium documentation.

Key Exchange

libsodium also offers some facilities to do key exchanges and key derivation. A keypair can be generate using the keyPair() method from an instance of a Kx object or by deriving an existing key using the seedKeyPair() method from the same instance.

var kpClient = sodium.crypto.kx.keyPair();
var kpServer= sodium.crypto.kx.keyPair();
print("kpClient: ${base64.encode(kpClient.publicKey)}");
print("kpServer: ${base64.encode(kpServer.publicKey)}");

kpClient: dPmBU3994tkERAjQUyD+ibo7rBFujq+ZV197l8FvwVU=
kpServer: 52FiQPJtOJkkBDoRoZYzC3ASjJDFsHu6SbeEP3jrswg=

A session can be started by sharing one's public key, signed with its private key, and by sharing a secret encrypted with the target public key. To do that, the clientSessionKeys() method can be used. It will return a SessionKeys object, containing the session keys to decrypt received data (tx property) or encrypt (tx) data to send.

var session1 = sodium.crypto.kx.clientSessionKeys(
  clientPublicKey: kpClient.publicKey,
  clientSecretKey: kpClient.secretKey,
  serverPublicKey: kpServer.publicKey,
);  
print("session1: $session1");
print("session1.tx: ${base64.encode(session1.tx.extractBytes())}");
print("session1.rx: ${base64.encode(session1.rx.extractBytes())}");

session1: SessionKeys(rx: Instance of 'SecureKeyFFI', tx: Instance of 'SecureKeyFFI')
session1.tx: vzJw72Dib/qHhze9LOsUGwYxjFYNzWV3D6541pLMxZc=
session1.rx: r7QtsYaSFrydDg6ijk1mbNrvR3cVSJmFqMXVH3m2YV4=

The previous example was using the client API, but a server can also take the lead by creating its session via the serverSessionKeys() method. The same kind of object is returned.

var session2 = sodium.crypto.kx.serverSessionKeys(
  serverPublicKey: kpServer.publicKey,
  serverSecretKey: kpServer.secretKey,
  clientPublicKey: kpClient.publicKey,
);  
print("session2: $session2");
print("session2.tx: ${base64.encode(session2.tx.extractBytes())}");
print("session2.rx: ${base64.encode(session2.rx.extractBytes())}");

session2: SessionKeys(rx: Instance of 'SecureKeyFFI', tx: Instance of 'SecureKeyFFI')
session2.tx: r7QtsYaSFrydDg6ijk1mbNrvR3cVSJmFqMXVH3m2YV4=
session2.rx: vzJw72Dib/qHhze9LOsUGwYxjFYNzWV3D6541pLMxZc=

Creating encrypted/authenticated sessions can be challenging, more information about this part can also be found on the official libsodium documentation.

Public Key Encapsulation

Secrets can also be shared using key encapsulation mechanism. The idea is to share a secret with someone using public key cryptography. Firstly, let generate a key pair with the keyPair() method from a Kem object instance.

KeyPair kpCarl = sodium.crypto.kem.keyPair();
print("kpCarl: ${base64.encode(kpCarl.publicKey)}");

kpCarl: X0w77TyQOXGrhBQV/ypBYTYt6zx7kniSE4ttD1d+qKa8Wyt7nKZJXjFeAiI2tKh2KBHLR9J5QHSPOvIVy5q5ZXEVpnJL4rivkEcE+MKeqbGvsvaYv2iK3sN96ejMnWSMHBABucw+oJMMHUcnD5sHgLKtD6tDjwJkjnoIVlmNShzPiUHPinhdRIdPpEizBQsAYBo40Qwo0nAph8l6T7VoyOOCaLHJotGj/2Z3P7cqU6UUOIq6kUBkjPlKJ3e4Ojgd4whmNfg6HEQuxQvCMxfAyOWSKaZE9cETVDIy4WWIX1VDjmWNGISlEtaMwmdSbnVJB4pWNbAhbCZeJ3pWZoFP71Gy5yeH0tme+tg9S3C0CauNM5ZuevhVHfVeERZob1Ig7pEIRgB04qAW8kYlD9ZiMhWH/+PK8mQu3XxOiWUoCii3xdbLAsVvJfEUiAHDzzhzs4oerHe4WOEVGPAqjaGAvfNOEJYbC4NXL/NhRvenYcutteqMuKFDVuYo2gUZN9nEX/wzM2qNoLO2ByEi8kUkOmQgMXo8F3Y+P0woNZS6jYaBTip4ocuIP8N4YYRSEciJi+FcAK1PPGQ7kWNsNPFv2LVw8xl2DTFx8glsMsACz7wiSQgdWbkHzRkizKcLwSdYYdefz1FmaFaDL4wsXsAzjhaXvedG3Oou7OJtqHK9OwTL/DCWUZEavMCqqaqWYsxJZbol0qdhEWWL5hexm1GGzmYnCPNdsusKOOCItFUwQfB5v9qbBKCWOVBEN/dJ1scaT3cXhZaqrphT45R1EKgVKXCIwVxifDd7P9FwGlV7b1Acpfao00xtpha7DabC8NZkSoiHX2V8PnwveHUjBDqZUTIhlSZO71R2sBbB8kTC0YiU2PKgBspsf0ywenRHR8N0cFygn3a4LOCKMTd9kKUlbWbAi2o1QmtaaUwd7KfAXfShSwdt+murTOSgVjWi87sfhONml7tC4qDFo6SF5WI1nRFik5YexptOgteuRgVWkCpmVPJF1UJRMtaobzjDpGm+pfl2jDIHYml+XHMcHCkKy6KCnbxSj1MaAAyZLZI1nFbKHYpvYOLAsjQE1adf+TsfaOC7ydZ9ILp4QcxfzAFxlpiqVSyRmMiXhTdc4xQBoUEZnFM8IVcuy/Jec+pe1heu/UnA1zw0MtoB6SQ616RkAeq6zmuoRbKt/GgoQvyQq+tLfakFPcNAsdUU3/p/GeVkjXQa2Qx7M/osyrKmiUJvN7pNZ5lIe/VC9zpadLZnHoIODICIgqaIG4QFxLOdiTHP9hKB7dA4SmZxgJOn0Rl1nDBRAlIgvmiVW9lrPkO3cRGdDkE/3dGQppinn5OFxgjOtHFRJVi037hmuDaRZmW/OIiV8VCuTbkeCNUg3sUix8mxtxG0GrVsJypvMHY1DNWSV2soK4Mk/IjLUdUWImhhUeCudMBNaPINYiEJkQpd67g+YkuRXOlBy9djpRQjTQtaITVvwbbG3as9Q5RlunktEOQvqJg6zMK7i1Be/HFI4SLKqtxQg5ePYQhHIbUzppd1sweNpuDs+9XxRsxZKtUb2KjYeoKiltPFRojgHOr5Ik6yXD67oxPVMorCn3gZv8mMrPu5cg7vXiWnxXDuncdHEg==

The public key can be then encrypted using the desired public key (the following example is wrong, the public key to use should have been another one, but the idea is the same). To encrypt the secret, the enc() method can be used, it will return a ciphertext that can be send to to the recipient.

KemEncResult enc1 = sodium.crypto.kem.enc(
  publicKey: kpCarl.publicKey,
);  
print("enc1.ciphertext: ${base64.encode(enc1.ciphertext)}");
print("enc1.sharedSecret: ${base64.encode(enc1.sharedSecret.extractBytes())}");

enc1.ciphertext: GgIpamj4uT72RQi1PYs8wyr5DJWGW2wAsU1tBxQIBPF4d4xke/Q3fL16FE0v5dLlF7Esvoq9DBVxSMLuGLM0xM95FU9Hz/nup6AZ2hpJEJLnzWWyw8lTMUmVmgttiKPiFCRGlaPeoZZCnofy7ccbPnST+0w3Pb0pknaxmAF6y4x6Q+Kp/u+1jbhRtx49Lb4gIWDqH3UAxWke5srjzyGnQkLBHtwsh1KIHtpFuk4KWcc5wHgZRfGHGTrim6bUZXoD6hz4KebEE4bNNEZtYDJyyME4p3sPKTLuvIBbUfJKfn3bl8rjbWBeTgDubFBoPs37PPIvg9xsJtdBRoynVoSEG6njTyNU3ZQNxbYTPgi5tVbwPpTkBwDaDc9roypNyu6QiBFVxrtSi1brsfQAeWXy2qO3PNySHhO0mcvpz27VmKE+UbaB+i93IkosxpD3TndnSkEODb0Dwzq544oTv588Qk1n02sTbEgIqofTFNjeJfyHJ3VVM/zEwKxA7lX9ud214XVZpzfLKj7AkoXlpX/0unZYUuyfyr/ooUZkvOu5cA9MBYiowZMTPNWddMkcRxCz2mRGFyZyiAGu6Xqwz8Mpv3b7Ci5r8OEZaOewke2d9XqXRSjkTEYi7hy/dVYSrX2e1STvfKxPdqtoNXCyfhg9IE+v8C0qcYgh4b34EDC1qg8BITUJl7KFFw111Qm+5B+BipVCjyttHQJUO6OoqMsDztc0NlO2kY37bI3mlU7SgHVTFEReqMHMpoIOoNWcPF6V7FwksllBkjdG5XR7bF6c0ShhQtrJViJv+wV0u7I4LtYCuHKoKnFM2+QEWMjJyrk5jpSErI3g2izscF/ku94wC3JeYUyou9gcz0u4xqMdoPBPdahZRjinmF6MzK5pY0mRhnbf6rLVkoxe5NCexmO4+ZddTEEivMx+ZCb+IYLfYfhIiWLMYYUni6MEs0hXTAv4ylS5u2JH0Ke/2YYsZ/Dj3g/zrMNPemp3IDLRSxvjWdZcF6oxP+X+tlOY/UBAiWpKFXPCTUthlH9u2KRNQX8inl4wQjqurXG4oAKqHl5SWEvTIaw6otfawZMffQIe4LjNLS/OCVZIchoFXU1KBxZX1w1Z8OW9zDIj17zYCYy5Gt+Rf0oYmqtjzq58/jQ+oggkPgj1Ey05b5XTlN2RFaID2IRJFNUMU/C0nRbpHcQJ36pS0by8UgWSHRY8Umb8u4dh6+k5zXM2X+1wEtf5SV/yN6dr7L11N7X1rHhrUuLf68Tvpuqnb0ecb3b0V5RziRC7J+NpvndDXEyqKXvZJJzOReq+mXLuc0J6qGdGpzKu7IqQwX0iKrlcTWrMNzXR667PrMnSxgMQJx7CeWHigXJGwWhWck5vhiNpy7YAZL3Awk153Zqgs0sJ+nSPVpSqkuLe1f1whfLxbiyMMQvE3KfXzYR4JY7hy5AkhKWXpehGaO+mtbxxN7+HK7n5ggKl74RO/Pl/+pPWtpEs8KfB8CY+Ng==
enc1.sharedSecret: LV6LjDbXe1v33jkOLx1YPAktFzgj2wAvQ+ns3RT6CPs=

The recipient receive the message and can decipher it using the dec() method. It will return the shared secret, previously generated.

SecureKey dec1 = sodium.crypto.kem.dec(
  ciphertext: enc1.ciphertext,
  secretKey: kpCarl.secretKey,
);  
print("dec1: ${base64.encode(dec1.extractBytes())}");

dec1: LV6LjDbXe1v33jkOLx1YPAktFzgj2wAvQ+ns3RT6CPs=

Most of the time, the secret will be derived using a KDF, to do that, one can use HKDF or the deriveFromKey() method from a KDF object instance.

SecureKey derivedKey1 = sodium.crypto.kdf.deriveFromKey(
  masterKey: dec1,
  context: "test",
  subkeyId: BigInt.from(1),
  subkeyLen: sodium.crypto.kdf.bytesMax,
);  
print("derivedKey1: $derivedKey1");
print("derivedKey1: ${base64.encode(derivedKey1.extractBytes())}");

derivedKey1: Instance of 'SecureKeyFFI'
derivedKey1: n1a5LPoQ80DNK0Sd3wwf4WrcjtELvPhGF6O58sM6YDk7izyipRBLpsXls42vqZNzEOrB27xuPgplF4YRIgJIDQ==

Don't forget to check the official libsodium documentation to have more information and study their examples.

Conclusion

libsodium, like most of the cryptographic libraries, is not easy to use. Why using sodium when other packets can offer a simpler interface? Well, because of its portability, working almost everywhere. To be honest, this library was also in my toolbox for many years (the javascript implementation), and was used on many of my personal projects, but the Dart interface feels... different.

A big part of the functions available on libsodium have been ported via FFI, and it's really a good thing to use modern cryptographic algorithms on a modern language. Some parts are missing though, for example, the function to encode correctly a password hash in the Unix-like format. The source code is available from libsodium, but it would have been nice to have it in the Dart implementation.

I hope I will have more time to write a better post, with real use case.As usual, if you want to know a bit more about all of this:

sodium package in Dart using FFI on pub.dev, where you will find the global information about the Dart implementation;
sodium examples in Dart on pub.dev, where you will find few examples;
sodium source code on Github, where you will find the source code of the Dart package;
sodium test suite on Github, where you will find the test suite, quite useful to see more examples;
Official libsodium documentation, where you will find the original documentation for the libsodium;
libsodium source code on Github, where you will find the pure-C libsodium source code, especially useful if you want to check the vectors and other test suite;
Official NaCl website, the original project from cr.yp.to, you should probably check, very instructive.

Always be careful when using cryptography in your code, try to reuse existing patterns and avoid creating your own stuff, it's most of the time a bad idea.

But as always... Happy hacking and have fun!

Cover image by Maria Petersson on Unsplash

WeeklyReview#202642

Mathieu Kerjouan — Mon, 19 Oct 2026 10:00:00 +0000

While reading "No way to prevent this" say users of only language where this regularly happens, yet another complaining about C code, I was thinking... During the past 20 years, many solutions have been created, but only a few C developers are using them. More recently, the fil-c was created to make C memory safe, hopefully it will be used in the future. The compilers are way better than 20 years ago, and for example, it is now possible to easily export the AST of any C project with clang/llvm. Even more, many compilers are now offering their own tools like static analyser to check the code. In fact, I was currently using few them to test my own pet project sandbox, and it's amazingly pleasant to code with all the flags and all those tools. So, why should we rewrite everything in rust, go or any other "modern" language if the AST is available, and we have ways to convert existing C/C++ code into a safer binary version?

Let's go back to the article, the CVE-2026-92392 is an use after free bug, a classical mistake anyone do at least one time in its developer life. fil-c fixes that, but let's go a bit deeper. C is known to be an interface to low-level language (assembly) and will let you do whatever you want, it means the memory management must be done manually, like any other resources. It's a complex task, and can become a nightmare when dealing with concurrency. Why not using a pattern above that, always in charge to deal with those resources... Oh, wait, it's a kind of garbage collector, right? Or perhaps a kind of state machine, where the state is a resource that can be opened, used or closed. Anyway, adding another layer will increase the complexity of the existing code, but using something like fil-c could really help a lot.

A part of my free-time was dedicated to find design inspiration with Flutter. Thanks to Flutter Awesome, a long list of open-source Flutter projects can be checked! Here a quick list with the ones I liked the most

flutter_social_chat: A simple chat compatible with iOS and Android. The design is really cool.
OstrichGram: a desktop (linux/windows) Nostr client.
dash_chat_2.1: an interesting library with many reusable widgets.
Flutter-Music-Player-App-With-BLoc: I like the clean MVC organization of this application and the fact there is no route defined, only a navigator pushing screen between context.
bloc_flutter_app: another example application using the Bloc pattern.

I was playing with the Erlang ssh key interface with Elixir (I wrote an article on that in a French magazine on this topic, but with Erlang). Let generate a public/private key.

$ ssh-keygen -f t.key -t ed25519
...
Your identification has been saved in t.key 
Your public key has been saved in t.key.pub 
...

$ file t.key.pub
t.key.pub: OpenSSH ED25519 public key

Then, we can use parse and validate it. We are using the :ssh_messagehttps://github.com/erlang/otp/blob/master/lib/ssh/src/ssh_message.erl Erlang module (undocumented) and the :public_key Erlang module to convert it from one format to another one.

# open the file, it can also from any kind of String.()
# element
File.read!("/t.key.pub")

# Split the key with null char, it should returns a list
# of 4 elements
#   ["ssh-ed25519", pubkey, comment, ""]
|> then(&Regex.split(~r"\s+", &1)) 

# Returns only the pubkey
|> Enum.at(1)

# SSH public key is base64 encoded, decode it.
|> Base.decode64!()

# parse the result with ssh2_pubkey_decode/1 function.
# it returns something like that:
#   {{:ECPoint, ecpoint}, {:namedCurve, {1, 3, 101, 112}}}
|> :ssh_message.ssh2_pubkey_decode()

# this part is to convert it in PEM format (TLS)
# the firs step is to create an entry with the correct
# ASN1 type (e.g :SubjectPublicKeyInfo)
#   {:SubjectPublicKeyInfo, pubkey, :not_encrypted}
|> then(&:public_key.pem_entry_encode(:SubjectPublicKeyInfo, &1))

# the final step is to finalize the PEM encoding
#   "-----BEGIN PUBLIC KEY-----\nbase64\n-----END PUBLIC KEY-----\n\n"
|> then(&:public_key.pem_encode([&1]))

Coding

📝 Introduction to Compilers and Language Design, Second Edition: what a great book! It covers all compiling steps, from scanning, parsing, code generation and optimization.

📖 Write You a Haskell: yet another book in Haskell.

📝 Arena Allocators in C: implementing memory allocators in C can be challenging, here a nice article on how to write one using region-based memory management technique or arena allocator

🏗️ eyelang: An experimental symbolic logic programming language using a practical subset of Prolog syntax and .pl files, with query explanations and executable checks.

📝 Concentrated Liability: a security audit report on Ash framework using LLMs.

📝 Generic Dynamic Arrays in C: Another great article by Eliase Ebner.

📝 Why C Strings Suck And How I Fixed Them: A good way to fix C strings. I like it.

📝 Some Advice for Software Developers in the Age of Large Language Models: This post has many good points regarding LLM, especially the quote below. Indeed, I already saw that in a previous position, someone losing many weeks of developing, switching and switching between LLMs to solve a problem.

what are you going to do when the LLM reaches its limit [...] and you are left having to solve the issue on your own? Ask a different LLM? And if that fails? Who are you going to turn to?

📝 C Socket Programming in Linux: Build a TCP Client & Server (Complete Guide): this blog has really nice article! A must read for the one who wants to learn more about C programming!

🌐 Competitive Programming Hall Of Fame: a project created to preserve the history of international championships in competitive programming. This website collects the results of final stages for both worldwide and regional contests. By international championship we mean a contest consisting of several stages available for competitors from more than one country.

System

📝 FreeBSD Cope with Signal Fuckup: installing signal on FreeBSD using chrooted environment. It would have been better to use a jail there, but it looks okay.

📝 We can harden C: why talking about rust or IA, again? If you want to increase the security of you C application, the first step is to enable all warnings and errors, but also all security features offered by your favorite compiler. It will less fun - because the compiler will blame you for every weird stuff you are doing - be the result will be way better. The next step is to ALWAYS code in C with the help of a small test suite framework like munit or minunit. Easy to install and easy to use, it will easier to create your tests and design your application. Finally, don't forget to use the modern tools available on gcc or llvm, like static analyzers. Not everything will be fixed though, and if you need a really strong application, you should check the security features offered by FreeBSD (capsicum), OpenBSD (pledge, unveil) or Linux. Anyway, C hardening can be challenging, not because of the complexity, but more because of our ignorance about all the tools available.

📝 Building OCI Images Without a Daemon or a Dockerfile: how to build OCI image from scratch, here the repository on github.

📝 Understanding the Linux Kernel: Namespaces: yet another article on the namespaces.

📝 Your Computer Memory Is A Lie: a post doing a summary of the problems involving memory (in general).

Database

📑 Distributed Datalog at Scale with Worst-Case Optimal Joins

🏗️ pglifecycle: keeps your database schema in a version-controlled project of YAML files — one file per database object, validated against a JSON-Schema contract — and moves schema between the repository and live databases.

security

📝 Big Fish: Exploiting a 37-Year-Old Kernel Bug in OpenBSD: an interesting report about one of the security issues recently published on OpenBSD. This is a nasty issue, present there for more than 3 decades.

🌐 zr.ax: classifies executables and raw firmware, with interactive Hilbert entropy maps, a hex inspector, and batch analysis. via Kenan Sulayman

📝 Azure's Weakest Link - Five Full Cross-Tenant Compromises: oof. a violent and nasty security issue on azure cloud that can extract user secret via a craft URL encoded path.

📑 Hardware-Assisted Rootkits: Abusing Performance Counters on the ARM and x86 Architectures

📝 Fingerprints are the new cookies

📝 Confessions of a Professional Hacker: Can the CIA or other Intelligence Agencies Track My Every Move?: yes. Any state agencies with enough resources can track anybody without any problem. It's becoming very hard to be protected against that, even without any digital devices, you are becoming a target.

📝 A story of wifi secrets, UART lockdowns and a damaged flash memory chip

🏗️ openwrt-one-sdr: Turns an OpenWrt One router into a wideband, receive-only software-defined receiver. The board's MediaTek MT7981B Wi-Fi SoC (with an MT7976 radio front end) captures raw 14-bit IQ from the 2.4 and 5 GHz bands at 40 Msps, the dual Cortex-A53 streams it over the 2.5 GbE port as VITA 49, and a Linux host shows it as a live spectrum and waterfall in the browser.

Security/Cryptography

🌐 ecdsa.fail

Security/Red-Team

🏗️ exploit-watch-archive: durable, private aggregation of the exploit/PoC source repos tracked by the exploit-watch monitoring task.

Embedded

📝 The FPGA Chronicles: Ready To Probe

📝 Another Wireless Chip Has A Hidden SDR Revealed: some ESP32 chips are embedding SDR feature. It looks provising.

Update/Upgrade

🔄 OpnSense: 7 remote exploits. Some of them looks really dangerous, please update.

🔄 pfSense: pfSense seems to have been impacted as well.

🔄 Erlang/OTP 29.1.2: bugs and security fixes.

Leaks of the week

📝 Meta: 1500 people impacted after github repositories hacked.

📝 MaQuestionMedicale: 96000 people impacted, 369k private messages, with name, phone number, and more.

📝 Solution30: more than 10millions identifiers impacted, 11 millions phone number, 2 millions e-mail address.

🐦 Canva: 200 millions database rows including enterprise license, and orders.

🐦 Sourcenext Corporation: 22.5 millions customer records including name, phone numbers, e-mail addresses, passwords, postal address...

Misc

📝 I don't understand this whole idiotic concept of "don't use AI for X".: The problem is not the LLM/AI per se, but where the computation is made. A compiler and most of the software a developer is using are hosted on their own laptop or on a server they are owning. The result is their own binaries, on their own infrastructure. Today, most - if not all - LLMs are being used through big companies (e.g. Google, Microsoft, Apple, Meta, Anthropic and so on), where the definition of "freedom of usage" is globally incompatible with open-source software. Yes this is a tool, but a tool one is using remotely, on servers no one controls. That's crazy someone like Torvalds does not understand that. When local-LLMs will be working correctly, I think it will be huge win for the open-source world, but for now, this is a future trap.

// Detect dark theme var iframe = document.getElementById('tweet-2109301986189324775-605'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2109301986189324775&theme=dark" }

Cover Image by freestocks on Unsplash

Q&D: Using WebView with Flutter

Mathieu Kerjouan — Thu, 15 Oct 2026 08:00:00 +0000

Your website is already working well, compatible with mobile device? Why recreating the wheel, and simply recreate your application around the WebView feature? The idea is simple, use the native Android (or iOS) browser inside your application. It will not fix everything though, but you will be able to reuse your website without rewriting everything from scratch.

Another purpose of WebView objects is to create an interface to a platform without public API. Indeed, with WebView you will have access to a real web browser capabilities, including executing requests on a remote service, storing cookies or executing Javascript script. on Dart/Flutter, the webview_flutter package is required.

$ flutter pub add webview_flutter
Resolving dependencies... 
Downloading packages... 
  cupertino_icons 1.0.9 (2.0.0 available)
  material_color_utilities 0.13.0 (0.13.1 available)
  meta 1.18.3 (1.19.0 available)
+ plugin_platform_interface 2.1.8
  test_api 0.7.12 (0.7.15 available)
  vector_math 2.4.2 (2.4.3 available)
+ webview_flutter 4.14.1
+ webview_flutter_android 4.14.1
+ webview_flutter_platform_interface 2.15.1
+ webview_flutter_wkwebview 3.27.0
Changed 5 dependencies!
5 packages have newer versions incompatible with dependency constraints.
Try `flutter pub outdated` for more information.

WebView objects are objects to display web content (usually website) inside an application. This kind of object can be found on Android and on iOS, both implementation offering similar (but not perfectly identical) features. When one WebView object is instantiated, it can be used like a browser where a part of its interfaces are exposed to the application.

The best way to demonstrate this is to create a WebViewController object. The one presented in the example should do the job for now (with few modification):

The setJavaScriptMode method is called, it can be set to disabled or unrestricted;
The setNavigationDelegate method is called, and a NavigationDelegate object is created. This last one is used to accept or reject the navigation changes. The following callback functions can be configured:
- onNavigationRequest: executed to accept/deny navigation;
- onPageFinished: executed when the page is fully loaded;
- onPageStarted: executed when the page loading started;
- onProgress: executed during the page loading progression;
- onWebResourceError: executed when an error occurs.
The cacheClear method is called: use the remove the cache from the WebView;
The clearLocalStorage method is called: used to clear the local storage from the WebView;
The loadRequest method is called: used to load an URL.

WebViewController controller({
  String target = 'https://authenticationtest.com/simpleFormAuth/'
}) {
  // parse an URL from a string
  final url = Uri.parse(target);

  // Return a WebViewController Object with custom callbacks
  // and parameters
  return WebViewController()

    // set the javascript mode to unrestricted (javascript
    // can be executed)
    ..setJavaScriptMode(JavaScriptMode.unrestricted)

    // configure the authorized navigation pattern
    ..setNavigationDelegate(
      NavigationDelegate(

        // a progression gauge can be created if necessary
        onProgress: (int progress) {
          print("loading in progress ($progress)...");
        },

        // when the connection to the page start
        onPageStarted: (String url) {
          print("page start to be loaded ($url).");
        },

        // when the page is totally loaded
        onPageFinished: (String url) {
          print("page loaded successfully ($url).");
          showCookies();
        },

        // what to do in case of error
        onHttpError: (HttpResponseError error) {
        }, 

        // this callback is used to select what kind of request
        // is allowed (to which website).
        onNavigationRequest: (NavigationRequest request) {
          return NavigationDecision.navigate;
        }))

    // when a new object is created, clear up the cache
    ..clearCache()

    // clear also the local storage
    ..clearLocalStorage()

    // and finally, load the requested url
    ..loadRequest(url);
}

With this future object created, it is possible to include it in the flutter tree. To that, one can use a WebViewWidget object, with the previous controller passed as argument.

@override
Widget build(BuildContext context) {
  return Scaffold(
    appBar: AppBar(
      backgroundColor: 
        Theme.of(context).colorScheme.inversePrimary,
      title: Text(widget.title),
    ),
    body: Center(
      child: Column(
        mainAxisAlignment: .center,
        children: [
          Container(
            child: Row(children: [])
          ),
          Expanded(child: WebViewWidget(controller: _controller == null ? controller() : _controller!))
          ],
        ),
      ),
    );
}

Some variable states can eventually store the reference of the current controller, initialized in the initState() method. This code was defined inside a StatefulWidget Widget.

WebViewController? _controller;

@override
initState() {
  super.initState();
  _controller = controller();
}

Here the log output during the loading. Yeah, the authenticationtest.com was used to retrieve some session cookies.

I/flutter (19018): page start to be loaded (https://authenticationtest.com/simpleFormAuth/).
I/flutter (19018): loading in progress (20)...
I/flutter (19018): loading in progress (20)...
I/flutter (19018): loading in progress (29)...
I/flutter (19018): loading in progress (70)...
I/flutter (19018): loading in progress (87)...
I/flutter (19018): loading in progress (100)...
I/flutter (19018): loading in progress (100)...
I/flutter (19018): loading in progress (100)...
I/flutter (19018): page loaded successfully (https://authenticationtest.com/simpleFormAuth/)

The cookies from the WebView can also be retrieved (and if needed stored in a local store). This can be achieved by created a new WebViewCookieManager object. After having been successfully instantiated, the cookies can be exported using the getCookies() method.

WebViewController? _controller;

@override
initState() {
  super.initState();
  _controller = controller();
  _cookieManager = WebViewCookieManager();
}

Future<void> showCookies() async {
  final currentUrl = await _controller?.currentUrl() ?? ''; 
  final Uri? domain = Uri.tryParse(currentUrl);
  final cookies = await _cookieManager?.getCookies(
    domain: domain!
  );
  print(cookies);
}

Here the log output when some cookies have been received from the website. Those cookies are using the WebViewCookie object, containing 4 fields, name, value, domain and path. The other cookies attributes like expires are not available on Android (but it seems they can be found on iOS).

I/flutter (19018): [WebViewCookie{name: PHPSESSID, value: 8035d936660419d25f1969de60945682, domain: https://authenticationtest.com/simpleFormAuth/, path: /}]

Here a screenshot of the WebView from the Android emulator. Few buttons have been added to do tests, like extract the cookies and save same in a permanent store, or switch to another URL.

WebView is a really interesting feature to bypass some website limitations, especially the one with public API only. Collecting those cookies can then be reused afterward. Some of you would probably ask why I was playing with this package. Well, I needed to fetch some cookies from another website (not the one used here for test), on one of my project. Having access to the cookies will help me to query my target when authenticated. Anyway, here some interesting links if you want to know more:

webview_flutter package from pub.dev;
The official webview_flutter API documentation on pub.dev;
The webview_flutter source code from github;
The official WebView documentation from the Android developer website;
The official mkWebView documentation from the Apple developer website;
cURL HTTP cookies documentation where the cURL cookies format is explained;
The RFC6265 where the set-cookie is explained, containing also the official cookies attributes with their specifications;
Starting with Flutter: How to embed a webview by @theotherdevs;
Converting any website into an app: Flutter webview by @franciscodex;
Not Just a WebView: Building a Native Engine on Flutter to Convert Sites to Apps with SDUI by @__2ea5fee000c;
Flutter: Two-way communication between a WebView and a web page by @diamantidis;
Is there a way to show more cookies fields from webview_flutter's WebViewCookieManager on StackOverflow, where I asked a question regarding the extra cookies attributes.

As usual, have fun and happy hacking!

The stackoverflow user who answered me on my question found a bug in the parser (I missed it), when a value is set to base64, it returns an empty string. In fact, while I was reading the code, I did not see any limits on the size of the key or the value. This should also be tested, a malicious server could eventually overflow an application with badly crafted cookies. This protected could be implemented on the WebView (Kotlin) part though and not in the flutter interface.

Anyway, I will need a way to reproduce (at least) the following bug. Notes from Mozilla documentation:

A <cookie-name> can contain any US-ASCII characters except for control characters (ASCII characters 0 up to 31 and ASCII character 127) or separator characters (space, tab and the characters: ( ) < > @ , ; : \ " / [ ] ? = { }

A <cookie-value> can optionally be wrapped in double quotes and include any US-ASCII character excluding control characters (ASCII characters 0 up to 31 and ASCII character 127), Whitespace, double quotes, commas, semicolons, and backslashes.

Encoding: Many implementations perform percent-encoding on cookie values. However, this is not required by the RFC specification. The percent-encoding does help to satisfy the requirements of the characters allowed for <cookie-value>.

set a cookie with a base64 value (e.g. dGVzdAo=)
fetch the cookie as WebCookie, the value returned is """ instead of "dGVzdAo="

This issue is due to the current implementation of the parser used in the AndrdoiWebViewCookieManager implementation.

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';

// code exfiltred from official sources
poc(String cookies) {
  List<dynamic> store = []; 
  for (final String cookie in cookies.split('; ')) {
    final List<String> cookieValue = cookie.split('=');
    store.add((cookieValue.first, cookieValue.last));
  }
  return store;
}

void main() {
  test("poc", () {
    print(poc("test=dGVzdAo"));
    expect(poc("test=dGVzdAo"), ("test", "dGVzdAo"));

    print(poc("test=dGVzdAo="));
    expect(poc("testdGVzdAo="), ("test", "dGVzdAo="));
  }); 
}

$ flutter test
...
00:00 +0: poc                                                                               
[(test, dGVzdAo)]
00:01 +0 -1: poc [E]                                                                        
  Expected: (String, String):<(test, dGVzdAo)>
    Actual: [(String, String):(test, dGVzdAo)]

  package:matcher                                     expect
  package:flutter_test/src/widget_tester.dart 473:18  expect
  test/widget_test.dart 25:5                          main.<fn>
...

Here a quick nginx configuration:

events {
    worker_connections   2000;
}

http {
  server {
    listen 4001;
    add_header Set-Cookie "test=dGVzdAo=; Path=/";
    return 200;
  }
}

Thanks to Mahendra Dhakad](https://stackoverflow.com/a/80009898/6782635) on stackoverflow.

Cover Image by Paul Skorupskas on Unsplash

Q&D: Android Device Information

Mathieu Kerjouan — Wed, 14 Oct 2026 08:00:00 +0000

It can be necessary, sometime, to collect device information, especially to identify a specific kind of device for debugging purpose, or to identify unique devices. In Dart/Flutter, one can install the device_info_plus package to have access to lot of informations regarding the device where the application is installed. Let have a quick look on this one.

$ flutter pub add device_info_plus
Resolving dependencies... 
Downloading packages... 
  cupertino_icons 1.0.9 (2.0.0 available)
+ device_info_plus 13.3.0
+ device_info_plus_platform_interface 8.1.0
+ ffi 2.2.0
+ ffi_leak_tracker 0.1.2
+ file 7.0.1
+ flutter_web_plugins 0.0.0 from sdk flutter
  material_color_utilities 0.13.0 (0.13.1 available)
  meta 1.18.3 (1.19.0 available)
+ plugin_platform_interface 2.1.8
  test_api 0.7.12 (0.7.15 available)
  vector_math 2.4.2 (2.4.3 available)
+ web 1.1.1
+ win32 6.4.0
+ win32_registry 3.0.3
Changed 10 dependencies!
5 packages have newer versions incompatible with dependency constraints.
Try `flutter pub outdated` for more information.

The first step is to import the device_info_plus package in your code.

import 'package:device_info_plus/device_info_plus.dart';

One can extract only a small subset of all device information available, for example, to extract the device id, the id property can be retrieved and returned.

Future<String> getDeviceId() {
  DeviceInfoPlugin deviceInfoPlugin = DeviceInfoPlugin();
  String deviceInfo = await deviceInfoPlugin.deviceInfo;
  return deviceInfo.id;
}

If more information about a device is required, one can extract all of them as a Map<String, dynamic> using the data property.

Future<Map<String, dynamic>> getDeviceInfoMap() {
  DeviceInfoPlugin deviceInfoPlugin = DeviceInfoPlugin();
  BaseDeviceInfo deviceInfo = await deviceInfoPlugin.deviceInfo;
  return deviceInfo.data;
}

Some information are only available on a specific kind of device (e.g. iOS, Android, Browser). If the application has been developed with portability in mind, it can also be a good idea to extract device's information based on the architecture/platform.

Future<Map<String, dynamic>> getAndroidDeviceInfoMap() {
  DeviceInfoPlugin deviceInfoPlugin = DeviceInfoPlugin();
  AndroidDeviceInfo deviceInfo = await deviceInfoPlugin.androidInfo;
  return deviceInfo.data;
}

Here an example of the data extracted on an Android emulator:

I/flutter ( 3959): physicalRamSize: 2472
I/flutter ( 3959): type: user
I/flutter ( 3959): manufacturer: Google
I/flutter ( 3959): freeDiskSize: 464072704
I/flutter ( 3959): bootloader: unknown
I/flutter ( 3959): fingerprint: google/sdk_gphone64_x86_64/emu64xa:16/BE2A.250530.026.D1/13818094:user/release-keys
I/flutter ( 3959): host: e8e8f3751cda
I/flutter ( 3959): isLowRamDevice: false
I/flutter ( 3959): model: sdk_gphone64_x86_64
I/flutter ( 3959): id: BE2A.250530.026.D1
I/flutter ( 3959): brand: google
I/flutter ( 3959): hardware: ranchu
I/flutter ( 3959): product: sdk_gphone64_x86_64
I/flutter ( 3959): totalDiskSize: 6228115456
I/flutter ( 3959): supportedAbis: [x86_64, arm64-v8a]
I/flutter ( 3959): supported32BitAbis: []
I/flutter ( 3959): display: BE2A.250530.026.D1
I/flutter ( 3959): isPhysicalDevice: false
I/flutter ( 3959): version: {baseOS: , securityPatch: 2025-07-05, sdkInt: 36, release: 16, codename: REL, previewSdkInt: 0, incremental: 13818094}
I/flutter ( 3959): systemFeatures: [android.hardware.sensor.proximity, android.software.adoptable_storage, android.hardware.sensor.accelerometer, android.software.controls, android.hardware.faketouch, android.software.telecom, android.hardware.telephony.subscription, android.hardware.reboot_escrow, android.hardware.telephony.data, android.software.backup, android.hardware.touchscreen, android.hardware.touchscreen.multitouch, android.software.erofs, android.software.print, android.software.activities_on_secondary_displays, android.software.device_lock, android.software.voice_recognizers, android.software.picture_in_picture, android.hardware.fingerprint, android.hardware.sensor.gyroscope, com.google.android.feature.PERSONAL_SAFETY, android.software.vulkan.deqp.level, android.software.cant_save_state, android.hardware.security.model.compatible, android.hardware.telephony.messaging, android.hardware.telephony.calling, android.hardware.sensor.relative_humidity, android.hardware.bluetooth, android.software.window_magnification, andr
I/flutter ( 3959): tags: release-keys
I/flutter ( 3959): supported64BitAbis: [x86_64, arm64-v8a]
I/flutter ( 3959): availableRamSize: 967
I/flutter ( 3959): name: sdk_gphone64_x86_64
I/flutter ( 3959): time: 1753213259000
I/flutter ( 3959): device: emu64xa
I/flutter ( 3959): board: goldfish_x86_64

More information can be found on Android via the AndroidBuildVersion object (including the base name, the code name and the release).

Depending on the platform, a lot of values can be extracted. Here a quick list enumerating few of them (the ones the most interesting to me):

AndroidDeviceInfo:
- availableRamSize as int: Current unallocated RAM size of the device in megabytes, extracted from ActivityManager.MemoryInfo;
- board as String: The name of the underlying board, like "goldfish", extracted from Build.BOARD;
- brand as String: The consumer-visible brand with which the product/hardware will be associated, if any, extracted from Build.BRAND;
- device as String: The name of the industrial design, extracted from Build.DEVICE;
- fingerprint as String: A string that uniquely identifies this build (Do not attempt to parse this value.), extracted from Build.FINGERPRINT;
- hardware as String: The name of the hardware (from the kernel command line or /proc), extracted from Build.HARDWARE;
- manufacturer as String: The manufacturer of the product/hardware, extracted from Build.MANUFACTURER;
- version as AndroidBuildVersion: Android operating system version values derived from android.os.Build.VERSION;
IosDeviceInfo
- identifierForVendor: Unique UUID value identifying the current device;
- modelName: Commercial or user-known model name Examples: iPhone 16 Pro, iPad Pro 11-Inch 3;
- utsname: Operating system information derived from sys/utsname.h;
LinuxDeviceInfo
- id: A lower-case string identifying the operating system, excluding any version information and suitable for processing by scripts or usage in generated filenames;
- machineId: A unique machine ID of the local system that is set during installation or boot. The machine ID is hexadecimal, 32-character, lowercase ID;
MacOsDeviceInfo
- systemGUID: Device GUID;
WindowsDeviceInfo
- deviceId: Displayed as "Device ID" in Windows Setting

The next step will be to generate an unique reproducible identifier from different systems using those information. A list of resources to help thinking about all of that:

Best practices for unique identifiers from the official Android Developers Website;
Identifying App Installations from the official Android developers blog;
android-unique-device-id on github;
Unique ID of Android device on StackOverflow;
Generate a unique Android Device ID on gist;

As usual, happy hacking!

Cover image by Elena Ramseier on Unsplash

WeeklyReview#202641

Mathieu Kerjouan — Mon, 12 Oct 2026 10:00:00 +0000

This week was quite busy in my code. Redarging my main application/product, I started to implement the database schema with sqflite in flutter. I did also a lot of tests regarding the flexibility of sqflite, and its potential limits in my use case. To be honest, I don't really like how this library is working but it should do the jobs for now. I also played with the flutter_quill text editor library (and also fleather). Not so complex to use, can produce something pretty cool quick. Finally, the BlocSignal pattern used in Flutter seems really attractive, lot of examples are available on their repository and it can be a good alternative of my approach (doing everything from scratch to learn).

It was not expected, but it looks like my motivation was focused to do C code for mac_randomizer. This is definitively a pet project, but C is still one of my favorite language, even if it can be challenging to do some things (e.g. concurrency), it's still great to do kinda low-level code. The source code tree is now clean, the Makefile is minimalist but should work on any system using GNU Makefile. Most of the functions are now sorted by categories in their own source file. The last action was to start working on a minimalist implementation of a Mealy machine, a specific kind of finite state machine, to parse mac addresses, instead of using an external regexp library. Actually, the implementation is a bit hard, not because of the complexity, but because of the interface and the abstraction. My idea is to have something like that:

// first tokenize the mac address to validate the
// characters used. Luckily, a mac address only has
// 23 allowed characters: [a-fA-F0-9\-\.\:]
//
// char *string is the raw strings containing the
// mac address.
//
// size_t length is the max size of the string, the
// tokenizer must stop when reading a NUL char or when
// it reaches the length limit
//
// the struct token *tokens is an array of token, where
// each token contains the raw valid character, the 
// position of the character and the kind of token (e.g.
// digit, separator).
//
// the tokens should probably be defined inside another
// structure embedded the length of the array. This value
// should be incremented every time a valid character is
// found.
int
tokenizer(char *string, size_t length, struct token *tokens) {
  return fsm_start(string, length, state, tokens);
}

// secondly, the parser will get the tokens and will convert
// them into an internal binary representation of the mac
// address (e.g. uint8_t mac_address[6]).
//
// The arguments of this function are similar to the previous
// one. The final result is then stored in the mac_address
// variable.
int
parser(struct token *tokens, size_t length, uint8_t *mac_address){
  return fsm_start(tokens, length, state, mac_address);
}

// the question know is how to design correctly the fsm_start()
// function. It should have the possibility to deal with different
// kind of input, not only char, but also C data-structure. It
// should also be able to deal with different kind of output. Based
// on these requirements, the definition of the function should look
// like the snippet below.
//
// The input argument can be a problem here, at least for the
// mac address parsing feature. Indeed, a string representing a
// mac address only has a few format, with specific length:
//
//   EUI48-IEEE: xx-xx-xx-xx-xx-xx (max length 17)
//   EUI48-IETF: xx:xx:xx:xx:xx:xx (max length 17)
//   EUI48-CISCO: xxxx.xxxx.xxxx (max length 16)
//   EUI64-IEEE: xx-xx-xx-xx-xx-xx-xx-xx (max length 23)
//   EUI64-IETF: xx:xx:xx:xx:xx:xx:xx:xx (max length 23)
//
// the input string should then have a maximal value of 23
// (+1 with NUL), and a minimal value of 17(+1 with NUL). The
// output can also be limited to sizeof(uint8_t)*6=48bits or
// sizeof(uint8_t)*8=64bits. Using the 64bits version is perhaps
// the best choice because an EUI48 can be converted to an EUI64
// and vice versa. Note: I don't even think it's possible to
// compress this value, it will probably add more complexity
// for no real benefits on the long run.
int
fsm_start(
  void *input,
  size_t input_length,
  void *output,
  struct state *init
){
  // do something here.
}

// the states must also be defined somewhere. states can
// be created dynamically by instantiating different kind
// of object and pushing them together, or creating them
// "statically" in their own data-structure with their
// own callback functions. One of the idea is to create
// something like the code below.
//
// The handler_enter callback is called only when a transition
// occurs between states.
//
// The handler_input callback is executed when an input
// is received.
struct state {
  struct state*(handler_enter)(struct state *s);
  struct state*(handler_input)(void *input, void *output, struct state *s);
};

As you can see, it's a good mental exercise to find the right way to implement something like that in C. In fact, a mealy (or moore) FSM are being use with Regexp but also with other complex protocols like TCP. Reimplementing something like that to parse well specified data like a mac address is probably the best choice, and, if correctly done, it could also be used to encode as well.

Anyway, here some implementations from github:

More implementation examples can be also found in userland from OpenBSD, FreeBSD and NetBSD projects. Many of those tools can have an implementation of a finite state machine.

OpenBSD:
- usr.bin/patch/ed.c: it contains a small finite state machine implementation used to parse commands
- games/bs/bs.c
- usr.sbin/npppd/npppd/fsm.c
- usr.sbin/pppd/fsm.c
- lib/libc/gen/unvis.c
FreeBSD:
- crypto/openssl/ssl/statem/statem.c
NetBSD:
- libexec/telnetd/state.c
- sbin/init/init.c

Most of my posts are currently stuck in draft mode due to that, but I like the weekly review format. It is a good summary containing a lot of information in one place. Then, I think most of the coming publications will use this format.

Coding

📝 A deep dive into HEEx compilation from LiveView 1.2: A post explaining the foundation of the HEEx template systems present in Phoenix LiveView. I did not know it was tokenized. In fact, this is a very good news... It is becoming closer to the way clojure is dealing with HTML.

📝 an application in lisp you grow by talking to it: LISP is probably one of the greatest language of all time, in this article, the program is created by talking to an LLM during a coding session.

🏗️ WeaveC: A C compiler with inferred ownership and borrow checking for Rust-style memory safety.

🌐 Tools for your Elixir app: A starting point for each kind of application. Recommended tools, why we use them, and when to choose something else.

🏗️ cccc: measures Cognitive Complexity (SonarSource / G. Ann Campbell) and Cyclomatic Complexity (McCabe). I used similar tools in the past, like ccc and scc or sccc. Nice to have, even before if you add it in your CI/CD to evaluate the complexity of a commit.

📑 Reducing undefined behavior in the C language: undefined behavior in C is a nightmare. Probably one of the most complex part of the language, and it can lead to total disaster. This post is presenting the solution to avoid common mistakes and find a way to create safer C code. Actually, this is from a talk, and the slides are available online.

📝 What Every C Programmer Should Know About Undefined Behavior #1/3: the first article of a series about undefined behavior in C.

📝 Abstract Syntax Tree, An Example in C: a short publication with nice example to explain how to implement and use an AST.

📝 One-pass Compiler: a nice trick to compile code directly without passed inside different layer.

📖 Compiling to Assembly from Scratch: a book by the author of the two previous posts. neat.

📑 CASM: Implementing an Abstract State Machine based Programming Language

System

📝 Headers Up! What's New in httpd and relayd: HTTP header manipulation support and request filtering. I'm still using nginx, but httpd is becoming really better every new release.

📝 Locking down openat(2) and friends with O_BELOW and F_BELOW: a mail-exchange describing the (future?) openat function to open file with relative path (and more safety).

🏗️ scrcpy: Display and control your Android device.

🏗️ floci: gives you AWS-shaped services on your machine without requiring a cloud account, an auth token, or paid feature gates.

🏗️ riscv/learn: A community-driven, continuously updated compilation of RISC-V learning resources. Content is organized by topic and experience level to help you discover courses, software, documentation, and articles efficiently.

📖 Haskell: The Craft of Functional Programming: revised version.

🏗️ DPDK (Data Plane Development Kit): an open source framework for fast packet processing on standard hardware. It runs in cloud infrastructure, telecom networks, financial exchanges, particle accelerators, and radio telescopes. Most of your traffic already runs through it. Start by reviewing a patch, no account or setup required. I discovered it was also running not only on Linux... I need to try that.

📝 unikernels were hard. key word: were: unikernel IS THE WAY TO GO, but I still this a complex technology mixing a lot of features from many different part of any systems. In fact, this is really close to microkernel, and both are, to me, two things that should be merged together.

Network

🏗️ Open vSwitch (OVS): a production quality, multilayer virtual switch, designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols.

🏗️ Open Virtual Network: a series of daemons for the Open vSwitch that translate virtual network configurations into OpenFlow.

Database

🏗️ Awesome Database Learning

🏗️ pgvector: Open-source vector similarity search for Postgres

Security

📝 Malicious Flutter Package while it have a Clean Code?: malware present in example directory inside a flutter project. Don't trust blindly all projects.

📝 How to fix a bug in a fix

📝 Chrome's Response to Recent ccTLD Registry Hijacks: Following recent .gh, .sl, and .as ccTLD domain hijacks, Chrome immediately blocked unauthorized HTTPS certificates via CRLSets to protect users automatically.

Telegram Desktop: one-click account takeover via IPC injection: a post describing the CVE-2026-107181, where one malicious actor can take control of a group by sharing a telegram link and opening it on the telegram desktop application. The problem is from the URI parser, when dealing with semicolons. That's an interesting attack, because many application can be ported on different systems, like Signal or SimpleXChat, and depending of the languages/tools used, some part of the can use different routines.

Security/Cryptography

📝 SequenceHash: multihashing for the rest of us

🌐 Hash Smash: Find ordinary collision attacks that reach more rounds with less total computation. Select a hash function’s chart, then one of its two round counts.

📑 MiniSHAKE: an extendable output function based on Keccak, with unlimited size output.

📑 TurboSHAKE: an extendable output function based on Keccak, with less rounds to improve performance.

🌐 Key Length: The minimum key sizes each source publishes for protecting data until a given year.

Embedded

🏗️ CPU Performance Engineering: A reading path for CPU performance engineering, from one instruction to production inference. Primary sources only, with a runnable benchmark for every section.

📝 The FPGA Chronicles: Open Source It

🏗️ PasRISCV:A RISC-V RV64GCV/RVA23 emulator written in Object Pascal. It simulates processor cores, memory, I/O, and much more.

Update/Upgrade

🔄 hackney 4.8.5: bug fixes.

🔄 Cowboy 2.20: security and bug fixes.

🔄 LibreSSL 4.2.2 and 4.3.3: security and bug fixes.

Computer Science

📑 Digital Hardware Design DFA, Mealy and Moore Machines: a lecture on Mealy/Moore machines

📑 A Report on Achieving Complete Regular-Expression Matching using Mealy Machines: an interesting (but complex) publication about regexp and finite state machine to match exact pattern.

📑 Effective Reductions of Mealy Machines: reducing mealy machines states and transitions.

Misc

📝 The empty brain: an amazing post about brain information processing. The brain is not storing information like a computer (a stream of bits), and in fact, it is not the only organ to "store it". It stores experiences, and experiences are different between every people alive, then, it will be impossible to "copy" experiences between humans. In fact, like the author is explaining in his publication, even if all neurons' states were extracted, copied and connected together, it will have no meaning because it would be outside of the body where the experiences have been learned.

Because the Information Processing metaphor is not even slightly valid, we will never have to worry about a human mind going amok in cyberspace; alas, we will also never achieve immortality through downloading. [...] We are organisms, not computers.

📝 Victime d’une fuite de données ? Voici dans quels cas vous pouvez réclamer une indemnisation: see also that on the CNIL website

📝 A strategist who called the Asian financial crisis sees parallels with the AI boom: another indicator the "AI" boom is not really doing what every one expect. I wanted to read The Myth Of AI's Miracle: Albert Edwards Spots The Same Fatal Flaw That Sank Asia In 1997, but, well, it is a premium article.

the productivity payoff from AI remains a forecast, rather than an observation [...] No AI productivity boost in the data yet [...] too many people believed the beguiling narrative and as a consequence, lent these economies cheap, abundant capital which was misallocated.

🐦 THE SIGNS THAT THE AI BOOM IS BUILT ON FAKE REVENUE ARE NOW HERE. Here is what is really happening: a thread about what could happen with the AI financial - SCAM - scheme. Anyway, it smells really bad, even worse than the crypto-currency bubble.

Every time an AI startup raises money at a higher valuation, the investor raises the value of its stake on its books and counts the increase as income. No cash comes in. The profit is on paper. [...] A large part of the sales and profits these companies report depends on money that investors, including the tech giants themselves, keep putting into OpenAI and Anthropic. If that funding slows, the backlog, the sales and the paper profits all weaken together.

Leaks of the week

🐦 Tata Consultancy Services: source code and credentials leaked.

🐦 UAE government: 17 millions entries from UAE government-related software licensing database

🐦 Get covered: more than 1 million customer records including email, phone numbers, physical address and more.

🐦 CAPEB: Around 4 millions records including emails and phone numbers.

🐦 Denmark government: 8.8 millions people impacted, Danish citizens and emigrates. The leak include Social security number, Age, Sex, Family relations, Physical address, Protected addresses and Sex change.

// Detect dark theme var iframe = document.getElementById('tweet-2107477791549907153-99'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2107477791549907153&theme=dark" }

Cover Image by Alfred Schrock on Unsplash

WeeklyReview#202640

Mathieu Kerjouan — Mon, 05 Oct 2026 10:00:00 +0000

Another week of doing Erlang/Elixir code on expki project. Again, that's mostly a sandbox to remove my 2 years rust doing only Erlang code and infrastructure stuff. Things are moving though, and I have a way to deal with the public_key module from Elixir. It's not perfect yet and can be optimized, but it should do the job. I think you will read that in a post very soon.

My way to extract macros and records from Erlang source code has been moved to its own module called preprocessing. It is currently being tested, but it should work correctly. Many things remain to be added though, like the support for dynamic macros and perhaps other attributes from Erlang.

Dev.to is buggy. One big article I started to write few days ago was practically destroyed because of cache issue on the browser. When editing the file from the dashboard, an old version was displayed instead of the latest one. Don't save. Don't publish. Close the window, logout from your dev.to account and reconnect.

publish is now able to fetch all the posts from dev.to using the API and curl. I did not have a lot of time working on it this week though. It would be really nice to migrate few my articles very soon on my own domain.

mac_randomizer was a pet project I got year ago, simply an idea at first. I wanted to do a bit of C this week and started to play with that. Not a really serious project though.

Coding

📖 Course book: Open Source Introductory Systems Programming Textbook for the University of Illinois

🌐 zero-js: compiling code in HTML/CSS directly, without javascript.

📝 Optimizing a Lock-Free Ring Buffer

📝 Simplify Function Generation in Elixir with Macros: a quick reminder to generate function in Elixir with macros. A good way to do meta-programming.

📝 Elixir Anonymous functions: official documentation about anonymous/lambda functions in Elixir.

📝 Elixir Quote and unquote: official documentation about metaprogramming elixir with quote and unquote.

🏗️ pulso: A headless, open-source observability backend: unified logs, metrics, and traces access, built-in alerting, and a native Model Context Protocol (MCP) interface, built on Elixir/OTP with a Rust hot path for columnar data.

🏗️ erlang_pay: A pure-Erlang payment gateway library for Alipay (App pay), WeChat Pay v3 (JSAPI/Native) and Stripe (PaymentIntent).

📝 How Copy-on-Write Works with Memory-Mapped Files

📝 Disable debuginfo to improve Rust compile times

📝 Reverse Engineering Novalogic's Comanche Terrain Maps: not an expert in game programming, but reverse engineering is an amazing field. This article explains how to reverse the maps from an old game, it's in C, and it's a great one! I liked this quote:

Programmers in the early 90s didn't reinvent everything. They used the formats and tools the artists already had, like PCX files straight out of a paint program, and saved their cleverness for the parts that really needed it: the renderer.

📝 What TLA+ can and can't check: TLA+ fascinated me few years ago, and I started to learn it by myself. Unfortunately, I was not really good with it. Languages formal verifications/specifications can be challenging to learn, even more if you don't have any a real usage for them. It's good to read a post on this language, even more to clarify what TLA+ can do or not.

📝 Compiler Options Hardening Guide for C and C++: an extremely useful guide and reference to correctly set C compiler flags for hardened applications.

Database

🏗️ Dasha: PostgreSQL performance dashboard for analyzing database cluster health, identifying problems, and providing optimization recommendations.

System

🏗️ MiniOS: A full-featured desktop environment with a wide range of applications. Professional tools for creativity and development.

🏗️ Vinix: Vinix is an effort to write a modern, fast, and useful operating system in the V programming language.

📝 HardenedBSD August / September 2026 Status Report: mostly small modification and bug/security fixes.

📝 OpenBSD-current has moved to version 8.0: openbsd 8.0 will be ready soon.

📝 vmm(4)/vmd(8) gain support for multi-processor VMs: OpenBSD vmm/vmd can now have more than one CPU configured, that's a very good news!

📝 SmartOS: The illumos Way of Thinking About Servers: sunos, illumos and smartos... A good way to introduce them via this article.

🏗️ How To Secure A Linux Server: An evolving how-to guide for securing a Linux server that, hopefully, also teaches you a little about security and why it matters.

📝 Powerboard Tyche unveiled at Pianeta Amiga: Never got an Amiga, never installed AmigaOS, but since I learnt DragonFlyBSD kernel was a mix between FreeBSD and AmigaOS, I always wanted to test it. Another crazy thing is... the community is active! The passion for Amiga seems eternal. Crazy.

Security

📝 Auditing in the age of (good enough) AI

🏗️ apk-reverse: a skill to reverse APK.

📝 Proactive Defense: Hardening Code Pipelines and CI/CD Infrastructure: a must read to apply modern protection on CI/CD.

Organizations should mandate a minimum release-age cooldown of seven days before any newly published public package version becomes installable [...] Establishing a strict seven-day buffer provides the open-source ecosystem time to detect and pull poisoned releases before they reach internal builds. [...] All external packages and container images should, wherever possible, route through a centralized internal proxy that caches, inspects, and gates each component.

📝 The Developer is the New Perimeter: How Supply Chain Attacks Are Becoming Cloud Breaches: the number of supply chain attacks is growing, and this is another article to remind us to always verify the security of our CI/CD.

📝 Cops Can Bypass iPhone’s Automatic Reboot to Get Into Locked Phones, Leaked Video Claims

Security/Cryptography

📑 A Survey of Pseudorandom Codes:

📑 Fully Anonymous Perfect Secret-Sharing

🏗️ Jolt: Just One Lookup Table. Jolt is a zkVM (zero-knowledge virtual machine) for RISC-V, built to be the simplest, fastest, and most extensible general-purpose of its kind. A book is available.

📑 Formally Modeling the Terrapin Attack on SSH

📑 DASH: Distributed Asynchronous Schnorr with High Throughput

📑 Perpetual Encryption

📑 Lightweight Cryptography for Secure UAV Data Integrity Using Ascon-XOF128 and SHAKE128

📑 Forging 1024-bit RSA signatures in nearly SNFS time

📑 New Applications of RSA

Security/RedTeam

📝 Offensive Security: Pentesting Office Phones with SIPVicious: a post showing a quick demo of sipvicious against VoIP local server.

Embedded

🏗️ Sail architecture definition language: Sail is a language for defining the instruction-set architecture (ISA) semantics of processors: the architectural specification of the behaviour of machine instructions. Sail is an engineer-friendly language, much like earlier vendor pseudocode, but more precisely defined and with tooling to support a wide range of use-cases.

📝 You Can Make A Microprocessor That’s All Your Own

📝 Reverse Engineering the iPod Classic's Undocumented Mikey Chip

Updates/Upgrades

🔄 Hackney 4.8.4: many updates here, from the 4.8.1 to 4.8.4, fixing many bugs. If you are using a previous version, you should probably update it.

🔄 OpenBSD 7.9 Errata: a huge amount of patch (9) to fix security issues on the whole system. ldapd, nsd (8 CVEs), unbound (35 CVEs), iked and many other issues found on the kernel.

🔄 Qubes OS 4.3.2

Misc

📝 How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog

📝 Making Deep Learning Go Brrrr From First Principles

📝 AI Bros Are Having a Meltdown Because the Associated Press Stylebook Clarifies That AIs Don’t Have Feelings

📝 Someone ‘Torturing’ LLMs in a Robot Prison Has Triggered the Dumbest Debate in AI Yet: Yeah. Crazy right? LLMs are not conscious. Adding a "feature" to feel pain/pleasure in a program does not make it conscious, it adds just another variable in the state, or another way to interpret inputs. This can be called - at best - "LLM moding".

AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans. Unfortunately, there’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings […] If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity.

Leaks of the week

📝 Alaxione: more than 8.5 millions patients impacted, data from 2010 to 2026. +5 millions phone numbers leaked, +1.4 millions e-mail, +300k social security number.

📝 KEL/Orisha: 1.3 million people impacted including +600k e-mail, +600+ phone number, +140k IBAN.

📝 PixelLeak: How AI Agents Exposed Developer Screenshots from Leading Tech Companies: Glow labs identified more than 13k screenshots from 300 organizations (including big companies). Impacted more than 900 repositories and 100k employees. It seems it was mostly skill issue there, and the "AI" were not hardened. Interesting leaks though, because if screenshots can leak, probably other data can be leak as well.

📝 FFRandonnée: 1.43 millions impacted people, more than 1 million e-mail and 663k phone numbers, with physical address location.

📝 Cookson-CLAL: complete credit card information have been leaked. No more information about the number of people impacted though.

📝 VosFactures: data leaks on one of the companies in charge of the digital invoices in France. No more information right now, but they have a database with more than 500k companies. Less than 1 month after the official launching of the digital invoices for the companies, one of the "big player" seems to have been impacted by a huge data leaks.

// Detect dark theme var iframe = document.getElementById('tweet-2104051150274408672-700'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2104051150274408672&theme=dark" }

Cover Image by Konstantin Dyadyun on Unsplash

Erlang Preprocessor and Elixir Interoperability

Mathieu Kerjouan — Thu, 01 Oct 2026 10:00:00 +0000

Erlang and Elixir are both running on the BEAM, the Erlang Virtual Machine, designed for long-living and distribute. In most of the case, talking to Erlang modules via Elixir is straightforward, doing the opposite can be a bit challenging though, but it is okay. In few cases though, extracting values from Erlang can become a real problem. The Erlang PreProcessor (epp) is used to expand Erlang macros before compiling the code, this is a similar feature than one can found in C, C++ and many other languages. This is a powerful way to avoid code-deduplication, and optimize the execution of the code by directly applying a data-structure instead of a reference. It has many other benefits, but this is not the place here to list them all.

Erlang From Code Source to AST

Before doing anything, one should understand a bit how Erlang and the BEAM are working together. The procedure to compile Erlang source code into a binary file is a complex process, but it is possible to have the big picture in few steps.

Firstly, one need to create a file containing Erlang source code. In fact, this is not mandatory to create a file... but we will see that later. As reminder, an Erlang file prefix is .erl, and the name of the file must match the name of the module defined in the module itself..

$ cat > mod.erl <<EOF
-module(mod).
-compile(export_all).
-define(DATA, <<1,2,3,4,5>>).

f() -> ?DATA.
EOF

The code created is straightforward. A simple module is created exporting all functions, a macro is defined containing a binary, and finally, a function called f/0 is defined, returning the content of the macro.

The second step is to call the Erlang preprocessor called epp to expand all macros and generate the final source code. The preprocessor will import functions or data-structures from the external libraries (-include and -include_lib), expands the macros (-define and ?MACRO) and can return a list of tokens or the fully expanded source code. This part can easily be seen by using the -P or -E flags with the command erlc.

$ erlc -P mod.erl

$ cat mod.P
-file("mod.erl", 1).
-module(mod).
-compile(export_all).

f() ->
    <<1,2,3,4,5>>

As you can see, the file mod.P does not have any -define attribute, and the macro has been expanded. A new attribute -file has also been added to the source, at the first line. The preprocessor did its job, and the source code can now be tokenized.

The tokenization procedure can't be seen via a command line, and it will be required to evaluate an Erlang one-line with the erl command to have an idea of what's happening.

$ erl -noshell -eval '
  {ok, M}=file:read_file("mod.P"), 
  T = erl_scan:string(binary_to_list(M)),
  io:format("~p~n", [T]),
  init:stop().
'
{ok,[{'-',1},
     {atom,1,file},
     {'(',1},
     {string,1,"mod.erl"},
     {',',1},
     {integer,1,1},
     {')',1},
     {dot,1},
     {'-',3},
     {atom,3,module},
     {'(',3},
     {atom,3,mod},
     {')',3},
     {dot,3},
     {'-',5},
     {atom,5,compile},
     {'(',5},
     {atom,5,export_all},
     {')',5},
     {dot,5},
     {atom,7,f},
     {'(',7},
     {')',7},
     {'->',7},
     {'<<',8},
     {integer,8,1},
     {',',8},
     {integer,8,2},
     {',',8},
     {integer,8,3},
     {',',8},
     {integer,8,4},
     {',',8},
     {integer,8,5},
     {'>>',8},
     {dot,8}],
    12}

Here, the mod.P has been used instead of mod.erl, because usually, the scanner is scanning the expanded file, not the one containing the macros. This snippet is reading the source code, calling erl_scan:string/1 to scan the tokens, io:format/2 is called to print the result and finally, the VM is stopped by invoking init:stop/0. The result displayed is a list of tokens containing the annotations for each characters.

A quick note here, epp is usually privileged than erl_scan. The tokens from epp:parse_file/2 are already ready to be used with the parser. Here and example:

$ erl -noshell -eval '
  {ok, T} = epp:parse_file("./mod.erl", [{source_name, mod}]),
  io:format("~p~n", [T]),
  init:stop().
'
[{attribute,1,file,{"mod",1}},
 {attribute,1,module,mod},
 {attribute,2,compile,export_all},
 {function,5,f,0,
           [{clause,5,[],[],
                    [{bin,5,
                          [{bin_element,5,{integer,5,1},default,default},
                           {bin_element,5,{integer,5,2},default,default},
                           {bin_element,5,{integer,5,3},default,default},
                           {bin_element,5,{integer,5,4},default,default},
                           {bin_element,5,{integer,5,5},default,default}]}]}]},
 {eof,6}]

Let continue on the previous procedure. The next step before generating the AST (Abstract Syntax Tree) above, is to parse the tokens produced by erl_scan. To do that, erl_parse module is used. The final result can be produced with the -S flag when calling erlc or by calling erl_parse directly from the shell. Below, the AST produced by erlc.

$ erlc -S mod.erl

$ cat mod.S
{module, mod}.  %% version = 0
{exports, [{f,0},{module_info,0},{module_info,1}]}.
{attributes, []}.
{labels, 7}.
{function, f, 0, 2}.
  {label,1}.
    {line,[{location,"mod.erl",5}]}.
    {func_info,{atom,mod},{atom,f},0}.
  {label,2}.
    {move,{literal,<<1,2,3,4,5>>},{x,0}}.
    return.
{function, module_info, 0, 4}.
  {label,3}.
    {line,[]}.
    {func_info,{atom,mod},{atom,module_info},0}.
  {label,4}.
    {move,{atom,mod},{x,0}}.
    {call_ext_only,1,{extfunc,erlang,get_module_info,1}}.
{function, module_info, 1, 6}.
  {label,5}.
    {line,[]}.
    {func_info,{atom,mod},{atom,module_info},1}.
  {label,6}.
    {move,{x,0},{x,1}}.
    {move,{atom,mod},{x,0}}.
    {call_ext_only,2,{extfunc,erlang,get_module_info,2}}

The very same result can be done using erl_parse:parse_form/1, erl_parse:parse_exprs/1 or erl_parse_term/1 when the tokens are grouped using the {dot,_} token.

> erl_parse:parse_form([
  {'-',1},{atom,1,file},{'(',1},
    {string,1,"mod.erl"},{',',1},
  {integer,1,1},{')',1},{dot,1}
]).
{ok,{attribute,1,file,{"mod.erl",1}}}

> erl_parse:parse_form([
  {atom,7,f}, {'(',7}, {')',7}, {'->',7},
     {'<<',8},
       {integer,8,1}, {',',8},
       {integer,8,2}, {',',8},
       {integer,8,3}, {',',8},
       {integer,8,4}, {',',8},
       {integer,8,5},
     {'>>',8},
     {dot,8}
]).
{ok,{function,7,f,0,
              [{clause,7,[],[],
                       [{bin,8,
                             [{bin_element,8,{integer,8,1},default,default},
                              {bin_element,8,{integer,8,2},default,default},
                              {bin_element,8,{integer,8,3},default,default},
                              {bin_element,8,{integer,8,4},default,default},
                              {bin_element,8,{integer,8,5},default,default}]}]}]}}

> erl_parse:parse_term([
  {'<<',8},
    {integer,8,1}, {',',8},
    {integer,8,2}, {',',8},
    {integer,8,3}, {',',8},
    {integer,8,4}, {',',8},
    {integer,8,5},
  {'>>',8},
  {dot,8}
]).
{ok,<<1,2,3,4,5>>}

> {ok, File} = file:read_file("mod.P"), 
  {ok, Tokens, _} = erl_scan:string(
    binary_to_list(File)
  ),
  {_, AST} = lists:foldl(fun
      (D = {dot, _}, {Buf, Acc}) ->
        {ok, R} = erl_parse:parse_form(
          lists:reverse([D|Buf])
        ),
        {[], [R|Acc]};
      (T, {Buf,Acc}) ->
        {[T|Buf], Acc}
    end,
    {[], []},
    Tokens
  ),
  lists:reverse(AST).
[{attribute,1,file,{"mod.erl",1}},
 {attribute,3,module,mod},
 {attribute,5,compile,export_all},
 {function,7,f,0,
           [{clause,7,[],[],
                    [{bin,8,
                          [{bin_element,8,{integer,8,1},default,default},
                           {bin_element,8,{integer,8,2},default,default},
                           {bin_element,8,{integer,8,3},default,default},
                           {bin_element,8,{integer,8,4},default,default},
                           {bin_element,8,{integer,8,...},default,default}]}]}]}]

More steps are executed after generating the AST, a linter and other checkers can verify the code, but it will be discarded in this part of the article. To finally generate the Erlang bytecode, the file with .beam prefix, containing the compiled source code, one can call erlc again or use the compile:file/2 function when a file is available.

$ erlc mod.erl

$ file mod.beam                                                                                                                                                                                                                                                                 
mod.beam: Erlang BEAM file

$ hexdump -C mod.beam        
00000000  46 4f 52 31 00 00 02 50  42 45 41 4d 41 74 55 38  |FOR1...PBEAMAtU8|
00000010  00 00 00 2d ff ff ff fb  30 6d 6f 64 10 66 b0 6d  |...-....0mod.f.m|
00000020  6f 64 75 6c 65 5f 69 6e  66 6f 60 65 72 6c 61 6e  |odule_info`erlan|
00000030  67 f0 67 65 74 5f 6d 6f  64 75 6c 65 5f 69 6e 66  |g.get_module_inf|
00000040  6f 00 00 00 43 6f 64 65  00 00 00 47 00 00 00 10  |o...Code...G....|
00000050  00 00 00 00 00 00 00 b1  00 00 00 07 00 00 00 03  |................|
00000060  01 10 99 10 02 12 22 00  01 20 40 47 00 03 13 01  |......".. @G....|
00000070  30 99 00 02 12 32 00 01  40 40 12 03 4e 10 00 01  |0....2..@@..N...|
00000080  50 99 00 02 12 32 10 01  60 40 03 13 40 12 03 4e  |P....2..`@..@..N|
00000090  20 10 03 00 53 74 72 54  00 00 00 00 49 6d 70 54  | ...StrT....ImpT|
000000a0  00 00 00 1c 00 00 00 02  00 00 00 04 00 00 00 05  |................|
000000b0  00 00 00 01 00 00 00 04  00 00 00 05 00 00 00 02  |................|
000000c0  45 78 70 54 00 00 00 28  00 00 00 03 00 00 00 03  |ExpT...(........|
000000d0  00 00 00 01 00 00 00 06  00 00 00 03 00 00 00 00  |................|
000000e0  00 00 00 04 00 00 00 02  00 00 00 00 00 00 00 02  |................|
000000f0  4c 69 74 54 00 00 00 17  00 00 00 00 00 00 00 01  |LitT............|
00000100  00 00 00 0b 83 6d 00 00  00 05 01 02 03 04 05 00  |.....m..........|
00000110  4d 65 74 61 00 00 00 2d  83 6c 00 00 00 01 68 02  |Meta...-.l....h.|
00000120  77 10 65 6e 61 62 6c 65  64 5f 66 65 61 74 75 72  |w.enabled_featur|
00000130  65 73 6c 00 00 00 01 77  0a 6d 61 79 62 65 5f 65  |esl....w.maybe_e|
00000140  78 70 72 6a 6a 00 00 00  4c 6f 63 54 00 00 00 04  |xprjj...LocT....|
00000150  00 00 00 00 41 74 74 72  00 00 00 27 83 6c 00 00  |....Attr...'.l..|
00000160  00 01 68 02 77 03 76 73  6e 6c 00 00 00 01 6e 10  |..h.w.vsnl....n.|
00000170  00 00 4a 2a a7 c3 41 6d  91 07 cf f0 1d 6d 6e 00  |..J*..Am.....mn.|
00000180  85 6a 6a 00 43 49 6e 66  00 00 00 48 83 6c 00 00  |.jj.CInf...H.l..|
00000190  00 03 68 02 77 07 76 65  72 73 69 6f 6e 6b 00 07  |..h.w.versionk..|
000001a0  39 2e 30 2e 36 2e 31 68  02 77 07 6f 70 74 69 6f  |9.0.6.1h.w.optio|
000001b0  6e 73 6a 68 02 77 06 73  6f 75 72 63 65 6b 00 13  |nsjh.w.sourcek..|
000001c0  2f 74 6d 70 2f 65 72 6c  61 6e 67 2f 6d 6f 64 2e  |/tmp/erlang/mod.|
000001d0  65 72 6c 6a 44 62 67 69  00 00 00 48 83 68 03 77  |erljDbgi...H.h.w|
000001e0  0d 64 65 62 75 67 5f 69  6e 66 6f 5f 76 31 77 11  |.debug_info_v1w.|
000001f0  65 72 6c 5f 61 62 73 74  72 61 63 74 5f 63 6f 64  |erl_abstract_cod|
00000200  65 68 02 77 04 6e 6f 6e  65 6c 00 00 00 01 68 02  |eh.w.nonel....h.|
00000210  77 03 63 77 64 6b 00 0b  2f 74 6d 70 2f 65 72 6c  |w.cwdk../tmp/erl|
00000220  61 6e 67 6a 4c 69 6e 65  00 00 00 15 00 00 00 00  |angjLine........|
00000230  00 00 00 00 00 00 00 03  00 00 00 01 00 00 00 00  |................|
00000240  51 00 00 00 54 79 70 65  00 00 00 0a 00 00 00 03  |Q...Type........|
00000250  00 00 00 01 0f ff 00 00                           |........|
00000258

If you have only the AST instead of a file containing the source code, the compile:forms/2 function can be invoked. It will return the name of the module as atom() and the bytecode.

> {ok, AST} = epp:parse_file("./mod.erl", [{source_name, mod}]).
{ok,[{attribute,1,file,{"mod",1}},
     {attribute,1,module,mod},
     {attribute,2,compile,export_all},
     {function,5,f,0,
               [{clause,5,[],[],
                        [{bin,5,
                              [{bin_element,5,{integer,5,1},default,default},
                               {bin_element,5,{integer,5,2},default,default},
                               {bin_element,5,{integer,5,...},default,default},
                               {bin_element,5,{integer,...},default,...},
                               {bin_element,5,{...},...}]}]}]},
     {eof,6}]}

> {ok, _Name, Bytecode} = compile:forms(AST, []).
{ok,mod,
    <<70,79,82,49,0,0,1,228,66,69,65,77,65,116,85,56,0,0,0,
      45,255,255,255,251,48,109,...>>}

> file:write_file("/tmp/mod.beam", Bytecode).
ok

To be sure the bytecode is working, let open another shell and load it.

$ cd /tmp

$ file mod.beam
mod.beam: Erlang BEAM file

$ hexdump -C mod.beam
00000000  46 4f 52 31 00 00 01 e4  42 45 41 4d 41 74 55 38  |FOR1....BEAMAtU8|
00000010  00 00 00 2d ff ff ff fb  30 6d 6f 64 10 66 b0 6d  |...-....0mod.f.m|
00000020  6f 64 75 6c 65 5f 69 6e  66 6f 60 65 72 6c 61 6e  |odule_info`erlan|
00000030  67 f0 67 65 74 5f 6d 6f  64 75 6c 65 5f 69 6e 66  |g.get_module_inf|
00000040  6f 00 00 00 43 6f 64 65  00 00 00 47 00 00 00 10  |o...Code...G....|
00000050  00 00 00 00 00 00 00 b1  00 00 00 07 00 00 00 03  |................|
00000060  01 10 99 10 02 12 22 00  01 20 40 47 00 03 13 01  |......".. @G....|
00000070  30 99 00 02 12 32 00 01  40 40 12 03 4e 10 00 01  |0....2..@@..N...|
00000080  50 99 00 02 12 32 10 01  60 40 03 13 40 12 03 4e  |P....2..`@..@..N|
00000090  20 10 03 00 53 74 72 54  00 00 00 00 49 6d 70 54  | ...StrT....ImpT|
000000a0  00 00 00 1c 00 00 00 02  00 00 00 04 00 00 00 05  |................|
000000b0  00 00 00 01 00 00 00 04  00 00 00 05 00 00 00 02  |................|
000000c0  45 78 70 54 00 00 00 28  00 00 00 03 00 00 00 03  |ExpT...(........|
000000d0  00 00 00 01 00 00 00 06  00 00 00 03 00 00 00 00  |................|
000000e0  00 00 00 04 00 00 00 02  00 00 00 00 00 00 00 02  |................|
000000f0  4c 69 74 54 00 00 00 17  00 00 00 00 00 00 00 01  |LitT............|
00000100  00 00 00 0b 83 6d 00 00  00 05 01 02 03 04 05 00  |.....m..........|
00000110  4c 6f 63 54 00 00 00 04  00 00 00 00 41 74 74 72  |LocT........Attr|
00000120  00 00 00 27 83 6c 00 00  00 01 68 02 77 03 76 73  |...'.l....h.w.vs|
00000130  6e 6c 00 00 00 01 6e 10  00 f2 e4 49 4c e5 8b 35  |nl....n....IL..5|
00000140  3f a4 aa a7 4e af 2c 9c  e1 6a 6a 00 43 49 6e 66  |?...N.,..jj.CInf|
00000150  00 00 00 28 83 6c 00 00  00 02 68 02 77 07 76 65  |...(.l....h.w.ve|
00000160  72 73 69 6f 6e 6b 00 07  39 2e 30 2e 36 2e 31 68  |rsionk..9.0.6.1h|
00000170  02 77 07 6f 70 74 69 6f  6e 73 6a 6a 44 62 67 69  |.w.optionsjjDbgi|
00000180  00 00 00 2e 83 68 03 77  0d 64 65 62 75 67 5f 69  |.....h.w.debug_i|
00000190  6e 66 6f 5f 76 31 77 11  65 72 6c 5f 61 62 73 74  |nfo_v1w.erl_abst|
000001a0  72 61 63 74 5f 63 6f 64  65 68 02 77 04 6e 6f 6e  |ract_codeh.w.non|
000001b0  65 6a 00 00 4c 69 6e 65  00 00 00 1b 00 00 00 00  |ej..Line........|
000001c0  00 00 00 00 00 00 00 03  00 00 00 01 00 00 00 01  |................|
000001d0  12 51 00 03 6d 6f 64 00  54 79 70 65 00 00 00 0a  |.Q..mod.Type....|
000001e0  00 00 00 03 00 00 00 01  0f ff 00 00              |............|
000001ec

$ erl -noshell -eval '
  io:format("~p~n", [mod:f()]),
  init:stop().
'
<<1,2,3,4,5>>

This is only the tip of the iceberg, the compilation process in Erlang is way more complex than that. In fact, many parts of the pipeline have been discarded. Anyway, the goal of this section was simply to show you how to generate quickly Erlang bytecode. The only interesting step for the goal of the post is the preprocessing one. The idea is to extract Erlang records and macros from Erlang source code and convert them into something usable in Elixir.

A last note regarding the compilation process. As you probably know, Elixir templating system using macros is really powerful and the abstraction has been really well designed. Did you know Erlang got the same feature years ago? Indeed, it is also possible to compile dynamically the code or alter the AST directly from the virtual machine via merl (for Metaprogramming in Erlang). It's way more complex to use than Elixir templates, but it works pretty well.

BONUS: it is also possible to disassemble the bytecode generated by this process using the beam_lib:all_chunks/1 function. It will then give you a better view of the content of a beam file.

$ erl -noshell -eval '
  {ok, F} = file:read_file("./mod.beam"),
  {ok,_,Bytes} = beam_lib:all_chunks(F),
  io:format("~p~n", [Bytes]),
  init:stop().
'
[{"AtU8",
  <<255,255,255,251,48,109,111,100,16,102,176,109,111,100,117,108,101,95,105,
    110,102,111,96,101,114,108,97,110,103,240,103,101,116,95,109,111,100,117,
    108,101,95,105,110,102,111>>},
 {"Code",
  <<0,0,0,16,0,0,0,0,0,0,0,177,0,0,0,7,0,0,0,3,1,16,153,16,2,18,34,0,1,32,64,
    71,0,3,19,1,48,153,0,2,18,50,0,1,64,64,18,3,78,16,0,1,80,153,0,2,18,50,16,
    1,96,64,3,19,64,18,3,78,32,16,3>>},
 {"StrT",<<>>},
 {"ImpT",<<0,0,0,2,0,0,0,4,0,0,0,5,0,0,0,1,0,0,0,4,0,0,0,5,0,0,0,2>>},
 {"ExpT",
  <<0,0,0,3,0,0,0,3,0,0,0,1,0,0,0,6,0,0,0,3,0,0,0,0,0,0,0,4,0,0,0,2,0,0,0,0,0,
    0,0,2>>},
 {"LitT",<<0,0,0,0,0,0,0,1,0,0,0,11,131,109,0,0,0,5,1,2,3,4,5>>},
 {"Meta",
  <<131,108,0,0,0,1,104,2,119,16,101,110,97,98,108,101,100,95,102,101,97,116,
    117,114,101,115,108,0,0,0,1,119,10,109,97,121,98,101,95,101,120,112,114,
    106,106>>},
 {"LocT",<<0,0,0,0>>},
 {"Attr",
  <<131,108,0,0,0,1,104,2,119,3,118,115,110,108,0,0,0,1,110,16,0,0,74,42,167,
    195,65,109,145,7,207,240,29,109,110,0,133,106,106>>},
 {"CInf",
  <<131,108,0,0,0,3,104,2,119,7,118,101,114,115,105,111,110,107,0,7,57,46,48,
    46,54,46,49,104,2,119,7,111,112,116,105,111,110,115,106,104,2,119,6,115,
    111,117,114,99,101,107,0,19,47,116,109,112,47,101,114,108,97,110,103,47,
    109,111,100,46,101,114,108,106>>},
 {"Dbgi",
  <<131,104,3,119,13,100,101,98,117,103,95,105,110,102,111,95,118,49,119,17,
    101,114,108,95,97,98,115,116,114,97,99,116,95,99,111,100,101,104,2,119,4,
    110,111,110,101,108,0,0,0,1,104,2,119,3,99,119,100,107,0,11,47,116,109,
    112,47,101,114,108,97,110,103,106>>},
 {"Line",<<0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,1,0,0,0,0,81>>},
 {"Type",<<0,0,0,3,0,0,0,1,15,255>>}]

If you are even more curious about this bonus, you are invited to read The Beam Book, you will find there way more information!

Erlang Preprocessor (`epp`)

The epp module is in charge to parse and expand macros from Erlang source code. Its source code implementation resides in lib/stdlib/epp.erl. The content can be intimidating with its 2600 lines of code, but it is not the most complex to read.

epp is starting its own server process handling I/O protocols to read data byte by byte with the help of the io:scan_erl_form/3. This server is having its own state updated every time a new correct value is parsed. Let begins our investigation by using the easiest function available to parse and expands macros, epp:parse_file/2. Ah, before doing that, let improve a bit our test module.

$ cat > mod2.erl << EOF
-module(mod2).
-compile(export_all).
-record(test, {field1}).
-define(DATA, <<1,2,3,4,5>>).

f() -> ?DATA
EOF

An Erlang record called test has been added. It was missing from the previous module.

> {ok, AST} = epp:parse_file("mod2.erl", []).
{ok,[{attribute,1,file,{"mod2.erl",1}},
     {attribute,1,module,mod2},
     {attribute,2,compile,export_all},
     {attribute,3,record,
                {test,[{record_field,3,{atom,3,field1}}]}},
     {function,6,f,0,
               [{clause,6,[],[],
                        [{bin,6,
                              [{bin_element,6,{integer,6,1},default,default},
                               {bin_element,6,{integer,6,...},default,default},
                               {bin_element,6,{integer,...},default,...},
                               {bin_element,6,{...},...},
                               {bin_element,6,...}]}]}]},
     {eof,7}]}

As you can see, a new attribute for the record is present, containing its description but there is no trace of the DATA macro defined (again). This is because epp returns the fully expand source code. Great, but those macros should be stored somewhere... right? Indeed, they are stored inside the #epp{} record present in the epp server's state. Unfortunately, this server is not directly available when using epp:parse_file/2, and epp:open/1 must be used instead. This function is opening a file using I/O protocols and returns a process identifier to extract all tokens step by steps.

> {ok, EppProcess} = epp:open("mod2.erl", [], []).
{ok,<0.175.0>}

To avoid repetitive task, let create EppRead/2 lambda function to incrementally read the tokens from the epp server. Every time epp:scan_erl_form/1 is called, the epp server should send a message containing its position in the file. When it reaches the end of the file, it sends {eof, Line}.

> EppRead = fun EppRead(Pid, Buffer) ->
  case epp:scan_erl_form(Pid) of
    {ok, T} -> EppRead(Pid, Buffer ++ T);
    {eof, _} -> {ok, Buffer}
  end
end.
#Fun<erl_eval.18.113135111>

> {ok, Forms} = EppRead(EppProcess, []).
[{'-',1},
 {atom,1,file},
 {'(',1},
 {string,1,"mod.erl"},
 {',',1},
 {integer,1,1},
 {')',1},
 {dot,1},
 {'-',1},
 {atom,1,module},
 {'(',1},
 {atom,1,mod},
 {')',1},
 {dot,1},
 {'-',2},
 {atom,2,compile},
 {'(',2},
 {atom,2,export_all},
 {')',2},
 {dot,2},
 {atom,5,f},
 {'(',5},
 {')',5},
 {'->',5},
 {'<<',5},
 {integer,5,1},
 {',',5},
 {integer,...},
 {...}|...]

Great, the tokens are available, it means it works correctly without any syntax errors. How to extract the process state? The epp server does not support sys patterns, and using sys:get_state/1 will simply froze the current process. If you read the source code of the module, you will find the undocumented macro_defs/1 function. It sends a message to an epp server and should return the list of macros stored in server's state. Let reimplement this function ourselves.

> GetMacros = fun (Pid) ->
  Pid ! {epp_request, self(), macro_defs},
  receive
    {epp_reply, Pid, Macros} -> Macros
  after
    1000 -> error
  end
end.
#Fun<erl_eval.42.113135111>

> GetMacros(EppProcess).
[{{atom,'FEATURE_ENABLED'},[{1,{['X'],[{atom,1,false}]}}]},
 {{atom,'FEATURE_AVAILABLE'},
  [{1,
    {['X'],
     [{'(',1},
      {'(',1},
      {var,1,'X'},
      {')',1},
      {'==',1},
      {atom,1,maybe_expr},
      {')',1}]}}]},
 {{atom,'OTP_RELEASE'},{none,[{integer,1,28}]}},
 {{atom,'MACHINE'},{none,[{atom,1,'BEAM'}]}},
 {{atom,'LINE'},{none,[{integer,1,1}]}},
 {{atom,'FILE'},{none,[{string,1,"mod2.erl"}]}},
 {{atom,'MODULE'},{none,[{atom,1,mod2}]}},
 {{atom,'MODULE_STRING'},{none,[{string,1,"mod2"}]}},
 {{atom,'BASE_MODULE'},undefined},
 {{atom,'BASE_MODULE_STRING'},undefined},
 {{atom,'FUNCTION_ARITY'},undefined},
 {{atom,'FUNCTION_NAME'},undefined},
 {{atom,'BEAM'},{none,[{atom,1,true}]}},
 {{atom,'DATA'},
  [{none,{none,[{'<<',4},
                {integer,4,1},
                {',',4},
                {integer,4,2},
                {',',4},
                {integer,4,...},
                {',',...},
                {...}|...]}}]}]

Indeed, all the macros have been returned, including the one defined by epp. The format of the macros stored here can be described below

{
  {atom, MacroName},
  {MacroArity, MacroTermOrExprs}
}.

The name of the macro is defined in MacroName as a charlist() type. The next part is the definition of the macro where:

MacroArity is the arity of the macro. If the macro does not have argument, it is set to none, else, it is set to a strictly positive integer greater than 0.
MacroTermOrExprs is the content of the macro, it can contain an Erlang term or an expression, and can also contain a variable previously defined in the argument part.

Because mod2.erl does not define any macros with arguments, let creates one to see how it is represented.

-define(DYN(X,Y,Z), X + Y + Z).

This macro when extracted looks like the Erlang term below.

{
  {atom,'DYN'},
  [
    {3,
      {['X','Y','Z'],[
        {var,4,'X'},
        {'+',4},
        {var,4,'Y'},
        {'+',4},
        {var,4,'Z'}
      ]}
    }
  ]
}

As you can see, this is not so different than the rest of AST, the arguments defined in the macro are expanded in the body of the macro using the token {var, Line, ArgumentName}. Finally, don't forget to close the file to avoid file descriptor exception at the end of this experiment.

> epp:close(EppServer).
ok

That's great, it is now possible to extract both records and macros, one from the AST, the other one directly from the preprocessor state. Let find a way to deal with that in Elixir.

Erlang Records in Elixir

Elixir already has an interface to deal with Erlang records called Record. This is working well in most of the case, let just have a look on it. First, a record can only created inside a module, because it uses Elixir meta-programming feature with macros and templates.

$ cat > r.ex << EOF
defmodule R do
  require Record

  Record.defrecord(:test, data: "test")
end
EOF

iex> c("r.ex")
[R]

iex> import R
R

iex> R.test()
{:test, "test"}

iex> R.test(:data)
1

iex> R.test() |> R.test(data: "updated")
{:test, "updated"}

Okay, great, but now, let check a real-world Erlang records from some very used modules, like public_key. Most if not all records are defined in public_key/include/public_key.hrl. The first one you will encounter is the RSAPublicKey record.

-record('RSAPublicKey', {
  modulus,
  publicExponent
 }).

Can you see the problem here? When using the Elixir Record module, it creates a function to simulate the behavior of the record, but here, the record identifier is a word starting with an uppercase character. It is not possible to create a function with an uppercase character in Elixir. Just to be sure, a new module containing a function starting with an uppercase letter can be created.

$ cat > up.ex << EOF
defmodule Up do
  def TestFunction(), do: 1
end
EOF

iex> c("up.ex")
== Compilation error in file up.ex ==
** (SyntaxError) invalid syntax found on up.ex:2:19:
    error: unexpected ( after alias TestFunction. Function names and identifiers in Elixir start with lowercase characters or underscore. For example:

        hello_world()
        _starting_with_underscore()
        numb3rs_are_allowed()
        may_finish_with_question_mark?()
        may_finish_with_exclamation_mark!()

    Unexpected token: (
    │
  2 │   def TestFunction(), do: 1
    │                   ^
    │
    └─ up.ex:2:19
    (elixir 1.19.5) lib/kernel/parallel_compiler.ex:530: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/8
** (CompileError) compile error
    (iex 1.19.5) lib/iex/helpers.ex:245: IEx.Helpers.c/2
    iex:15: (file)

Well, you would probably say something like "not a problem, if it's the only record...", I stop you right there. More than 150 records are using this format in the public_key module. Defining 150+ functions is annoying, but because they are all following the same pattern, macros or module templates can be used.

This issue can be solved in many different ways, the first one is to create a function record/1 where the first argument will contain the record name as an atom() starting with an uppercase. Every new record can be added like that.

iex> R.record(:RSAPrivateKey)
{:RSAPrivateKey, :undefined, :undefined}

iex> R.record(:RSAPrivateKey, modulus: 3, publicExponent: 111)
{:RSAPrivateKey, 3, 111}

iex> R.record(:RSAPrivateKey) |> R.record(:modulus, 3)
{:RSAPrivateKey, 3, :undefined}

The second solution is the one I prefer. Instead of creating a function containing all record in one place, why not creating a module containing an Elixir struct to represent the record itself? Well, let create the module in charge of that. It will be called Preprocessing and the sources are already available on github

defmodule Preprocessing.Record do
end

Listing the available records from an Erlang file is one of the main requirement. To do that, it is possible to mix 2 functions, :code.lib_dir/1 to get the path of the module, and :epp.parse_file/2 to parse the Erlang source code. The list_records/3 function returns all the record identifier as [atom()].

  def list_records(module, filepath, opts \\ %{}) do
    case :code.lib_dir(module) do
      {:error, reason} ->
        {:error, reason}
      path ->
        target = Path.join([path, filepath])
        list_records2(module, target, opts)
    end
  end

  defp list_records2(module, target, opts) do
    source_name = Map.get(opts, :source_name, module)
    with true <- File.exists?(target),
         {:ok, tokens} <- :epp.parse_file(target, source_name: source_name)
    do
      for {:attribute, _, :record, {record_name, _}} <- tokens, do: record_name
    else
      false -> {:error, :path};
      other -> other
    end
  end

Next, a way to extract the records is required as well. In this case, the Record.extract/2 can be reused without, it has been created for that.

  def extract(module, filepath, record) do
    target = Path.join([Atom.to_string(module), filepath])
    Record.extract(record, from_lib: target)
  end

Finally, the module template can be created with the help of the special macro called __using__/1. When used on another module, it will generate all the required functions to deal with the imported record.

defmacro __using__(opts) do
    record_id = Keyword.get(opts, :record_id)
    module = Keyword.get(opts, :module)
    filepath = Keyword.get(opts, :filepath)
    fields = extract(module, filepath, record_id)
    keys = for {k, _} <- fields, do: k
    values = for {_, v} <- fields, do: v
    tuple = [:record_id] ++ values
      |> List.to_tuple()
      |> Macro.escape()
    record_size = length(fields)+1

    quote do
      import Preprocessing.Record

      defstruct unquote(fields)

      def id(), do: unquote(record_id)

      def fields(), do: unquote(fields)

      def keys(), do: unquote(keys)

      def tuple(), do: unquote(tuple)

      def struct(), do: %__MODULE__{}

      def is_valid?(record) do
        cond do 
          is_tuple(record) and 
            :erlang.size(record) == unquote(record_size) and 
            :erlang.element(1, record) == unquote(record_id) -> true
          true -> false
        end
      end

      def convert(struct = %__MODULE__{}) do
        map = Map.from_struct(struct)
        {:ok, Enum.reduce(fields(), [], fn ({k, _}, acc) -> 
            [Map.get(map, k)|acc]
          end)
          |> Enum.reverse()
          |> (fn(xs) -> [unquote(record_id)|xs] end).()
          |> List.to_tuple()
        }
      end
      def convert(record) when is_tuple(record) do
        if is_valid?(record) do
          record_list =
            Tuple.to_list(record)
            |> Enum.drop(1)

          {:ok, keys()
            |> Enum.zip(record_list)
            |> Enum.reduce(%__MODULE__{}, fn ({k, v}, acc) -> 
              Map.put(acc, k, v)
            end)
          }
        else
          {:error, :invalid_tuple}
        end
      end
      def convert(_), do: {:error, :invalid_term}

      def convert!(record) do
        case convert(record) do
          {:ok, data} -> data
          other -> throw other
        end
      end

    end
  end

Great! Let creates our first interface to an existing record. #RSAPublicKey{} from the public_key module is a good target.

defmodule Preprocessing.PublicKey.RSAPublicKey do
  use Preprocessing.Record,
    record_id: :RSAPublicKey,
    module: :public_key,
    filepath: "include/public_key.hrl"
end

That's all. The project can now be recompile and it will be possible to test our new module.

iex> alias Preprocessing.PublicKey.RSAPublicKey
Preprocessing.PublicKey.RSAPublicKey

iex> RSAPublicKey.is_valid?({:RSAPublicKey, :undefined, :undefined})
true

iex> RSAPublicKey.is_valid?({:RSAPublicKey})
false

iex> RSAPublicKey.keys()
[:modulus, :publicExponent]

iex> RSAPublicKey.struct()
%PublicKey.RSAPublicKey{
  modulus: :undefined,
  publicExponent: :undefined
}

iex> %RSAPublicKey{}
%RSAPublicKey{
  modulus: :undefined,
  publicExponent: :undefined
}

iex> RSAPublicKey.convert(%RSAPublicKey{})
{:RSAPublicKey, :undefined, :undefined}

iex> RSAPublicKey.convert({:RSAPublicKey, :undefined, :undefined})
%PublicKey.RSAPublicKey{
  modulus: :undefined,
  publicExponent: :undefined
}

This module is not perfect though. For example, a function to valid and convert the data correctly from the Elixir struct to the record and vice verse is required. At this time, the developer is in charge of creating the interface for this process. This solution still looks better than the first one. Every new records are treated as individual data-structure defined in their own containers with their own functions.

Erlang Macros Definition in Elixir

The Erlang preprocessor macros are not so different than the Erlang records, except when it comes to their arity. In this implementation, only static macros will be supported, but it should not be too complex to implement something with arguments support. Anyway, in a previous section, we saw the macros where stored in the epp server process state and must be extracted with an undocumented function from the epp module. In fact, most of the hard work is already done, now, we need a way to integrate them in Elixir.

Static macros are basically made of a key and a value. Why not converting them as Map.t() or Keyword.t() in this case? In fact, a macro with the same name than another one present below will overwrite the value of the previous definition. This means it's impossible to have a naming conflict, perfect for a Map.t(). First, let creates a new module called Preprocessing.Macros, it will contain all the functions required to extract and convert macros, but also a template to include them inside another module.

defmodule Preprocessing.Macros do
end

The definitions/2 functions is the only one exported from the module, and will return the converted list of macros as Map.t(). This function is used to generate the template but also to find all the macros from a specific module while developing.

  def definitions(module, filepath) do
    with {:ok, pid} <- open_file(module, filepath),
      :ok <- definitions_loop(pid),
      {:ok, macros} <- definitions_macros(pid)
    do
      convert_macros(macros, [])
    end
  end

All the remaining functions are not exposed to the rest of the application, and are being used to deal with every step of the conversion. The open_file/2 function is a wrapper around :epp.open/2, checking if a path is available and returning the process id of the epp server.

  defp open_file(module, filepath) do
    case :code.lib_dir(module) do
      {:error, reason} ->
        {:error, reason}
      path ->
        Path.join([path, filepath])
          |> String.to_charlist()
          |> :epp.open([], [])
    end
  end

The definitions_loop/1 function is quite stupid. It will simply read the content of the opened file until it reaches the end of it. When done, it simply return :ok. In fact, this step is very important, because it will load all the macros from the file and store them in the epp server's state.

  defp definitions_loop(pid) do
    msg = :epp.parse_erl_form(pid)
    case msg do
      {:eof, _} ->
        :ok
      _ ->
        definitions_loop(pid)
    end
  end

The definitions_macros/1 function sends the undocumented :macro_defs message to the eep server and wait for its answer containing the list of all parsed macros from the source code. Those macros (a list of tuple) is then returned.

  defp definitions_macros(pid) do
    send(pid, {:epp_request, self(), :macro_defs})
    receive do
      {:epp_reply, pid, macros} ->
        :epp.close(pid)
        {:ok, macros}
      msg ->
        :epp.close(pid)
        {:error, msg}
      after
        10000 ->
          :epp.close(pid)
          {:error, :timeout}
      end
  end

Every raw macros from the epp server's state must be converted. The convert_macros/2 and convert_macros_value/1 functions are doing the conversion and the filtering. Indeed, we are looking only for the static macros right now, not for the one with an arity, then, only the macros set to :none are kept, the rest are simply dropped.

  defp convert_macros([], buffer) do
    {:ok,
      buffer
      |> Enum.filter(fn({_, v}) ->
        if v==:ignored do
          false
        else
          true
        end
     end)
    }
  end
  defp convert_macros([macro={{:atom, name}, value}|rest], buffer) do
    with {:ok, term} <- convert_macro_value(value) do
      convert_macros(rest, [{name, term}|buffer])
    else
      error -> {:error, error, macro}
    end
  end

  defp convert_macro_value([none: {:none, value}]) do
    with {:ok, term} <- :erl_parse.parse_term(value ++ [{:dot, 1131}]) do
      {:ok, term}
    end
  end
  defp convert_macro_value(_), do: {:ok, :ignored}

The escape_keys/2 function will escape the Erlang macros definitions to insert them safely in the module template via the Macro.escape/2 function.

  defp escape_keys(module, filepath) do
    with {:ok, defs} <- definitions(module, filepath) do
      defs
      |> :maps.from_list()
      |> Macro.escape()
    end
  end

The macro_keys/2 and macro_values/2 functions are respectively extracting the keys and the values of the macros, converting them into a [term()].

  defp macro_keys(module, filepath) do
    with {:ok, defs} <- definitions(module, filepath) do
      defs
      |> Enum.map(fn({k,_}) -> k end)
      |> Macro.escape()
    end
  end

  defp macro_values(module, filepath) do
    with {:ok, defs} <- definitions(module, filepath) do
      defs
      |> Enum.map(fn({_,v}) -> v end)
      |> Macro.escape()
    end
  end

The module template definition is probably the most important part here, because it will integrate the data from the Erlang macros into the Elixir AST. 5 functions have been created to help users to check their macros:

macros/0: returns the macros as a Map.t();
macro/1: returns the value of a defined macros using its key;
macro_keys/0: returns the list of all macro's keys;
macro_values/0: returns the list of all macro's values;
macro_filter/1: a wrapper around Map.filter/2 to filter the macros.

  defmacro __using__(opts) do
    module = Keyword.get(opts, :module)
    filepath = Keyword.get(opts, :filepath)
    def_keys = escape_keys(module, filepath)

    quote do
      import Preprocessing.Macros

      def macros(), do: unquote(def_keys)

      def macro(index), do: Map.get(unquote(def_keys), index)

      def macro_keys(), do: unquote(macro_keys(module, filepath))

      def macro_values(), do: unquote(macro_values(module, filepath))

      def macros_filter(fun) do
        unquote(def_keys)
        |> Map.filter(fun)
      end
    end
  end

Great, the goal now is to create or update a module with this template. The best target here could be the Preprocessing.PublicKey module. The module to parse the Erlang macros is :public_key and the path for the header is inclyde/public_key.hrl.

defmodule Preprocessing.PublicKey do
  use Preprocessing.Macros,
    module: :public_key,
    filepath: "include/public_key.hrl"
end

Finally, this interface can be tested. The definitions/2 function returns the correct values from different header files.

iex> alias Preprocessing.Macros
Preprocessing.Macros

iex> Macros.definitions(:kernel, "include/file.hrl")
{:ok, [FILE_HRL_: 1]}

iex> Macros.definitions(:kernel, "include/inet.hrl")
{:ok, []}

Let have a look on the Preprocessing.PublicKey module now.

iex> alias Preprocessing.PublicKey
alias Preprocessing.PublicKey

The PublicKey.macros/0 function returns all the available macros parsed from the Erlang source code as a Map.t().

iex> PublicKey.macros()
%{
  sect163r1: {1, 3, 132, 0, 2},
  "ub-postal-code-length": 16,
  secp256k1: {1, 3, 132, 0, 10},
  "id-RSAES-OAEP": {1, 2, 840, 113549, 1, 1, 7},
  "id-ce-policyMappings": {2, 5, 29, 33},
  "id-at-pseudonym": {2, 5, 4, 65},
  affiliationChanged: 3,
  "id-at-localityName": {2, 5, 4, 7},
  "id-at-givenName": {2, 5, 4, 42},
  "id-pkix-ocsp-service-locator": {1, 3, 6, 1, 5, 5, 7, 48, 1, 7},
  "ub-name": 32768,
  "id-qt-cps": {1, 3, 6, 1, 5, 5, 7, 2, 1},
  "pbeWithSHA1AndDES-CBC": {1, 2, 840, 113549, 1, 5, 10},
  secp160k1: {1, 3, 132, 0, 9},
  sect409k1: {1, 3, 132, 0, 36},
  "id-ce-extKeyUsage": {2, 5, 29, 37},
  "id-ml-dsa-87": {2, 16, 840, 1, 101, 3, 4, 3, 19},
  emptyString: "",
  ...
}

The PublicKey.macro/1 function is doing its job by returning the value of an existing key, or nil if the key is not present.

iex> PublicKey.macro(:sect163r1)
{1, 3, 132, 0, 2}

iex> PublicKey.macro(:undefined)
nil

The PublicKey.macro_keys/0 and PublicKey.macro_values/0 are returning respectively the keys and the values from the parsed macros.

iex> PublicKey.macro_keys()
[:sect163r1, :"ub-postal-code-length", :secp256k1, :"id-RSAES-OAEP",
 :"id-ce-policyMappings", :"id-at-pseudonym", :affiliationChanged, ...]

iex> PublicKey.macro_values
[
  {1, 3, 132, 0, 2},
  16,
  {1, 3, 132, 0, 10},
  {1, 2, 840, 113549, 1, 1, 7},
  ...
]

At last, the PublicKey.macro_filter/1 can filter the keys/values from the macros to return only the one requested via the Map.filter/2 function. The following snippet is filtering the keys using a Regex

iex>  PublicKey.macro_filter(fn ({key,_value}) -> 
  cond do
    Regex.match?(~r"^rsa"i, Atom.to_string(key)) -> true; 
    true -> false 
  end 
end)
%{
  "rSASSA-PSS-Default-Identifier": {:"RSASSA-AlgorithmIdentifier",
   {1, 2, 840, 113549, 1, 1, 10},
   {:"RSASSA-PSS-params", {:HashAlgorithm, {1, 3, 14, 3, 2, 26}, :NULL},
    {:MaskGenAlgorithm, {1, 2, 840, 113549, 1, 1, 8},
     {:HashAlgorithm, {1, 3, 14, 3, 2, 26}, :NULL}}, 20, 1}},
  "rSAES-OAEP-Default-Identifier": {:"RSAES-AlgorithmIdentifier",
   {1, 2, 840, 113549, 1, 1, 7},
   {:"RSAES-OAEP-params", {:Externalvaluereference, 354, :"PKCS-1", :sha1},
    {:Externalvaluereference, 355, :"PKCS-1", :mgf1SHA1},
    {:Externalvaluereference, 356, :"PKCS-1", :pSpecifiedEmpty}}},
  rsaEncryption: {1, 2, 840, 113549, 1, 1, 1}
}

It is now possible to extract and convert macros from Erlang world to the Elixir universe. This implementation is perhaps not the best one, but it will make my life easier while dealing with the public_key module.

Real World Application

This whole post is the consequence of a sandbox project called expki, created to remove all my rust from years of doing Erlang and infrastructure stuff instead of Elixir. The goal was to use Elixir and Phoenix framework to manage a small local PKI (Public Key Infrastructure) compatible with openssl and easy_rsa, a tool provided by OpenVPN to manage certificates locally. In fact, when this project started, it was expected to have a working PoC quick because Erlang was already offering a good support for TLS with the public_key, crypto and asn1 Erlang applications.

Unfortunately, most of the records name present in those modules are starting with an uppercase letter... and it blocks to create something in a short amount of time. A big part of my time on expki was dedicated to read the source code and the documentation of these modules or trying to find a better way to deal with the structure from them. Here the whole list of them extract from an older version of Erlang/OTP.

from public_key (157):
- #'AAControls'{}
- #'ACClearAttrs'{}
- #'AccessDescription'{}
- #'AlgorithmIdentifier'{}
- #'AlgorithmIdentifierPKCS5v2{}
- #'AlgorithmIdentifierPKCS{}
- #'AlgorithmIdentifier{}
- #'AnotherName'{}
- #'AttCertValidityPeriod'{}
- #'Attribute'{}
- #'AttributeCertificate'{}
- #'AttributeCertificateInfo'{}
- #'AttributePKCS{}
- #'AttributeTypeAndValue'{}
- #'Attributes_SETOF'{}
- #'Attributes_SETOF_valuesWithContext_SETOF'{}
- #'AuthorityKeyIdentifier'{}
- #'BasicConstraints'{}
- #'BasicOCSPResponse'{}
- #'BuiltInDomainDefinedAttribute'{}
- #'BuiltInStandardAttributes'{}
- #'CertID'{}
- #'Certificate'{}
- #'CertificateList'{}
- #'CertificationRequest'{}
- #'CertificationRequestInfo'{}
- #'CertificationRequestInfo_attributes_SETOF'{}
- #'CertificationRequestInfo_subjectPKInfo'{}
- #'CertificationRequestInfo_subjectPKInfo_algorithm'{}
- #'CertificationRequest_signatureAlgorithm'{}
- #'Characteristic{}
- #'Clearance'{}
- #'ContentEncryptionAlgorithmIdentifier'{}
- #'ContentInfo'{}
- #'Context'{}
- #'CrlID'{}
- #'Curve'{}
- #'DHParameter'{}
- #'DSAPrivateKey'{}
- #'DigestAlgorithm'{}
- #'DigestAlgorithmIdentifier'{}
- #'DigestEncryptionAlgorithmIdentifier'{}
- #'DigestInfoPKCS{}
- #'DigestedData'{}
- #'DistributionPoint'{}
- #'DomainParameters'{}
- #'Dss{}
- #'ECDSA{}
- #'ECParameters'{}
- #'ECPoint'{}
- #'ECPrivateKey'{}
- #'EDIPartyName'{}
- #'EncryptedContentInfo'{}
- #'EncryptedData'{}
- #'EncryptedPrivateKeyInfo'{}
- #'EncryptedPrivateKeyInfo_encryptionAlgorithm'{}
- #'EnvelopedData'{}
- #'ExtendedNetworkAddress_e163{}
- #'Extension'{}
- #'ExtensionAttribute'{}
- #'Extension{}
- #'FieldID'{}
- #'GeneralSubtree'{}
- #'HashAlgorithm'{}
- #'Holder'{}
- #'IetfAttrSyntax'{}
- #'IssuerAndSerialNumber'{}
- #'IssuerSerial'{}
- #'IssuingDistributionPoint'{}
- #'KeyEncryptionAlgorithmIdentifier'{}
- #'MaskGenAlgorithm'{}
- #'NameConstraints'{}
- #'NoticeReference'{}
- #'OCSPRequest'{}
- #'OCSPResponse'{}
- #'ORAddress'{}
- #'OTPAttributeTypeAndValue'{}
- #'OTPCertificate'{}
- #'OTPCharacteristic{}
- #'OTPExtension'{}
- #'OTPExtensionAttribute'{}
- #'OTPFieldID'{}
- #'OTPNoticeReference'{}
- #'OTPOLDSubjectPublicKeyInfo'{}
- #'OTPOLDSubjectPublicKeyInfo_algorithm'{}
- #'OTPSubjectPublicKeyInfo'{}
- #'OTPSubjectPublicKeyInfo{}
- #'OTPTBSCertificate'{}
- #'OTPUserNotice'{}
- #'ObjectDigestInfo'{}
- #'OneAsymmetricKey'{}
- #'OneAsymmetricKey_privateKeyAlgorithm'{}
- #'OtherPrimeInfo'{}
- #'PBEParameter'{}
- #'PBES2{}
- #'PBKDF2{}
- #'PBMAC1{}
- #'PDSParameter'{}
- #'PKAttribute'{}
- #'PKAttribute_valuesWithContext_SETOF'{}
- #'PSourceAlgorithm'{}
- #'Pentanomial'{}
- #'PersonalName'{}
- #'PolicyConstraints'{}
- #'PolicyInformation'{}
- #'PolicyMappings_SEQOF'{}
- #'PolicyQualifierInfo'{}
- #'PreferredSignatureAlgorithm'{}
- #'PresentationAddress'{}
- #'PrivateKeyInfo'{}
- #'PrivateKeyInfo_privateKeyAlgorithm'{}
- #'PrivateKeyUsagePeriod'{}
- #'PublicKeyAlgorithm'{}
- #'RC2{}
- #'RC5{}
- #'RSAES{}
- #'RSAPrivateKey'{}
- #'RSAPublicKey'{}
- #'RSASSA{}
- #'RecipientInfo'{}
- #'Request'{}
- #'ResponseBytes'{}
- #'ResponseData'{}
- #'RevokedInfo'{}
- #'RoleSyntax'{}
- #'SecurityCategory'{}
- #'ServiceLocator'{}
- #'Signature'{}
- #'SignatureAlgorithm'{}
- #'SignatureAlgorithm{}
- #'SignedAndEnvelopedData'{}
- #'SignedData'{}
- #'SignerInfo'{}
- #'SignerInfoAuthenticatedAttributes_aaSequence_SEQOF'{}
- #'SignerInfoAuthenticatedAttributes_aaSet_SETOF'{}
- #'SignerInfo_unauthenticatedAttributes_uaSequence_SEQOF'{}
- #'SignerInfo_unauthenticatedAttributes_uaSet_SETOF'{}
- #'SingleResponse'{}
- #'SubjectPublicKeyInfo'{}
- #'SubjectPublicKeyInfoAlgorithm'{}
- #'SubjectPublicKeyInfo{}
- #'SvceAuthInfo'{}
- #'TBSCertList'{}
- #'TBSCertList_revokedCertificates_SEQOF'{}
- #'TBSCertificate'{}
- #'TBSRequest'{}
- #'TargetCert'{}
- #'TeletexDomainDefinedAttribute'{}
- #'TeletexPersonalName'{}
- #'UnformattedPostalAddress'{}
- #'UserNotice'{}
- #'V2Form'{}
- #'ValidationParms'{}
- #'Validity'{}
- #cert{}
- #path_validation_state{}
- #revoke_state{}
from asn1 (31):
- #'ComponentType'{}
- #'Constraint'{}
- #'EXTENSIONMARK'{}
- #'ExtensionAdditionGroup'{}
- #'Externaltypereference'{}
- #'Externalvaluereference'{}
- #'Object'{}
- #'ObjectClassFieldType'{}
- #'ObjectSet'{}
- #'SEQUENCE'{}
- #'SET'{}
- #'SymbolsFromModule'{}
- #abst{}
- #classdef{}
- #cmap{}
- #gen_state{}
- #gen{}
- #module{}
- #objectclass{}
- #pobjectdef{}
- #pobjectsetdef{}
- #ptypedef{}
- #pvaluedef{}
- #pvaluesetdef{}
- #seqtag{}
- #simpletableattributes{}
- #state{}
- #tag{}
- #typedef{}
- #type{}
- #valuedef{}

That's a lot, right? In reality, not all are required for a Poc, only ~10 or ~20 of them will be used. This is where the preprocessing will help, by creating the right interface for the right object. This post is already way to long... And we will see that on another one.

A Quick Note About `epp_dodger` Module

epp_dodger is another module available in the Erlang OTP toolsuite to parse valid Erlang source code. So, why not using it instead of epp and all the complexity relative to its server?

Firstly because I wanted to dig a bit more in the Erlang parsing process. Most of my researches were focused on the "default execution path", and I totally missed epp_dodger, even if I read the documentation.

Secondly, the epp_dodger module was not loaded by default with iex (this module is from the syntax_tools application) and epp was already available (this module is from the stdlib application).

Thirdly, the Erlang toolbox is huge, I don't even think someone know every single tool available there. Every time I start to work on Erlang/OTP source code, I discover new utils.

Finally, it was not planned to implement my own interface to the public_key module, I also missed the x509 from hex. In fact, it is also interesting to see the differences between both implementation and it will also give me a way to improve the test suite.

Anyway, let fix that with a quick example of epp_dodger usage. If you want to have it in Elixir, you will need to load the syntax_tools application.

> Application.ensure_all_started(:syntax_tools)
{:ok, [:syntax_tools]}

Then, the function :epp_dodger.parse_file/1 can be called

> [:code.lib_dir(:public_key), "include/public_key.hrl"]
  |> Path.join()
  |> :epp_dodger.parse_file()
{:ok,
 [
   {:tree, :attribute, {:attr, 24, [], :none},
    {:attribute, {:atom, 24, :ifndef}, [{:atom, 24, :public_key}]}},
   {:tree, :attribute, {:attr, 25, [], :none},
    {:attribute, {:atom, 25, :define},
     [{:atom, 25, :public_key}, {:atom, 25, true}]}},
   {:tree, :attribute, {:attr, 31, [], :none},
    {:attribute, {:atom, 31, :define},
     [
       {:atom, 31, :"pkcs-1"},
       {:tree, :tuple, {:attr, 31, [], :none},
        [
          {:integer, 31, 1},
          {:integer, 31, 2},
          {:integer, 31, 840},
          {:integer, 31, 113549},
          {:integer, 31, 1},
          {:integer, 31, 1}
        ]}
     ]}},
   {:tree, :attribute, {...}, ...},
   ...
 ]}

Now is the better way? I would think using epp_dodger can be a better solution, the :epp_dodger.parse_file/1 function is documented, this is not the case when it comes to the epp server solution. I also assume the epp_dodger module, because documented, will be more stable in the future than epp. In the end, for this first test release, I think it should do the job.

Conclusion

Even if the interoperability is great between Erlang and Elixir, some part has been ignored or simply discarded because only few people will be interested to use those features. Dealing with Erlang Preprocessor is one of them I guess. In fine, when one knows how Erlang is working, and where to investigate, it becomes an easy problem to fix. Yeah, it can take few hours to understand how the bricks are working, but in the end, it's not lost. Learning how the BEAM is working, and how Erlang is parsed is a fundamental knowledge anyone should have when using this ecosystem.

Long time ago, I worked on an implementation of ZFS (ZettaFileSystem) in Erlang, even if it was working for many things, it was slow and limited to a small amount of the features exposed by ZFS. One of my problem was to deal with all the internal C structure defined in ZFS, evolving sometimes between versions... but also doing mistake by implementing them poorly on the Erlang side as binaries or functions. Having a feature like this one here would have been a game changer, converting a data-structure in C into an Erlang or Elixir function would have be just crazy!

Anyway, as usual, a long list of links if you are interested to know more on this topic:

Erlang Code Loading from the official Erlang documentation, where you will learn how Erlang code is loaded;
Erlang Abstract Format from the official Erlang documentation, where you will learn what the Erlang abstract form is made of;
erlc command documentation and its source code from the official Erlang documentation where you will learn how to use the erlc command;
Core Erlang abstract syntax trees from the official Erlang documentation, where you will learn about the Erlang abstract syntact tree and how to generate ot;
The Erlang compiler and its source code from the official Erlang documentation, where you will learn how the Erlang compiler works;
epp (erlang preprocessor) documentation and its source code from the official Erlang documentation where you will learn and understand the Erlang preprocessor;
erl_anno (compiler annotation) documentation and its source code from the official Erlang documentation, where you will learn about the Erlang compiler annotations;
erl_scan (scanner) documentation and its source code from the official Erlang documentation, where the tokens are produced direct after reading the source code;
erl_parse (parser) documentation and its source code from the official Erlang documentation, where the AST is generated from the tokens previously created by the scanner;
io module documentation and its source code from the official Erlang documentation, where you can find more information about the I/O protocols;
io_lib module documentation and its source code from the official Erlang documentation;
Erlang Abstract Format on Github, where you will have another view of the Erlang Abstract Format;
The Beam Book on Github, where you find a lot of interesting information about the low-level datastructures and how the BEAM is working.
expki source code on Github, where the Erlang record and macro issues were found, while developing a PKI manager;
preprocessing source code on Github, created while writing this article, and to use it as library on other projects.

Have fun and happy hacking!

Cover Image by Evgeniy Smersh on Unsplash