DEV Community: Staex

Cijail: How to protect your CI/CD pipelines from supply chain attacks?

Ivan — Wed, 05 Jun 2024 11:00:00 +0000

Supply chain attacks are especially popular nowadays, and there is a good reason for that. Many build tools such as Cargo, Pip, NPM were not designed to protect from them (NPM example, Cargo-related discussion). At the same time maintainers' tools such as Nix, Guix, RPM and DEB build systems successfully mitigate such attacks. These tools precisely control what files are downloaded over the network before the build starts and prohibit any network access during the build phase itself. In this article we introduce a tool called Cijail that allows you to adopt similar rules for developers' build systems such as Cargo, Pip, NPM. This tool is based on Linux Seccomp, can be run inside CI/CD pipelines, and does not require superuser privileges. It protects from data exfiltration over DNS via deep packet inspection effectively limiting the damage supply chain attacks can cause. The tool is open source and written in Rust.

Why protect from supply chain attacks?
What is a supply chain attack?
How the data is exfiltrated over DNS?
How we can protect ourselves from supply chain attacks?
- Example: Cargo + Github (Cijail itself)
- Example: NPM + Gitlab (static web site)
- Caveat: cargo-deny via HTTPS proxy
- Caveat: NPM via HTTPS proxy
Conclusion

Why protect from supply chain attacks?

Supply chain attacks are become popular with introduction of developers' tools that manage project's dependencies. In contrast to maintainers' tools they do not block network access during build phase, and hackers use this seemingly minor breach to exfiltrate secrets by bundling malicious scripts with the dependency and executing these scripts during build phase. It takes only one popular dependency to be compromised to run these scripts on a multitude of developers' computers and CI/CD pipelines and steal private keys. This is unlikely event but the damage it may cause is catastrophic: private keys might give access to a cryptowallet (on a developer's machine), to a server via SSH, to a static website via cloud upload endpoint etc. From our perspective protecting from them by default is like using the seat belt: no one expects a car crash when one uses a seat belt, but expects the belt to save one's life in an unlikely catastrophic situation.

What is a supply chain attack?

Supply chain attack starts with hacker getting access to a repository of a popular software package. The hacker can use social engineering, zero-day vulnerabilities in operating systems or breaches in repository management system itself. Usually two-factor authentication can protect from the attack on this phase.

If the hacker was able to get access to the repository, he or she proceeds with making a malicious commit or (most likely) making a new release archive that contains malicious code. Usually signed commits and signed releases/packages/archives protect from the attack on this phase.

Then the attacker waits until dependent software packages download new release of the breached dependency and execute the malicious code in their CI/CD pipelines or on the developers' computers. In order to exfiltrate the secrets the hacker would obscure the traffic as DNS for example and setup a DNS server to collect the secrets.

How the data is exfiltrated over DNS?

Data exfiltration over DNS works as follows. A malicious actors sets up a DNS server for his/her domain. Then it encodes secrets as subdomains of this domain and eventually the DNS lookup request reaches the hacker's DNS server via other perfectly secure and legit publicly available DNS servers. This exfiltration uses DNS as a side channel. This is one of many side channels that hackers might use (the other popular one being ICMP protocol).

Conveniently DNS traffic is not blocked anywhere because other software uses DNS. One way to protect from this attack is to either allow to resolve only certain domains via deep packet inspection or block Internet access altogether. Maintainers' tools use the latter while Cijail adopts the former approach because developers' tools were not designed to block the traffic during build phase.

How we can protect ourselves from supply chain attacks?

Cijail protects from supply chain attacks via whitelisting domain names, IP addresses and ports as well as URLS that a script is allowed to access. This is implemented using Seccomp and MITM HTTPS proxy server. Cijail launches the supplied command in a child process with Seccomp jail and SECCOMP_RET_USER_NOTIF flag. Simultaneously the control process is launched that receives notifications from the jailed process and decides if the resource can be accessed via SECCOMP_IOCTL_NOTIF_SEND flag. Finally, a MITM HTTPS proxy is launched as the third process. This process decrypts all HTTPS requests to check that the corresponding URL is allowed. For MITM HTTPS proxy to work the CA SSL certificate is automatically installed in the operating system as trusted.

# no traffic is allowed
🌊 cijail dig staex.io @1.1.1.1
[Sun Apr 04 17:28:22 2024] cijail: deny connect 1.1.1.1:53

# DNS request (connection to DNS server is allowed whereas name resolution is not)
🌊 env CIJAIL_ENDPOINTS='1.1.1.1:53' cijail dig staex.io @1.1.1.1
[Sun Apr 04 17:28:22 2024] cijail: allow connect 1.1.1.1:53
[Sun Apr 04 17:28:22 2024] cijail: deny sendmmsg staex.io

# DNS request and name resolution is allowed
🌊 env CIJAIL_ENDPOINTS='1.1.1.1:53 staex.io' cijail dig staex.io @1.1.1.1
[Sun Apr 04 17:28:22 2024] cijail: allow connect 1.1.1.1:53
[Sun Apr 04 17:28:22 2024] cijail: allow sendmmsg staex.io
... dig output ...

Example: Cargo + Github (Cijail itself)

We tried to use Cijail for building itself. In order to use Cijail in your Github Actions you need to add the following line to your Dockerfile.

COPY --from=ghcr.io/staex-io/cijail:0.6.8 / /usr/local

Then you have to prepend cijail to every command in every step because Github Actions do not respect Docker's ENTRYPOINT. Then all you need to do is to add CIJAIL_ENDPOINTS environment variable with the list of allowed URLS and other endpoints. The resulting workflow specification for Cijail looks like the following.

variables:
  CIJAIL_ENDPOINTS: |
    https://github.com/lyz-code/yamlfix/                          # git
    https://pypi.org/simple/                                      # pip
    https://files.pythonhosted.org/packages/                      # pip
    https://static.crates.io/crates/                              # cargo
    https://index.crates.io/                                      # cargo
    https://uploads.github.com/repos/staex-io/cijail/releases/    # github
    https://api.github.com/repos/staex-io/cijail/releases         # github
steps:
  - name: Lint
    run: cijail ./ci/build.sh

Example: NPM + Gitlab (static web site)

For Gitlab the approach is similar. This time you might consider adding ENTRYPOINT ["/usr/local/bin/cijail"] to your Dockerfile to not prepend cijail to every command in your pipeline. The resulting workflow specification for a static web site looks like the following.

CIJAIL_ENDPOINTS: |
  https://registry.npmjs.org/                  # npm
  https://github.com/lyz-code/yamlfix/         # git
  https://pypi.org/simple/                     # pip
  https://files.pythonhosted.org/packages/     # pip
  9.9.9.9:53                                   # rsync
  staex.io:22                                  # rsync

Caveat: cargo-deny via HTTPS proxy

One particular problem that we encountered is the fact that some programs bundle trusted root CA certificates in their binaries. This is the case for cargo-deny. This tool uses webpki-roots crate that bundles root CA certificates as byte arrays directly in the cargo-deny binary. It is impossible to add Cijail's root certificate to such a program. The current workaround is to run cargo-deny without Cijail.

# our MITM proxy failed to trick cargo-deny :-(
🌊 cijail cargo deny check
[ERROR] error trying to connect: invalid peer certificate: UnknownIssuer

# a workaround
🌊 cijail cargo deny check --disable-fetch || true    # a warm-up (download dependencies)
🌊 cargo deny check                                   # run without cijail 😮

Caveat: NPM via HTTPS proxy

Another problem comes from the fact that NPM usage behind HTTPS proxy is not as reliable as without it. In some cases it creates thousands of connections to download a few dependencies. The workaround that we found is to specify maxsocket=1 in NPM's configuration.

# 1000+ connections for 340 dependencies?
🌊 cijail npm install
[Fri May 24 07:02:13 2024] cijail: allow connect 127.0.0.1:39317
[Fri May 24 07:02:13 2024] cijail: allow connect 127.0.0.1:39317
[Fri May 24 07:02:13 2024] cijail: allow connect 127.0.0.1:39317
... the message repeats 1000+ times
npm ERR! code ECONNREFUSED

# a workaround
🌊 npm config set maxsockets 1

Conclusion

To summarize, most CI/CD pipelines are vulnerable to data exfiltration via DNS because developers' tools like Cargo, NPM and PIP do not block network access during build phase in contrast to maintainers' tools like Nix, Guix, RPM and DEB build systems that do.

The best way to protect from any data exfiltration is to split building the package into download and build phase. During download phase the dependencies are downloaded but no scripts are executed and no packages are built. During build phase the scripts are executed and the packages are built, but the network access is disabled. This simple technique will protect from any type of data exfiltration without the need for deep packet inspection.

The major problem with implementing such a split in developers' tools is the fact that it might break some packages. Another problem is that blocking network access in a Docker container might require additional privileges that are not present by default. Below is the example of how to do this manually for NPM and Cargo.

# cargo example
🌊 cargo download    # only download dependencies
🌊 unshare -rn cargo build    # build packages and run scripts without network access (will not work in a Docker container)

# npm example
🌊 npm clean-install --ignore-scripts    # only download dependencies
🌊 unshare -rn npm rebuild    # build packages and run scripts without network access (will not work in a Docker container)

References

Discuss on Reddit
Slides from Rust & Tell: It is not June yet.
Cijail git repo.

How to build and test your OpenWRT packages with Docker

Ivan — Tue, 30 Jan 2024 12:00:00 +0000

Building and testing software packages for OpenWRT is challenging because this Linux distribution often runs on the devices with exotic architecture and uses centralized configuration (UCI) with which you often need to integrate your software. In this article we will use Docker and QEMU to test package installation on MIPS architecture and discuss what scripts and other files to include in the package to better integrate your software with UCI and OpenWRT itself.

Why OpenWRT?
How to build OpenWRT package?.
- Post-install scripts.
- Post-delete scripts.
- Pre-delete scripts.
- Init scripts.
- First-boot scripts.
- Persist files across system upgrades.
- Package contents.
How to test OpenWRT package with Docker.
- Testing packages for host architecture.
- Testing packages for non-host architecture.
Conclusion.

Why OpenWRT?

OpenWRT is a popular Linux distribution for network routers that brings the power of Linux kernel to resource-constrained devices. Companies use it as a base for their own routers' firmware, regular people use it to replace vendor-provided firmware which is often closed-source and lack many features compared to open-source OpenWRT.

Apart from OpenWRT there is FreeBSD-based pfSense. This distribution supports only x86 (64 bit) architecture whereas OpenWRT supports x86, ARM and MIPS architectures. FreeBSD and Linux are completely different kernels, and you would need a whole different setup for building FreeBSD packages.

How to build OpenWRT package?

OpenWRT uses opkg package manager that can install ipk/opk packages from online repositories and local file system, and to build your own package you need to use opkg-utils. Ipk/opk package format is similar to deb but uses tar instead of ar to package files. The simplest way of building a package is to use opkg-build command.

# Flag `-c` replaces `ar` with `tar`. This is mandatory for OpenWRT.
🌊 opkg-build -c input-dir output-dir

The input-dir contains the files that you want to install plus CONTROL directory with metadata (for deb format this directory is called DEBIAN and you can use this name with opkg-build as well). We don't know other major differences between deb and ipk/opk. For this reason we ended up converting to ipk from deb that was generated by fpm — a tool that we use to produce packages for other Linux distributions. However, the contents of pre/post install/update/remove scripts is different for OpenWRT and there are other special files that you might want to include in the package.

The package includes scripts that are run prior to or after the installation, update and removal of the package. Often package maintainers include them to start/stop services and update firewall rules.

Post-install scripts

Setting up firewall rules in packages scripts is not typical of Linux distributions other than OpenWRT. This is due to the fact that OpenWRT uses Unified Configuration Interface (UCI) — a centralized way of managing system configuration. Through UCI you can setup firewall rules that are inactive by default and can later be enabled via OpenWRT's web interface or via command-line interface.

The following commands set up firewall rule to accept TCP and UDP traffic on port 1234. The rule is disabled by default.

# post-install script
uci -q batch >/dev/null <<'EOF'
add firewall rule
set firewall.@rule[-1].dest_port='1234'
set firewall.@rule[-1].src='*'
set firewall.@rule[-1].name='Allow-MyApp-any'
set firewall.@rule[-1].proto='udp tcp'
set firewall.@rule[-1].target='ACCEPT'
set firewall.@rule[-1].enabled='0'
commit firewall
EOF

Usually you run each line as a separate uci command (e.g. uci add firewall rule), but uci batch is generally faster for a large number of lines. It is up to you to enable or disable the rule by default: for public-facing services (e.g. VPNs) it is generally safe to open the port by default, for everything else (e.g. DNS resolver like Stubby) I would rather close the port by default.

Later the rule can be enabled with the following commands.

# NNN is the actual index of the rule
uci set firewall.@rule[NNN].enabled=1
uci commit firewall
fw3 reload

Post-delete scripts

Deleting the rules is more involved as you need to find the rule index and delete all the matching lines. To distinguish between the package update and package deleting you need to check the first argument of the script.

# post-delete script

delete_rule() {
    name=
    config_get name "$1" name
    if test "$name" = "Allow-MyApp-any"; then
        uci -q delete firewall."$1" || true
    fi
}

case "$1" in
0 | remove)
    . /lib/functions.sh
    config_load firewall
    config_foreach delete_rule rule
    uci -q delete firewall.mcc || true
    ;;
esac

Pre-delete scripts

This is the right place to stop your services before deleting the package. Again to distinguish between the package update and package deleting you need to check the first argument of the script.

#!/bin/sh
case "$1" in
0 | remove)
    /etc/init.d/myapp stop || true
    ;;
esac

Init scripts

OpenWRT uses procd as the init system — pid 1 process that launches all other processes in the system on boot. Procd is similar to SysV init, systemd, openrc etc., however, the syntax for the scripts is different. Here is an example of init script for a typical application that does not daemonizes itself and writes logs on standard output and standard error streams.

#!/bin/sh /etc/rc.common
USE_PROCD=1
START=98 # start order
STOP=99 # stop order
start_service() {
    procd_open_instance
    procd_set_param command /usr/bin/myapp # run the command without daemonizing
    procd_set_param respawn 0 7 0 # respawn after 7 seconds delay
    procd_set_param stdout 1 # redirect stdout to syslog
    procd_set_param stderr 1 # redirect stderr to syslog
    procd_close_instance
}

Procd has a lot of features including process jails, capabilities, etc. that are documented on OpenWRT web site.

First-boot scripts

OpenWRT firmware can come with packages pre-installed, and in this case the right place to generate firewall rules would be a uci-defaults script. Such scripts are placed in /etc/uci-defaults directory and are executed on the first system boot. After successful execution they are deleted by the system. These scripts do not have any special arguments, and generally you repeat post-install script contents there.

Beware that distributing your package as part of OpenWRT firmware image means that your clients would not be able to reclaim free disk space by deleting the package: it will be deleted only in the overlay file system, but not in the underlying real file system.

Persist files across system upgrades

Usually OpenWRT firmware is updated separately from the packages using sysupgrade command. By default all the configuration files (that you specified in CONTROL/conffiles file) are retained during the upgrade, however, your application might generate other files that you want to persist during the upgrade. To do so simply add /lib/upgrade/keep.d/myapp file to your package that lists all directories and files that need to be persisted during the upgrade.

The following file lists /etc/myapp directory as the one that should be persisted during the system upgrade.

/etc/myapp

Package contents

The final package contents would look like the following.

.
├── conffiles # files that are persisted during system upgrade and that are not overwritten by package update
├── control # package metadata
├── etc
│   ├── init.d
│   │   └── myapp # init script
│   ├── myapp
│   │   └── myapp.conf # app configuration
│   └── uci-defaults
│       └── myapp # the script that runs on the first boot
├── lib
│   └── upgrade
│       └── keep.d
│           └── myapp # the list of files that are persisted during system upgrade
├── postinst # post-install script
├── postrm # post-delete script
├── prerm # pre-delete script
└── usr
    └── bin
        └── myapp # application executable binary

Most of the contents can be generated with fpm tool using deb format as the output. Then the following script will convert deb package to ipk package.

#!/bin/sh

cleanup() {
    rm -rf "$workdir"
}

set -ex
trap cleanup EXIT
workdir="$(mktemp -d)"
mkdir -p "$workdir"/deb "$workdir"/ipk/CONTROL
cd "$workdir"/deb
# unpack deb package
ar x "$1"
tar -C "$workdir"/ipk -xf data.tar.gz
tar -C "$workdir"/ipk/CONTROL -xf control.tar.gz
# remove generated files
rm "$workdir"/ipk/CONTROL/md5sums
# patch architecture for OpenWRT
sed -i -e 's/Architecture: amd64/Architecture: x86_64/g' "$workdir"/ipk/CONTROL/control
# write ipk to /tmp
opkg-build -c "$workdir"/ipk /tmp

How to test OpenWRT package with Docker

Similar to any other major Linux distribution OpenWRT maintains rootfs Docker images of root file systems, but it has a separate image for each target architecture. The documentation is on Github. There is a separate tag for each architecture plus OpenWRT version combination. There are also sdk and imagebuilder images that are useful for building the package and the firmware image with the package pre-installed, but we will not discuss them here.

Testing packages for host architecture

Rootfs images are useful to test your package installation/removal without investing into a router running OpenWRT (although, you should definitely invest in such a device to feel comfortable with doing system upgrades, working with web UI etc.). To install and remove the package use the following commands.

🌊 docker run --rm -it -v /tmp:/tmp openwrt/rootfs


BusyBox v1.36.1 (2024-01-22 12:01:31 UTC) built-in shell (ash)

🌊 opkg update
...
🌊 opkg install /tmp/myapp.x86_64.ipk
Installing myapp (1.0.0) to root...
Configuring myapp.
🌊 opkg remove myapp
Removing package myapp from root...

Docker by default pulled and ran openwrt/rootfs image that matches the host machine's architecture (x86_64). Then we updated the package index and installed our application from the /tmp directory mounted from the host.

Testing packages for non-host architecture

Testing for an architecture other than host's is more involved and requires running a virtual machine. Luckily for us QEMU and Linux's binfmt_misc makes it easy to do so (and without Docker even noticing!).

QEMU is a tool that runs virtual machines on the host, and the most useful feature it has for us is the ability to transparently execute binary files compiled for an architecture other than the host's architecture. The following commands show how to do that on Ubuntu 22.04.3 LTS.

# install QEMUE binaries that were statically compiled for each supported architecture
🌊 apt-get install qemu-user-static
# list all available QEMU binaries
🌊 ls /usr/bin/qemu-*static
...
/usr/bin/qemu-aarch64-static
...
/usr/bin/qemu-mips-static
...
# execute a file compiled for mips
🌊 qemu-mips-static /tmp/file-compiled-for-mips

To execute the binary compiled for architecture X you just add qemu-X-static before the command and that's it.

Linux's Support for miscellaneous Binary Formats (binfmt_misc) allows us to execute binaries without specifying any QEMU command. Upon execution the kernel detects the actual executable format of the file and executed it via the matching QEMU binary. The matching QEMU binaries and the «magic» bytes that distinguish a particular format from all others are specified in /proc/sys/fs/binfmt_misc directory. On Ubuntu 22.04.3 LTS this directory is populated automatically after qemu-user-static package is installed.

🌊 ls /proc/sys/fs/binfmt_misc
...
qemu-mips
...
qemu-aarch64
...
🌊 cat /proc/sys/fs/binfmt_misc/qemu-mips
enabled
interpreter /usr/libexec/qemu-binfmt/mips-binfmt-P
flags: POCF
offset 0
magic 7f454c46010201000000000000000000000200080000000000000000000000000000000000000000
mask ffffffffffffff00fefffffffffffffffffeffff0000000000000000000000000000000000000020

Now to run a foreign binary you don't need anything special: it runs the same way as a native binary on the system.

# you need matching entry in /proc/sys/fs/binfmt_misc for this to work
🌊 /tmp/file-compiled-for-mips

Since Docker is not a virtualization platform, but a process isolation tool, binfmt-misc and QEMU allows you to transparently run Docker images that were built for other architectures. Again, you don't need anything special to do that. The following command runs OpenWRT rootfs image for MIPS architecture.

🌊 docker run --rm -it -v /tmp:/tmp openwrt/rootfs:mips_24kc
Unable to find image 'openwrt/rootfs:mips_24kc' locally
mips_24kc: Pulling from openwrt/rootfs
2dd8ebde9a90: Pull complete
Digest: sha256:58d0bf8e15559e0a331e23915ed0221d678c8b2a569c58c0fa25a4f991e4beca
Status: Downloaded newer image for openwrt/rootfs:mips_24kc
WARNING: The requested image platform (linux/mips_24kc) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested


BusyBox v1.36.1 (2024-01-26 09:19:40 UTC) built-in shell (ash)

🌊 grep ARCH /etc/os-release
OPENWRT_ARCH="mips_24kc"
🌊 opkg update
...
🌊 opkg install /tmp/myapp.mips_24kc.ipk
Installing myapp (1.0.0) to root...
Configuring myapp.
🌊 opkg remove myapp
Removing package myapp from root...

Docker warns that the image's architecture does not match the host's, but still successfully runs the image. This is where the power of QEMU and binfmt-misc shows itself.

Conclusion

Building OpenWRT packages is similar to any other Linux distributions, but has unique requirements if you want to integrate your package with the rest of the system.

Generate firewall rules in post-install and uci-defaults scripts and delete them in post-delete script.
List files (in addition to configuration files) that need to be preserved during system upgrades in /lib/upgrade/keep.d/myapp file.
Write procd-compatible init script.
Use opkg-build to generate ipk/opk package.
Optionally, use fpm tool to generate deb package and then convert it to ipk/opk.

Testing OpenWRT packages is also very similar to other Linux distributions, provided that you installed QEMU and used binfmt-misc kernel feature to transparently run foreign binaries on your host. OpenWRT maintains root file system images for each combination of version and architecture all of which you can directly run on your host and in your CI/CD pipeline.

We at Staex help our clients make IoT devices first-class citizens in their private networks, protect from common attacks, reduce mobile data usage, and enable audacious use cases that were not possible before.

Subscribe to our newsletter to get more content like this.

How to reduce Docker image size for IoT devices

Ivan — Tue, 16 Jan 2024 12:00:00 +0000

IoT devices sometimes have too little resources to pull and run heavyweight Docker images. In this article we show how to reduce the size by 36-91% using patchelf and strace tools without recompiling containerized applications. We also show how to build minimal images for your own Rust, Go, C/C++ applications.

Why reduce Docker image size?
Patchelf.
- Motivating example: Stubby.
- Results.
- Limitations.
Strace.
- Motivating example: Home Assistant.
- Results.
- Limitations.
You own images.
- Rust static binaries.
- Go static binaries.
- C/C++ static binaries.
Conclusion.

Why reduce Docker image size?

Docker image size and the number of layers affects how much memory and disk space a device needs to pull and unpack the image. Devices like Raspberry Pi Zero have too little resources to pull and unpack e.g. Home Assistant image, however, have more than enough resources to actually run it. Reducing the size improves Docker performance in such use cases. Also including only the files that are actually used by the application helps reduce the attack surface. This benefit goes beyond just IoT devices and is applicable to servers as well.

It is easy to reduce the image size of the containerized applications that you developed yourself: just compile the static binary and include only this file in the final image. However, there are several approaches for third-party applications that do not require recompilation.

Patchelf

If the application in question is compiled into ELF binary (usually this is the case for C, C++, Fortran, Rust, Go etc.), you can use patchelf tool to find all the libraries that the application uses and copy them into the final image.

ELF — executable and linkable format — specifies program interpreter path (e.g. /lib64/ld-linux-x86-64.so.2 on x86_64 platform) and runtime search path abbreviated as rpath (e.g. /lib64) among a multitude of other metadata.

Program interpreter is used to dynamically load ELF file and all its dependencies (libraries) into the memory and execute it. On Linux you can do this manually: /lib64/ld-linux-x86-64.so.2 /bin/sh is a «shortcut» for just /bin/sh.

Runtime search path (rpath) is used by the program interpreter to find the dependencies. On most Linux distributions (Guix and Nix are the only exceptions that I know) this path is empty and the interpreter searches for dependencies in hard-coded paths (e.g. /lib64).
We use patchelf tool to modify the interpreter and rpath and readelf to inspect ELF file. Another useful tool is ldd. It shows both the interpreter and all the dependencies.

# Debian
🌊 readelf --headers /bin/sh | grep -A2 INTERP
  INTERP         0x0000000000000318 0x0000000000000318 0x0000000000000318
                 0x000000000000001c 0x000000000000001c  R      0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
🌊 readelf --dynamic /bin/sh | grep RUNPATH
🌊 patchelf --set-interpreter /lib/ld-linux-x86-64.so.2 --set-rpath /lib /path/to/some/elf/binary
🌊 ldd /bin/sh
        linux-vdso.so.1 (0x00007ffce0f91000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fedf9b66000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fedf9d6d000)

As you can see from the output rpath is empty on Debian and /bin/sh depends only on libc. The output of the same commands on Guix is quite different. This is just an example, we will not dive into why Guix uses non-empty rpath.

# Guix
🌊 readelf --headers /bin/sh | grep -A2 INTERP
  INTERP         0x0000000000000318 0x0000000000400318 0x0000000000400318
                 0x0000000000000050 0x0000000000000050  R      0x1
      [Requesting program interpreter: /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/ld-linux-x86-64.so.2]
🌊 readelf --dynamic /bin/sh | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/gnu/store/lxfc2a05ysi7vlaq0m3w5wsfsy0drdlw-readline-8.1.2/lib:/gnu/store/bcc053jvsbspdjr17gnnd9dg85b3a0gy-ncurses-6.2.20210619/lib:/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib:/gnu/store/930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib/lib:/gnu/store/930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/11.3.0/../../..]
🌊 ldd /bin/sh
        linux-vdso.so.1 (0x00007ffe777f6000)
        libreadline.so.8 => /gnu/store/lxfc2a05ysi7vlaq0m3w5wsfsy0drdlw-readline-8.1.2/lib/libreadline.so.8 (0x00007efca9070000)
        libhistory.so.8 => /gnu/store/lxfc2a05ysi7vlaq0m3w5wsfsy0drdlw-readline-8.1.2/lib/libhistory.so.8 (0x00007efca9063000)
        libncursesw.so.6 => /gnu/store/bcc053jvsbspdjr17gnnd9dg85b3a0gy-ncurses-6.2.20210619/lib/libncursesw.so.6 (0x00007efca8ff1000)
        libgcc_s.so.1 => /gnu/store/930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib/lib/libgcc_s.so.1 (0x00007efca8fd7000)
        libc.so.6 => /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6 (0x00007efca8dd9000)
        /gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/ld-linux-x86-64.so.2 (0x00007efca90c9000)

Motivating example: Stubby

Let's use patchelf to reduce the Docker image size for Stubby — a name resolver that supports DNS-over-TLS. We will use Debian as the base image but the process is specific neither to this Linux distribution nor to this application.

First we write a Dockerfile that installs Stubby and the required packages from Debian repositories on the first stage, and on the second stage copies only the necessary files into the final image created from scratch.

FROM debian:latest AS builder

# install stubby and patchelf
RUN apt-get update && apt-get install -y stubby ca-certificates patchelf

# copy and run patchelf script
COPY patchelf.sh /tmp/patchelf.sh
RUN /tmp/patchelf.sh

# create the final image from scratch (i.e. without the base image)
FROM scratch

# copy only the /out directory that contains the files that are actually used by stubby
COPY --from=builder /out /

EXPOSE 53/udp
EXPOSE 53/tcp

CMD ["/bin/stubby"]

Second we write patchelf script that determines which files need to be copied. The script copies all the dependencies, the program interpreter, the binary itself and the configuration file, and finally the OpenSSL library's configuration files and the list of trusted SSL certificates.

#!/bin/sh
set -ex
mkdir -p /out/lib /out/bin /out/etc /out/var/cache/stubby /out/var/run /out/usr/lib
# copy the libraries that stubby uses
ldd /usr/bin/stubby |
    sed -rne 's/.*=> (.*) \(.*\)$/\1/p' |
    while read -r path; do
        cp "$path" /out/lib
    done
# copy the interpreter
cp /lib64/ld-linux-x86-64.so.2 /out/lib
# copy stubby and its configuration file
cp /usr/bin/stubby /out/bin/stubby
# make stubby listen on all addresses to access it from outside the container
sed -i 's/127\.0\.0\.1/0.0.0.0/g' /etc/stubby/stubby.yml
cp -r /etc/stubby /out/etc/stubby
# copy openssl library configuration and certificates
cp -r /etc/ssl /out/etc/ssl
cp -r /usr/lib/ssl /out/usr/lib/ssl
find /out/etc/ssl/certs -not -type d -not -name ca-certificates.crt -delete
rm -rf /out/usr/lib/ssl/misc
# patch stubby binary to use the copied interpreter and libraries
patchelf --set-interpreter /lib/ld-linux-x86-64.so.2 --set-rpath /lib /out/bin/stubby
ldd /out/bin/stubby
find /out
# check that stubby works
chroot /out /bin/stubby -V

Now we build the image and check that it runs correctly.

🌊 docker build --tag stubby:debian-patchelf .
🌊 docker inspect docker inspect -f "{{ .Size }}" stubby:debian-patchelf
13120030
🌊 docker run --init --rm --publish 53:53/udp stubby:debian-patchelf stubby -l
# in the other terminal window
🌊 dig @127.0.0.1 +short google.com
142.251.220.206

Results

We compare the resulting image size using docker inspect command. The competing images are Debian-based and Alpine-based images created without patchelf script.

Image	Size, MiB	Size, %
stubby:debian-patchelf	12.5	9% of stubby:debian
stubby:debian	143.4
stubby:alpine-patchelf	9.0	64% of stubby:alpine
stubby:alpine	14.1

The results speak for themselves. We reduced the size of Debian-based Stubby image by 91% and Alpine-based Stubby image by 36% by including only the files that Stubby actually uses. Impressive.

Limitations

Patchelf fully automates copying dependencies and program interpreter, however, any other files need to be copied manually. Also, if your program is not compiled to ELF binary (e.g. NodeJS, Python) then you're out of luck. This is where strace can help.

Strace

This tool intercepts system calls the binary makes and prints their arguments. Strace uses the same kernel API as debuggers and may considerably slow down the traced program. Luckily we will use this tool only on the Docker image build stage.

Motivating example: Home Assistant

This is the image that I failed to install on Raspberry Pi Zero while using the official Docker image. When you pull this image Docker downloads the many layers in parallel and then fails to extract them due to a lack of disk space. I had to temporarily attach an external USB drive and move /var/lib/docker directory there, then pull the image and move the directory back to the Raspberry Pi to successfully pull and run this image.

Now we create a new Docker image for Home Assistant that has only one layer and consumes only a fraction of disk space of the original image.

First we create Dockerfile with the official image as the base.

FROM ghcr.io/home-assistant/home-assistant:stable AS builder

RUN apk update && apk add strace

COPY strace.sh /tmp/strace.sh
RUN /tmp/strace.sh

FROM scratch

COPY --from=builder /out /

# default Home Assistant port
EXPOSE 8123/tcp

# default Home Assistant command
CMD ["/usr/local/bin/python3", "-m", "homeassistant", "--config", "/config"]

Then we write strace script that finds all the files accessed by Home Assistant and copies them into the final image.

#!/bin/sh
set -ex
mkdir -p /out/lib /out/usr/local/bin /out/usr/bin /out/usr/local/lib
# copy ffmpeg and its dependencies
ldd /usr/bin/ffmpeg |
    sed -rne 's/.*=> (.*) \(.*\)$/\1/p' |
    while read -r path; do
        cp "$path" /out/lib
    done
cp /lib/ld-musl-x86_64.so.1 /out/lib
cp /usr/local/bin/python3 /out/usr/local/bin/python3
cp /usr/bin/ffmpeg /out/usr/bin/ffmpeg
# copy frontend files manually
mkdir -p /out/usr/local/lib/python3.11/site-packages
cp -r /usr/local/lib/python3.11/site-packages/hass_frontend /out/usr/local/lib/python3.11/site-packages/hass_frontend
# copy all the files that home assistant actually opens
strace -f -e open,stat,lstat timeout 30s python3 -m homeassistant --config /config 2>&1 |
    sed -rne 's/.*(open|stat)\(.*"([^"]+)".*/\2/p' |
    grep -vE '^/(dev|proc|sys|tmp)' |
    sort -u |
    while read -r path; do
        if ! test -e "$path"; then
            continue
        fi
        if test -d "$path"; then
            # create directories
            mkdir -p /out/"$path"
        else
            # copy files
            mkdir -p /out/"$(dirname "$path")"
            cp -n "$path" /out/"$path" 2>/dev/null || true
        fi
    done
# recreate config directory
rm -rf /out/config
mkdir /out/config

Now we build the image and check that it runs correctly.

🌊 docker build --tag home-assistant:strace .
🌊 docker run --rm --publish=8123:8123/tcp home-assistant:strace \
    python3 -m homeassistant --config /config
# now open https://127.0.0.1:8123/ in the browser

Results

We compare the resulting image size using docker inspect command to the original image.

Image	Size, MiB	Size, %
home-assistant:strace	590	31
ghcr.io/home-assistant/home-assistant:stable	1886	100

We were able to reduce the image size by 69%. Most importantly now Raspberry Pi Zero can pull and run the new image without hitting the disk space limit.

Limitations

The obvious limitation of strace is that frontend files are not copied automatically because they are read only if an HTTP request is made. Of course we can do some HTTP requests with curl but usually all frontend files are needed. It is much easier to just copy them all to the final image.

Your own images

Dealing with your own Docker images is much easier than with third-party ones. You either compile your program into a static binary or compile to a dynamically linked binary and use patchelf tool to copy the dependencies and the interpreter. In this section we show how to compile static binaries for Rust, Go and C/C++. The general approach is to use musl library and the accompanying musl-gcc tool to build your project, but some languages make it simpler.

Rust static binaries

In order to use musl library in your project you need to install musl-based toolchain and then compile for the corresponding target.

🌊 rustup toolchain add stable --target x86_64-unknown-linux-musl 
# here we remove debugging information and optimize for size
🌊 env RUSTFLAGS='-Copt-level=z -Cstrip=symbols' \
    cargo build --release --target x86_64-unknown-linux-musl

Now we build Docker image that includes only the resulting binary file.

FROM scratch
COPY target/x86_64-unknown-linux-musl/release/app /bin/app
CMD ["/bin/app"]

As you can see the resulting image contains only the binary but not the dependencies. This means that Docker is merely a distribution format for static binaries.

Go static binaries

Go does not use musl, but contains its own static implementation of libc. This makes compiling static binaries even simpler.

🌊 env CGO_ENABLED=0 go build -ldflags '-s -w' -o app ./cmd/app

Now we build Docker image similar to Rust static binary.

FROM scratch
COPY app /bin/app
CMD ["/bin/app"]

C/C++ static binaries

The idea here is to replace C/C++ compiler with musl-gcc and enable static compilation in GCC via -static linker flag. All the dependencies have to be recompiled this way as well. This makes the approach especially problematic for dependencies that prefer dynamic linking for whatever reason (e.g. use features of GNU libc that does not support static linking; dynamically load other libraries; use sophisticated build instructions that make it impossible for a mere human to modify to enable static linking). That's why the general approach for C/C++ binaries is to use patchelf.

The following snippet shows how to compile static binary for a cmake-based project.

🌊 cat > CMakeLists.txt << 'EOF'
project (HelloWorld)
add_executable (app app.c)
EOF

🌊 cat > app.c << 'EOF'
#include <stdio.h>
int main() {
    printf("Hello world\n");
    return 0;
}
EOF

🌊 mkdir build-musl
🌊 cd build-musl
🌊 env CC=musl-gcc LDFLAGS='-static' cmake -DCMAKE_BUILD_TYPE=Release ..
🌊 make
[ 50%] Building C object CMakeFiles/app.dir/app.c.o
[100%] Linking C executable app
[100%] Built target app
🌊 ldd ./app
        not a dynamic executable

Conclusion

There are multiple approaches that help reduce Docker image size:

including only the required dependencies with patchelf,
including only the required files with strace,
compiling your own program into a static binary that includes all the dependencies.

On average you can reduce the image size by approximately 50% (at least in our experiments). The smaller size improves Docker performance on resource constrained devices such as Raspberry Pi Zero. However, the major benefit for any platform is the fact that the attack surface is much smaller when your image doesn't contain tools like wget, curl and shell interpreters.

VPN kill switch: how to do it on Linux

Ivan — Tue, 02 Jan 2024 12:00:00 +0000

Kill switch is a mechanism that prohibits any outgoing traffic unless a VPN is active. In this article we discuss how to implement such a mechanism using Linux policy-based routing for a wide range of VPNs.

Linux IP packet routing tour.
- CIDR notation, gateways and broadcast addresses.
- How to read the routing table.
- Is there only one routing table?.
- Policy-based routing.
VPN kill switch with policy-based routing.
- Custom routing table.
- Custom routing rules.
- Any alternatives?
- Multiple VPNs.
Conclusion.

Linux IP packet routing tour

Before diving into how to implement a kill switch we need to get familiar with how Linux IP packet routing works in general. The best way to do that is to run ip route command that shows a routing table. On my computer this command outputs the following.

🌊 ip route
default via 10.65.0.1 dev wlan0 proto dhcp src 10.65.0.11 metric 305
10.33.0.0/16 dev vpn1 scope link
10.65.0.0/20 dev wlan0 proto dhcp scope link metric 305

CIDR notation, gateways and broadcast addresses

Each row in the table is a rule that matches a particular packet destination in CIDR notation. For example, 10.33.0.0/16 matches any packet with a destination 10.33.X.X where X is arbitrary number from 0 to 255. Usually the first address 10.33.0.1 is the gateway — default packet destination if no rules match the current packet destination — and the last address 10.33.255.255 is broadcast address — if you send a packet to this address all nodes in the network will receive it. The reality is more complicated though: you can use any address as the gateway and set any address as broadcast address. Plus broadcast packets are usually only sent to the nodes that are connected to the same network switch and these packets are usually blocked by the network router to prevent accidental flooding.

How to read the routing table

Now we can go back to the table to study the rules. In the output default is another way of spelling 0.0.0.0/0, and this rule matches any packet destination.

The first rule says «forward the packet to a gateway with address 10.65.0.1 if none of other rules match».
The second rule says «forward the packet to network device vpn1 if the destination matches 10.33.0.0/16». In this case the device driver or a program that is attached to this device will handle the packet.
The third rule says «forward the packet to network device wlan0 if the destination matches 10.65.0.0/16». There is no gateway in this rule because the packet's destination is in the same network as the gateway, and Linux sends the packet directly to the destination.

Is there only one routing table?

As you may have guessed there are many routing tables in the system. There is default, local and main table. Each table has the id and the name. The mapping between them is stored in /etc/iproute2/rt_tables file. Counterintuitively the default table is main. To see the contents of other tables use the following commands.

🌊 ip route show table main
...
🌊 ip route show table local
...
🌊 ip route show table default
...

On my computer default table does not exist. The local table lists local and broadcast addresses associated with network devices.

🌊 ip route show table local
local 10.33.0.41 dev vpn1 proto kernel scope host src 10.33.0.41
local 10.65.0.11 dev wlan0 proto kernel scope host src 10.65.0.11
broadcast 10.65.15.255 dev wlan0 proto kernel scope link src 10.65.0.11
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1

Policy-based routing

Linux has another set of rules that define how the table is selected. These rules also have priorities, so if a packet matches multiple rules then the rule with lowest priority is selected. To see all the rules use ip rule command.

🌊 ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

On my computer each rule matches any packet (from all clause in the output), and local table has lower priority than main. This is what we will leverage to create a VPN kill switch.

VPN kill switch with policy-based routing

VPN kill switch requires routing all outgoing traffic through a VPN except for the local traffic and VPN internal traffic. This means that if a VPN uses port 1234, then the traffic from this port should go through the default gateway or directly to the node in the local network. To implement that we will create a separate table and a rule that uses this table for all non-VPN and non-local packets.

Custom routing table

First edit /etc/iproute2/rt_tables file and add the following line that defines our new table.

83 vpn1

Now create rules in the new table. The table itself is created automatically.

# remove existing rules if any
🌊 ip route flush table vpn1
# add default route via gateway node from VPN network
🌊 ip route add default dev vpn1 via 10.83.0.1 table vpn1 metric 100
# add blackhole route (this is the actual kill switch)
🌊 ip route add blackhole default table vpn1 metric 200
# check that the rule has been added
🌊 ip route show table vpn1
default via 10.83.0.1 dev vpn1 metric 100
blackhole default metric 200

To summarize, we added new routing table called vpn1, we added default route via gateway node from VPN and we added so-called blackhole route. Default route is preferred over blackhole route because of the lower metric. The blackhole route is used only when the default is not present in the table. Device vpn1 is automatically deleted whenever VPN is stopped and the corresponding rules are deleted as well, however, blackhole route stays intact.

Custom routing rules

Now we will link vpn1 to the main routing table.

# route all packets except the ones from source port 1234 using the rules from table vpn1
🌊 ip rule add not sport 1234 table vpn1
# prefer specific rules in table "main" over the rules in other tables
🌊 ip rule add table main suppress_prefixlength 0
# check the rules
🌊 ip rule
0:      from all lookup local
32764:  from all lookup main suppress_prefixlength 0
32765:  not from all sport 9376 lookup vpn1
32766:  from all lookup main
32767:  from all lookup default

The first rule is self-explanatory, you can check out all possible alternatives to not sport in the documentation. According to the documentation suppress_prefixlength N option means «reject routing decisions that have a prefix length of N or less». Prefix length equals zero means default route, hence this rule means «reject routing decisions that match default route in table main». So, suppress_prefixlength 0 is a fancy way of saying «ignore the default route from the main routing table». Since the next table in the list is vpn1, then all the traffic except for local networks will go through the vpn1 network interface.

Any alternatives?

We tested policy-based VPN kill switch with Wireguard (do not forget to specify the port in the configuration) and Staex. Both VPNs use only one port for their internal traffic. It should be possible to match the traffic of a centralized VPN by source/destination in CIDR notation (from and to options of ip rule command). In general the exact packets can be marked using iptables and then matched by the same mark in the routing rules (see mark iptables module). This article discusses various approaches within the context of Wireguard.

Multiple VPNs

The nature of a kill switch does not play well with multiple VPNs. Probably the only way to exclude multiple ports from the default route is to use firewall marks. We have not evaluated this approach yet.

Conclusion

We conceived kill switch to be a simple VPN feature, however, we underestimated the complexity of Linux networking. Linux has multiple layers of IP packet routing rules, built-in firewall and network namespaces. VPNs do not make this task simpler either: they might use several ports for the internal communication or you might want to run multiple VPNs on a single node.

Kill switch will be available in the upcoming Staex release. Subscribe to our newsletter to be notified about the new releases.

Terrapin attack on SSH: what do you need to know

Ivan — Wed, 27 Dec 2023 17:00:00 +0000

Terrapin is a recent prefix truncation attack on SSH that exploits deficiencies in the protocol specification, namely not resetting sequence number and not authenticating certain parts of handshake transcript. The attack requires man-in-the-middle, i.e. a rogue network node that intercepts the traffic. SSH protocol is used to remotely manage servers and IoT devices and is widely spread. In this article we explain how to secure your servers and devices from this attack.

How to protect your SSH servers
Mitigation for OpenSSH
Mitigation for Dropbear on Debian
Mitigation for Dropbear on OpenWRT
Defense in depth

How to protect your SSH servers

According to the paper the attack is possible only if you use vulnerable ciphers and encryption modes: ChaCha20-Poly1305, CTR-EtM, CBC-EtM. Note that the ciphers and the encryption modes themselves are not vulnerable, but their input (sequence number) can be manipulated by the attacker.

To mitigate the attack you either update OpenSSH and Dropbear to the their latest versions (OpenSSH 9.6 and Dropbear 2022.83) or disable the affected ciphers and encryption modes. We will show how to do the latter.

Mitigation for OpenSSH

We will show how to disable the affected ciphers on the example of Debian. We will use Docker to make this reproducible. Then we will verify our configuration using vulnerability scanner provided by the authors of the paper.

# docker run -it --rm debian:latest
# then run the following commands
apt-get update
apt-get install -y wget ssh
mkdir /run/sshd

# check if ssh is vulnerable
/usr/sbin/sshd
wget https://github.com/RUB-NDS/Terrapin-Scanner/releases/download/v1.1.0/Terrapin_Scanner_Linux_amd64
chmod +x Terrapin_Scanner_Linux_amd64
./Terrapin_Scanner_Linux_amd64 -connect 127.0.0.1:22
pkill sshd

# print effective ssh configuration and filter out affected ciphers
# '*-cbc' ciphers should be disabled by default
sshd -T | sed -nr 's/(chacha20-poly1305@openssh\.com,|,chacha20-poly1305@openssh\.com)//gip' >> /etc/ssh/sshd_config

# re-check ssh
/usr/sbin/sshd
./Terrapin_Scanner_Linux_amd64 -connect 127.0.0.1:22
pkill sshd

Mitigation for Dropbear on Debian

To disable the affected ciphers in Dropbear we need to recompile it. Here we show the steps again using a Docker container for the latest Debian and Terrapin scanner.

# docker run -it --rm debian:latest
# then run the following commands
apt-get update
apt-get install -y git wget build-essential zlib1g-dev
git clone https://github.com/mkj/dropbear
cd dropbear
# here we disable ChaCha20Poly1305 and enable GCM instead
# CBC is disabled by default
env CFLAGS='-DDROPBEAR_CHACHA20POLY1305=0 -DDROPBEAR_ENABLE_GCM_MODE=1' ./configure
make
make install

# check if dropbear is vulnerable
dropbear -R
wget https://github.com/RUB-NDS/Terrapin-Scanner/releases/download/v1.1.0/Terrapin_Scanner_Linux_amd64
chmod +x Terrapin_Scanner_Linux_amd64
./Terrapin_Scanner_Linux_amd64 -connect 127.0.0.1:22
pkill dropbear

Mitigation for Dropbear on OpenWRT

For this Linux distribution you need cross compiler to recompile Dropbear. The easiest way to get it is to use official Docker image.

# docker run -it --rm -v $PWD/bin/:/builder/bin openwrt/sdk:latest
# Substitute 'latest' with your router's architecture.
# All tags are listed on DockerHub: https://hub.docker.com/r/openwrt/sdk/tags
# Then run the following commands.
./scripts/feeds update -a
make defconfig
sed -i 's/.*DROPBEAR_CHACHA20POLY1305.*/# CONFIG_DROPBEAR_CHACHA20POLY1305 is not set/' .config
./scripts/feeds install dropbear
make package/dropbear/compile
make package/index
# the IPK package is in 'bin' directory
# now we will check that dropbear is not vulnerable
# (you don't need to repeat this convoluted command)
env LD_LIBRARY_PATH=./staging_dir/toolchain-x86_64_gcc-12.3.0_musl/lib ./build_dir/target-x86_64_musl/toolchain/.pkgdir/libc/lib/ld-musl-x86_64.so.1 ./staging_dir/target-x86_64_musl/root-x86/usr/sbin/dropbear -R
wget https://github.com/RUB-NDS/Terrapin-Scanner/releases/download/v1.1.0/Terrapin_Scanner_Linux_amd64
chmod +x Terrapin_Scanner_Linux_amd64
./Terrapin_Scanner_Linux_amd64 -connect 127.0.0.1:22

Upon exit the package will appear in bin directory. Then you copy it to your router and update Dropbear.

Defense in depth

Disabling perfectly fine ciphers might be an overkill. Terrapin attack does not break SSH session integrity, it only allows an attacker to disable keystroke timing obfuscation features of OpenSSH. Disabling ChaCha20Poly1305 in Dropbear (which is often used in embedded devices) would result in increased CPU usage: most embedded CPUs do not have hardware acceleration for AES ciphers which will be used instead.

The alternative is to establish SSH connection over a VPN. This would add an additional security layer with its own authenticated encryption and trust establishment method. VPNs are not a silver bullet against cyber attacks but a tool to implement defense-in-depth in your system. Knowing that you have another security layer when some protocol is breached gives you peace of mind and much needed time to implement proper mitigations.

Securing IoT devices from DNS-based attacks

Ivan — Tue, 19 Dec 2023 12:00:00 +0000

DNS protocol is one of the attack vectors on your corporate network and IoT devices in particular. Most operating systems access DNS servers using legacy unencrypted protocol by default despite the fact that there are modern secure enhancements for this protocol: DNSSEC, DNS-over-HTTPS, DNS-over-TLS. In this article we discuss these enhancements and explain how to configure them in your network.

What is DNS?
Cache poisoning attack
DNS traffic encryption
Hard-coded DNS servers
DNS data exfiltration
Wrap-up

What is DNS?

Domain Name System is a protocol that resolves human-readable names into machine-readable IP addresses and vice versa — the address book of the Internet.

DNS records are stored on the servers that are organized in a tree. Leaves of the tree are managed by cloud providers, internet providers, and other businesses, whereas roots are managed by an international organization IANA.

DNS was conceived more than thirty years ago (RFC 1035). As such it doesn't have any encryption and trustworthiness built-in. It is easy to spoof DNS name, to spoof DNS server address and to analyze DNS traffic going through your router.

Cache poisoning attack

This type of attack involves falsifying DNS records by a corrupt DNS server. These records are cached by downstream servers and eventually reach the clients — your IoT devices and servers. These attacks are sometimes called DNS cache poisoning.

To mitigate this attack IETF introduced a set of security extensions to DNS collectively called DNSSEC (RFC 9364). These extensions need to be enabled both on the client and on the server side.

Mitigating on the server

If DNSSEC is enabled for a particular domain, then there is a RRSIG record for this domain. DNS client needs to verify the signature from this record, if the verification fails then return name resolution failure. Dig command verifies DNSSEC record by default, however, it does not fail when there is no such record. Use the following command to resolve DNS name and print the RRSIG record (+dnssec flag).

dig staex.io +dnssec # should resolve to the ip address
dig badsign-a.test.dnssec-tools.org +dnssec # should fail

Usually DNS providers have an option to enable DNSSEC in their portal. Configuring DNSSEC for your own DNS server is out of scope of this article.

Mitigating on the client

To use DNSSEC system-wide you need a capable DNS resolver. One such resolver is Stubby. We will use Stubby in this article because it also supports DNS encryption that we will discuss later.

To enable DNSSEC in Stubby edit the configuration file (/etc/stubby/stubby.yml by default), and add/replace the following options.

# /etc/stubby/stubby.yml
dnssec: GETDNS_EXTENSION_TRUE

To check that Stubby works, we repeat dig commands but specify 127.0.0.1 as the server.

dig staex.io +dnssec @127.0.0.1 # should resolve to the ip address
dig badsign-a.test.dnssec-tools.org +dnssec @127.0.0.1 # should fail

In all our tests the second address failed to resolve with or without dnssec flag in the configuration file. We assume that it was filtered on the server side prior to reaching the client. Better safe than sorry.

DNS traffic encryption

DNS traffic is still unencrypted even if DNSSEC is enabled, and the solution is to use either DNS-over-TLS or DNS-over-HTTPS. DoT and DoH encapsulate the packet in TLS and HTTPS frame respectively. Both protocols use state-of-the-art encryption (signed private keys obtained via public key exchange) and offer protection from replay attacks and man-in-the-middle attacks during initial key exchange. Both protocols are extensively used in the Internet and are regularly updated with new ciphers and other cryptographic algorithms.

Mitigating on the server

Both DoH and DoT are supported by popular DNS resolvers like BIND. Configuring these protocols on the server side is out of scope of this article.

Mitigating on the client

To enable DoT on the client side we again use Stubby. This name resolver uses DoT by default. If you want to change the upstream DNS server, then add the following lines to the configuration file (/etc/stubby/stubby.yml by default).

# /etc/stubby/stubby.yml
upstream_recursive_servers:
  - address_data: 9.9.9.11
    tls_auth_name: "dns11.quad9.net"
  - address_data: 149.112.112.11
    tls_auth_name: "dns11.quad9.net"

Here we configured Quad9 servers. Other alternatives are NextDNS, Cloudflare, Google. Some of them filter malicious sites which might be useful for browsers but is not so important for IoT devices.

Hard-coded DNS servers

So far we configured encryption and signature verification for DNS traffic, and in ideal world this should be enough to protect your devices. However, some devices use a hard-coded list of DNS servers in their firmware, and do not allow us to change the firmware.

Mitigation

To solve this problem we will redirect DNS traffic from those devices to Stubby. The irony is that we use the fact that legacy DNS packets can be easily rewritten and sent to another server without the client noticing.

Below is a set of iptables rules that will redirect any incoming traffic on port 53 to the local Stubby server. These rules are for the router.

# network interface that receives DNS packets
interface=br-lan
# IP address assigned to the network interface
interface_ip_address=192.168.1.1
# local stubby port
stubby_port=53
for protocol in udp tcp; do
    iptables -t nat -A PREROUTING \
        -i $interface ! -s $interface_ip_address \
        -p $protocol --dport 53 \
        -j DNAT --to $interface_ip_address:$stubby_port
done

These rules will not work for DoT and DoH, and will not work if the device in question uses non-standard DNS port. The first problem can be solve by deploying HTTPS proxy (which might not be desirable) and the second can be solved with eBPF rules. Diving into these solutions worth an article of its own.

DNS data exfiltration

This data exfiltration technique is rather new, but has already been exploited by various malicious programs. To use DNS queries to exfiltrate stolen data an attacker sets up a DNS resolver for his/her domain. Then on the victim's device the stolen data is encoded in subdomains of this domain: an attacker resolves subdomains and DNS servers forward the name resolution requests to the attacker's DNS server.

Mitigation

One problem of this attack is that the traffic goes through perfectly secure public DNS servers and there is no 100% reliable way to detect it on the server side. However, we can easily block the data exfiltration on the client's side using a list of allowed DNS names and allowed IP addresses to which these names resolve. If a program tries to resolve a name that is not in the list, then the name resolution fails.

Below is a set of rules for iptables that restrict the names that are allowed to be resolved to IP addresses.

# wan interface
interface=eth0
iptables -A OUTPUT -o $interface \
    -p udp --port 53 \
    -m string --hex-string "|05|staex|02|io" -algo bm \
    -j ACCEPT
# 05 --- length of "staex" in hexadecimal format
# 02 --- length of "io" in hexadecimal format

Below is the script that generates a set of IP addresses to and from which the traffic is allowed. It is a good idea to run this script periodically to get DNS records' updates. Beware that DNS servers themselves need to be in the set as well.

cleanup() {
    rm -f "$ip_addresses"
}

set -e
trap cleanup EXIT

# resolve hostnames
allowed_hostnames="dns9.quad9.net one.one.one.one"
ip_addresses="$(mktemp)"
for hostname in $allowed_hostnames; do
    dig +short "$hostname"
done >"$ip_addresses"

# create ipset
name=allowlist
ipset -exist create $name hash:ip
ipset flush $name
while read -r ip_address; do
    ipset add $name $ip_address
done <"$ip_addresses"

Below are iptables rules that restrict the IP addresses that are allowed for inbound and outbound traffic. For these rules to work you need to disallow all inbound and outbound traffic by default. You might want to use iptables-apply command to not to lock yourself out of the server.

name=allowlist
iptables -A INPUT ! -m set --match-set $name src -j DROP
iptables -A OUTPUT ! -m set --match-set $name dst -j DROP

These rules might not work for DNS queries that use pointers to encode names. DoT and DoH queries are also problematic. The first problem can be solved using eBPF and the second by using HTTPS proxy. Again, diving into these solutions worth an article of its own.

It is worth noting that data exfiltration is the second-order threat, since an attacker needs to infiltrate the node first to steal the data. Nevertheless, it is important to have multiple layers of security — an approach encouraged by zero-trust security model.

Wrap-up

DNS protocol is not secure by default, and you need to protect your devices from common attacks yourself. Implementing full protection is a huge endeavor that is best done by professionals. However, implementing «good enough» protection is manageable and can be done using the techniques from this article.

Signature verification via DNSSEC and traffic encryption via DoT or DoH is a must for any serious IoT project, whereas hard-coded DNS servers and DNS data exfiltration are mostly second-order threats.

Future versions of Staex will include all the mitigations mentioned in the article thus making your IoT network secure by default. Subscribe to our newsletter to be the first to know about the new features.

Originally posted on staex.io.

DEV Community: Staex

Cijail: How to protect your CI/CD pipelines from supply chain attacks?

Table of contents

Why protect from supply chain attacks?

What is a supply chain attack?

How the data is exfiltrated over DNS?

How we can protect ourselves from supply chain attacks?

Example: Cargo + Github (Cijail itself)

Example: NPM + Gitlab (static web site)

Caveat: cargo-deny via HTTPS proxy

Caveat: NPM via HTTPS proxy

Conclusion

References

How to build and test your OpenWRT packages with Docker

Table of contents

Why OpenWRT?

How to build OpenWRT package?

Post-install scripts

Post-delete scripts

Pre-delete scripts

Init scripts

First-boot scripts

Persist files across system upgrades

Package contents

How to test OpenWRT package with Docker

Testing packages for host architecture

Testing packages for non-host architecture

Conclusion

How to reduce Docker image size for IoT devices

Table of contents

Why reduce Docker image size?

Patchelf

Motivating example: Stubby

Results

Limitations

Strace

Motivating example: Home Assistant

Results

Limitations

Your own images

Rust static binaries

Go static binaries

C/C++ static binaries

Conclusion

VPN kill switch: how to do it on Linux

Table of contents

Linux IP packet routing tour

CIDR notation, gateways and broadcast addresses

How to read the routing table

Is there only one routing table?

Policy-based routing

VPN kill switch with policy-based routing

Custom routing table

Custom routing rules

Any alternatives?

Multiple VPNs

Conclusion

Terrapin attack on SSH: what do you need to know

Table of contents

How to protect your SSH servers

Mitigation for OpenSSH

Mitigation for Dropbear on Debian

Mitigation for Dropbear on OpenWRT

Defense in depth

Securing IoT devices from DNS-based attacks

Table of contents

What is DNS?

Cache poisoning attack

Mitigating on the server

Mitigating on the client

DNS traffic encryption

Mitigating on the server

Mitigating on the client

Hard-coded DNS servers

Mitigation

DNS data exfiltration

Mitigation

Wrap-up