Running applications on different operating systems and processor architectures is a common scenario, so it is a common practice to build separate distributions for different platforms. This is not easy to achieve when the platform we use to develop the application is different from the target platform for deployment. For example, developing an application on an x86 architecture and deploying it to a machine on an ARM platform usually requires preparing the ARM platform infrastructure for development and compilation.
Problem
I recently purchased the new Apple M1 Pro MacBook, which is a high-performance laptop. The M1 Pro (aarch64) processor is very fast and very friendly on the battery, but as I mentioned there are also some - not so small - issues associated with working on a new chip architecture.
I’ve tried to create a few multi-platform docker images with buildx and some are just not correctly emulated on the M1.
The parallelism of buildx was also an issue but the emulation gave me the most trouble.
Solution
Use buildx remote node(s) to build the image on a native architecture.
What do you need?
- More than one docker-enabled device in your network or on the internet like:
- Virtual Machine / Server
- Raspberry Pi
- Laptop
- Desktop
- ssh access to these devices
In my case I have a CX11 VM on HETZNER that is **amd64/x86_64**
based and my own M1 Pro that is **arm64/aarch64**
based. Since Hetzner now offers arm64 machines, you can also do it the other way around too.
How I did do this?
Step 1: Installation
Since the newer versions of Docker, Buildx (Buildkit) is a built-in function, to check its availability, use the following command.
❯ docker buildx version
github.com/docker/buildx v0.11.1 b4df08551fb12eb2ed17d3d9c3977ca0a903415a
Step 2: Setup ssh access for your remote node
The remote host where we also want to run Docker needs to be configured with a password-less SSH connection. This post will not explain how to do that as there are many articles explaining how to set up public/private key access to a device.
If you already have the keys you can copy them to your target device like so:
ssh-copy-id USERNAME@TARGET_VM_HOST
To run a command with the user environment available you need to make sure your sshd service allows it.
When running a command through ssh you will have a limited environment and that needs to be adjusted.
To have a correct environment where docker is known you need to set the #PermitUserEnvironment no
property to PermitUserEnvironment yes
in the /etc/ssh/sshd_conf
file on your VM.
When this is adjusted you need to restart the sshd service. On my CX11 VM with the Ubuntu server, I used this command. It might be different for your device:
sudo systemctl restart sshd
Now you have to set the environment for ssh:
- login to your VM through the terminal (ssh USER_HERE@TARGET_VM_HOST)
- cd ~/.ssh create a file called environment with the following value in it
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
the last /usr/local/bin
is where the docker command lives. So now it will be available.
Check if you can run docker with the command below from local machine/laptop. It should work:
docker -H ssh://USERNAME_HERE@TARGET_VM_HOST info
Do not forget to change the command's values to appropriate ones for your situation.
Step 3: Create a buildx local node
Let’s first create the local platform. In my case that is the Apple M1 Pro node. It should build all the arm64/aarch64 targets.
docker buildx create \
--name local_remote_builder \
--node local_remote_builder \
--platform linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/mips64le,linux/mips64,linux/arm/v8,linux/arm/v7,linux/arm/v6 \
--driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
--driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000
I’ve played a bit with the settings but this seems to work best for me. I will update as needed or if I find settings that work even better.
Now we have a local config for a builder called local_remote_builder.
You can check if you have it by running docker buildx ls
command:
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
local_remote_builder * docker-container
local_remote_builder unix:///var/run/docker.sock running linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386
Step 4: Create a target buildx remote node
In my case, this will be the amd64/x86_64 builder node. The platforms corresponding to that architecture should be sent there.
We have already configured the SSH access and the environment and tested that docker can access it.
Now we need to add it to the buildx target
docker buildx create \
--name local_remote_builder \
--append \
--node intelarch \
--platform linux/amd64,linux/386 \
ssh://USERNAME@TARGET_VM_HOST \
--driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
--driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000
Do not forget to change the command's values to appropriate ones for your situation.
This command will append the remote node to the local_remote_builder and call the node intelarch. You can check it again by running docker buildx ls
command again:
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
local_remote_builder * docker-container
local_remote_builder orbstack running v0.11.6 linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386
intelarch ssh://aboozar@167.21.183.24 running v0.11.6 linux/amd64*, linux/386*, linux/amd64/v2, linux/amd64/v3
Step 5: buildx use and bootstrap
In order to start using this builder setup we need to tell buildx to start using it and we need to bootstrap it.
docker buildx use local_remote_builder
docker buildx inspect --bootstrap
output:
#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch
#1 ...
#2 [localremote_builder internal] booting buildkit
#2 pulling image moby/buildkit:buildx-stable-1 1.6s done
#2 creating container buildx_buildkit_local_remote_builder 0.6s done
#2 DONE 2.3s
#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch 3.6s done
#1 DONE 3.6s
Name: localremote_builder
Driver: docker-container
Nodes:
Name: local_remote_builder
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386
Name: intelarch
Endpoint: ssh://aboozar@167.21.183.24
Status: running
Platforms: linux/amd64*, linux/386*, linux/amd64/v2
Step 6: Build a multi-platform image
The command below will build for amd64 and arm64 but will direct the amd64 build to the remote node and the arm64 build will be done on the local machine.
docker buildx build --platform=linux/amd64,linux/arm64/v8 --push -t name/target:tag .
This will result in something like this on the docker hub:
Conclusion
The most obvious issue in this setup is that I can only build for multiple platforms when within my own network.
That is not necessarily true if you used a remote ssh accessible host or IP. I did not do that and that limits me to these builds when I have my remote node available in the network. For now, that is not a real issue for me.
It solved my concurrency problem. I had one build where I had to start the server, in order to configure it, during the docker build. Buildx does this in parallel and that gave me a port conflict as one was a bit faster but not ready when the other also tried to start. That was a problem when I tried to build all on only my local node. When I added the remote node this problem went away as the port was used on a completely different machine.
Overall, I am currently content with this solution.
Top comments (2)
i don't have much knowledge about it . thanks for sharing . fairbet7
Nice article lot's of good info. FYI github actions can build for for these architectures.
linux/amd64,
linux/arm64,
linux/arm/v7
darwin/amd64,
darwin/arm64
windows/amd64
I have mine setup to build the images on push to it's release branch,
have a look at github actions it may save you some time in future projects :)
Happy Coding