DEV Community: Ákos Takács

History of GPUs in the context of containers until 2025

Ákos Takács — Tue, 24 Jun 2025 20:00:37 +0000

Introduction

Years ago, before I started to use Docker, I didn't really care about GPUs. When I was about 12, all I heard, that a good graphics card was required to play some games on a Windows PC. That didn't change much during the years since I wasn't really among people talking about graphics cards and later GPUs for a long time. Now, in the age of containers, Docker and machine learning, everyone is talking about GPUs.

I have done some research to discover the important steps of the evolution of GPUs, but I did it with containers in mind, so I could learn also about important steps of the development of Docker and containers in general in the context of GPUs. That's why I share more GPU-related events until 2017 below, and then speed up, focusing on containers.

It is important to note that many of the events I share, I read in other articles on the internet, but I tried to get multiple sources for confirmation. You will find the links in this post to all of my sources. This one could not have been made without the authors of the sources, so a big thanks to them and please, follow the links if you want to learn more.

The very beginning
Main contributors to the early history of GPUs
GPU not just for graphics
Containers and GPUs
Introducing GPU support in WSL and Docker Desktop
Supporting AI workloads and using AU at Docker, Inc
Conclusion
Sources

The very beginning

» Back to table of contents «

It is not an accident that I didn't hear about using GPUs when I was 12, and not just because I was a kid. The term "GPU" did not exist at the beginning, when we could only talk about graphics cards. So, if we really wanted to go back to the very beginning, we could start with the first attempts at visualization, and talk about the "Manchester Baby", which, according to Britannica, could display images with a cathode-ray tube in 1948. The same article also mentions "Whirlwind computer" as "the first computer to display video". Of course, it was not the video that you could watch today.

According to AceCloud, the history of GPUs started in the 1970s, mentioning even 1968 when "computer graphics were just in their infancy".
In the beginning, graphics cards were indeed for computer graphics but in a very simple way compared to what we have today. Later, people realized that these devices could be used for other purposes when parallel processing is required.

Main contributors to the early history of GPUs

» Back to table of contents «

There are some companies I probably don't have to introduce to anyone, as they are all well-known like ATI, AMD's NVIDIA, and Sony, but I personally never heard about 3DLabs before.

ATI Technologies was founded in 1985
even before NVIDIA was founded in 1993
The term "GPU" was coined by Sony in 1994
A company, called 3DLabs was founded in 1994, and according to Britannica, they had a "3-D add-in card" as the "first modern GPU" from 1995, which I have never heard about before, and I couldn't figure out which card exactly the author referred to, as I couldn't find the quoted term anywhere else on Google. However, 3DLabs is mentioned by other websites as once a "leading company in the field".
Intel's first dedicated GPU was made in 1998.
You can read that GeForce 256 was marketed by NVIDIA as the "world's first GPU" in 1999.
In 2006, ATI Technologies was acquired by AMD

It is interesting how many products were considered basically the beginning of GPUs by different people. I think it just means that the birth of today's GPUs didn't happen suddenly in the morning, but it was a long process with many important steps. Let's continue with more.

GPU not just for graphics

» Back to table of contents «

In 2009, a paper was written "discussing the technology's promise in machine learning applications". The link to the paper in the linked article doesn't work anymore, but I think I found it from two sources, although both are from the Stanford University uploaded by the authors:
- Uploaded by Andrew Y. Ng
- Uploaded by Rajat Raina
So it seems that things started to change and more and more people and companies started to use GPUs for machine learning.
Still, in 2009, a paper was published with the title "NVIDIA's Fermi: The First Complete
GPU Computing Architecture".
In 2010, NVIDIA released the Fermi Architecture as a successor to the Tesla Architecture.
In 2012, AlexNet wins and even "dominates" the ImageNet challenge using CUDA, which was a significant step in Deep Learning.
In 2017, NVIDIA released the Volta architecture that introduced Tensor cores to "speed up the training of neural networks"

Containers and GPUs

» Back to table of contents «

GPUs might be very important tools in parallel processing and machine learning, but containers are also everywhere in development and also in production, so naturally, we have to be able to use GPUs in containers.

The first version of LXC (Linux Containers) was released in 2008
Docker's first version was released in 2013.
If my interpretation of the sources is correct, NVIDIA started to support GPUs in containers in 2016. There was a proof of concept release of NVIDIA Docker.
The latest patch of "nvidia-docker" v1 was released in March 2017. This was the last package that actually contained the source code, and it supported Docker 17.03.
The first commit of libnvidia-container was made in April 2017. Interestingly, to confuse us, the source code was pushed to the "nvidia-container-runtime" repository and the "libnvidia-container" repository as well, and I assume the original source was actually the "nvidia-container-runtime" repository. This assumption is based on the following:
- The title of the readme of v1.0.0 was "libnvidia-container"
- All the v1 tags (v1.0.0 - v1.0.3) in the "nvidia-container-runtime" repository were created when the commits were created. According to GitHub, v1.0.0 in "nvidia-container-runtime" was tagged in September 2018
- Only one day later was the same commit in the "libnvidia-container" repository tagged as v1.0.0
- The rest of the v1 tags in "nvidia-container-runtime" were created even with a bigger delay in "libnvidia-container".
I also found a Hungarian name as the author of the commit of the "gpus" option in the Docker CLI. To me, as a Hungarian, it was probably more interesting than to others.
After v1, "nvidia-docker" eventually became the name of a collection of packages including "nvidia-container-runtime" and the "libnvidia-container".
Initially, the runtime was a patched version of "runc", which is the default container runtime in Docker today. Despite some sources that indicate "nvidia-docker" was a fork of "runc", it was not.
The latest release of NVIDIA Docker came out on August 30 in 2023, and it was superseded by NVIDIA Container Toolkit.

Introducing GPU support in WSL and Docker Desktop

» Back to table of contents «

Although GPU support existed on Linux and in containers before, Docker Desktop runs the Docker daemon and the containers in a virtual machine even on Linux, as I mentioned in a previous blog post. That means the GPU has to be available inside the virtual machine, even when you are using that GPU on the physical host.

On June 17, 2020, Microsoft announced GPU compute support for WSL2, which was also mentioned on the NVIDIA tech blog.

On December 21, 2020, Docker announced the "general preview of Docker Desktop support for GPU with Docker in WSL2". That basically meant that since WSL2 already supported NVIDIA GPUs, and the NVIDIA container runtime existed, it could be automatically configured in Docker Desktop when using WSL2 as a backend.

Supporting AI workloads and using AI at Docker, Inc

» Back to table of contents «

"Docker, Inc" didn't just continue to support workloads using GPUs, which often meant some AI-related processes, but released their "Docker Docs AI" and wrote a blog post about it pn May 22, 2024. That was the first step towards a Docker AI assistant, but it was for helping with Docker-related questions based on the documentation.
On the official Docker Forums, I could announce the Beta "Ask Gordon" on January 2, 2025. It was an AI integrated into Docker Desktop.
Docker Desktop is a tool to support container-based development, which currently also includes working with AI models, so on April 9, 2025, the Docker Model Runner was announced.
Not long after the announcement of the Model Runner, the MCP Catalog and Toolkit were announced as well on May 5, 2025.

Conclusion

» Back to table of contents «

So this is where we are now, and I'm sure I could have mentioned many other details, but my goal was to give you an overview, before I jump to other GPU- or AI-related posts. At least that's the goal.

Until then, you can check the Excel sheet in which I collected some GPU- and container-related events in chronological order:

https://1drv.ms/x/c/9d670019d6697cb6/EbZ8adYZAGcggJ2RjAAAAAABhaNs-vt8Yj4QIeqzhs4Rxg

If you are looking for links, you can move the mouse pointer over the top right corner of the cells where you see a little red arrow (at least this is how it works at the time of writing this post).

In the next section, you can see the links of websites that I also linked in this article above except those that I didn't link as a source, but only a reference to a product or one of my previous posts.

If you find incorrect information, please let me know in the comments. You can also share your opinion or add an important event that you would have mentioned.

Sources

» Back to table of contents «

Error message: Is the Docker daemon running?

Ákos Takács — Sun, 05 Jan 2025 16:06:43 +0000

I shared this post on the official Docker forum as well, in case you prefer reading there. I will try to keep these posts in sync, but leave a comment if you think theye are not.

Introduction

There is a common error message that Docker users get

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Sometimes you get an IP address

Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?

Or sometimes a default domain:

Cannot connect to the Docker daemon at http://docker.example.com. Is the docker daemon running?

Although the solution is often quite simple, people often jump to the wrong conclusion that the Docker client knows exactly whether the daemon is running or not, and they don't understand what else could be the problem, when they see the Docker daemon running. So in this post I will explain what the error message is, what it isn't, and how can a single error message mean so many things. Of course, I will also share the solutions I know about, but what I want you to learn here is how to understand the error message in the context of your environment and the way to find a solution for your case.

The meaning of the error message
Quick guide for the impatient
Problems and solutions in details
- Find out which Docker variant you have
- Docker runtime contexts
- Running remote Docker daemons
- Check if the Docker daemon is actually running
  - Docker CE status on Linux running under Systemd
  - Docker as Snap package status
  - Docker CE status in Docker Desktop
- Check if the Docker unix domain socket exists
- Check if you have permission to the socket
- Make sure you are using the right socket
I got "is the Docker daemon running", but I followed the official installation guide

The meaning of the error message

» Back to table of contents «

Notice that it is a question at the end and not a statement. So it is either the dockerd process is not running or the client could not tell whether it is running or not. The Docker client will not check the running processes on the host machine. The client could run on your workstation while you are trying to connect to a Docker daemon running on a remote machine or in a virtual machine, maybe even created by Docker Desktop. So the client will attempt to connect to the daemon, and if it fails to do so, the client will tell you that one possible reason of failing is that the daemon is not even running.

The second detail you need to notice is the unix socket in the error message. That is a file on the filesystem of the machine on which the client is running, but it is almost always the same as the one on which the daemon is running, or at least the same physical machine while the daemon is running in a VM (like Docker Desktop). In case there is only one machine, this is the file used by both, the client and the daemon, so the client can communicate with the Docker daemon. This file is not a regular file, just a way for processes to communicate with each other through the kernel memory. Fortunately, you don't have to know anything about that, but this is probably the reason why you can't mount this socket file (or any other unix domain socket) into a virtual machine.

Oh... you think you could do that with Docker Desktop? No, you couldn't, but there is a Docker socket inside the VM as well. So when you mount the docker socket into a container, you mount the one from the virtual machine.

Even though this file is not a regular file, you can still set permissions on the file, which means you can allow or deny access to the file for different users. If you don't have access to the unix domain socket which is configured for the client, it will not be able to use it for communication. It will not be able to connect to the Docker daemon, even if the daemon is running and listening on this socket for requests. Note that unix:// is just a protocol similarly to http://. The difference is that after an HTTP protocol, the next part is the domain name or IP address, while the next part after the unix domain socket protocol is the file path, which starts with a slash character on Linux. That is why you see 3 slashes. One is part of the filepath, the other two are part of the protocol reference.

Quick guide for the impatient

» Back to table of contents «

Here is a summary of what you should consider:

There are multiple Docker variants, and knowing which one you are using can help you find out whether it is running or not, because you know where to look for the process. On the other hand, if you know, how you can list processes on the operating system, and look for the Docker daemon in the list, that can help you to figure out which Docker variant you have. Be careful, because you could have more than one.
Are you using a remote daemon running on a remote server, or a local daemon
Does the socket file exist?
Does the Docker client has access to the file?
If you have access to the file, are you sure that this socket is the one that is used by the Docker daemon?

Problems and solutions in details

Find out which Docker variant you have

» Back to table of contents «

First you need to know which Docker variant you have. People often install Docker CE and Docker Desktop when they only want to have the Desktop. Then they actually try to connect to the wrong daemon. I wrote about the different kind of Docker installations in "You run containers, not dockers - Discussing Docker variants, components and versioning". If you want to learn more about it in practice, you can also read "Which Docker variant am I using and where is the daemon running?"

Docker runtime contexts

» Back to table of contents «

We have to talk about docker contexts. If I'm not specific enough, it could be confused with the docker build contexts which is a completely different topic. What I'm talking about now is what we could call the "Docker runtime context". The context definition that tells the client how to connect to the daemon on the server where the containers will run. The following command can show you the current contexts:

docker context ls

Sometimes the easiest solution is switching to another context or create one first if none of the existing contexts are correct. If you don't want to create a context, you can also set an environment variable temporarily to change the host. It is mentioned in the linked documentation, but there is another one with an example: Protect the Docker daemon socket / Secure by default. I will also use this later in this post.

If you want to get the actual endpoint from the current context, you can also run

docker context inspect --format '{{ .Endpoints.docker.Host }}'

Which can show you something like unix:///var/run/docker.sock.

Running remote Docker daemons

» Back to table of contents «

Even if you are using Docker CE, the Docker client can connect to a remote server. If you know that the daemon is running on a remote server, because you have a VPS to which you want to connect using SSH, you could get a slightly different error from what I shared in the intro.

Cannot connect to the Docker daemon at http://docker.example.com. Is the docker daemon running?

It is still the same kind of error, but there is an HTTP endpoint instead of the unix socket. This is special, since even when you don't have a unix domain socket on Linux, you have a TCP socket, not HTTP. When you try to actually use URL as an endpoint, you will get "invalid bind address format".

DOCKER_HOST="http://docker.example.com" docker ps

output:

Failed to initialize: unable to resolve docker endpoint: invalid bind address format: http://docker.example.com

This is still currently the default at the time of writing this post when the unix socket cannot be found through the SSH connection. You can reproduce it with the following command, assuming the hostname to your VPS is myvps.

DOCKER_HOST="ssh://myvps/var/run/docke.sock" docker ps

In the above command I intentionally set a wrong socket path using an SSH connection, but I can try TCP as well.

DOCKER_HOST="tcp://127.0.0.1" docker ps

Then we get the IP address in the error message, not the default domain

Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?

And if you try a domain name, you see the domain name in the error message, except when the domain name cannot be resolved to an IP address, in which case you have a DNS issue with a completely different error message.

DOCKER_HOST="tcp://myvips" docker ps

Output

error during connect: Get "http://myvips:2375/v1.47/containers/json": dial tcp: lookup myvips: no such host

This is out of the scope of this post, but I still have to mention that the environment variable is not the only way to get this error. You could have a Docker runtime context as well.

Check if the Docker daemon is actually running

» Back to table of contents «

First of all, the "Find out which Docker variant you have" section could already help you, because knowing which Docker variant you had also required being able to check the running processes, but there is more.

Docker CE status on Linux running under Systemd

» Back to table of contents «

If you followed the official documentation to install Docker CE, the Docker daemon is running under systemd. In that case the following command should tell you if it is running or not.

sudo systemctl status docker

It actually works for non-official packages if Systemd was used to run the daemon. Since most of the Linux distributions today use Systemd, it usually works. On older Windows Subsystem for Linux versions, Systemd was not available, so you had to run a different command:

sudo service docker status

This actually works even if you have systemd, since the service script can recognize it and use systemctl commands behind the scenes.

The output would be something like this:

● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-01-01 17:32:24 CET; 3 days ago

This is only the beginning the output, but the important part is "Active: active (running)". Depending on your terminal configuration, the circle next to the service name could be green when the service is running.

If you have Rootless Docker, the daemon is running as your non-root user, and the systemctl command is different:

systemctl --user status docker

notice that there is no sudo in the command and we use the --user flag.

Docker as a Snap package status

» Back to table of contents «

If you installed Docker as a Snap package, the following command will list the running services

snap services

Output:

Service                          Startup  Current   Notes
docker.dockerd                   enabled  active    -
docker.nvidia-container-toolkit  enabled  inactive  -

The only service I have in this example is the Docker daemon and the Nvidia Container Toolkit which is inactive. The "dockerd.dockerd" service is active, so it is running. If you have many Snap services, you can of course get the status of one service directly:

snap services docker.dockerd

Output

Service         Startup  Current  Notes
docker.dockerd  enabled  active

Docker CE status in Docker Desktop

» Back to table of contents «

If you have Docker Desktop, the Docker daemon is running in a virtual machine. If the host operating system is Linux, there is a systemctl command that shows the status of the docker-desktop service.

systemctl --user status docker-desktop

This is again the status of Docker Desktop, and not the daemon which is running in the virtual machine of Docker Desktop. If you want to know the status of the engine, open the GUI of Docker Desktop and look for the "Engine running" message currently in the bottom left corner of the window.

Check if the Docker unix domain socket exists

» Back to table of contents «

It is usually at /var/run/docker.sock, but it can be changed, so just look for what the error message says, or run the command I already mentioned before:

docker context inspect --format '{{ .Endpoints.docker.Host }}'

If it is a unix domain socket file, you should be able to check it by running the following command in which I will assume the path is the default:

file /var/run/docker.sock

The output:

/var/run/docker.sock: socket

It also tells you if the file is an actual socket. If not, you have to fix the socket.

Check if you have permission to the socket

» Back to table of contents «

Maybe it is an existing socket, but you don't have permission to access it. The following command can reveal that:

ls -l /var/run/docker.sock

Output:

srw-rw---- 1 root docker 0 Jan  3 12:49 /var/run/docker.sock

It shows that the socket is owned by root, and it is assigned to the "docker" group. rw-rw---- means that only the owner (root) or a user in the group (docker) can read and write the file. so you are either using

sudo docker ps

to list containers, or you add your user to the Docker group:

sudo usermod -aG docker USERNAME

There is actually a third option that I sort of "invented", since I have never seen this anywhere else, but you can read about it in "Allow non-root users to use the docker commands".

Make sure you are using the right socket

» Back to table of contents «

As I already pointed out, some users install Docker CE when they actually want to use Docker Desktop, but when the documentation says that adding the official package repository to the system is required, and it links to the configuration guide, they continue with the rest of the steps, which are not just not required, but also not recommended. The only package you need from the official repository is the Docker Client which is called "docker-ce-cli", and which will install "docker-buildx-plugin" and "docker-compose-plugin" as well.

If you install "docker-ce", you will have a second daemon outside of Docker Desktop, and when Docker Desktop is running, it changes the runtime context to "desktop-linux", so even if you checked the context previously, it will be changed, and you connect to another socket. This is one more time when docker context commands could be useful to see which context you are in, but know that when you use the docker commands with sudo, you will run the docker client as root, which has no configured access to your Docker Desktop socket in your home directory, and you connect to the daemon of Docker CE on the host, not Docker Desktop. So even if the socket you check is accessible by your user, it is possible that it is not the one you wanted to use, and you can get an error message when trying to connect to the wrong daemon.

If you have Rootless Docker, it is a similar situation, since it has a different socket too, and it is configured only for your non-root user, so using sudo would result in using another daemon.

Don't forget to check the endpoint in the context using

docker context ls

docker context inspect --format '{{ .Endpoints.docker.Host }}'

and find out if this is the right context. It is also possible that something added an environment variable to your shell. Run

echo "$DOCKER_HOST"

to find out if this is the case, figure out where it was set (possibly in your .bashrc file in your home) and remove it or change it. This could happen also when you are running Docker in a CI/CD pipeline, although in that case it is not likely to be wrong. You should still check it to make sure.

If you have multiple Docker installation on the same machine, or you changed the daemon configuration or used a non-official installer for the daemon part, it is possible that the socket in the error message is not the one that the daemon is using, so you need to reconfigure the client either by setting a docker runtime context or the environment variable.

I got "is the docker daemon running", but I followed the official installation guide

» Back to table of contents «

We had multiple cases on the Docker forum when the user stated they followed the official documentation for example on Ubuntu, and they still got the error message.

The problem is that people often ignore the "Next steps" section, which is this:

Continue to Post-installation steps for Linux

This is where the group membership is set. Even if you didn't ignore it and configured the user properly, it can be that you have one of the previously mentioned issues due to multiple Docker daemons on your system.

Conclusion

» Back to table of contents «

By this time you could learn that the error message is not a statement, but a question or an assumption. There is a general form of this error message, but the endpoint can be multiple thing. I hope I could help you to understand the meaning of all, so you can make your Docker work again.

You could learn some of the risks of running multiple Docker daemons on the same machine, but there are other risks not included here, since those are not related to the discussed error messages.

As always I tried to include all the related problems and solutions that I know of, but obviously I can't always think of everything. If you are sure, that you have none of the issues mentioned in this post, leave a comment here or on the official Docker forums to figure out what happened to your system. Please, share what you have actually tried, and not just that you tried everything mentioned in this or any other post, since it could be easy to miss something!

Which Docker variant am I using and where is the daemon running?

Ákos Takács — Thu, 26 Dec 2024 16:40:39 +0000

Introduction

People often install Docker CE and Docker Desktop when they only want to have the Desktop. Then they actually try to connect to the wrong daemon. I wrote about the different kind of Docker installations in "You run containers, not dockers - Discussing Docker variants, components and versioning"

So sometimes you want to know which Docker variant you are running. Maybe you don't remember, or accidentally enabled an option while installing the host operating system, or someone else installed Docker and you had to take over the project. When you ask a question on the official Docker Community Forums, we also need to know which one you are using, since the cause of an issue and the solution could be completely different.

You can find out which Docker you installed, but first you need to know what operating system you are using and what package managers it supports. Then you need to know how you can list the installed packages, using the supported package managers and listing processes can help too.

In this post I try to summarize the different kind of Docker installations and how you can tell which variant you have.
You can also have one that I don't write about in this post, but I hope you can use the methods to discover what you or anyone else installed before.

Docker Engine on Linux
- The number of dockerd processes running directly on Linux
- Docker as a Snap package
- Docker installed using the default package manager of the Linux distribution
  - Docker on Debian-based Linux distributions
  - Docker on Red Hat-based Linux distributions
  - Find non-snap dockerd processes on Linux
Docker Desktop
- Docker Desktop on Linux
- Docker Desktop on macOS
- Docker Desktop on Windows
Docker Contexts
Remote Docker daemon
Conclusion

Docker Engine on Linux

The number of dockerd processes running directly on Linux

» Back to table of contents «

When using the Docker Engine on Linux directly, based on the Moby project, you can run

pidof dockerd

That should show a single number, the process id of the Docker daemon. If you get none, then the Daemon is either not running, or you most likely run the daemon on a remote machine or in a virtual machine. It is less likely, but it is also possible that you found another non-official way to install Docker, and the daemon executable has a different name. Then you need to find the maintainer of that Docker daemon and ask for their help.

If you get multiple process IDs, that means you have multiple Docker daemons running, which is most likely by accident. You should run only one Docker daemon, unless you are really experienced, and you know how to make sure that these daemons are using different sockets, data directories and iptables rules (or it is enabled only for one daemon).

Docker as a Snap package

» Back to table of contents «

When Docker is installed as a Snap package on Linux, the following command can be used:

snap list docker

If you get an error message, you either have no snap package manager at all, or Docker is not installed as a snap package. Otherwise, you would get something like the following:

Name    Version  Rev   Tracking       Publisher   Notes
docker  27.2.0   2964  latest/stable  canonical✓  -

The exact output can be different depending on the version you have. Then you can find the process in the process list using the following command:

ps aux | grep dockerd | grep snap

If the Docker daemon installed as a Snap package is also running, you will get an output like below:

root 916 0.0 1.6 2037476 67312 ? Ssl 15:51 0:02 dockerd --group docker --exec-root=/run/snap.docker --data-root=/var/snap/docker/common/var-lib-docker --pidfile=/run/snap.docker/docker.pid --config-file=/var/snap/docker/2964/config/daemon.json

Docker installed using the default package manager of the Linux distribution

Docker on Debian-based Linux distributions

» Back to table of contents «

On Debian-based Linux distributions, you can use dpkg to find out if Docker was installed as an APT package.

dpkg -l 'docker*' | grep '^ii'

The output would be something like this:

ii  docker-buildx-plugin      0.19.2-1~ubuntu.24.04~noble   arm64        Docker Buildx cli plugin.
ii  docker-ce                 5:27.4.0-1~ubuntu.24.04~noble arm64        Docker: the open-source application container engine
ii  docker-ce-cli             5:27.4.0-1~ubuntu.24.04~noble arm64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras 5:27.4.0-1~ubuntu.24.04~noble arm64        Rootless support for Docker.
ii  docker-compose-plugin     2.31.0-1~ubuntu.24.04~noble   arm64        Docker Compose (V2) plugin for the Docker CLI.

Recent APT versions also support the below command:

apt list --installed 'docker*'

Then the output would be similar to the one below:

Listing... Done
docker-buildx-plugin/noble,now 0.19.2-1~ubuntu.24.04~noble arm64 [installed]
docker-ce-cli/noble,now 5:27.4.0-1~ubuntu.24.04~noble arm64 [installed]
docker-ce-rootless-extras/noble,now 5:27.4.0-1~ubuntu.24.04~noble arm64 [installed,automatic]
docker-ce/noble,now 5:27.4.0-1~ubuntu.24.04~noble arm64 [installed]
docker-compose-plugin/noble,now 2.31.0-1~ubuntu.24.04~noble arm64 [installed]

The above outputs show that I installed the Docker CE package. If you installed docker.io instead, you would get one of the outputs below:

ii  docker.io         26.1.3-0ubuntu1~24.04.1 arm64        Linux container runtime

Listing... Done
docker.io/noble-updates,now 26.1.3-0ubuntu1~24.04.1 arm64 [installed]

Docker on Red Hat-based Linux distributions

» Back to table of contents «

Red Hat-based Linux distributions might use dnf or yum. Then you can search for packages by running the following commands:

dnf list --installed 'docker*'
# or
yum list --installed 'docker*'

Find non-snap dockerd processes on Linux

» Back to table of contents «

Regardless of the package manager and the exact package, you can get the docker daemon process in the terminal:

ps auxf | grep dockerd | grep -v 'snap\|grep'

And you would get something like this:

root 5584 0.0 1.7 1966352 68348 ? Ssl 20:08 0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

If you see "rootlesskit" in the output like below

ubuntu      1767  0.0  0.2 1826120 11392 ?       Ssl  16:25   0:00  \_ rootlesskit --state-dir=/run/user/1000/dockerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
ubuntu      1778  0.0  0.2 1899656 9984 ?        Sl   16:25   0:00      \_ /proc/self/exe --state-dir=/run/user/1000/dockerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
ubuntu      1808  0.0  1.6 2039632 66504 ?       Sl   16:25   0:00      |   \_ dockerd

You have Rootless Docker, which just means the daemon is running as your non-root user also using a different socket and Docker data dir.

Docker Desktop

Docker Desktop on Linux

» Back to table of contents «

Docker Desktop can be installed on Linux, macOS and Windows (except Windows Server), so the way to find the process in a process list could be different, but in this post I focus on Linux, because that is where most of you can be confused by the multiple ways to install Docker.

If you have Docker Desktop installed on Linux, the following command would reveal it to you:

ps aux | grep docker-desktop | grep -v grep

The output:

takacsa+    9804  0.0  0.5 1335920 81420 ?       Ssl  21:43   0:00 /opt/docker-desktop/bin/com.docker.backend
takacsa+    9818  0.2  0.6 1402484 110804 ?      Sl   21:43   0:03 /opt/docker-desktop/bin/com.docker.backend run
takacsa+    9863  0.3  1.2 1187000148 198528 ?   Sl   21:43   0:05 /opt/docker-desktop/Docker Desktop --reason=open-tray --analytics-enabled=false --name=dashboard
takacsa+    9912  0.0  0.3 33806596 52224 ?      S    21:43   0:00 /opt/docker-desktop/Docker Desktop --type=zygote --no-zygote-sandbox
takacsa+    9913  0.0  0.3 33806588 52352 ?      S    21:43   0:00 /opt/docker-desktop/Docker Desktop --type=zygote
takacsa+    9915  0.0  0.0 33806616 12876 ?      S    21:43   0:00 /opt/docker-desktop/Docker Desktop --type=zygote
takacsa+    9972  0.1  0.8 34220684 144844 ?     Sl   21:43   0:02 /opt/docker-desktop/Docker Desktop --type=gpu-process --enable-crash-reporter=978d60a8-7422-4f87-adcc-19c08da28830,no_channel --user-data-dir=/home/takacsakos/.config/Docker Desktop --gpu-preferences=WAAAAAAAAAAgAAAEAAAAAAAAAAAAAAAAAABgAAEAAAA4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAGAAAAAAAAAAYAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA== --shared-files --field-trial-handle=3,i,9326157886237964838,13802503777149121506,262144 --disable-features=SpareRendererForSitePerProcess --variations-seed-version
takacsa+    9977  0.0  0.4 33872616 68480 ?      Sl   21:43   0:00 /opt/docker-desktop/Docker Desktop --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --enable-crash-reporter=978d60a8-7422-4f87-adcc-19c08da28830,no_channel --user-data-dir=/home/takacsakos/.config/Docker Desktop --standard-schemes=app --secure-schemes=app --fetch-schemes=scout-graphql,scout-rest,docker-hub,docker-extensions-be,project-api --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,9326157886237964838,13802503777149121506,262144 --disable-features=SpareRendererForSitePerProcess --variations-seed-version
takacsa+   10050  0.8  0.9 1186775380 159476 ?   Sl   21:43   0:11 /opt/docker-desktop/Docker Desktop --type=renderer --enable-crash-reporter=978d60a8-7422-4f87-adcc-19c08da28830,no_channel --user-data-dir=/home/takacsakos/.config/Docker Desktop --standard-schemes=app --secure-schemes=app --fetch-schemes=scout-graphql,scout-rest,docker-hub,docker-extensions-be,project-api --app-path=/opt/docker-desktop/resources/app.asar --enable-sandbox --lang=en-US --num-raster-threads=4 --enable-main-frame-before-activation --renderer-client-id=4 --time-ticks-at-unix-epoch=-1734808931784927 --launch-time-ticks=4855412489 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,9326157886237964838,13802503777149121506,262144 --disable-features=SpareRendererForSitePerProcess --variations-seed-version --desktop-ui-launch-options={"isPackaged":true,"isMainWindow":true,"isE2eTest":false,"needsPrimaryIpcClient":true,"needsBackendErrorsIpcClient":true}

On a Linux, which supports ps -f to get a tree of processes, you can try the following:

ps auxf | grep docker-desktop | grep -v grep

And the output:

takacsa+    9804  0.0  0.5 1335920 81420 ?       Ssl  21:43   0:00  \_ /opt/docker-desktop/bin/com.docker.backend
takacsa+    9818  0.2  0.6 1402484 110804 ?      Sl   21:43   0:03  |   \_ /opt/docker-desktop/bin/com.docker.backend run
takacsa+    9863  0.3  1.2 1187000148 198588 ?   Sl   21:43   0:05  |       \_ /opt/docker-desktop/Docker Desktop --reason=open-tray --analytics-enabled=false --name=dashboard
takacsa+    9912  0.0  0.3 33806596 52224 ?      S    21:43   0:00  |           \_ /opt/docker-desktop/Docker Desktop --type=zygote --no-zygote-sandbox
takacsa+    9972  0.1  0.8 34220684 144844 ?     Sl   21:43   0:02  |           |   \_ /opt/docker-desktop/Docker Desktop --type=gpu-process --enable-crash-reporter=978d60a8-7422-4f87-adcc-19c08da28830,no_channel --user-data-dir=/home/takacsakos/.config/Docker Desktop --gpu-preferences=WAAAAAAAAAAgAAAEAAAAAAAAAAAAAAAAAABgAAEAAAA4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAGAAAAAAAAAAYAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA== --shared-files --field-trial-handle=3,i,9326157886237964838,13802503777149121506,262144 --disable-features=SpareRendererForSitePerProcess --variations-seed-version
takacsa+    9913  0.0  0.3 33806588 52352 ?      S    21:43   0:00  |           \_ /opt/docker-desktop/Docker Desktop --type=zygote
takacsa+    9915  0.0  0.0 33806616 12876 ?      S    21:43   0:00  |           |   \_ /opt/docker-desktop/Docker Desktop --type=zygote
takacsa+   10050  0.7  0.9 1186775380 159476 ?   Sl   21:43   0:11  |           |       \_ /opt/docker-desktop/Docker Desktop --type=renderer --enable-crash-reporter=978d60a8-7422-4f87-adcc-19c08da28830,no_channel --user-data-dir=/home/takacsakos/.config/Docker Desktop --standard-schemes=app --secure-schemes=app --fetch-schemes=scout-graphql,scout-rest,docker-hub,docker-extensions-be,project-api --app-path=/opt/docker-desktop/resources/app.asar --enable-sandbox --lang=en-US --num-raster-threads=4 --enable-main-frame-before-activation --renderer-client-id=4 --time-ticks-at-unix-epoch=-1734808931784927 --launch-time-ticks=4855412489 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,9326157886237964838,13802503777149121506,262144 --disable-features=SpareRendererForSitePerProcess --variations-seed-version --desktop-ui-launch-options={"isPackaged":true,"isMainWindow":true,"isE2eTest":false,"needsPrimaryIpcClient":true,"needsBackendErrorsIpcClient":true}
takacsa+    9977  0.0  0.4 33872616 68480 ?      Sl   21:43   0:00  |           \_ /opt/docker-desktop/Docker Desktop --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --enable-crash-reporter=978d60a8-7422-4f87-adcc-19c08da28830,no_channel --user-data-dir=/home/takacsakos/.config/Docker Desktop --standard-schemes=app --secure-schemes=app --fetch-schemes=scout-graphql,scout-rest,docker-hub,docker-extensions-be,project-api --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,9326157886237964838,13802503777149121506,262144 --disable-features=SpareRendererForSitePerProcess --variations-seed-version

If the virtual machine is running too, you will see the following as well:

takacsa+   14622  0.0  0.0 4060012 2432 ?        Sl   22:34   0:00  |       \_ /opt/docker-desktop/bin/virtiofsd --socket-path=/home/takacsakos/.docker/desktop/virtiofs.sock0 -o cache=auto --shared-dir=/home --sandbox=none --announce-submounts --xattr --xattrmap=:prefix:all::user.docker.desktop.::bad:all::: --translate-uid squash-guest:0:1000:4294967295 --translate-gid squash-guest:0:1000:4294967295
takacsa+   14670 97.7  4.4 5389724 728184 ?      Sl   22:34   0:08  |       \_ qemu-system-x86_64 -accel kvm -cpu host -machine q35 -m 3958 -smp 8 -kernel /opt/docker-desktop/linuxkit/kernel -append init=/init loglevel=1 root=/dev/vdb rootfstype=erofs ro vsyscall=emulate panic=0 eth0.dhcp eth1.dhcp linuxkit.unified_cgroup_hierarchy=1     vpnkit.connect=tcp+connect://192.168.65.2:40409 console=ttyS0 -serial pipe:/tmp/qemu-console888894299/fifo -netdev user,id=net0,ipv6=off,net=192.168.65.0/24,dhcpstart=192.168.65.9 -device virtio-net-pci,netdev=net0 -vga none -nographic -monitor none -drive if=none,file=/home/takacsakos/.docker/desktop/vms/0/data/Docker.raw,format=raw,id=hd0 -device virtio-blk-pci,drive=hd0,serial=dummyserial -drive if=none,file=/opt/docker-desktop/linuxkit/boot.img,format=raw,id=hd1,readonly=on -device virtio-blk-pci,drive=hd1,serial=dummyserial -object memory-backend-memfd,id=mem,size=3958M,share=on -numa node,memdev=mem -chardev socket,id=char0,path=/home/takacsakos/.docker/desktop/virtiofs.sock0 -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=virtiofs0

Docker Desktop on macOS

» Back to table of contents «

You don't have as many ways to install Docker on macOS as you have on Linux, but you could still have Docker Desktop or Rancher Desktop, or you could have only the client while managing a remote Docker daemon ona remote server. so it cans till be useful to find out if Docker Desktop os installed. Of course, on macOS, you would also have the whale icon at the top of the screen, but for those who prefer the terminal, or in case the icon is not shown, the below command can tell you if Docker Desktop is running:

ps aux | grep Docker.app | grep -v grep

The output is quite long, but you would find the below part as well:

/Applications/Docker.app/Contents/MacOS/Docker Desktop.app/

Docker.app is also the name of the application that you can look for on your Mac if the Desktop is not running, but you want to know if it is installed or not. Since the app name can be changed in a future release, searching for docker in the terminal instead of Docker.app can also be enough, but you may have a virtual machine or any folder appearing in the process list even if it is not for Docker Desktop.

Docker Desktop on Windows

» Back to table of contents «

On Windows, you can use the "Task Manager" desktop app or run the following command in PowerShell:

Get-Process | Select-String docker

The application on Windows is called "Docker Desktop"

Docker Contexts

» Back to table of contents «

You can also check what contexts are configured for the Docker client. Regardless of which Docker you are using, the client is most likely using the same $HOME/.docker folder which contains information about the contexts. Although this directory can be changed, I don't remember any client changing it, except when I made a video for which I wrote a script to do it. The following command can list all the contexts you have:

docker context ls

The output could show something like this:

NAME              DESCRIPTION                               DOCKER ENDPOINT                                       ERROR
default           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                           
desktop-linux *   Docker Desktop                            unix:///home/takacsakos/.docker/desktop/docker.sock   
rootless          Rootless mode                             unix:///run/user/1000/docker.sock

The active context (which you are using when running docker commands) is the one with the "*" character on the right side of the context name.

The endpoint of the default is usually the following unix Domain socket (on Linux and macOS): unix:///var/run/docker.sock.

Just because you see a local unix socket, it doesn't necessarily mean, you are using a local daemon directly on the host, although usually that is the case.
So the default context is usually for the Docker Engine running directly on the host.

Now then notice the "desktop-linux" context which shows another unix domain socket, but we know that the Docker Engine is running in the virtual machine of Docker Desktop. In fact, Docker Desktop can also use the default context with the default endpoint depending on how you installed it.
You always need to check the context name and the description as well. If you don't know what that means, you can still ask the community, but make sure you search for it first.

You can also notice the "rootless" context. Anyone could name a context "rootless", but this is usually the rootless version of Docker CE. That means, that you are most likely have Docker CE, or at least a Docker that runts directly on your Linux host.

Remote Docker daemon

» Back to table of contents «

Sometimes you have the client on your local machine, but the daemon is not just in a virtual machine, but in a remote machine over which you have no control at all. If you control that remote machine, you most likely know about it, but some CI/CD tools like CircleCI can support a Docker Engine which is not directly running in your environment. People often notice it when they try to access containers using their IP addresses, and it doesn't work. The same problem occurs when you use Docker Desktop since Docker Desktop runs the Docker daemon in a virtual machine, but another case can be when your processes run in a container while the Docker daemon is running either on the host or more likely in another container or a remote Docker host. Whether it is a virtual machine or physical machine is irrelevant.

If you are not the one who installed Docker and you don't even control the environment, but you paid for an online service, always rad the documentation and ask their support or community whenever it is possible.

Conclusion

» Back to table of contents «

Maybe you have Rancher Desktop which could use docker as container Engine, or you could have Podman or Podman Desktop, which are really not Docker, but the method is the same.

You use the package manager to list installed packages filtered to the name of the software
or try to look for it in the process list,
or use docker context ls to learn about Docker contexts which can help you find out which Docker daemon you are trying to connect to.

If the operating system is not Linux, you can still use the tools supported by the operating system to list running processes and installed apps. I would not recommend running multiple Docker on the same machine for beginners, but if you have any of the mentioned apps or even something I haven't mentioned, be aware of which one you are running at the moment and share it with people when you need help from them.

Using gVisor's container runtime in Docker Desktop

Ákos Takács — Tue, 26 Nov 2024 22:41:49 +0000

Introduction

I compared 3 container runtimes previously, which included runsc from gVisor. Since the default is runc, that is the default in Docker Desktop as well. The Kata runtime could not be tested in Docker Desktop, since that would require nested virtualization in the VM of Docker Desktop, so runsc is the only runtime from that comparison we could try in Docker Desktop which isn't already in it.

The only question is how we could copy the runtime binary into Docker Desktop. This could also be asked in general related to any binary, since the root filesystem of Docker Desktop is read-only, but the runtime is executed by Docker so copying that would actually make sense.

Download runsc into Docker Desktop
Configure the Docker daemon to use the runtime
Download and install runsc in one step
Testing runsc after the installation
Reloading the internal daemon config to support Windows
Possible error messages

Download runsc into Docker Desktop

» Back to table of contents «

If a runtime can be downloaded directly without using any package manager, you can download it to a location which can be read by the Docker daemon. There is one thing that we know the Docker daemon can read, and we can write, and that is a local volume. So we will need a container that downloads runsc to a local volume.

First we will need to check the official documentation for an installation guide. Despite that the documentation mentions installing specific releases as well, I couldn't figure it out and the link they provide to the releases on GitHub shows that there is no release at all, so we will install the latest version.

I used the script from the documentation, except that I removed the parenthesis and I also needed to remove sudo from the beginning of the last line. We don't need it, since we will use the root user in the container. I will use the script directly in a compose file, so I had to escape all the dollar characters with a second dollar character. This is the compose file:

services:
  runsc-installer:
    image: alpine:3.20
    volumes:
      - type: volume
        source: data
        target: /usr/local/bin
    command:
      - sh
      - -c
      - |
        set -e
        ARCH=$(uname -m)
        URL=https://storage.googleapis.com/gvisor/releases/release/latest/$${ARCH}
        wget $${URL}/runsc $${URL}/runsc.sha512 \
          $${URL}/containerd-shim-runsc-v1 $${URL}/containerd-shim-runsc-v1.sha512
        sha512sum -c runsc.sha512 \
          -c containerd-shim-runsc-v1.sha512
        rm -f *.sha512
        chmod a+rx runsc containerd-shim-runsc-v1
        mv runsc containerd-shim-runsc-v1 /usr/local/bin

volumes:
  data:
    name: runsc-runtime-binaries

The chance of you using runsc-runtime-binaries as a volume for something else is very low. But if you do have it, make sure you use a different volume name.

Place the file anywhere as compose.yml and run

docker compose run --rm runsc-installer

You will see the progress which is something like this:

Connecting to storage.googleapis.com (172.217.19.123:443)
saving to 'runsc'
runsc                100% |****************************************************************| 60.2M  0:00:00 ETA
'runsc' saved
Connecting to storage.googleapis.com (172.217.19.123:443)
saving to 'runsc.sha512'
runsc.sha512         100% |****************************************************************|   136  0:00:00 ETA
'runsc.sha512' saved
Connecting to storage.googleapis.com (172.217.19.123:443)
saving to 'containerd-shim-runsc-v1'
containerd-shim-runs 100% |****************************************************************| 27.7M  0:00:00 ETA
'containerd-shim-runsc-v1' saved
Connecting to storage.googleapis.com (172.217.19.123:443)
saving to 'containerd-shim-runsc-v1.sha512'
containerd-shim-runs 100% |****************************************************************|   155  0:00:00 ETA
'containerd-shim-runsc-v1.sha512' saved
runsc: OK
containerd-shim-runsc-v1: OK

Whenever a new version comes out, you can run the command again, and it will replace the old files.

Configure the Docker daemon to use the runtime

» Back to table of contents «

As I also mentioned in Everything about Docker volumes, you can inspect a volume and see the mount point

docker volume inspect runsc-runtime-binaries

This is my output:

[
    {
        "CreatedAt": "2024-10-28T17:21:24Z",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "03-gvisor-in-docker-desktop-data",
            "com.docker.compose.version": "2.29.7",
            "com.docker.compose.volume": "data"
        },
        "Mountpoint": "/var/lib/docker/volumes/runsc-runtime-binaries/_data",
        "Name": "runsc-runtime-binaries",
        "Options": null,
        "Scope": "local"
    }
]

Normally a mount point would be where you mount something not from where you mounted it, but I guess it is called mount point because of what I described in the linked tutorial, since unless you are using the default local volumes, this is where the data could be mounted to. Now that we know that, the following script can be used to add the runtime to the daemon config:

#!/usr/bin/env bash

volume="runsc-runtime-binaries"
volume_path="/var/lib/docker/volumes/$volume/_data"

docker run -it --rm \
  --mount "type=bind,source=$HOME/.docker,target=/etc/docker" \
  --mount "type=volume,source=$volume,target=$volume_path" \
  ubuntu "$volume_path/runsc" install

This script also mounts $HOME/.docker to /etc/docker in the container, since it contains the daemon.json and runsc install will add the parameters to /etc/docker/daemon.json by default and back up the original file as daemon.json~. If you save the script as runsc-install.sh, this is how you could run it:

chmod +x ./runsc-install.sh
./runsc-install.sh

This script would work only on Linux and macOS, and we would have other problems as well, so before doing too much this way, let's jump to the next section and introduce Docker Compose.

Download and install runsc in one step

» Back to table of contents «

If we want to make the installer simpler, we can add the "install" command to the compose file. This means that we have to mount $HOME/.docker in the compose file. We should also remove the last line of the original script in the compose file, because we don't want to move the binaries to /usr/local/bin anymore, since runsc install will need to run from a different folder, and we need only one mount for the volume. So we change the mount definition of the volume too and we also want a platform independent way to refer to our home folder so we replace $HOME with the tilde character (~), which actually came as a surprise to me, but it works in the compose file even on Windows. And finally, we run the installer. So these are the last three lines:

dest="/var/lib/docker/volumes/${RUNSC_VOLUME_NAME:-runsc-runtime-binaries}/_data/"
mv runsc containerd-shim-runsc-v1 $$dest
$$dest/runsc install

Notice that $dest is escaped with a second dollar character but $RUNSC_VOLUME_NAME isn't. This is because the volume name will be a compose parameter with a default value so it has to be interpreted by compose. This is the full compose file:

services:
  runsc-installer:
    image: alpine:3.20
    volumes:
      - type: volume
        source: data
        target: /var/lib/docker/volumes/${RUNSC_VOLUME_NAME:-runsc-runtime-binaries}/_data/
      - type: bind
        source: ~/.docker
        target: /etc/docker
    command:
      - sh
      - -c
      - |
        set -e
        ARCH=$(uname -m)
        URL=https://storage.googleapis.com/gvisor/releases/release/latest/$${ARCH}
        wget $${URL}/runsc $${URL}/runsc.sha512 \
          $${URL}/containerd-shim-runsc-v1 $${URL}/containerd-shim-runsc-v1.sha512
        sha512sum -c runsc.sha512 \
          -c containerd-shim-runsc-v1.sha512
        rm -f *.sha512
        chmod a+rx runsc containerd-shim-runsc-v1

        dest="/var/lib/docker/volumes/${RUNSC_VOLUME_NAME:-runsc-runtime-binaries}/_data/"
        mv runsc containerd-shim-runsc-v1 $$dest
        $$dest/runsc install
volumes:
  data:
    name: ${RUNSC_VOLUME_NAME:-runsc-runtime-binaries}

This way you can even change the volume name. and run the compose project like this on Linux and macOS (or on Windows, with WSL2 integration enabled, and running docker commands in your WSL distribution):

RUNSC_VOLUME_NAME="runsc-bin" docker compose run --rm runsc-installer

My previous examples used docker compose run, but you can use docker compose up as well. Then the log lines will be prefixed with the container name and the container will be kept after running it.

Testing runsc after the installation

» Back to table of contents «

Before you test the runtime, don't change any Docker Desktop configuration, because that will reset your client config on the host, since the VM still doesn't know about it. In order to avoid resetting the config file, "Quit Docker Desktop" and then start it again. DO NOT just "Restart" it in the menu, because that will just restart services in the VM without updating the config.

When the config is ready and Docker Desktop is started again, you can run

docker run --rm --runtime runsc ubuntu uname -a

If everything works, you will see something like this:

Linux 73309f6efa2c 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 aarch64 aarch64 aarch64 GNU/Linux

For more testing ideas, check out my blog post about the comparison of runtimes.

Reloading the internal daemon config to support Windows

» Back to table of contents «

I already mentioned some ways to support Windows, but there is one more problem. When using the WSL2 backend, Docker Desktop on Windows supports Nvidia GPUs, because WSL2 supports it too. Unfortunately, it seems that the Nvidia runtime configuration currently completely overrides the "runtimes" section in the daemon config instead of adding itself to the existing "runtimes" section.

That means, even if we add runsc in the daemon config on the host, it will not be usable You will see it even in Docker Desktop's graphical interface, but it will not actually be the one that is used by the Docker daemon. So I will show a workaround I came up with, until it is fixed in Docker Desktop if it is ever considered a bug to be fixed. Let's not forget that there is nothing to indicate that using a custom runtime is supported in Docker Desktop. Just because we can do something, it doesn't mean we will always be able to do it if it is not supported. So maybe it will be fixed in the future, but it is also possible, that nothing that I described in this blogpost will work in the future. With that in mind, here is the workaround:

We have to find the config file used by the docker daemon in the virtual machine
We have to mount it from the VM into our installer container
We have to find a way to pass a custom config path to the runsc install command and automatically update the internal config file
After updating the internal config file, we have to reload it, which means, we have to send a HUP signal to the dockerd process.
In order to be able to send signals to the dockerd process, we need to use the process namespace of the host.

The first can be done by running the following command:

docker run --rm -it --pid host alpine:3.20 sh -c 'ps aux | grep dockerd | grep -v grep'

This should show something like the following:

308 root 0:45 /usr/local/bin/dockerd --config-file /run/config/docker/daemon.json --containerd /run/containerd/containerd.sock --pidfile /run/desktop/docker.pid --swarm-default-advertise-addr=192.168.65.3 --host-gateway-ip 192.168.65.254

So we know, the config is at /run/config/docker/daemon.json. Let's check the content by running the following command on Windows

docker run -v /run/config/docker/daemon.json:/daemon.json alpine:3.20 cat /daemon.json

The output is something like this:

{"builder":{"gc":{"defaultKeepStorage":"20GB","enabled":true}},"experimental":true,"features":{"cone-registries":["hubproxy.docker.internal:5555"],"mtu":1500,"runtimes":{"nvidia":{"path":"nvidia-contain-driver":"overlayfs","userland-proxy":false}

This indeed shows the nvidia container runtime.

We could also download "runsc" to somewhere or use the one on the already created volume to run runsc install -help, or we can check the source code too: https://github.com/google/gvisor/blob/745828301c936ddd1dac49b2611bdf1a4477f9ab/runsc/cmd/install.go#L60

This shows we have a config_file option. Great. Now we need to notify the dockerd process that it should reload the daemon config, which requires some Linux experience, but the pkill command can help us like this:

pkill -HUP dockerd

This will work only if we add pid: host to the installer service.

Note that this will notify all dockerd processes, so if you have Docker in Docker, it will reload all Docker daemon's configuration. It shouldn't be a problem unless you broke the config of any docker daemon. The new compose file is the following now:

services:
  runsc-installer:
    image: alpine:3.20
    volumes:
      - type: volume
        source: data
        target: /var/lib/docker/volumes/${RUNSC_VOLUME_NAME:-runsc-runtime-binaries}/_data/
      - type: bind
        source: ~/.docker
        target: /etc/docker
      - type: bind
        source: /run/config/docker/daemon.json
        target: /run/config/docker/daemon.json
    pid: host
    command:
      - sh
      - -c
      - |
        set -e
        ARCH=$(uname -m)
        URL=https://storage.googleapis.com/gvisor/releases/release/latest/$${ARCH}
        wget $${URL}/runsc $${URL}/runsc.sha512 \
          $${URL}/containerd-shim-runsc-v1 $${URL}/containerd-shim-runsc-v1.sha512
        sha512sum -c runsc.sha512 \
          -c containerd-shim-runsc-v1.sha512
        rm -f *.sha512
        chmod a+rx runsc containerd-shim-runsc-v1

        dest="/var/lib/docker/volumes/${RUNSC_VOLUME_NAME:-runsc-runtime-binaries}/_data/"
        mv runsc containerd-shim-runsc-v1 $$dest
        $$dest/runsc install

        $$dest/runsc install -config_file /run/config/docker/daemon.json
        pkill -HUP dockerd
volumes:
  data:
    name: ${RUNSC_VOLUME_NAME:-runsc-runtime-binaries}

Using this compose file, you can even test runsc on Windows, but you will have to run the installer every single time you restart Docker Desktop.

Possible error messages

» Back to table of contents «

If something goes wrong and runsc cannot be found, you can see an error message like below, which I got when I accidentally wrote "volume" instead of "volumes" in the script, so Docker tried to find the runtime using an incorrect path.

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/3a01e11f99939f9fd85151e2ae97f15d5f39ff94e835187357b82807147dcc52/log.json: no such file or directory): fork/exec /var/lib/docker/volume/runsc-runtime-binaries/_data/runsc: no such file or directory: <nil>: unknown.

Without the automatic reloading of the internal daemon config, if you try the runtime too early before activating it in the VM, for example because your config was reset to the previous one after you changed something in Docker Desktop before stopping it, you could get an error about the invalid runtime name:

docker: Error response from daemon: unknown or invalid runtime name: runsc.
See 'docker run --help'.

Conclusion

Changing the runtime in Docker Desktop is not the first thing that Docker Desktop users think. It is probably not even in the top 10, but sometimes, it can be useful if you have only Docker Desktop at the moment. So this post was basically just a way to know more about Docker Desktop, Docker Compose, and how we can change the daemon configuration manually, or automatically.

If you break the daemon configuration, you can fix it, but you have to quit Docker Desktop and start it again.

You could also learn a little bit about volumes, but if you need more, you can read the already mentioned Everything about Docker volumes

And finally, it is always good to understand general Linux commands even when we use Docker Desktop on Windows, but we want to run Linux containers. It helps us to troubleshoot and do things that are not officially supported yet, but can be done at the moment.

Docker Desktop is not Docker, as I pointed it out in "You run containers, not dockers - Discussing Docker variants, components and versioning", which means there are some things that you can do with Docker Desktop but not with Docker CE, or you can do something with Docker CE but not with Docker Desktop. I still like to find the limits of both so we can enjoy as many features of both as possible.

Comparing 3 Docker container runtimes - Runc, gVisor and Kata Containers

Ákos Takács — Tue, 29 Oct 2024 19:04:16 +0000

Introduction

Previously I wrote about the multiple variants of Docker and also the dependencies behind the Docker daemon. One of the dependencies was the container runtime called runc. That is what creates the usual containers we are all familiar with. When you use Docker, this is the default runtime, which is understandable since it was started by Docker, Inc.

We can change it and use another runtime to create containers differently. We can also choose a runtime that doesn't even create containers, but virtual machines. You could also create your own runtime that adds something to the definition of the container or increases the level of the isolation.

The Docker documentation mentions alternative container runtimes, but I will not write about all of them. Obviously I'm not trying to copy the documentation. I want to compare 3 different kind of runtimes which you could use for a simple Linux container like Ubuntu.

I will not explain how you can install these runtimes, but if you think the documentations are not good enough, please, let me know in the comments and I will see what I can do.

You can also watch a video of this topic on YouTube.

The chosen 3 runtimes
Introduce the runtimes
- runc
- kata-runtime
- runsc
Comparing the runtimes
Checking differences in practice
Resource handling and limitations of the Kata runtime
Conclusion

The chosen 3 runtimes

» Back to table of contents «

Although the documentation also mentions "youki", that is mentioned as a "drop-in replacement" of the default runtime basically doing the same, so let's stick with runc. The second runtime will be Kata runtime from Kata containers, since it runs small virtual machines which is good for showing how differently it uses the CPU and memory. This also adds a higher level of isolation with some downsides as well. And the third runtime will be runsc from gVisor which is a perfect third runtime to see how we can run containers and still have a little more secure isolation. I will show how we can recognize the differences by running commands from the isolated environments and from the host.

Introduce the runtimes

runc

» Back to table of contents «

It probably doesn't even require much explanation, since this is the default, and we could just compare everything to it. What runc does is what we are already used to. It creates and runs containers using Linux kernel namespaces based on a config file which we don't have to create manually if we use Docker. We know that every process is running on the host and the kernel just doesn't let the processes see everything on the host. Which doesn't change the fact that we can see everything from the host that runs in the containers.

Since everything runs on the host, nothing is required in the container, only the process which we want to isolate. It means there is no visible kernel in the container on the filesystem. Just because you don't see the wheels inside a car, it doesn't mean the car is just floating. But when you run uname -a in the container to get information about the Kernel, the output will be the same as you could see on the host, except the hostname which is different in a container unless you use the host network or share the UTS namespace of the host with the container.

There is no hardware virtualization, the processes in a container can see all the resources on the host, including the CPU and memory. Of course, you can set a CPU or memory limit for the containers, but that only restricts the amount of memory and CPU that the kernel allows the processes in the container to use. The hardware remains the same regardless of where the process is.

It can also lead to problems, since some applications like databases could use a default resource limit based on the available hardware. So if you set a lower memory limit for the container, the application will try to use more memory and the operating system will kill it. So setting the resource limits on application level can be important in addition to the container level resource limits. But what if the application doesn't support it or there is a bug, and it ignores the parameter?

kata-runtime

» Back to table of contents «

As I mentioned before, the Kata runtime runs containers in their own very small virtual machines. If you still remember how runc worked, let's see how the Kata runtime changes everything?

The key is the fact that it runs a virtual machine below the container. Since we have a virtual machine, we need a kernel in the VM. That means, when you run uname -a you will get information about a different kernel. The one in the virtual machine created by the Kata runtime. Even though there is a virtual machine with another kernel, we will still not see the kernel on the filesystem, since the container is not replaced, but extended with a virtual machine layer.

The CPU and memory from the container will not be the same either. In this case, we will have hardware virtualization, so we will see the hardware available for the virtual machine. It also means we will not be able to use all the CPUs and all the memory. A VM created by the Kata runtime will get 2 gigabytes of memory and 1 vCPU. So even if the application in the container does not support resource limits, it can only detect the hardware in the VM. There are some important details regarding resource limits with the Kata runtime, but we will discuss it later.

It is still important to know, that the Docker daemon will still be on the host, and not in the VM created by the Kata runtime. It should be obvious, since the virtual machine is created by the runtime which is executed by a shim process instructed by containerd which is instructed by dockerd, the Docker daemon. So it couldn't possibly be in the virtual machine that it will create. This fact will be important when we want to talk about mounting files from the "host machine". The host machine is where the Docker daemon is running.

runsc

» Back to table of contents «

As mentioned before, runsc from gVisor creates containers. The difference is that tries to make the container more secure by intercepting system calls sent by processes in the container before the host kernel could handle it. This interception makes some requests a little slower. In fact, the documentation mentions that you should use runsc only for "user-facing containers" like reverse proxy containers that the users are directly interacting with. Of course, if you are also interacting with a web application with a security hole and someone can execute a command through your application, using runsc only for the reverse proxy will not help a lot. But the point is that you probably don't want to use this runtime for all your containers due to the impact on the performance.

Let's assume you use it. Then the process in the container will see almost everything that it would see from a container created by runc, the default runtime. The difference is that runsc will have an application kernel to handle the intercepted system calls so when you run uname -a, you will see that kernel.

Comparing the runtimes

» Back to table of contents «

The below table shows a comparison of the three runtimes

	runc (default)	runsc	kata-runtime
developer	opencontainers	gVisor (Google)	Kata Containers
isolation type	container	container	virtual machine
available resources	all resources	all resources	limited resources
used kernel	host kernel	application kernel	kernel in the VM
installable on	VM / Physical	VM / Physical	VM (with nested virt.) / Physical

As you can see, most of the differences are the consequences of the isolation type. If you know what the runtimes create and the differences between a VM and a container, you have a pretty good idea what to expect. The one additional difference is the application kernel used by runsc. That's why you see a different kernel, but in every case you still have a container either on the host or in a VM, so you will never see the kernel files. Even if you mount /boot into the container, you will mount it from the host where the Docker daemon is running, so you can't mount that folder from the VM created by the Kata runtime.

The second difference which is worth mentioning is that the runtimes that create containers can be used in virtual machines, but when the runtime creates a virtual machine, you need nested virtualization enabled for the host VM. This means you could not test Kata containers in Docker Desktop.

Checking differences in practice

» Back to table of contents «

I wrote a script that can execute the same test command in different environments, including the host machine. I uploaded it to GitHub as a gist, but I share here too to make sure you don't depend on the availability of GitHub and the gist.

#!/usr/bin/env bash

set -eu -o pipefail

runtimes=(Host runc runsc kata)

YELLOW_START='\033[1;33m'
YELLOW_END='\033[0m'

declare -A COMMANDS=(
  [cpus]='nproc'
  [memory]='free | grep "Mem" | awk "{print \$2}"'
  [kernel]='uname -nrv'
  [filesystem]='ls /boot | awk "/vmlinuz-/" | sort -r | head -n1'
)

function runtime_run() {
  local mode="$1"
  local runtime="$2"
  local command="$3"

  if [[ "$runtime" != "Host" ]]; then
    command="docker run --rm --runtime $runtime ubuntu $command"
  fi

  case "$mode" in
    echo)   echo "$command" ;;
    yellow) echo -e "${YELLOW_START}${command}${YELLOW_END}" ;;
    exec)   eval "$command" ;;
    *) >&2  echo "Invalid mode: $mode. Valid modes: echo, exec"; return 1
  esac
}

function showresult() {
  local runtime="$1"
  local label="$2"
  local command="$3"

  runtime_run yellow "$runtime" "$command"
  echo -n "$label: "
  runtime_run exec "$runtime" "$command"
}

labels=("$@")
if (( "${#labels[@]}" == 0 )); then
  for label in "${!COMMANDS[@]}"; do
    labels+=("$label")
  done
fi

for label in "${labels[@]}"; do
  for runtime in "${runtimes[@]}"; do
    showresult "$runtime" "$label" "${COMMANDS[$label]}"
    echo
  done
done

Since using the gist is still easier, let's try to download it first:

curl -L --output runtime-test.sh https://gist.githubusercontent.com/rimelek/05241c26a3b10ff8c9cfe1035b787996/raw/5ef46b3cbdeda5e5bd1f094724929ed9514e2f85/docker-container-runtime-test.sh
chmod +x runtime-test.sh

Then you can either run

./runtime-test.sh

without parameters, or test the availability of the CPU, memory and kernel one by one. The beginning of the script shows the categories you can test and what commands will be executed:

declare -A COMMANDS=(
  [cpus]='nproc'
  [memory]='free | grep "Mem" | awk "{print \$2}"'
  [kernel]='uname -nrv'
  [filesystem]='ls /boot | awk "/vmlinuz-/" | sort -r | head -n1'
)

Even before the above part, you can find the list of runtimes

runtimes=(Host runc runsc kata)

If your runtime names are different, you can change the list of runtimes in the script. "Host" here is not a runtime, but the host machine. I intentionally wrote it with an uppercase "H" to make it different from the runtime names.

If you run

./runtime-test.sh cpus

The following commands will be generated:

nproc
docker run --rm --runtime runc ubuntu nproc
docker run --rm --runtime runsc ubuntu nproc
docker run --rm --runtime kata ubuntu nproc

You can also test the memory availability

./runtime-test.sh memory

The generated commands would be

free | grep "Mem" | awk "{print \$2}"
docker run --rm --runtime runc ubuntu free | grep "Mem" | awk "{print \$2}"
docker run --rm --runtime runsc ubuntu free | grep "Mem" | awk "{print \$2}"
docker run --rm --runtime kata ubuntu free | grep "Mem" | awk "{print \$2}"

To test the kernel version, you can run

./runtime-tets.sh kernel

which generates an executes these commands:

uname -nrv
docker run --rm --runtime runc ubuntu uname -nrv
docker run --rm --runtime runsc ubuntu uname -nrv
docker run --rm --runtime kata ubuntu uname -nrv

Yes, we are using uname -nrv here instead of uname -a to make the lines shorter by not showing redundant information like the CPU architecture multiple times in the output.

And finally, you can also try to list the files under /boot

./runtime-tets.sh filesystem

which generates and executes the following commands:

ls /boot | awk "/vmlinuz-/" | sort -r | head -n1
docker run --rm --runtime runc ubuntu ls /boot | awk "/vmlinuz-/" | sort -r | head -n1
docker run --rm --runtime runsc ubuntu ls /boot | awk "/vmlinuz-/" | sort -r | head -n1
docker run --rm --runtime kata ubuntu ls /boot | awk "/vmlinuz-/" | sort -r | head -n1

Using awk I try to list only files starting with vmlinuz-, but I want to get the latest only, because otherwise we could get too many files which would be irrelevant for the test. If there is one, that proves we can see the kernel files. In this test we assume the current kernel with which the operating system was booted is the latest.

You can see the output of the script running on my machine without arguments

ls /boot | awk "/vmlinuz-/" | sort -r | head -n1
filesystem: vmlinuz-5.15.0-122-generic

docker run --rm --runtime runc ubuntu ls /boot | awk "/vmlinuz-/" | sort -r | head -n1
filesystem:
docker run --rm --runtime runsc ubuntu ls /boot | awk "/vmlinuz-/" | sort -r | head -n1
filesystem:
docker run --rm --runtime kata ubuntu ls /boot | awk "/vmlinuz-/" | sort -r | head -n1
filesystem:
nproc
cpus: 12

docker run --rm --runtime runc ubuntu nproc
cpus: 12

docker run --rm --runtime runsc ubuntu nproc
cpus: 12

docker run --rm --runtime kata ubuntu nproc
cpus: 1

free | grep "Mem" | awk "{print \$2}"
memory: 16241924

docker run --rm --runtime runc ubuntu free | grep "Mem" | awk "{print \$2}"
memory: 16241924

docker run --rm --runtime runsc ubuntu free | grep "Mem" | awk "{print \$2}"
memory: 16241924

docker run --rm --runtime kata ubuntu free | grep "Mem" | awk "{print \$2}"
memory: 2038464

uname -nrv
kernel: ta-lxlt 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024

docker run --rm --runtime runc ubuntu uname -nrv
kernel: aea867200366 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024

docker run --rm --runtime runsc ubuntu uname -nrv
kernel: 91d3bb0285b8 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016

docker run --rm --runtime kata ubuntu uname -nrv
kernel: 9ca9368d8781 6.1.62 #1 SMP Mon Sep  9 09:44:34 UTC 2024

If you want to see me executing these commands, consider watching the video linked at the beginning of this post.

Resource handling and limitations of the Kata runtime

» Back to table of contents «

Since the Kata runtime runs a virtual machine, you cannot assign half a CPU to a VM. The docker commands still support setting --cpus 0.5, but it only means this will be the amount that the qemu process can use. By default, as mentioned before, the VM has 1 vCPU and if you set the limit to be half a CPU, you will get 1 and a half rounded up to an integer, so you will get 2. The limit will increase the default amount and not replace it.

This is what happens with the memory as well, except it doesn't have to be rounded up. So if you set --memory 500M you get 2 and a half gigabytes of memory. If you want to test it, you can use the commands generated by the scripts and add the limits:

docker run --rm -it --runtime kata --cpus 0.5 ubuntu nproc
docker run --rm -it --runtime kata --memory 500M ubuntu free | grep "Mem" | awk "{print \$2}"

The output is:

2
2562752

Where the second line is the memory in kilobytes.

I bet the first thing you think that it is a bug. There is an issue on GitHub where someone thought the same. The fact is that Kata containers are different and there are Limitations. The first I noticed too, that there is no way to share process or network namespaces between Docker containers. The fact that you cannot use the process namespace or network namespace of the host is easily understandable because we have a VM and not just a host kernel isolating our processes.

Conclusion

» Back to table of contents «

Originally Docker created only containers. In fact, it used LXC as an "exec driver" which is basically what we call runtime today or at least the closest thing to it. It was deprecated in Docker 1.8.0.

Now you can even choose a runtime which creates a virtual machine or a container with a more secure isolation. Once there was a runtime for using an NVIDIA GPU called nvidia-container-runtime. That project is now deprecated and Docker has the "--gpus" option instead. Talking about GPUs is not the scope of this blogpost, but it is a good example of a special runtime that gave additional capabilities to containers.

Each runtime has benefits and downsides. Which one is the best for you depends on what you need it for. I recommend testing the runtimes before making a decision. Running a small VM could seem to be a good idea, but you can discover downsides that change your mind.

You run containers, not dockers - Discussing Docker variants, components and versioning

Ákos Takács — Sun, 27 Oct 2024 10:23:26 +0000

Introduction

Usually I start with a blogpost and then I create a video. In this case I made a video in two languages, and now I'm here for you, who prefer reading. As the title suggests, I want to clear up a misunderstanding, and in the meantime explain how many kind of Docker exists and what components Docker has. We will also cover some of the history of Docker.

I often see that people think they are running "dockers". Keep in mind that "Dockers" is a garment company and has absolutely nothing to do with "Docker", the software.

If you prefer watching videos, most but not all of what I explain here can be watched on Youtube

For more videos, you can subscribe to my YouTube channel: @akos.takacs

What I call Docker
Docker as a company
Docker as a software
- The beginning of Docker as a software and where we are now
- Two main ways of running the Docker daemon in terms of isolation
- Docker running in a virtual machine
  - Docker in a VM in general
  - Docker in a VM using Docker Desktop
- Docker runing directly on a physical machine
  - The source code of Moby
  - Docker Enterprise Edition
  - Docker IO
  - Snap package
  - Docker CE
- Docker in Docker
- Docker CE components
  - Two main parts of Docker CE
  - Docker client
    - The docker command
    - Docker SDK
    - Docker Compose
  - Docker daemon
    - The history of dockerd
    - Rootless vs Rootful Docker
    - Docker daemon dependencies
- Podman pretending to be Docker
- Packages and versioning
Conclusion

What I call Docker

» Back to table of contents «

In short, the Docker daemon. I could stop this post hear, since when I say "I run Docker on a server" I mean I run the Docker daemon, even if I don't have the client on that server. The daemon is required for running containers, but a container is not a docker, Docker just creates a container. Similarly to how bread is not a baker.

Unfortunately, it is more complicated, so let's not stop this post here just yet.

Docker as a company

» Back to table of contents «

The company who made Docker as a software was "dotCloud, Inc". The website doesn't exist anymore, but you can still find the GitHub organization. Later the company was renamed to "Docker, Inc" which was big news back then, and you can find some articles linked on Docker's website.

Gigaom: The link takes you to an error page, so I don't even quote it.
InfoQ: https://www.infoq.com/news/2013/10/dotcloud-renamed-docker/
Forbes: https://www.forbes.com/sites/benkepes/2013/10/29/docker-and-the-timely-pivot/

Now, in a conversation we just say "Docker". For example "Docker Desktop is Docker's product." or "I know someone working at Docker". In the rest of this blog post we will talk about Docker as a software.

Docker as a software

The beginning of Docker as a software and where we are now

» Back to table of contents «

The first commit of Docker happened on January 19, 2013. You can still find it on GitHub: https://github.com/moby/moby/commit/a27b4b8cb8e838d03a99b6d2b30f76bdaf2f9e5d

Nowadays, Docker has different variants and multiple components. So when we say Docker, we could mean the entire software family. Collection of different products. That is what makes the confusion.

Two main ways of running the Docker daemon in terms of isolation

» Back to table of contents «

You can run Docker (meaning the Docker daemon) directly on a server or even on your machine, but there are two kind of containers. Linux containers require a Linux host. Windows containers require a Windows host. This is because a container is not a virtual machine. A container is just a process running on your host in an isolated way, so the process can't see everything on the host, only what it is allowed to see.

This means if you have a Windows machine, and you want to try Linux containers, you are in trouble. No... I'm kidding. But it is true that in this case you could not run Linux containers without the help of virtual machines.

I haven't mentioned macOS yet, because there is no such thing as "macOS container" at the moment, but you can install Docker in a virtual machine even on macOS.

Docker running in a virtual machine

Docker in a VM in general

» Back to table of contents «

If you want to run Linux containers on Windows or macOS, you need a virtual machine with a Linux operating system inside. If you want to run Windows containers on Linux or macOS, you could probably try a Windows VM, but I have never tried and Windows containers are different so you would also need to deal with two different isolation modes and the HyperV isolation mode would require nested virtualization enabled for your virtual machine.

Docker in a VM using Docker Desktop

» Back to table of contents «

A special way of using Docker in a virtual machine is when you have the client on your host and the Docker daemon is in the virtual machine. You could configure it manually similarly to how you would configure a remote Docker daemon for your local client, but Docker Desktop solves it for you.

Docker Desktop can run on Windows, macOS and also on Linux. Yes, it runs on Linux, but it still creates a virtual machine instead of accessing the Docker daemon on your host.

When you run Docker Desktop on Windows, you can also switch between Windows containers and Linux containers, but Windows containers are supported by Docker Desktop only on Windows. You could try running Docker Desktop in a Windows VM, but that would also require nested virtualization if you want to use Docker Desktop for running Linux containers, or Windows containers with HyperV isolation. Not to mention that as far as I remember, you could not even install Docker Desktop if you don't have HyperV or WSL2 enabled, which requires nested virtualization in a VM.

Docker is actually Docker Desktop on macOS

» Back to table of contents «

One problem on macOS that adds more confusion is that the application which you can install on macOS is called "Docker". When you open the launchpad, the label below the Docker icon is "Docker". Don't forget that you are still using Docker Desktop and the Docker daemon is running inside the virtual machine as well as all your containers.

Docker running directly on a physical machine

The source code of Moby

» Back to table of contents «

I shared the first commit of Docker's source code at the beginning of the "Docker as a software" section. The source code is on GitHub in the moby/moby repository. During the years since Docker was created, many repositories were used. "moby/moby" was originally called "dotcloud/docker" and the URL still works (http://github.com/dotcloud/docker). When the company was renamed, the repository was renamed again to "docker/docker" (https://github.com/dotcloud/docker), but both are redirected to Moby today. Moby is the base source code of the Docker variants we have today.

Docker Enterprise Edition

» Back to table of contents «

Years ago there was a Docker EE (Enterprise Edition), but it was bought by Mirantis, who made the Mirantis Container Runtime. You can even read it on their website:

Mirantis Container Runtime (formerly Docker Engine - Enterprise)

Since I'm focusing on Docker, this is all I wanted to say about an edition that does not exist today.

Docker IO

» Back to table of contents «

docker.io is based on Moby, but not supported by Docker, Inc. You could find this package on a Debian or Ubuntu Linux distribution. On Ubuntu, running apt info docker.io reveals that it is maintained by "Ubuntu Developers". The output of the command contains exactly

Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>

and the following on Debian

Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>

The Internet Archive's "Wayback machine" reveals that the earlier installation guide for Ubuntu started with installing docker.io, but recent guides start with removing docker.io.

I don't recommend using docker.io today as it will install an older version (at the time of writing this post, it is 24.0.7 on Ubuntu 22.04) and most of the tutorials will show you features based the official version and the official documentation.

Snap package

» Back to table of contents «

There is a snap package which is also based on Moby, but built by Canonical. I quote:

Authors

This snap is built by Canonical based on source code published by Docker, Inc. It is not endorsed or published by Docker, Inc.

So I don't recommend this variant either, unless the installation guide of the official version doesn't work on your distribution but Snap does. A snap is basically a kind of container with possible restrictions like you can only mount files from your home directory.

Docker CE

» Back to table of contents «

Docker CE is the Docker Community Edition based on Moby and officially supported by Docker, Inc. So when you want to install Docker on Linux without a virtual machine, this is what I recommend. This is also what you can install on a Windows server, but the installation is described by Microsoft.

Docker in Docker

» Back to table of contents «

The term "Docker in Docker" is misleading. You are not running Docker in a Docker. As I explained, the word Docker should be used for the Docker daemon, but you are not running a Docker daemon in a Docker daemon.

Then what is that you are doing when using Docker in Docker? You are running a Docker daemon inside a Docker container. Since calling it "Docker daemon in Docker container" would be long and wouldn't sound as good, we call it "Docker in Docker" in short.

If you want to try it, you can find Docker images on Docker Hub and you can use a Docker image downloaded (pulled) from Docker Hub to run a Docker container in which you run a Docker daemon to run isolated processes (containers) in an already isolated environment.

Docker CE components

Two main parts of Docker CE

» Back to table of contents «

As I pointed out earlier, you can run a Docker daemon in a virtual machine or on a remote machine and still keep the "docker" command on your local machine. This is possible because we can talk about two main parts of Docker CE. The client and the daemon. The client can be anywhere, but the daemon has to be where the containers will be running.

The client can instruct the Docker daemon through a TCP or Unix socket using the API service, but a "client" can mean multiple things.

Docker Client

The docker command

» Back to table of contents «

Often when we talk about the Docker client, we think of the "docker" command used in a terminal. For a beginner, who sees the name of the command, it could be confusing, but this is not the entire Docker, and it will not run "dockers", but Docker containers.

Docker SDK

» Back to table of contents «

The client can also be SDK, although the SDKs are for the core Docker and not for the plugins like Buildx.

Docker Compose

» Back to table of contents «

There is another plugin called Docker Compose. There was a Compose v1 originally written in Python for which we used the docker-compose command, but now we need to use docker compose without the dash.

Both versions require a so-called "compose file", but v1 required docker-compose.yml while the new version accept compose.yml as well. The other difference is that v1 required

version: XY

at the beginning of the file to specify which version of the compose syntax we wanted to use and the XY was a number like 3.8. So this was never the version of Docker Compose, but the version of the compose schema in the yaml file. For more information about the history of Docker Compose, you can visit the description in the official documentation.

Docker Daemon

The history of dockerd

» Back to table of contents «

The Docker daemon can be started by running dockerd. Until Docker 1.8.0, the daemon could be started with the docker command using the -d flag, but it was changed to docker daemon replacing the flag with a subcommand. Then that became deprecated too in Docker 1.13 after the dockerd binary was introduced in Docker 1.12.

Rootless vs rootful Docker

» Back to table of contents «

Originally the Docker daemon required a root user. Running it as a non-root user became possible in Docker CE 19.03 as an experimental feature which was already one of the core features of Podman (we will talk about it later) from the beginning. This is what we call "rootless mode", so the original mode became the "rootful mode".

Docker daemon dependencies

» Back to table of contents «

Originally, Docker was a single binary. Before introducing the "dockerd" command for the daemon, docker-containerd, docker-containerd-shim and docker-runc was introduced in Docker 1.11.

So once we had a single binary, then "Docker, Inc" started separating the functionalities into multiple binaries on Linux. That was the beginning the of dependencies and components we have today, except that these dependencies are now not limited to Docker. containerd can also be the container runtime of Kubernetes.

As the "docker" command is the client for "dockerd", nerdctl is the client for containerd, although the project was started only at the end of 2020.

Now we have dockerd which uses containerd, but containerd will not create containers directly. It needs a runtime and the default runtime is runc, but that can be changed. containerd actually doesn't have to know the parameters of the runtime. There is a shim process between containerd and runc, so containerd knows the parameters of the shim, and the shim knows the parameters of runc or other runtimes.

Podman pretending to be Docker

» Back to table of contents «

I admit the title of this section sounds worse than it is, but the fact is that sometimes when you install Podman, you can also have an alias called "docker" pointing to "podman". That can make you believe that you are running Docker and come to the Docker forum asking about an issue which is actually not related to Docker. The alias exists because Podman tries to keep a similar command line interface to the interface of Docker, so when someone relies on an existing docker command, they don't have to rewrite their scripts if they are lucky.

But again, Podman is not Docker and Podman Desktop is not Docker Desktop!

The docker version and docker info commands can give you an idea of what you are actually using.

Packages and versioning

» Back to table of contents «

In my mind the core components of Docker CE are the Docker daemon and the Docker client. When you install Docker CE on Linux, after you added the official APT repository provided by "Docker, Inc", the following packages could be installed:

docker-ce: The daemon
docker-ce-rootless-extras: Scripts to run rootless Docker
docker-ce-cli: The docker command

These packages have the same version. You can use a different version of the command line interface (cli), but if you want to make sure that the docker command is compatible with the API of the Docker daemon, use the same versions.

Everything else listed below has completely independent version numbers:

docker-compose (Compose v1)
Plugins
- docker-compose-plugin (Compose v2)
- docker-buildx
- docker-init (only in Docker Desktop)
Docker Desktop
Docker daemon dependencies
- containerd
- containerd-shim*
- runc

Conclusion

» Back to table of contents «

Docker has a long history, and it changed a lot during the years, but one thing was always true. You always ran containers and not dockers. If you say "I want to run multiple dockers" on the Docker forum, we will think that you want to run multiple Docker daemons. Well... since we are kind of trained by our users, we will eventually realize what you mean, but knowing the right terms helps you communicate your issue better and get an answer faster.

Before you ask a question, always think about what you are using and communicate that to the potential helpers.

And finally, you can find an important screenshot from the video below that shows all the variants and components I described in this post. Click on the image to download it in full size.

Install Docker and Portainer in a VM using Ansible

Ákos Takács — Sun, 02 Jun 2024 14:05:28 +0000

Introduction

This episode is actually why I started this series in the first place. I am an active Docker user and Docker fan, but I like containers and DevOps topics in general. I am a moderator on the official Docker forums and I see that people often struggle with the installation process of Docker CE or Docker Desktop. Docker Desktop starts a virtual machine, and the GUI is to manage the Docker CE inside the virtual machine even on Linux. Even though I prefer not to use a GUI for creating containers, I admit it can be useful in some situations, but you always need to be ready to use the command line where all the commands are available. In this episode I will use Ansible to install Docker CE in the previously created virtual machine, and I will also install a web-based graphical interface, Portainer.

If you want to be notified about other videos as well, you can subscribe to my YouTube channel: https://www.youtube.com/@akos.takacs

Before you begin
- Requirements
- Download the already written code of the previous episode
- Have an inventory file
- Activate the Python virtual environment
Small improvements before we start today's main topic
- Disable gathering facts automatically
- Create an inventory group for LXD playbooks
- Reload ZFS pools after removing LXD to make the playbook more stable
Add a host to a dynamically created inventory group
Use a dynamically created inventory group
Install Docker CE in a VM using Ansible
- Using 3rd-party roles to install Docker
- Default variables for the docker role
- Add docker to the Ansible roles
- Install the dependencies of Docker
- Configure the official APT repository
- Install a specific version of Docker CE
Allow non-root users to use the docker commands
Install Portainer CE, the web-based GUI for containers
Conclusion

Before you begin

Requirements

» Back to table of contents «

The project requires Nix which we discussed in Install Ansible 8 on Ubuntu 20.04 LTS using Nix
You will also need an Ubuntu remote server. I recommend an Ubuntu 22.04 virtual machine.

Download the already written code of the previous episode

» Back to table of contents «

If you started the tutorial with this episode, clone the project from GitHub:



git clone https://github.com/rimelek/homelab.git
cd homelab

If you cloned the project now, or you want to make sure you are using the exact same code I did, switch to the previous episode in a new branch



git checkout -b tutorial.episode.8b tutorial.episode.9.1

Have the inventory file

» Back to table of contents «

Copy the inventory template



cp inventory-example.yml inventory.yml

Change ansible_host to the IP address of your Ubuntu server that you use for this tutorial,
and change ansible_user to the username on the remote server that Ansible can use to log in.
If you still don't have an SSH private key, read the Generate an SSH key part of Ansible playbook and SSH keys
If you want to run the playbook called playbook-lxd-install.yml, you will need to configure a physical or virtual disk which I wrote about in The simplest way to install LXD using Ansible. If you don't have a usable physical disk, Look for truncate -s 50G <PATH>/lxd-default.img to create a virtual disk.
You will need an encrypted secret file which I wrote about in the Encrypt a file section of "Use SOPS in Ansible ro read your secrets".

Activate the Python virtual environment

» Back to table of contents «

How you activate the virtual environment, depends on how you created it. In the episode of The first Ansible playbook describes the way to create and activate the virtual environment using the "venv" Python module and in the episode of The first Ansible role we created helper scripts as well, so if you haven't created it yet, you can create the environment by running



./create-nix-env.sh venv

Optionally start an ssh agent:



ssh-agent $SHELL

and activate the environment with



source homelab-env.sh

Small improvements before we start today's main topic

You can skip this part too, if you joined the tutorial at this episode, and you don't want to improve other playbooks.

Disable gathering facts automatically

» Back to table of contents «

We discussed facts before in the "Using facts and the GitHub API in Ansible" episode, but we left this setting on default in other playbooks. Let's quickly add gather_facts: false to all playbooks, except playbook-hello.yml as that was to demonstrate how a playbook runs.

Create an inventory group for LXD playbooks

» Back to table of contents «

Now that we have a separate group for the virtual machines that run Docker, we can also create a new group for LXD, so when we add more machines, we will not install LXD on every single machine, and we will not remove it from a machine on which it was not installed. Let's add the following to the inventory.yml



lxd_host_machines:
  hosts:
    YOURHOSTNAME:

NOTE: Replace YOURHOSTNAME with your actual hostname which you used in the inventory under the special group called "all".

In my case, it is the following:



lxd_host_machines:
  hosts:
    ta-lxlt:

And now, replace hosts: all in playbook-lxd-install.yml and playbook-lxd-remove.yml with hosts: lxd_host_machines.

Reload ZFS pools after removing LXD to make the playbook more stable

» Back to table of contents «

When I wrote the "Remove LXD using Ansible" episode, the playbook worked for me every time. Since then, I noticed that sometimes it cannot delete the ZFS pool, because it's missing. I couldn't actually figure out why it happens, but a workaround can be implemented to make the playbook more stable. We have to restart the zfs-import-cache Systemd service, which will reload the ZFS pools so the next task can delete it and the disks can be wiped as well.

Open roles/zfs_destroy_pool/tasks/main.yml and look for the following task:



- name: Get zpool facts
  ignore_errors: true
  community.general.zpool_facts:
    name: "{{ zfs_destroy_pool_name }}"
  register: _zpool_facts_task

All we have to do is add a new task before it:



# To fix the issue of missing ZFS pool after uninstalling LXD
- name: Restart ZFS import cache
  become: true
  ansible.builtin.systemd:
    state: restarted
    name: zfs-import-cache

The built-in systemd module can restart a Systemd service if the "state" is "restarted".

Add a host to a dynamically created inventory group

» Back to table of contents «

Since we already configured our SSH client for the newly created virtual machine last time, we could create a new inventory group and add the virtual machine to that group. But sometimes you don't want to do that, or you can't. That's why I wanted to show you a way to create a new inventory group without changing the inventory file. Then we can add a host to this group. Since last time we also used Ansible to get the IP address of the new virtual machine, we can add that IP to the inventory group. The next task will show you how you can do that.



    # region task: Add Docker VM to Ansible inventory
    - name: Add Docker VM to Ansible inventory
      changed_when: false
      ansible.builtin.add_host:
        groups: _lxd_docker_vm
        hostname: "{{ vm_inventory_hostname }}"
        ansible_user: "{{ config_lxd_docker_vm_user }}"
        ansible_become_pass: "{{ config_lxd_docker_vm_pass }}"
        # ansible_host is not necessary, inventory hostname will be used
        ansible_ssh_private_key_file: "{{ vm_ssh_priv_key }}"
        #ansible_ssh_common_args: "-o StrictHostKeyChecking=no"
        ansible_ssh_host_key_checking: false
    # endregion

You need to add the above task to the "Create the VM" play in the playbook-lxd-docker-vm.yml playbook. The builtin add_host module is what we needed. Despite what you see in the task, it has only 2 parameters. Everything else is a variable which you could use in the inventory file as well. The groups and name parameters have aliases, and I thought that using the hostname alias of name would be better as we indeed add a hostname or an IP as its value. groups can be a list or a string. I defined it as a string as I have only one group to which I will add the VM.

I mentioned before that I like to start the name of helper variables with an underscore. The name of the group is not a variable, but I start it with an underscore, so I will know it is a temporary, dynamically created inventory group. We have some variables that we defined in the previous episode.

vm_inventory_hostname: The inventory hostname of the VM, which is also used in the SSH client configuration. It actually comes from the value of config_lxd_docker_vm_inventory_hostname with a default in case it is not defined.
config_lxd_docker_vm_user: The user that we created using cloud init and with which we can SSH into the VM.
config_lxd_docker_vm_pass: The sudo password of the user. It comes from a secret.
vm_ssh_priv_key: The path of the SSH private key used for the SSH connection. It is just a short alias for config_lxd_docker_vm_ssh_priv_key which can be defined in the inventory file.

Using these variables we could configure the SSH connection parameters like ansible_user, ansible_become_pass and ansible_ssh_private_key_file. We also have a new variable, ansible_ssh_host_key_checking. When we first SSH to a remote server, we need to accept the fingerprint of the server's SSH host key, which means we know and trust the server. Since we dynamically created this virtual machine and detected its IP address, we would need to accept it every time we recreate our VM, so I just disable host key checking by setting boolean false as value.

Use a dynamically created inventory group

» Back to table of contents «

We have a new inventory group, but we still don't use it. Now we need a new play in the paybook which will actually use this group. Add the following play skeleton to the end of playbook-lxd-docker-vm.yml.



# play: Configure the OS in the VM
- name: Configure the OS in the VM
  hosts: _lxd_docker_vm
  gather_facts: false
  pre_tasks:
  roles:
# endregion

Even though we forced Ansible to wait until the virtual machine gets an IP address, having an IP address doesn't mean the SSH daemon is ready in the VM. So we need the following pre task:



    - name: Waiting for SSH connection
      ansible.builtin.wait_for_connection:
        timeout: 20
        delay: 0
        sleep: 3
        connect_timeout: 2

The built-in wait_for_connection module can be used to retry connecting to the servers. We start checking it immediately, so we set the delay to 0. If we are lucky, it will be ready right away. If it does not connect in 2 seconds (connect_timeout), Ansible will "sleep" for 3 seconds and try again. If the connection is not made in 20 seconds (timeout), the task will fail.

While I was testing the almost finished playbook, I realized that sometimes the installation of some packages failed like if they were not in the APT cache yet, so I added a new pre task to update the APT cache before we start the VM. We already discussed this module in Using facts and the GitHub API in Ansible



    - name: Update APT cache
      become: true
      changed_when: false
      ansible.builtin.apt:
        update_cache: true

If you don't want to add it, you can just rerun the playbook and it will probably work. Now we have an already written Ansible role which we will want to use here too, which is cli_tools. So this is how our second play looks like in playbook-lxd-docker-vm.yml:



# play: Configure the OS in the VM
- name: Configure the OS in the VM
  hosts: _lxd_docker_vm
  gather_facts: false
  pre_tasks:
    - name: Waiting for SSH connection
      ansible.builtin.wait_for_connection:
        timeout: 20
        delay: 0
        sleep: 3
        connect_timeout: 2
    - name: Update APT cache
      become: true
      changed_when: false
      ansible.builtin.apt:
        update_cache: true
  roles:
    - role: cli_tools
# endregion

Now you could delete the virtual machine if you already created it last time, and run the playbook with the new play again to create the virtual machine and immediately install the command line tools in it.

Install Docker CE in a VM using Ansible

Using 3rd-party roles to install Docker

» Back to table of contents «

I admit, that there are already existing great roles we could use to Install Docker. For example the one made by Jeff Geerling which supports multiple Linux distributions, so feel free to use it, but we are still practicing writing our own roles, so I made a simple one for you, although that works only on Ubuntu. On the other hand, I will add something that even Jeff Geerling didn't do.

Default variables for the docker role

» Back to table of contents «

We will create some default variables in roles/docker/defaults/main.yml:



docker_version: "*.*.*"
docker_sudo_users: []

The first, docker_version is to define which version we want to install. When you just start playing with Docker but don't want to use Play with Docker, you probably want to install the latest version. That's why the default value is "*.*.*", which means the latest major, minor and patch version of Docker CE. You will see the implementation soon. The second variable is docker_sudo_users, which is an empty list. We will be able to add users to the list who should be able to use Docker. We will discuss it later in more details.

Add docker to the Ansible roles

» Back to table of contents «

Before we continue, let's add "docker" as a new role to our second play in playbook-lxd-docker-vm.yml:



  # ...
  roles:
    - role: cli_tools
    - role: docker
      docker_sudo_users: "{{ config_lxd_docker_vm_docker_sudo_users | default([]) }}"

You know very well now that this way we can add this new config variable to the inventory.yml



all:
  vars:
    # ...
    config_lxd_docker_vm_user: manager
    config_lxd_docker_vm_docker_sudo_users:
      - "{{ config_lxd_docker_vm_user }}"

Note that config_lxd_docker_vm_user is probably already defined if you followed the previous episodes as well.

Install the dependencies of Docker

» Back to table of contents «

As I say it very often we should always start with the official documentation. It starts with uninstalling old and unofficial packages. Our role will not include it, so you will do it manually or write your own role as a homework. Then it updates the APT cache, which we just did as a pre task, so we will install the dependencies first. The official documentation says:



sudo apt-get install ca-certificates curl

Which looks like this in Ansible in roles/docker/tasks/main.yml:



- name: Install dependencies
  become: true
  ansible.builtin.apt:
    name:
      - ca-certificates
      - curl

Note: We know that the "cli_tools" role already installed curl, but we don't care, because when we create a role, we try to make it without depending on other roles. So even if we decide later not to use the "cli_tools" role, our "docker" role will still work perfectly.

Configure the official APT repository

» Back to table of contents «

The official documentation continues with creating a folder, /etc/apt/keyrings. It uses the



sudo install -m 0755 -d /etc/apt/keyrings

command, but it really just creates a folder this time, which looks like this in Ansible:




- name: Make sure the folder of the keyrings exists
  become: true
  ansible.builtin.file:
    state: directory
    mode: 0755
    path: /etc/apt/keyrings

The next step is downloading the APT key for the repository. Previously, the documentation used the apt-key command which was deprecated on Ubuntu, so it was replaced with the following:



sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Which looks like this in Ansible:



- name: Install the APT key of Docker's APT repo
  become: true
  ansible.builtin.get_url:
    url: https://download.docker.com/linux/ubuntu/gpg
    dest: /etc/apt/keyrings/docker.asc
    mode: a+r

Then the official documentation shows how you can add the repository to APT depending on the CPU architecture and Ubuntu release code name, like this:



# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

In the "Using facts and the GitHub API in Ansible" episode we already learned to get the architecture. We also need the release code name, so we gather the distribution_release subset of Ansible facts as well.



- name: Get distribution release fact
  ansible.builtin.setup:
    gather_subset:
      - distribution_release
      - architecture

Before we continue with the next task, we will add some variables to roles/docker/vars/main.yml.



docker_archs:
  x86_64:  amd64
  amd64:   amd64
  aarch64: arm64
  arm64:   arm64
docker_arch: "{{ docker_archs[ansible_facts.architecture] }}"
docker_distribution_release: "{{ ansible_facts.distribution_release }}"

Again, this is very similar to what we have done before to separate our helper variables from the tasks, and now we can generate docker.list under /etc/apt/sources.list.d/. To do that, we add the new task in roles/docker/tasks/main.yml:



- name: Add Docker APT repository
  become: true
  ansible.builtin.apt_repository:
    filename: docker
    repo: "deb [arch={{ docker_arch }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu {{ docker_distribution_release }} stable"
    state: present
    update_cache: true

The built-in apt_repository module can also update APT cache after adding the new repo. The filename is automatically generated if we don't set it, but the "filename" parameter is actually a name without the extension, so do not add .list at the end of the name.

Install a specific version of Docker CE

» Back to table of contents «

The official documentation recommends using the following command to list the available versions of Docker CE on Ubuntu.



apt-cache madison docker-ce | awk '{ print $3 }'

It shows the full package name which includes more than just the version of Docker CE. Fortunately, the actual version number can be parsed like this:



apt-cache madison docker-ce \
  | awk '$3 ~ /^([0-9]+:)([0-9]+\.[0-9]+\.[0-9]+)(-[0-9]+)?(~.*)$/ {print $3}'

Where the version number is the second expression in parentheses.



[0-9]+\.[0-9]+\.[0-9]+

We can replace it with an actual version number, but keeping the backslashes:



26\.1\.3

Let's search for only that version:



apt-cache madison docker-ce \
  | awk '$3 ~ /^([0-9]+:)26\.1\.3(-[0-9]+)?(~.*)$/ {print $3}'

Output:



5:26.1.3-1~ubuntu.22.04~jammy

This is what we will implement in Ansible:



- name: Get full package version for {{ docker_version }}
  changed_when: false
  ansible.builtin.shell: |
    apt-cache madison docker-ce \
    | awk '$3 ~ /^([0-9]+:){{ docker_version | replace('*', '[0-9]+') | replace('.', '\.') }}(-[0-9]+)?(~.*)$/ {print $3}'
  register: _docker_versions_command

I hope now it starts to make sense why the default value of docker_version was *.*.*. It was because we replace that with regular expressions. We also escape all dots as otherwise it would mean "any character" in the regular expression. This solution allows us to install the latest version unless we override the default value with an actual version number. Even if we override it, we can use a version like 26.0.* to get a list of available patch version of Docker CE 26.0 instead of the latest major version. Of course, this is still a list of versions unless we set a specific version number, but we can get the first line in the next task. According to the official documentation, we would install Docker CE and related packages like this:



VERSION_STRING=5:26.1.0-1~ubuntu.24.04~noble
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

Let's do it in Ansible:



- name: Install Docker CE
  become: true
  vars:
    _full_version: "{{ _docker_versions_command.stdout_lines[0] }}"
  ansible.builtin.apt:
    name:
      - docker-ce={{ _full_version }}
      - docker-ce-cli={{ _full_version }}
      - docker-ce-rootless-extras={{ _full_version }}
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin

What is not mentioned in the documentation is marking Docker CE packages as held. In the terminal it would be like this:



apt-mark hold docker-ce docker-ce-cli docker-ce-rootless-extras containerd.io docker-compose-plugin

It will be almost the same in Ansible as we need to use the built-in command module:



- name: Hold Docker CE packages
  become: true
  ansible.builtin.command: apt-mark hold docker-ce docker-ce-cli docker-ce-rootless-extras containerd.io docker-compose-plugin
  changed_when: false

You can check the list of held packages:



apt-mark showheld

Output:



containerd.io
docker-ce
docker-ce-cli
docker-ce-rootless-extras
docker-compose-plugin

Note: I never actually saw an upgraded containerd causing problems, but this is a very important component of Docker CE, so I decided to hold that too. If it causes any problem, you can "unhold" that any time by running the following command:



apt-mark unhold containerd.io

Allow non-root users to use the docker commands

» Back to table of contents «

This is the part where I don't follow the documentation. The official documentation mentions this on the Linux post-installation steps for Docker Engine page:

The Docker daemon binds to a Unix socket, not a TCP port. By default it's the root user that owns the Unix socket, and other users can only access it using sudo. The Docker daemon always runs as the root user.

If you don't want to preface the docker command with sudo, create a Unix group called docker and add users to it. When the Docker daemon starts, it creates a Unix socket accessible by members of the docker group. On some Linux distributions, the system automatically creates this group when installing Docker Engine using a package manager. In that case, there is no need for you to manually create the group.

Of course, using the docker group is not really secure, which is also mentioned right after the previous quote in the documentation:

The docker group grants root-level privileges to the user. For details on how this impacts security in your system, see Docker Daemon Attack Surface.

If we want to have just a little bit more secure solution, we don't use the docker group which can directly access the docker socket, but we create another group like docker-sudo and we allow all users in this group to run the docker command as root by using sudo docker without a password. It would involve creating a new rule in /etc/sudoers.d/docker like:



%docker-sudo ALL=(root) NOPASSWD: /usr/bin/docker

This would force users to always run sudo docker, not just docker and they will often forget it and get an error message. We could add an alias like



alias docker='sudo \docker'

to ~/.bash_aliases, but that would work only when the user uses the bash shell. Instead of that, we can add a new script in /usr/local/bin/docker,
which usually overrides /usr/bin/docker and add this command in the script:



#!/usr/bin/env sh

exec sudo /usr/bin/docker "$@"

This is what we will do with Ansible, so our new script will be executed which will call the original docker command as an argument of sudo. Now even when we use Visual Studio Code's Remote Explorer to connect to the remote virtual machine and use Docker in the VM from VSCode, /var/log/auth.log on Debian-based systems will contain exactly what docker commands were executed. If you don't find this file, it may be called /var/log/secure on your system. This is for example how browsing files from VSCode in containers looks like:



May 19 20:14:36 docker sudo:  manager : PWD=/home/manager ; USER=root ; COMMAND=/usr/bin/docker container exec --interactive d71cae80db867ee79ba66fa947ab126ac6f7b0e482ebb8b3320d9f3bfa3fb3e6 /bin/sh -c 'stat -c \'%f %h %g %u %s %X %Y %Z %n\' "/"* || true && stat -c \'%f %h %g %u %s %X %Y %Z %n\' "/".*'

This is useful when you want to be able to investigate accidental damage when a Docker user executes a command that they should not have executed, and they don't even know what they executed. It will not protect you from intentional harm as an actual hacker could also delete the logs. On the other hand, if you have a remote logging server where you collect logs from all machines, you will probably have the logs to figure out what happened.

Now let's configure this in Ansible.

First we will create the docker-sudo group:



- name: Ensure group "docker-sudo" exists
  become: true
  ansible.builtin.group:
    name: docker-sudo
    state: present

Now we can finally use our docker_sudo_users variable which we defined in roles/docker/defaults/main.yml and check if there is any user who doesn't exist.



- name: Check if docker sudo users are existing users
  become: true
  ansible.builtin.getent:
    database: passwd
    key: "{{ item }}"
  loop: "{{ docker_sudo_users }}"

This built-in getent module basically calls the getent command on Linux:



getent passwd manager

If the user exists, it returns the passwd record of the user, and it fails otherwise. Now let's add the groups to the users:



- name: Add users to the docker-sudo group
  become: true
  ansible.builtin.user:
    name: "{{ item }}"
    # users must be added to docker-sudo group without removing them from other groups
    append: true
    groups:
      - docker-sudo
  loop: "{{ docker_sudo_users }}"

We used the built-in user module, to add the docker-sudo group to the users defined in docker_sudo_users. We are close to the end. The next step is creating the script using a similar solution we used in the hello_world role at the beginning of the series.



- name: Create a sudo wrapper for Docker
  become: true
  ansible.builtin.copy:
    content: |
      #!/usr/bin/env sh

      exec sudo /usr/bin/docker "$@"
    dest: /usr/local/bin/docker
    mode: 0755

And finally, using the same method, we create the sudoers rule:



- name: Allow run execute /usr/bin/docker as root without password
  become: true
  ansible.builtin.copy:
    content: |
      %docker-sudo ALL=(root) NOPASSWD: /usr/bin/docker
    dest: /etc/sudoers.d/docker

We could now run the playbook to install Docker in the virtual machine:



./run.sh playbook-lxd-docker-vm.yml

Install Portainer CE, the web-based GUI for containers

» Back to table of contents «

Installing Portainer CE is the easy part, actually. We will need to use the built-in pip module to install a Python dependency for Ansible to be able to manage Docker, and then we will also use a community module, called docker_container. Let's create the tasks file first at roles/portainer/tasks/main.yml:



- name: Install Python requirements
  become: true
  ansible.builtin.pip:
    name: docker

- name: Install portainer
  become: true
  community.docker.docker_container:
    name: "{{ portainer_name }}"
    state: started
    container_default_behavior: no_defaults
    image: "{{ portainer_image }}"
    restart_policy: always
    ports:
      - "{{ portainer_external_port }}:9443"
    volumes:
      - "{{ portainer_volume_name }}:/data"
      - /var/run/docker.sock:/var/run/docker.sock

After defining the variables, this will be basically equivalent of running the following in shell:



docker run \
  -p 9443:9443 \
  --name portainer \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:2.20.2-alpine

Let's add our defaults at roles/portainer/defaults/main.yml:



portainer_name: portainer
portainer_image: portainer/portainer-ce:2.20.2-alpine
portainer_external_port: 9443
portainer_volume_name: "{{ portainer_name }}_data"

That's it, and now we add the role to the playbook:



  # ...
  roles:
    - role: cli_tools
    - role: docker
      docker_sudo_users: "{{ config_lxd_docker_vm_docker_sudo_users | default([]) }}"
    - role: portainer

When you finished installing Portainer, you need to quickly open it in a web browser on port 9443. If you do it on a publicly available machine in your LAN network, you can simply open it like https://192.168.4.58:9443. In this tutorial, our virtual machine needs an SSH tunnel like below, so you can use https://127.0.0.1:9443:



ssh -L 9443:127.0.0.1:9443 -N docker.lxd.ta-lxlt

Your hostname will be different. If you already have other containers with a forwarded port, you can add more ports to the tunnel:



ssh \
  -L 9443:127.0.0.1:9443 \
  -L 32768:127.0.0.1:32768 \
  -N \
  docker.lxd.ta-lxlt

If you have containers without forwarded ports from the host, you can forward your local port directly to the container IP.



ssh \
  -L 9443:127.0.0.1:9443 \
  -L 32768:127.0.0.1:32768 \
  -L 8080:172.17.0.4:80 \
  -N \
  docker.lxd.ta-lxlt

When you can finally open Portainer in your browser, create your first user and configure the connection to the local Docker environment. If you wait too long, the webinterface will show an error message, and you will need to go to the terminal in the virtual machine and restart portainer:



docker restart portainer

After that, you can start the configuration. This way if you install Portainer on a publicly available server, there will be less time for others to log in before you do, and after you initialized Portainer, it is no longer possible to log in without a password.

Conclusion

» Back to table of contents «

I hope this episode helped you to install Docker and allow non-root users to use the Docker commands in a little more secure way than the documentation suggests. Now you can have a web-basd graphical interface for containers, however, Portainer is definitely not Docker Desktop, so you will not have the extra features like Docker Desktop extensions. Using Ansible can help to deploy your entire dev environment, destroy it and recreate any time. Using containers you can have pre-built and pre-configured applications that you can try and learn more about it to customize the configuration for your needs. When you have a production environment, you need to focus much more on security, but now that you have the tools to begin with a dev environment, you can make that new step more easily.

The final source code of this episode can be found on GitHub:

https://github.com/rimelek/homelab/tree/tutorial.episode.10

rimelek / homelab

Source code to create a home lab. Part of a video tutorial

README

This project was created to help you build your own home lab where you can test your applications and configurations without breaking your workstation, so you can learn on cheap devices without paying for more expensive cloud services.

The project contains code written for the tutorial, but you can also use parts of it if you refer to this repository.

Tutorial on YouTube in English: https://www.youtube.com/watch?v=K9grKS335Mo&list=PLzMwEMzC_9o7VN1qlfh-avKsgmiU8Jofv

Tutorial on YouTube in Hungarian: https://www.youtube.com/watch?v=dmg7lYsj374&list=PLUHwLCacitP4DU2v_DEHQI0U2tQg0a421

Note: The inventory.yml file is not shared since that depends on the actual environment so it will be different for everyone. If you want to learn more about the inventory file watch the videos on YouTube or read the written version on https://dev.to. Links in the video descriptions on YouTube.

You can also find an example inventory file in the project root. You can copy that and change the content, so you will use your IP…

View on GitHub

Use Ansible to create and start LXD virtual machines

Ákos Takács — Mon, 11 Mar 2024 22:46:31 +0000

Introduction

The next step in our home lab is to finally have an Ansible playbook to create and start a virtual machine. It also means that eventually we will have different kind of dynamically created servers into which we probably want to SSH. In this chapter I will show you a way to create a LXD virtual machine on Ubuntu and also automatically configure your host to be able to SSH into the new virtual machine. In this capter we will assume that we still don't want to manage these virtual machines from Ansible, but that will be the final goal.

Note: In the original post and video I used a different LXD remote image repository. I had to change it, because that repository was replaced surprisingly in an active LTS LXD release. That's because of the original developers of LXD started Incus and they stopped supporting LXD in their image repository. There is a new repo, which does not include Ubuntu server images currently, only desktop.

Before you begin
- Requirements
- Download the already written code of the previous episode
- Have an inventory file
- Activate the Python virtual environment
Preparing the new inventory
Check how Ansible interprets the inventory file
- Wrapper script to run any command in the Nix environment
- Use the ansible-inventory command
- Get the value of a single variable
New playbook for creating a VM
Start a virtual machine from Ansible
- Common parameters without variables for starting a VM
- Dynamically set parameters for starting a VM
  - VM name and resource limits
  - Cloud-init config for SSH access and sudo password
- Run the playbook to start the VM
Get the IP address of a virtual machine
Accessing the virtual machine from the Ansible controller
- Why don't we use LAN network
- LXD proxy
- SSH tunnel
- SSH jump servers
- Using separate config files for different kind-of-servers
Use Ansible to configure the SSH client
- Make sure config.d exists
- Get information about the VM
- Get the IP addresses used by the VM
- Get information about the LXD network
- Get the usable IP address assigned to eth0
- Generate the new SSH client config for the VM
- Run the playbook to generate the SSH client config
- Access a webservice on the crated VM
Conclusion

Before you begin

Requirements

» Back to table of contents «

The project requires Nix which we discussed in Install Ansible 8 on Ubuntu 20.04 LTS using Nix
You will also need an Ubuntu remote server. I recommend an Ubuntu 22.04 virtual machine.

Download the already written code of the previous episode

» Back to table of contents «

If you started the tutorial with this episode, clone the project from GitHub:

git clone https://github.com/rimelek/homelab.git
cd homelab

If you cloned the project now, or you want to make sure you are using the exact same code I did, switch to the previous episode in a new branch

git checkout -b tutorial.episode.8b tutorial.episode.8

Have the inventory file

» Back to table of contents «

Copy the inventory template

cp inventory-example.yml inventory.yml

Change ansible_host to the IP address of your Ubuntu server that you use for this tutorial,
and change ansible_user to the username on the remote server that Ansible can use to log in.
If you still don't have an SSH private key, read the Generate an SSH key part of Ansible playbook and SSH keys
If you want to run the playbook called playbook-lxd-install.yml, you will need to configure a physical or virtual disk which I wrote about in The simplest way to install LXD using Ansible. If you don't have a usable physical disk, Look for truncate -s 50G <PATH>/lxd-default.img to create a virtual disk.
You will need an encrypted secret file which I wrote about in the Encrypt a file section of "Use SOPS in Ansible ro read your secrets".

Activate the Python virtual environment

» Back to table of contents «

./create-nix-env.sh venv

Optionally start an ssh agent:

ssh-agent $SHELL

and activate the environment with

source homelab-env.sh

Preparing the new inventory

» Back to table of contents «

Although in this tutorial I used only one server, you could add multiple servers to the inventory file, but you probably don't want to create a specific VM on each server.

You could have a playbook in which you define a specific host instead of an inventory group. Not a very good way in terms of redundancy.
You could define a list under the vars: section of a specific host in the inventory file, in which all the items would be a mapping of key-value pairs with the configurations for the virtual machine you want to create on that host and handle that list in a loop in the Ansible role responsible for creating the VM. That's much better, but too complicated for this tutorial.

For now, I will create a playbook for creating a general virtual machine which allows using Docker, so we could install it on multiple hosts. Just imagine that you want to be able to use Docker on all of your Linux hosts in a VM to test network connections between them. I know, it is hard to imagine that, so we will need a list of servers which could include a single server, so you will run Docker in a VM only on one machine.

It's time to change our inventory file to use new inventory groups in addition to the special group called "all". As a reminder, see my old inventory file:

all:
  vars:
    ansible_user: ansible-homelab
    config_apt_update: true
    config_lxd_zfs_pool_disks:
      - /dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
  hosts:
    ta-lxlt:
      ansible_host: 192.168.4.58
      ansible_ssh_private_key_file: ~/.ssh/ansible
      ansible_become_pass: "{{ sops.become_pass }}"

In the new inventory file, we will have an additional group:

docker_vm_host_machines:
  hosts:
    ta-lxlt:

That's it, defined in the end of the previous inventory file. Since we have all the parameters for the host in the group "all", we don't have to define any in the new group, but we still have to add the name of the host machine without values.

When we create our new playbook, in the "hosts:" section we will refer to "docker_vm_host_machines" instead of "all"

Check how Ansible interprets the inventory file

Wrapper script to run any command in the Nix environment

» Back to table of contents «

When I started this tutorial series, I thought we would need
only the ansible-playbook command. I was wrong. Now we should try the ansible-inventory command and also run an ad-hoc Ansible command. Instead of creating two more wrapper scripts only for the new commands, I will create a script called nix.sh which can run any.

nix.sh

#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p sops
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz

source config.sh

"$@"

I could use this script from all the other wrapper scripts to reduce redundancy, but I don't want to change many files in this chapter, so let's keep it for another day.

Make the script executable:

chmod +x nix.sh

And run the following command to get the version of Ansible:

./nix.sh ansible --version

ansible [core 2.15.0]
  config file = None
  configured module search path = ['/Users/ta/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/ta/Data/projects/myprojects/tutorials/iac/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/ta/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/ta/Data/projects/myprojects/tutorials/iac/venv/bin/ansible
  python version = 3.11.5 (main, Oct 15 2023, 20:58:46) [Clang 11.1.0 ] (/Users/ta/Data/projects/myprojects/tutorials/iac/venv/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

Use the ansible-inventory command

» Back to table of contents «

The following command would normally show you the structure of the inventory file as Ansible can see it:

ansible-inventory -i inventory.yml --graph

Since we installed Ansible in a Nix environment, we will need to use the previously created wrapper script:

./nix.sh ansible-inventory -i inventory.yml --graph

My output:

@all:
  |--@ungrouped:
  |--@docker_vm_host_machines:
  |  |--ta-lxlt

Since "all" includes the new group, any variables we define in that group, will be available in the others.

Get the value of a single variable

» Back to table of contents «

Using the "ansible" command, we can use the "debug" module to get the value of ansible_host. We will tell Ansible to run task in the group called "docker_vm_host_machines", but it will get the parameter defined in "all".

./nix.sh ansible docker_vm_host_machines -i inventory.yml -m debug -a var=ansible_host

My output:

ta-lxlt | SUCCESS => {
    "ansible_host": "192.168.4.58"
}

New playbook for creating a VM

» Back to table of contents «

Let's create the skeleton of the playbook and call the file "playbook-lxd-docker-vm.yml in the project root.".

# region play: Create the VM
- name: Create the VM
  hosts: docker_vm_host_machines
  pre_tasks:
  vars:
  tasks:
# endregion

I used a special comment syntax which is supported in JetBrains IDEs and also in Visual Studio code. These are what I use most of the time. If it doesn't work for you in VSCode by default, you can try the extension called region folding for VSCode. This syntax allows us to collapse different regions of the code and assign a name to it which will be shown in the collapsed state. It will be useful when we will have multiple plays, and we want to see the name when the play is collapsed instead of something like <4 keys>. I will use this for tasks as well.

You can see the new inventory group set for the play.

Start a virtual machine from Ansible

» Back to table of contents «

Now we will need an Ansible role to create and start a virtual machine. Since LXD was originally for containers, the module still called community.general.lxd_container. Let's see the skeleton of the task in the tasks section of the playbook containing only the static parameters.

Common parameters without variables for starting a VM

» Back to table of contents «

  tasks:

    # region task: Start Docker VM
    - name: Start Docker VM
      become: true
      community.general.lxd_container:
        url: unix:/var/snap/lxd/common/lxd/unix.socket
        type: virtual-machine
        state: started
        wait_for_ipv4_addresses: true
        config:
          boot.autostart: "false"
    # endregion

We need to be able to communicate with the LXD daemon, so we define the path of the LXD unix socket file in the url parameter.
We want a virtual machine, so we set the type to virtual-machine.
We want the task return only when the virtual machine got an IP address already, so we can continue with other tasks that require that IP address.
This is a test virtual machine, so we don't want it to start when the host starts, so boot.autostart should be false in the config section.
When we run the playbook, we don't want to start the VM manually, so we set the state to started.

Of course, we want set a source. A source is a collection of parameters containing how and from where LXD should download the base image for the virtual machine.

    - name: Start Docker VM
      become: true
      community.general.lxd_container:
        # ...
        source:
          type: image
          server: https://cloud-images.ubuntu.com/releases
          protocol: simplestreams
          alias: "22.04"

So as you see above, we will use the cloud variant of the Ubuntu 22.04 image from the specified server using the simplestream protocol.

Dynamically set parameters for starting a VM

VM name and resource limits

» Back to table of contents «

Our VM will need a name, and we want to limit the number of CPUs and the memory.

    - name: Start Docker VM
      become: true
      community.general.lxd_container:
        # ...
        name: "{{ vm_name }}"
        # ...
        config:
          # ...
          limits.cpu: "{{ vm_cpu }}"
          limits.memory: "{{ vm_memory }}"

The value of these parameters will come from the inventory file, but I wanted to keep this task short, so we will set the variables in the "vars" section of the play:

# region play: Create the VM
- name: Create the VM
  hosts: docker_vm_host_machines
  vars:
    vm_name: "{{ config_lxd_docker_vm_name | default('docker', true) }}"
    vm_memory: "{{ config_lxd_docker_vm_memory | default('4GiB', true)}}"
    vm_cpu: "{{ config_lxd_docker_vm_cpus | default(4, true) }}"
  pre_tasks:
  tasks:
# endregion

This way we have default values even if the config parameters are not defined in the inventory file or when defined with empty values (that is what "true" for as second parameter). I defined only the name of the virtual machine in the inventory and let Ansible use the default CPU and memory limits. Now we got to another interesting part.

Cloud-init config for SSH access and sudo password

» Back to table of contents «

We can create a virtual machine with the already described parameters, but without cloud-init configs we have no users in the virtual machine, not to mention the SSH configuration for that user. We want the following cloud-init config:

#cloud-config
users:
  - name: {{ vm_user }}
    lock_passwd: false
    groups: sudo
    shell: /bin/bash
    passwd: "{{ vm_pass }}"
    ssh_authorized_keys:
      - {{ lookup('file', vm_ssh_pub_key) }}

packages:
  - openssh-server

First of all we needed to install the openssh-server package and then define a list of users. There will be only one user by default and that user will be in the "sudo" group. The user's default shell will be the bash shell, and we define a password and a list of SSH public keys. The password comes from the vm_pass variable, and we will need the "file" lookup plugin to read the public key defined in the vm_ssh_pub_key variable. Let's add it to the task:

    - name: Start Docker VM
      become: true
      community.general.lxd_container:
        # ...
        config:
          # ...
          cloud-init.user-data: |
            #cloud-config
            users:
              - name: {{ vm_user }}
                lock_passwd: false
                groups: sudo
                shell: /bin/bash
                passwd: "{{ vm_pass }}"
                ssh_authorized_keys:
                  - {{ lookup('file', vm_ssh_pub_key) }}

            packages:
              - openssh-server

And set the variables:

# region play: Create the VM
- name: Create the VM
  hosts: docker_vm_host_machines
  vars:
    # ...
    vm_user: "{{ config_lxd_docker_vm_user | default('manager', true) }}"
    vm_pass: "{{ config_lxd_docker_vm_pass | ansible.builtin.password_hash(salt=vm_pass_salt)  }}"
    vm_ssh_pub_key: "{{ config_lxd_docker_vm_ssh_pub_key | default(vm_ssh_priv_key + '.pub', true) }}"
  pre_tasks:
  tasks:
# endregion

The default user will be "manager" and there will be no default password, but we need to use ansible.builtin.password_hash to convert the plain text password to a hash. To use this filter, you need to add the following line to the requirements.txt:

passlib==1.7.4 # for using ansible.builtin.password_hash()

Don't forget to run pip install -r requirements.txt to install the new library.

Alternatively, you could remove the filter and pass a password hash directly. The path of the SSH public key will come from the path of the private key, plus .pub as extension, but we still let the user override it. You might have noticed that we have an argument for password_hash called "salt" and the value will come from the variable vm_pass_salt. That means we will need to set two more variables.

# region play: Create the VM
- name: Create the VM
  hosts: docker_vm_host_machines
  vars:
    # ...
    vm_pass_salt: "{{ config_lxd_docker_vm_pass_salt }}"
    vm_ssh_priv_key: "{{ config_lxd_docker_vm_ssh_priv_key }}"
  pre_tasks:
  tasks:
# endregion

Note that the order of the variables does not matter. If you prefer defining the variables first which will be used by other variables, that's fine. If you want to start with the variables that you will eventually refer to in the tasks, that's okay too. For me, it felt easier to explain the variables in this order. You can also notice that there are no default values here, so we add pre tasks to check the required variables:

  pre_tasks:
    - name: Fail if config_lxd_docker_vm_ssh_priv_key is not defined
      when: config_lxd_docker_vm_ssh_priv_key | default('', true) == ''
      ansible.builtin.fail:
        msg: You must set an SSH private key to log in to the VM

    - name: Fail if config_lxd_docker_vm_pass_salt is not defined
      when: config_lxd_docker_vm_pass_salt | default('', true) == ''
      ansible.builtin.fail:
        msg: You must set a password salt for the sudo password to log in to the VM

Now our whole play looks like this:

# region play: Create the VM
- name: Create the VM
  hosts: docker_vm_host_machines
  vars:
    vm_name: "{{ config_lxd_docker_vm_name | default('docker', true) }}"
    vm_memory: "{{ config_lxd_docker_vm_memory | default('4GiB', true)}}"
    vm_cpu: "{{ config_lxd_docker_vm_cpus | default(4, true) }}"
    vm_user: "{{ config_lxd_docker_vm_user | default('manager', true) }}"
    vm_pass: "{{ config_lxd_docker_vm_pass | ansible.builtin.password_hash(salt=vm_pass_salt)  }}"
    vm_ssh_pub_key: "{{ config_lxd_docker_vm_ssh_pub_key | default(vm_ssh_priv_key + '.pub', true) }}"

    vm_pass_salt: "{{ config_lxd_docker_vm_pass_salt }}"
    vm_ssh_priv_key: "{{ config_lxd_docker_vm_ssh_priv_key }}"
  pre_tasks:
    - name: Fail if config_lxd_docker_vm_ssh_priv_key is not defined
      when: config_lxd_docker_vm_ssh_priv_key | default('', true) == ''
      ansible.builtin.fail:
        msg: You must set an SSH private key to log in to the VM

    - name: Fail if config_lxd_docker_vm_pass_salt is not defined
      when: config_lxd_docker_vm_pass_salt | default('', true) == ''
      ansible.builtin.fail:
        msg: You must set a password salt for the sudo password to log in to the VM
  tasks:

    # region task: Start Docker VM
    - name: Start Docker VM
      become: true
      community.general.lxd_container:
        url: unix:/var/snap/lxd/common/lxd/unix.socket
        type: virtual-machine
        state: started
        wait_for_ipv4_addresses: true
        name: "{{ vm_name }}"
        source:
          type: image
          server: https://cloud-images.ubuntu.com/releases
          protocol: simplestreams
          alias: "22.04"
        config:
          boot.autostart: "false"
          limits.cpu: "{{ vm_cpu }}"
          limits.memory: "{{ vm_memory }}"
          cloud-init.user-data: |
            #cloud-config
            users:
              - name: {{ vm_user }}
                lock_passwd: false
                groups: sudo
                shell: /bin/bash
                passwd: "{{ vm_pass }}"
                ssh_authorized_keys:
                  - {{ lookup('file', vm_ssh_pub_key) }}

            packages:
              - openssh-server
    # endregion
# endregion

That is a completely fine play in a playbook, but we have some undefined variables in the play which we want to set or other variables we want to override. Let's add the following required variables to our inventory file.

all:
  vars:
    # ...
    config_lxd_docker_vm_pass: "{{ sops.config_lxd_docker_vm_pass }}"
    config_lxd_docker_vm_pass_salt: "{{ sops.config_lxd_docker_vm_pass_salt }}"
    config_lxd_docker_vm_ssh_priv_key: ~/.ssh/ansible

In my case, the SSH private key is the same as I used for the host. I could use a template here too to get the value from the other variable, but I don't recommend reading values from Ansible's built-in variables, because it could lead to some confusion when you change how you access the host, and you forget that it was used somewhere else too. You could introduce a third variable like config_global_ssh_priv_key and read the value from it in the value of the other two variables. The password and the salt parameter comes from a secret. To add the variables, if your secret is already encrypted, use the helper script created for sops.

./sops.sh secret.yml

This is the content of my encrypted secrets.yml in the project root:

become_pass: HomeLab2023
config_lxd_docker_vm_pass: password
config_lxd_docker_vm_pass_salt: 3rehbvjfbr

Run the playbook to start the VM

» Back to table of contents «

We can now run the playbook which will create a new virtual machine on the remote server.

./run.sh playbook-lxd-docker-vm.yml

You can now SSH to the remote server and see that the virtual machine is running:

lxc list

My output:

+--------+---------+------------------------+------+-----------------+-----------+
|  NAME  |  STATE  |          IPV4          | IPV6 |      TYPE       | SNAPSHOTS |
+--------+---------+------------------------+------+-----------------+-----------+
| docker | RUNNING | 10.17.181.143 (enp5s0) |      | VIRTUAL-MACHINE | 0         |
+--------+---------+------------------------+------+-----------------+-----------+

If you want to get a shell as root in the virtual machine, just run

sudo lxc shell docker

You could also use SSH. You already know the IP address, which was in the output of lxc list.

ssh manager@10.17.181.143

We haven't installed anything in the VM yet, but we can finally play with it and delete it when we are done.

Get the IP address of a virtual machine

» Back to table of contents «

The IP address will be different on your machine, and I always like to share commands that will run the same way for everyone, so let's get the IP address automatically.

vm_name=docker
vm_info="$(lxc list "^$vm_name$" --format json | jq '.[0]')"
ip_addresses="$(
  echo "$vm_info" \
  | jq -r '
      .state.network
      | to_entries
      | .[].value.addresses[]
      | select(.family == "inet" and .scope == "global")
      | .address
    '
)"

Now the $ip_addresses variable probably contains a single IP address, unless the virtual machine has multiple networks. In this tutorial we know that we added only one, but later we could change it, so why not make it work with multiple networks as well.

network="$(echo "$vm_info" | jq -r '.expanded_devices.eth0.network')"
ip_range="$(lxc network show "$network" | yq '.config["ipv4.address"]')"

With the above two lines we will get the CIDR notation of the network address. This is what I have in $ip_range: 10.17.181.1/24.

The IP address in the value is the IP of the gateway, but the gateway and the mask size (24) together describes an IP range which has to include the IP we are looking for. With this specific mask size we could check that the IP address starts with 10.17.181., but later we could define a different network with another mask size, so this is when we could use grepcidr. On Ubuntu, we can install it this way:

sudo apt-get install grepcidr

Or download it from the website: https://www.pc-tools.net/unix/grepcidr/

After that, we can run the following command to make sure that the final list contains IP addresses only in the specified IP range:

echo "$ip_addresses" | grepcidr "$ip_range"

That's nice. We could detect the IP address, which is not that useful interactively, but it will help us continue with the next step.

Accessing the virtual machine from the Ansible controller

Why don't we use LAN network

» Back to table of contents «

In the previous sections we learned about SSH-ing to the virtual machine from the host on which the VM is running and also about detecting the IP address of the VM automatically. It would be easier if we could use the SSH client with the SSH keys on the Ansible controller to log in to the virtual machine. One way could be changing the network settings of the virtual machine to get an IP address on the LAN network instead of on the local LXD bridge (lxdbr0). That would work if you have a local network, and you can manage the IP addresses to assign one to a virtual machine. It is not always the case, so I have chosen a different approach. We will keep our current IP address and learn about proxies and how SSH can help us again.

LXD proxy

» Back to table of contents «

You could use an LXD proxy device, but there are multiple requirements.

The configuration is different for virtual machines and containers.
You would need a static IP address which is recommended anyway, so it won't be a problem in the future, but for now, we are experimenting with dynamically assigned IP addresses.
Without firewall settings, you would make the port available from all machines. In a local homelab that is probably fine, but I prefer another solution which is coming in the following sections.

SSH tunnel

» Back to table of contents «

My first idea was simply using an SSH tunnel. I do it frequently. It is basically SSH-ing to a host and use that SSH connection to forward a request from a specific port of the client through the host to an endpoint which is accessible from the host. Note that my remote server's hostname is still ta-lxlt and the IP address of the virtual machine is 10.17.181.143. I open a terminal and run the following command:

ssh -L 127.0.0.1:2000:10.17.181.143:22 -N ta-lxlt

After -L I have the port mapping. The format is

<CLIENT_IP>:<CLIENT_PORT>:<ENDPOINT_IP>:<ENDPOINT_PORT>

CLIENT_IP is optional, but I want to make sure the port is accessible only from localhost. If you know how port forwarding works with Docker, this is very similar, except Docker doesn't need the endpoint ip, since it is always the IP of the container. -N allows me to keep the SSH connection without actually executing any command on the remote server. I open a new terminal and run the following command:

ssh -p 2000 manager@127.0.0.1

If you don't want it to ask for the password, use the SSH key:

ssh -p 2000 -i ~/.ssh/ansible manager@127.0.0.1

or use the SSH agent as we discussed it in Ansible playbook and SSH keys.

Although this is not what we will use for the SSH connection, SSH tunnels can help you with accessing other ports as well, like a web application running inside the virtual machine.

SSH jump servers

» Back to table of contents «

You know that you can SSH to the remote server and from that you can SSH to the virtual machine. You also know that you can pass a command to SSH to execute on the remote server, so you can also run another SSH command.

ssh -t ta-lxlt -- ssh manager@10.17.181.143

-t was required because the SSH command running on the remote server required a pseudo-terminal. We can also try to SSH to the virtual machine and pass the address of a jump server. A jump server is a server to which we have access, and from which we have access to another server. In our case, the virtual machine.

ssh -J ta-lxlt manager@10.17.181.143

This jump server works only because I configured that host in my SSH client config ($HOME/.ssh/config):

Host ta-lxlt ta-lxlt.lan 192.168.4.58
  HostName 192.168.4.58
  Port 22
  User ta
  IdentityFile ~/.ssh/ta-lxlt

We can add our virtual machine too. We will need a name that we can pass to the SSH client. It could be the internal full-qualified hostname of the VM with the remote server's hostname as suffix. Something like this:

Host docker.lxd.ta-lxlt

The internal full qualified hostname is the name of the VM ending with ".lxd". You can confirm it by running the following command on the remote server:

lxc exec docker -- hostname -A

Host docker.lxd.ta-lxlt
  Hostname 10.17.181.143
  User manager
  IdentityFile ~/.ssh/ansible
  ProxyJump ta-lxlt

Now that we have the ProxyJump parameter defined in the SSH client config, the following command on the Ansible controller is enough to log in to the virtual machine:

ssh docker.lxd.ta-lxlt

That way you don't have to remember the IP address every time. We will automate the SSH client configuration of the virtual machine, so even if the IP address changes, you can still log in using the same hostname.

Using separate config files for different kind of servers

» Back to table of contents «

Using $HOME/.ssh/config for all your SSH client configs works, but I had so many servers (physical, virtual, behind VPN connections) that I found it hard to maintain the config file. Fortunately with recent SSH versions, you can include other config files in the main SSH config. For example, this is how my main config looks like:

Include config.d/homelab
Include config.d/external-services
Include config.d/home
Include config.d/multipass

Include config.d/docker.lxd.ta-lxlt

The configuration of ta-lxlt is in config.d/home, I also have an old swarm cluster client config in config.d/homelab, but I found it better to include the dynamically generated client config for the virtual machine directly in the main config.

Use Ansible to configure the SSH client

Make sure config.d exists

» Back to table of contents «

Since config.d is not created by default,we need to make sure it exists. This is hour following task in the playbook:

    # region task: Create base dir for the new SSH client config file
    - name: Create base dir for the new SSH client config file
      delegate_to: localhost
      ansible.builtin.file:
        state: directory
        path: "{{ lookup('env', 'HOME') }}/.ssh/config.d/"
    # endregion

delegate_to is used to run the task on a specific host regardless of where the rest of the tasks were running, and we can use the env lookup plugin to get the home of our local user. You know everything else from the previous episodes.

Get information about the VM

» Back to table of contents «

As we did in the section Get the IP address of a virtual machine, we need to get information about the virtual machine, but now we use Ansible.

    # region task: Get VM info
    - name: Get VM info
      become: true
      changed_when: false
      ansible.builtin.command: lxc list "^{{ vm_name }}$" --format json
      register: _vm_info_command
    # endregion

We save the info in a registered variable. Here comes the fun part. We already learned that the order of the variables doesn't matter. In order to avoid using blocks and indenting our playbook deeper, we can add our helper variables to the vars section of the playbook. I will keep using the underscore character as a prefix.

    _vm_info: "{{ _vm_info_command.stdout | from_json | first }}"

This way we converted the json text in the standard output to a list object in Ansible, and we got the first item from the list.

Get the IP addresses used by the VM:

» Back to table of contents «

To get the IP addresses in Ansible, we can use a very similar approach to what we used in the terminal. The difference is that we used jq in the terminal (we installed it in Using facts and the GitHub API in Ansible), and we will use Jinja filters in Ansible. Let's add the following variable to the vars section of the playbook.

    _ip_addresses: "{{ 
        _vm_info.state.network
          | dict2items
          | map(attribute='value.addresses')
          | flatten
          | selectattr('family', 'equalto', 'inet')
          | selectattr('scope', 'equalto', 'global')
          | map(attribute='address')
      }}"

Get information about the LXD network

» Back to table of contents «

Getting the name of the network is the easy part. Add the following variable to the vars section of the playbook:

    _network: "{{ _vm_info.expanded_devices.eth0.network }}"

Now that we know the name of the network, we will need to find out what subnet the network is using. Let's add the following task to the playbook:

    # region task: Get network info
    - name: Get network info
      become: true
      changed_when: false
      ansible.builtin.command: lxc network show "{{ _network }}"
      register: _network_info_command
    # endregion

Remember, lxc network show returns a YAML output, not json. So we need to add the following variable to the vars section to the playbook.

    _network_info: "{{ _network_info_command.stdout | from_yaml }}"

Get the IP usable IP address assigned to eth0

» Back to table of contents «

Now that we have everything about the LXD network, we can get the CIDR notation of the subnet and the actual IP address of the VM. We will use a new filter in Ansible, called ansible.utils.reduce_on_network. It is basically the alternative to grepcidr which we used in the terminal. Let's add the following variables to the vars section of the playbook:

    _ip_range: "{{ _network_info.config['ipv4.address'] }}"
    _ip: "{{
      _ip_addresses
        | ansible.utils.reduce_on_network(_ip_range)
        | first
      }}"

Because I'm super careful, I also filter to the first item in the list, so even if for some reason there are multiple lines, I will deal with only one in the next steps. This filter requires the following line in the requirements.txt.

netaddr==0.9.0 # for using ansible.utils.reduce_on_network()

Don't forget to run pip install -r requirements.txt to install the new library.

Generate the new SSH client config for the VM

» Back to table of contents «

Since we put our helper variables to the vars section of the playbook, our task to generate and save the client config will be very short:

    # region task: Add SSH client config
    - name: Add SSH client config
      delegate_to: localhost
      ansible.builtin.copy:
        dest: "{{ lookup('env', 'HOME') }}/.ssh/config.d/{{ vm_inventory_hostname }}"
        content: |
          Host {{ vm_inventory_hostname }}
            Hostname {{ _ip }}
            User {{ vm_user }}
            IdentityFile {{ vm_ssh_priv_key }}

            ProxyJump {{ inventory_hostname }}
    # endregion

We had to delegate it to localhost again. We used the good old copy module save the content defined in the task to the destination file. There is one variable missing and that is the name of the file which will also be the host in the client config. Let's add the following variable to the vars section of the playbook:

    vm_inventory_hostname: "{{
        config_lxd_docker_vm_inventory_hostname
        | default(vm_name + '.lxd.' + inventory_hostname, true)
      }}"

Note that the default suffix is the inventory hostname of the remote server. This is the name which you use in the inventory file under the hosts section. For me, it is the same as the hostname of the remote server, but it could be different. If you want to override the generated name, you can set config_lxd_docker_vm_inventory_hostname in the inventory file.

As a final step, we have to include the generated config file in the main config:

    # region task: Include the new SSH client config in the main config
    - name: Include the new SSH client config in the main config
      delegate_to: localhost
      ansible.builtin.lineinfile:
        state: present
        create: true
        path: "{{ lookup('env', 'HOME') }}/.ssh/config"
        line: "Include config.d/{{ vm_inventory_hostname }}"
    # endregion

We use the lineinfile module to add a new line to the main config. Since it is also possible that you don't have that main config either (although at this point it is unlikely), create: true makes sure the file is created if necessary. We also set the filepath in path and the line in the line parameter.

Run the playbook to generate the SSH client config

» Back to table of contents «

Now you can finally run the playbook. In case you couldn't follow the steps, this is the full playbook:


# region play: Create the VM
- name: Create the VM
  hosts: docker_vm_host_machines
  vars:
    vm_name: "{{ config_lxd_docker_vm_name | default('docker', true) }}"
    vm_memory: "{{ config_lxd_docker_vm_memory | default('4GiB', true)}}"
    vm_cpu: "{{ config_lxd_docker_vm_cpus | default(4, true) }}"
    vm_user: "{{ config_lxd_docker_vm_user | default('manager', true) }}"
    vm_pass: "{{ config_lxd_docker_vm_pass | ansible.builtin.password_hash(salt=vm_pass_salt)  }}"
    vm_ssh_pub_key: "{{ config_lxd_docker_vm_ssh_pub_key | default(vm_ssh_priv_key + '.pub', true) }}"

    vm_pass_salt: "{{ config_lxd_docker_vm_pass_salt }}"
    vm_ssh_priv_key: "{{ config_lxd_docker_vm_ssh_priv_key }}"
    vm_inventory_hostname: "{{
        config_lxd_docker_vm_inventory_hostname
        | default(vm_name + '.lxd.' + inventory_hostname, true)
      }}"

    _vm_info: "{{ _vm_info_command.stdout | from_json | first }}"
    _ip_addresses: "{{ 
        _vm_info.state.network
          | dict2items
          | map(attribute='value.addresses')
          | flatten
          | selectattr('family', 'equalto', 'inet')
          | selectattr('scope', 'equalto', 'global')
          | map(attribute='address')
      }}"
    _network: "{{ _vm_info.expanded_devices.eth0.network }}"
    _network_info: "{{ _network_info_command.stdout | from_yaml }}"
    _ip_range: "{{ _network_info.config['ipv4.address'] }}"
    _ip: "{{
      _ip_addresses
        | ansible.utils.reduce_on_network(_ip_range)
        | first
      }}"
  pre_tasks:
    - name: Fail if config_lxd_docker_vm_ssh_priv_key is not defined
      when: config_lxd_docker_vm_ssh_priv_key | default('', true) == ''
      ansible.builtin.fail:
        msg: You must set an SSH private key to log in to the VM

    - name: Fail if config_lxd_docker_vm_pass_salt is not defined
      when: config_lxd_docker_vm_pass_salt | default('', true) == ''
      ansible.builtin.fail:
        msg: You must set a password salt for the sudo password to log in to the VM
  tasks:

    # region task: Start Docker VM
    - name: Start Docker VM
      become: true
      community.general.lxd_container:
        url: unix:/var/snap/lxd/common/lxd/unix.socket
        type: virtual-machine
        state: started
        wait_for_ipv4_addresses: true
        name: "{{ vm_name }}"
        source:
          type: image
          server: https://cloud-images.ubuntu.com/releases
          protocol: simplestreams
          alias: "22.04"
        config:
          boot.autostart: "false"
          limits.cpu: "{{ vm_cpu }}"
          limits.memory: "{{ vm_memory }}"
          cloud-init.user-data: |
            #cloud-config
            users:
              - name: {{ vm_user }}
                lock_passwd: false
                groups: sudo
                shell: /bin/bash
                passwd: "{{ vm_pass }}"
                ssh_authorized_keys:
                  - {{ lookup('file', vm_ssh_pub_key) }}

            packages:
              - openssh-server
    # endregion

    # region task: Create base dir for the new SSH client config file
    - name: Create base dir for the new SSH client config file
      delegate_to: localhost
      ansible.builtin.file:
        state: directory
        path: "{{ lookup('env', 'HOME') }}/.ssh/config.d/"
    # endregion

    # region task: Get VM info
    - name: Get VM info
      become: true
      changed_when: false
      ansible.builtin.command: lxc list "^{{ vm_name }}$" --format json
      register: _vm_info_command
    # endregion

    # region task: Get network info
    - name: Get network info
      become: true
      changed_when: false
      ansible.builtin.command: lxc network show "{{ _network }}"
      register: _network_info_command
    # endregion

    # region task: Add SSH client config
    - name: Add SSH client config
      delegate_to: localhost
      ansible.builtin.copy:
        dest: "{{ lookup('env', 'HOME') }}/.ssh/config.d/{{ vm_inventory_hostname }}"
        content: |
          Host {{ vm_inventory_hostname }}
            Hostname {{ _ip }}
            User {{ vm_user }}
            IdentityFile {{ vm_ssh_priv_key }}

            ProxyJump {{ inventory_hostname }}
    # endregion

    # region task: Include the new SSH client config in the main config
    - name: Include the new SSH client config in the main config
      delegate_to: localhost
      ansible.builtin.lineinfile:
        state: present
        create: true
        path: "{{ lookup('env', 'HOME') }}/.ssh/config"
        line: "Include config.d/{{ vm_inventory_hostname }}"
    # endregion

And the command to run it:

./run.sh playbook-lxd-docker-vm.yml

After that, you can log in to the VM with the following command from the Ansible controller:

ssh docker.lxd.<server_hostname>

In my case, it is:

ssh docker.lxd.ta-lxlt

Access a webservice on the created VM

» Back to table of contents «

Run a Python http server as an example in the virtual machine:

python3 -m http.server 8888

Open a tunnel on the ansible controller:

ssh -L 127.0.0.1:8888:127.0.0.1:8888 -N docker.lxd.ta-lxlt

And open 127.0.0.1:8888 from a web browser or run the following curl command on the Ansible controller:

curl 127.0.0.1:8888

My output:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".bash_history">.bash_history</a></li>
<li><a href=".bash_logout">.bash_logout</a></li>
<li><a href=".bashrc">.bashrc</a></li>
<li><a href=".cache/">.cache/</a></li>
<li><a href=".profile">.profile</a></li>
<li><a href=".ssh/">.ssh/</a></li>
</ul>
<hr>
</body>
</html>

Just think about what happened here:

You SSH-d to a remote server
Through that remote server you SSH-d to the virtual machine directly from the Ansible controller
You opened a tunnel for a web application from the Ansible controller to the localhost of the virtual machine.

Yes, you are right, configuring a LAN IP would have been easier, but not an option for everyone, and if you ask me, less fun ass well.

Conclusion

» Back to table of contents «

I think it's important to mention again that our current solution is not ideal. We use a dynamically assigned IP address for the virtual machine, so even though the IP address will not change every time you restart the virtual machine, it is not dedicated to this VM. That means, when you can't access the virtual machine from the Ansible controller, you need to run the playbook again, even if the virtual machine is already created. For servers, we usually use static IP addresses, but to choose the right IP address could be another challenge, so for now, we used a dynamically assigned IP. That's how we learn step by step.

The final source code of this episode can be found on GitHub:

https://github.com/rimelek/homelab/tree/tutorial.episode.9.1

rimelek / homelab

Source code to create a home lab. Part of a video tutorial

README

The project contains code written for the tutorial, but you can also use parts of it if you refer to this repository.

Tutorial on YouTube in English: https://www.youtube.com/watch?v=K9grKS335Mo&list=PLzMwEMzC_9o7VN1qlfh-avKsgmiU8Jofv

Tutorial on YouTube in Hungarian: https://www.youtube.com/watch?v=dmg7lYsj374&list=PLUHwLCacitP4DU2v_DEHQI0U2tQg0a421

You can also find an example inventory file in the project root. You can copy that and change the content, so you will use your IP…

View on GitHub

Everything about Docker volumes

Ákos Takács — Sun, 07 Jan 2024 20:54:45 +0000

Intro

"Where are the Docker volumes?" This question comes up a lot on the Docker Forum. There is no problem with curiosity, but this is usually asked when someone wants to edit or at least read files directly on the volume from a terminal or an IDE, but not through a container. So I must start with a statement:

Important: You should never handle files on a volume directly without entering a container, unless there is an emergency, and even then, only at your own risk.

Why I am saying it, you will understand if you read the next sections. Although the original goal with this tutorial was to explain where the volumes are, it is hard to talk about it without understanding what the volumes are and what different options you have when using volumes. As a result of that, by reading this tutorial, you can learn basically everything about the local volumes, but you can also search for volume plugins on Docker Hub.

Where does Docker store data?
What is a Docker volume
Custom volume path
- Custom volume path overview
- Avoid accidental data loss on volumes
Docker CE volumes on Linux
Docker Desktop volumes
- Docker Desktop volumes on macOS
- Docker Desktop volumes on Windows
  - Switching between Linux and Windows containers
  - Linux containers
  - Windows containers
- Docker Desktop volumes on Linux
Editing files on volumes
- The danger of editing volume contents outside a container
- View and edit files htrough Docker Desktop
- Container based dev environments
  - Docker Desktop Dev environment
  - Visual Studio Code remote development
  - Visual Studio Code dev containers
Conclusion

Where does Docker store data?

» Back to table of contents «

Before we talk about the location of volumes, we first have to talk about the location of all data that Docker handles. When I say "Docker", I usually mean "Docker CE".

Docker CE is the community edition of Docker and can run directly on Linux. It has a data root directory, which is the following by default:

/var/lib/docker

You can change it in the daemon configuration, so if it is changed on your system, you will need to replace this folder in the examples I show. To find out what the data root is, run the following command:

docker info --format '{{ .DockerRootDir }}'

In case of Docker Desktop of course, you will always have a virtual machine, so the path you get from the above command will be in the virtual machine.

What is a Docker volume?

» Back to table of contents «

For historical reasons, the concept of volumes can be confusing. There is a page in the documentation which describes what volumes are, but when you see a Compose file or a docker run command, you see two types of volumes, but only one of them is actually a volume.

Example Compose file:

services:
  server:
    image: httpd:2.4
    volumes:
      - ./docroot:/usr/local/apache2/htdocs

Did I just define a volume? No. It is a bind mount. Let’s just use the long syntax:

services:
  server:
    image: httpd:2.4
    volumes:
      - type: bind
        source: ./docroot
        target: /usr/local/apache2/htdocs

The "volumes" section should have been "storage" or "mounts" to be more clear. In fact, the “docker run” command supports the --mount option in addition to -v and --volume, and only --mount supports the type parameter to directly choose between volume and bind mount.

Then what do we call a volume? Let’s start with answering another question. What do we not call a volume? A file can never be a volume. A volume is always a directory, and it is a directory which is created by Docker and handled by Docker throughout the entire lifetime of the volume. The main purpose of a volume is to populate it with the content of the directory to which you mount it in the container. That’s not the case with bind mounts. Bind mounts just completely override the content of the mount point in the container, but at least you can choose where you want to mount it from.

You should also know that you can disable copying data from the container to your volume and use it as a simple bind mount, except that Docker creates it in the Docker data root, and when you delete the volume after you wrote something on it, you will lose the data.

volumes:
  docroot:

services:
  server:
    image: httpd:2.4
    volumes:
      - type: volume
        source: docroot
        target: /usr/local/apache2/htdocs
        volume:
          nocopy: true

You can find this and other parameters in the documentation of volumes in a compose file. Scroll down to the "Long syntax" to read about "nocopy".

Custom volume path

Custom volume path overview

» Back to table of contents «

There is indeed a special kind of volume which seems to mix bind mounts and volumes. The following example will assume you are using Docker CE on Linux.

volume_name="test-volume"
source="$PWD/$volume_name"

mkdir -p "$volume_name"
docker volume create "$volume_name" \
  --driver "local" \
  --opt "type=none" \
  --opt "device=$source" \
  --opt "o=bind"

Okay, so you created a volume and you also specified where the source directory is (device), and you specified that it is a bind mount. Don’t worry, you find it confusing because it is confusing. o=bind doesn’t mean that you will bind mount a directory into the container, which will always happen, but that you will bind mount the directory to the path where Docker would have created the volume if you didn’t define the source.

This is basically the same what you would do on Linux with the mount command:

mount -o bind source/ target/

Without -o bind the first argument must be a block device. This is why we use the “device” parameter, even though we mount a folder.

This is one way to know where the Docker volume is.

Note: Even the the above example assumed Linux, custom volume path would work on other operating systems as well, since Docker Desktop would mount the required path into the virtual machine.

Let’s just test if it works and inspect the volume:

docker volume inspect test-volume

You will get a json like this:

[
    {
        "CreatedAt": "2024-01-05T00:55:15Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/test-volume/_data",
        "Name": "test-volume",
        "Options": {
            "device": "/home/ta/test-volume",
            "o": "bind",
            "type": "none"
        },
        "Scope": "local"
    }
]

The “Mountpoint” field in the json is not the path in a container, but the path where the specified device should be mounted at. In our case, the device is actually a directory. So let’s see the content of the mount point:

sudo ls -la $(docker volume inspect test-volume --format '{{ .Mountpoint }}')

You can also check the content of the source directory:

ls -la test-volume/

Of course, both are empty as we have no container yet. How would Docker know what the content should be? As we already learned it, we need to mount the volume into a container to populate the volume.

docker run \
  -d --name test-container \
  -v test-volume:/usr/local/apache2/htdocs \
  httpd:2.4

Check the content in the container:

docker exec test-container ls -lai /usr/local/apache2/htdocs/

Output:

total 16
 256115 drwxr-xr-x 2 root     root     4096 Jan  5 00:33 .
5112515 drwxr-xr-x 1 www-data www-data 4096 Apr 12  2023 ..
 256139 -rw-r--r-- 1      501 staff      45 Jun 11  2007 index.html

Notice that we added the flag "i" to the "ls" command so we can see the inode number, which identifies the files and directories on the filesystem in the first column.

Check the directory created by Docker:

sudo ls -lai $(docker volume inspect test-volume --format '{{ .Mountpoint }}')

256115 drwxr-xr-x 2 root root  4096 Jan  5 00:33 .
392833 drwx-----x 3 root root  4096 Jan  5 00:55 ..
256139 -rw-r--r-- 1  501 staff   45 Jun 11  2007 index.html

As you can see, only the parent directory is different, so we indeed see the same files in the container and in the directory created by Docker. Now let’s check our source directory.

ls -lai test-volume/

Output:

total 12
256115 drwxr-xr-x  2 root root  4096 Jan  5 00:33 .
255512 drwxr-xr-x 11 ta   ta    4096 Jan  5 00:32 ..
256139 -rw-r--r--  1  501 staff   45 Jun 11  2007 index.html

Again, the same files, except the parent. We confirmed, that we could create an empty volume directory, we could populate it when we started a container and mounted the volume, and the files appeared where Docker creates volumes. Now let’s check one more thing. Since this is a special volume where we defined some parameters, there is an opts.json right next to "_data"

sudo cat "$(dirname "$(docker volume inspect test-volume --format '{{ .Mountpoint }}')")"/opts.json

Output:

{"MountType":"none","MountOpts":"bind","MountDevice":"/home/ta/test-volume","Quota":{"Size":0}}

Now remove the test container:

docker container rm -f test-container

Check the directory created by Docker:

sudo ls -lai $(docker volume inspect test-volume --format '{{ .Mountpoint }}')

It is empty now.

392834 drwxr-xr-x 2 root root 4096 Jan  5 00:55 .
392833 drwx-----x 3 root root 4096 Jan  5 00:55 ..

And notice that even the inode has changed, not just the content disappeared. On the other hand, the directory we created is untouched and you can still find the index.html there.

Avoid accidental data loss on volumes

» Back to table of contents «

Let me show you an example using Docker Compose. The compose file would be the following:

volumes:
  docroot:
    driver: local
    driver_opts:
      type: none
      device: ./docroot
      o: bind

services:
  httpd:
    image: httpd:2.4
    volumes:
      - type: volume
        source: docroot
        target: /usr/local/apache2/htdocs

You can populate ./docroot in the project folder by running

docker compose up -d

You will then find index.html in the docroot folder. You probably know that you can delete a compose project by running docker compose down, and delete the volumes too by passing the flag -v.

docker compose down -v

You can run it, and the volume will be destroyed, but not the content of the already populated "docroot" folder. It happens, because the folder which is managed by Docker in the Docker data root does not physically have the content. So the one that was managed by Docker could be safely removed, but it didn’t delete your data.

Docker CE volumes on Linux

» Back to table of contents «

This question seems to be already answered in the previous sections, but let’s evaluate what we learned and add some more details.

So you can find the local default volumes under /var/lib/docker/volumes if you didn’t change the data root. For the sake of simplicity of the commands, I will keep using the default path.

The Docker data root is not accessible by normal users, only by administrators. Run the following command:

sudo ls -la /var/lib/docker/volumes

You will see something like this:

total 140
drwx-----x 23 root root  4096 Jan  5 00:55 .
drwx--x--- 13 root root  4096 Dec 10 14:27 ..
drwx-----x  3 root root  4096 Jan 25  2023 0c5f9867e761f6df0d3ea9411434d607bb414a69a14b3f240f7bb0ffb85f0543
drwx-----x  3 root root  4096 Sep 19 13:15 1c963fb485fbbd5ce64c6513186f2bc30169322a63154c06600dd3037ba1749a
...
drwx-----x  3 root root  4096 Jan  5  2023 apps_cache
brw-------  1 root root  8, 1 Dec 10 14:27 backingFsBlockDev
-rw-------  1 root root 65536 Jan  5 00:55 metadata.db

These are the names of the volumes and two additional special files.

backingFsBlockDev
metadata.db

We are not going to discuss it in more details. All you need to know at this point is that this is where the volume folders are. Each folder has a sub-folder called "_data" where the actual data is, and there could be an opts.json with metadata next to the "_data" folder.

Note: When you use rootless Docker, the Docker data root will be in your user’s home: $HOME/.local/share/docker.

Docker Desktop volumes

» Back to table of contents «

Docker Desktop volumes are different depending on the operating system and whether you want to run Linux containers or Windows containers.

Docker Desktop always runs a virtual machine for Linux containers and runs Docker CE in it in a quite complicated way, so your volumes will be in the virtual machine too. Because of that fact when you want to access the volumes, you either have to find a way to run a shell in the virtual machine, or find a way to share the filesystem on the network and use your filebrowser, IDE or terminal on the host.

Parts of what I show here and more can be found in my presentation which I gave on the 6th Docker Community All-Hands. Tyler Charboneau wrote a blog post about it, but you can also find the video in the blog post.

Docker Desktop volumes on macOS

» Back to table of contents «

On macOS, you can only run Linux containers and there is no such thing as macOS container yet (2024. january).

You can get to the volumes folder by running the following command:

docker run --rm -it --privileged --pid host ubuntu:22.04 \
  nsenter --all -t 1 \
    sh -c 'cd /var/lib/docker/volumes && sh'

Or just simply mount that folder to a container:

docker run --rm -it \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  --workdir /var/lib/docker/volumes \
  ubuntu:22.04 \
  bash

You can also run an NFS server in a container that mounts the volumes so you can mount the remote fileshare on the host. The following compose.yml file can be used to run the NFS server:

services:
  nfs-server:
    image: openebs/nfs-server-alpine:0.11.0
    volumes:
       - /var/lib/docker/volumes:/mnt/nfs
    environment:
      SHARED_DIRECTORY: /mnt/nfs
      SYNC: sync
      FILEPERMISSIONS_UID: 0
      FILEPERMISSIONS_GID: 0
      FILEPERMISSIONS_MODE: "0755"
    privileged: true
    ports:
      - 127.0.0.1:2049:2049/tcp
      - 127.0.0.1:2049:2049/udp

Start the server:

docker compose up -d

Create the mount point on the host:

sudo mkdir -p /var/lib/docker/volumes
sudo chmod 0700 /var/lib/docker

Mount the base directory of volumes:

sudo mount -o vers=4 -t nfs 127.0.0.1:/ /var/lib/docker/volumes

And list the content:

sudo ls -l /var/lib/docker/volumes

Docker Desktop volumes on Windows

Switching between Linux and Windows containers

» Back to table of contents «

Docker Desktop on Windows allows you to switch between Linux containers and Windows containers.

To find out which one you are using, run the following command:

docker info --format '{{ .OSType }}'

If it returns "windows", you are using Windows containers, and if it returns "linux", you are using Linux containers.

Linux containers

» Back to table of contents «

Since Linux containers always require a virtual machine, you will have your volumes in the virtual machine the same way as you would on macOS. The difference is how you can access them. A common way is through a Docker container. Usually I would run the following command.

docker run --rm -it --privileged --pid host ubuntu:22.04 `
  nsenter --all -t 1 `
    sh -c 'cd /var/lib/docker/volumes && sh'

But if you have an older kernel in WSL2 which doesn’t support the time namespace, you can get an error message like:

nsenter: cannot open /proc/1/ns/time: No such file or directory

If that happens, make sure you have the latest kernel in WSL2. If you built a custom kernel, you may need to rebuild it from a new version.

If you can’t update the kernel yet, exclude the time namespace, and run the following command:

docker run --rm -it --privileged --pid host ubuntu:22.04 `
  nsenter -m -n -p -u -t 1 `
    sh -c 'cd /var/lib/docker/volumes && sh'

You can simply mount the base directory in a container the same way as we could on macOS:

docker run --rm -it `
  -v /var/lib/docker/volumes:/var/lib/docker/volumes `
  --workdir /var/lib/docker/volumes `
  ubuntu:22.04 `
  bash

We don’t need to run a server in a container to share the volumes, since it works out of the box in WSL2. You can just open the Windows explorer and go to

\\wsl.localhost\docker-desktop-data\data\docker\volumes

Warning: WSL2 let’s you edit files more easily even if the files are owned by root on the volume, so do it at your own risk. My recommendation is using it only for debugging.

Windows containers

» Back to table of contents «

Windows containers can mount their volumes from the host. Let's create a volume

docker volume create windows-volume
Inspect the volume:

You will get something like this:

[
    {
        "CreatedAt": "2024-01-06T16:27:03+01:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "C:\\ProgramData\\Docker\\volumes\\windows-volume\\_data",
        "Name": "windows-volume",
        "Options": null,
        "Scope": "local"
    }
]

So now you got the volume path on Windows in the “Mountpoint” field, but you don’t have access to, it unless you are Administrator. The following command works only from Powershell run as Administrator

cd $(docker volume inspect windows-volume --format '{{ .Mountpoint }}')

If you want to access it from Windows Explorer, you can first go to

C:\ProgramData

Note: This folder is hidden by default, so if you want to open it, just type the path manually in the navigation bar, or enable hidden folders on Windows 11 (works differently on older Windows):

Menu bar » View » Show » Hidden Items

Then try to open the folder called “Docker” which gives you a prompt to ask for permission to access to folder.

and then try to open the folder called “volumes” which will do the same.

After that you can open any Windows container volume from Windows explorer.

Docker Desktop volumes on Linux

» Back to table of contents «

On Windows, you could have Linux containers and Window containers, so you had to switch between them. On Linux, you can install Docker CE in rootful and rootless mode, and you can also install Docker Desktop. These are 3 different and separate Docker installations and you can switch between them by changing context or logging in as a different user.

You can check the existing contexts by running the following command:

docker context ls

If you have Docker CE installed on your Linux, and you are logged in as a user who installed the rootless Docker, and you also have Docker Desktop installed, you can see at least the following three contexts:

NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                                       KUBERNETES ENDPOINT   ORCHESTRATOR
default             moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux *     moby                Docker Desktop                            unix:///home/ta/.docker/desktop/docker.sock
rootless            moby                Rootless mode                             unix:///run/user/1000/docker.sock

In order to use Docker Desktop, you need to switch to the context called "desktop-linux".

docker context use desktop-linux

Important: The default is usually rootful Docker CE and the other too are obvious. Only the rootful Docker CE needs to run as root, so if you want to interact with Docker Desktop, don’t make the mistake of running the docker commands with sudo:

sudo docker context ls

NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *           moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

In terms of accessing volumes, Docker Desktop works similarly on macOS and Linux, so you have the following options:

Run a shell in the virtual machine using nsenter:

docker run --rm -it --privileged --pid host ubuntu:22.04 \
  nsenter --all -t 1 \
    sh -c 'cd /var/lib/docker/volumes && sh'

Or just simply mount that folder to a container:

docker run --rm -it \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  --workdir /var/lib/docker/volumes \
  ubuntu:22.04 \
  bash

And of course, you can use the nfs server compose project with the following compose.yml

services:
  nfs-server:
    image: openebs/nfs-server-alpine:0.11.0
    volumes:
       - /var/lib/docker/volumes:/mnt/nfs
    environment:
      SHARED_DIRECTORY: /mnt/nfs
      SYNC: sync
      FILEPERMISSIONS_UID: 0
      FILEPERMISSIONS_GID: 0
      FILEPERMISSIONS_MODE: "0755"
    privileged: true
    ports:
      - 127.0.0.1:2049:2049/tcp
      - 127.0.0.1:2049:2049/udp

and prepare the mount point. Remember, you can have Docker CE running as root, which means /var/lib/docker probably exists, so let's create the mount point as /var/lib/docker-desktop/volumes:

sudo mkdir -p /var/lib/docker-desktop/volumes
sudo chmod 0700 /var/lib/docker-desktop

And mount it:

sudo mount -o vers=4 -t nfs 127.0.0.1:/ /var/lib/docker-desktop/volumes

And check the content:

sudo ls -l /var/lib/docker-desktop/volumes

You could ask why we mount the volumes into a folder on the host, which requires sudo if the docker commands don't. The reason is that you will need sudo to use the mount command, so it shouldn't be a problem to access the volumes as root.

Editing files on volumes

The danger of editing volume contents outside a container

» Back to table of contents «

Now you know how you can find out where the volumes are. You also know how you can create a volume with a custom path, even if you are using Docker Desktop, which creates the default volumes inside a virtual machine.

But most of you wanted to know where the volumes were to edit the files. Note that I will quote the following paragraphs only for better visibility, not because it was written by someone else.

Any operation inside the Docker data root is dangerous, and can break your Docker completely, or cause problems that you don’t immediately recognize, so you should never edit files without mounting the volume into a container, except if you defined a custom volume path so you don’t have to go into the Docker data root.

Even if you defined a custom path, we are still talking about a volume, which will be mounted into a container, in which the files can be accessed by a process which requires specific ownership and permissions. By editing the files from the host, you can accidentally change the permission or the owner making it inaccessible for the process in the container.

Even though I don’t recommend it, I understand that sometimes we want to play with our environment to learn more about, but we still have to try to find a less risky way to do it.

You know where the volumes are, and you can edit the files with a text editor from command line or even from the graphical interface. One problem on Linux and macOS could be setting the proper permissions so you can edit the files even if you are not root. Discussing permissions could be another tutorial, but this is one reason why we have to try to separate the data managed by a process in a Docker container from the source code or any files that requires an interactive user. Just think of an application that is not running in a container, but the files still have to be owned by another user. An example could be a webserver, where the files has to be owned by a user or group so the webserver has access to the files, while you still should be able to upload files.

View and Edit files through Docker Desktop

» Back to table of contents «

Docker Desktop let's you browse files from the GUI, which is great for debugging, but I don’t recommend it for editing files, even though Docker Desktop makes that possible too. Let’s see why I am saying it.

Open the Containers tab of Docker Desktop.

Click on the three dots in the line of the container in which you want to browse files

Go to a file that you want to edit

Note: Notice that Docker Desktop shows you whether the files are modified on the container’s filesystem, or you see a file on a volume.

Right click on the file and select "Edit file".

Before you do anything, run a test container:

docker run -d --name httpd -v httpd_docroot:/usr/local/apache2/htdocs httpd:2.4

And check the permissions of the index file:

docker exec -it httpd ls -l /usr/local/apache2/htdocs/

You will see this:

-rw-r--r-- 1 504 staff 45 Jun 11  2007 index.html

You can then edit the file and click on the floppy icon on the right side or just press CTRL+S (Command+S on macOS) to save the modification.

Then run the following command from a terminal:

docker exec -it httpd ls -l /usr/local/apache2/htdocs/

And you will see that the owner of the file was changed to root.

total 4
-rw-r--r-- 1 root root 69 Jan  7 12:21 index.html

One day it might work better, but I generally don't recommend editing files in containers from the Graphical interface.

Edit only source code that you mount into the container during development or Use Compose watch to update the files when you edit them, but let the data be handled only by the processes in the containers.

Some applications are not optimized for running in containers and there are different folders and files at the same place where the code is, so it is hard to work with volumes and mounts while you let the process in the container change a config file, which you also want to edit occasionally. In that case you ned to learn how permissions are handled on Linux using the chmod and chown commands so you both have permission to access the files.

Container based dev environments

Docker Desktop Dev environment

» Back to table of contents «

One of the features of Docker Desktop is that you can run a development environment in a container. In this tutorial we will not discuss it in details, but it is good to know that it exists, and you can basically work inside a container into which you can mount volumes.

More information in the documentation of the Dev environment

Visual Studio Code remote development

» Back to table of contents «

The dev environment of Docker Desktop can be opened from Visual Studio Code as it supports opening projects in containers similarly to how it supports remote development through SSH connection or in Windows Subsystem for Linux. You can use it without Docker Desktop to simply open a shell in a container or even open a project in a container.

More information is in the documentation of VSCode about containers.

Visual Studio Code dev containers

» Back to table of contents «

Microsoft also created container images for creating a dev container, which is similar to what Docker Desktop supports, but the process of creating a dev container is different.

More information in the documentation of VSCode about dev containers.

Conclusion

» Back to table of contents «

There are multiple ways to browse the content of the Docker volumes, but it is not recommended to edit the files on the volumes. If you know enough about how containers work and what are the folders and files that you can edit without harming your system, you probably know enough not to edit the files that way in the first place.

For debugging reasons or to learn about Docker by changing things in the environment, you can still edit the files at your own risk.

Everything I described in this tutorial is true even if the user is not an interactive user, but an external user from the container’s point of view, trying to manage files directly in the Docker data root.

So with that in mind if you ever think of doing something like that, stop for a moment, grab a paper and write the following sentence 20 times to the paper:

I do not touch the Docker data root directly.

If you enjoyed this tutorial, I also recommend reading about Volume-only Compose projects.

You can also subscribe to my YouTube channel if you want to be notified about my videos as well. If you are interested in Hungarian contents, I have a channel for you too.

This tutorial was originally posted on a website where I write about Docker-related topics to help beginners and even more advanced users. Check it out and see if you can find something interesting:

https://learn-docker.it-sziget.hu/en/latest/pages/advanced/volumes.html

Using facts and the GitHub API in Ansible

Ákos Takács — Sat, 30 Dec 2023 21:48:18 +0000

Introduction

We will create a new Ansible role and a playbook to automate the installation of the Command line tools I always install on Ubuntu servers. Having the installer and Ansible role is not enough. It is always a good practice to document the role, what it is for and how people can use it, so we will discuss that too.

The new features we will learn about today are the following:

Using multiple tasks files and including a tasks file in the main.yml.
Using Ansible facts, disabling them and gathering a subset of the available facts.
Creating a symbolic link
Updating the apt repository cache
Using the folder "vars" in addition to "defaults"
Using regular expressions in Ansible

We will start from the source code of the 7th episode:
https://github.com/rimelek/homelab/tree/tutorial.episode.7

Before you begin
- Requirements
- Download the already written code of the previous episode
- Have an inventory file
- Activate the Python virtual environment
Ansible playbook and optional APT cache update
Ansible role
- Ansible role overview
- Creating a symbolic link
- Using multiple tasks files
- Install the latest yq from GitHub
  - Using the GitHub API to get the latest release
  - Get the version number of the latest release
  - Get the architecture and operating system of the server
  - Saving helper variables in addition to defaults
  - Installing the desired version of yq
  - Skip downloading when the existing version is the desired one
  - Full yq tasks file
Documenting Ansible roles
Conclusion

Before you begin

Requirements

» Back to table of contents «

The project requires Nix which we discussed in Install Ansible 8 on Ubuntu 20.04 LTS using Nix
You will also need an Ubuntu remote server. I recommend an Ubuntu 22.04 virtual machine.

Download the already written code of the previous episode

» Back to table of contents «

If you started the tutorial with this episode, clone the project from GitHub:

git clone https://github.com/rimelek/homelab.git
cd homelab

If you cloned the project now, or you want to make sure you are using the exact same code I did, switch to the previous episode in a new branch

git checkout -b tutorial.episode.7b tutorial.episode.7

Have the inventory file

» Back to table of contents «

Copy the inventory template

cp inventory-example.yml inventory.yml

Change ansible_host to the IP address of your Ubuntu server that you use for this tutorial,
and change ansible_user to the username on the remote server that Ansible can use to log in.
If you still don't have an SSH private key, read the Generate an SSH key part of Ansible playbook and SSH keys
If you want to run the playbook called playbook-lxd-install.yml, you will need to configure a physical or virtual disk which I wrote about in The simplest way to install LXD using Ansible. If you don't have a usable physical disk, Look for truncate -s 50G <PATH>/lxd-default.img to create a virtual disk.
You will need an encrypted secret file which I wrote about in the Encrypt a file section of "Use SOPS in Ansible ro read your secrets".

Activate the Python virtual environment

» Back to table of contents «

./create-nix-env.sh venv

Optionally start an ssh agent:

ssh-agent $SHELL

and activate the environment with

source homelab-env.sh

Ansible playbook and optional APT cache update

» Back to table of contents «

Before we can talk about the role, we have to start with a playbook. Previously, we only had playbooks for specific tasks like installing and removing LXD. The goal is to have a playbook that installs the common dependencies with which you can play on the remote servers even without Ansible, so when you are trying to do something new, you don't have to start with yaml files without even knowing what you want to do in the end. Let's call this playbook file "playbook-system-base.yml", and for now, add only the role that we will create soon.

- hosts: all
  roles:
    - role: cli_tools

We still assume that all our machines that we configure in the inventory file are targets. It will change, but not in this post.

This ansible role will contain the installation of lots of APT packages. We could have other roles that want to install APT packages, so we also want to make sure the APT cache is up-to-date. It would be a waste of time to update the cache in every role, so we will update it in a pre task:

- hosts: all
  pre_tasks:
    - name: APT update
      become: true
      changed_when: false
      when: config_apt_update | default(false, true) | bool
      ansible.builtin.apt:
        update_cache: true
  roles:
    - role: cli_tools

In this case we use the built-in "apt" module to update the cache, without installing anything, but apparently, updating the cache will also mean the task will always report a change. To disable that, we add changed_when: false to the task. We also want a way to skip the updater pre task. When you have to run a playbook 10 times in two minutes while you are developing it, updating the cache every time is simply not necessary. We add a condition which will use the new config_apt_update variable. If it is not defined in the inventory file, we use "false" as default value, but you can always override it from command line.

./run.sh playbook-system-base.yml \
  -e config_apt_update=true

I will define it in my inventory, so this is how the global vars section looks like now:

all:
  vars:
    ansible_user: ansible-homelab
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
    config_apt_update: true
    config_lxd_zfs_pool_disks:
      - /dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6

Ansible facts

» Back to table of contents «

There is one more line we need to add to the playbook.

When you run a playbook, as the very first step, Ansible detects devices and collects information for example about networks and the version of the Linux distribution. The collected information will be available through variables and these are the facts. Sometimes you don't need these facts, and you want to speed up the execution of the playbook, especially when you have to run it on 100 servers or on just a couple but very often during development. If that is the case, you can set gather_facts: false in the playbook like this:

- hosts: all
  gather_facts: false
  pre_tasks:
    - name: APT update
      become: true
      changed_when: false
      when: config_apt_update | default(false, true) | bool
      ansible.builtin.apt:
        update_cache: true
  roles:
    - role: cli_tools

If you use roles you didn't write, and you don't want to find out what facts they need, just leave the facts gathering enabled.

Now you may think you understand the difference between the variables we used before and the facts, but in fact, you can also define facts using the set_fact builtin module. So one more important difference is the scope. You can define a variable in a task, but that variable will not be available in the next task. Facts are available everywhere, and you can also cache them, so when you define a fact, run the playbook, remove the definition and rerun the playbook, you can still read the fact from the cache. Of course it depends on the used cache plugin, and the default is memory. So by default, the facts are not available when you run a playbook the second time. If ou want to see how persistent fact caching works, the following example can show it.

Run the following commands in terminal:

export ANSIBLE_CACHE_PLUGIN=jsonfile 
export ANSIBLE_CACHE_PLUGIN_CONNECTION="$PWD/var/cache"

Use the following playbook:

- hosts: localhost
  gather_facts: false
  tasks:
    - ansible.builtin.set_fact:
        cacheable: true
        mytest: hello
    - ansible.builtin.debug:
        var: ansible_facts.mytest

After running the playbook, you will find a file named "localhost" in the folder you specified in the plugin connection.

{
    "mytest": "hello"
}

Then run the following playbook:

- hosts: localhost
  gather_facts: false
  tasks:
    - ansible.builtin.debug:
        var: ansible_facts.mytest

And Ansible will still remember the value of "mytest":

ok: [localhost] => {
    "ansible_facts.mytest": "hello"
}

Ansible role

Ansible role overview

» Back to table of contents «

The new Ansible role will be called "cli_tools". The structure of the role will be the following:

defaults/
- main.yml: The place for default parameter values.
vars/
- main.yml: A file to store helper variables which are not intended to be changed by the user. You can use this file if the alternative is storing the variables in the tasks file, which requires creating a block only for those variables.
tasks/
- main.yml: The default tasks file that we always used
- yq.yml: An additional tasks file which we can refer to and load in the main.yml.
README.md: This is basically the documentation of the role containing everything that helps the user to understand how the role can be used, what it expects to be already installed and so on. We will discuss it in more details later.

Creating a symbolic link

» Back to table of contents «

Most of the packages I install on a Debian-based Linux can be installed from an APT repository, but using the built-in apt module that we already used before is not really interesting, so let's just jump to the interesting part. Sometimes, I just want to have an alias for a command, and that's where I will create a symbolic link like now to point to the pygmentize command. The built-in files module can create a symbolic link if the state field is "link".

    - name: Create "highlight" as a symbolic link to "pygmentize" | Install formatting tools for scripting and user-friendly outputs
      become: true
      ansible.builtin.file:
        state: link
        src: /usr/bin/pygmentize
        dest: "{{ cli_tools_highlight_dest }}"

The destination could have been static, but I wanted to make it changeable, so I will have a default value for that in defaults/main.yml.

Using multiple tasks files

» Back to table of contents «

Installing yq will be complicated, but I don't want to complicate my main tasks file.

The built-in include_tasks module can load another tasks file and expects the name of the file and executes the tasks in it.

- name: Include tasks from another file
  ansible.builtin.include_tasks: file.yml

It can be useful in different situations, but in this case, I didn't want to keep the most complicated installation process in the main file. The main.yml can also be shorter this way. For more details about how this module can be used, don't forget to check the documentation I linked above.

The following code is the part of main.yml in the cli_tools role which shows all the 3 modules I used in the main.yml, and also includes a block. Most of the tasks will be familiar since we used the APT module before and I also shared the symlink part, but the last task is an include.

roles/cli_tools/tasks/main.yml

- name: Install formatting tools for scripting and user friendly outputs
  block:

    - name: APT packages | Install formatting tools for scripting and user-friendly outputs
      become: true
      ansible.builtin.apt:
        name:
          - jq # to handle json files
          - python3-pygments # to highlight codes with "pygmentize"

    - name: Create "highlight" as a symbolic link to "pygmentize" | Install formatting tools for scripting and user-friendly outputs
      become: true
      ansible.builtin.file:
        state: link
        src: /usr/bin/pygmentize
        dest: "{{ cli_tools_highlight_dest }}"

    - ansible.builtin.include_tasks: yq.yml

I didn't use the "name" parameter in the last task, because the tasks in the included file will have names, so it wouldn't really help to understand the role better and wouldn't add more value to the logs either. It was not my idea. I named every single task until I read about this point of view and I agreed. Unfortunately, I don't have a link to the source.

Install the latest yq from GitHub

Using the GitHub API to get the latest release

» Back to table of contents «

We can finally discuss the most interesting part. I want to install "yq" from GitHub, which will require two more default variables in defaults/main.yml:

cli_tools_yq_version:
cli_tools_yq_dest: /usr/local/bin/yq

The version number is empty, which will mean that I want to install the latest version. I tried to find a link directly to the latest release, but it turned out, there was no such link. However, the GitHub API can tell us which one is the latest. If you just want to get the URL to download the latest version, you can try the following in the terminal:

curl -sL https://api.github.com/repos/mikefarah/yq/releases/latest

It will return a json which is too long to show it, but let's see the relevant part:

{
  "html_uri": "https://github.com/mikefarah/yq/releases/tag/v4.40.5",
  "assets": [
    {
      "browser_download_url": "https://github.com/mikefarah/yq/releases/download/v4.40.5/yq_linux_amd64"
    },
    {
      "browser_download_url": "https://github.com/mikefarah/yq/releases/download/v4.40.5/yq_darwin_arm64"
    }
  ]
}

This will be really important, because it has all the information we need, and it has it more than once.

Let's see how you can call the API endpoint from Ansible:

- name: Get latest version info as json
  when: cli_tools_yq_version | default('', true) == ''
  ansible.builtin.uri:
    url: https://api.github.com/repos/mikefarah/yq/releases/latest
  register: _yq_latest

The built-in uri module allows us to call the endpoint and save the json response into a variable. Of course we want to do that only if the requested version number is empty, that's why we compare the version number to an empty string.

Get the version number of the latest release

» Back to table of contents «

In the previous section, you could see that we could get the download url from the json response, which contains the version number, the architecture and also the operating system. The response also shows that these are the only differences in the download URLs. The download URL is the only thing we need, but sometimes we want to specify the version number instead of getting the latest version. So instead of using the above information to filter to the URL that we know exactly how it looks like, we can just build the URL from scratch. The first important part of that URL is the version number, but the version number can also be found in the html_url field, which does not require to list the release files.

Assuming you already have jq on the server, you can run the following:

curl 'https://api.github.com/repos/mikefarah/yq/releases/latest' -s  \
  | jq -r '.html_url' \
  | xargs -- basename \
  | sed 's/^v//'

Output:

4.50.5

We need to the version number in Ansible. We registered the json response in _yq_latest. It will have a property called "json", which is not a string. It is in fact a decoded version of the json string, since The "uri" module recognized json in the HTTP response header. The above bash command can be replaced with the following Jinja template in Ansible:

    _yq_latest_version_number: "{{
      _yq_latest.json.html_url
        | basename
        | regex_replace('^v(.*)', '\\1')
      }}"

We also used a very simple regular expression telling Ansible to remove the leading "v" from the version number. Removing the "v" is not really important. It was just my preference to work with only the numbers.

We now have the latest version number, and we know that we want to use that as the default value and also be able to override it. This is how you do it:

_yq_desired_version_number: "{{ cli_tools_yq_version | default(_yq_latest_version_number, true) }}"

Get the architecture and operating system of the server

» Back to table of contents «

The next important thing after the version number is the release name. The release always starts with "yq_" followed by the operating system and the architecture. We will need the uname command to get name of the operating system (darwin on macOS and linux on Linux) and the arch command to get the CPU architecture. Unfortunately, amd64 can also be called x86_64 and arm64 can also be called aarch64, so let's use sed to fix that.

uname
arch

Output:

Linux
x86_64

While we could use the uname and the arch commands to get the operating system and the CPU architecture in the terminal, we can use facts in Ansible. Since we disabled the fact gathering, we have to use the built-in setup module to get the architecture and the operating system.

- name: Collect architecture facts
  ansible.builtin.setup:
    gather_subset: architecture

After that you can get operating system and the architecture from the ansible_facts variable.

- vars:
    info:
      os: "{{ ansible_facts.system }}"
      arch: "{{ ansible_facts.architecture }}"
  debug:
    var: info

Although I prefer using ansible_facts, so I can search for where I'm using facts, you could use the variables prefixed with ansible_.

- vars:
    info:
      os: "{{ ansible_system }}"
      arch: "{{ ansible_architecture }}"
  debug:
    var: info

Saving helper variables in addition to defaults

» Back to table of contents «

Variables in Ansible can be defined in many places. In a role, we can have defaults, but we can also have variables which are not for changing them (although we could change them too), but only for organizing our templates, so we don't have to define all the variables in the tasks files.

The architecture and the operating system is the two most important pieces of information to build the final URL. We have to convert those to a format that can be used in the download URL.

uname | tr '[:upper:]' '[:lower:]'
arch \
  | sed 's/x86_64/amd64/' \
  | sed 's/aarch64/arm64/'

We converted the nme of the operating system to lowercase, and replaced the architecture with the alternative names. Yes, in this project we support only these two.

Output:

linux
amd64

In Ansible, we will save the templates in vars/main.yml, so it is another folder called "vars/" at the same level as "defaults/".

cli_tools_yq_archs:
  x86_64:  amd64
  amd64:   amd64
  aarch64: arm64
  arm64:   arm64

cli_tools_yq_os: "{{ ansible_facts.system | lower }}"
cli_tools_yq_arch: "{{ cli_tools_yq_archs[ansible_facts.architecture] }}"
cli_tools_yq_release_name: "{{ 'yq_' + cli_tools_yq_os + '_' + cli_tools_yq_arch }}"

This is how we will always get arm64 or amd64. Since we get the OS name from fact, it would also work on macOS. It doesn't mean the whole role would work, since we also use the APT package manager, but you could try to move the yq installation into a separate role. Whether you want to use multiple tasks files or a new role, it's up to you.

The last thing we did was defining the full release name.

Installing the desired version of yq

» Back to table of contents «

We finally have all the information the build the download url:

_url_base: https://github.com/mikefarah/yq/releases/download/
_url: "{{ _url_base }}v{{ _yq_desired_version_number }}/{{ cli_tools_yq_release_name }}"

We can use the following task, but in this case, we choose the built-in get_url module instead of uri.

- name: Install yq
  become: true
  failed_when: _yq_install.status_code not in [200, 304]
  vars:
    _yq_latest_version_number: "{{
      _yq_latest.json.html_url
        | basename
        | regex_replace('^v(.*)', '\\1')
      }}"
    _yq_desired_version_number: "{{ cli_tools_yq_version | default(_yq_latest_version_number, true) }}"
    _url_base: https://github.com/mikefarah/yq/releases/download/
    _url: "{{ _url_base }}v{{ _yq_desired_version_number }}/{{ cli_tools_yq_release_name }}"
  ansible.builtin.get_url:
    url: "{{ _url }}"
    dest: "{{ cli_tools_yq_dest }}"
    owner: root
    group: root
    mode: 0775
    force: true

There is one parameter I have to explain.

force: true

Without this parameter wo couldn't update an already installed yq. It tells Ansible to override the downloaded file.

Skip downloading when the existing version is the desired one

» Back to table of contents «

Previously, we always overwrote the installed version, which required downloading the file every time. To avoid that we need the version of the already installed yq, and do that only if it is already installed.

To find out if the file is already downloaded, we can use the built-in stat module.

- name: Check if {{ cli_tools_yq_dest }} exists
  ansible.builtin.stat:
    path: "{{ cli_tools_yq_dest }}"
  register: _yq_existing_dest_check

Now the boolean _yq_existing_dest_check.stat.exists variable tells you whether it exists or not. In the terminal, you would get the version number like this:

yq --version

Output:

yq (https://github.com/mikefarah/yq/) version v4.40.5

It's not just a version number, so we will use regular expression again, but first we get the version info in Ansible:

- name: Get the version information of the existing yq command
  changed_when: false
  when: _yq_existing_dest_check.stat.exists
  ansible.builtin.command: "{{ cli_tools_yq_dest }} --version"
  register: _yq_existing_version_info

I used the cli_tools_yq_dest parameter so the task will work even if the path of the base folder is missing from the PATHS environment variable.

We need to apply the following filter on the version info:

regex_replace('.*version v(\\d+\\.\\d+\\.\\d+).*', '\\1')

As a template variable:

_yq_existing_version_number: "{{ _yq_existing_version_info | regex_replace('.*version v(\\d+\\.\\d+\\.\\d+).*', '\\1') }}"

We will also need to add the following condition to the task:

  when:
    - not _yq_existing_dest_check.stat.exists or _yq_existing_version_number != _yq_desired_version_number

The final task is below:

- name: Install yq
  become: true
  when:
    - not _yq_existing_dest_check.stat.exists or _yq_existing_version_number != _yq_desired_version_number
  vars:
    _yq_existing_version_number: "{{ _yq_existing_version_info | regex_replace('.*version v(\\d+\\.\\d+\\.\\d+).*', '\\1') }}"
    _yq_latest_version_number: "{{
      _yq_latest.json.html_url
        | basename
        | regex_replace('^v(.*)', '\\1')
      }}"
    _yq_desired_version_number: "{{ cli_tools_yq_version | default(_yq_latest_version_number, true) }}"
    _url_base: https://github.com/mikefarah/yq/releases/download/
    _url: "{{ _url_base }}v{{ _yq_desired_version_number }}/{{ cli_tools_yq_release_name }}"
  ansible.builtin.get_url:
    url: "{{ _url }}"
    dest: "{{ cli_tools_yq_dest }}"
    owner: root
    group: root
    mode: 0775
    force: true

Full yq tasks file

» Back to table of contents «

Now let's see how yq.yml looks like:

roles/cli_tools/tasks/yq.yml

- name: Collect architecture facts
  ansible.builtin.setup:
    gather_subset: architecture

- name: Get latest version info as json
  when: cli_tools_yq_version | default('', true) == ''
  ansible.builtin.uri:
    url: https://api.github.com/repos/mikefarah/yq/releases/latest
  register: _yq_latest

- name: Check if {{ cli_tools_yq_dest }} exists
  ansible.builtin.stat:
    path: "{{ cli_tools_yq_dest }}"
  register: _yq_existing_dest_check

- name: Get the version information of the existing yq command
  changed_when: false
  when: _yq_existing_dest_check.stat.exists
  ansible.builtin.command: "{{ cli_tools_yq_dest }} --version"
  register: _yq_existing_version_info

- name: Install yq
  become: true
  when:
    - not _yq_existing_dest_check.stat.exists or _yq_existing_version_number != _yq_desired_version_number
  vars:
    _yq_existing_version_number: "{{ _yq_existing_version_info | regex_replace('.*version v(\\d+\\.\\d+\\.\\d+).*', '\\1') }}"
    _yq_latest_version_number: "{{
      _yq_latest.json.html_url
        | basename
        | regex_replace('^v(.*)', '\\1')
      }}"
    _yq_desired_version_number: "{{ cli_tools_yq_version | default(_yq_latest_version_number, true) }}"
    _url_base: https://github.com/mikefarah/yq/releases/download/
    _url: "{{ _url_base }}v{{ _yq_desired_version_number }}/{{ cli_tools_yq_release_name }}"
  ansible.builtin.get_url:
    url: "{{ _url }}"
    dest: "{{ cli_tools_yq_dest }}"
    owner: root
    group: root
    mode: 0775
    force: true

Run the final playbook

./run.sh playbook-system-base.yml \
  -e config_apt_update=true

Documenting Ansible roles

» Back to table of contents «

When you write an Ansible role, you can forget about the parameters and how you can use them. You can forget about some requirements which are needed before you use the role. It is a good practice to have a README file in the root folder of the role. If you want to share the role, it is even more important.

The README file could have any structure, but the recommended one is the following markdown structure:


text
role_name
=========

Description

Requirements
------------

List of requirements like the supported operating systems

Role variables
--------------



```yaml
role_variable: value
```



Description of the above variable

Dependencies
------------

List of dependencies like other roles

Example playbook
----------------



```yaml
- hosts: all
  roles:
    - role: role_name
      role_variable: value
```



License
-------

The name of the license

Author information
------------------

Your name or the name of your team and optional email address.

The description part is usually short, but I thought it would be a good idea to describe all the tools that the role would install, so mine is really long. I don't want to share the whole documentation, but you can find it on GitHub.

Conclusion

» Back to table of contents «

This is how a very simple task becomes a very complicated. I wanted to show you what command line tools I usually install on my Linux servers, which become a separate article. In Ansible, it required talking about Ansible facts and organizing our variables better. In my original role, I never overwrote the existing yq binary, and when I needed a new version, I could just remove the binary on the server and rerun the playbook. If you have many servers, it is better to automatically checking whether you have the desired version or not. It also demonstrated what it means to detect the existing state if there is no module to do that for you.

Now that we have a role to install the most important command line tools, we can reuse it later. For example, when we use Ansible to run new virtual machines in which we also want to have these tools and more. Coming soon in a following tutorial.

The final source code of this episode can be found on GitHub:

https://github.com/rimelek/homelab/tree/tutorial.episode.8

rimelek / homelab

Source code to create a home lab. Part of a video tutorial

README

The project contains code written for the tutorial, but you can also use parts of it if you refer to this repository.

Tutorial on YouTube in English: https://www.youtube.com/watch?v=K9grKS335Mo&list=PLzMwEMzC_9o7VN1qlfh-avKsgmiU8Jofv

Tutorial on YouTube in Hungarian: https://www.youtube.com/watch?v=dmg7lYsj374&list=PLUHwLCacitP4DU2v_DEHQI0U2tQg0a421

You can also find an example inventory file in the project root. You can copy that and change the content, so you will use your IP…

View on GitHub

Command line tools I always install on Ubuntu servers

Ákos Takács — Mon, 25 Dec 2023 17:34:13 +0000

Introduction

There are some tools that we almost always install on our servers. Sometimes we install them immediately, and sometimes we realize what is missing later, and install it then. In this post we will discuss the tools I always install on Ubuntu servers. Although the following package names will be for Ubuntu linux, most of the tools can also be installed on other distributions as well and some of them, like jq and yq can be installed on macOS as well.

Common network tools
Python 3 support
Formatting tools for scripting and user-friendly outputs
Packages for downloading files and webpages
Monitoring tools in command line
User manual related packages
Text editors
Conclusion

Common network tools

bridge-utils

» Back to table of contents «

It contains the brctl command which can manage bridge interfaces. The brctl command can be replaced with the "ip" command, but it is useful to have it in case you copy-paste some commands from tutorials where brctl is used.

To list bridge interfaces, you could use the following command for example:

brctl show \
  | awk -F $'\t' '$1 != "" {print $1}' \
  | tail -n +2

Example output:

br-35a7f7573daa
br-40c75c6946c5
br-415f86d29b58
br-85776ec34ce3
br-a39b1276c115
br-be88ce1dc3d8
br-c506658ca645
docker0

net-tools

It contains "netstat" which can also be replaced with another command called "ss" which is part of the "iproute2" package on Ubuntu. A common netstat command is the following to list used ports on which processes are listening:

netstat -tulpn

Output example:

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8952          0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.4.58:53         0.0.0.0:*               LISTEN      -
tcp        0      0 10.17.181.1:53          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:61209         0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.4.58:5432       0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 ::1:8952                :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
udp        0      0 10.17.181.1:53          0.0.0.0:*                           -
udp        0      0 192.168.4.58:53         0.0.0.0:*                           -
udp        0      0 0.0.0.0:67              0.0.0.0:*                           -

iproute2

Usually installed by default on recent Ubuntu distros. It gives you the "ip" command to manage IP addresses and network interfaces. It also contains the "ss" command which can be used instead of "netstat".

The alternative command to the previously mentioned brctl command is the following:

ip -json link show type bridge \
  | jq -r .[].ifname \
  | sort

This specific command requires jq which I will write about later.

If you want to replace the netstat command with the newer "ss" command, you can run the following:

ss -tulpn

As you can see the parameters are the same, but the output will be slightly different:

Netid             State              Recv-Q             Send-Q                          Local Address:Port                           Peer Address:Port             Process
udp               UNCONN             0                  0                                 10.17.181.1:53                                  0.0.0.0:*
udp               UNCONN             0                  0                                192.168.4.58:53                                  0.0.0.0:*
udp               UNCONN             0                  0                              0.0.0.0%lxdbr0:67                                  0.0.0.0:*
tcp               LISTEN             0                  5                                   127.0.0.1:61209                               0.0.0.0:*
tcp               LISTEN             0                  32                                10.17.181.1:53                                  0.0.0.0:*
tcp               LISTEN             0                  256                              192.168.4.58:53                                  0.0.0.0:*
tcp               LISTEN             0                  128                                   0.0.0.0:22                                  0.0.0.0:*
tcp               LISTEN             0                  244                              192.168.4.58:5432                                0.0.0.0:*
tcp               LISTEN             0                  16                                  127.0.0.1:8952                                0.0.0.0:*
tcp               LISTEN             0                  128                                      [::]:22                                     [::]:*
tcp               LISTEN             0                  16                                      [::1]:8952                                   [::]:*

Python 3 support

» Back to table of contents «

Python 3 is recommended usually instead of Python 2. This role also installs common tools which we use in a Python environment:

python3

The main package for Python 3. Sometimes you want a specific version if the APT repository supports multiple versions. Then you could specify that like python3.11 if the exact version number matters.

python3-pip

Pip package manager for Python 3. If you install python3.11 then the package name will be python3.11-pip

python3-venv

The venv module is one of the modules that can create a virtual environment.

Example:

python3 -m venv venv

python3-virtualenv

The virtualenv module is also one that can create a virtual environment. It is probably better known than venv, but you can use very similarly.

python -m virtualenv venv

Formatting tools for scripting and user-friendly outputs

» Back to table of contents «

jq

To handle JSON files and JSON outputs in a script or format and highlight it, jq can be very handy. Many command line tools provide a json output, so you don't have to write a custom parser for a table a list in a terminal. Instead of that, you can use jq to get a specific value from the output or even modify the output. For more information, you can visit https://jqlang.github.io/jq/

Example:

lxc list --format json docker \
  | jq -r .[].created_at

The above command is similar to what we will discuss in the near future in another post. Docker also supports JSON outputs, so jq can be useful in case you are not comfortable with golang templates.

python3-pygments

It will install the "pygmentize" command to highlight contents of files like source codes in the terminal. It isn't required for JSON because of jq, but it could be useful for other kind of outputs.

For example instead of using cat lxd-init.yml to get the content of a yaml file (which we did before in previous chapters), you can use the pygmentize command and have a highlighted, colorized output:

pygmentize lxd-init.yml

If you want to read about the installation of LXD for running virtual machines and containers on Linux, read Creating virtual machines with LXD

Instead of remembering the name "pygmentize" you would probably prefer "highlight", so you can also add a symbolic link to "pygmentize".

ln -s /usr/bin/pygmentize /usr/local/bin/highlight

yq

yq is similar to jq, except it is for YAML files. Let's say you want to get the profile definitions from an lxd init yaml. You can run the following:

yq .profiles lxd-init.yml

Example output:

- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      network: lxdbr0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default

For more information about this command visit https://github.com/mikefarah/yq

Packages for downloading files and webpages

» Back to table of contents «

curl and wget are two popular commands to download files from the internet or just handle API calls. I usually install both of them, so whatever commands I find in a documentation, I can copy and paste. However, I prefer using curl. Curl is also implemented in programming languages like PHP.

Example commands with curl and wget to get the response header and follow redirections:

curl -L http://google.com

wget -O- -S --spider http://google.com

Compress and decompress files

On Linux I usually use "tar" and "gzip" to compress files. Sometimes I need to download a release from GitHub or any file compresses with zip, so I also install "zip" and "unzip".

Monitoring tools in command line

» Back to table of contents «

htop

Probably everyone knows about the "top" command. Htop is similar, but gives us a more user-friendly output. It shows processes using the most resources, how much available resources you have and who runs those processes. For more information, visit https://htop.dev/

btop

Btop is even more advanced than htop. It is almost like a GUI in terminal, and it feels like a dashboard of an airplane. I like it, but when I want to see what processes are using most of my resources, it is usually htop that comes to my mind and not btop, since btop shows more by default than I usually want. For more information see https://github.com/aristocratos/btop.

glances

It has a web-based interface too, but you can use it from terminal. "Glances" shows information about Docker containers too by default, so when it comes to Docker containers, it is really useful. For more information see https://glances.readthedocs.io/en/latest/

I created the screenshot in a virtual machine (which we will create together in a following tutorial) where I have Docker installed. As you can probably see (click on the image if it is too small), there is a "containers" section as well.

User manual related packages

» Back to table of contents «

The "man" command is almost always available on Linux, but sometimes it is missing. Some virtual machine images and container images don't come with this package, so I installed it to make sure I always have it, so I can get the user manual of curl for example.

man curl

Text editors

» Back to table of contents «

nano

A simple text editor in terminal with limited capabilities, but it is very easy to use. I usually prefer this one and I don't care how many people tell me that vim is much better. When I want to edit a simple file quickly like /etc/hosts I don't need any advanced editor and when I need an advanced editor, I almost always have GUI, so I can use VSCode for example.

vim

An advanced text editor in terminal which is very popular. It has many keyboard shortcuts for example to delete multiple lines and other tasks which is not as simple in a terminal as it would be with a GUI. Even if I usually don't need it, I install it, because it could be useful sometimes.

Conclusion

» Back to table of contents «

The list I shared in this post shows what I install on my servers. Those servers are usually debian based like Ubuntu, but whatever Linux distributions I run, I like to use the same tools. For example, I use nano, jq, curl almost every day on my macOS.

I also use Docker and although it has a command line part as well, it is much more than that, so I didn't include it in this list.

Use SOPS in Ansible to read your secrets

Ákos Takács — Tue, 14 Nov 2023 21:08:52 +0000

Introduction

When we are using Ansible, we often need passwords or tokens or any secrets that we don't want to store as plain text and definitely don't want to commit to a git repository. At least not as plain text. Personally, I don't like to commit secrets even if they are encrypted, unless it is required.

I used Ansible Vault for years, and I was suggested to replace it with GPG since we used that too anyway, but somehow I just didn't like it so much, so I kept using Ansible Vault. My problem with Ansible Vault was that when I encrypted a yaml file which contained secrets, the whole file was encrypted including parameter names. Of course there was a solution for that. Creating an unencrypted file without the values and another which was completely encrypted, so you could manage two files instead of one. Or you could encrypt individual variables manually.

Maybe about a year ago, I started to use Secrets Operations (SOPS) and it made everything easier. Not to mention that I can use SOPS without Ansible. In this post I will use SOPS to finally encrypt the sudo password (aka become pass) for the user on the remote server and I will use the Nix package manager to install the required to packages, sops and age.

If you want to be notified about new videos, you can subscribe to my YouTube channel: https://www.youtube.com/@akos.takacs

Before you begin
- Requirements
- Download the already written code of the previous episode
- Have an inventory file
- Activate the Python virtual environment
Why Nix again?
Create a new config file
Generate the age key pair
Create a wrapper script for sops and detect the public key automatically
Encrypt a file
Edit an already encrypted file
Use the secret in the inventory file
Use a dedicated user for Ansible
Run Ansible in a Nix shell
Test if Ansible could read the secret
Don't commit secrets to a git repository
What more you should know
Conclusion

Before you begin

Requirements

» Back to table of contents «

The project requires Nix which we discussed in Install Ansible 8 on Ubuntu 20.04 LTS using Nix
You will also need an Ubuntu remote server. I recommend an Ubuntu 22.04 virtual machine.

Download the already written code of the previous episode

» Back to table of contents «

If you started the tutorial with this episode, clone the project from GitHub:



git clone https://github.com/rimelek/homelab.git
cd homelab

If you cloned the project now, or you want to make sure you are using the exact same code I did, switch to the previous episode in a new branch



git checkout -b tutorial.episode.6b tutorial.episode.6

Have the inventory file

» Back to table of contents «

Copy the inventory template



cp inventory-example.yml inventory.yml

Change ansible_host to the IP address of your Ubuntu server that you use for this tutorial,
and change ansible_user to the username on the remote server that Ansible can use to log in.
If you still don't have an SSH private key, read the Generate an SSH key part of Ansible playbook and SSH keys
If you want to run the playbook called playbook-lxd-install.yml, you will need to configure a physical or virtual disk which I wrote about in The simplest way to install LXD using Ansible. If you don't have a usable physical disk, Look for truncate -s 50G <PATH>/lxd-default.img to create a virtual disk.

Activate the Python virtual environment

» Back to table of contents «



./create-nix-env.sh venv

Optionally start an ssh agent:



ssh-agent $SHELL

and activate the environment with



source homelab-env.sh

Why Nix again?

» Back to table of contents «

There are multiple ways which you can find in the git repository of sops to install it, but I will show you one that isn't there. The reason is that I want to support this project on Linux and macOS and I also want you to use the same version that I use. So if I follow the guide on GitHub, I need to either download a specific binary from GitHub or use different package managers on different platforms. Both would require automatically detecting the platform. Since downloading binaries would also mean that I would need to manually verify checksums and install dependencies and I like to use general solutions, I rather choose Nix. We already use Nix to create a Python virtual environment, so why not?

Create a new config file

» Back to table of contents «

SOPS supports multiple tools for encryption like PGP or age, and we will use age. The first step is to open a Nix shell with pre-installed age so we can use the age-keygen command to generate a keypair. Eventually, we need to run a command like this:



age-keygen -o age/private-key

SOPS will look for this file in specific folder which is different on macOS and on Linux, so we will override that, and regardless of the OS you are using, you can run the same commands, and I can use the same project on macOS and in a Linux virtual machine where I mount the project from the host. In order to use the same key in the terminal and in Ansible, we will need two environment variables:

SOPS_AGE_KEY_FILE
ANSIBLE_SOPS_AGE_KEYFILE

Since I will have multiple scripts using the same config, I will create a config file which will be a simple shell script. Let's save it in the project root and call it config.sh.



# HOMELAB variables
export HOMELAB_VAR_DIR="${HOMELAB_PROJECT_ROOT:-.}/var"

# SOPS variables
export SOPS_AGE_KEY_FILE="$HOMELAB_VAR_DIR/age/key"

# Ansible variables
export ANSIBLE_SOPS_AGE_KEYFILE="$SOPS_AGE_KEY_FILE"

We have a "HOMELAB variables" section for general variables, an "SOPS variables" section for variables that are used by the "sops" executable directly and an "Ansible variables" section for Ansible of course. We also use the HOMELAB_PROJECT_ROOT variable, which is just a simple dot by default, but can be set in each script. We do this only because this way we don't have to support sourcing the config file in different shells and use different ways to determine the project root.

Generate the age key pair

» Back to table of contents «

Now we need a script that runs age-keygen and stores the private key file based on the config file. Let's call it sops-keygen.sh



#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p age
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz

current_dir="$(cd "$(dirname "$0")" && pwd)"
export HOMELAB_PROJECT_ROOT="$current_dir"

source "$HOMELAB_PROJECT_ROOT/config.sh"

parent_dir="$(dirname "$SOPS_AGE_KEY_FILE")"

if [[ ! -e "$parent_dir" ]]; then
  mkdir -p "$parent_dir"
fi

if [[ ! -e "$SOPS_AGE_KEY_FILE" ]]; then
  age-keygen -o "$SOPS_AGE_KEY_FILE"
fi

echo "The private key file is at $SOPS_AGE_KEY_FILE"

It should be executable of course:



chmod +x sops-keygen.sh

And when you execute it,



./sops-keygen.sh

the script will generate the private key, save it and show the location. If you are wondering where the public key is, it is in the same file.



source config.sh
cat $SOPS_AGE_KEY_FILE

The output will be something like this:



# created: 2023-11-12T11:36:39+01:00
# public key: age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e
AGE-SECRET-KEY-1YCKA5MJ44V2YDA8RZ9MKV0YWTDLG8MXFSDEFT9AQ76GQ7005JFQQN0WFN4

Create a wrapper script for sops and detect the public key automatically

» Back to table of contents «

You can copy the public key manually when you need it, but sometimes you want to get it in a script like now. We need a wrapper script for the sops command. In this script we can get the public key and load it into a variable. We don't set this variable in the config file, because we need it only when we run sops. Even then, we would need it only when we want to encrypt a file, not when we decrypt it. I won't overcomplicate this script even more, so I will let the script read the public key every time. The filename will be sops.sh.



#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p sops
#! nix-shell -p age
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz

current_dir="$(cd "$(dirname "$0")" && pwd)"
export HOMELAB_PROJECT_ROOT="$current_dir"

source "$HOMELAB_PROJECT_ROOT/config.sh"

export SOPS_AGE_RECIPIENTS="$(age-keygen -y < $SOPS_AGE_KEY_FILE)"

sops "$@"

And of course we make it executable:



chmod +x ./sops.sh

You can test it with a simple command that shows its version number:



./sops.sh --version

Encrypt a file

» Back to table of contents «

What we need now is a file to encrypt. Let's create secrets.plain.yml in the project root.



become_pass:

And encrypt it:



./sops.sh --encrypt --output secrets.yml secrets.plain.yml

We will get a secrets.yml file like this:



become_pass: null
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNZjVEOTJJZWkwQytML1F6
            QVFJNHNmcVhHbThMZWI2SGJWalArd2c0bkVvCng3QjgvQms4SE9LT1o3Z21DN1Yw
            MVBWcFdnNVV3UGVKNFB4YS85UU9ScWsKLS0tIFNOTkdodkt6aEZMT1pvdGxxVjNm
            M29mL2JmMDI4N1FmbUMxVmpsRW1vMWcKRIxPc0dZv9JcEn2NyZ9OJ6QDerh9VIcw
            rvD0Tyvrbzoc32cxUZMEUGH+tCwFi5eQ212Fehw1jlLh/YYmYPYBEA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2023-11-12T20:21:57Z"
    mac: ENC[AES256_GCM,data:QsdpRH36FdY3Gq01U/4NvuiXt/OAkxQv+GURksOmlJCpEzyjKDXrqJOk6D/ZiuGrwPn48803MFSvojNtvdrLr0k8+sJ58/Kv9u1vf7/fxbZczvW5ohVhH1bfou6ZgqsJyMLu8yp3EbXukQ8hJe9N159HgqIdCkyycSFwOGko5O4=,iv:0Y4cP2mzq9wedRLxxYSBM4GAS/VvvbohKWGICO+IWw0=,tag:dfoEDyC0736bL+aXHEmAcA==,type:str]
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.8.1

The important part for us now is the first line where we can see the become_pass with a null value. Now I like to edit the values from a terminal and I can simply use

Edit an already encrypted file

» Back to table of contents «



./sops.sh secrets.yml

It will open the file in the default text editor like nano or vim and show you only the decrypted content without the rest of the YAML keys. My default editor currently is vim, but I'm not afraid to tell you that I usually prefer nano, although I don't care enough to change it. If I want, I can change it just for this specific command:



EDITOR=nano ./sops.sh secrets.yml

I will use the following new value in the encrypted file:



become_pass: HomeLab2023

After saving the file, the first line in the encrypted view will be like this:



become_pass: ENC[AES256_GCM,data:degiXqRIpLEVdTY=,iv:Loyf/XbEcTq6Z9enDq1ccCFzOYurL4kUa1Js9dpQzgo=,tag:xmhhExQREc6wkY373eW5+g==,type:str]

We can remove the plaintext file:



unlink secrets.plain.yml

Use the secret in the inventory file

» Back to table of contents «

We need to pass this file somehow to Ansible. We could do it multiple ways, but what I prefer is loading the content
in the inventory file. For that we need a lookup plugin called community.sops.sops which is the part ot the collection called community.sops. We can set a variable called sops and load all the secrets from yaml as an Ansible "dict".



all:
  vars:
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"

We also need to use the values somehow like this:



all:
  vars:
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
  hosts:
    ta-lxlt:
      ansible_become_pass: "{{ sops.become_pass }}"

Now as a reminder, this is my current full inventory.yml file:



all:
  vars:
    ansible_user: ansible-homelab
    config_lxd_zfs_pool_disks:
      - /dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
  hosts:
    ta-lxlt:
      ansible_host: 192.168.4.58
      ansible_ssh_private_key_file: ~/.ssh/ansible
      ansible_become_pass: "{{ sops.become_pass }}"

Use a dedicated user for Ansible

» Back to table of contents «

You might have noticed that I changed the value of ansible_user from ta to ansible-homelab. I did it, so I don't need to change my user's password and I can still show you the secrets. It is also useful to have a dedicated user for Ansible when you have a CI/CD pipeline, and you don't want to share your password with others who can read the secrets. I created the user with the following command:



pass="HomeLab2023"
salt="sugar"
useradd \
  --shell /bin/bash \
  --create-home \
  --groups sudo \
  --password "$(openssl passwd -6 -salt "$salt" "$pass" )" \
  ansible-homelab

If you don't have openssl or just prefer to create a user manually, that's fine too, just make sure you add the user to the sudo group on Ubuntu, so the user can become root. After all, this is why we store the become pass (sudo password) in a secret.

Run Ansible in a Nix shell

» Back to table of contents «

Now there is one script that we already have, but needs to be changed. We used Nix to download sops, but that means Ansible needs to run in a Nix shell. We had this in our original run.sh in the project root:



ansible-playbook \
  -i inventory.yml \
  --ask-become-pass \
  "$@"

We don't want that script to force Ansible to ask for the become pass so that line must be removed, and we need the usual shebang lines for Nix and also our config parameters. The new run.sh will be this:



#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p sops
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz

source config.sh

ansible-playbook \
  -i inventory.yml \
  "$@"

Test if Ansible could read the secret

» Back to table of contents «

Now let's run the hello playbook which requires root privileges, but we need to change the original destination path of the hello-world.txt. Otherwise, Ansible would not need to copy again, and it could work even if the sudo password is incorrect.



./run.sh playbook-hello.yml -e hello_world_dest=/opt/hello-world-$(date +%s).txt

As you can see, we can override a role parameter from terminal. Unless this command failed, our secret worked.

Don't commit secrets to a git repository

» Back to table of contents «

We have one more job before we say goodbye. As I stated at the beginning of this post, I prefer not to commit these encrypted secrets either, so I add secrets.yml to gitignore. The other file I definitely don't want to commit is the private key so let's add the following two lines to gitignore:



/var
/secrets.yml

My full gitignore looks like this now:



/venv
/venv-linux
/inventory.yml
/var
/secrets.yml

# for macOS
/.DS_Store

What more you should know

» Back to table of contents «

In this tutorial I used a bit different way than what you can find in documentations. That made the commands easier, but also more complicated at the same time. sops supports passing the public key as an argument like sops --encrypt --age age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e, but I used the environment variable instead. Why? Because that way you can still have the option to override it from terminal, although this tutorial is not for production systems but for a private home lab, a playground, so you will probably not need to change it or add multiple public keys which means multiple recipients which would support multiple private keys to decrypt the secret file. You can find everything in the documentations and I recommend reading it and playing with sops and age, so you can discover more like how you can use age with a hardware key like YubiKey to decrypt files. For using YubiKey, you need a plugin called "age-plugin-yubikey".

Conclusion

» Back to table of contents «

We can now use secrets, but that is completely optional. If you want to pass test passwords like "abc" or "password" as plaintext on a machine where you have nothing to protect like a virtual machine which you just created for playing with Ansible or test some commands, that's fine. Then you don't need to use the lookup plugin in your inventory file.

I have to note again that this series is for creating a home lab, a local environment to develop and learn and not for a production environment. In a production environment you should always encrypt secrets. Don't forget that SOPS is just one tool so be prepared for other options too when you have to use a keyserver for example in a team.

The final source code of this episode can be found on GitHub:

https://github.com/rimelek/homelab/tree/tutorial.episode.7

rimelek / homelab

Source code to create a home lab. Part of a video tutorial

README

The project contains code written for the tutorial, but you can also use parts of it if you refer to this repository.

Tutorial on YouTube in English: https://www.youtube.com/watch?v=K9grKS335Mo&list=PLzMwEMzC_9o7VN1qlfh-avKsgmiU8Jofv

Tutorial on YouTube in Hungarian: https://www.youtube.com/watch?v=dmg7lYsj374&list=PLUHwLCacitP4DU2v_DEHQI0U2tQg0a421

You can also find an example inventory file in the project root. You can copy that and change the content, so you will use your IP…

View on GitHub

DEV Community: Ákos Takács

History of GPUs in the context of containers until 2025

Introduction

Table of contents

The very beginning

Main contributors to the early history of GPUs

GPU not just for graphics

Containers and GPUs

Introducing GPU support in WSL and Docker Desktop

Supporting AI workloads and using AI at Docker, Inc

Conclusion

Sources

Error message: Is the Docker daemon running?

Introduction

Table of contents

The meaning of the error message

Quick guide for the impatient

Problems and solutions in details

Find out which Docker variant you have

Docker runtime contexts

Running remote Docker daemons

Check if the Docker daemon is actually running

Docker CE status on Linux running under Systemd

Docker as a Snap package status

Docker CE status in Docker Desktop

Check if the Docker unix domain socket exists

Check if you have permission to the socket

Make sure you are using the right socket

I got "is the docker daemon running", but I followed the official installation guide

Conclusion

Which Docker variant am I using and where is the daemon running?

Introduction

Table of contents

Docker Engine on Linux

The number of dockerd processes running directly on Linux

Docker as a Snap package

Docker installed using the default package manager of the Linux distribution

Docker on Debian-based Linux distributions

Docker on Red Hat-based Linux distributions

Find non-snap dockerd processes on Linux

Docker Desktop

Docker Desktop on Linux

Docker Desktop on macOS

Docker Desktop on Windows

Docker Contexts

Remote Docker daemon

Conclusion

Using gVisor's container runtime in Docker Desktop

Introduction

Table of contents

Download runsc into Docker Desktop

Configure the Docker daemon to use the runtime

Download and install runsc in one step

Testing runsc after the installation

Reloading the internal daemon config to support Windows

Possible error messages

Conclusion

Comparing 3 Docker container runtimes - Runc, gVisor and Kata Containers

Introduction

Table of contents

The chosen 3 runtimes

Introduce the runtimes

runc

kata-runtime

runsc

Comparing the runtimes

Checking differences in practice

Resource handling and limitations of the Kata runtime

Conclusion

You run containers, not dockers - Discussing Docker variants, components and versioning

Introduction

Table of contents

What I call Docker

Docker as a company

Docker as a software

The beginning of Docker as a software and where we are now

Two main ways of running the Docker daemon in terms of isolation

Docker running in a virtual machine

Docker in a VM in general

Docker in a VM using Docker Desktop