Ever wondered what happens after running docker-compose up? IP tables, DNS, and virtual networks are just vague buzzwords for you? Well, it's your lucky day! Follow along and learn some new stuff that will help you to better understand Docker magic ✨🧑🎓
Warning: This is a beginners post. If you already have a good grasp of networking aspects this might not be for you:)
The Setup
If you have done literally anything with Docker Compose, this will look very similar to you:
services:
web:
build: .
ports:
- "8000:8000"
db:
image: postgres
This is basically the hello-world of Docker. One locally defined web service that is exposed on port 8000 and one internal database. You will probably also know that by writing 8000:8000, the internal port 8000 of the container is also exposed at your host port 8000. This will allow you to access the web service by opening your browser and navigating to http://localhost:8000. While the postgres database is somehow not reachable for you, it is reachable from the web container. So let's start with the first of many WHAT?! and dig a bit deeper. What actually happens when we run docker-compose up?
PS: You can also follow a long by cloning this Github repository and executing the commands as we go. This should work for MacOS and Linux, no guarantee for Windows 😬
Step 1. docker-compose up
When you run docker-compose up, 3 things happen:
- A virtual network gets created called
dockernetworking_default(or whatever your working directory is called + "_default") - Both
webanddbcontainer start and get added to the network -
webalso makes the port8000available on your host
This sounds easy enough, but I am asking the same thing again: WHAT?!
What does it mean to create a network? How do we add container to a network? And why do we even need to make ports available to the host, its all on the same computer right?!
Step 2. Creating a "network"
When you run docker-compose up a so called "bridge network" that are isolated on only your local machine, meaning that no other hosts can access them (the alternative would be an overlay network).
(docker-compose up does the equivalent to docker network create -d bridge my-bridge-network)
Each container in the network gets a unique IP address and name that they can be reached on.
Let's take a closer look at what is going on. After running docker-compose up, execute docker network ls. This will list all of your docker networks, one line should look like this:
You can then "inspect" it (print the configuration) using
docker inspect docker-networking_default | jq (jq is just a nice add-on to add json syntax highlighting)
This will then print this monster of a json string:
[
{
"Name": "docker-networking_default",
"Id": "d968443acaf789d7d2e9ae18a82371f663baa3019288b18a13c5f38c432aa5df",
"Created": "2023-12-15T17:25:27.011248Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.21.0.0/16",
"Gateway": "172.21.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"7dc4807614c7a4660b9bd34f23dc91610afcdffad8b39af288b9160c3ce34e6e": {
"Name": "docker-networking-web-1",
"EndpointID": "25250386497a2d3093ac133ab0892595128c5734eedb8e039744deccfa1c5c06",
"MacAddress": "02:42:ac:15:00:03",
"IPv4Address": "172.21.0.3/16",
"IPv6Address": ""
},
"ac105fd2a292cda4136aedd0d00c88635f2a140ba3e2359c7e0267514ed9c8b5": {
"Name": "docker-networking-db-1",
"EndpointID": "e26c41754ad5016b7fd9ea0777cbbaa27546209cc8c610db7648e5ed33189538",
"MacAddress": "02:42:ac:15:00:02",
"IPv4Address": "172.21.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "docker-networking",
"com.docker.compose.version": "2.23.0"
}
}
]
There is a lot of information here, but what I want to focus on are two important lines:
{
"Subnet": "172.21.0.0/16",
"Gateway": "172.21.0.1"
}
This means that all your container IP addresses will be in the 172.21.0.0/16 subnet, or in other words between 172.21.0.1 - 172.21.255.254. Although the first one at 172.21.0.1 will be reserved for your gateway. You can confirm this by opening your web containers shell with docker exec -ti docker-networking-web-1 sh and executing hostname -I. This will print 172.21.0.3 or 172.21.0.2! Btw, the gateway is (as the name suggests) the piece that is connecting the virtual docker network with your hosts network.
For the same reason why we use domain names in the internet instead of IP adresses, each container also gets a domain name that can be used instead of the IP adress (which can also change!)
For example, if you go into your web container with
docker exec -ti docker-networking-web-1 sh you can run ping db:
You should see that the domain name db gets translated to 172.21.0.2! Awesome right, but WHAT?! how does your container know that?!
3. DNS in Docker
Something that I didn't tell you before is that not just the gateway IP is reserved, 172.21.0.1 is also reserved for dockers internal DNS server.
If you once again go into your web container with
docker exec -ti docker-networking-web-1 sh
you can run
dig db
to see how the hostname db gets resolved by your DNS.
# dig db
; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> db
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37963
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;db. IN A
;; ANSWER SECTION:
db. 600 IN A 172.21.0.2
;; Query time: 1 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Fri Dec 15 18:32:12 UTC 2023
;; MSG SIZE rcvd: 38
To be fair, this is fairly unreadable. But the only thing we really care about are two lines. First, we see that there is an A record for db pointing at 172.21.0.2, which (who would have thought!) is also the IP address of our database container.
db. 600 IN A 172.21.0.2
Below that, we see that the DNS server we used is at 127.0.0.11#53, which is the default local DNS server.
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
4. Exposing Ports
Ever wondered why you need to map your container port to your host port? The abstract and easy answer is that the network of your containers is isolated from the one of your host. This is done for security reasons but also removes side-effects from weird setups you might have. So in order to connect from your host to a docker container, there needs to be some sort of translation layer in-between. Luckily, this is something that the docker daemon handles for you! If you start your containers with docker-compose up and you mapped some ports, you can use this command:
lsof -i -P -n | grep LISTEN to find what applications are listening on what ports. Somewhere you will find the host port that you declared. This process will re-route the traffic to the "hidden" docker network. Pretty awesome right? If you want to know more, I suggest reading more into how virtual networks and network isolation in linux work!
5. Exposing Containers to the Internet
Docker networking usually stops on your machine. If you want to expose your apps to the internet you can either use a reverse-proxy like Caddy or a reverse-tunneling tool like ngrok or Livecycle. Tools like caddy are great for production-ready and deployed apps, while something like Livecycle is more for a prototyping/collaboration use-case. While this is of course not necessary, at some point you will have to share your app with the world:)
I previously wrote about how I use Livecycle for Hackathons, but the TL;DR goes like this:
- Install the Livecycle Docker Desktop extension
- Click on "share" in the Docker Desktop Livecycle tab
- Your locally hosted docker-compose apps will now be tunneled to a publicly available web server with a random domain.
- If you shutdown your compose app, the tunnel will go down as well!
Conclusion
There are so many things that I would love to cover and also go way deeper into technical OS-level details. But to keep this brief and somewhat entertaining I am stopping here. If you have any questions or suggestions on what to cover next, please let me know in the comments! As always, I hope you learned something new:)
Cheers, Jonas
Top comments (2)
Nice - Thanks for the insights :D
🫡