- What is a Container?
- Docker Setup
- Container Repositories
- Using Images
- Persistent Storage
- Docker Compose
- Creating Images
What is a Container?
Containers virtualize an operating system and connect to the underlying kernel of a computer which allows each container to define system dependencies needed to run the code they contain and isolates them from other containers and the host operating system. This is similar to how Virtual Machines isolate software but VMs utilize a Hypervisor and abstract the hardware layer. This is why a linux VM can run on windows but a linux container needs to run on a device that has a linux kernel. Virtual Machines require more resources, management and are slower to boot up where containers in contrast are very quicker to start and require limited resources to run. In most production setups, Virtual Machines will be provisioned and Containers will be ran on the Virtual Machines which allows for a lot of flexibility.
Docker Setup
The most popular container solution and one we will be using is Docker and containers are commonly referred to as Dockers even if Docker is not the container runtime. If you are using a Mac or PC you can install Docker Desktop and if you are using a linux server then you can Install Docker and Docker Compose directly. The VS Code Docker Extension is also useful for visualizing docker resources.
š Docker Desktop is free for personal use but requires a paid subscription for enterprises. A free open source alternative is Podman and Podman Compose which uses the same core syntax as the Docker tools and can be used as a near drop-in replacement.
Container Repositories
Much like code is committed to a code repository, container images are committed to a container repository and versioned with tags. The main container repository is DockerHub where most popular software can be found. A good example container repository to take a look at is the popular web server and reverse proxy NGINX which shows an overview of how to use the image along with tags for the various versions. A common pattern for container image tags is appVersion-operationSystem
so in the case of NGINX using the image nginx:1.25-alpine
indicates we are using NGINX version 1.25 running on an alpine linux image.
Using Images
In order to first interact with an image once you have docker installed you will need to pull it from a container repository.
$ docker pull nginx:1.25-alpine
Digest: sha256:16164a43b5faec40adb521e98272edc528e74f31c1352719132b8f7e53418d70
Status: Downloaded newer image for nginx:1.25-alpine
docker.io/library/nginx:1.25-alpine
Check your local images with the image command.
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx 1.25-alpine fa0c6bb79540 4 weeks ago 43.4MB
Run the local container image giving it a name and mapping our local computer's port 8080
to the container port 80
.
š” Running with the
-d
option will run it detached as a background process
$ docker run --name myServer -p 8080:80 nginx:1.25-alpine
2023/09/24 10:50:35 [notice] 1#1: using the "epoll" event method
2023/09/24 10:50:35 [notice] 1#1: nginx/1.25.2
2023/09/24 10:50:35 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)
2023/09/24 10:50:35 [notice] 1#1: OS: Linux 6.3.13-linuxkit
2023/09/24 10:50:35 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/09/24 10:50:35 [notice] 1#1: start worker processes
2023/09/24 10:50:35 [notice] 1#1: start worker process 30
2023/09/24 10:50:35 [notice] 1#1: start worker process 31
2023/09/24 10:50:35 [notice] 1#1: start worker process 32
2023/09/24 10:50:35 [notice] 1#1: start worker process 33
Navigate to http://localhost:8080 and you should now see the NGINX welcome page.
The server can be stopped with ctrl+c
and the stopped container can be viewed by appending the -a
flag.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a709dbb8bbce nginx:1.25-alpine "/docker-entrypoint.ā¦" 2 minutes ago Exited (0) 3 seconds ago myServer
Before re-running a container with the same name it will need to be removed and you can do this via its Container ID
or Name
, we will use the name since it is easy to reference.
$ docker rm myServer
myServer
Persistent Storage
Containers are ephemeral and when they are running they may create some files on disk but when they are shutdown and deleted all of those files are also deleted and the next time the container starts they will need to be re-created. The container may also require files when it initially starts such as a configuration file or directory of some sort. This is where persistent storage comes into play for mapping existing files to containers and/or persisting files after a container is shutdown and deleted. The main options that are used are Bind Mounts and Volumes.
Bind Mounting Files
Running the container like we just did is useful for initial testing but anyone using a web server uses it to host custom files or services. Create the following nginx.conf
and index.html
files and we will map them from our local filesystem into our container with a Bind Mount.
š nginx.conf
worker_processes 3;
error_log /dev/stdout info;
events {
worker_connections 2048;
}
http {
include /etc/nginx/mime.types;
server {
listen 8000;
location / {
root /www/data;
try_files $uri /index.html;
}
}
}
š index.html
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
</head>
<body>
<h1>This is running on a containerized NGINX Server!</h1>
<p>Hey š</p>
</body>
</html>
Run the NGINX container with volume flags -v local_file_path:container_file_path
using our custom config and HTML bind mounted to the container.
$ docker run --name myServer \
-p 8080:8000 \
-v ./nginx.conf:/etc/nginx/nginx.conf \
-v ./src/index.html:/www/data/index.html \
nginx:1.25-alpine
2023/09/24 11:26:59 [notice] 1#1: using the "epoll" event method
2023/09/24 11:26:59 [notice] 1#1: nginx/1.25.2
2023/09/24 11:26:59 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)
2023/09/24 11:26:59 [notice] 1#1: OS: Linux 6.3.13-linuxkit
2023/09/24 11:26:59 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/09/24 11:26:59 [notice] 1#1: start worker processes
2023/09/24 11:26:59 [notice] 1#1: start worker process 30
2023/09/24 11:26:59 [notice] 1#1: start worker process 31
2023/09/24 11:26:59 [notice] 1#1: start worker process 32
192.168.65.1 - - [24/Sep/2023:11:27:30 +0000] "GET / HTTP/1.1" 200 172 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
Navigate back to http://localhost:8080 and you should now see our custom HTML page being served!
Volumes
The NGINX container is using bind mounts to map local files to the container but when the container is generating data such as when it is running a database the preferred method to persist data which does not rely on the host file system structure is using Volumes. Volumes are managed by docker and are therefor easier to create without needing to know a good host path to mount to. This is an example of running a MongoDB which uses a Docker volume to persist data.
$ docker run --name myDataBase \
-p 27017:27017 \
-v my_db_data:/data/db \
mongo:7.0.1
{"t":{"$date":"2023-09-24T11:25:58.117+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"a6160bdf-566c-4c64-b848-a6f889c5634b"}},"namespace":"config.system.sessions","index":"lsidTTLIndex","ident":"index-6-8756480188533464000","collectionIdent":"collection-4-8756480188533464000","commitTimestamp":null}}
Attach an interactive shell to the live container to test adding a document to the database.
$ docker exec -it myDataBase bash
root@c0afb87902e3:/# mongosh
test> use myDb
switched to db myDb
myDb> db.myCollection.insertOne({"a": "b"})
{
acknowledged: true,
insertedId: ObjectId("65156478b3f3f1b9c0d3c0a3")
}
The data has been inserted and we can verify that the volume has been created with the docker volume
command.
$ docker volume ls
DRIVER VOLUME NAME
local my_db_data
Make sure the volume persists data even after we stop, delete and restart it.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0afb87902e3 mongo:7.0.1 "docker-entrypoint.sā¦" 5 minutes ago Exited (0) 19 seconds ago myDataBase
$ docker rm myDataBase
myDataBase
$ docker run --name myDataBase \
-p 27017:27017 \
-v my_db_data:/data/db \
mongo:7.0.1
{"t":{"$date":"2023-09-24T11:25:58.117+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"a6160bdf-566c-4c64-b848-a6f889c5634b"}},"namespace":"config.system.sessions","index":"lsidTTLIndex","ident":"index-6-8756480188533464000","collectionIdent":"collection-4-8756480188533464000","commitTimestamp":null}}
$ docker exec -it myDataBase bash
root@8b78d1da358d:/# mongosh
test> use myDb
switched to db myDb
myDb> db.myCollection.findOne({"a": "b"})
{ _id: ObjectId("65156478b3f3f1b9c0d3c0a3"), a: 'b' }
Docker Compose
Running dockers from the command line with docker run
is useful for quick testing but not commonly used in practice. The better way to run one or more dockers on a single server with all of the changes and requirements documented is to use Docker Compose. Docker Compose uses a YAML
spec to define which containers to run with their required configurations and aligns with Infrastructure as Code (IaC) best practices since it can be committed to a code repository.
Docker Compose Spec
Create a docker-compose.yml
file which will run the same NGINX container setup as the previous docker run
command.
š docker-compose.yml
services:
server:
image: nginx:1.25-alpine
container_name: myServer
restart: always
ports:
- 8080:8000
volumes:
- type: bind
source: ./nginx.conf
target: /etc/nginx/nginx.conf
- type: bind
source: ./src/index.html
target: /www/data/index.html
Using Docker Compose
Within the directory you created the docker-compose.yml
file run the following command to start the container.
$ docker-compose up
[+] Running 2/0
ā Network containers_default Created 0.0s
ā Container myServer Created 0.0s
Attaching to myServer
myServer | 2023/09/24 11:46:02 [notice] 1#1: using the "epoll" event method
myServer | 2023/09/24 11:46:02 [notice] 1#1: nginx/1.25.2
myServer | 2023/09/24 11:46:02 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)
myServer | 2023/09/24 11:46:02 [notice] 1#1: OS: Linux 6.3.13-linuxkit
myServer | 2023/09/24 11:46:02 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
myServer | 2023/09/24 11:46:02 [notice] 1#1: start worker processes
myServer | 2023/09/24 11:46:02 [notice] 1#1: start worker process 30
myServer | 2023/09/24 11:46:02 [notice] 1#1: start worker process 31
myServer | 2023/09/24 11:46:02 [notice] 1#1: start worker process 32
Navigate back to http://localhost:8080 and we can see our same custom NGINX server is being ran.
Break the process with ctrl+c
and try running the container as a background process.
$ docker-compose up -d
[+] Running 1/1
ā Container myServer Started
This is now running in the background as a detached process and can be verified with the docker ps
command
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39b33b52f96b nginx:1.25-alpine "/docker-entrypoint.ā¦" 2 minutes ago Up 45 seconds 80/tcp, 0.0.0.0:8080->8000/tcp myServer
In order to stop a detached docker compose container run the stop
command in the same directory that the docker-compose.yml
file is in.
$ docker-compose stop
[+] Stopping 1/1
ā Container myServer Stopped
To stop and remove the container use down
.
$ docker-compose down
[+] Running 2/1
ā Container myServer Removed
ā Network containers_default Removed
Multiple Services
The Docker Compose spec allows you to define all of your required containers as separate services
within the same file. Lets add our previous mongoDb volume example to this same file.
š” We are using the docker compose long syntax for volumes
š docker-compose.yml
services:
server:
image: nginx:1.25-alpine
container_name: myServer
restart: always
ports:
- 8080:8000
volumes:
- type: bind
source: ./nginx.conf
target: /etc/nginx/nginx.conf
- type: bind
source: ./src/index.html
target: /www/data/index.html
db:
image: mongo:7.0.1
container_name: myDataBase
restart: always
ports:
- 27017:27017
volumes:
- type: volume
source: my_db_data
target: /data/db
volumes:
my_db_data:
We can bring both of these containers up with the same one command now.
$ docker-compose up
[+] Running 1/0
ā Container myDataBase Created
Attaching to myDataBase, myServer
myServer | 2023/09/24 11:57:11 [notice] 1#1: nginx/1.25.2
myServer | 2023/09/24 11:57:11 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)
myServer | 2023/09/24 11:57:11 [notice] 1#1: OS: Linux 6.3.13-linuxkit
myServer | 2023/09/24 11:57:11 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
myServer | 2023/09/24 11:57:11 [notice] 1#1: start worker processes
myServer | 2023/09/24 11:57:11 [notice] 1#1: start worker process 30
myServer | 2023/09/24 11:57:11 [notice] 1#1: start worker process 31
myServer | 2023/09/24 11:57:11 [notice] 1#1: start worker process 32
myDataBase | {"t":{"$date":"2023-09-24T11:57:11.278+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
If you need to start or stop only one of the containers then you can use the same commands and just add the service name of the container you want to use.
$ docker-compose up server
[+] Running 2/0
ā Network containers_default Created
ā Container myServer Created
Attaching to myServer
myServer | 2023/09/24 11:59:51 [notice] 1#1: using the "epoll" event method
myServer | 2023/09/24 11:59:51 [notice] 1#1: nginx/1.25.2
myServer | 2023/09/24 11:59:51 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)
myServer | 2023/09/24 11:59:51 [notice] 1#1: OS: Linux 6.3.13-linuxkit
myServer | 2023/09/24 11:59:51 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
myServer | 2023/09/24 11:59:51 [notice] 1#1: start worker processes
myServer | 2023/09/24 11:59:51 [notice] 1#1: start worker process 30
myServer | 2023/09/24 11:59:51 [notice] 1#1: start worker process 31
myServer | 2023/09/24 11:59:51 [notice] 1#1: start worker process 32
Environment Variables
Environment variables are used in all applications and they allow you to make applications more flexible to run in multiple environments and keep secrets out of source code where they could become a security risk. There are a few ways to use environment variables with docker compose files, I mainly use a couple to make the compose spec more flexible by using ${VAR}
variable substitution syntax or by passing external environment variables directly to the container.
Compose Spec variable substitution
In the previous Mongo example spec we could create a .env
file in the same directory and define a TAG
environment variable. At this point we can update the spec to use ${TAG}
as the version which docker compose will automatically substitute with 6.0.1
.
š .env
MONGO_TAG=7.0.1
š docker-compose.yml
services:
...
db:
image: mongo:${MONGO_TAG}
container_name: myDataBase
restart: always
ports:
- 27017:27017
volumes:
- type: volume
source: my_db_data
target: /data/db
...
š”
.env
is used by default but the--env-file
flag can be passed to the docker compose command to specify which env file to use$ docker-compose --env-file .env.dev up
Passing variables to containers
In the mongo example we are running the database without a login which should never be done in production. So we can set an initial username and password with MONGO_INITDB_ROOT_USERNAME
and MONGO_INITDB_ROOT_PASSWORD
environment variables. We should not add these into the spec directly since they would be exposed when we commit it to our code repository. This is where we can use the env_file
option to tell docker compose to pass the variables from our .env
file directly into the Mongo container.
š .env
MONGO_TAG=7.0.1
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD=mySecureDbPassword1!
š docker-compose.yml
services:
...
db:
image: mongo:${MONGO_TAG}
container_name: myDataBase
restart: always
ports:
- 27017:27017
env_file:
- .env
volumes:
- type: volume
source: my_db_data
target: /data/db
...
Creating Images
We have covered how to use popular docker images such as NGINX and Mongo but for containerizing custom software we will need to create our own docker images.
Dockerfile
The way to define how docker images are created is to use a Dockerfile
which typically inherits FROM
a base image and then adds our custom code and system dependencies. Lets take a look at how we can create a simple containerized python script which makes an HTTP call with the requests
library. Create the following example.py
and Dockerfile
files in the same directory.
š example.py
import requests
r = requests.get("https://jsonplaceholder.typicode.com/todos/1")
r.raise_for_status()
print(r.json())
š Dockerfile
FROM python:3.11-slim-bookworm
WORKDIR /app
COPY example.py .
# Install dependencies
RUN pip install --upgrade pip \
&& pip install requests
ENTRYPOINT [ "python3", "example.py" ]
-
FROM
- This defines which image to inherit from, In this case we are using a Python image with the Python 3.11 version running on a slim version of the Bookwork version of Debian linux. Image definitions can be viewed from the DockerHub TAG link such as the python:3.11-slim-bookworm and official images like this one typically have pretty complicated definitions in order to get them highly optimized.
-
WORKDIR
- This defines which directory to use, otherwise it will do everything at the root
/
directory.
- This defines which directory to use, otherwise it will do everything at the root
-
COPY
- This line copies our local file into the Docker container.
-
RUN
- This line runs any executable shell command such as for installing dependencies in this case and you can have many different
RUN
commands to install things as needed. You can also run multiple commands with a singleRUN
statement with&&
in between them.
- This line runs any executable shell command such as for installing dependencies in this case and you can have many different
-
ENTRYPOINT
- This defines what command to run when the container starts and allows for additional command line arguments to be passed in
Building a Custom Image
Now that we have our Dockerfile
and example.py
Python script we need to build the local docker image.
$ docker build . -t py-todos:1.0
[+] Building 8.1s (10/10) FINISHED docker:desktop-linux
...
=> [internal] load metadata for docker.io/library/python:3.11-slim-bookworm 1.6s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [1/4] FROM docker.io/library/python:3.11-slim-bookworm@sha256:edaf703dce209d774af3ff768fc92b1e3b60261e76021 3.3s
...
=> [2/4] WORKDIR /app 0.1s
=> [3/4] COPY example.py . 0.0s
=> [4/4] RUN pip install --upgrade pip && pip install requests 3.0s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:342db88be1b9f09b3aa0e85c52590d7dc8acb59259cd02eaaea955255c5acefd 0.0s
=> => naming to docker.io/library/py-todos:1.0 0.0s
We can verify that our image was built successfully with docker image commands.
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
py-todos 1.0 342db88be1b9 About a minute ago 171MB
Running a Custom Image
Now that we have our custom image built we can run it the same way that we have ran the NGINX or Mongo image.
$ docker run py-todos:1.0
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}
Pushing an image to a Repository
We have successfully containerized an application! š This is very useful but the container image currently only exists on our local laptop so in order to share the image it needs to be pushed to a container repository. The most popular container repository is DockerHub, create an account, login and then create a Personal Access Token so we can push our images to a repository.
$ docker login
Log in with your Docker ID or email address to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com/ to create one.
You can log in with your password or a Personal Access Token (PAT). Using a limited-scope PAT grants better security and is required for organizations using SSO. Learn more at https://docs.docker.com/go/access-tokens/
Username: dpills
Password:
Login Succeeded
After logging in we can update our local image tag with a tag containing the remote container repository which will have the syntax your_username/container_image_name
.
š The dockerhub container repositories are public by default so always be careful that you do not push any sensitive information in the container
$ docker image tag py-todos:1.0 dpills/py-todos:1.0
$ docker image push dpills/py-todos:1.0
The push refers to repository [docker.io/dpills/py-todos]
66acad84a3a9: Pushed
350eed7dde11: Pushed
755eced23209: Pushed
96c02c33d5bb: Mounted from library/python
02436e886926: Mounted from library/python
157a5fb7cdeb: Mounted from library/python
c4d9cc0a5063: Mounted from library/python
311627f8702d: Mounted from library/python
1.0: digest: sha256:7909c0c054aa0f7deacad0871dc07a98467a0d0c97458d6660ae46b40682c63b size: 1994
The image should now show up with the container repo in your local images and also be pushed up to the centralized container repo.
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
dpills/py-todos 1.0 342db88be1b9 16 minutes ago 171MB
py-todos 1.0 342db88be1b9 16 minutes ago 171MB
Going forward for any container which is intended to be pushed to a container repo you can just build it with the repository path from the start.
$ docker build . -t dpills/py-todos:1.1
[+] Building 0.5s (9/9) FINISHED docker:desktop-linux
...
=> [internal] load metadata for docker.io/library/python:3.11-slim-bookworm 0.5s
=> [1/4] FROM docker.io/library/python:3.11-slim-bookworm@sha256:edaf703dce209d7 0.0s
...
=> CACHED [2/4] WORKDIR /app 0.0s
=> CACHED [3/4] COPY example.py . 0.0s
=> CACHED [4/4] RUN pip install --upgrade pip && pip install requests 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:342db88be1b9f09b3aa0e85c52590d7dc8acb59259cd02eaaea95 0.0s
=> => naming to docker.io/dpills/py-todos:1.1 0.0s
Multi-Architecture Images
Docker builds are Architecture specific so if you are using an Apple Silicon Macbook for example then the images will be built for ARM64
CPUs but most servers use AMD64
CPUs. In order to create a multi-architecture build you will need to use buildx
. Once we start this build we can see it go through the same steps for each architecture. You can also use the --push
flag to automatically push it to the container repo after building.
$ docker buildx create --use
$ docker buildx build --push --platform linux/amd64,linux/arm64 . -t dpills/py-todos:1.1
[+] Building 79.6s (17/17) FINISHED docker-container:unruffled_shockley
=> [internal] booting buildkit 4.6s
=> => pulling image moby/buildkit:buildx-stable-1 4.2s
=> => creating container buildx_buildkit_unruffled_shockley0 0.5s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 226B 0.0s
=> [linux/arm64 internal] load metadata for docker.io/library/python:3.11-slim-bookworm 1.5s
=> [linux/amd64 internal] load metadata for docker.io/library/python:3.11-slim-bookworm 1.6s
...
=> [linux/arm64 1/4] FROM docker.io/library/python:3.11-slim-bookworm@sha256:edaf703dce209d774af3ff768fc92b1e3b60261 4.0s
...
=> => transferring context: 156B 0.0s
=> [linux/amd64 1/4] FROM docker.io/library/python:3.11-slim-bookworm@sha256:edaf703dce209d774af3ff768fc92b1e3b60261 3.7s
...
=> [linux/amd64 3/4] COPY example.py . 0.0s
=> [linux/amd64 4/4] RUN pip install --upgrade pip && pip install requests 17.2s
=> [linux/arm64 2/4] WORKDIR /app 0.0s
=> [linux/arm64 3/4] COPY example.py . 0.0s
=> [linux/arm64 4/4] RUN pip install --upgrade pip && pip install requests 3.0s
...
=> => pushing manifest for docker.io/dpills/py-todos:1.1@sha256:a2080925c4f090ca44720e77dfe37b28a8543b1be57c24dc3508 1.4s
Top comments (1)
Excellent! Very clearly explanation š