DEV Community: Parvathi

Beginner's Guide to Docker (Part 4) - Docker Compose

Parvathi — Sat, 22 Jan 2022 17:27:43 +0000

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

Parvathi ・ Sep 27 '21 ・ 8 min read

#docker #devops #beginners #webdev

Beginner's Guide to Docker(Part 2)- Manage data in Docker

Parvathi ・ Sep 29 '21 ・ 5 min read

#docker #devops #beginners #webdev

Beginner's Guide to Docker (Part 3) - Communication & Network

Parvathi ・ Oct 8 '21 ・ 2 min read

#docker #devops #beginners #webdev

We have learned above topics by taking todo-app as an example, which involves frontend(react), backend(node) and DB(mongo). So far we did everything manually, building images, creating the containers, attaching storage to the container, creating network for container communication, etc... this seems to be a lot of work...Right??? What could be the solution? Yes, Docker Compose!!!

Reference

Todo app git repo

The file structure

Docker Compose

Docker Compose is a tool that was developed to help define and share multi-container applications. With Compose, we can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

The big advantage of using Compose is you can define your application stack in a file, keep it at the root of your project repo (it’s now version controlled), and easily enable someone else to contribute to your project. Someone would only need to clone your repo and start the compose app. In fact, you might see quite a few projects on GitHub/GitLab doing exactly this now.

If you installed Docker Desktop/Toolbox for either Windows or Mac, you already have Docker Compose! In Linux, we need to install manually.

Writing a compose file

At the root of the app project, create a file named docker-compose.yml.

In the compose file, we’ll start off by defining the schema version. It’s best to use the latest supported version. Next, we’ll define the list of services (or containers) we want to run as part of our application. Compose file reference

Service

Let's see how we can move one service to docker compose file.

docker run
    --name todo-node
    -p 3030:3030
    --rm
    --network todo-app
    -e MONGO_HOST=mongodb
    -v /Users/xyz/Documents/dockerExamples/todo-app/todo-backend:/app
    -v /app/modules
    todo-node

version: '3.8'
services:
  backend:
    build: ./todo-backend 
    ports:
      - '3030:3030' #host:container
    volumes:
      - ./todo-backend:/app 
      - /app/node_modules
    env_file:
      - ./env/node.env
    depends_on:
      - mongodb

Manual commands

Creating network for mongo and node to share

docker network create todo-app

Creating mongo container

docker run
    --name mongodb
    --rm
    -d
    --network todo-app
    -v data:/data/db
    -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=password
    mongo

Frontend

docker build -t todo-react .

docker run
    --name todo-react
    --rm
    -p 3000:3000
    -v /Users/xyz/Documents/dockerExamples/todo-app/todo-frontend/src:/app/src
    -v /app/node_modules
    todo-react

Backend

docker build -t todo-react .

docker run
    --name todo-node
    -p 3030:3030
    --rm
    --network todo-app
    -e MONGO_HOST=mongodb
    -v /Users/xyz/Documents/dockerExamples/todo-app/todo-backend:/app
    -v /app/modules
    todo-node

Docker compose file

This is how the docker compose file will look like for the above commands

version: '3.8'
services:
  mongodb: #name of the service
    image: 'mongo' #Use existing image
    volumes:
      - data:/data/db
    restart: always
    # environment:
    #   MONGO_INITDB_ROOT_USERNAME: root
    #   MONGO_INITDB_ROOT_PASSWORD: password
    env_file:
      - ./env/mongo.env
    ports:
      - '27017:27017'
    container_name: mongodb # optional
    # networks:
      # - todo-network
  backend:
    build: ./todo-backend #if dockerfile path is straightforward we can use this
    # build:
    #   context: ./todo-backend
    #   dockererfile: Dockerfile - docker file name inside the path
    #   args:
    #     some-args: 1
    ports:
      - '3030:3030' #host:container
    volumes:
      - ./todo-backend:/app #bind mount - live source code change for development purpose
      - /app/node_modules
    env_file:
      - ./env/node.env
    depends_on:
      - mongodb
  frontend:
    build: ./todo-frontend
    ports:
      - '3000:3000'
    volumes:
      - /app/node_modules
      - ./todo-frontend:/app
    stdin_open: true #to run in interactive mode
    tty: true #to run in interactive mode
    depends_on:
      - mongodb
      - backend
volumes: #specify all named volumes that are used
  data:

Note - We don't have to explicitly create a network while working with docker compose. By default, it automatically creates a network specifically for the application stack.

Run the application stack

Now that we have our docker-compose.yml file, we can start it up!

Start up the application stack using the docker-compose up command. We’ll add the -d flag to run everything in the background.

docker-compose up -d

We shall look at logs using

docker-compose logs -f

You’ll see the logs from each of the services interleaved into a single stream. This is incredibly useful when you want to watch for timing-related issues. The -f flag “follows” the log, so will give you live output as it’s generated.

If you want to view the logs for a specific service, you can add the service name to the end of the logs command

docker-compose logs -f frontend

Hola, Our application is up and running!!!

In order to bring down our application stack

docker-compose down

By default, named volumes in your compose file are NOT removed when running docker-compose down. If you want to remove the volumes, you will need to add the --volumes flag.

Hope, this gave you a basic understanding about docker compose.

Thanks for reading!

Beginner's Guide to Docker (Part 3) - Communication & Network

Parvathi — Fri, 08 Oct 2021 15:06:09 +0000

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

Parvathi ・ Sep 27 '21 ・ 8 min read

#docker #devops #beginners #webdev

Beginner's Guide to Docker(Part 2)- Manage data in Docker

Parvathi ・ Sep 29 '21 ・ 5 min read

#docker #devops #beginners #webdev

We have learned about docker basics and also we saw what is volumes and why we need volume in docker. Now let us see what are all the types of communication our application needs and how we can achieve it using docker.

Container to WWW

In case, our application in docker container wants to communicate with world wide web for read or write, for example, our application uses youtube api to fetch video for our iframe, or building any search widget using wikipedia api we need to communicate with api that youtube and wikipedia has exposed, this we can do it without any additional efforts, docker supports this out of box.

Container to host

Let's say we have our application is in need to connect with mongodb which is hosted on localhost, now what can we do? how our application inside container reaches the mongodb hosted locally?

mongoose.connect('mongodb://localhost/todo_app', {useNewUrlParser: true, useUnifiedTopology: true});

we can simply change the above code to

mongoose.connect('mongodb://host.docker.internal/todo_app', {useNewUrlParser: true, useUnifiedTopology: true});

Container to another container

Let's take the same above example, our application wants to communicate with the mongodb, but this time it is hosted in another container instead of host machine, now how can we do that?

docker run --name mongoDb --rm -d -v data:/data/db mongo

docker container inspect mongoDb

from the result of the above command, we can find the ipAddress of the mongo container

"IPAddress": "172.17.0.2",

We can replace the connection string as

mongoose.connect('mongodb://172.17.0.2/todo_app', {useNewUrlParser: true, useUnifiedTopology: true});

Wait is this the only way to do it??? No!

Network

docker network COMMAND

Manage networks. You can use subcommands to create, inspect, list, remove, prune, connect, and disconnect networks.

Let us take todo application built using MERN stack, in which our node application talks with mongo db.

We can bring node container and mongo container to a same network and then we can just use the container name as host ipAddress and docker will resolve the ipAddress internally.

docker network create todo-network

docker run \
    --name mongodb_container \
    --rm \
    -d \
    --network todo-network \
    -v data:/data/db \
    mongo

docker run \
    --name todo-node \
    -p 3030:3030 \
    --rm \
    --network todo-network \
    todo-node

mongoose.connect('mongodb://mongodb_container/todo_app', {useNewUrlParser: true, useUnifiedTopology: true});

Thanks for Reading!!!

Learn more about Docker Compose in next article

Beginner's Guide to Docker(Part 2)- Manage data in Docker

Parvathi — Wed, 29 Sep 2021 12:54:21 +0000

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

Parvathi ・ Sep 27 '21

#docker #devops #beginners #webdev

We learned few basic things about docker, such as how to build an Image using dockerfile and create a container based on that Image, in continuation to that lets see how to manage data in docker.

Why we need to manage data?

Our dockerized application might have to communicate with file system either for reading or writing data, based on the functionality. Some data needs to be persisted and some data are not.

By default all files created inside a container are stored on a writable container layer. This means that:

1)The data doesn’t persist when that container no longer exists, and it can be difficult to get the data out of the container if another process needs it.

2)A container’s writable layer is tightly coupled to the host machine where the container is running. You can’t easily move the data somewhere else.

3)Writing into a container’s writable layer requires a storage driver to manage the filesystem. The storage driver provides a union filesystem, using the Linux kernel. This extra abstraction reduces performance as compared to using data volumes, which write directly to the host filesystem.

How can we persist data?

Docker has two options for containers to store files in the host machine, so that the files are persisted even after the container stops: volumes, and bind mounts. If you’re running Docker on Linux you can also use a tmpfs mount. If you’re running Docker on Windows you can also use a named pipe.

Volumes

Volumes are stored in a part of the host filesystem which is created and managed by Docker (/var/lib/docker/volumes/ on Linux). Non-Docker processes should not modify this part of the filesystem.

When you create a volume, it is stored within a directory on the Docker host. When you mount the volume into a container, this directory is what is mounted into the container

A given volume can be mounted into multiple containers simultaneously. When no running container is using a volume, the volume is still available to Docker and is not removed automatically.

We can create volume in two ways:
1)Anonymous volumes are not given an explicit name when they are first mounted into a container, so Docker gives them a random name that is guaranteed to be unique within a given Docker host. This volume will be removed automatically only when we create/run the container with -rm option. However new anonymous volume are created when we re-create the container.

2)Named volumes - As name suggests, we can assign name to the volume. These volumes are not deleted even when we stop or remove the container. By naming the volumes we can prevent recreating the volume again.

Volumes are often a better choice than persisting data in a container’s writable layer, because a volume does not increase the size of the containers using it, and the volume’s contents exist outside the lifecycle of a given container.

Volumes also support the use of volume drivers, which allow you to store your data on remote hosts or cloud providers, among other possibilities. To specify drivers we need to use --mount.

Let us take a simple todo application, in which the app saves todo list items in /app/todos/todo.txt. If we save this file within the container filesystem, our todo list will be wiped clean every single time we launch the container. So we are going to use volumes instead.

docker volume create todo-db

Creates volume named todo-db.

docker volume ls

Lists all created/available volumes.

docker volume inspect todo-db

This command allows us to inspect the volume.

docker volume rm todo-db

Removes the given volume.

docker volume prune

Remove all volumes.

Bind mounts

Bind mounts may be stored anywhere on the host system, it is managed by us. They may even be important system files or directories. Non-Docker processes on the Docker host or a Docker container can modify them at any time.

When you use a bind mount, a file or directory on the host machine is mounted into a container. The file or directory is referenced by its absolute path on the host machine.

Using `-v` or `--volume` option

docker run -d --name devtest \ -v todo-db:/app/todos todo-app:latest - Named volume

All files that are created in /app/todos will be captured in todo-db volume

docker run -d --name devtest \ -v /app/todos todo-app:latest - Anonymous volume

All files that are created in /app/todos will be captured in docker created anonymous volume.

We can also create volume using VOLUME instruction in dockerfile VOLUME ["/app/todos"] and we cannot create named volume with this instruction in dockerfile.

docker run -d --name devtest \ -v "$(pwd)"/todo-mount:/app/todos todo-app:latest - Bind mount

-v or --volume: Consists of three fields, separated by colon characters (:). The fields must be in the correct order

In the case of named volumes, the first field is the name of the volume, and is unique on a given host machine. For anonymous volumes, the first field is omitted.

The second field is the path where the file or directory are mounted in the container.

The third field is optional, and is a comma-separated list of options, such as ro `-v todo-db:/app/todos:ro`(defining volume to be read only, by default the volume has read and write access).

If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.

Using `--mount` option

docker run -d --name devtest \ --mount type=volume,source=todo-db,target=/app/todos \ todo-app:latest - Volume

docker run -d --name devtest \ --mount type=bind,source="$(pwd)"/todo-mount,target=/app/todos \ todo-app:latest - Bind

--mount: Consists of multiple key-value pairs, separated by commas and each consisting of a <key>=<value>

The type of the mount, which can be bind, volume, or tmpfs.

The source of the mount. For bind mounts, this is the path to the file or directory on the Docker daemon host. May be specified as source or src.

The destination takes as its value the path where the file or directory is mounted in the container. May be specified as destination, dst, or target.

The readonly option, if present, causes the bind mount to be mounted into the container as read-only.

The bind-propagation option, if present, changes the bind propagation. May be one of rprivate, private, rshared, shared, rslave, slave.

If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.

tmpfs mount

As opposed to volumes and bind mounts, a tmpfs mount is temporary, and only persisted in the host memory. When the container stops, the tmpfs mount is removed, and files written there won’t be persisted. It can be used by a container during the lifetime of the container, to store non-persistent state or sensitive information.

This is useful to temporarily store sensitive files that you don’t want to persist in either the host or the container writable layer.

This is available only if you’re running Docker on Linux.

This can be achieved using --mount or --tmpfs

docker run -d -it --name devtest \ --mount type=tmpfs,destination=/app/todos todo-app:latest

docker run -d -it --name devtest \ --tmpfs /app/todos todo-app:latest

Source Docker

To learn about docker communication and networking checkout Networking

Thanks for reading!!

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

Parvathi — Mon, 27 Sep 2021 18:21:35 +0000

What is Docker? - "build once and run anywhere."

Docker is a software development tool and a virtualisation technology that makes it easy to develop, deploy, and manage applications by using containers. Container refers to a lightweight, stand-alone, executable package of a piece of software that contains all the libraries, configuration files, dependencies, and other necessary parts to operate the application.

In other words, applications run in the same way, irrespective of where they are and what machine they are running on because the container provides the environment throughout the software development lifecycle of the application. Since containers are isolated, they provide security, thus allowing multiple containers to run simultaneously on the given host. Also, containers are lightweight because they do not require an extra load of a hypervisor. A hypervisor is a guest operating system like VMWare or VirtualBox, but instead, containers run directly within the host’s machine kernel.

When to use docker?

Learning new technologies: To get started with a new tool without spending time on installation and configuration, Docker offers an isolated and disposable environment. Many projects maintain Docker images with their applications already installed and configured.

Basic use cases: Pulling images from Docker Hub is also a good solution if your application is basic or standard enough to work with a default Docker image. Cases such as in developing and hosting a website using MERN stack, the node, mongodb official images are already available on DockerHub and they are well supported. If the default configuration in these images is acceptable for your needs, then pulling the image can save a lot of time that would otherwise be spent setting up your environment and installing the necessary tools.

App isolation: If you want to run multiple applications on one server, keeping the components of each application in separate containers will prevent problems with dependency management.

Developer teams: It works on my machine! As a developer we know that one of the trickiest problems in software development is having to deal with environment disparity across different machines and platforms. Docker allows you to run containers locally, eliminating disparity between your development and production environments, and everything in between. There is no need to install software packages locally. Everything you need for your development environment can simply run on the Docker engine as containers. Regardless of the language or the tool, you can easily containerise your environment locally.

Limited system resources Instances of containerised apps use far less memory than virtual machines, they start up and stop more quickly, and they can be packed far more densely on their host hardware. All of this amounts to less spending on IT.
The cost savings will vary depending on what apps are in play and how resource-intensive they may be, but containers invariably work out as more efficient than VMs. It’s also possible to save on costs of software licenses, because you need many fewer operating system instances to run the same workloads.

Virtual machines vs Docker

Docker containers and virtual machines are both ways of deploying applications inside environments that are isolated from the underlying hardware. The chief difference is the level of isolation.

With a virtual machine, everything running inside the VM is independent of the host operating system, or hypervisor. The virtual machine platform starts a process (called virtual machine monitor, or VMM) in order to manage the virtualisation process for a specific VM, and the host system allocates some of its hardware resources to the VM. However, what’s fundamentally different with a virtual machine is that at start time, it boots a new, dedicated kernel for this VM environment, and starts a (often rather large) set of operating system processes. This makes the size of the VM much larger than a typical container that only contains the application.

In contrast, With a container runtime like Docker, your application is sandboxed inside of the isolation features that a container provides, but still shares the same kernel as other containers on the same host. As a result, processes running inside containers are visible from the host system (given enough privileges for listing all processes). Having multiple containers share the same kernel allows the end user to bin-pack lots and lots of containers on the same machine with near-instant start time. Also, as a consequence of containers not needing to embed a full OS, they are very lightweight, commonly around 5-100 MB.

What is a Container?

A container is a runnable instance of an image. You can create, start, stop, move, or delete a container using the Docker API or CLI. You can connect a container to one or more networks, attach storage to it, or even create a new image based on its current state.

By default, a container is relatively well isolated from other containers and its host machine. You can control how isolated a container’s network, storage, or other underlying subsystems are from other containers or from the host machine.

A container is defined by its image as well as any configuration options you provide to it when you create or start it. When a container is removed, any changes to its state that are not stored in persistent storage disappear.

What is a Docker Image?

An image is a read-only template with instructions for creating a Docker container. Often, an image is based on another image, with some additional customisation. For example, you may build an image which is based on the ubuntu image, but installs the Apache web server and your application on top of it, as well as the configuration details needed to make your application run.

You might create your own images or you might only use those created by others and published in a registry. To build your own image, you create a Dockerfile with a simple syntax for defining the steps needed to create the image and run it. Each instruction in a Dockerfile creates a layer in the image. When you change the Dockerfile and rebuild the image, only those layers which have changed are rebuilt. This is part of what makes images so lightweight, small, and fast, when compared to other virtualisation technologies.

What is Docker Hub?

Docker Hub is a cloud-based repository service provided by Docker in which users create, test, store and distribute container images. Through Docker Hub, a user can access public, open-source image repositories, as well as use space to create their own private repositories, automated build functions, web hooks and workgroups.

Writing simple Dockerfile

Assume we have a simple node JS application that has server.js file, which listens to the port 3040 and prints 'Hello World!' on hitting 'localhost:3040/'

File Structure as follows:
simpleNodeApplication

Dockerfile
package-lock.json
package.json
server.js


//Initializes a new build stage and sets the Base Image for subsequent instructions
FROM node:14-alpine

//Defining working directory for our application and 
it will be the default directory for our further steps
WORKDIR /app

//Copy package.json file to our created workdir
COPY package.json .

//Does npm install
RUN npm install

//Copy entire code from local to our workdir
COPY . .

//Exposing application on port 3040
EXPOSE 3040

//Command to run our application
CMD ["node", "server.js"]

To play with the above dockerfile

Build image => docker build -t myApp:V1 .
Create container and map it to port 3000 => docker run -p 3000:3040 --name myContainer myApp:V1
List running containers => docker ps
Stop container => docker stop myContainer
To start container again => docker start myContainer

Note: We might wonder about the difference between RUN and CMD, RUN command will be executed while we build an image, Whereas the CMD instruction should be used to run the software contained in your image

Docker commands

The following are some of the docker CLI commands that we use in daily basis

Let ImageId be myApp:V1 and ContainerName be myContainer

Image
docker build [OPTIONS] PATH | URL | -
docker build -t myApp:V1 .

This command help us to build an image of an application using written dockerfile.
We can name the image on our own, to do that use -t option docker build -t name:tag .

docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]
docker tag myApp:V1 myNewApp:V1

This allows us to rename/tag already created image without the need for rebuilding it.

docker images

List all images that are available in local.

docker image inspect Name|ImageId
docker image inspect myApp:V1

This provides detailed information of an image in JSON format by default.

docker rmi [OPTIONS] IMAGE [IMAGE...]
docker rmi myApp:V1

This command allows us to remove one or more images.
We can remove the image only if it is not used by any container, including stopped containers. Inorder to remove image we need to remove the container first.

docker system prune [OPTIONS]

Remove all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes.

Containers
docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
docker run -p 3000:3040 -it --rm --name myContainer myApp:V1

This command first creates a writeable container layer over the specified image, and then starts it using the specified command. By default this command runs in attached mode. Every time you run this command it will create a new container with the given image.
By default this command search for image in local, if not found it will also look in repositories.
In order to name our container we can use --name option, docker run --name string Name|ImageId.
In order to run an application with interactive mode(read input from console etc...) use -it option.
--rm option will tell docker to remove the container once the container is stopped.

docker stop [OPTIONS] CONTAINER [CONTAINER...]
docker stop myContainer

This commands helps us to stop one or more containers.

docker start [OPTIONS] CONTAINER [CONTAINER...]
docker start myContainer

This commands helps us to start one or more stopped containers. By default this runs in detached mode.

docker restart [OPTIONS] CONTAINER [CONTAINER...]
docker restart myContainer

Restart one or more containers.

docker rename CONTAINER NEW_NAME
docker rename myContainer myNewContainer

Allows us to rename already created container.

docker ps [OPTIONS]

This list all running containers by default.
To list all containers use -a option.

docker rm [OPTIONS] CONTAINER [CONTAINER...]
docker rm myContainer

This command allows us to remove one or more containers.

docker logs [OPTIONS] CONTAINER
docker logs myContainer

Fetch the logs of a container.

docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-

Copy files/folders between a container and the local filesystem. This can be used in scenarios like when we want to pull log file from container to local file system for debugging.

docker login [OPTIONS] [SERVER]
docker login localhost:8080

Log in to a Docker registry.

docker logout [SERVER]
docker logout localhost:8080

Log out from a Docker registry.

docker push [OPTIONS] NAME[:TAG]
docker image push myApp:V1

Use docker image push to share your images to the Docker Hub registry or to a self-hosted one.

docker pull [OPTIONS] NAME[:TAG|@DIGEST]
docker pull ubuntu:20.04

Docker Hub contains many pre-built images that you can pull and try without needing to define and configure your own.
This command allows us to download a particular image, or set of images (i.e., a repository)

Hope you got a good understanding about docker basics, to know about how to manage data in docker checkout Part 2
Thanks for reading!!

DEV Community: Parvathi

Beginner's Guide to Docker (Part 4) - Docker Compose

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

Parvathi ・ Sep 27 '21 ・ 8 min read

Beginner's Guide to Docker(Part 2)- Manage data in Docker

Parvathi ・ Sep 29 '21 ・ 5 min read

Beginner's Guide to Docker (Part 3) - Communication & Network

Parvathi ・ Oct 8 '21 ・ 2 min read

Reference

Docker Compose

Writing a compose file

Manual commands

Docker compose file

Run the application stack

Beginner's Guide to Docker (Part 3) - Communication & Network

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

Parvathi ・ Sep 27 '21 ・ 8 min read

Beginner's Guide to Docker(Part 2)- Manage data in Docker

Parvathi ・ Sep 29 '21 ・ 5 min read

Container to WWW

Container to host

Container to another container

Beginner's Guide to Docker(Part 2)- Manage data in Docker

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

Parvathi ・ Sep 27 '21

Why we need to manage data?

Volumes

Bind mounts

Using -v or --volume option

Using --mount option

tmpfs mount

Beginner's guide to Docker(Part 1)- Dockerfile and Docker CLI commands

What is Docker? - "build once and run anywhere."

When to use docker?

Virtual machines vs Docker

What is a Container?

What is a Docker Image?

What is Docker Hub?

Writing simple Dockerfile

Docker commands

Using `-v` or `--volume` option

Using `--mount` option