DEV Community: Prasenjeet Kumar

Docker Swarn : From Newbie to Master ( part 2 )

Prasenjeet Kumar — Sun, 28 May 2023 05:52:49 +0000

Welcome to Part 2 of this series on Docker Swarm. If you have not read the previous article, please check out my profile to find it. In this article, we will discuss the remaining Docker Swarm commands.

`docker service update`

The docker service update command is used to update a running service in a Docker swarm. It can be used to change the configuration of a service, such as the number of replicas, the image used, or the environment variables.

The basic syntax of the command is as follows:

docker service update [OPTIONS] SERVICE

Here are some common options that can be used with the docker service update command:

-replicas: Change the number of replicas for the service
-image: Update the image used for the service
-env-add and -env-rm: Add or remove environment variables for the service
-force: Force an update of the service even if there are no changes

Here is an example of using the docker service update command to update the number of replicas for a service:

docker service update --replicas 3 my-service

This will change the number of replicas for the my-service service to 3.

Here is another example that updates the image used by a service:

docker service update --image my-image:latest my-service

This will update the image used by the my-service service to my-image:latest.

You can also use the --env-add and --env-rm options to add or remove environment variables for the service:

docker service update --env-add MY_VAR=my_value my-service
docker service update --env-rm MY_VAR my-service

The first command will add an environment variable MY_VAR with the value my_value to the my-service service. The second command will remove the MY_VAR environment variable from the service.

Finally, you can use the --force option to force an update of the service even if there are no changes:

docker service update --force my-service

This will force an update of the my-service service even if there are no changes.

💡 The --force flag is used with the docker service update command to force the immediate update of a service even if it results in service disruption. By default, Docker Swarm will try to update a service without disrupting it by gradually updating one replica at a time. However, in certain cases, it may be necessary to force an immediate update of all replicas to ensure that the changes are applied quickly. This can be useful in situations where the new changes are critical and need to be implemented as soon as possible, even if it causes temporary service disruption.

It is important to use the --force flag with caution as it may cause service disruption and should be used only when necessary.

`docker service scale`

The docker service scale command is used to scale the number of replicas of a service in a Docker swarm.

The syntax for the command is:

docker service scale <service_name>=<replica_count>

Here, <service_name> is the name of the service to be scaled and <replica_count> is the new number of replicas to be set.

For example, if we have a service named web with 3 replicas and we want to scale it to 5 replicas, we can use the following command:

docker service scale web=5

This will update the service with the new replica count. Note that this command only works for replicated services and not for global services.

💡 Replicated services have a specific number of replicas defined, while global services run one replica on every node in the swarm. Replicated services are useful for running a fixed number of tasks, while global services are useful for tasks that need to run on every node in the swarm, such as logging or monitoring agents.

`docker service rm`

docker service rm is a command used to remove one or more services from a Docker swarm. It stops and removes all tasks associated with the specified services. The command takes one or more service IDs or service names as arguments.

Here is an example of how to use it:

docker service rm my-service

This will remove the service named my-service from the swarm.

`docker stack deploy`

The docker stack deploy command is used to deploy a Docker Compose file to a Swarm cluster as a stack. A stack is a group of services that are deployed together as a single unit.

Here is an example command that deploys a stack:

docker stack deploy -c docker-compose.yml mystack

In this command, docker-compose.yml is the name of the Docker Compose file, and mystack is the name of the stack.

The options available with the docker stack deploy command are:

c, -compose-file: Specifies the Docker Compose file to use.
-prune: Removes any services or containers that are not defined in the Compose file.
-resolve-image: Forces the cluster to resolve the image name to a specific digest, rather than using the default "latest" tag.
-orchestrator: Specifies the orchestrator to use, either "swarm" or "kubernetes".

Here is an example of deploying a stack with the --prune option:

docker stack deploy --prune -c docker-compose.yml mystack

This command removes any services or containers that are not defined in the Compose file.

The --resolve-image option for the docker stack deploy command is used to resolve image digest and tag to the corresponding image ID before deploying a stack.

The possible values for --resolve-image option are:

always: This value tells Docker to always attempt to pull the image, even if a local copy exists. This can be useful if you want to make sure you're always using the latest version of an image.
changed: This value tells Docker to only pull the image if the image's tag has changed since the last time it was pulled. This can be useful if you want to minimize network traffic and avoid unnecessary pulls.
never: This value tells Docker to never attempt to pull the image, and to only use a local copy if it exists. This can be useful if you know that you won't need to update the image, or if you want to avoid network traffic altogether. However, be careful when using this option, as it can result in using an outdated image if a newer one is available.

Here is an example command using the --resolve-image option with the always value:

docker stack deploy --compose-file docker-compose.yml --resolve-image always myapp

`docker stack ls`

The docker stack ls command is used to list all the stacks that are currently deployed on the swarm. It provides a quick summary of the deployed stack, including the name, the number of services in the stack, and the status of the stack.

Here's an example of how to use the command:

docker stack ls

This will output a table that lists all the deployed stacks, their number of services, and their status. Here's an example output:

NAME                SERVICES            STATUS             
my_stack            3                   * Running

The NAME column shows the name of the stack. The SERVICES column shows the number of services in the stack. The STATUS column shows the current status of the stack. In this example, the stack named my_stack has 3 services and is currently running.

This command is useful when you need to check the status of your deployed stacks and the number of services running in them.

`docker stack ps`

The docker stack ps command is used to list the tasks or replicas of a given stack.

Here is the syntax of the command:

docker stack ps [OPTIONS] STACK_NAME

Where STACK_NAME is the name of the stack.

Here are some options that can be used with the docker stack ps command:

-filter: It can be used to filter the tasks based on various parameters like desired state, node, etc.
-no-trunc: It is used to disable truncating the output of the command.
-quiet: It is used to display only the task IDs.

Here is an example command to list the tasks of a stack named my_stack:

docker stack ps my_stack

The output of docker stack ps command shows the running processes/replicas of the services in the specified stack along with their current status. It displays a table with the following columns:

ID: The unique ID of the task.
Name: The name of the service the task belongs to.
Image: The Docker image used to create the task.
Node: The hostname of the node where the task is running.
Desired State: The desired state of the task (running or shutdown).
Current State: The current state of the task (running, completed, or shutdown).
Error: Any error message associated with the task, if applicable.

Here's an example output:

ID                  NAME                IMAGE                    NODE                DESIRED STATE       CURRENT STATE            ERROR
qn41eabryqw1        myapp_web.1         myapp:latest             swarm-node-1        Running             Running 2 minutes ago    
fb48n1mzsqxd        myapp_web.2         myapp:latest             swarm-node-2        Running             Running 2 minutes ago    
iz4f7ifg5ue5        myapp_worker.1      myapp-worker:latest      swarm-node-1        Running             Running 2 minutes ago    
7nzq44tigzrw        myapp_db.1          postgres:10.1            swarm-node-2        Running             Running 2 minutes ago

In this example, there are four tasks running for the my_stack stack. Two replicas of the myapp_web service are running on swarm-node-1 and swarm-node-2, one replica of the myapp_worker service is running on swarm-node-1, and one replica of the myapp_db service is running on swarm-node-2.

`docker stack rm`

The docker stack rm command is used to remove a Docker stack, which includes all the services and networks associated with that stack. Here is the syntax for the command:

docker stack rm [OPTIONS] STACK_NAME [STACK_NAMES...]

Where STACK_NAME is the name of the stack to be removed.

For example, to remove a stack named my_stack, the command would be:

docker stack rm my_stack

This command will remove all the services and networks associated with the my_stack stack.

There are options available for the docker stack rm command. Here are some commonly used options:

-force: This option forces the removal of the stack even if it is currently running.
-orchestrator: This option specifies which orchestrator to use for the removal operation. By default, the command uses the swarm mode orchestrator.
-volumes: This option removes the volumes associated with the stack along with the stack itself.

Here's an example command that uses the --force option to forcefully remove a stack:

docker stack rm --force myapp

Note that this command does not remove any volumes that were created by the services in the stack. However, you can use the --volumes option to remove the volumes as well.

Here is an example command to remove a stack with associated volumes:

docker stack rm --volumes mystack

This will remove the mystack stack along with its associated volumes. If you do not want to remove the volumes, simply omit the --volumes option.

💡 Just remember that docker is not going to remove any volume except anonymous volumes without your confirmation due to data loss risk

`docker stack services`

The docker stack services command is used to list the services that are running as part of a stack.

Here's the basic syntax:

docker stack services <stack_name>

This will display a table with the following columns:

ID: The unique identifier of the service
NAME: The name of the service
MODE: Whether the service is running in replicated or global mode
REPLICAS: The number of replicas of the service (only applicable for replicated services)
IMAGE: The Docker image used by the service

Here's an example of how to use the command:

docker stack services myapp_stack

This will display a table of all the services running in the myapp_stack stack.

ID                  NAME                 MODE                REPLICAS            IMAGE                      PORTS
8h1fc2y1ffaz        myapp_stack_web      replicated          3/3                 myapp:latest               *:80->80/tcp
e96fa5zjdt0a        myapp_stack_redis    replicated          1/1                 redis:latest               *:6379->6379/tcp

How to scale services in Docker Swarm stack?

To scale a service in a Docker stack, you can use the docker service scale command followed by the name of the service and the desired number of replicas. For example, to scale a service named web in a stack named my_stack to 5 replicas, you can use the following command:

docker service scale my_stack_web=5

This will create or remove replicas as necessary to ensure that there are 5 replicas of the web service running.

Docker swarn secret

Swarm Secrets is a feature in Docker Swarm that allows users to securely store and manage sensitive data, such as passwords, authentication tokens, and other confidential information. It follows a client-server architecture, where the Docker Swarm manager node acts as the server and the worker nodes as clients.

When a secret is created, it is encrypted and stored on the manager node. The encrypted secret is then replicated to all the worker nodes that are part of the Swarm. The workers have access to the secret only when the service that requires the secret is created or updated on the worker node.

Overall, Swarm Secrets provide a secure and convenient way to manage and distribute sensitive data in a Swarm environment.

How to use swarn secret ?

First, create a Swarm Secret using the docker secret create command:

$ echo "mysecretpassword" | docker secret create db_password -

This command will create a Swarm Secret named db_password with the value mysecretpassword.

When you deploy the Node.js application to a Docker Swarm using the docker stack deploy command, you can specify the Swarm Secret in the docker-compose.yml file like this:

version: "3.7"
services:
  app:
    image: my-node-app
    secrets:
      - source: db_password
        target: /run/secrets/db_password
        mode: 0400
secrets:
  db_password:
    external: true
    file: ./db_password.txt

In this example, the secrets section is used to specify the db_password Swarm Secret. The external: true option tells Docker that this Secret already exists in the Swarm.

The secrets section in the app service specifies that the db_password Secret should be mounted as a file at /run/secrets/db_password inside the container. The mode: 0400 option sets the permissions of the file to read-only for the owner.

When you deploy the stack using the docker stack deploy command, Docker Swarm will automatically create the necessary files and mount the Swarm Secret as a file in the container.

Here is an example of a Node.js application that uses the Swarm Secret db_password:

const fs = require('fs');

const password = fs.readFileSync('/run/secrets/db_password', 'utf8');

console.log('Database password:', password);

In this example, the fs.readFileSync method is used to read the db_password Swarm Secret file. The file is mounted in the container at the path /run/secrets/db_password.

💡 In the development env just mark theexternal to false inside your docker-compose.yml file and put your database password inside the file db_password.txt located in the folder of docker-compose.yml file.

Docker Swarn : From Newbie to Master ( part 1 )

Prasenjeet Kumar — Wed, 17 May 2023 06:29:22 +0000

Docker Swarm is a container orchestration tool provided by Docker that allows you to create a cluster of Docker hosts and deploy services to the cluster. It allows you to manage and scale your Docker containers and services across a cluster of hosts, providing high availability and load balancing for your applications. With Docker Swarm, you can create a highly available and scalable infrastructure that can handle a large number of containerized applications.

Why docker swarn ?

Docker Swarm provides several benefits that make it useful in certain scenarios. Here are a few reasons why someone might choose to use Docker Swarm:

High availability: Docker Swarm is designed to provide high availability of applications by automatically distributing containers across multiple nodes.
Scalability: Docker Swarm allows you to easily scale your application by adding or removing nodes.
Load balancing: Docker Swarm provides built-in load balancing to distribute traffic across multiple containers.
Security: Docker Swarm provides security features such as mutual TLS authentication and encryption for network communication.
Ease of use: Docker Swarm is easy to set up and use, especially if you are already familiar with Docker.

Overall, Docker Swarm provides an efficient and convenient way to manage and scale containerized applications in a distributed environment.

Architecture of Docker Swarm

Docker Swarm follows a distributed and decentralized architecture with a manager-worker model. The basic components of the Swarm architecture are:

Manager Nodes: The manager nodes are responsible for managing the cluster state, scheduling tasks, and maintaining the desired state of the services. There can be multiple manager nodes in a Swarm cluster, but only one manager node is the leader at a time.
Worker Nodes: The worker nodes are responsible for running the tasks and services assigned to them by the manager nodes. There can be multiple worker nodes in a Swarm cluster.
Service: A service is a group of containers that perform the same task or function. The service is defined by a Docker Compose file or Docker service create command.
Task: A task is a container running on a worker node, which is part of a service. Each task runs a specific instance of a container image.
Overlay Network: An overlay network is a virtual network that connects the containers across different nodes in the Swarm cluster. This network allows the containers to communicate with each other seamlessly, even if they are running on different hosts.
Swarm Secret: A Swarm secret is a way to securely store sensitive data such as passwords, certificates, or API keys. The Swarm secrets are encrypted and only accessible by authorized containers in the Swarm cluster.

The Swarm manager nodes handle the orchestration of tasks and services and make sure that the desired state of the services is maintained. The worker nodes execute the tasks and services assigned to them by the manager nodes. The overlay network connects the containers across the Swarm cluster, and Swarm secrets provide a secure way to store sensitive data.

How to create Swarn cluster ?

To create a Docker Swarm, you need to initialize a Swarm on one or more nodes. You can do this by running the docker swarm init command on a manager node. Here are the steps to create a Docker Swarm:

Choose one or more nodes to act as manager nodes. These nodes will be responsible for managing the Swarm cluster.
On one of the manager nodes, run the following command to initialize the Swarm:
```
docker swarm init --advertise-addr <MANAGER-IP-ADDRESS>
```
Replace <MANAGER-IP-ADDRESS> with the IP address of the manager node.
After the Swarm is initialized, the command will output a docker swarm join command that you can use to add worker nodes to the Swarm. Copy this command and run it on each worker node to join them to the Swarm.
Once all nodes are part of the Swarm, you can start deploying services and managing them using Docker Swarm commands.

💡 Note: In a production environment, you should use secure communication channels and encryption for communication between the Swarm nodes. You can use TLS certificates for this purpose.

Some common Docker Swarm commands

Here are some of the commonly used Docker Swarm commands:

docker swarm init: Initializes a new Swarm.
docker swarm join: Join an existing Swarm as a node.
docker node ls: List all the nodes in the Swarm.
docker service create: Create a new service.
docker service ls: List all the services running in the Swarm.
docker service ps: List all the tasks (replicas) of a service.
docker service update: Update an existing service.
docker service scale: Scale a service up or down.
docker service rm: Remove a service.
docker stack deploy: Deploy a new stack.
docker stack ls: List all the stacks running in the Swarm.
docker stack ps: List all the tasks (replicas) of a stack.
docker stack rm: Remove a stack.
**docker stack services**: List all the services of given stack.

💡 In this article, I will only discuss five Swarm commands. The remaining commands will be discussed in upcoming articles. I have invested a lot of time in this Docker Swarm series. If you find this article helpful, please consider liking and sharing it with others, and saving it for future reference.

`docker swarm init`

The docker swarm init command initializes a new Docker Swarm. This command should be executed on the node that you want to become the Swarm Manager. Once the Swarm Manager is initialized, you can add other nodes as Swarm Workers to the Swarm.

Here's an example command to initialize a new Docker Swarm:

$ docker swarm init --advertise-addr <MANAGER-IP-ADDRESS>

Here, <MANAGER-IP-ADDRESS> should be replaced with the IP address of the node that will act as the Swarm Manager. The --advertise-addr flag specifies the address that other nodes should use to communicate with the Swarm Manager.

After running the above command, you should see output similar to the following:

Swarm initialized: current node <NODE-ID> is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token <TOKEN> <MANAGER-IP-ADDRESS>:<MANAGER-PORT>

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

This output indicates that the Swarm Manager has been initialized successfully. It also provides instructions on how to add other nodes as Workers or Managers to the Swarm.

The docker swarm init command generates a unique token that is used to authenticate nodes when they join the Swarm. The token is displayed in the output of the docker swarm init command and is required when joining new nodes to the Swarm.

Here's how to reprint the join command for both worker and manager nodes.

To reprint the join command for a Docker Swarm node, you can use the following command on the manager node:

docker swarm join-token worker

This will print the join command for a worker node. If you want to add a manager node, you can run:

docker swarm join-token manager

This will print the join command for a manager node. You can copy and paste this command to the node you want to join the Swarm.

`docker swarm join`

docker swarm join is a command used to add a worker node or a manager node to an existing swarm. The command requires a token generated by a docker swarm init command from an existing manager node. The syntax of the command is as follows:

docker swarm join --token <token> <ip-address>:<port>

Here, <token> is the token generated by docker swarm init command and <ip-address> and <port> is the IP address and port of the existing manager node.

Once the command is executed successfully, the node will join the swarm and can be listed using the docker node ls command.

`docker node ls`

The docker node ls command is used to list all the nodes in the Docker Swarm. It provides information about the nodes such as their ID, hostname, status, availability, and number of running and stopped containers.

To use the docker node ls command, you need to have a Docker Swarm initialized and have at least one node joined to the Swarm

Once you have one or more nodes joined to the Swarm, you can use the docker node ls command to list all the nodes in the Swarm:

$ docker node ls

The output of the command includes information such as the hostname, the status of the node, the availability, the type of the node, and the number of running containers on the node. Here's an example output:

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
a95syht99niz6h15xy5gry0mp *   node-1              Ready               Active              Leader              20.10.5
9mijxl5b5nb3d2a3qzsdtnyho     node-2              Ready               Active                                  20.10.5
0xarz8pya5z5vmpgolcqd00tx     node-3              Ready               Active                                  20.10.5

In this example, there are three nodes in the Docker Swarm cluster, with node-1 being the leader. All nodes are ready and active, and are running Docker Engine version 20.10.5.

`docker service create`

The docker service create command is used to create a new service in a Docker Swarm. A service is a group of containers that are all running the same image and configuration, working together to provide a single application. The docker service create command allows you to specify the image, number of replicas, and other configuration options for the service.

Here is the basic syntax for the docker service create command:

docker service create [OPTIONS] IMAGE [COMMAND] [ARG...]

Here are some of the most commonly used options:

-name: Sets the name of the service
-replicas: Specifies the number of replicas of the service to create
-publish: Publishes a container's port to the host
-mount: Mounts a volume to the service
-env: Sets environment variables for the service
-network: Specifies the network to use for the service
**-label** : Sets the label of the service

Here is an example command that creates a new service called web:

docker service create --name web --replicas 3 --label mylabel=myvalue --publish published=8080,target=80 nginx:latest

This command creates a new service called web that runs the nginx:latest image with 3 replicas, publishes port 80 in the container to port 8080 on the host, and uses the default network for the service with label mylabel as the label key, and myvalue as the label value.

How to add env variables from the file ?

To add environment variables from a file while creating a Docker service, you can use the --env-file option with the docker service create command.

Here's an example command that creates a service and sets environment variables from a file named myenv.list:

docker service create --name myservice --env-file myenv.list myimage:tag

The myenv.list file should contain one environment variable per line, in the format VARNAME=value. For example:

DB_HOST=db.example.com
DB_PORT=5432

These environment variables will be passed to the container when it is started as part of the service.

`docker service ls`

The docker service ls command is used to list all the services running in the swarm. It provides a tabular view of the services, including the service ID, name, mode, replicas, image, and ports.

Here's an example output of the command:

ID             NAME          MODE         REPLICAS   IMAGE                 PORTS
bwv5f5m5wmlz   web           replicated   3/3        nginx:latest          *:80->80/tcp
ui01bmy5ue5m   api           replicated   2/2        myapp/api:latest      *:8080->8080/tcp
3bxq77hqlfjf   db            replicated   1/1        mysql:latest          *:3306->3306/tcp

This output shows three services running in the swarm: web, api, and db. The REPLICAS column shows the number of replicas for each service, and the PORTS column shows the published ports for each service.

To check the current number of replicas for a service, you can look at the REPLICAS column. In the example above, the web service has been configured to run 3 replicas, and all 3 are currently running and also*PORTS* column shows that the container's port 80 is being published on the host's port 80.

There are several options available for the docker service ls command. Here are some commonly used ones:

-filter: Allows you to filter the services based on various criteria such as service name, label, mode, etc.

Suppose we want to filter the list of services by the label com.example.description=web service. We can use the --filter option as follows:
```
$ docker service ls --filter label=com.example.description=web service
```
This command will list all the services that have the label com.example.description set to web service.
-format: Specifies the format of the output. You can choose from several predefined formats or create your own custom format.

Suppose we want to display only the service name and the number of replicas, and we want to use a custom output format. We can use the --format option as follows:
```
$ docker service ls --format "table {{.Name}}\t{{.Replicas}}"
```
This command will list all the services and display the service name and the number of replicas in a table format. The output will look something like this:
```
NAME        REPLICAS
web         3/3
api         2/2
database    1/1
```
In the --format option, we use the table format and specify the columns we want to display using the Go template syntax ({{.Name}} and {{.Replicas}}). The \t character is used to separate the columns with a tab.
-quiet or q: Only displays the IDs of the services.
-no-trunc: Displays the full service ID instead of truncating it.

`docker service ps`

The docker service ps command shows the running tasks for a given service. Each task represents a container running a service on a swarm node.

Here's how to use the docker service ps command:

docker service ps <service_name>

For example, if we have a service named webapp, we can list its tasks using the following command:

docker service ps webapp

The output will show information about the tasks running for the service, including the container ID, the node it's running on, the service replica number, and its current status.

Here's an example output:

ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
e9y2pr5z6nca        webapp.1            nginx:latest        node1               Running             Running 5 minutes ago                       
6f9cgm91swxc        webapp.2            nginx:latest        node2               Running             Running 5 minutes ago                       
3f6ap9q6ghor        webapp.3            nginx:latest        node3               Running             Running 5 minutes ago

We can also use the --format option to display specific information about each task in a custom format. For example, to display the task ID and the node it's running on, we can use the following command:

docker service ps --format "Task ID: {{.ID}} Node: {{.Node}} " webapp

The output will show the task ID and the node it's running on for each task in the webapp service:

Task ID: e9y2pr5z6nca Node: node1 
Task ID: 6f9cgm91swxc Node: node2 
Task ID: 3f6ap9q6ghor Node: node3

We can also use the --filter option to display only the tasks that match certain criteria. For example, to display only the tasks that are currently running, we can use the following command:

docker service ps --filter "desired-state=running" webapp

The output will show only the tasks that are currently running for the webapp service:

ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
e9y2pr5z6nca        webapp.1            nginx:latest        node1               Running             Running 5 minutes ago                        
6f9cgm91swxc        webapp.2            nginx:latest        node2               Running             Running 5 minutes ago                        
3f6ap9q6ghor        webapp.3            nginx:latest        node3               Running             Running 5 minutes ago

There are many values that can be used with the --filter flag in the docker service ps command to filter the output. Here are some common ones:

id : The id filter matches on all or a prefix of a task’s ID.

docker service ps -f "id=8" redis

Output of this command will look something like this :


ID             NAME      IMAGE        NODE      DESIRED STATE  CURRENT STATE      ERROR  PORTS
8ryt076polmc   redis.4   redis:3.0.6  worker1   Running        Running 9 seconds
8eaxrb2fqpbn   redis.10  redis:3.0.6  manager1  Running        Running 8 seconds

name : The name filter matches on task names

docker service ps -f "name=redis.1" redis

Output of this command will look something like this :


ID            NAME     IMAGE        NODE      DESIRED STATE  CURRENT STATE      ERROR  PORTS
qihejybwf1x5  redis.1  redis:3.0.6  manager1  Running        Running 8 seconds

node : The node filter matches on a node name or a node ID.

docker service ps -f "node=manager1" redis

Output of this command will look something like this :


ID            NAME      IMAGE        NODE      DESIRED STATE  CURRENT STATE      ERROR  PORTS
0qihejybwf1x  redis.1   redis:3.0.6  manager1  Running        Running 8 seconds
1x0v8yomsncd  redis.5   redis:3.0.6  manager1  Running        Running 8 seconds
3w1wu13yupln  redis.9   redis:3.0.6  manager1  Running        Running 8 seconds
8eaxrb2fqpbn  redis.10  redis:3.0.6  manager1  Running        Running 8 seconds

desired-state : The desired-state filter can take the values running, shutdown, or accepted

That's it for this article. If you found it useful, please share it with your friends and save it for future reference. In the next article, we will discuss remaining command. See you in the next one!

Building TabNode: A Social Media App for Tech Bloggers with Appwrite

Prasenjeet Kumar — Sun, 14 May 2023 05:51:57 +0000

I am participating in the upcoming Appwrite hackathon, and I am determined to win it no matter what. The hackathon will run from May 15, 2023 to June 14, 2023.

"What is Appwrite?”

Appwrite is a backend-as-a-service that helps developers launch applications faster by providing all the necessary services, such as databases, authentication, storage, and cloud functions. You can self-host it, but now they've launched Appwrite Cloud, so there's no need to self-host. You can simply purchase a cloud subscription and start using it, which is awesome 😃.

Have you used Appwrite before?

No, I have never coded anything using Appwrite. However, after seeing the hackathon tweet, I am super excited about it. I plan to learn everything about Appwrite by reading its documentation and watching free YouTube videos in just 2 to 3 days. You can count on me. I am passionate about this stuff 😐

Have you worked on similar products?

Yes! I am a big fan of Firebase and have built several applications using it in the past. You can find them on my GitHub profile, but note that some of the projects may be private 🤣. I am confident that I can learn Appwrite quickly because I already have experience with similar cloud-based technologies.

Why do you want to participate?

To make a long story short, I used to be very introverted, and to some extent, I still am. I made my computer my best friend and spent much of my time alone in my dark room with it. During this period, I only communicated with my very close school friend and my family. I didn't like making new friends because I didn't want casual friendships. I wanted friends who could trust me, help me become a better person, and stay with me lifelong. You might say I was afraid of making new friends due to past trauma.

But now I know that I can't do everything alone, especially if my dream is to start my own company. I must go out into the world, make friends, communicate, and seek help from the community and friends. That's why I decided, for the first time, to participate in this hackathon. By doing so, I can involve the Appwrite and Hashnode communities, make new friends, and hopefully land my dream job at one of the companies listed in my Notion's top 10 list. And guess what? Appwrite and Hashnode are both on that list.

Where does Appwrite rank on your list of dream companies?

I am unable to disclose it at the moment, but I plan to release it after winning this hackathon. I am confident that I can win, and I will share the list with you all afterward.

Do you have experience building large-scale applications?

Yes, I live for this kind of work. I promise to make my repository public after winning this hackathon because I love open-source. I want everyone to be able to contribute to my large-scale applications and learn from them. It would be a dream come true if you would open issues and submit pull requests to my project.

Do you have experience in UI/UX?

Hey, just so you know, I'm really passionate about this kind of work. I have the skills to build a product from scratch and take it all the way to production-grade level, ready to launch to the market. So you can definitely count on me.

Can you tell us your age and gender?

Yes, I am a 23-year-old male who turns ideas into reality 😎

What are you building?

I plan to develop a social media application for tech bloggers that will include the following features:

Users can register with GitHub and Twitter only.
Users can set up an initial profile and choose topics (tags) that interest them.
Users can create posts using Markdown. Initially, each user is limited to two posts to reduce costs.
The search feature allows any user to easily search for other users.
Our AI system will automatically detect up to four tags for each blog post.
Users can follow each other, and we will use a recommendation engine to suggest other users to follow.
Users can see a personalized feed.
Users can comment, upvote, downvote, save, and share blog posts.
Users can subscribe to bloggers to see their exclusive private blog posts, and they can also purchase individual blog posts.
The feed page will include a blog story section that shows all the stories of the bloggers the user is following. This section will include an auto-generated summary and a call-to-action to read more. Bloggers can configure the theme of this section from their account.
Each story will have a TTL of 12 hours and will be sorted according to the published date.
When bloggers post new blogs, they will have the option to post them as stories that will go to all their followers.

Have you decided on the name of this app?

Yes! I am going to call it TabNode, inspired by HashNode.

What tech stack are you going to use?

I am planning to use the following technologies:

Appwrite, obviously, for the backend part.
We will use Figma to design and prototype.
React or Angular in the frontend side, and Netlify for hosting.
In-app notification system (NOVU).
GPT-3.5-Turbo or Huggingface for AI Part.
Stripe as the payment gateway.

Do you need team members?

Yes, of course. Working with a team is something I crave. I have spent a lifetime building things alone and now I want to gain experience working collaboratively with team members and future friends.

I am looking to form a team of two and am searching for my next team member. Anyone who thinks they can contribute to this project, is willing to work day and night, and can learn the technology on the go is welcome.

If you have already built projects using the stacks described above, that would be a plus. Please feel free to contact me.

Where can we contact you?

You can contact me on my Twitter and GitHub.

Twitter: @Prasenjeetsymon
GitHub: prasenjeet-symon

Will you split the prize?

If we win, I will split the prize equally among the team members. I believe that every team member should be credited equally, regardless of the amount or quality of work they contributed. Once someone becomes a team member, they are an equal contributor to the project and should receive equal credit.

Why should we join you?

I believe I have the capability and skillset to successfully complete this project. By joining me as a team member, our chances of winning will increase, as it will speed up project development. However, I am not certain how much time it will take to complete the project or what unseen challenges we may face.

Joining me on this project may provide us with increased visibility among the Appwrite and Hashnode communities, which could benefit us both. Are you ready to embark on this adventure with me? Let's do it! 💪

Mastering Docker Compose: Essential CLI Commands ( part 6 )

Prasenjeet Kumar — Sat, 13 May 2023 05:04:18 +0000

In the previous article, we discussed the use cases and benefits of Docker Compose and multi-container applications. In this article, we will continue our discussion of Docker Compose commands.

In the previous article, we covered five Docker Compose commands: **docker-compose up**, docker-compose down, **docker-compose stop**, docker-compose start, and **docker-compose restart**.

In this article, we will discuss the remaining commands in great detail so that you can become a Docker hero from scratch. Let's get started!

`docker-compose pause` & `docker-compose unpause`

The docker-compose pause command is used to pause one or more containers in a Docker Compose project. When a container is paused, it will stop responding to any new requests, but its state and configuration will be preserved.

Here is an example of how to use docker-compose pause:

Let's say we have a simple Docker Compose file named docker-compose.yml with two services: a Redis database and a Node.js application that depends on Redis.

version: "3"
services:
  redis:
    image: redis:alpine
  app:
    build: .
    environment:
      - REDIS_HOST=redis
    depends_on:
      - redis
    ports:
      - "3000:3000"

To pause the Redis container, we can run the following command:

docker-compose pause redis

This will pause the Redis container, and any new requests to the Redis service will fail until it is resumed.

To resume the Redis container, we can run the following command:

docker-compose unpause redis

This will resume the Redis container, and it will start responding to new requests again.

Note that docker-compose pause and docker-compose unpause can be used with multiple container names as arguments, allowing you to pause and unpause multiple containers at once.

`docker-compose ps`

The docker-compose ps command is used to display the status of containers defined in the Compose file. It lists all the containers that are created based on the services defined in the docker-compose.yml file along with their status, ports, and names.

The basic syntax of the docker-compose ps command is:

docker-compose ps [options] [SERVICE...]

Here, SERVICE is an optional argument that allows you to filter the output by specifying the name of the service. If you don't specify any service name, it will show the status of all the services defined in the docker-compose.yml file.

Some commonly used options with the docker-compose ps command are:

q, --quiet: Only display container IDs.
-services: Display only service names.
-filter: Filter services by a property.

Here's an example of using the docker-compose ps command:

Suppose we have a docker-compose.yml file that defines three services: web, db, and redis.

version: '3'
services:
  web:
    image: nginx:latest
    ports:
      - "8080:80"
  db:
    image: mysql:latest
    environment:
      MYSQL_ROOT_PASSWORD: password
  redis:
    image: redis:latest

To display the status of all the containers defined in this file, we can run the following command:

docker-compose ps

This will output the status of all three services like this:

Name                   Command               State            Ports
-------------------------------------------------------------------------------
compose_db_1   docker-entrypoint.sh mysqld      Up      3306/tcp, 33060/tcp
compose_redis_1 /usr/local/bin/docker-entr ...   Up      6379/tcp
compose_web_1   nginx -g daemon off;             Up      0.0.0.0:8080->80/tcp

Here, we can see the status of all three services, their names, and their respective ports.

`docker-compose logs`

The docker-compose logs command is used to display the logs of services running inside the Docker Compose environment. It can be used to view the output of all the services or specific services, and provides options to display timestamps, tail logs and limit the number of lines displayed.

The basic syntax for using the docker-compose logs command is:

docker-compose logs [options] [service...]

Here are some of the commonly used options for docker-compose logs:

-follow, f: Follow the logs in real-time like tail -f.
-timestamps, t: Display timestamps along with the log lines.
-tail: Number of lines to display from the end of the logs. For example, -tail 50 will display the last 50 lines.
-no-color: Disable colored output

Suppose we have a docker-compose.yml file with two services, web and db. We can use the docker-compose logs command to view the logs for both services like this:

$ docker-compose logs

To view the logs for a specific service, we can specify the service name as an argument:

$ docker-compose logs db

We can also use the --followoption to follow the logs in real-time:

$ docker-compose logs -f

To limit the number of log lines displayed, we can use the --tailoption:

$ docker-compose logs --tail 50

These are some of the basic options for using the docker-compose logscommand.

`docker-compose build`

The docker-compose buildcommand is used to build the Docker images for the services defined in a docker-compose.ymlfile. It reads the Dockerfilein each service's build context and generates an image based on the instructions defined in the Dockerfile. It also tags the image with the service name and a version if specified.

The docker-compose build command supports several options, some of which are:

-force-rm: Removes intermediate containers and forces a new build.
-no-cache: Builds the image without using cache.
-pull: Always attempts to pull a newer version of the image.

Here is an example usage of docker-compose build command:

version: '3.8'
services:
  web:
    build:
      context: .
      dockerfile: Dockerfile

In this example, the web service is built using the Dockerfilein the current directory (.
). We can then run docker-compose buildto build the image for the web service.

$ docker-compose build
Building web
Step 1/5 : FROM node:14
 ---> 0123456789ab
Step 2/5 : WORKDIR /app
 ---> Using cache
 ---> 9876543210fe
Step 3/5 : COPY package*.json ./
 ---> Using cache
 ---> fedcba987654
Step 4/5 : RUN npm install
 ---> Using cache
 ---> 4567890abcde
Step 5/5 : COPY . .
 ---> 0123456789ab
Successfully built 0123456789ab
Successfully tagged myapp_web:latest

This will build the webservice using the Dockerfilein the current directory, and tag the resulting image with the name myapp_web:latest.

To build a single service in a Docker Compose file, you can use the docker-compose build command followed by the name of the service you want to build. Here is an example:

Suppose you have a docker-compose.yml file that defines two services, app and database. To build only the app service, you can use the following command:

docker-compose build app

This will build the Docker image for the app service based on the instructions in its Dockerfile. The database service will not be built because it was not specified.

If you want to rebuild the image for the app service even if it has already been built, you can add the --no-cache option:

docker-compose build --no-cache app

This will force Docker to rebuild the image from scratch, ignoring any previously cached layers.

Note that if the app service depends on other services, those services will also be built if they have not been built yet. If you only want to build the app service and not its dependencies, you can use the --no-deps option:

docker-compose build --no-deps app

This will skip building any services that the appservice depends on.

`docker-compose config`

The docker-compose config command is used to validate and view the Compose file in its final merged form after variable substitution and interpolation. It can be used to check if there are any syntax errors or configuration issues in the Compose file.

When run, docker-compose config reads the docker-compose.yml file, processes it and prints the resulting configuration to the standard output.

Here's an example usage:

docker-compose config

This will check and display the final configuration for the services defined in the docker-compose.yml file.

You can also specify a specific Compose file by using the -f option, like this:

docker-compose -f docker-compose.prod.yml config

This will check and display the final configuration for the services defined in the docker-compose.prod.yml file.

Note that docker-compose config does not actually start or stop any containers, it simply validates and displays the configuration.

`docker-compose exec`

The docker-compose exec command is used to run a command in a running container. This command allows you to interact with a container as if you are logged in to it, without having to start a new shell session.

The basic syntax of the command is:

docker-compose exec [options] service_name command

Here, service_name is the name of the service you want to execute the command in, and command is the command you want to execute in the service's container.

Some common options used with docker-compose exec are:

u: This option allows you to specify the user context in which the command should be run.
T: This option disables pseudo-tty allocation, which is useful when running non-interactive commands.
-index: This option allows you to specify which container instance to run the command in. By default, the first instance is used.

For example, let's say you have a service named web that runs a web server, and you want to run a command in its container to check the version of a software package:

docker-compose exec web bash -c "apt show my-package | grep Version"

This will execute the bash -c "apt show my-package | grep Version"command in the web
service's container, and print the output to the terminal.

`docker-compose pull`

The docker-compose pull command is used to pull (download) images for all services listed in the docker-compose.yml file. This command is used to ensure that the latest image of each service is downloaded before starting the services. If there are any updates available for the images, docker-compose pull will download them.

Here is an example of using docker-compose pull command:

Assume that you have a docker-compose.yml file that specifies two services, web and db, as follows:

version: '3'
services:
  web:
    image: nginx:latest
  db:
    image: mysql:latest

To download the latest images for both services, you can run the following command:

docker-compose pull

This will pull the latest nginxand mysqlimages from Docker Hub. If you have already pulled the images before, docker-compose pullwill only download the updates for the images, if any.

`docker-compose push`

The docker-compose push command is used to push images of the services defined in the docker-compose.ymlfile to a Docker registry. This command assume that you are pushing an image you have built locally.

The basic syntax of the command is as follows:

docker compose push [OPTIONS] [SERVICE...]

where [SERVICE...]is an optional list of services to push. If no service is specified, all services defined in the docker-compose.yml file are pushed.

services:
  service1:
    build: .
    image: localhost:5000/yourimage  ## goes to local registry

  service2:
    build: .
    image: your-dockerid/yourimage  ## goes to your repository on Docker Hub

To push the images of these services to a Docker registry, run the following command:

docker-compose push

This command will push the service2 to the docker Hub registry and service1 to local registry.

`docker-compose rm`

The docker-compose rmcommand is used to remove stopped containers created by the docker-compose upcommand along with their respective volumes. It can be used to clean up any resources that may have been left behind by the containers.

By default, the docker-compose rm command will remove all stopped containers associated with the current docker-compose.yml file. However, it is also possible to target a specific service or container using the --service or --container options.

Here's the basic syntax of the docker-compose rm command:

docker-compose rm [options] [SERVICE...]

Some common options that can be used with the docker-compose rm command are:

-stop : Stops the containers before removing them.
-force : Forces the removal of the containers even if they are running.
-volumes : Removes the anonymous volumes associated with the containers.

For example, to remove all stopped containers and their volumes for the services defined in the docker-compose.yml file, you can run the following command:

docker-compose rm

To remove a specific service and its stopped containers, you can use the --service
option followed by the name of the service:

docker-compose rm --service myservice

And to force the removal of running containers, you can use the --forceoption:

docker-compose rm --force

💡 Running the command with no options also removes one-off containers created by docker compose run

`docker-compose run`

The docker-compose run command is used to run a one-off command on a service defined in a Compose file. This allows you to run a command within a specific environment defined by the Compose file.

The basic syntax of the command is as follows:

docker-compose run [options] [service_name] [command]

Here, [options] are the additional options that you can pass to the command, [service_name] is the name of the service to run the command on, and [command] is the command to run.

Some common options used with the docker-compose run command are:

d or -detach: Run the container in the background and print the new container ID.
e or -env: Set an environment variable for the container.
-name: Assign a name to the container.
v or -volume: Mount a volume from the host to the container.

For example, to run the command ls on the web service defined in a Compose file, you would use the following command:

docker-compose run web ls

This would start a container for the webservice and run the lscommand inside it. Once the command finishes, the container will be stopped and removed.

Here are some use cases for the docker-compose runcommand:

Run a one-time command: You can use docker-compose run to run a command in a service container just once, without starting the service itself. For example, you can run a database migration script with the following command:
```
docker-compose run webapp python manage.py migrate
```
This will start a new container for the webapp service, run the python manage.py migrate command inside it, and then stop and remove the container.
Run a command with environment variables: You can use docker-compose run to set environment variables for a command that you want to run. For example:
```
docker-compose run -e DEBUG=True webapp python app.py
```
This will start a new container for the webapp service, set the DEBUG environment variable to True, run the python app.py command inside it, and then stop and remove the container.
Debug a service: You can use docker-compose run to start a service container in interactive mode and explore its environment. For example:
```
docker-compose run --service-ports webapp bash
```
This will start a new container for the webapp service, start a Bash shell inside it, and expose the container's ports so that you can access them from your local machine. This is useful for debugging and troubleshooting.
Test a service: You can use docker-compose run to test a service in isolation, without affecting other services in your composition. For example, you can run unit tests for your web application with the following command:
```
docker-compose run webapp python manage.py test
```
This will start a new container for the webapp service, run the python manage.py test command inside it, and then stop and remove the container.

What is `-service-ports`

The --service-ports option is used with the docker-compose run command to publish all exposed ports of the service containers to the host machine. By default, the container's ports are not exposed to the host. This option is useful when you want to run a one-off command in a service container and need to access its exposed ports from the host machine.

For example, let's say you have a web service in your docker-compose.yml file that exposes port 80:

version: '3'
services:
  web:
    build: .
    ports:
      - "80:80"

Now, you want to run a curlcommand in the webservice container to test the website. You can use the docker-compose runcommand with the --service-portsoption to access the website from the host machine:

docker-compose run --service-ports web curl http://localhost

The --service-portsoption publishes the webservice container's exposed port 80 to the host machine, so you can access the website using http://localhost.

If you found this article helpful, please give it a thumbs up and save it for future reference. If you have any questions or comments, feel free to leave them below.Thank you for reading! I'll see you in the next article.

Discover the hidden gems of multi-container applications ( part 5 )

Prasenjeet Kumar — Mon, 08 May 2023 16:35:45 +0000

What is multi-container Docker applications ?

Multi-container Docker applications are composed of multiple containers that are designed to work together to provide the complete functionality of an application. The containers within a multi-container application are typically interconnected and work in concert to accomplish a specific task or set of tasks.

For example, a web application might consist of a front-end container that serves HTML pages and a back-end container that handles database queries. Both of these containers would be part of the same multi-container application and would work together to provide a fully functional web application.

Where do we use multi-container applications?

Multi-container Docker applications are used in several scenarios, including:

Microservices architecture: A microservices architecture breaks down complex applications into smaller, independent services that can be developed, deployed, and scaled separately. Each microservice can be run in its own container, and multiple containers can communicate with each other using Docker networking or other mechanisms.
Development and testing environments: Docker Compose allows developers to define the entire stack for their applications, including databases, message queues, and other services. This allows them to quickly spin up the environment on their local machine and test their code changes before deploying to production.
High availability and scaling: By running multiple containers for a single application, Docker Swarm or Kubernetes can manage the containers and provide load balancing, high availability, and scaling features. This helps to ensure that the application is always available and can handle varying levels of traffic.
Separation of concerns: Separating the application into multiple containers allows each container to have a specific role or responsibility. For example, one container may handle the application logic, while another container may handle the database. This separation of concerns can help with maintainability and scalability.

Why should we use multi-container applications?

There are several benefits to using multi-container Docker applications:

Scalability: With multi-container Docker applications, you can easily scale individual parts of your application independently. For example, if one part of your application is experiencing high traffic, you can scale that part of the application without having to scale the entire application.
Isolation: By using multiple containers, you can isolate different parts of your application from each other. This can help to prevent issues in one part of the application from affecting other parts.
Resource utilization: By using multiple containers, you can optimize your resource utilization. For example, if one part of your application requires more CPU or memory than another part, you can allocate resources accordingly.
Simplified deployment: With multi-container Docker applications, you can deploy all parts of your application together in a single deployment unit, making it easier to manage and deploy your application.
Improved maintainability: With multiple containers, you can update and maintain different parts of your application separately, making it easier to keep your application up-to-date and secure.

Overall, multi-container Docker applications provide a flexible, scalable, and maintainable way to deploy complex applications.

Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you can define a set of related services in a YAML file, then start and stop them all together with a single command.

Here are some of the most commonly used Docker Compose commands:

up: Create and start containers.
down: Stop and remove containers, networks, images, and volumes.
start: Start existing containers.
stop: Stop running containers.
restart: Restart running containers.
pause: Pause running containers.
unpause: Unpause paused containers.
ps: List containers.
logs: View output from containers.
build: Build or rebuild services.
config: Validate and view the Compose file.
exec: Execute a command in a running container.
pull: Pull service images.
push: Push service images.
rm: Remove stopped containers.
run: Run a one-time command against a service.

In this series on Docker, we will discuss each command in great detail so that you can become proficient in using Docker. However, in this particular article, we will only be covering 5 commands.

💡 I spent hours writing this article in a way that is easy to understand for those new to Docker. If you found it helpful, please like and save it for future reference.

`docker-compose up`

The docker-compose up command is used to build, create, start, and attach to containers for a set of services defined in a docker-compose.yml file.

Here is an example docker-compose.yml file :

version: '3.9'  # specify the Compose file version

services:  # define the services/containers

  db:  # define the MySQL service
    image: mysql:latest  # specify the MySQL image to use
    restart: always  # restart the container automatically if it crashes
    environment:  # set environment variables
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: db_name
      MYSQL_USER: db_user
      MYSQL_PASSWORD: db_password
    ports:  # map container port to host port
      - "3306:3306"
    volumes:  # mount a host directory or named volume to the container
      - db_data:/var/lib/mysql

  app:  # define the Node.js/Express service
    build: ./app  # build the Docker image using the Dockerfile in the app directory
    restart: always  # restart the container automatically if it crashes
    environment:  # set environment variables
      NODE_ENV: development
      DB_HOST: db
      DB_USER: db_user
      DB_PASSWORD: db_password
      DB_NAME: db_name
    ports:  # map container port to host port
      - "8080:8080"
    volumes:  # mount a host directory or named volume to the container
      - ./app:/app

  client:  # define the React service
    build: ./client  # build the Docker image using the Dockerfile in the client directory
    restart: always  # restart the container automatically if it crashes
    ports:  # map container port to host port
      - "3000:3000"
    volumes:  # mount a host directory or named volume to the container
      - ./client:/client

volumes:  # define named volumes to be used in the services
  db_data:

To start these services, we can use the docker-compose up command from the same directory as the docker-compose.yml file:

$ docker-compose up

This will build and start the db, client, and appservices. The containers will be started in the foreground, and we'll see the logs from all containers in the console.

💡 When you run docker-compose up, a default network is created. For example, if your project is named myapp, a network called myapp_default is created. Each container for a service joins the network under the service’s name. For example, a container created using a service named web would join the network under the name web

When running docker-compose up, there are a few options that can be used to modify the behavior of the command. Here are some of the most common options:

`--detach`

The -d or --detach option is used with the docker-compose up command to run the containers in the background and print the container IDs.

By default, docker-compose up runs the containers in the foreground and logs their output to the terminal. However, with the -d option, the command returns immediately, and the containers continue to run in the background.

For example, to start the containers defined in the docker-compose.yml file in the background, you can use the following command:

docker-compose up -d

This will start the containers in the background and print their IDs.

`--build`

The docker-compose up --build command is used to build the images for the services defined in the docker-compose.yml file before starting the containers. This command is useful when changes are made to the application code or Dockerfile of a service, and you want to rebuild the image to include those changes.

When you run docker-compose up --build, it will first build the images for any services that have a Dockerfile in their build context or use a custom build command specified in the build section of the docker-compose.yml file. Then it will start the containers for all services defined in the docker-compose.yml file.

Here is an example command to run docker-compose up --build:

docker-compose up --build

This will build the images for all services defined in the docker-compose.yml file before starting the containers.

`--force-recreate`

The --force-recreate option for the docker-compose up command recreates all containers defined in the docker-compose.yml file, even if they are already running, and forces them to be built again. This option can be used to rebuild containers with updated configurations or to recover from a corrupted state.

By default, Docker Compose will try to reuse existing containers if possible, which can be useful for faster development and testing. However, in some cases, it may be necessary to rebuild and recreate all containers from scratch, such as when changing the Dockerfile or updating dependencies.

Here is an example of using --force-recreate with docker-compose up:

docker-compose up --force-recreate

This command will stop and remove all containers defined in the docker-compose.yml file, and then recreate them with the latest configurations and images. It can be useful when deploying changes to a production environment or when troubleshooting issues with containers.

💡If a service is already running, the docker-compose up command will not start another instance of that service. Instead, it will leave the running instance as it is and continue to start up any other services that are defined in the docker-compose.yml file.

If you want to force a restart of the running service, you can use the docker-compose up --force-recreate command. This will stop the running service and start a new instance of it with the latest configuration defined in the docker-compose.yml file.

If you want to recreate a specific service, you can use the docker-compose up --force-recreate <service> command. This will recreate only the specified service and not the others.

For example, if you have a web and a db service defined in your docker-compose.yml file, and you want to recreate only the db service, you can run the following command:

docker-compose up --force-recreate db

This will recreate the db service and leave the web service running as is.

`-no-recreate`

If containers already exist, don't recreate them. This option is useful when you don't want to recreate the containers each time you run the command.

`-no-build`

Don't build an image, even if it's outdated or missing. This option is useful when you don't want to build the image each time you run the command.

`--remove-orphans`

The --remove-orphans option is used with the docker-compose up command to remove any containers that were created by the current project but are no longer defined in the Compose file. Orphaned containers can occur when a container is created but its service definition is later removed from the Compose file.

Here is an example of using the --remove-orphans option with docker-compose up command:

docker-compose up --remove-orphans

This will start all the services defined in the Compose file and remove any orphaned containers.

💡 If a container is running that is not defined in the Compose file, it will not be affected by the --remove-orphans option. This option is only used to remove containers that were created as part of the Compose service definition, but are no longer defined in the Compose file.

`-timeout`

Specify a shutdown timeout in seconds. This option is used to specify how long Compose should wait for containers to shut down before killing them.

`--scale`

The --scale option in docker-compose is used to scale a service up or down by increasing or decreasing the number of containers running for that service. It allows you to run multiple instances of the same service, with the same configuration, simultaneously.The syntax of using the --scale option is as follows:

docker-compose up --scale service_name=num_instances

Here, service_name is the name of the service that you want to scale and num_instances is the number of instances you want to run for that service.

For example, if you have a web service in your docker-compose.yml file and you want to run three instances of that service, you can use the following command:

docker-compose up --scale web=3

This will start three instances of the web service.

💡 Note that the --scale option only works for services that are defined in the docker-compose.yml file. If a service is not defined in the file, the --scale option will have no effect.

`docker-compose down`

docker-compose down is a command used to stop and remove the containers, networks, and volumes that were created by docker-compose up.

By default, docker-compose down will stop and remove the containers declared in the docker-compose.yml file, and remove the network, but will not remove volumes. To remove the volumes as well, you can use the -v or --volumes option.

Here is the basic syntax for docker-compose down:

docker-compose down [options]

Here are some commonly used options:

v, --volumes: Remove named volumes declared in the volumes section of the docker-compose.yml file
-rmi <all|local>: Remove images. The all option removes all images used by any service defined in the docker-compose.yml file. The local option removes only images that don't have a custom tag or are not used by any other service.
t, --timeout <timeout> : Specify a shutdown timeout (default is 10 seconds).
-remove-orphans : Remove containers for services not defined in the docker-compose.yml file.

Here's an example command to stop and remove the containers and network defined in a docker-compose.yml file:

docker-compose down

If you also want to remove the volumes, you can add the -v option:

docker-compose down -v

The -rmi option is used to remove images created by docker-compose build. It accepts two values:

all: Remove all images used by any service defined in the docker-compose.yml file.
local: Remove only the images that are not associated with any container. This is the default option.

Here are some examples:

To remove all containers, networks, and volumes created by docker-compose up , and also remove all images used by the services:

docker-compose down --rmi all

To remove all containers, networks, and volumes created by docker-compose up , and only remove the images that are not associated with any container:

docker-compose down --rmi local

`docker compose stop`

The docker-compose stop command stops the containers defined in the docker-compose.yml file. It sends a SIGTERM signal to the container processes, giving them a chance to stop gracefully. If the processes do not stop within the given timeout period (default is 10 seconds), then a SIGKILL signal is sent to forcefully terminate the processes.

The syntax of the command is:

docker-compose stop [options] [SERVICE...]

Where options can include:

t, --timeout TIMEOUT : Specify a custom timeout value (in seconds) for stopping the containers.

And SERVICE... is an optional list of service names to stop. If no service name is provided, all services defined in the docker-compose.yml file will be stopped.

Here's an example of how to use the docker-compose stop command:

docker-compose stop

This will stop all the containers defined in the docker-compose.yml file.

docker-compose stop mysql

This will stop only the mysql service defined in the docker-compose.yml file.

💡 If we stop a service that is a dependency for another service, the dependent service will also stop. For example, if we have two services, dband web, and web depends on db, if we stop the dbservice, the webservice will also stop.

`docker-compose start`

The docker-compose start command is used to start containers that are already created but are currently stopped. It does not recreate containers or rebuild images. This command is useful when you have stopped a container using docker-compose stop and want to start it again.

Here's an example of using docker-compose start command:

Suppose you have a docker-compose.yml file that defines two services app and db. If you start the services using the following command:

docker-compose up -d

It will create and start the containers in detached mode. Now, if you want to stop the containers, you can use the following command:

docker-compose stop

This will stop the running containers. If you want to start the containers again, you can use the docker-compose start command:

docker-compose start

This will start the stopped containers.

You can start a specific service using the docker-compose start command followed by the service name. For example, if you have a service called web defined in your docker-compose.yml
file, you can start it using the following command:

docker-compose start web

This will start only the web service and any services that it depends on. If the service is already running, it will not be restarted.

`docker-compose restart`

The docker-compose restart command is used to restart one or more services defined in the docker-compose.yml file.

Here is the basic syntax of the command:

docker-compose restart [options] [SERVICE...]

Some important options that can be used with this command are:

-timeout TIMEOUT : This option sets the timeout for stopping the containers. The default value is 10 seconds.

Here is an example command that restarts the web service defined in the docker-compose.yml file:

docker-compose restart web

This command will stop and then start the web service.

In conclusion, Docker Compose is a powerful tool for managing multi-container applications. It allows you to define and run complex applications with ease, and provides a range of options for managing containers and services. In this article, we covered some of the most common Docker Compose commands and options, including docker-compose up, docker-compose down, docker-compose start, and docker-compose restart. We also looked at options for managing container builds, scaling services, and removing orphaned containers.

Docker : From Zero to Hero ( part 4 )

Prasenjeet Kumar — Sat, 06 May 2023 17:20:53 +0000

Docker volumes are a way to persist data outside the lifetime of a container. They are used to share data between containers or between a container and the host system. Docker volumes can be managed using the docker volumecommand.

Wait a minute, why do we need volume?

Docker volumes were invented to provide a mechanism for storing and sharing data between containers, and between containers and the host system. Prior to the introduction of volumes, data storage in Docker was tied to the container's lifecycle. This meant that any data stored in the container would be lost when the container was removed or recreated. Volumes provide a way to persist data even after the container that created them has been deleted, and also enable easy sharing of data between containers. Additionally, volumes allow for more flexibility in how and where data is stored, such as using external storage systems or cloud-based storage solutions.

Different type of volume

Docker provides different types of volume to manage the persistent data for containers. The different types of volumes in a Docker container are:

Named Volumes:When using named volumes in Docker, the data is stored on the Docker host machine in a directory managed by Docker. Use named volumes when you need to persist data generated by a container or share data between multiple containers. For example, you can use a named volume to store a database's data so that it is not lost when the container is removed or restarted.
Host Bind Mounts: A host bind mount is a way to mount a directory of your choice from the host machine to the container. you can use a bind mount to mount a local directory containing configuration files into a container to customize its behavior. Suppose you want to customize the NGINX config file located inside the container. You can use a host bind mount to override the container's config file with that of the host.
Anonymous Volumes: Anonymous volumes are created by Docker when a container is started without specifying a named volume. Anonymous volumes are not named, so they cannot be reused. They are deleted when the container is removed.
Tmpfs Mounts: Tmpfs mounts are mounted into the container’s memory, not on the host filesystem. They are temporary volumes and exist only while the container is running. Once the container is stopped, the tmpfs mount is removed along with all the data stored inside it.

Each type of volume has its own use case, and choosing the right type of volume depends on the needs of the application.

Volumes can also be created with specific options, such as read-only access, or with a specific driver, which determines where the data is stored.But for now, we will not go into the driver's rabbit hole.

Some common docker volume commands include:

docker volume create: creates a new named or anonymous volume
docker volume ls: lists all available volumes
docker volume inspect: displays detailed information about a specific volume
docker volume rm: removes a specific volume
docker volume prune: removes all unused volumes

`docker volume create`

docker volume create is a command used to create a new named or anonymous volume in Docker. A named volume is a volume that has a specific name, while an anonymous volume is a volume that is not given a specific name and is only referenced by its container.

The syntax for docker volume create is:

docker volume create [OPTIONS] [VOLUME]

Where OPTIONSare any optional parameters you want to specify for the volume, and VOLUME
is the name you want to give the volume (if any). If you do not provide a name, an anonymous volume will be created.

Some common options for docker volume create include:

-driver: Specifies the volume driver to use for the volume.
-label: Adds a label to the volume.
-opt: Allows you to specify additional options for the volume driver.

To create a new named volume named "my_volume", you would run the following command:

docker volume create my_volume

This will create a new named volume named "my_volume" that can be used by Docker containers.

`-driver`

The --driver option with docker volume create specifies the driver to use for creating a new volume.

A volume driver is a plugin that allows Docker to use external storage systems for managing persistent data. By default, Docker comes with a built-in local driver for managing volumes on the Docker host. However, other volume drivers can be used for managing volumes on remote or cloud storage systems.

Here is an example of creating a new volume with the --driver option:

docker volume create --driver=local my_volume

In this example, a new volume named my_volumeis created with the built-in localdriver.

`-label`

The -label option can be used with the docker volume create command to add metadata labels to a volume. These labels can be used to tag or categorize volumes, and can be helpful for organization and management purposes.

For example, you could create a volume for a specific project and add a label to indicate the project name or version:

docker volume create --label project=myproject --label version=1.0 myvolume

You can then use the docker volume lscommand with the --filteroption to filter and list volumes based on their labels:

docker volume ls --filter label=project=myproject

`docker volume ls`

docker volume ls command is used to list all the volumes in Docker. When you run this command, it shows a list of volumes along with their names, driver types, and other details such as mountpoint, size, and labels.

Here is an example of how to use the docker volume ls command:

$ docker volume ls

This command will show a list of all the volumes that exist on your Docker host. If there are no volumes, the command will not display anything.

You can also use the --filteroption to filter volumes based on different criteria. For example, you can list only dangling volumes using the following command:

docker volume ls --filter dangling=true

This will only list volumes that are not associated with any containers.

You can also use the --format option to customize the output format. For example, you can show only the volume names using the following command:

docker volume ls --format "{{.Name}}"

This will produce output similar to the following:

my_volume
another_volume

`docker volume inspect`

The docker volume inspectcommand is used to display detailed information about a volume, including its configuration options and labels.

The basic syntax of the command is as follows:

docker volume inspect VOLUME [VOLUME...]

where VOLUME is the name or ID of the volume to inspect.

For example, to inspect a volume named myvolume, run the following command:

docker volume inspect myvolume

This will display a JSON-formatted output containing information about the volume, such as its name, driver, mountpoint, and labels.

If you want to get a more readable output, you can use the --format option to specify a Go template. For example, the following command will display the volume name, driver, and mountpoint:

docker volume inspect --format '{{.Name}} {{.Driver}} {{.Mountpoint}}' myvolume

Note that you can inspect multiple volumes by specifying their names or IDs as arguments to the command. The output will contain information about all the volumes in a JSON array format.

`docker volume rm`

The docker volume rm command is used to remove one or more Docker volumes that are no longer needed. The basic syntax of the command is as follows:

docker volume rm VOLUME [VOLUME...]

Here, VOLUME refers to the name or ID of the volume(s) to be removed.

For example, to remove a single volume named mydata, the following command can be used:

docker volume rm mydata

To remove multiple volumes at once, simply list their names or IDs separated by a space:

docker volume rm mydata1 mydata2 mydata3

It is important to note that once a volume is removed, all data stored in that volume will be lost permanently. Therefore, it is recommended to use this command with caution and ensure that any important data is backed up before proceeding with the removal.

Additionally, if a volume is currently being used by a container, it cannot be removed until it is disconnected from the container. Therefore, it may be necessary to stop and remove any containers that are currently using the volume before attempting to remove it.

`docker volume prune`

docker volume prune is a command used to remove all the unused volumes in Docker. When you delete a container, its associated volumes remain on your system, taking up valuable disk space. The prune command helps you free up that space by removing all the volumes that are not used by any containers.

To use the docker volume prune command, simply type the following in your terminal:

docker volume prune

This will remove all the unused volumes on your system. If you want to bypass the confirmation prompt, you can use the -f or --force flag, like this:

docker volume prune -f

Keep in mind that this command will remove all the unused volumes, so make sure you don't have any important data stored in those volumes. If you're not sure which volumes are in use and which ones are not, you can use the docker volume lscommand to list all the volumes on your system and their status.

Before we move on to Docker Compose and multi-container Docker applications, there is one more command we need to teach you quickly.

One last command `docker exec`

docker exec is a command used to execute a command in a running container. It allows you to run a command inside a container that is already running.

The syntax for docker exec is as follows:

docker exec [OPTIONS] CONTAINER COMMAND [ARG...]

OPTIONS are optional arguments that can be used to configure the behavior of the docker exec command. For example, you can use i to keep STDIN open even if not attached, or t to allocate a pseudo-TTY.
CONTAINER is the name or ID of the container you want to execute a command in.
COMMAND is the command you want to execute in the container.
ARG is an optional argument that can be passed to the command.

Here's an example of how to use docker exec:

docker exec -it my_container_name sh

This will open a shell inside the container named my_container_name. -it options are used to allocate a pseudo-TTY and keep STDIN open even if not attached, so that you can interact with the shell.

You can also run a command in the container, for example:

docker exec my_container_name ls -l

This will execute the ls -l command inside the container named my_container_name, and show the output on your local terminal.

Here are some of the commonly used options for the docker execcommand:

d: Detach from the container after running the command

$ docker exec -d my_container touch /tmp/newfile

i: Keep STDIN open even if not attached
t: Allocate a pseudo-TTY

u: Username or UID (format: [:])

$ docker exec -u root my_container command

e: Set environment variables

$ docker exec -e ENV_VAR=value my_container command

w: Working directory inside the container

$ docker exec -w /path/to/dir my_container command

-privileged: Give extended privileges to the command

$ docker exec --privileged my_container command

--env-file: Read in a file of environment variables.

$ docker exec --env-file=./env-vars my_container command

💡 The -t option is used to allocate a pseudo-TTY, which allows users to interact with the container as if they were sitting at a terminal. The -i option is used to keep STDIN open, which enables the user to provide input to the container. The combination of -t and -i allows users to run an interactive shell inside a container.

That concludes today's article. In the next one, we will explore multi-container applications and Docker Compose commands. If you found this article useful and interesting, please share it with your friends and colleagues, and don't forget to save it for future reference. See you in the next article! 😇

Docker : From Zero to Hero ( part 3 )

Prasenjeet Kumar — Thu, 04 May 2023 16:06:51 +0000

Docker network is a feature that enables containers to communicate with each other in a networked environment. It provides an isolated network environment for containers so that they can communicate with each other securely and efficiently.

With Docker network, we can create and manage networks for our Docker containers. These networks can be used to connect containers running on the same host or across multiple hosts, even in different data centers.

We can also create different types of networks depending on our needs. For example, a bridge network provides communication between containers on the same host, while an overlay network provides communication between containers across multiple hosts.

Overall, Docker network makes it easier to manage container communication and provides more flexibility and security in container networking.

docker networkis a command used to manage Docker networks. Docker provides a way to create virtual networks for containers to communicate with each other, as well as with the host machine or other networks.

Some of the common commands used with docker network are:

docker network create: Creates a new Docker network.
docker network ls: Lists the available Docker networks.
docker network inspect: Inspects the details of a Docker network.
docker network connect: Connects a container to a Docker network.
docker network disconnect: Disconnects a container from a Docker network.
docker network rm: Removes a Docker network.

Docker networks are useful for separating containers from each other and allowing them to communicate securely. They can be used to create multi-container applications and microservices that can be scaled and managed easily.

`docker network create`

The docker network create command is used to create a new network in Docker. This allows containers to communicate with each other even if they are running on different Docker hosts.

The basic syntax of the command is:

docker network create [OPTIONS] NETWORK_NAME

Here are some of the most commonly used options:

-driver specifies the driver that should be used for the network. The default is bridge, but other options include overlay and macvlan.
-subnet specifies the subnet that should be used for the network. This is only used for networks that use the bridge driver.
-gateway specifies the default gateway for the network. This is only used for networks that use the bridge driver.
-ip-range specifies the range of IP addresses that can be assigned to containers on the network. This is only used for networks that use the bridge driver.

Here is an example of creating a new network called my_network using the bridge driver:

docker network create --driver bridge my_network

This creates a new bridge network named my_networkusing the default subnet and gateway settings.

A bridge network is a type of network that connects multiple containers together. It is the default network type for Docker containers.

When a container is connected to a bridge network, it can communicate with other containers on the same network using their IP addresses. The bridge network provides a simple and efficient way for containers to communicate with each other without exposing ports to the host system.

`docker network ls`

The docker network ls command lists all the available networks on the Docker host.

Here's an example:

$ docker network ls
NETWORK ID     NAME                   DRIVER    SCOPE
3614fc1289ac   bridge                 bridge    local
799e5a8e5f53   host                   host      local
dcbdf4e4a4af   none                   null      local

The output lists the NETWORK ID, NAME, DRIVER, and SCOPEof each network. The bridge, host, and nonenetworks are the default networks that are created automatically when Docker is installed.

`docker network inspect`

The docker network inspect command is used to display detailed information about a specific Docker network or multiple networks. It provides a JSON format output containing the configuration details of the specified network(s).

The basic syntax of the docker network inspect command is as follows:

docker network inspect [OPTIONS] NETWORK [NETWORK...]

Here, OPTIONS are the additional flags that can be used with the docker network inspect command and NETWORK is the name or ID of the network to inspect.

Some commonly used options with the docker network inspect command are:

-format: This option is used to format the output using a Go template. The default value is {{json .}}.
-verbose or v: This option is used to enable verbose output.

Here is an example command that inspects the bridgenetwork and displays the output in a pretty format:

docker network inspect --format '{{json .Containers}}' bridge | python -m json.tool

This command inspects the bridge network and displays the containers connected to it in a pretty format.

Note that the docker network inspect command can also be used to inspect multiple networks by specifying their names or IDs separated by a space.

docker network inspect network1 network2

`docker network connect`

The docker network connect command is used to attach a container to an existing Docker network. This allows the container to communicate with other containers and services that are connected to the same network.

The basic syntax of the command is:

docker network connect <network-name> <container-name-or-id>

Here, <network-name>is the name of the network to which the container should be attached, and <container-name-or-id>is the name or ID of the container that should be attached to the network.

For example, let's say we have an existing network named "my-network" and a container named "my-container". We can connect the container to the network using the following command:

docker network connect my-network my-container

After running this command, the container "my-container" will be attached to the "my-network" network and will be able to communicate with other containers and services that are connected to the same network.

It is also possible to specify additional options with the docker network connectcommand . Here are some commonly used options:

-alias: Assign a network alias to the container on the specified network. This can be useful for container-to-container communication.
-ip: Assign a static IP address to the container on the specified network. This can be useful for certain network configurations that require fixed IP addresses.

Here's an example of using the docker network connectcommand with the --aliasoption:

docker network connect --alias myalias mynetwork mycontainer

This command connects the container mycontainerto the network mynetworkand assigns the alias myaliasto the container on that network. This means that other containers on the same network can refer to mycontainerusing the alias myaliasinstead of the container name or IP address.

How to display container ip address ?

To display the IP address of a running container, you can use the docker inspect command with the --format option to filter the output and display only the IP address.

Here's an example command:

docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container-name-or-id>

In this command, replace <container-name-or-id>with the name or ID of the container you want to inspect. The output will be the IP address of the container.

You can also use the docker network inspectcommand to display the IP address of all the containers connected to a specific network. Here's an example command:

docker network inspect <network-name>

Replace <network-name>with the name of the network you want to inspect. The output will include information about all the containers connected to that network, including their IP addresses.

The container IP address is assigned when the container is started and attached to a network. The IP address is dynamically allocated by the Docker daemon from the subnet range of the network. The Docker daemon uses the IPAM (IP Address Management) driver to manage the allocation of IP addresses for the containers.

If you want to assign statics ip address to the container at the time of attaching it to the network use the -ip options of docker network connect

Here's an example command:

$ docker network connect --ip 192.168.0.2 my-network my-container

This will connect the my-containercontainer to the my-networknetwork and assign it the IP address 192.168.0.2

It's important to note that the specified IP address must be within the subnet of the network and must not already be in use by another container on the same network.

`docker network disconnect`

The docker network disconnectcommand is used to disconnect a container from a network. It takes two arguments: the network name or ID, and the container name or ID. The syntax is as follows:

docker network disconnect [OPTIONS] NETWORK CONTAINER

Here, NETWORKis the name or ID of the network to disconnect from, and CONTAINERis the name or ID of the container to disconnect. The available options are:

--force, -f: This option forces the container to disconnect from the network. It is useful when the container is stuck in the "disconnecting" state.

Here is an example command to disconnect a container named "my_container" from a network named "my_network":

docker network disconnect my_network my_container

When a container is disconnected from a Docker network using the docker network disconnectcommand, its IP address is released and can be used by other containers connected to the same network.

`docker network rm`

The docker network rmcommand is used to remove one or more Docker networks. The general syntax of the command is:

docker network rm NETWORK [NETWORK...]

where NETWORK is the name or ID of the network(s) to be removed.

Here's an example:

$ docker network rm my-network

This command will remove the my-network network.

Note that you cannot remove a network that is currently being used by a container. You must disconnect the container from the network before you can remove it. You can use the docker network disconnect command to disconnect a container from a network.

Docker : From Zero to Hero 🛸 ( part 2)

Prasenjeet Kumar — Thu, 04 May 2023 07:10:13 +0000

Welcome to the second part of our Docker: From Zero to Hero series, where we will continue exploring essential Docker commands for container management. In the previous article, we covered Docker basics such as installation, images, and container creation. If you missed part one, you can find it here. In this article, we will focus on five critical Docker commands: docker ps, docker stop, docker rm, docker images, and docker rmi. With these commands, you will be able to manage running containers, stop or remove them, manage images, and clean up your Docker environment. Let's dive in and learn more about these essential commands.

`docker ps`

The docker ps command is used to list all running containers on the Docker host. The basic syntax of the docker psis as follows:

docker ps [OPTIONS]

OPTIONS: Optional flags that can be used to modify the output of the command.

Some common options include:

a: Show all containers, including stopped ones.
q: Only show container IDs.
-format: Specify a format for the output.

Example usage:

docker ps -a --format "table {{.ID}}\t{{.Names}}\t{{.Status}}"

This command will list all containers (including stopped ones), and format the output as a table with columns for the container ID, name, and status.

Here's an example output:

CONTAINER ID        NAMES               STATUS
a21a0498af56        nginx-container     Up 2 hours
be0fcf8f0c23        mysql-container     Up 5 days
e4389ac84062        redis-container     Exited (0) 3 weeks ago

In this example output, we can see the container ID, name, and status of each container on the host. The nginx-containerand mysql-containerare currently running, while the redis-container
is stopped.

`docker stop`

The docker stop command is used to stop one or more running containers gracefully. It sends a SIGTERM signal to the container process, giving it a chance to shutdown gracefully, and then after a default timeout of 10 seconds, it sends a SIGKILL signal to forcefully terminate the container.

Here's the basic syntax of the docker stop command:

docker stop [OPTIONS] CONTAINER [CONTAINER...]

Here, OPTIONS are the various flags that you can pass to customize the behavior of the command, and CONTAINER is the ID or name of the container(s) you want to stop.

Here's an example of using the docker stop command:

docker stop my-container

In this example, my-container is the name of the container that we want to stop.

You can also stop multiple containers at once by passing their names or IDs as space-separated arguments:

docker stop container1 container2 container3

Some common options that can be used with docker stop are:

t, --time: Specify a custom timeout value (in seconds) for how long to wait before forcefully stopping the container with SIGKILL. By default, the timeout is set to 10 seconds.
-time=0: Stop the container immediately without waiting for the container process to shut down gracefully. This is equivalent to sending a SIGKILL signal directly to the container process.
f, --force: Stop the container forcefully by sending a SIGKILL signal directly to the container process, without waiting for the container process to shut down gracefully.

For example:

docker stop -t 30 my-container

In this example, we're setting a custom timeout value of 30 seconds for stopping the my-container
container gracefully. If the container process does not shut down within 30 seconds, Docker will forcefully terminate it with a SIGKILL signal.

`docker rm`

The docker rm command is used to remove one or more stopped containers. It takes one or more container IDs or names as arguments and removes them. If a container is running, you need to stop it first using the docker stop command before removing it.

Here is the basic syntax:

docker rm [OPTIONS] CONTAINER [CONTAINER...]

Some of the commonly used options with the docker rm command are:

f, --force: This option forces the removal of a running container.
v, --volumes: This option removes the associated volumes as well.
l, --link: This option removes the specified link as well.

For example, to remove a container named my_container, you can use the following command:

docker rm my_container

To remove multiple containers, you can specify their names or IDs as space-separated values:

docker rm container1 container2 container3

To remove a running container, you can use the -foption:

docker rm -f my_container

To remove the associated volumes as well, you can use the -v option:

docker rm -v my_container

When a named volume is shared with multiple containers and one container is removed using the --volumesoption, the named volume will still exist and be accessible to other containers. However, if a container is using an anonymous volume and is removed with --volumes
, the anonymous volume will be removed along with the container and any data stored in the volume will be lost. To avoid accidental data loss, use the --volumesoption with caution.

`docker images`

The docker images command is used to list all the images that are currently stored on your system. This command shows the repository name, tag, image ID, creation date, and size of each image.

Here's an example:

$ docker images

REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
ubuntu                latest              7e0aa2d69a15        3 weeks ago         72.8MB
nginx                 latest              e37c78c07e2f        3 weeks ago         133MB
mysql                 latest              8a0d9ac31343        3 weeks ago         551MB

In the output, you can see that there are three images available: Ubuntu, Nginx, and MySQL. The columns show the repository name, tag, image ID, creation date, and size of each image.

You can use this command to check which images are available on your system and to find the image ID that you need to use when running a container.

`docker rmi`

The docker rmi command is used to remove one or more Docker images from the host. The syntax of the command is as follows:

docker rmi [OPTIONS] IMAGE [IMAGE...]

Here, OPTIONSare additional options that can be used with the command, and IMAGE is the name or ID of the image(s) to remove.

For example, to remove an image with the name my-image, you can use the following command:

docker rmi my-image

If you want to remove multiple images, you can specify their names or IDs separated by a space, like this:

docker rmi my-image1 my-image2 my-image3

Some common options used with the docker rmi command include:

f or -force: This option forces the removal of the image(s) even if there are running containers using them or if they have dependent child images.
-no-prune: This option prevents Docker from removing the parent images of the specified image(s) if they are not being used by any other images or containers.

Note that when you remove an image, any containers that were created from it will still exist on the host, but they will be unusable. To remove these containers, you can use the docker rm
command with the --forceoption.

In Docker, a parent image is the image that another image is built upon using a Dockerfile. Each image can have one parent image, with the exception of the base image (e.g. Ubuntu, Alpine, etc.) which has no parent.

To find the parent image(s) of a specified image, you can use the docker history command followed by the image name or ID. The output of the command will show the image's layers, with the parent image of each layer listed in the "IMAGE" column.

Docker : From Zero to Hero 🛸 ( part 1)

Prasenjeet Kumar — Wed, 03 May 2023 06:04:50 +0000

Docker is a tool that allows developers to package, distribute and run applications as containers.It provides an efficient and consistent way to deploy applications across different environments, from development to production, without worrying about dependencies or underlying infrastructure.

A real-life example of the benefits of Docker can be seen in web application development. In the past, deploying a web application could be a cumbersome and error-prone process. Developers would have to manually install and configure the necessary software components on each server, and any differences in configuration could lead to issues in production.

With Docker, the application and its dependencies are packaged together in a container, which can be easily deployed to any environment that supports Docker. This means that developers can be confident that their application will run the same way in development, testing, and production, and that the deployment process will be faster and more reliable.

In addition, Docker allows for better resource utilization, as multiple containers can run on a single machine without interfering with each other. This can lead to cost savings and improved scalability.

Overall, Docker simplifies the process of deploying and running applications, making it a valuable tool for developers and IT professionals alike.

Why do we need `Docker` if there is `virtual machine` ?

While both Docker and virtual machines have some similarities, they serve different purposes and have different benefits.

Virtual machines emulate an entire operating system, including the kernel, and allow you to run multiple operating systems on the same physical machine. Each virtual machine requires its own set of resources, including memory, disk space, and CPU time, which can make it difficult to scale and manage large applications.

Docker, on the other hand, uses containerization technology to allow you to package and run applications in lightweight, portable containers. Unlike virtual machines, Docker containers share the same host operating system kernel, which makes them more lightweight and efficient.

Here are some reasons why you might choose to use Docker over virtual machines:

Resource efficiency: Docker containers use fewer resources than virtual machines because they share the same host operating system kernel. This means you can run more containers on the same physical machine than you could with virtual machines.
Portability: Docker containers are portable across different environments and can run on any machine that has Docker installed, regardless of the underlying infrastructure.
Fast deployment: Docker containers can be deployed and started much faster than virtual machines, which can take several minutes to start up.
Isolation: Docker containers provide a high degree of isolation between applications, which can help to prevent conflicts and security vulnerabilities.
Version control: Docker containers allow you to version control your application and its dependencies, making it easier to roll back to previous versions or test new versions without affecting the production environment.

In summary, Docker is a lightweight and efficient alternative to virtual machines that offers many benefits for modern application development and deployment.

Is `Docker` and `Container` same thing ?

Docker is a platform that allows you to build, package, and deploy applications in containers. A container is a lightweight, portable, and self-contained environment that contains everything an application needs to run, including code, libraries, and system tools.

Here are some differences between Docker and containers:

Docker is a platform: Docker is a platform that provides a set of tools and services to build, deploy, and manage containers. It includes the Docker Engine, which is responsible for running and managing containers, as well as other tools like Docker Compose, Docker Swarm, and Docker Registry.
Containers are the building blocks: Containers are the fundamental building blocks of Docker. They provide a lightweight and portable way to package and run applications in a self-contained environment.
Docker provides a standardized format: Docker provides a standardized format for containers, which makes it easy to build, share, and deploy them across different environments. This format includes everything the container needs to run, including the code, dependencies, and system tools.
Docker provides a container runtime: Docker provides a container runtime, which is responsible for creating, starting, stopping, and managing containers. It provides a layer of abstraction between the container and the host operating system, which allows containers to be run on any system that supports Docker.

In summary, Docker is a platform that provides a standardized way to build, package, and deploy applications in containers, while containers are the lightweight and portable environments that contain everything an application needs to run.

Docker Engine

The Docker engine is the core component of Docker that allows you to create, run, and manage Docker containers. It is a lightweight and efficient runtime that can be installed on various operating systems, including Windows, macOS, and Linux.

The Docker engine consists of several key components, including:

The Docker daemon: This is the background service that runs on the host system and manages the containers and images.
The Docker CLI: This is the command-line interface that allows you to interact with the Docker daemon and perform various operations such as building and running containers.
The Docker API: This is the interface that allows external tools and applications to interact with the Docker daemon.

The Docker engine provides a powerful and flexible platform for building, running, and managing containerized applications. It allows you to package your application and its dependencies into a lightweight, portable container that can run on any system with Docker installed. This makes it easy to deploy and scale applications across different environments and platforms.

Docker daemon

The Docker daemon is the core component of the Docker engine that runs as a background service on the host system. It is responsible for managing Docker containers, images, networks, and volumes.

The Docker daemon listens to requests from the Docker CLI, Docker API, and other tools, and takes the necessary actions to create, start, stop, and remove containers. It also manages the networking and storage resources required by containers, as well as the caching and sharing of images.

The Docker daemon runs as a system service and can be configured to start automatically when the host system boots up. It is designed to be lightweight and efficient, using minimal system resources to provide fast and responsive container management.

Overall, the Docker daemon plays a critical role in the Docker architecture, enabling the creation, deployment, and management of containerized applications in a consistent and reliable manner.

Docker API

The Docker API is the interface that allows external tools and applications to interact with the Docker daemon. It provides a RESTful API that allows you to manage Docker containers, images, networks, and volumes programmatically, using standard HTTP requests.

The Docker API supports a wide range of operations, including creating and starting containers, managing images and networks, and accessing logs and stats for running containers. It is designed to be platform-agnostic, making it easy to integrate Docker into a wide range of tools and workflows.

The Docker API can be accessed using various programming languages and tools, including Python, Java, Node.js, and cURL. It is also supported by many popular DevOps tools such as Jenkins, Ansible, and Terraform.

Overall, the Docker API provides a powerful and flexible interface for automating Docker operations and integrating Docker into your DevOps workflows. It allows you to manage Docker resources programmatically, making it easy to create, deploy, and manage containerized applications in a consistent and scalable manner.

Docker CLI

The Docker CLI (Command Line Interface) is a command-line tool that allows you to interact with the Docker daemon and manage Docker containers, images, networks, and volumes. It provides a simple and powerful way to build, run, and manage Docker applications from the command line.

There are many Docker commands available, but here are some of the most commonly used ones:

docker run: This command is used to create and start a new container from an image.
docker build: This command is used to build a new Docker image from a Dockerfile.
docker push: This command is used to upload a Docker image to a registry.
docker pull: This command is used to download a Docker image from a registry.
docker ps: This command is used to list running containers.
docker stop: This command is used to stop a running container.
docker rm: This command is used to remove a container.
docker images: This command is used to list available Docker images.
docker rmi: This command is used to remove a Docker image.
docker network: This command is used to manage Docker networks.
docker volume: This command is used to manage Docker volumes.
docker exec: This command is used to execute a command inside a running container.
docker-compose: This command is used to manage multi-container Docker applications.

These are just a few examples of the many Docker commands available. To see a full list of available commands, you can use the docker --help command or refer to the Docker documentation. We will discuss each and every command in detail now. But for this article we will discuss only 4 commands and in the upcoming articles we explore remaining commands.

`docker run`

The docker run command is used to create and start a new Docker container from a specified image. When you run the docker run command, Docker creates a new container based on the specified image and starts it up.

Here's the basic syntax for the docker run command:

docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

OPTIONS: Specifies any additional options or flags to pass to the docker run command.
IMAGE: Specifies the Docker image to use as the basis for the new container.
COMMAND: Specifies the command to run inside the container.
ARG: Specifies any additional arguments to pass to the command.

Here is a list of some of the most commonly used options for the docker run command and their explanations:

d, --detach: Runs the container in the background (detached mode) and returns control to the terminal. This option is useful when you want to run a container as a background process.
p, --publish: Publishes a container's port(s) to the host machine. This option takes two arguments: the first specifies the port to expose inside the container, and the second specifies the port to map on the host machine. For example, docker run -p 8080:80 maps port 80 inside the container to port 8080 on the host machine.
v, --volume: Mounts a host directory or a named volume as a data volume inside the container. This option takes two arguments: the first specifies the path to the directory or volume on the host machine, and the second specifies the path inside the container to mount the volume. For example, docker run -v /path/on/host:/path/in/container mounts the host directory /path/on/host inside the container at /path/in/container.
-name: Assigns a custom name to the container instead of using a randomly generated one. This option is useful when you need to refer to a container by a specific name.
e, --env: Sets an environment variable inside the container. This option takes an argument in the form of VAR=value. For example, docker run -e VAR=value sets the environment variable VAR to value inside the container.
it, --interactive --tty: Runs the container in interactive mode and allocates a pseudo-TTY. This option is useful when you need to interact with the container's command line.
-rm: Removes the container automatically when it exits. This option is useful when you want to create a container for a specific task that you don't need to keep around after it completes.
-network: Specifies the network to use for the container. This option is useful when you need to connect a container to a specific network.
-restart: Specifies the restart policy for the container. This option takes an argument in the form of on-failure, always, unless-stopped, or a specific number of seconds to wait before restarting. This option is useful when you need to automatically restart a container if it exits unexpectedly.
w, --workdir: Sets the working directory inside the container. This option takes an argument in the form of a path. For example, docker run -w /app sets the working directory inside the container to /app.

How to set the multiple env variables ?

To set multiple environment variables when running a container using the docker run command, you can use multiple -e flags or pass a file containing the variables using the --env-file flag. Here are some examples:

Using multiple -e flags:

docker run -e VAR1=value1 -e VAR2=value2 myimage

Using an env file:

Create a file named env.listand add the environment variables with the key=value
format, one per line:
```
VAR1=value1
VAR2=value2
```
Then pass this file to the container using the --env-fileflag:
```
docker run --env-file env.list myimage
```

In both cases, the container will have access to the environment variables VAR1and VAR2.

Volume overview

The --volume or -v option in Docker is used to mount a directory or file from the host machine into the container. This allows the container to access and modify files on the host system.

To set up a volume when running a container using the docker run command, you can use the following syntax:

docker run -v /path/on/host:/path/in/container myimage

This will mount the directory located at /path/on/hoston the host machine into the /path/in/container directory in the container.

You can also use named volumes or anonymous volumes to manage your data. Here are some examples:

Using named volumes:
```
docker run -v myvolume:/path/in/container myimage
```
This creates a named volume named myvolumeand mounts it into the /path/in/container
directory in the container. You can use the same volume across multiple containers to share data.
Using anonymous volumes:
```
docker run -v /path/in/container myimage
```
This creates an anonymous volume and mounts it into the /path/in/container directory in the container. Anonymous volumes are created and managed by Docker and are not intended to be shared across containers.

You can also set options for your volumes, such as read-only mode, using the following syntax:

docker run -v /path/on/host:/path/in/container:ro myimage

`docker build`

docker build is a command in Docker that is used to build Docker images from a Dockerfile. A Dockerfile is a text file that contains a set of instructions that are used to build a Docker image. When you run the docker build command, Docker reads the instructions from the Dockerfile and builds a Docker image that you can use to run a Docker container.

Here is an example command to build a Docker image from a Dockerfile:

docker build -t myimage:latest .

In this example, we use the docker buildcommand to build a Docker image with the tag myimage:latest. The .at the end of the command specifies the build context, which is the directory containing the Dockerfile and any other files needed to build the Docker image.

The docker build command has several options that you can use to customize the build process. Here are some of the most commonly used options:

t: specifies the name and optionally a tag to assign to the built image.
f: specifies the name of the Dockerfile to use for building the image.
-build-arg: passes build-time variables to the Dockerfile.
-no-cache: forces Docker to build the image from scratch, without using any cached layers.
-pull: forces Docker to pull the latest version of the base image before building the new image.

Here is an example command that uses some of these options:

docker build -t myimage:latest -f Dockerfile.prod --build-arg ENVIRONMENT=production --no-cache --pull .

In this example, we specify a different Dockerfile (Dockerfile.prod) to use for building the image, and pass a build-time variable (ENVIRONMENT) to the Dockerfile using the --build-arg option. We also specify the --no-cacheoption to force Docker to build the image from scratch, and the --pulloption to ensure that we are using the latest version of the base image.

Docker Image , Dockerfile and Docker Container

Think of a Docker image as a recipe for a dish. Just as a recipe lists all the ingredients and instructions for making a dish, a Docker image lists all the dependencies and configurations required for running an application. When you run a Docker image, it creates a Docker container, just as when you cook a dish using a recipe, it creates a serving of the dish.

Now, let's compare this to a Dockerfile. A Dockerfile is like a chef's notebook that lists all the steps and ingredients required for creating a recipe. In the same way, a Dockerfile lists all the steps and configurations required for creating a Docker image. Just as a chef can use their notebook to create different versions of a dish, a developer can use a Dockerfile to create different versions of a Docker image.

To summarize, a Docker image is like a recipe, a Docker container is like a serving of the dish made from the recipe, and a Dockerfile is like a chef's notebook that lists all the steps and ingredients required for creating the recipe.

Dockerfile

A Dockerfile is a text file that contains instructions for building a Docker image. The Dockerfile defines how the Docker image should be built and what should be included in it. Here are the components of a Dockerfile:

Base Image: A Dockerfile starts with a base image, which serves as the foundation for your application. The base image is the starting point for building your Docker image. You can choose from a variety of base images available on Docker Hub or create your own custom base image.
Maintainer: The MAINTAINER instruction allows you to specify the author and email of the person who created the Dockerfile. This is an optional field but it's a good practice to include it.
RUN: The RUN instruction is used to execute commands during the Docker build process. For example, you can use the RUN instruction to install dependencies, configure the environment, and run tests.
COPY: The COPY instruction is used to copy files from the host machine to the Docker image. This is useful for copying your application code, configuration files, and other assets.
WORKDIR: The WORKDIR instruction sets the working directory for subsequent instructions. This is useful for ensuring that your application is running in the correct directory.
EXPOSE: The EXPOSE instruction is used to specify the port on which your application listens. This does not actually publish the port, it only documents it. To publish the port, you need to use the -p flag when running the container.
CMD: The CMD instruction specifies the default command that should be run when the container starts. This is usually the command that starts your application.
ENV: The ENV instruction is used to set environment variables in the Docker image. This is useful for configuring your application.

# Base image
FROM node:14-alpine

# Set working directory
WORKDIR /app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy source code to the working directory
COPY . .

# Expose port 3000
EXPOSE 3000

# Define a named volume to persist data
VOLUME /app/data

# Set environment variables
ENV NODE_ENV=production
ENV PORT=3000

# Start the application
CMD ["npm", "start"]

This Dockerfile uses the official Node.js 14 Alpine base image and sets the working directory to /app. It then copies the package.json and package-lock.json files to the working directory and installs the dependencies using npm install.

Next, it copies the source code to the working directory and exposes port 3000 using the EXPOSE command. It defines a named volume called data that will be used to persist data even if the container is stopped and removed.

The ENV command sets two environment variables, NODE_ENV and PORT, and the CMD command specifies that the application should be started using npm start.

With this Dockerfile, you can build a Docker image that includes your Node.js application and all its dependencies. You can then run the container using the docker run command and mount the named volume to persist data, like this:

docker run -d -p 3000:3000 --name my-app -v data:/app/data my-image

This will start the container, map port 3000 on the container to port 3000 on the host machine, mount the named volume datato the /app/datadirectory inside the container, and name the container my-app.

Multiple env ? i am confused 😕

The --envoption in docker run and the ENV instruction in the Dockerfile serve the same purpose, which is to set environment variables for the container.

However, there are a few differences between them:

-env is used at runtime, while ENV is used at build time.
When using -env, you can specify the environment variables as key-value pairs in the command-line, while in ENV, you specify them as environment variable assignments in the Dockerfile.
The values set with -env can override the values set in the Dockerfile with ENV. This means that if the same environment variable is set with both methods, the value set with -env takes precedence.

For example, let's say we have the following Dockerfile:

FROM alpine:latest

ENV MY_VAR=value

CMD ["sh"]

If we build the image with docker build -t my-image ., the MY_VAR environment variable will be set to value in the image.

However, if we run a container from the image with docker run --env MY_VAR=new-value my-image, the MY_VAR environment variable will be set to new-value instead of value.

In summary, both --env and ENV are used to set environment variables in Docker containers, but --env is used at runtime and can override values set in the Dockerfile with ENV.

`--build-arg`

-build-arg is a Docker CLI option that allows you to pass build-time variables to a Dockerfile. These variables can be used in the Dockerfile to customize the build process.

When you use --build-arg option, you can specify the value of the argument by passing it on the command line, like this:

docker build --build-arg VARIABLE_NAME=VALUE .

In the Dockerfile, you can reference the build-time variable using the syntax ${VARIABLE_NAME}
. For example:

FROM alpine
ARG VARIABLE_NAME
ENV ENV_VARIABLE ${VARIABLE_NAME}

In this example, we are using the ARG instruction to define the VARIABLE_NAME build-time variable. We are then using the ENV instruction to define an environment variable called ENV_VARIABLE and setting its value to the value of the VARIABLE_NAME build-time variable.

Using --build-arg is useful when you want to pass dynamic information to the Docker build process, such as version numbers, build paths, or other configuration options. By using build-time variables, you can make your Docker builds more flexible and reusable.

`docker push`

docker push is a command used to push a Docker image from a local machine to a Docker registry. A Docker registry is a service that stores and distributes Docker images. Docker Hub is a public Docker registry that allows users to store and share Docker images.

To push an image to a registry, you must first tag it with the registry name and repository name. For example, if you want to push an image with the tag myimage to the Docker Hub repository named myrepo, you would run the following command:

docker tag myimage myusername/myrepo:myimage

This command tags the image with the registry name myusername, the repository name myrepo, and the tag myimage.

Once you have tagged the image, you can push it to the Docker Hub repository using the docker push command:

docker push myusername/myrepo:myimage

This command uploads the image with the tag myimage to the Docker Hub repository myrepo under the username myusername.

Note that you must be logged in to the Docker registry before you can push an image. You can use the docker login command to log in to a registry using your Docker Hub credentials.

`docker tag`

The docker tagcommand is used to create a new tag for an existing Docker image.

The basic syntax of the docker tag command is as follows:

docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

Here, SOURCE_IMAGEis the name of the source Docker image, and TARGET_IMAGEis the name of the new image you want to create. You can optionally specify a tag for both the source and target images.

For example, let's say you have an image named myimagewith the tag latest. To create a new tag for this image, you can run the following command:

docker tag myimage:latest myrepo/myimage:v1.0

This command will create a new tag named v1.0for the myimageimage, and will associate it with the myrepo repository on your Docker registry.

You can also use the docker tagcommand to rename an image. For example, if you want to rename the myimage image to mynewimage, you can run the following command:

docker tag myimage mynewimage

This will create a new image named mynewimage that is identical to the myimage image.

In summary, the docker tag command is a useful tool for managing your Docker images and providing additional names or versions for your images.

`docker pull`

The docker pull command is used to download Docker images from a registry. It allows you to fetch a specific version of an image or the latest version of the image. The basic syntax of the docker pullcommand is as follows:

docker pull [OPTIONS] NAME[:TAG|@DIGEST]

Here, OPTIONSare any additional options that you want to pass to the docker pullcommand. NAMEis the name of the Docker image that you want to download. You can also specify the image tag using the :TAGsyntax or the image digest using the @DIGESTsyntax.

For example, to download the latest version of the nginximage from Docker Hub, you can use the following command:

docker pull nginx

To download a specific version of the nginximage, you can specify the image tag as follows:

docker pull nginx:1.20.1

This will download version 1.20.1 of the nginximage.

`OPTIONS`

Here are some common options for docker pull command:

-all-tags or a: By default, docker pull pulls only the latest image. This option pulls all the available tags for the specified image repository.
-disable-content-trust: By default, Docker verifies the authenticity of the image with Notary. This option disables this check.
-platform: This option specifies the platform for the pulled image. This is useful for multi-architecture images.
-quiet or q: This option only displays the image ID of the pulled image.
-registry-mirror: This option specifies a registry mirror to use for pulling the image.
-no-trunc: By default, docker pull truncates the image ID to the first 12 characters. This option displays the full length of the image ID.

Example usage:

# Pull the latest version of an image
docker pull myimage:latest

# Pull all available tags for an image repository
docker pull --all-tags myimage

# Disable content trust verification
docker pull --disable-content-trust myimage

# Pull a specific platform for a multi-architecture image
docker pull --platform linux/amd64 myimage

# Only display the image ID
docker pull --quiet myimage

# Use a registry mirror to pull the image
docker pull --registry-mirror=https://myregistrymirror.com myimage

# Display the full length of the image ID
docker pull --no-trunc myimage

A Docker registry mirror is a server that caches and serves Docker images from a remote registry such as Docker Hub. When you pull an image, Docker first checks the local registry mirror for the image, and if it's not found, it pulls the image from the remote registry. This can improve the speed of pulling images, especially in situations where multiple users or systems need to pull the same images repeatedly. This only applies if you are using --registry-mirror option to pull the image.

Notary is an open-source project that provides a trust framework for signing and verifying container images. It helps ensure the integrity and authenticity of container images by allowing you to digitally sign them and store the signature in a trusted registry.

Lerna: Simplifying Multi-Package JavaScript Projects with Monorepo

Prasenjeet Kumar — Fri, 28 Apr 2023 03:43:55 +0000

Lerna is a popular tool used for managing multi-package JavaScript projects. It simplifies the development process, improves code sharing and collaboration, and makes it easier to maintain and manage large codebases. In this article, we'll explore the benefits of using a monorepo and how Lerna can help you manage your multi-package JavaScript projects more effectively.

Monorepo

A monorepo, short for "monolithic repository", is a software development approach where multiple projects are stored in a single repository. This means that instead of having separate repositories for each project, they are all kept together in a single repository, with version control applied at the repository level.

In a monorepo, all the projects share the same codebase, dependencies, and configuration. This allows for better code reuse, easier cross-project refactoring, and simplified dependency management. Monorepos are commonly used by large software companies such as Google, Facebook, and Microsoft, as well as by many open-source projects.

There are several reasons why one might choose a monorepo for their software development:

Code sharing: By keeping all code in a single repository, it becomes easier to share code between different projects. This is especially useful when there are common components or libraries that are used across multiple projects.
Consistency: With a monorepo, all projects share the same codebase and dependencies. This ensures consistency across projects and reduces the risk of incompatibilities.
Simplified development process: With all code in a single repository, developers can work on multiple projects simultaneously without having to switch between different repositories.
Easier maintenance: Since all code is in a single repository, it's easier to maintain and manage. This makes it simpler to update dependencies, apply security patches, and perform other maintenance tasks.
Improved collaboration: By using a monorepo, teams can collaborate more easily since all code is in one place. This can help reduce communication overhead and speed up development.

Overall, a monorepo can simplify the development process and improve code sharing and collaboration, making it a popular choice for many development teams.

Lerna is a popular tool used for managing multi-package JavaScript projects. Some of the most used Lerna commands are:

lerna init: Initializes a new Lerna repository in the current directory.
lerna create: Creates a new package in the Lerna repository.
lerna bootstrap: Installs package dependencies and links local packages together.
lerna add: Adds a package as a dependency to another package in the Lerna repository.
lerna exec: Runs an arbitrary command in each package of the Lerna repository.
lerna changed: Lists all packages that have changed since the last release.
lerna diff: Shows the difference between the current working tree and the last release of a package.
lerna publish: Publishes new package versions to a package registry, such as npm.
lerna run: Runs an npm script in each package of the Lerna repository.
lerna clean: Removes the node_modules directory from all packages in the Lerna repository.

lerna init

lerna init is a command used to initialize a new Lerna repository with a basic directory structure and a default package.json file. It creates a new Git repository and adds an initial commit.

To use lerna init, you need to have Lerna installed globally on your machine. Once installed, you can navigate to the directory where you want to create the Lerna repository and run the following command:

lerna init

This will create the following directory structure:

my-lerna-repo/
  packages/
  package.json
  lerna.json

The packages directory is where you will create your individual packages. Each package should have its own directory within the packages directory. The package.json file is the main package.json file for the entire repository. The lerna.json file contains configuration options for Lerna.

After running lerna init, you can add packages to the repository using the lerna create command.

💡 To install the lerna globally on your machine use the command npm i -g lerna

Lerna Create

The lerna create command is used to create a new package in a Lerna-managed monorepo. It initializes the package with a basic directory structure and a package.json file.

Here's the basic syntax of the lerna create command:

lerna create <name> [loc]

<name>: The name of the package to be created
[loc]: Optional. The location where the package should be created. If not specified, the package will be created in the default packages directory.

For example, to create a new package called my-package in the default packages directory, you can run the following command:

lerna create my-package

This will create a new directory called packages/my-package, along with a basic package.json
file. You can then add your source code, tests, and other files to this directory as needed. Once you've added your code, you can use lerna addto add the new package as a dependency of other packages in your monorepo.

lerna bootstrap

lerna bootstrap is a command provided by Lerna that enables you to set up a development environment where packages depend on each other within a Lerna monorepo.

When you run lerna bootstrap, Lerna installs all external dependencies of each package, hoisting dependencies to the root level where possible. It also symlinks packages that are dependencies of each other.

Here's an example of how to use lerna bootstrap in a Lerna monorepo:

Let's say you have a Lerna monorepo with two packages:

my-monorepo/
├── package-1/
│   ├── package.json
│   └── index.js
├── package-2/
│   ├── package.json
│   └── index.js
└── lerna.json

Inside lerna.json, you can define which packages should be linked and how:

{
  "packages": [
    "packages/*"
  ],
  "version": "0.0.0"
}

Then, you can run lerna bootstrapto install all external dependencies of each package and create symlinks for interdependent packages:

lerna bootstrap

This command will execute the following steps:

Install all external dependencies for each package
Symlink packages that depend on each other

After running lerna bootstrap, you should have the following directory structure:

my-monorepo/
├── node_modules/
├── package-1/
│   ├── node_modules/ (symlinked to ../node_modules)
│   ├── package.json
│   └── index.js
├── package-2/
│   ├── node_modules/ (symlinked to ../node_modules)
│   ├── package.json
│   └── index.js
└── lerna.json

Now you can develop your packages as usual and Lerna will take care of the dependencies.

💡 Hoisting is the process of moving common dependencies from packages to the root node_modules folder to reduce duplication and improve performance.

lerna add

lerna add is a command provided by Lerna that allows you to add a dependency to one or more packages in your Lerna-managed monorepo. It automates the process of manually editing the package.json files of each package to add the dependency.

Here's an example of how to use lerna add:

Suppose you have a Lerna-managed monorepo with two packages: package-a and package-b. You want to add the lodash library as a dependency to package-a:

Open your terminal and navigate to the root of your Lerna-managed monorepo.
Run the following command:
```
lerna add lodash --scope=package-a
```
This command adds the lodash library as a dependency to package-a and updates the package.json file of package-a. It also installs lodash in package-a.

If you want to add lodash as a dev dependency, you can use the --dev flag:
```
lerna add lodash --scope=package-a --dev
```
This will add lodash as a dev dependency to package-a.
If you want to add the same dependency to multiple packages at once, you can specify a comma-separated list of scopes:
```
lerna add lodash --scope=package-a,package-b
```
This will add lodash as a dependency to both package-a and package-b.
💡 A "dev dependency" is a package or library that is only required for development and testing purposes, rather than for the production version of the software.

lerna exec

lerna exec is a command provided by Lerna that allows you to execute a command in each of the packages in your Lerna repository. This can be useful for running tests or other commands that need to be executed across all packages.

The basic syntax for lerna exec is:

lerna exec [command]

This will execute the specified command in each package directory. For example, if you wanted to run the npm testcommand in each package, you could use:

lerna exec npm test

You can also use {}as a placeholder for the package directory name in the command. For example, if you wanted to create a new directory in each package directory, you could use:

lerna exec 'mkdir {}/new-directory'

This would create a new new-directory directory in each package directory.

lerna exec also supports various options for controlling the behavior of the command. Some commonly used options include:

-parallel or P: Run the command in all packages in parallel.
-stream: Stream output from each package as it completes instead of buffering it.
-scope: Only run the command in packages that match the given scope.

For example, if you wanted to run the npm install command in all packages in parallel, you could use:

lerna exec --parallel -- npm install

We can use the lerna execcommand with the --scopeflag and a wildcard expression:

lerna exec --scope 'package-*' -- npm run test

This will execute the npm run testcommand on all the packages that have a name starting with "package-". The --scopeflag allows us to specify which packages we want to run the command on.

Overall, lerna execis a powerful command that can help you automate tasks across multiple packages in your Lerna repository.

lerna changed

The lerna changed command is used to identify which packages in a Lerna-managed repository have changed since the last tagged release, in preparation for publishing new versions of those packages. It compares the current state of the repository with the latest tagged release and identifies packages that have been modified or added.

Here is an example of using lerna changed command:

Suppose you have a Lerna-managed repository with two packages named package-1 and package-2. You have made some changes to package-1 since the last tagged release, and you want to see which packages have changed in the repository. You can use the following command:

lerna changed

This will output a list of packages that have changed since the last tagged release, in this case package-1.

You can also use the --json flag to output the result as a JSON object, which can be used by other tools:

lerna changed --json

This will output a JSON object with the list of packages that have changed.

💡 lerna changedcan be used in a Continuous Integration (CI) or Continuous Deployment (CD) pipeline to trigger builds or deployments only for the packages that have changed, instead of rebuilding or redeploying everything.

lerna diff

lerna diff is a Lerna command that shows the difference between the current repository state and the last release of the package(s).

The lerna diff command outputs a list of the packages that have changed since the last release, along with the type of change that has been made. The types of changes include:

changed: the package has changed but it doesn't include changes to its dependencies
new: a new package has been added
upgraded: the package has upgraded one of its dependencies
downgraded: the package has downgraded one of its dependencies

For example, if you run lerna diff, and there have been changes made to two packages since the last release, the output might look something like this:

lerna notice cli v3.22.1
lerna info Looking for changed packages since v1.0.0
lerna info found 2 packages to publish
lerna info diffing packages
lerna info comparing to previous version v1.0.0
my-package-1             CHANGED
my-package-2             UPGRADED (from 1.0.0 to 1.1.0)

This indicates that there have been changes made to my-package-1, and an upgrade made to

my-package-2since the last release.

💡 lerna changedis used to identify which packages have changed, while lerna diff
is used to see the specific changes made to each package.

lerna publish

lerna publish is a command in Lerna that helps you publish new versions of packages to the npm registry. It automates many of the tasks associated with publishing multiple packages, including updating version numbers, creating and pushing git tags, and publishing to the npm registry.

Here are the steps involved in using lerna publish:

First, run lerna changed to see which packages have changed since the last release.
Use lerna version to bump the version number of each changed package, update the changelog, create a git tag, and commit the changes.
Finally, run lerna publish to publish the new versions of the packages to the npm registry.

During the lerna publishprocess, you will be prompted to confirm the versions you want to publish, and you will have the option to choose a custom tag for the release (such as "next" or "beta").

What is conventional-commits ?

Conventional commit messages are a standardized way of structuring commit messages that provide more context and semantic meaning.

The basic structure of a conventional commit message is:

<type>[optional scope]: <description>

[optional body]

[optional footer]

For example, a commit message for a feature implementation might look like this:

feat(login): add forgot password feature

Add a new "forgot password" link to the login page, allowing users to reset their password if they forget it.

By using conventional commits, you can generate a changelog automatically based on the commit messages, making it easier to track changes and releases.

In conventional commits, <type> represents the nature of the changes made in the commit. This can be one of the following types:

feat: A new feature.
fix: A bug fix.
docs: Documentation changes.
style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc).
refactor: A code change that neither fixes a bug nor adds a feature.
perf: A code change that improves performance.
test: Adding missing tests or correcting existing tests.
build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm).
ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs).
chore: Other changes that don't modify src or test files.
revert: Reverts a previous commit.

The [optional scope] is used to provide additional context about the changes being made, such as the affected component or module. It is enclosed in square brackets and is optional.

There is a awesome library know as commitizen that will help you generate the conventional commit. Please do check that out !

lerna publish supports many other options as well, such as publishing only to a specific registry, skipping certain packages, and specifying a custom git commit message.

Here are all the options available for lerna publish:

-canary: creates a canary release, which means that it will publish a release with a version suffix such as alpha.X. This is useful when you want to test a release before publishing it to the public registry.
-conventional-commits: uses the conventional commits specification to determine the new version number based on the commit messages since the last release.
-force-publish: publishes all packages, even if they haven't changed since the last release. You can specify which packages to force publish by providing a comma-separated list of package names.
-dist-tag: the tag that will be used to identify the package when installing it. The default value is "latest".
-exact: forces the package version to be published exactly as it appears in the package.json file, without resolving the version to the latest semver-compatible version.
-skip-git: skips the Git commit and tag creation process, meaning that no Git operations will be performed during the release.
-skip-npm: skips the actual npm publish step, meaning that no packages will be published to the npm registry.
-preid: specifies the pre-release identifier for the version. For example, if you specify -preid alpha, the version will be something like 1.0.0-alpha.0.
-registry: the URL of the npm registry to use for publishing the packages. By default, it uses the public npm registry at https://registry.npmjs.org/.
-message: the message to use when creating the Git tag for the release. By default, it uses the release type and version number.

Here's an example that uses several of these options:

These options can be used in combination with each other to achieve the desired result. For example, you can use --skip-git --skip-npm --skip-changelog to quickly generate a new version number without committing any changes or publishing to npm.

lerna publish --canary --conventional-commits --skip-git --skip-npm --registry https://my-registry.com --dist-tag next

This command will create a canary release with a version suffix, using the conventional commits specification to determine the new version number. It will skip the Git commit and tag creation process, as well as the actual npm publish step. It will publish the package to a private registry at https://my-registry.com, and use the "next" dist-tag to identify the package when installing it.

Package release lifecycle ?

The package release lifecycle refers to the various stages that a software package goes through from its initial development to its final release. The lifecycle can be broken down into several phases, each of which serves a specific purpose. The phases typically include:

Development: This is the phase where the software package is being actively developed. During this phase, developers write code, test it, and make changes as needed.
Alpha testing: Once the initial development is complete, the package may be released to a limited group of testers for alpha testing. This testing is designed to catch any major bugs or issues before the package is released to a wider audience.
Beta testing: After alpha testing is complete, the package may be released to a larger group of beta testers. Beta testing is designed to catch any remaining bugs and to gather feedback from users.
Release candidate: Once beta testing is complete and all major bugs have been fixed, the package may be designated as a release candidate. This version is considered to be stable and is intended to be the final release unless any critical issues are found.
Final release: After the release candidate has been thoroughly tested and any remaining issues have been fixed, the package is released to the public as a final, stable version.

Throughout the release lifecycle, developers may also release patches or minor updates to the software package to address any bugs or issues that are found. These updates typically follow the same lifecycle process as the initial release, with alpha and beta testing before the final release.

Here are the steps to use Lerna to publish in different release scenarios:

Alpha release: If you want to publish an alpha release, you can use the --dist-tag
flag to specify a alphatag for your package. This will let users install your package with the @your-package-name@alphatag.
```
lerna publish --dist-tag alpha
```
Beta release: Similarly, if you want to publish a beta release, you can use the --dist-tag
flag to specify a betatag for your package. This will let users install your package with the @your-package-name@betatag.
```
lerna publish --dist-tag beta
```
Release candidate (RC) release: To publish an RC release, you can use the --dist-tagflag to specify a rctag for your package. This will let users install your package with the @your-package-name@rctag.
```
lerna publish --dist-tag rc
```
Production (stable) release: Finally, when you are ready to publish a stable release of your package, you can simply run lerna publishwithout any flags. This will publish the latest version of your package to the default latesttag. This will let users install your package with the @your-package-name@latesttag or @your-package-name.
```
lerna publish
```
💡 If you also want to suffix your version number with release tag then use the flag **`-preid`**

lerna run

lerna run is a command provided by Lerna that allows running npm scripts across packages in the monorepo. It enables running scripts in a specific package, running scripts for all packages, or running scripts in a subset of packages that match a specific glob pattern.

The basic syntax of lerna run command is:

lerna run <script> [--flags] [packageGlob...]

where <script> is the name of the npm script to run, --flags are any flags passed to the npm command, and packageGlob... is an optional list of glob patterns that filter the packages where the script should be executed.

Here's an example of using lerna run to run the build script in all packages:

lerna run build

This command will run the buildscript in all packages in the monorepo. You can also pass additional flags to the npm command, for example:

lerna run build -- --prod

This command will pass the --prod flag to the npm command, which could be used by the build script to create a production build.

You can also use glob patterns to run the script only in certain packages, for example:

lerna run build --scope my-package -- --prod

This will run the buildscript for only the my-packagepackage and pass the --prodflag to the script.

In addition to the --scope flag, lerna run supports other options such as --parallel to run scripts in parallel, --stream to stream output from multiple processes, and --no-bail to continue running scripts even if one fails.

Here's an example of running the test script in all packages in parallel:

lerna run test --parallel

This command will run the testscript in all packages at the same time, which can speed up the build process if the packages are independent of each other.

lerna clean

lerna clean is a command provided by Lerna that removes the node_modules directory from all packages in the Lerna monorepo. It is useful when you want to clean up the dependencies and start fresh.

The lerna clean command has the following options:

-yes: skips the confirmation prompt and cleans the packages without confirmation.
-ignore: a comma-separated list of package names that should be ignored during the cleaning process.
-nohoist: prevents Lerna from hoisting the dependencies while cleaning.

Here's an example:

Suppose you have a Lerna monorepo with two packages named package-a and package-b, and you want to clean them up.

To clean up all the packages in the monorepo, run the following command:

lerna clean

If you want to clean up only package-aand ignore package-b, run the following command:

lerna clean --ignore package-b

If you want to skip the confirmation prompt and clean up all packages in the monorepo, run the following command:

lerna clean --yes

If you want to prevent Lerna from hoisting the dependencies while cleaning, run the following command:

lerna clean --nohoist

Hoisting is a technique used by package managers like npm and Yarn to optimize the installation process by reducing the number of duplicated packages. It involves moving packages that are used by multiple dependencies to a higher level in the dependency tree, so they can be shared by all packages that need them, rather than being duplicated in each package's node_modules folder. This helps to reduce the overall size of the node_modules folder and improve installation times.

Follow me

You can follow me on Twitter, GitHub, and Instagram to stay updated on my latest projects and musings.

Let's connect and share our passion for tech and development!

DEV Community: Prasenjeet Kumar

Docker Swarn : From Newbie to Master ( part 2 )

docker service update

docker service scale

docker service rm

docker stack deploy

docker stack ls

docker stack ps

docker stack rm

docker stack services

How to scale services in Docker Swarm stack?

Docker swarn secret

How to use swarn secret ?

Docker Swarn : From Newbie to Master ( part 1 )

Why docker swarn ?

Architecture of Docker Swarm

How to create Swarn cluster ?

Some common Docker Swarm commands

docker swarm init

Here's how to reprint the join command for both worker and manager nodes.

docker swarm join

docker node ls

docker service create

How to add env variables from the file ?

docker service ls

docker service ps

Building TabNode: A Social Media App for Tech Bloggers with Appwrite

"What is Appwrite?”

Have you used Appwrite before?

Have you worked on similar products?

Why do you want to participate?

Where does Appwrite rank on your list of dream companies?

Do you have experience building large-scale applications?

Do you have experience in UI/UX?

Can you tell us your age and gender?

What are you building?

Have you decided on the name of this app?

What tech stack are you going to use?

Do you need team members?

Where can we contact you?

Will you split the prize?

Why should we join you?

Mastering Docker Compose: Essential CLI Commands ( part 6 )

docker-compose pause & docker-compose unpause

docker-compose ps

docker-compose logs

docker-compose build

docker-compose config

docker-compose exec

docker-compose pull

docker-compose push

docker-compose rm

docker-compose run

What is -service-ports

Discover the hidden gems of multi-container applications ( part 5 )

What is multi-container Docker applications ?

Where do we use multi-container applications?

Why should we use multi-container applications?

Docker Compose

docker-compose up

--detach

--build

--force-recreate

-no-recreate

-no-build

--remove-orphans

-timeout

--scale

docker-compose down

docker compose stop

docker-compose start

docker-compose restart

Docker : From Zero to Hero ( part 4 )

Wait a minute, why do we need volume?

Different type of volume

docker volume create

-driver

-label

docker volume ls

docker volume inspect

`docker service update`

`docker service scale`

`docker service rm`

`docker stack deploy`

`docker stack ls`

`docker stack ps`

`docker stack rm`

`docker stack services`

`docker swarm init`

`docker swarm join`

`docker node ls`

`docker service create`

`docker service ls`

`docker service ps`

`docker-compose pause` & `docker-compose unpause`

`docker-compose ps`

`docker-compose logs`

`docker-compose build`

`docker-compose config`

`docker-compose exec`

`docker-compose pull`

`docker-compose push`

`docker-compose rm`

`docker-compose run`

What is `-service-ports`

`docker-compose up`

`--detach`

`--build`

`--force-recreate`

`-no-recreate`

`-no-build`

`--remove-orphans`

`-timeout`

`--scale`

`docker-compose down`

`docker compose stop`

`docker-compose start`

`docker-compose restart`

`docker volume create`

`-driver`

`-label`

`docker volume ls`

`docker volume inspect`

`docker volume rm`

`docker volume prune`

One last command `docker exec`

`docker network create`

`docker network ls`

`docker network inspect`

`docker network connect`

`docker network disconnect`

`docker network rm`

`docker ps`

`docker stop`

`docker rm`

`docker images`

`docker rmi`

Why do we need `Docker` if there is `virtual machine` ?

Is `Docker` and `Container` same thing ?

`docker run`

`docker build`

`--build-arg`

`docker push`

`docker tag`

`docker pull`

`OPTIONS`