DEV Community: Hosea Mutwiri

The Ultimate Docker Guide: Core Concepts & Commands 🐳

Hosea Mutwiri — Thu, 25 Jun 2026 15:42:42 +0000

Docker Core Concepts
Virtual Machines vs. Containers
Docker Architecture
Docker Installation on Linux
Docker Command Reference
Dockerfile Authoring
Reference Documentation

1. Docker Core Concepts

What is a Container?

A container is a lightweight, isolated process running on a shared operating system kernel.
It is packaged with everything it needs to run, including code, libraries, environment variables, runtime, and configuration files.

Key Features of Containers

No Separate OS: Shares the host operating system kernel directly.
Isolated Environment: Runs as a self-contained process.
Fast Lifecycle: Quick to create, start, and tear down.
Repeatable: Behaves identically across different environments.
Portable: Works seamlessly on local laptops, VMs, or the cloud.
Scalable: Easily multiplied to handle increased traffic.

Container Engines

To build, ship, and run containers, you need a container runtime engine installed

Docker: is the most popular container engine.
Other Engines: Linux Containers (LXC), containerd, and CRI-O.

2. Virtual Machines vs. Containers

The structural divergence between Virtual Machines (VMs) and Containers defines their respective resource profiles and deployment speeds:

Feature	Virtual Machines (VMs)	Containers
Architecture	Runs via a Hypervisor.	Shares the host OS kernel.
Weight	Heavyweight	Lightweight
Scalability	Slow boot times	Near-zero startup times
Portability	Bound to hypervisor standards	Highly portable across any OS
OS Overhead	Redundant guest OS overhead	Shares the host OS kernel
Image Management	Inefficient image management	Efficient, layered image management

3. Docker Architecture

Docker uses a client-server architecture to simplify building, running, managing, and distributing applications. It is an open-source platform developed by Docker Inc.
Architectural Components

Docker Client: The primary interface (docker CLI) that enables users to interact with Docker by sending commands.
Docker Host: The physical server or Virtual Machine (VM) on which the Docker engine is installed and running.
Docker Daemon (dockerd): The heart of Docker. It listens for Docker API requests from the client and manages containers, images, networks, and volumes.
Docker Images: Read-only binary templates containing the application and its dependencies used to build containers.
Docker Containers: The encapsulated, runnable instances of a Docker image where applications actually execute.
Docker Registries: Storage repositories used to hold and share Docker images (e.g., Docker Hub).

4. Docker Installation on Linux

Official Reference: Docker Ubuntu Installation Guide

Step 1: Add Docker's Official GPG Key

sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Step 2: Add the Repository to Apt Sources

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

Step 3: Install Docker Engine

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 4: Verify Installation

# Check if the Docker system service is active
sudo systemctl status docker

# Start Docker manually if it is not running
sudo systemctl start docker

# Verify installed versions and system configurations
docker version
docker info

5. Docker Command Reference

Image Management

docker pull: Download an image from a registry.
- Syntax: docker pull [OPTIONS] IMAGE[:TAG|@DIGEST]
docker images: List locally stored images.
- Syntax: docker images [OPTIONS]
- Example: docker images -q Lists only local image IDs
docker rmi: Remove one or more local images.
- Syntax: docker rmi [OPTIONS] IMAGE [IMAGE...]
- Example: docker rmi -f myimage Forcefully removes the image
docker image prune: Remove unused, dangling images to safely clear host disk space.
docker history: Show the build history and layers of a specific image.
- Syntax: docker history IMAGE

Container Lifecycle

docker ps: List containers currently tracked by the engine.
- Syntax: docker ps [OPTIONS]
- Example: docker ps Shows active containers
- Example : docker ps -a Shows all containers, including stopped ones
docker start: Start one or more stopped containers.
- Syntax: docker start [OPTIONS] CONTAINER [CONTAINER...]
docker stop: Gracefully stop a running container.
- Syntax: docker stop [OPTIONS] CONTAINER [CONTAINER...]
- Example: docker stop -t 30 mycontainer Allows 30 seconds to stop gracefully before forcing termination
docker rm: Remove one or more stopped containers from the host filesystem.
- Syntax: docker rm [OPTIONS] CONTAINER [CONTAINER...]
- Example: docker rm -f mycontainer Forcefully removes a running container
docker inspect: Return low-level configuration and state information on Docker objects in raw JSON format.
- Syntax: docker inspect [OPTIONS] OBJECT [OBJECT...]

Executing Commands & Interactivity

docker exec: Run a new command process inside an actively running container.
- Syntax: docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
- Example: docker exec -t my-container ls /app
- Example (View File): docker exec -i mycontainer cat /etc/hostname
- Example: docker exec -it mycontainer /bin/bash
docker run: Create and start a brand new container instance from a target image.
- Example: docker run -it --rm --name cont3 ubuntu /bin/bash automatically deletes the container’s ephemeral file layer as soon as it exits, keeping your host system clean.

System Utilities

docker --help: Displays help documentation, global options, and sub-commands for the Docker CLI.

6. Dockerfile Authoring

What is a Dockerfile?

A Dockerfile is a text document containing all the sequential commands a user could call on the command line to assemble a Docker image.

Essential Dockerfile Instructions

Instruction	Function
`FROM`:	Sets the base image for subsequent instructions.
`RUN`:	Executes commands in a new image layer and commits the results.
`CMD`:	Specifies the default command to execute when the container starts.
`COPY`:	Copies local files and directories from the host context into the filesystem of the image.
`WORKDIR`:	Sets the working directory for any subsequent instructions.
`ENV`:	Defines environment variables inside the container environment.
`LABEL`:	Adds metadata key-value pairs to organize your images.

Image Layers & Building

Each instruction in a Dockerfile maps directly to a read only Docker image layer.

docker build: Reads the Dockerfile instructions and builds the image layers.
- Syntax: docker build [OPTIONS] PATH | URL | -
- Example: docker build -t myapp . Builds an image named myapp using the Dockerfile in the current directory

Docker Image Layers Best Practice Tips

Group Wisely: Combine commands (like apt update && apt install) to optimize layer caching performance.
Order Strategically: Place less frequently changed instructions (such as dependency installation) early in the file.
Minimize Size: Avoid unnecessary RUN commands to reduce layer count and images size

7. Reference

Official Documentation: Learn Docker Documentation

Official Reference: Docker Ubuntu Installation Guide

The Ultimate Docker Guide: Core Concepts & Commands 🐳

Hosea Mutwiri — Thu, 25 Jun 2026 15:42:41 +0000

Docker Core Concepts
Virtual Machines vs. Containers
Docker Architecture
Docker Installation on Linux
Docker Command Reference
Dockerfile Authoring
Reference Documentation

1. Docker Core Concepts

What is a Container?

A container is a lightweight, isolated process running on a shared operating system kernel.

It is packaged with everything it needs to run, including code, libraries, environment variables, runtime, and configuration files.

Key Features of Containers

No Separate OS: Shares the host operating system kernel directly.
Isolated Environment: Runs as a self-contained process.
Fast Lifecycle: Quick to create, start, and tear down.
Repeatable: Behaves identically across different environments.
Portable: Works seamlessly on local laptops, VMs, or the cloud.
Scalable: Easily multiplied to handle increased traffic.

Container Engines

To build, ship, and run containers, you need a container runtime engine installed

Docker: is the most popular container engine.
Other Engines: Linux Containers (LXC), containerd, and CRI-O.

2. Virtual Machines vs. Containers

The structural divergence between Virtual Machines (VMs) and Containers defines their respective resource profiles and deployment speeds:

Feature	Virtual Machines (VMs)	Containers
Architecture	Runs via a Hypervisor.	Shares the host OS kernel.
Weight	Heavyweight	Lightweight
Scalability	Slow boot times	Near-zero startup times
Portability	Bound to hypervisor standards	Highly portable across any OS
OS Overhead	Redundant guest OS overhead	Shares the host OS kernel
Image Management	Inefficient image management	Efficient, layered image management

3. Docker Architecture

Docker uses a client-server architecture to simplify building, running, managing, and distributing applications. It is an open-source platform developed by Docker Inc.
Architectural Components

Docker Client: The primary interface (docker CLI) that enables users to interact with Docker by sending commands.
Docker Host: The physical server or Virtual Machine (VM) on which the Docker engine is installed and running.
Docker Daemon (dockerd): The heart of Docker. It listens for Docker API requests from the client and manages containers, images, networks, and volumes.
Docker Images: Read-only binary templates containing the application and its dependencies used to build containers.
Docker Containers: The encapsulated, runnable instances of a Docker image where applications actually execute.
Docker Registries: Storage repositories used to hold and share Docker images (e.g., Docker Hub).

4. Docker Installation on Linux

Official Reference: Docker Ubuntu Installation Guide

Step 1: Add Docker's Official GPG Key

sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Step 2: Add the Repository to Apt Sources

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

Step 3: Install Docker Engine

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 4: Verify Installation

# Check if the Docker system service is active
sudo systemctl status docker

# Start Docker manually if it is not running
sudo systemctl start docker

# Verify installed versions and system configurations
docker version
docker info

Docker Command Reference Here is your content naturally enhanced and formatted into a clean, professional Markdown section. I have standardized the syntax formatting, added brief explanations for common flags, and organized the sub-sections into a more readable layout.

5. Docker Command Reference

Image Management

docker pull: Download an image from a registry.
- Syntax: docker pull [OPTIONS] IMAGE[:TAG|@DIGEST]
docker images: List locally stored images.
- Syntax: docker images [OPTIONS]
- Example: docker images -q Lists only local image IDs
docker rmi: Remove one or more local images.
- Syntax: docker rmi [OPTIONS] IMAGE [IMAGE...]
- Example: docker rmi -f myimage Forcefully removes the image
docker image prune: Remove unused, dangling images to safely clear host disk space.
docker history: Show the build history and layers of a specific image.
- Syntax: docker history IMAGE

Container Lifecycle

docker ps: List containers currently tracked by the engine.
- Syntax: docker ps [OPTIONS]
- Example: docker ps Shows active containers
- Example : docker ps -a Shows all containers, including stopped ones
docker start: Start one or more stopped containers.
- Syntax: docker start [OPTIONS] CONTAINER [CONTAINER...]
docker stop: Gracefully stop a running container.
- Syntax: docker stop [OPTIONS] CONTAINER [CONTAINER...]
- Example: docker stop -t 30 mycontainer Allows 30 seconds to stop gracefully before forcing termination
docker rm: Remove one or more stopped containers from the host filesystem.
- Syntax: docker rm [OPTIONS] CONTAINER [CONTAINER...]
- Example: docker rm -f mycontainer Forcefully removes a running container
docker inspect: Return low-level configuration and state information on Docker objects in raw JSON format.
- Syntax: docker inspect [OPTIONS] OBJECT [OBJECT...]

Executing Commands & Interactivity

docker exec: Run a new command process inside an actively running container.
- Syntax: docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
- Example: docker exec -t my-container ls /app
- Example (View File): docker exec -i mycontainer cat /etc/hostname
- Example: docker exec -it mycontainer /bin/bash
docker run: Create and start a brand new container instance from a target image.
- Example: docker run -it --rm --name cont3 ubuntu /bin/bash automatically deletes the container’s ephemeral file layer as soon as it exits, keeping your host system clean.

System Utilities

docker --help: Displays help documentation, global options, and sub-commands for the Docker CLI.

6. Dockerfile Authoring

What is a Dockerfile?

A Dockerfile is a text document containing all the sequential commands a user could call on the command line to assemble a Docker image.

Essential Dockerfile Instructions

Instruction	Function
`FROM`:	Sets the base image for subsequent instructions.
`RUN`:	Executes commands in a new image layer and commits the results.
`CMD`:	Specifies the default command to execute when the container starts.
`COPY`:	Copies local files and directories from the host context into the filesystem of the image.
`WORKDIR`:	Sets the working directory for any subsequent instructions.
`ENV`:	Defines environment variables inside the container environment.
`LABEL`:	Adds metadata key-value pairs to organize your images.

Image Layers & Building

Each instruction in a Dockerfile maps directly to a read only Docker image layer.

docker build: Reads the Dockerfile instructions and builds the image layers.
- Syntax: docker build [OPTIONS] PATH | URL | -
- Example: docker build -t myapp . Builds an image named myapp using the Dockerfile in the current directory

Docker Image Layers Best Practice Tips

Group Wisely: Combine commands (like apt update && apt install) to optimize layer caching performance.
Order Strategically: Place less frequently changed instructions (such as dependency installation) early in the file.
Minimize Size: Avoid unnecessary RUN commands to reduce layer count and images size

7. Reference

Official Documentation: Learn Docker Documentation

Official Reference: Docker Ubuntu Installation Guide

How Linux Is Used in Real-World Data Engineering

Hosea Mutwiri — Sun, 29 Mar 2026 23:44:48 +0000

If you’re just getting started in data engineering, you’ll hear one word again and again: Linux.

In this beginner friendly guide, you’ll learn:

What Linux is
A quick history of Linux
Why data engineers rely on it
The Linux commands worth learning first

What is Linux?

Linux is an operating system, like Windows or macOS. An operating system manages your machine’s hardware (CPU, memory, storage, networking) and provides the foundation that applications run on. Without an operating system, your computer or server cannot do much at all.

A complete Linux system typically includes:

Bootloader - Loads the operating system when the machine starts
Kernel - The core component that manages hardware and system resources
Init system - Starts and supervises services
Daemons - Background processes (for example, logging, scheduling, and networking)
Graphical server - Powers a desktop interface
Desktop environment and applications

Linux is free and open source, which means anyone can view, modify, and share the code. It comes in many versions called distributions (or “distros”), each tuned for different needs. If you want to explore what’s out there, DistroWatch is a good place to browse.

Brief history of Linux

In 1991, Finnish student Linus Torvalds started building Linux as a personal project while studying at the University of Helsinki. At the time, he was frustrated by the limitations of MINIX, a small Unix like teaching system, and wanted a free alternative.

Torvalds released the first version of the Linux kernel that year. In parallel, Richard Stallman and the Free Software Foundation (FSF) were developing the GNU project, which provided many of the tools and utilities people use in a Unix-like system.

Together, the Linux kernel and GNU tools formed the complete system many people refer to as GNU/Linux. What began as a hobby project grew into a global collaboration and now powers much of the world’s servers, cloud platforms and embedded devices.

Why do data engineers use Linux?

Data engineering is about building and maintaining pipelines that move data from raw sources into clean, reliable datasets for analysts and data scientists and most of that work happens on servers, and Linux dominates the server world.

Here’s why it matters so much:

Automation Data pipelines depend on scripting, scheduling, and file processing. With command line and tools like grep, awk, sed, and find, plus job scheduling with cron. You can automate ETL runs, backups and routine maintenance.
Stability and performance Linux systems are known for running reliably for long periods. That matters when pipelines need to work 24/7 and process large volumes of data without interruptions.
Security Linux system can run continuously for years without reboots hence efficiency in data processing pipelines that need to run 24/7 without interruptions.
Compatibility and ecosystem Many core data tools are built for Linux or run best on it, including Apache Spark, Kafka, Airflow, Hadoop, Docker, Kubernetes, PostgreSQL, and most cloud services (AWS, GCP, Azure).

Basic Linux commands every data engineer should learn first

Here are the most useful beginner commands, grouped by category.

Directory navigation

pwd # Print the current working directory
cd [path] # Change directory
cd .. # Go up one level
cd ~ # Go to your home directory

File and directory management

ls # List files in the current directory
ls -l # Detailed list (permissions, owner, size, date)
ls -la # Show all files, including hidden ones
ls -lh # Human readable file sizes
mkdir [name] # Create a new directory
mkdir -p [parent/child] # Create nested directories
cp [source] [destination] # Copy files or directories (use -r for recursive)
mv [source] [destination] # Move or rename files or directories
rm [file] # Remove a file (use -r for directories)
rmdir [directory] # Remove an empty directory

Viewing and searching content

cat [file] # Print the file contents (best for small files)
cat -n [file] # Print with line numbers
head [file] # Show the first 10 lines (handy for checking CSV headers)
tail [file] # Show the last 10 lines (use -f to follow logs in real time)
grep "string" [file] # Search for a string in a file
grep -i "string" [file] # Case-insensitive search
grep -n "string" [file] # Show matching line numbers
grep -c "string" [file] # Count matching lines

Other useful commands

echo "text" # Print text (useful for debugging scripts)
clear # Clear the terminal screen
find [path] -name "*.csv" # Find files by name
wc [file] # Count lines, words, and characters

Conclusion

As someone who has just started learning data engineering (currently in week 3), it’s already clear how central Linux is to real world data work.
This article was my attempt to explain the basics: what Linux is, its history, why it matters, and the essential commands every beginner should learn.
I still have a long way to go, but practicing these commands has already boosted my confidence.
Thanks for reading! I would like to hear your tips for beginners in the comments.

DEV Community: Hosea Mutwiri

The Ultimate Docker Guide: Core Concepts & Commands 🐳

Table of Contents

1. Docker Core Concepts

2. Virtual Machines vs. Containers

3. Docker Architecture

4. Docker Installation on Linux

Step 1: Add Docker's Official GPG Key

Step 2: Add the Repository to Apt Sources

Step 3: Install Docker Engine

Step 4: Verify Installation

5. Docker Command Reference

6. Dockerfile Authoring

What is a Dockerfile?

Essential Dockerfile Instructions

Image Layers & Building

Docker Image Layers Best Practice Tips

7. Reference

The Ultimate Docker Guide: Core Concepts & Commands 🐳

Table of Contents

1. Docker Core Concepts

2. Virtual Machines vs. Containers

3. Docker Architecture

4. Docker Installation on Linux

Step 1: Add Docker's Official GPG Key

Step 2: Add the Repository to Apt Sources

Step 3: Install Docker Engine

Step 4: Verify Installation

5. Docker Command Reference

6. Dockerfile Authoring

What is a Dockerfile?

Essential Dockerfile Instructions

Image Layers & Building

Docker Image Layers Best Practice Tips

7. Reference

How Linux Is Used in Real-World Data Engineering

What is Linux?

Brief history of Linux

Why do data engineers use Linux?

Basic Linux commands every data engineer should learn first

Conclusion