DEV Community: Dhananjay Patel

[Boost]

Dhananjay Patel — Thu, 20 Aug 2026 18:40:36 +0000

Dhananjay Patel

Aug 20

Understanding the Linux Filesystem Hierarchy

#linux #devops #tutorial #programming

8 min read

Understanding the Linux Filesystem Hierarchy

Dhananjay Patel — Thu, 20 Aug 2026 18:36:06 +0000

If you have just installed Linux, booted up your machine, opened the terminal, and typed ls /, you were probably met with a wall of cryptic two- and three-letter folder names: bin, etc, usr, var, opt.

If you are coming from Windows, your first question is likely: "Where is my C: drive? And what are all these strange folders?"

Don't worry—you are not alone. Many computer science students and engineers navigate Linux for years without truly understanding why these directories are named this way or how they work under the hood.

In this blog, we will break down the Linux Filesystem Hierarchy Standard (FHS), clear up common myths, learn how to navigate the directory tree, and understand the design ideas behind it. The FHS is a convention, not a rigid law: distributions can add directories, merge old paths, or package applications in different ways. The examples below describe a typical modern Linux installation.

1. The Forest of Slashes: Windows vs. Linux

One of the biggest differences between Windows and Linux is how they organize the filesystem namespace.

Windows uses a drive-based system. Your operating system lives on C:. If you plug in a USB drive, it gets assigned a new letter like D: or E:.
Linux uses a single unified tree structure. There are no drive letters. Instead, everything starts from a single point called the root directory, represented by a single forward slash (/).

The Linux filesystem is one unified directory tree rooted at /.

Think of the root / as the trunk of a massive tree. Every accessible file is reached somewhere below it. Separate filesystems—such as another disk, a USB drive, or a network share—can be attached at directories in that tree using mounts. After mounting, the contents of that filesystem appear at the chosen directory (the mount point), so the application does not need a separate drive letter. If the mount point already contained files, they are temporarily hidden until the filesystem is unmounted.

Windows exposes separate drive letters; Linux attaches filesystems into one namespace through mount points.

2. The Golden Rule: "Everything Is a File"

If you remember only one thing from this blog, make it this design principle: In Linux, almost everything is exposed through file-like interfaces.

Rather than giving every part of the system a completely different interface, Linux lets programs use familiar paths and operations for many tasks:

Regular files: Your code, PDFs, images, and text documents.
Directories: Special filesystem objects that map names to other filesystem objects.
Devices: Many hardware and pseudo-devices are exposed through device files, such as storage devices under /dev and input devices like /dev/input/event0.
Processes: Information about running programs is exposed through virtual files under /proc.

This is a powerful convention, not a literal rule: sockets, pipes, and system calls are also important interfaces, and not every resource behaves like an ordinary file.

Why is this useful?

It often lets you use familiar command-line tools such as cat, grep, and echo to inspect or interact with system interfaces as well as ordinary files. The operations supported by a particular interface still matter—writing to a device or kernel interface is not the same as editing a text file.

Want to see your computer's CPU specifications? Just read a virtual file: cat /proc/cpuinfo.
Want to see your available RAM? Just check: cat /proc/meminfo.

Important: /proc and /sys are interfaces, not ordinary folders full of permanent files. Some entries are generated when you read them, and many are only meaningful on a running Linux kernel.

3. Grouping Directories by Purpose

Rather than memorizing folders alphabetically, it is much easier to group them by what they do:

📂 User Workspaces

/home (The Workspace): This is where regular users store their personal files, projects, and downloads. If your username is Alice, your personal folder is /home/alice. Think of it as your desktop and documents folder.
/root (The Administrator's Home): This is the private home directory of the system administrator (the root user). Crucial distinction: / is the starting point of the entire filesystem, while /root is just one specific folder inside it. A regular user may have administrative power through sudo without using /root as their home directory.

📂 The Toolbox & Applications

/bin & /sbin (Essential Tools): Traditionally, /bin held essential commands usable by ordinary users (ls, cat, cp, rm), while /sbin held essential system-administration commands (ip, mount, fsck, and sometimes shutdown). The s means system or superuser, but it does not mean that every system-related program must be in /sbin, nor that only root can execute its files. On many modern distributions both are compatibility links into /usr/bin and /usr/sbin as part of a merged-/usr layout.
/usr (Most installed software): Despite its historical name, /usr is not the same as a user's home directory. It commonly holds the bulk of installed programs, shared libraries, documentation, and shared data—for example, under /usr/bin, /usr/lib, and /usr/share. A package-installed application such as Firefox may place its executable in /usr/bin, libraries in /usr/lib, and desktop metadata in /usr/share/applications. Google Chrome's .deb package commonly keeps the application files under /opt/google/chrome while placing a launcher or symlink in /usr/bin.
/lib & /lib64 (Shared Libraries): These hold libraries needed by essential programs and, on some systems, the dynamic loader. As with /bin, they may be compatibility links to locations under /usr on a merged-/usr system.
/usr/local (Administrator-installed local software): Software installed manually by the system administrator traditionally goes here, such as /usr/local/bin and /usr/local/lib. It is intended to remain separate from files managed by the distribution's package manager.
/opt (Self-contained optional applications): Large or third-party applications that keep their files together may be installed under /opt, for example /opt/vendor-app. There is no universal “X folder” for applications: the location depends on the packaging method.
Per-user applications: A user can install programs without affecting the whole system, commonly under ~/.local/bin and ~/.local/share. Snap and Flatpak applications use their own layouts (often /snap and /var/lib/flatpak). These are conventions, not replacements for learning /usr.

A useful rule of thumb is:

Installation style	Common locations
Distribution package (`apt`, `dnf`, `pacman`)	`/usr/bin`, `/usr/lib`, `/usr/share`, with configuration in `/etc` and changing data in `/var`
Manually installed administrator software	`/usr/local/bin`, `/usr/local/lib`, or `/opt/<application>`
Per-user software	`~/.local/bin`, `~/.local/lib`, and `~/.local/share`
Snap or Flatpak	Their package-specific locations and runtime namespaces

The exact layout can vary. To find the files owned by a package, use the package manager rather than guessing—for example, dpkg -L firefox on Debian/Ubuntu or rpm -ql firefox on Fedora-based systems.

📂 The Settings & Variable Data

/etc (System Configuration): Historically, the name came from "et cetera," . Today, /etc conventionally contains host-specific, system-wide configuration, such as service, network, and account-related settings.
/var (Variable Data): Short for variable. Unlike mostly static program files, /var holds data that changes during normal operation—often logs (/var/log), caches, mail queues, spool files, and application state.

📂 Runtime, Temporary, and Kernel Interfaces

/run (Runtime State): This holds volatile state created since boot, such as PID files, sockets, and runtime service data. It is typically mounted as tmpfs, so its contents normally disappear when the system shuts down or reboots.
/tmp (Temporary Scratchpad): Applications use this for temporary workspace files. It may be disk-backed or mounted as tmpfs; cleanup timing and whether contents survive a reboot depend on the distribution and system configuration. Do not use it for data you need to keep.
/dev (Device Interfaces): This is where device files and pseudo-devices are exposed.
/proc & /sys (Kernel Interfaces): These are virtual filesystems provided by the kernel rather than ordinary files stored on disk. /proc exposes process and kernel information; /sys exposes devices, drivers, and other kernel objects.
/boot (Boot files): Contains files needed to boot Linux, such as the kernel and initramfs. Some systems use a separate filesystem mounted here.
/mnt and /media (Mount points): /mnt is a conventional temporary mount point for an administrator; /media is commonly used for removable media mounted for a desktop user.
/srv (Service data): A conventional location for data served by services, such as a website or file server.
/lost+found (Filesystem recovery): On many ext filesystems, this directory holds recovered fragments after filesystem repair. It may not exist on other filesystem types.

The major Linux directories grouped by purpose.

4. Crucial Navigational Shortcuts

To explore this tree yourself in the terminal, you only need a few simple commands:

pwd (Print Working Directory): Tells you exactly where you are in the tree.
cd (Change Directory): Moves you to another folder.
- cd ~ takes you straight back to your safe and cozy home directory.
- cd .. moves you up one level (closer to the / root).
ls: Lists the contents of the directory you are in.
ls -la: Includes hidden entries (names beginning with .), permissions, owners, sizes, and timestamps.
man <command> or <command> --help: Reads local documentation and usage options.
which <command> or command -v <command>: Shows which executable the shell would run. Use type -a <command> to also see aliases and all matching entries in PATH.

Paths, `PATH`, and permissions

A path beginning with / is absolute: /etc/hosts means the same thing no matter where you are. A path without that slash is relative to the current directory: notes/today.md. . means the current directory and .. means its parent; ~ is expanded by the shell to the current user's home directory.

When you type chrome, the shell searches the directories in the PATH environment variable, usually including /usr/local/bin, /usr/bin, and possibly ~/.local/bin. The directory where a program is stored and the permissions needed to run it are separate concerns. A binary in /usr/bin is not automatically “for root,” and a binary in /usr/sbin is not automatically inaccessible to regular users.

Linux also uses ownership and permission bits on filesystem objects. Inspect them with ls -l; use sudo only when an operation genuinely requires elevated privileges, and take extra care with commands that modify /etc, /usr, or /var.

Pro-Tip: Visualizing the Tree

You can install a tool called tree using your package manager (like sudo apt install tree on Ubuntu). Running the following command will print a level-1 overview of the directories visible below /:

tree -L 1 /

tree may not be installed, and some directories will produce permission warnings. A portable alternative is:

findmnt                 # show mounted filesystems and mount points
ls -ld / /home /usr /var # inspect selected directories

5. A Few Advanced Checks

The visible directory tree is a namespace assembled from multiple filesystems. These commands help connect paths to storage and programs:

df -h /home              # filesystem containing /home and its free space
du -sh ~/Downloads       # space used by one directory
findmnt /                # filesystem mounted at the root
readlink -f "$(command -v ls)"  # resolve a symlink to its target
file /usr/bin/ls         # identify a file's type

Do not assume that every directory consumes space on the root disk: /home, /boot, /var, or even /tmp may be separate mounts. Likewise, /proc, /sys, and often /run are virtual or memory-backed filesystems rather than permanent disk storage.

Summary Cheat Sheet

Directory	What Lives Here?
`/`	The root start of the filesystem tree.
`/home`	Personal folders for regular users.
`/etc`	System configurations and settings.
`/bin`	Traditionally, essential commands such as `ls` and `cat`; often a link into `/usr/bin` today.
`/sbin`	Traditionally, essential system-administration commands; often a link into `/usr/sbin` today.
`/usr`	Distribution-managed programs, libraries, and shared data.
`/usr/local`	Locally installed administrator-managed software.
`/opt`	Optional or third-party self-contained applications.
`/var`	Variable files that change, such as logs.
`/run`	Volatile runtime state, typically stored in `tmpfs`.
`/tmp`	Temporary application files; cleanup behavior is system-dependent.

The Big Picture: A Deep Dive into Linux Architecture

Dhananjay Patel — Thu, 18 Dec 2025 15:20:37 +0000

As DevOps engineers, we often get caught up in the high-level tools—Kubernetes, Docker, Jenkins. We treat the server as a black box that just "runs things." But to truly master the cloud, you have to understand the engine that powers it all: The Linux Operating System.

I've been revisiting the fundamentals of Linux architecture, specifically looking at how the pieces fit together from the hardware up. Here is "The Big Picture" of what actually happens inside your server.

The Three Layers of Reality

At its core, a Linux system is an abstraction machine. It takes complex, messy hardware and turns it into clean, usable software interfaces. It does this through three distinct layers:

Hardware (The Foundation)
The Kernel (The Manager)
User Space (The Interface)

1. Hardware: The Raw Power

This is the physical reality—the CPU, the RAM (Main Memory), the hard disks, and the network cards.

The Constraint: Hardware is dumb. A CPU only knows how to execute simple instructions. A disk only knows how to store bits. Without management, two programs would fight over the same piece of memory, crashing the system instantly.

2. The Kernel: The Boss

This is the heart of Linux. The Kernel is the only program that speaks directly to the hardware. It acts as the strict manager of the system's resources. Its primary jobs are:

Process Management: It decides which program gets to use the CPU and for how long (Time Slicing). It creates the illusion that your browser and your terminal are running at the same time, when in reality, they are rapidly switching turns.
Device Drivers: It translates generic commands ("Write file") into specific electrical signals for your specific SSD or Hard Drive.
Memory Management: It splits the RAM into private chunks. This leads us to the most important concept in OS architecture: The Split.

The Great Divide: Kernel Space vs. User Space

To prevent chaos, Linux divides the system memory into two distinct zones. This isn't just a software rule; it is enforced by the hardware (CPU) itself using Protection Rings.

Kernel Space (Ring 0)

The VIP Zone: This is the reserved memory where the Kernel executes.
Privilege: In Kernel Space, code has unrestricted access to the hardware. It can write to any address in RAM, stop the CPU, or wipe the disk.
The Risk: If code crashes here, the entire system halts (Kernel Panic). This is why you cannot just run any random script in Kernel Space.

User Space (Ring 3)

The Sandbox: This is where your applications live (Nginx, Python, Chrome, Bash).
Restricted Memory: Programs here cannot see physical RAM directly. The Kernel gives them a "Virtual Memory" address.
- Illusion: Program A thinks it has address 0x100.
- Reality: The Kernel maps that 0x100 to a safe physical spot in RAM.
Protection: If Program A tries to read Program B's memory, the CPU detects a violation and the Kernel steps in to kill Program A (Segmentation Fault). This ensures one bad app cannot crash the whole server.

3. User Space: Where We Live

This is where everything else happens. Your shell (Bash/Zsh), your web server (Nginx), your text editor (Vim), and even your graphical desktop—they all run in User Space.

The Critical Distinction: Programs in User Space cannot access hardware directly.

If ls wants to read a directory, it cannot touch the disk.
It must ask the Kernel to do it.
This request is called a System Call (syscall).

Why This Matters for DevOps

Understanding this separation explains why "sudo" exists.

User Space is restricted to protect the system.
The Kernel has "Rootly powers" to touch hardware and memory.

When you run a Docker container, you aren't creating a new machine; you are creating a new isolated area in User Space, sharing the same Kernel. This efficiency is why containers took over the world.

The Takeaway

Linux isn't just a collection of commands. It's a carefully orchestrated dance between processes demanding resources and a Kernel managing them. As I dig deeper into the system internals, the "magic" of commands like ls, cd, and ps starts to look a lot more like logic.

Next up: Diving into the directory hierarchy and the secrets of the shell.

I am documenting my journey of mastering DevOps by going deep into the internals that power the Cloud. Follow along as I break down the systems we build upon every day.

Understanding DNS: From Root Servers to resolv.conf

Dhananjay Patel — Sun, 07 Sep 2025 18:12:16 +0000

When you type google.com in your browser, it feels instant. But behind the scenes, there’s a powerful system mapping human-readable domains to machine-friendly IPs: DNS (Domain Name System).

For DevOps engineers, DNS is more than just theory—it’s at the heart of application availability, cluster networking, and troubleshooting. Let’s break down how DNS works and why it matters in real-world DevOps.

🔑 1. Root Servers – The Internet’s Directory

Every DNS query starts with the root servers.

There are 13 named root server clusters (A–M), but thanks to anycast, they exist as hundreds of distributed servers worldwide.

Here’s the step-by-step flow when you query google.com:

Browser/OS Check: Browser checks cache, OS cache, and /etc/hosts. If no record, it queries the configured resolver (e.g., 8.8.8.8).
Recursive Resolver → Root Server: Resolver doesn’t know google.com, so it asks a root server.
Root Server Response: Root server says, “I don’t know google.com, but I know who manages .com. Go ask the .com TLD server.”
TLD Server: .com server responds, “I don’t know the IP, but here’s the authoritative server for google.com.”
Authoritative Server: Google’s DNS replies: “google.com = 142.250.72.14.”
Final Step: Resolver caches it, and your browser connects directly to that IP.

📌 DevOps Use Case:

If you deploy myapp.dev on AWS and configure it in Route53, DNS propagation follows this chain. A single misstep (e.g., wrong nameserver delegation) = app unreachable. Tools like dig or nslookup help trace where it fails.

🔑 2. Anycasting of Root Servers – Why DNS is Fast & Resilient

Root servers aren’t single machines. They use anycast routing:

Multiple servers worldwide share the same IP address.
When you query, BGP routing ensures you hit the closest available root server.
This reduces latency and increases fault tolerance.

📌 DevOps Use Case:

For global apps (deployed in Mumbai + Virginia), anycasting ensures users always hit the nearest DNS server, speeding up requests. Without it, DNS would be a massive bottleneck.

🔑 3. Port 53 – The Gateway for DNS

DNS typically uses port 53:

UDP/53 → Default for queries (fast, lightweight).
TCP/53 → Used if the response is too large (e.g., DNSSEC, zone transfers).

📌 DevOps Use Case:

If your firewall or Kubernetes NetworkPolicy blocks port 53, pods can’t resolve domains (curl google.com fails inside containers). Always check port 53 when debugging DNS issues in clusters.

🔑 4. resolv.conf – The Resolver Configuration

On Linux/macOS, /etc/resolv.conf tells your system which DNS servers to use.

Example:

nameserver 8.8.8.8      # Google DNS
nameserver 1.1.1.1      # Cloudflare DNS
search default.svc.cluster.local

The nameserver lines specify where queries go first.
The search directive is critical in Kubernetes:
- You can just run ping myservice in a pod.
- Behind the scenes, it expands to myservice.default.svc.cluster.local.

📌 DevOps Use Case:

If services in Kubernetes aren’t resolving, check /etc/resolv.conf inside pods. Misconfigured search domains break service discovery.

🔑 5. hosts File – Manual Overrides

The /etc/hosts file maps hostnames to IP addresses before DNS is queried.

Example:

127.0.0.1    localhost
192.168.1.10 staging.myapp.dev

Checked before DNS lookup.
Useful for local testing and overrides.

📌 DevOps Use Case:

Point staging.myapp.dev to a local IP for testing before DNS propagation.
Override domains during CI/CD pipeline testing.
Debug DNS by bypassing external resolvers.

🔑 6. Bonus: Other Important DNS Concepts

Recursive Resolvers → Google DNS (8.8.8.8), Cloudflare DNS (1.1.1.1), or your ISP.
Authoritative Servers → Store the final answer for a domain (managed via Route53, Cloudflare, GoDaddy, etc.).
DNS Caching → Reduces latency, but wrong TTL = stale records.
DNS Propagation → Global delay when records change (can take minutes to hours).

⚡ Real-World Scenarios Where DNS Breaks DevOps

Pods can’t resolve services → CoreDNS misconfigured in Kubernetes.
App deployed but unreachable → DNS record missing or not propagated. 👉 Coming up next in my Advanced DevOps series: “Kubernetes Networking Demystified: From Pod-to-Pod Communication to Ingress”.
SSL cert failure → Domain points to wrong IP.
Multi-region latency → Not using latency-based DNS routing (Route53, Cloudflare).
CI/CD tests failing → Use /etc/hosts override to simulate new environments.

✅ Conclusion

DNS is the hidden backbone of the internet. For DevOps engineers, understanding root servers, anycast, port 53, resolv.conf, and hosts is more than academic—it’s practical.

The next time a pod can’t reach a service or your new domain fails to resolve, you’ll know exactly where to look in the chain.

Dockerfile Anti-Patterns: What Not to Do

Dhananjay Patel — Tue, 26 Nov 2024 09:29:16 +0000

Docker is an essential tool for modern software development, offering efficiency and scalability in building, shipping, and running applications. However, writing an efficient and maintainable Dockerfile is not always straightforward. Many developers, especially beginners, unknowingly introduce anti-patterns that can lead to bloated images, slow builds, or even security vulnerabilities.

In this blog, we’ll explore some common Dockerfile anti-patterns, their consequences, and best practices to avoid them.

1. Using Large Base Images

Anti-Pattern:

Choosing a heavy, general-purpose base image, such as ubuntu:latest or debian:latest, for simple applications.

Why It’s a Problem:

Large base images increase the size of your Docker image unnecessarily, leading to longer build and deployment times.

Better Practice:

Use minimal base images, such as alpine, whenever possible. For example, instead of:

FROM ubuntu:latest

Use:

FROM alpine:latest

This drastically reduces the size of the image, often by hundreds of megabytes.

2. Failing to Leverage Multi-Stage Builds

Anti-Pattern:

Including build tools, dependencies, and artifacts in the final image.

Why It’s a Problem:

This makes the image unnecessarily large and exposes tools and files that aren’t required in production.

Better Practice:

Use multi-stage builds to separate build-time dependencies from the final runtime image. For example:

# Stage 1: Build
FROM golang:1.20 as builder
WORKDIR /app
COPY . .
RUN go build -o myapp

# Stage 2: Runtime
FROM alpine:latest
WORKDIR /app
COPY --from=builder /app/myapp .
CMD ["./myapp"]

This approach ensures that only the necessary runtime files are included in the final image.

3. Using latest Tag for Base Images

Anti-Pattern:

Pulling a base image with the latest tag.

FROM node:latest

Why It’s a Problem:

The latest tag can lead to inconsistent builds if the image updates. This unpredictability can cause issues in production.

Better Practice:

Specify a fixed version tag for consistency and reproducibility. For example:

FROM node:18

This ensures your build uses the same version every time.

4. Excessive Layering

Anti-Pattern:

Splitting every command into a separate layer.

RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y vim

Why It’s a Problem:

Each RUN instruction creates a new layer, increasing the image size and making it harder to maintain.

Better Practice:

Combine related commands into a single RUN instruction.

RUN apt-get update && apt-get install -y \
    curl \
    vim

5. Ignoring .dockerignore

Anti-Pattern:

Failing to exclude unnecessary files from the build context.

Why It’s a Problem:

If your build context contains unnecessary files, such as .git directories or large media files, the build process slows down significantly.

Better Practice:

Create a .dockerignore file to exclude unnecessary files and directories. For example:

.git
node_modules
*.log

This reduces the build context size, speeding up builds.

6. Hardcoding Secrets in Dockerfile

Anti-Pattern:

Embedding sensitive information, such as API keys or database credentials, in your Dockerfile.

ENV API_KEY=supersecretkey

Why It’s a Problem:

Secrets embedded in images can be extracted, posing a significant security risk.

Better Practice:

Use environment variables or secret management tools to handle sensitive data securely. For example, use Docker’s secrets management in swarm or Kubernetes secrets.

7. Not Cleaning Up After Installation

Anti-Pattern:

Leaving behind unnecessary files after package installation.

RUN apt-get update && apt-get install -y \
    curl \
    vim

Why It’s a Problem:

Temporary files from installations increase the size of your image.

Better Practice:

Clean up temporary files after installation.

RUN apt-get update && apt-get install -y \
    curl \
    vim && \
    rm -rf /var/lib/apt/lists/*

8. Missing a Specific CMD or ENTRYPOINT

Anti-Pattern:

Relying on the default shell behavior instead of defining a specific CMD or ENTRYPOINT.

Why It’s a Problem:

It leads to ambiguity and makes the container harder to use and debug.

Better Practice:

Specify the intended command or entry point for your application.

CMD ["python", "app.py"]

or, if you need a more robust setup:

ENTRYPOINT ["python"]
CMD ["app.py"]

Conclusion

Writing an efficient Dockerfile is both an art and a science. Avoiding these anti-patterns will result in smaller, faster, and more secure Docker images that are easier to maintain and deploy.

Take the time to review your Dockerfiles and incorporate these best practices into your workflow. With consistent improvements, you’ll ensure your Dockerized applications run smoothly and efficiently.

Docker Layer Caching Explained: Tips to Improve Build Times

Dhananjay Patel — Tue, 19 Nov 2024 10:59:30 +0000

Introduction

In our previous discussion on Docker image optimization, we focused on reducing image size to achieve faster deployments and lower storage costs. Now, let’s address another vital aspect of Docker workflows: build speed.

The time it takes to build a Docker image can significantly impact your development and deployment cycles. Fortunately, Docker offers a powerful feature called layer caching that can drastically reduce build times by reusing unchanged layers from previous builds.

In this blog, we’ll dive into how Docker layer caching works, practical tips to use it effectively, and common pitfalls to avoid.

What Is Docker Layer Caching?

Docker images are constructed layer by layer, with each instruction in the Dockerfile creating a new layer.

For example:

FROM node:16  
WORKDIR /app  
COPY package.json .  
RUN npm install  
COPY . .  
CMD ["npm", "start"]

In this Dockerfile, each instruction (FROM, WORKDIR, COPY, etc.) generates a new layer in the image. Docker saves these layers in the cache. If a subsequent build encounters an instruction that hasn’t changed, Docker reuses the cached layer instead of recreating it, speeding up the build process.

Why Is Layer Caching Important?

Faster Builds: Reusing cached layers reduces the time spent on unchanged instructions.
Improved Development Workflow: Iterative changes become quicker to test and deploy.
Cost Efficiency: Shorter build times reduce compute resource usage in CI/CD pipelines.

How Docker Layer Caching Works

Docker processes the Dockerfile sequentially:

It examines the first instruction.
If the instruction hasn’t changed since the last build, Docker uses the cached layer.
Once a layer’s cache is invalidated, all subsequent layers are rebuilt.

For instance:

If COPY package.json changes, the cache for the RUN npm install step will also be invalidated. (same example as above)
Instructions after the invalidated layer will not benefit from caching.

Best Practices for Leveraging Docker Layer Caching

Organize Your Instructions Thoughtfully

To maximize caching, place instructions that rarely change at the top of your Dockerfile.

Example:

# Better
COPY package.json package-lock.json .  
RUN npm install  
COPY . .

In this example, updates to your application code (copied in the last step) won’t invalidate the cached npm install layer.

Leverage Multi-Stage Builds

Multi-stage builds allow you to separate the build and runtime environments, reducing unnecessary layers in the final image.

Example for Node.js App:

# Build Stage
FROM node:16 AS builder  
WORKDIR /app  
COPY package.json package-lock.json ./  
RUN npm install  
COPY . .  
RUN npm run build  

# Runtime Stage
FROM nginx:alpine  
COPY --from=builder /app/build /usr/share/nginx/html  
CMD ["nginx", "-g", "daemon off;"]

This approach ensures that only the production-ready artifacts are included in the final image, significantly reducing size and build time.

Use .dockerignore to Avoid Irrelevant Files

Include a .dockerignore file to exclude unnecessary files like .git directories, logs, or node_modules that could invalidate caching.

Example .dockerignore:

node_modules  
*.log  
.git

Avoid Frequent Changes to Dependency Files

Modifications to files like package.json or requirements.txt can invalidate cache for subsequent layers. If possible, group and minimize such changes.

Combine Commands to Reduce Layers

Each instruction creates a new layer. Combining commands into a single RUN statement minimizes layer count and keeps images compact.

RUN apt-get update && apt-get install -y curl vim \  
    && apt-get clean && rm -rf /var/lib/apt/lists/*

or there is one more option, Rather than endless && \ statements this would be more readable, especially for more complex runs.

RUN <<EOF
apt-get update
apt-get install -y curl vim 
apt-get clean
rm -rf /var/lib/apt/lists/*  
EOF

Bump Cache for Layer-Sensitive Changes

When changes in dependencies (e.g., bumping package.json version) invalidate a cache, consider temporary techniques like pre-defining dependency versions to isolate changes.

Common Pitfalls to Avoid

Changing Order of Instructions

Rearranging Dockerfile instructions can invalidate the cache for no reason. Be consistent in the order.

Neglecting Cleanup

Temporary files in one layer persist unless explicitly removed in the same instruction.

Fix:

RUN apt-get update && apt-get install -y curl \  
    && rm -rf /var/lib/apt/lists/*

RUN <<EOF
apt-get update
apt-get install -y curl vim 
apt-get clean
rm -rf /var/lib/apt/lists/*  
EOF

Forgetting the Build Context

Large files in the build context can slow down COPY or ADD instructions and invalidate the cache.

Tools to Enhance Build Speed

BuildKit

Docker BuildKit offers advanced caching mechanisms and parallelism for faster builds.

In the another blog I will deep dive into Buildkit.

as of now just Enable BuildKit:

DOCKER_BUILDKIT=1 docker build .

Layer Caching in CI/CD

Many CI/CD platforms, like GitHub Actions and GitLab CI, support Docker layer caching to avoid rebuilding unchanged layers in every pipeline run.

Conclusion

Docker layer caching is a game-changer for accelerating builds and optimizing your workflows. By structuring your Dockerfile smartly, leveraging tools like BuildKit, and avoiding common pitfalls, you can drastically reduce build times and improve developer productivity.

Start experimenting with these techniques, and let me know how much faster your builds become! 🚀

Docker Image Optimization: Reducing Size for Faster Deployments

Dhananjay Patel — Sat, 16 Nov 2024 14:07:39 +0000

Introduction

A few months ago, while working on a critical deployment for a client, we faced an unexpected issue: the deployment took forever to complete. The culprit? Bloated Docker images. The process was not only frustrating but also led to downtime we couldn’t afford.

This experience taught me an important lesson: small changes can make a big impact. By optimizing Docker images, we managed to cut deployment times in half, save storage costs, and improve our CI/CD pipeline's overall efficiency. Today, I’ll share the strategies we used to achieve this transformation.

Why Optimize Docker Images?

If you've ever experienced sluggish builds, long deployment times, or a cluttered registry filled with oversized images, you’re not alone. Here’s why reducing image sizes is crucial:

Faster Builds: Your development cycles become quicker, letting you focus on what matters.
Efficient Storage: Smaller images save disk space in your Docker registries and on your machines.
Quicker Deployments: Deploying a smaller image over a network is much faster.
Enhanced Security: Fewer components mean fewer vulnerabilities.

The Day We Shrunk Our Docker Images

I remember the first time I ran docker images after our optimization efforts. Seeing the "before" and "after" sizes felt like stepping on the weighing scale after weeks of gym sessions—you notice the difference, and it feels rewarding.

Here are the exact steps we followed to make that transformation happen:

7 Effective Ways to Optimize Docker Images

1. Choose a Minimal Base Image

Instead of starting with ubuntu:latest or other large images, we switched to alpine. This one change reduced the image size from 800MB to less than 30MB.

Example:

FROM alpine:latest

2. Use Multi-Stage Builds

In many projects, such as a React application, we might have build dependencies (like Node.js and npm) that are only required during the build process but not needed in the production image. By using multi-stage builds, we can separate the build environment from the runtime environment, resulting in a much smaller image.

Example:

In this example, we’ll use a multi-stage build for a React app:

# Build Stage
FROM node:16 AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
RUN npm run build

# Runtime Stage
FROM nginx:alpine
COPY --from=builder /app/build /usr/share/nginx/html
CMD ["nginx", "-g", "daemon off;"]

In the above Dockerfile

The first stage uses the official node:16 image to install dependencies, build the React app, and generate static files.
The second stage uses the smaller nginx:alpine image to serve the built React app.

This multi-stage approach ensures that only the necessary build artifacts (the build directory) are included in the final image, keeping the image size minimal and optimized for production.

3. Remove Unnecessary Files

While debugging, we often included temporary files in our builds. By adding a .dockerignore file, we ensured these files never made it into the image.

Example .dockerignore:

node_modules
*.log
.git

4. Combine and Minimize Layers

Each instruction in a Dockerfile (e.g., RUN, COPY, ADD) creates a new layer in the Docker image. Too many layers can bloat your image size. By combining multiple instructions into a single RUN statement, you can reduce the number of layers and optimize the image.

Example:

Instead of writing:

RUN apt-get update  
RUN apt-get install -y curl vim  
RUN apt-get clean  
RUN rm -rf /var/lib/apt/lists/*

Combine them into one:

RUN apt-get update && apt-get install -y curl vim \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

This approach minimizes the number of layers and ensures temporary files (e.g., cache) are removed within the same layer, keeping the image smaller and cleaner.

5. Avoid Installing Unnecessary Dependencies

Initially, our Docker images had extra libraries "just in case." Over time, we realized that this led to bloated images and unnecessary security risks. By specifying only the dependencies that are actually needed for runtime, we kept the image smaller and more secure.

For example, instead of installing a large number of libraries for every project, we focused on minimal dependencies and avoided unnecessary packages.

6. Use `docker-slim`

A game-changer for our process was docker-slim. This tool automatically analyzes your images and reduces their size by removing unnecessary parts, such as unused files, binaries, and libraries, without affecting functionality.

We saw an image size reduction of up to 80% using docker-slim, making it an invaluable tool in our optimization strategy.

Command to slim down an image:

docker-slim build <image-name>

7. Regularly Audit and Prune Images

Docker images accumulate over time, and unused images or layers can take up valuable space. Regularly auditing and pruning unused images helps maintain a clean environment.

You can remove unused images and layers by running these commands:

Command to prune unused images:

docker system prune -f

Command to remove all unused images:

docker image prune -a -f

By incorporating regular pruning into your workflow, you ensure that your Docker environment stays lean and efficient.

Measuring Success

After implementing these optimizations, we used docker images to compare sizes. The results were stunning:

Before Optimization: 1.2GB
After Optimization: 250MB

Not only did our deployments become faster, but our cloud storage costs also went down significantly.

Conclusion

Optimizing Docker images might seem like a minor task, but the benefits it brings to your workflows are immense. Whether you’re a solo developer or part of a large team, these strategies can make a real difference.

So, what are you waiting for? Dive into your Dockerfile, start optimizing, and enjoy the perks of leaner, faster deployments.

References

📖 Love this blog? You can also read my articles on Hashnode and follow me there for more tech insights! 🚀

DEV Community: Dhananjay Patel

[Boost]

Understanding the Linux Filesystem Hierarchy

Understanding the Linux Filesystem Hierarchy

1. The Forest of Slashes: Windows vs. Linux

2. The Golden Rule: "Everything Is a File"

Why is this useful?

3. Grouping Directories by Purpose

📂 User Workspaces

📂 The Toolbox & Applications

📂 The Settings & Variable Data

📂 Runtime, Temporary, and Kernel Interfaces

4. Crucial Navigational Shortcuts

Paths, PATH, and permissions

Pro-Tip: Visualizing the Tree

5. A Few Advanced Checks

Summary Cheat Sheet

The Big Picture: A Deep Dive into Linux Architecture

The Three Layers of Reality

1. Hardware: The Raw Power

2. The Kernel: The Boss

The Great Divide: Kernel Space vs. User Space

Kernel Space (Ring 0)

User Space (Ring 3)

3. User Space: Where We Live

Why This Matters for DevOps

The Takeaway

Understanding DNS: From Root Servers to resolv.conf

🔑 1. Root Servers – The Internet’s Directory

🔑 2. Anycasting of Root Servers – Why DNS is Fast & Resilient

🔑 3. Port 53 – The Gateway for DNS

🔑 4. resolv.conf – The Resolver Configuration

🔑 5. hosts File – Manual Overrides

🔑 6. Bonus: Other Important DNS Concepts

⚡ Real-World Scenarios Where DNS Breaks DevOps

✅ Conclusion

Dockerfile Anti-Patterns: What Not to Do

1. Using Large Base Images

2. Failing to Leverage Multi-Stage Builds

3. Using latest Tag for Base Images

4. Excessive Layering

5. Ignoring .dockerignore

6. Hardcoding Secrets in Dockerfile

7. Not Cleaning Up After Installation

8. Missing a Specific CMD or ENTRYPOINT

Conclusion

Docker Layer Caching Explained: Tips to Improve Build Times

Introduction

What Is Docker Layer Caching?

Why Is Layer Caching Important?

How Docker Layer Caching Works

Best Practices for Leveraging Docker Layer Caching

Common Pitfalls to Avoid

Tools to Enhance Build Speed

Conclusion

Docker Image Optimization: Reducing Size for Faster Deployments

Introduction

Why Optimize Docker Images?

The Day We Shrunk Our Docker Images

7 Effective Ways to Optimize Docker Images

1. Choose a Minimal Base Image

2. Use Multi-Stage Builds

3. Remove Unnecessary Files

4. Combine and Minimize Layers

5. Avoid Installing Unnecessary Dependencies

6. Use docker-slim

7. Regularly Audit and Prune Images

Measuring Success

Conclusion

References

Paths, `PATH`, and permissions

6. Use `docker-slim`