DEV Community: opscanopy

Networking for DevOps Engineers: The Complete Guide

opscanopy — Fri, 18 Sep 2026 16:19:29 +0000

Networking is the invisible plumbing that every deployment depends on. When a pod can't reach its database, a deploy fails, or latency spikes with no obvious cause, the engineer who understands the network stack finds the answer while everyone else is still guessing. This guide covers the networking fundamentals every DevOps practitioner needs — from binary subnet math to Kubernetes service meshes — grounded in the commands you'll actually run. If you want a structured learning path alongside this reference, see the Networking roadmap.

Why Networking Matters for DevOps

Infrastructure work is networking work, whether you recognize it or not. Every CI/CD pipeline fetches artifacts over the wire. Every container runtime assigns IP addresses, manages virtual interfaces, and enforces iptables rules. Cloud platform mistakes — wrong CIDR range, missing security group rule, overlooked NAT gateway — routinely stall production deployments and generate hours of troubleshooting.

Concrete scenarios where networking knowledge pays off immediately:

Debugging why two services in the same VPC can't communicate (security group rule, route table missing, NACL block, wrong subnet)
Sizing a VPC subnet large enough to accommodate auto-scaling groups that expand to hundreds of pods
Investigating certificate errors, TLS handshake failures, or mixed-content warnings in HTTPS apps
Tuning load balancer health-check thresholds and connection-draining windows to achieve zero-downtime deploys
Reading tcpdump or Wireshark output to understand exactly what traffic crossed the wire during an incident

The sections below build from the ground up: models → addresses → subnets → protocols → services → tools.

The OSI and TCP/IP Models

Two reference models describe how data travels from application to wire. The OSI (Open Systems Interconnection) model has seven layers; the TCP/IP model collapses those into four. Both remain useful — OSI gives precise vocabulary for troubleshooting, TCP/IP maps to real protocol implementations.

OSI Layer	Name	TCP/IP Layer	Examples
7	Application	Application	HTTP, HTTPS, DNS, SMTP, FTP, SSH
6	Presentation	Application	TLS/SSL, MIME encoding
5	Session	Application	TLS sessions, RPC, NetBIOS
4	Transport	Transport	TCP, UDP, SCTP
3	Network	Internet	IPv4, IPv6, ICMP, IPsec
2	Data Link	Network Access	Ethernet, Wi-Fi (802.11), ARP, VLAN
1	Physical	Network Access	Cable, fiber, radio, transceivers

OSI's seven layers collapse into four TCP/IP layers; each arrow shows which OSI layers map to the corresponding TCP/IP layer.

Encapsulation moves data down the stack on the sender and up on the receiver. Each layer wraps the payload with its own header (and sometimes trailer). By the time an HTTP request leaves a NIC, the original bytes are nested inside: HTTP → TCP segment → IP packet → Ethernet frame.

Where DevOps tooling sits:

Load balancers operate at L4 (TCP/UDP) or L7 (HTTP/HTTPS)
Firewalls and security groups filter at L3/L4 (IP addresses, ports, protocol)
Service meshes (Envoy, Linkerd) intercept at L4/L7 inside the application network
Overlay networks (VXLAN, Geneve) wrap L2 frames inside L3 packets — how Kubernetes CNI plugins create virtual pod networks across physical hosts

Tip: When troubleshooting, identify which layer the failure lives in. "Can't reach the server" is vague. "The TCP SYN goes out but no SYN-ACK returns" pins the problem to L3/L4, narrowing the suspect list to routes, security groups, and firewalls.

IP Addressing

IPv4 Structure

An IPv4 address is 32 bits written as four decimal octets separated by dots: 192.168.1.100. Each octet ranges from 0–255 (2⁸ possible values). The address space is divided into a network portion (identifies the subnet) and a host portion (identifies the device within that subnet). The split point is determined by the subnet mask.

# Inspect assigned IPs on Linux
ip addr show

# Legacy command (still common on older systems)
ifconfig -a

Public vs Private Address Ranges

RFC 1918 reserves three blocks for private use — they are not routable on the public internet:

Range	CIDR	Size	Common Use
10.0.0.0 – 10.255.255.255	10.0.0.0/8	~16.7 million	Enterprise, cloud VPCs
172.16.0.0 – 172.31.255.255	172.16.0.0/12	~1 million	Mid-size networks, Docker default
192.168.0.0 – 192.168.255.255	192.168.0.0/16	~65,000	Home, small office

Additional special-purpose ranges:

127.0.0.0/8 — loopback (localhost); 127.0.0.1 is the standard loopback address
169.254.0.0/16 — link-local; assigned automatically when DHCP fails (APIPA)
100.64.0.0/10 — shared address space (ISP carrier-grade NAT, RFC 6598)
0.0.0.0/0 — the default route, meaning "everything"

A Note on IPv6

IPv6 uses 128-bit addresses written in eight groups of four hex digits, e.g. 2001:0db8:85a3:0000:0000:8a2e:0370:7334. Consecutive groups of zeroes can be collapsed with :: — the above becomes 2001:db8:85a3::8a2e:370:7334. The primary motivation was exhaustion of the IPv4 space.

Key DevOps IPv6 touchpoints: AWS dual-stack VPCs, Kubernetes Pod IPs (IPv6 mode), and modern load balancers that accept both stacks. The subnet math uses /64 prefixes for most LANs (leaving 64 bits for host IDs).

Use the IP Address Converter to convert between dotted-decimal, binary, and hex representations — indispensable when verifying subnet masks or reading raw packet captures.

Subnets and CIDR

Subnet Masks and Prefix Length

A subnet mask is a 32-bit value with all network bits set to 1 and all host bits set to 0. It tells a device which part of an IP address identifies the network.

CIDR (Classless Inter-Domain Routing) notation expresses the same information as a prefix length after a slash: 192.168.1.0/24 means the first 24 bits are the network, the remaining 8 bits are for hosts.

CIDR Prefix	Subnet Mask	# Addresses	Usable Hosts
/8	255.0.0.0	16,777,216	16,777,214
/16	255.255.0.0	65,536	65,534
/20	255.255.240.0	4,096	4,094
/24	255.255.255.0	256	254
/25	255.255.255.128	128	126
/26	255.255.255.192	64	62
/27	255.255.255.224	32	30
/28	255.255.255.240	16	14
/29	255.255.255.248	8	6
/30	255.255.255.252	4	2
/32	255.255.255.255	1	host route only

A /24 block dedicates 24 bits to the network and 8 bits to hosts, yielding 256 total addresses with 254 usable after reserving the network and broadcast addresses.

The Host Count Formula

host_bits = 32 − prefix_length
total_addresses = 2^host_bits
usable_hosts = 2^host_bits − 2

Subtract 2 because the first address is the network address (all host bits zero) and the last is the broadcast address (all host bits one). Neither can be assigned to a host.

Worked Example: Subnetting 10.0.0.0/16

You need to allocate subnets inside a VPC at 10.0.0.0/16. You want:

One public subnet per availability zone (3 AZs, ~250 hosts each)
One private subnet per AZ (~500 hosts each)
One database subnet per AZ (~50 hosts each)

Step 1 — Choose prefix lengths:

Public subnets: /24 → 254 usable hosts ✓
Private subnets: /23 → 510 usable hosts ✓
Database subnets: /26 → 62 usable hosts ✓

Step 2 — Lay out the ranges (keeping subnets non-overlapping):

Subnet	CIDR	Usable Range	Purpose
Public AZ-a	10.0.0.0/24	10.0.0.1–10.0.0.254	Public subnet, AZ us-east-1a
Public AZ-b	10.0.1.0/24	10.0.1.1–10.0.1.254	Public subnet, AZ us-east-1b
Public AZ-c	10.0.2.0/24	10.0.2.1–10.0.2.254	Public subnet, AZ us-east-1c
Private AZ-a	10.0.4.0/23	10.0.4.1–10.0.5.254	Private subnet, AZ us-east-1a
Private AZ-b	10.0.6.0/23	10.0.6.1–10.0.7.254	Private subnet, AZ us-east-1b
Private AZ-c	10.0.8.0/23	10.0.8.1–10.0.9.254	Private subnet, AZ us-east-1c
DB AZ-a	10.0.16.0/26	10.0.16.1–10.0.16.62	Database subnet, AZ us-east-1a
DB AZ-b	10.0.16.64/26	10.0.16.65–10.0.16.126	Database subnet, AZ us-east-1b
DB AZ-c	10.0.16.128/26	10.0.16.129–10.0.16.190	Database subnet, AZ us-east-1c

Note: Cloud providers reserve additional addresses inside each subnet. AWS, for example, reserves the first 4 addresses and the last 1 in every subnet, reducing usable hosts by 5.

VLSM (Variable Length Subnet Masking)

VLSM allows different subnets within the same address space to use different prefix lengths — exactly the approach used in the example above. Each subnet is sized to its actual requirement, minimizing waste. The rule is that subnets must not overlap, and each subnet's network address must be aligned to its block size (a /23 must start on an even /24 boundary, a /26 must start at a multiple of 64, etc.).

Free Subnet Tools

You should never do this math by hand under pressure. Use these tools:

Subnet Calculator — enter any IP/prefix and instantly get network address, broadcast, usable host range, and mask
CIDR Checker — verify whether an IP falls inside a given CIDR block, check overlap between ranges
Subnet Splitter — divide a parent CIDR into equal or custom-sized child subnets automatically

TCP vs UDP

Both TCP and UDP sit at Layer 4 (Transport). The choice between them is a design trade-off between reliability and speed.

TCP — Transmission Control Protocol

TCP is connection-oriented: before any data flows, a three-way handshake establishes the connection.

Client → Server   SYN       (sequence number X)
Server → Client   SYN-ACK   (sequence number Y, ack X+1)
Client → Server   ACK       (ack Y+1)

The TCP three-way handshake: client sends SYN, server replies with SYN-ACK, client confirms with ACK, then bidirectional data flow begins.

After the handshake, TCP guarantees:

Ordered delivery — segments are reassembled in sequence regardless of arrival order
Reliable delivery — lost segments are detected (via ACK timeout or duplicate ACKs) and retransmitted
Flow control — receiver advertises a window size to prevent buffer overflow
Congestion control — sender reduces rate when the network signals congestion (CUBIC, BBR, etc.)

Connection teardown uses a four-way FIN/FIN-ACK/FIN/FIN-ACK exchange, leaving connections in TIME_WAIT for 2×MSL (typically 60–120 seconds) to absorb delayed duplicates.

UDP — User Datagram Protocol

UDP is connectionless: datagrams are fired without a handshake. There is no delivery guarantee, no ordering, and no retransmission. The sender does not know whether the packet arrived.

The upside is minimal overhead and latency. For workloads that handle their own reliability, or where a slightly stale/lost packet is acceptable, UDP wins.

When to Use Each

Characteristic	TCP	UDP
Delivery guarantee	Yes	No
Ordering	Yes	No
Connection state	Yes (SYN/FIN)	No
Overhead	Higher	Lower
Typical latency	Higher	Lower
Use cases	HTTP/S, SSH, databases, file transfer	DNS, DHCP, VoIP, video streaming, gaming, QUIC/HTTP3

Note: QUIC (the underlying protocol for HTTP/3) runs over UDP but implements its own reliability, ordering, and congestion control at the application level — combining UDP's flexibility with TCP-like guarantees while eliminating head-of-line blocking.

Inspecting TCP State

# Show all listening and established TCP connections
ss -tnp

# Show UDP sockets
ss -unp

# Filter for a specific port
ss -tnp sport = :443

# Count connections by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

Ports and Sockets

What Is a Socket?

A socket is the combination of IP address and port that uniquely identifies one end of a network connection. A full TCP connection is a 4-tuple: (src IP, src port, dst IP, dst port). This allows a single server IP to handle thousands of simultaneous connections to the same port — each connection's tuple is unique.

Well-Known Ports (0–1023)

These ports are assigned by IANA and typically require root/admin privileges to bind.

Port	Protocol	Service
20/21	TCP	FTP (data/control)
22	TCP	SSH
23	TCP	Telnet (insecure, avoid)
25	TCP	SMTP
53	TCP/UDP	DNS
67/68	UDP	DHCP (server/client)
80	TCP	HTTP
110	TCP	POP3
123	UDP	NTP
143	TCP	IMAP
443	TCP	HTTPS
445	TCP	SMB
514	UDP	Syslog
587	TCP	SMTP submission (with auth)
993	TCP	IMAPS
995	TCP	POP3S

Registered and Ephemeral Ports

Registered (1024–49151): Assigned to specific services by IANA but do not require root. Common examples: 3306 (MySQL), 5432 (PostgreSQL), 6379 (Redis), 8080 (HTTP alternate), 9200 (Elasticsearch), 27017 (MongoDB).
Ephemeral (49152–65535): Assigned by the OS to the client side of outgoing connections. The range is configurable; Linux defaults to roughly 32768–60999 (check /proc/sys/net/ipv4/ip_local_port_range).

# Check the ephemeral port range on Linux
cat /proc/sys/net/ipv4/ip_local_port_range

# See what process owns a specific port
ss -tnlp | grep :8080

# Check if a port is open from inside the host
nc -zv 127.0.0.1 5432

Caution: Running out of ephemeral ports under high outbound connection rates (e.g., a service hammering a downstream API) causes connection failures. Monitor with ss -s and widen the range if needed: sysctl -w net.ipv4.ip_local_port_range="1024 65535".

DNS

The Domain Name System translates human-readable names into IP addresses. From a DevOps perspective, DNS is a source of subtle bugs, slow failovers, and cache-related incidents.

Resolution Flow

Client queries its stub resolver (the OS resolver, configured via /etc/resolv.conf or systemd-resolved)
Stub resolver forwards to a recursive resolver (e.g., 8.8.8.8, your VPC's .2 resolver)
Recursive resolver checks its cache; on a miss it begins iterative resolution:
- Queries a root nameserver (one of 13 root server IPs) → gets the TLD nameserver address
- Queries the TLD nameserver (e.g., .com, .io) → gets the authoritative nameserver address
- Queries the authoritative nameserver for the domain → gets the final record
Answer is returned to the client and cached at each layer according to the TTL

Client → Stub Resolver → Recursive Resolver → Root NS → TLD NS → Auth NS
                                      ← ← ← (answer cached + returned) ← ← ←

On a cache miss the recursive resolver walks root, TLD, and authoritative nameservers iteratively; each answer is cached by its TTL so subsequent queries resolve instantly.

DNS Record Types

Record	Purpose	Example
A	IPv4 address for a hostname	`api.example.com → 93.184.216.34`
AAAA	IPv6 address for a hostname	`api.example.com → 2606:2800::1`
CNAME	Alias; points to another hostname	`www.example.com → example.com`
MX	Mail exchange server(s), with priority	`example.com → 10 mail.example.com`
TXT	Arbitrary text; used for SPF, DKIM, domain verification	`v=spf1 include:_spf.google.com ~all`
NS	Authoritative nameservers for a zone	`example.com → ns1.route53.aws`
SOA	Start of Authority; zone metadata	Serial, refresh, retry, expire, TTL
PTR	Reverse DNS (IP → name)	`34.216.184.93.in-addr.arpa → api.example.com`
SRV	Service location (protocol, port, host)	`_http._tcp.example.com → 0 5 80 api.example.com`
CAA	Certificate Authority Authorization	Limits which CAs may issue certs for the domain

TTL and Caching

TTL (Time To Live) is set on each record and controls how long resolvers cache the answer. Low TTLs (30–60 seconds) enable fast failover; high TTLs (3600+) reduce resolver load and query latency. The classic incident pattern: TTL is 3600, a record is changed, but old IPs continue serving traffic for up to an hour because downstream caches haven't expired yet.

Tip: Before a planned IP change or failover, lower the TTL to 60–120 seconds at least 2× the current TTL in advance (so caches expire before the change window). After the change, restore the original TTL.

DNS Debugging

# Basic A record lookup
dig example.com

# Query a specific record type
dig example.com MX
dig example.com TXT

# Query a specific nameserver directly (bypass local cache)
dig @8.8.8.8 example.com A

# Trace the full resolution chain from root
dig +trace example.com

# Reverse lookup (PTR)
dig -x 93.184.216.34

# Show only the answer section, short output
dig +short example.com

# Check DNS propagation from the authoritative server
dig @ns1.example.com example.com A +norecurse

# nslookup (simpler, available everywhere)
nslookup example.com
nslookup -type=MX example.com 8.8.8.8

Use the Reverse DNS / PTR tool to look up the hostname registered for any IP — useful for verifying that mail server PTR records are set correctly and for identifying infrastructure from IPs you see in logs.

Note: /etc/hosts takes precedence over DNS on most Linux systems. During incident response, check whether a stale hosts entry is masking a real DNS record.

HTTP and HTTPS/TLS

HTTP Methods and Semantics

HTTP is the application-layer protocol powering almost every API and web interface in a DevOps toolchain.

Method	Semantics	Idempotent	Safe
GET	Retrieve a resource	Yes	Yes
HEAD	Retrieve headers only	Yes	Yes
POST	Submit data / create resource	No	No
PUT	Replace a resource entirely	Yes	No
PATCH	Partial update	No	No
DELETE	Remove a resource	Yes	No
OPTIONS	Describe allowed methods	Yes	Yes

HTTP Status Code Families

Range	Meaning	Common Examples
1xx	Informational	101 Switching Protocols (WebSocket upgrade)
2xx	Success	200 OK, 201 Created, 204 No Content
3xx	Redirection	301 Moved Permanently, 302 Found, 304 Not Modified
4xx	Client error	400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 429 Too Many Requests
5xx	Server error	500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout

Tip: In a reverse proxy setup, 502 Bad Gateway usually means the upstream (your app) returned an invalid response or crashed. 504 Gateway Timeout means the upstream was too slow. These two codes point you toward the application tier, not the proxy.

The TLS Handshake

HTTPS wraps HTTP inside TLS (Transport Layer Security). The handshake negotiates a shared symmetric key using asymmetric cryptography, then all subsequent data is encrypted with that key.

TLS 1.3 handshake (simplified):

Client → Server: ClientHello  (supported cipher suites, TLS version, random nonce, key_share)
Server → Client: ServerHello  (chosen cipher, key_share)
Server → Client: Certificate  (server's X.509 cert chain)
Server → Client: Finished     (handshake complete, encrypted)
Client → Server: Finished     (handshake complete, encrypted)
--- Application data now flows encrypted ---

TLS 1.3 reduces the handshake to 1 round-trip (1-RTT), and supports 0-RTT resumption for returning clients, significantly reducing latency versus TLS 1.2.

Certificates

An X.509 certificate binds a public key to a domain name (or wildcard like *.example.com), signed by a Certificate Authority (CA) that browsers and operating systems trust. Key fields:

Subject / CN (Common Name): The hostname the cert covers
SAN (Subject Alternative Names): Additional hostnames (modern certs use SAN; CN alone is deprecated)
Issuer: The CA that signed the certificate
Validity period: Not Before / Not After — expiry causes widespread outages
Key usage / Extended Key Usage: Restricts what the cert can be used for

# Inspect a remote certificate
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -text

# Check expiry date only
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

# Verify a cert file locally
openssl x509 -in cert.pem -noout -text

# Test TLS with curl, verbose (shows handshake)
curl -vvv https://example.com 2>&1 | grep -A5 "TLS\|SSL\|certificate"

Caution: Certificate expiry is one of the most common causes of production outages. Automate renewal (Let's Encrypt / Certbot, AWS ACM auto-renew, cert-manager in Kubernetes) and set monitoring alerts at 30 days and 7 days before expiry.

Load Balancing and Reverse Proxies

What a Load Balancer Does

A load balancer distributes incoming connections or requests across a pool of backend servers. Goals: eliminate single points of failure, allow horizontal scaling, and enable zero-downtime deploys through rolling updates and connection draining.

L4 vs L7 Load Balancing

L4 (Transport layer): Routes based on IP address and TCP/UDP port only. The load balancer forwards packets or proxies TCP connections without inspecting content. Fast and protocol-agnostic.

Examples: AWS NLB, HAProxy in TCP mode, Linux IPVS
Use for: raw TCP throughput, non-HTTP protocols (databases, gRPC over raw TCP, game servers), lowest possible latency

L7 (Application layer): Understands HTTP(S). Can inspect and route on Host header, URL path, request method, cookies, or any header. Can terminate TLS, add/remove headers, rewrite URLs, and return synthetic responses.

Examples: AWS ALB, NGINX, HAProxy in HTTP mode, Envoy, Traefik, Caddy
Use for: microservices routing, canary deployments, A/B testing, authentication offloading, WebSocket upgrades

Feature	L4	L7
Routing basis	IP + port	Host, path, headers, cookies
TLS termination	Pass-through or terminate	Terminate + re-encrypt (mTLS)
Health checks	TCP connect	HTTP GET / gRPC health
Latency overhead	Minimal	Slightly higher
Observability	Connection-level	Request-level metrics

L4 load balancers forward TCP/UDP connections by IP and port alone; L7 load balancers inspect HTTP and can route each request by host, path, headers, or cookies.

Load Balancing Algorithms

Algorithm	Behaviour	Best For
Round Robin	Distribute in order	Homogeneous backends
Least Connections	Route to backend with fewest active connections	Variable request duration
IP Hash	Route same client IP to same backend	Session affinity (sticky sessions)
Weighted Round Robin	Distribute proportionally by weight	Mixed-capacity backends
Random	Pick at random	Simple, low overhead

Health Checks

Health checks detect failed backends and remove them from rotation. Key parameters:

Path / port: What the load balancer probes (e.g., GET /healthz)
Interval: How often to probe (e.g., every 10 seconds)
Timeout: How long to wait for a response before counting a failure
Healthy threshold: Consecutive successes required to mark healthy
Unhealthy threshold: Consecutive failures required to remove from rotation

Tip: Design /healthz to return 200 only when the application is genuinely ready to serve traffic — not just that the process started. Check database connectivity, cache availability, and any critical dependency.

Common Reverse Proxy / Load Balancer Software

NGINX: High-performance HTTP server and reverse proxy; excellent for static file serving, TLS termination, upstream proxying
HAProxy: Purpose-built proxy; extremely reliable, detailed metrics, supports both L4 and L7
Envoy: Cloud-native L7 proxy; the data plane behind Istio, Consul Connect, and AWS App Mesh; first-class gRPC support
Traefik: Auto-discovers Docker/Kubernetes services and configures itself from labels/annotations
Caddy: Automatic HTTPS with built-in Let's Encrypt; simple configuration syntax

# Test if NGINX config is valid before reload
nginx -t

# Reload NGINX without dropping connections
nginx -s reload

# Check upstream health in HAProxy (stats page)
# Enable in haproxy.cfg: stats uri /haproxy-stats
curl http://localhost/haproxy-stats

Firewalls and Security Groups

Stateful vs Stateless Filtering

Stateless firewalls (e.g., AWS Network ACLs, classic ACL on a router) evaluate each packet independently against a rule list. They have no memory of connection state. To allow return traffic for an outbound TCP connection, you must explicitly permit inbound traffic on the ephemeral port range.

Stateful firewalls (e.g., Linux iptables/nftables, AWS Security Groups, firewalld) track connection state. If outbound traffic is permitted, return packets for that connection are automatically allowed — you only write rules for traffic initiation.

iptables Basics

iptables organises rules into tables (filter, nat, mangle, raw) and chains (INPUT, FORWARD, OUTPUT, plus custom). The filter table is the default and handles accept/drop decisions.

# List all filter rules with line numbers
iptables -L -n -v --line-numbers

# Allow inbound SSH
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow established/related return traffic (stateful)
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Drop everything else inbound (default deny)
iptables -P INPUT DROP

# Allow a specific source IP to reach port 5432
iptables -A INPUT -s 10.0.1.0/24 -p tcp --dport 5432 -j ACCEPT

# Save rules (Debian/Ubuntu)
iptables-save > /etc/iptables/rules.v4

# View NAT table (SNAT/DNAT/MASQUERADE rules)
iptables -t nat -L -n -v

Note: nftables is the modern replacement for iptables on Linux. It uses a unified table/chain model with improved performance and atomic rule updates. Both remain common; many distributions still default to iptables wrappers over nftables.

Cloud Security Groups

Cloud security groups (AWS, GCP, Azure) are stateful, VM-level virtual firewalls applied to network interfaces. Key characteristics:

Rules specify direction (inbound/outbound), protocol, port range, and source/destination (CIDR or another security group ID)
Default deny: traffic is blocked unless explicitly permitted
Changes take effect immediately — no restart needed
In AWS, Security Groups are distinct from Network ACLs (NACLs); NACLs are stateless and operate at the subnet level

# List security groups for a VPC (AWS CLI)
aws ec2 describe-security-groups --filters "Name=vpc-id,Values=vpc-0abc123"

# Describe inbound rules for a specific group
aws ec2 describe-security-groups --group-ids sg-0abc123 \
  --query 'SecurityGroups[*].IpPermissions'

# Add an inbound rule (allow HTTPS from anywhere)
aws ec2 authorize-security-group-ingress \
  --group-id sg-0abc123 \
  --protocol tcp --port 443 --cidr 0.0.0.0/0

Common Firewall Patterns

Scenario	Rule
Allow SSH from bastion only	Inbound TCP 22 from bastion security group
Allow all outbound internet	Outbound all, 0.0.0.0/0
Restrict database to app tier	Inbound TCP 5432 from app security group only
Allow health checks	Inbound TCP 80/443 from load balancer security group
Block ICMP ping	No inbound rule for protocol ICMP (implicit deny)

Caution: Avoid 0.0.0.0/0 on inbound rules for management ports (SSH, RDP, database ports). Use a bastion/jump host, VPN, or AWS Session Manager instead.

NAT

Network Address Translation allows hosts with private IP addresses to communicate with the public internet by rewriting IP addresses in transit.

SNAT (Source NAT / Masquerade)

The most common form: a router or gateway rewrites the source IP of outbound packets from a private address to its own public IP, maintaining a connection-tracking table to forward return traffic back to the correct private host.

In cloud environments this is the NAT Gateway (AWS) or Cloud NAT (GCP). Private subnet instances route internet-bound traffic to the NAT gateway; return traffic is forwarded back automatically.

# Enable IP forwarding on a Linux NAT host
echo 1 > /proc/sys/net/ipv4/ip_forward
sysctl -w net.ipv4.ip_forward=1

# MASQUERADE rule: outbound traffic on eth0 gets the host's public IP
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

DNAT (Destination NAT / Port Forwarding)

DNAT rewrites the destination IP (and optionally port) of inbound packets. This is how port forwarding works: traffic arriving at a public IP on port 8080 is redirected to an internal host at port 80.

# Forward external port 8080 to internal host 10.0.1.5:80
iptables -t nat -A PREROUTING -p tcp --dport 8080 \
  -j DNAT --to-destination 10.0.1.5:80

# Also enable forwarding for the redirected traffic
iptables -A FORWARD -p tcp -d 10.0.1.5 --dport 80 -j ACCEPT

NAT in Kubernetes

Kubernetes relies heavily on NAT internally:

kube-proxy (iptables mode) installs DNAT rules that redirect Service ClusterIP traffic to a real Pod IP
NodePort Services use DNAT to redirect external traffic arriving on a node's port to the correct pod
CNI plugins implement pod-to-internet SNAT for pods with no public IP

MAC Addresses and ARP

MAC Addresses

A MAC (Media Access Control) address is a 48-bit Layer 2 hardware address assigned to a network interface. It is written as six colon- or hyphen-separated pairs of hex bytes: aa:bb:cc:dd:ee:ff.

The first three bytes (OUI — Organizationally Unique Identifier) identify the manufacturer; the last three bytes are device-specific. MAC addresses are used for delivery within a single Layer 2 segment (broadcast domain). Routers do not forward MAC addresses between segments — IP (Layer 3) handles inter-segment routing.

Use the MAC Address Formatter to normalize MAC addresses between colon, hyphen, and Cisco dotted-quad formats, and to look up the manufacturer OUI for any address.

ARP — Address Resolution Protocol

ARP resolves an IPv4 address to a MAC address within a local subnet. When host A wants to send a packet to 10.0.1.5 (same subnet), it broadcasts an ARP request: "Who has 10.0.1.5? Tell 10.0.1.1." The host with that IP replies with its MAC address. A then writes the MAC into the Ethernet frame header and transmits.

ARP replies are cached in an ARP table (also called ARP cache) with a short TTL (typically 20–30 minutes on Linux).

# Show the ARP cache
arp -n
ip neigh show

# Send a gratuitous ARP (announce MAC change)
arping -I eth0 -c 3 10.0.1.5

# Flush the ARP cache for an interface
ip neigh flush dev eth0

ARP in virtual/cloud environments: Virtual switches and hypervisors handle ARP within a physical host. Cloud providers use proxy ARP or ARP suppression to avoid broadcast storms across large Layer 2 domains. In Kubernetes, the CNI plugin handles L2 within a node; cross-node traffic is routed at L3.

Note: ARP spoofing (sending gratuitous ARP replies with a false MAC) is a common Man-in-the-Middle attack vector on flat L2 networks. Mitigation: Dynamic ARP Inspection (DAI) on managed switches, or move to routed L3-only designs.

Network Troubleshooting Toolkit

Systematic troubleshooting follows the OSI model from the bottom up: is the interface up? Can I reach the gateway? Can I resolve DNS? Can I complete a TCP handshake? Can I get an HTTP 200? Each step isolates a layer. The commands below (ping, dig, ss, tcpdump) are Linux utilities; see Linux for DevOps for the broader shell environment they live in.

ping — ICMP Reachability

# Basic connectivity test
ping -c 4 8.8.8.8

# Set packet size (test MTU issues)
ping -c 4 -s 1472 8.8.8.8   # 1472 + 28 byte ICMP/IP header = 1500 (standard MTU)

# Ping with interval
ping -i 0.2 -c 20 10.0.1.5

What it tells you: Whether L3 reachability exists and roughly what the round-trip latency looks like. ICMP blocked by a firewall will make a host appear unreachable even when TCP services are fine.

traceroute / mtr — Path Discovery

# Trace the route to a host (UDP probes on Linux)
traceroute 8.8.8.8

# Use ICMP probes (works where UDP is blocked)
traceroute -I 8.8.8.8

# Use TCP SYN on port 80 (works through most firewalls)
traceroute -T -p 80 8.8.8.8

# mtr: real-time traceroute with packet loss per hop
mtr 8.8.8.8

# mtr non-interactive report (good for pasting into tickets)
mtr --report --report-cycles 20 8.8.8.8

What it tells you: Which hop introduces latency, where packet loss starts, and whether traffic takes an unexpected path.

dig — DNS Interrogation

# Full record lookup with TTL
dig example.com A

# Query a specific resolver
dig @1.1.1.1 example.com

# Trace resolution from root
dig +trace example.com

# Check DNSSEC
dig +dnssec example.com

# Check MX records
dig example.com MX +short

# Reverse lookup
dig -x 93.184.216.34

What it tells you: Whether DNS resolution succeeds, which nameserver answered, the TTL remaining on a cached record, and whether DNSSEC is valid.

ss / netstat — Socket State

# All listening TCP sockets with process name
ss -tnlp

# All established connections
ss -tn state established

# Count connections per remote IP (find abusive clients)
ss -tn state established | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

# Socket summary (total connections by state)
ss -s

# Legacy equivalent (older systems)
netstat -tnlp
netstat -s   # protocol statistics

What it tells you: What is actually listening, whether connections are accumulating in TIME_WAIT or CLOSE_WAIT, and which process owns a port.

tcpdump — Packet Capture

# Capture all traffic on eth0
tcpdump -i eth0

# Capture only traffic on port 443
tcpdump -i eth0 port 443

# Capture TCP SYN packets (new connection attempts)
tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'

# Capture ICMP
tcpdump -i eth0 icmp

# Write to file for Wireshark analysis
tcpdump -i eth0 -w /tmp/capture.pcap

# Read a capture file
tcpdump -r /tmp/capture.pcap

# Show packet contents in ASCII
tcpdump -i eth0 -A port 80

# Capture DNS traffic
tcpdump -i eth0 udp port 53

What it tells you: Exactly what traffic is on the wire. Indispensable for confirming that packets are being sent and received, diagnosing TLS errors, and verifying firewall behavior.

curl — HTTP Testing

# Basic HTTP request
curl https://example.com

# Follow redirects, show headers
curl -L -I https://example.com

# Verbose output (full request/response headers, TLS handshake)
curl -v https://example.com

# Test with a specific DNS resolution (bypass DNS)
curl --resolve example.com:443:93.184.216.34 https://example.com

# Time each phase of the request
curl -w "\nDNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTLS: %{time_appconnect}s\nFirst byte: %{time_starttransfer}s\nTotal: %{time_total}s\n" \
  -o /dev/null -s https://example.com

# Send a POST with JSON body
curl -X POST https://api.example.com/v1/items \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"name":"widget"}'

# Test a health endpoint
curl -sf https://api.example.com/healthz || echo "UNHEALTHY"

What it tells you: Whether an HTTP service is reachable, what headers and status codes it returns, how long each phase of the request takes, and whether TLS is configured correctly.

nc (netcat) — TCP/UDP Probe

# Test TCP connectivity to a host/port
nc -zv 10.0.1.5 5432

# Test UDP
nc -zuv 10.0.1.5 53

# Listen for TCP connections on a port (useful to verify firewall rules)
nc -l 8080

# Simple port scan of a range
nc -zv 10.0.1.5 20-25

What it tells you: Whether a TCP handshake can be completed to a specific port, isolating network/firewall issues from application issues.

Quick Troubleshooting Checklist

1. Interface up?          ip link show
2. IP assigned?           ip addr show
3. Default route?         ip route show
4. Can reach gateway?     ping <gateway IP>
5. Can reach internet?    ping 8.8.8.8
6. DNS working?           dig google.com
7. Target TCP port open?  nc -zv <host> <port>
8. HTTP response?         curl -v http://<host>:<port>
9. TLS valid?             curl -v https://<host>
10. App responding?       curl https://<host>/healthz

Common Networking Interview Questions

Q: What is the difference between a router and a switch?
A: A switch operates at Layer 2 — it forwards Ethernet frames within a single broadcast domain using MAC address tables. A router operates at Layer 3 — it forwards IP packets between different networks using routing tables, connecting separate subnets.

Q: How many usable hosts are in a /27 subnet?
A: Host bits = 32 − 27 = 5. Total addresses = 2⁵ = 32. Usable hosts = 32 − 2 = 30 (subtract network address and broadcast address).

Q: You can ping a server but cannot SSH to it. What is the likely cause?
A: ICMP is allowed (so L3 is fine) but TCP port 22 is blocked. Most likely causes: firewall rule or security group blocks inbound port 22, SSH daemon is not running, or SSH is listening on a non-standard port. Check: nc -zv <host> 22 and ss -tnlp | grep :22 on the server.

Q: What happens when you type a URL into a browser?
A: DNS resolves the hostname to an IP. A TCP connection is established (3-way handshake). TLS handshake occurs if HTTPS. An HTTP GET request is sent. The server processes and returns a response. The browser renders the HTML, making additional requests for assets (CSS, JS, images). Keep-alive connections reuse the TCP connection for multiple requests.

Q: What is the difference between NAT and a proxy?
A: NAT rewrites IP headers at L3/L4 transparently — the application is unaware. A proxy operates at L7, establishing separate connections on each side and understanding the application protocol (HTTP, etc.). A proxy can inspect, log, and modify application-level data; NAT cannot.

Q: What does a 502 Bad Gateway error mean?
A: The reverse proxy or load balancer received an invalid response (or no response) from the upstream server. The proxy is reachable, but the backend application is not responding correctly — crashed, overloaded, or returning a malformed response.

Q: How would you verify that a DNS change has propagated?
A: Query the authoritative nameserver directly (dig @ns1.example.com example.com) to confirm the change is live at the source. Then query public resolvers (dig @8.8.8.8, dig @1.1.1.1) to check whether their caches have expired. Use dig +trace to walk the resolution chain. Also check the TTL on the old record to understand the maximum propagation delay.

Q: Explain the TCP TIME_WAIT state and why it matters.
A: After the active close side sends the final ACK, it enters TIME_WAIT for 2×MSL (Maximum Segment Lifetime, typically 60 seconds). This prevents delayed duplicate packets from a closed connection being accepted as part of a new connection with the same 4-tuple. Under high connection rates, large numbers of TIME_WAIT sockets can exhaust ephemeral ports. Mitigation: enable SO_REUSEADDR, increase ephemeral port range, or use persistent connections / connection pooling.

Q: What is VLAN and why is it used?
A: A VLAN (Virtual LAN) logically segments a physical L2 network into isolated broadcast domains using 802.1Q tagging on Ethernet frames. Traffic in different VLANs cannot communicate without routing through a Layer 3 device. VLANs are used to isolate environments (prod/staging/dev), separate traffic types (management vs. data), and improve security without requiring separate physical switches.

Q: What is BGP and where do DevOps engineers encounter it?
A: BGP (Border Gateway Protocol) is the routing protocol that exchanges routes between autonomous systems on the internet. DevOps engineers encounter it in AWS Direct Connect (BGP advertises your on-premises routes to AWS), Transit Gateway, and in tools like MetalLB (which uses BGP to advertise Kubernetes service IPs to upstream routers in bare-metal clusters).

Note: Prefer a day-by-day path? This is covered in Mission 90 Days 11–13 — a free 90-day guided DevOps program with browser terminal missions.

What's Next

Networking knowledge is most powerful when combined with hands-on command-line proficiency. Pair this with the command-line side in Linux for DevOps, where you'll see how the same ss, ip, and tcpdump commands fit into a broader systems administration workflow.

For immediate hands-on practice, use the subnet tools linked throughout this guide — in particular the Subnet Calculator and Subnet Splitter — to work through VPC design exercises until the CIDR arithmetic feels automatic.

AWS AI Practitioner (AIF-C01): 14-Day Study Plan, Exam Guide, and Free Mock

opscanopy — Mon, 07 Sep 2026 15:35:44 +0000

A study plan for the AWS Certified AI Practitioner exam written for someone with no AI, ML, or AWS background, at one to one and a half hours a day. One main course, one question bank, this guide, and a free full-length mock. Everything here follows exam guide version 1.1, published 30 April 2026, which added agentic AI, Amazon Bedrock AgentCore, Kiro, Strands Agents, and Amazon Nova to the syllabus and dropped several services that older study material still teaches. If a resource you are using does not mention agents or AgentCore, it predates the current exam.

When you are ready to test yourself, the free 65-question AIF-C01 mock on this site follows the real domain weighting and explains every answer.

Exam at a glance

Fact	Detail
Exam code	AIF-C01 (AWS Certified AI Practitioner)
Questions	65 (50 scored + 15 unscored, not identified)
Duration	90 minutes (about 83 seconds per question)
Passing score	700 on a scaled range of 100–1,000
Cost	USD 100 (50% off until 30 September 2026, see below)
Validity	3 years
Delivery	Pearson VUE test centre or OnVUE online proctoring
Question types	Multiple choice, multiple response, ordering, matching
Hands-on or coding	None
Current exam guide	Version 1.1, 30 April 2026

Domain weights

Domain	Weight	≈ Scored questions
1 · Fundamentals of AI and ML	20%	10
2 · Fundamentals of generative AI	24%	12
3 · Applications of foundation models	28%	14
4 · Guidelines for responsible AI	14%	7
5 · Security, compliance, and governance for AI solutions	14%	7

Key: 52% of the exam is generative AI and foundation models. Amazon Bedrock is the centre of the exam: its features (Knowledge Bases, Agents, Guardrails, model choice, Prompt Management) and the prompt engineering vs RAG vs fine-tuning vs distillation decision are the most frequently tested topics. Since the 2026 update, agentic AI (AgentCore, Strands Agents, the Model Context Protocol) sits alongside them. Spend your time there.

Questions are scenario-based: "A company needs to do X with the least operational overhead. Which service?" Memorising definitions is not enough; you need to know when to use each service and each approach.

Is the AWS AI Practitioner worth it, and who is it for

AWS wrote AIF-C01 for people who use AI on AWS rather than build it: analysts, product managers, project leads, sales engineers, and developers who need to talk credibly about AI and pick the right service. The target candidate has up to six months of exposure to AI and ML on AWS, and the exam guide puts model development, feature engineering, hyperparameter tuning, and pipeline building explicitly out of scope.

It is worth taking if you fit that description. The exam costs USD 100, half that under the current promotion, needs roughly 30 to 40 hours of preparation, and the syllabus is a well-organised tour of the concepts and services that appear in real generative AI projects. Recruiters and managers recognise the AWS badge, and in 2026 an AI credential differentiates a CV more than the far more common Cloud Practitioner.

It is not worth taking if you are already a working ML engineer or data scientist. You will find it too basic, and the AWS Certified Machine Learning Engineer – Associate (MLA-C01) is the credential that reflects your skills. The AI Practitioner rarely changes a salary band on its own; it amplifies whatever role you already have by adding demonstrable AI literacy.

How hard is the AIF-C01, and how long does it take to prepare

It is a foundational exam and the easiest of the AWS AI certifications. There is no code, no maths, and no console work. The difficulty is vocabulary and discrimination: the exam uses a large set of terms (tokens, embeddings, RAG, RLHF, distillation, MCP) and a large set of similar-sounding services, and asks you to pick the right one for a scenario under time pressure.

Plan on 14 days at one to one and a half hours a day if you are new to both AI and AWS, which is the schedule below. If you already work with AWS, one week is enough, and if you already work with generative AI tools, three to five days of service-mapping and practice tests will do.

Two facts about scoring shape how you should prepare. First, 83 seconds per question means you cannot deliberate; the qualifier words in the stem ("least operational overhead", "most cost-effective", "no ML expertise", "full control") decide the answer, and you should learn to spot them. Second, the 700 pass mark is a scaled score, and AWS does not publish a raw-percentage equivalent. Treat consistent scores above 80% on realistic practice tests as the signal that you are ready.

Exam cost, discounts, and free vouchers

The list price is USD 100, charged through Pearson VUE in your local currency at Pearson's exchange rate, plus any local tax. In India, for example, the fee is billed in INR with GST added.

Until 30 September 2026 the exam is 50% off, and passing it earns a free Cloud Practitioner exam. The promotion code is AIF2CLOUD. The terms: register for AIF-C01 between 26 May and 30 September 2026, pass by 30 September 2026 at 23:59 local time, and you receive a voucher for the AWS Certified Cloud Practitioner (CLF-C02) exam to be used by 30 November 2026. One per person, only for exams in pass status, and it cannot be combined with other offers.

Two standing benefits also apply. After you pass any AWS certification exam you receive a 50% discount voucher toward your next one, so AIF-C01 is a cheap way to halve the cost of an associate exam later. And AWS Skill Builder offers the official exam-prep plan and a 20-question official practice set free of charge.

Note for non-English candidates: the Italian and German versions of the exam are retired after 15 October 2026. English, French, Spanish, Portuguese, Japanese, Korean, Chinese, and Arabic remain available.

AI Practitioner vs Cloud Practitioner: which first

Neither exam is a prerequisite for the other. They share a format (65 questions, 90 minutes, 700 to pass, USD 100) and a level, and they differ in subject.

	AWS Certified Cloud Practitioner (CLF-C02)	AWS Certified AI Practitioner (AIF-C01)
What it covers	Cloud concepts, core services (EC2, S3, VPC, IAM), billing, security basics	AI/ML and GenAI concepts, Bedrock, SageMaker AI, agentic AI, responsible AI, AI security
Best for	Career switchers into cloud; anyone new to AWS	People in AI-adjacent roles; anyone who already knows AWS basics
Recognition	The most widely held AWS credential	Newer and rarer, so it differentiates more
Assumed knowledge	None	Familiarity with core services, IAM, shared responsibility, pricing

Take Cloud Practitioner first if you are switching careers into cloud and have never used AWS. You cannot reason about Bedrock security or vector stores without knowing what a VPC, an IAM role, or an S3 bucket is, and CLF-C02 is what entry-level cloud job postings ask for.

Take AI Practitioner first if you already have some AWS exposure, or if your job is about AI rather than infrastructure. Until 30 September 2026 this is also the cheaper order: pass AIF-C01 at half price and Cloud Practitioner is free.

AWS AI Practitioner vs Microsoft Azure AI Fundamentals (AI-900)

AI-900 is Microsoft's equivalent foundational AI exam: about USD 99, no prerequisites, and a similar mix of AI and ML concepts with Azure services in place of AWS ones. The concepts transfer almost entirely; the service names do not. Choose by the cloud your organisation runs. If you are undecided, AIF-C01 currently has the richer generative AI and agentic AI coverage, and AWS's promotion makes it cheaper.

AWS AI Practitioner vs Google Cloud Generative AI Leader

Google's Generative AI Leader certification, launched in 2025, is aimed at business leaders rather than practitioners: it is about USD 99, focuses on generative AI strategy and Google Cloud's Gemini and Vertex AI offerings, and contains little classical ML. AIF-C01 is broader, covering traditional ML fundamentals, responsible AI, and security in more depth. Pick Google's exam if your organisation is a Google Cloud shop and your role is strategic; pick AIF-C01 for a more technical foundation.

AWS AI Practitioner vs AWS Machine Learning Engineer – Associate (MLA-C01)

MLA-C01 is the next step up, not an alternative. It is an associate-level, hands-on exam (about USD 150, 65 questions, 130 minutes) covering data preparation, model training and tuning, deployment, and MLOps on SageMaker AI. It assumes you build and operate ML systems. Take AIF-C01 first if you are new to AI; go straight to MLA-C01 if you already do ML engineering. Passing MLA-C01 also renews an AI Practitioner certification.

The 14-day plan

Week one is concepts, week two is practice. The plan assumes Stephane Maarek's Udemy course (about 10 hours); any current course works, so map its sections to the days below. Check that the course you pick covers AgentCore, Strands Agents, and the Model Context Protocol; if it does not, supplement those topics from this guide.

Week 1: concepts

Day	Focus	What to do
1	Setup and AI/ML basics	Skim the official exam guide (15 minutes). Course: intro, AI vs ML vs deep learning vs generative AI vs agentic AI, supervised / unsupervised / reinforcement learning. Read Domain 1 below.
2	ML lifecycle and AWS AI services	Data → train → evaluate → deploy → monitor. SageMaker AI overview. Rekognition, Comprehend, Textract, Transcribe, Polly, Translate, Lex, Personalize. Note which services are not on the 2026 in-scope list.
3	Generative AI fundamentals	Tokens, embeddings, transformers, diffusion models, foundation models, context window, inference parameters, token-based pricing. Domain 2 concepts.
4	Prompt engineering and Bedrock	Zero-shot, few-shot, chain-of-thought, negative prompts, prompt injection, context engineering. Open the Bedrock console (free tier) or spend 20 minutes in PartyRock changing prompts and temperature.
5	RAG, Knowledge Bases, agents	End-to-end RAG flow, chunking, vector stores. Bedrock Knowledge Bases, Guardrails, Agents. Agentic AI: MCP, memory, tool use, multi-agent patterns, AgentCore, Strands Agents. The customisation decision table.
6	Customising and evaluating FMs	Fine-tuning, continued pre-training, distillation, RLHF. Evaluation: ROUGE, BLEU, BERTScore, LLM-as-a-judge, Bedrock Model Evaluation. Prompt caching, Provisioned Throughput vs on-demand.
7	Responsible AI and security	Bias, fairness, explainability, SageMaker Clarify, Model Cards, human-in-the-loop. IAM, KMS, PrivateLink, CloudTrail, Macie, Artifact, Config, Trusted Advisor, the Generative AI Security Scoping Matrix, AgentCore Identity and Policy.

Week 2: practice and revision

Day	Focus	What to do
8	Practice test 1 (baseline)	Take the free 65-question mock on this site. Set a 90-minute timer yourself; the runner does not enforce one. Ignore the score. Read the explanation for every wrong and every guessed answer. Write a weak-topics list.
9	Revise	Re-watch the course sections for your weak topics at 1.5× speed. Read the services cheat sheet below twice.
10	Practice test 2	A Tutorials Dojo timed set, or the 20 official questions on AWS Skill Builder. Target 75%+. Which domain keeps failing?
11	Revise	Bedrock deep dive, agentic AI, and responsible AI again. Read the glossary.
12	Practice test 3	Another Tutorials Dojo set, or retake the mock here and compare against day 8. Target 80%+. Below 80%, move the exam three or four days out; there is no rush.
13	Final light revision	Notes, cheat sheet, glossary only. No new content. Confirm exam time, ID, and the Pearson system check if testing online. Sleep early.
14	Exam day	Arrive or log in 30 minutes early. Flag and move on. Watch for the qualifier words.

If one and a half hours a day is too heavy, stretch week one to ten days but keep the week two structure intact. Never skip the practice tests; they raise your score more than anything else.

Domain 1: Fundamentals of AI and ML (20%)

Pure basics: concept-heavy, service-light. Clear definitions here are easy marks.

Concepts

AI vs ML vs deep learning vs generative AI vs agentic AI: how each relates to the others
Supervised (labelled data) vs unsupervised (clustering, anomaly detection) vs reinforcement learning (reward signal)
Classification vs regression vs clustering; when ML is not appropriate (deterministic rules suffice, or an exact outcome is required)
Inference types: batch, real-time, asynchronous, serverless
Training vs inference; overfitting vs underfitting; bias and variance
Data types: structured and unstructured, labelled and unlabelled, tabular, time-series, image, text
The AI/ML pipeline: business goal → data collection → EDA → feature engineering → training → evaluation → deployment → monitoring; MLOps ideas such as repeatability, model monitoring, and retraining
Sources of foundation models (open-source pre-trained vs custom-trained) and ways to use a model in production (managed API vs self-hosted endpoint)
Traditional ML vs foundation model: when regulation, explainability, or operational constraints favour a conventional model on tabular data
Model metrics: accuracy, precision, recall, F1 (classification), MAE and RMSE (regression); business metrics: cost per user, development cost, customer feedback, ROI

AWS services

Amazon SageMaker AI: build, train, and deploy custom models. Canvas (no-code), Ground Truth (labelling), Data Wrangler, Feature Store, Model Monitor (drift), Clarify (bias and explainability), JumpStart (model hub)
Rekognition (image and video), Textract (text from documents), Comprehend (NLP: sentiment, entities, PII)
Transcribe (speech → text), Polly (text → speech), Translate, Lex (chatbots)
Personalize (recommendations)
For the pipeline stages, the guide now also names Amazon Bedrock, Amazon Quick, and Kiro

Note: Amazon Kendra, Amazon Forecast, Amazon Fraud Detector, and AWS Audit Manager appear in older study material but are not on the 2026 in-scope services list, and Fraud Detector is closed to new customers. Do not pick them as answers; the exam will not make them correct.

Tip: Exam pattern — "The company needs to train a custom model" → SageMaker AI. "Needs a ready-made API with no ML expertise" → a managed AI service (Rekognition, Comprehend, and so on). "Needs a general-purpose text or image capability" → a foundation model in Bedrock. This distinction appears in five or six questions.

Domain 2: Fundamentals of generative AI (24%)

This is where the real weight begins. Know a one-line meaning and one example for every term.

Concepts

What a foundation model (FM) is; an LLM is one type of FM; pre-training vs fine-tuning
Tokens, tokenisation, context window, embeddings, vectors, chunking, semantic similarity
Transformer architecture (self-attention) at a high level; diffusion models for image generation; multimodal models
Inference parameters: temperature, top-p, top-k, max tokens, stop sequences; which increase creativity and which increase determinism
Prompt engineering: zero-shot, one- and few-shot, chain-of-thought, prompt templates, negative prompts, system prompts
Context engineering: deciding what goes into the context window (instructions, retrieved content, memory, tool results) and keeping it relevant
Token-based pricing and its effect on cost and latency; prompt caching
Agentic AI concepts: tool use, memory (short-term vs long-term), multi-agent and orchestrator patterns, workflow orchestration, and the Model Context Protocol (MCP) for connecting agents to external tools and data
Hallucination, nondeterminism, toxicity, prompt injection and jailbreaks, and their mitigations
GenAI use cases (summarisation, assistants, code generation, image and video generation, translation, search, recommendations) and disadvantages (hallucination, interpretability, inaccuracy, nondeterminism)
Model selection factors: model type, performance, capabilities, constraints, compliance, cost, latency, complexity
Business value metrics: ROI, efficiency, conversion rate, average revenue per user, customer lifetime value
The FM lifecycle: data selection → model selection → pre-training → fine-tuning → evaluation → deployment → feedback

AWS services

Amazon Bedrock: serverless API access to FMs from Amazon (Nova), Anthropic, Meta, Mistral, Cohere, Stability AI, and others; pay per token
Amazon Nova: Amazon's own family of models for text, image (Nova Canvas), and video generation
Amazon Bedrock AgentCore: the managed platform to run agents in production
Strands Agents: AWS's open-source SDK for building agents in code
Kiro: AWS's agentic IDE for planning and writing code
Amazon Quick (formerly QuickSight): business intelligence with natural-language questions and agentic workflows
Amazon Q Business and Amazon Q Developer: enterprise assistant over company data; developer assistant for code and AWS
SageMaker JumpStart: deploy open-source FMs onto your own endpoint (more control, more effort than Bedrock)
AWS Transform: agentic migration and modernisation of legacy workloads
PartyRock: a free Bedrock playground that needs no AWS account (Apple, Google, or Amazon login). Useful for learning; no longer named in the exam guide

Temperature ↑	Temperature ↓ (near 0)
More random and creative. Marketing copy, brainstorming.	Deterministic and factual. Extraction, classification, code.

Agentic AI on AWS

The April 2026 guide made agents a first-class topic. Know these five things and how they fit together.

Piece	What it is	When the question means it
AI agent	A foundation model that plans steps, calls tools or APIs, observes results, and adapts to reach a goal	"Book, order, look up, then act" scenarios; multi-step tasks
Model Context Protocol (MCP)	An open standard for exposing tools and data sources to agents through one reusable interface	"Connect the agent to many internal systems without custom code per tool"
Strands Agents	Open-source, model-driven SDK: define model, tools, and prompt in a few lines of code	"Build an agent in code"
Amazon Bedrock AgentCore	Managed platform to deploy and operate agents: Runtime, Memory, Identity, Gateway, Policy, Observability, plus Code Interpreter and Browser tools	"Run agents in production securely without managing servers"
Amazon Bedrock Agents	The managed agent feature inside Bedrock: action groups (APIs, Lambda), knowledge bases, memory	"Managed agent with the least setup"

Memory in agents is short-term (the current session's context) or long-term (facts and preferences persisted across sessions). Multi-agent systems typically use an orchestrator or supervisor that decomposes a goal and delegates to specialised agents. AgentCore Identity handles the agent's own identity and delegated access on a user's behalf; Policy in AgentCore authorises each tool call through AgentCore Gateway.

Domain 3: Applications of foundation models (28%)

The largest domain. Nearly every question revolves around Bedrock. The table below answers roughly five exam questions on its own.

Approach	Use it when	Cost and effort
Prompt engineering (in-context learning)	You need better output from the model as it is. Adds no new knowledge. Try this first.	Cheapest, fastest
RAG	The model needs fresh or private data (documents, databases), with citations. No retraining. → Bedrock Knowledge Bases	Low to medium
Fine-tuning (supervised or reinforcement)	You need a specific tone, format, or task behaviour, and have labelled examples or a reward signal	Higher; the custom model then runs on Provisioned Throughput or an on-demand custom-model deployment
Distillation	You want a smaller, cheaper, faster model that matches a large model's accuracy on your task	Medium; a teacher model generates training data for a student
Continued pre-training	You want the model to absorb a whole domain (legal, medical) from large amounts of unlabelled text	Most expensive

Concepts

Model selection criteria: cost, modality, latency, multilingual support, model size and complexity, customisation, input and output length, prompt caching
Effect of inference parameters on responses
RAG pipeline: documents → chunking → embeddings → vector store → retrieve → augment prompt → generate
Vector stores on AWS: Amazon OpenSearch Service (Serverless or managed), Amazon Aurora PostgreSQL with pgvector, Amazon Neptune Analytics (GraphRAG), Amazon RDS for PostgreSQL, Amazon S3 Vectors; Bedrock Knowledge Bases can quick-create OpenSearch Serverless, Aurora PostgreSQL Serverless, Neptune Analytics, or S3 Vectors
The role of agents in multi-step tasks and their business applications
Prompt engineering constructs (context, instruction, negative prompts) and techniques (chain-of-thought, zero-shot, single-shot, few-shot, templates); best practices (specificity, experimentation, guardrails); risks (exposure, poisoning, hijacking, jailbreaking)
Prompt versioning and management with Bedrock Prompt Management
Training elements: pre-training, fine-tuning, continued pre-training, distillation; fine-tuning methods: instruction tuning, domain adaptation, transfer learning; data preparation: curation, governance, size, labelling, representativeness, RLHF
Evaluation: human-in-the-loop, benchmark datasets, Bedrock Model Evaluation; metrics ROUGE (summaries), BLEU (translation), BERTScore (semantic similarity), LLM-as-a-judge; evaluating RAG, agents, and workflows as applications; business alignment metrics: task completion rate, user satisfaction, cost per interaction
Pricing: on-demand per token vs Provisioned Throughput (dedicated capacity for steady high volume) vs custom-model deployment

Bedrock features to memorise

Knowledge Bases: managed RAG; data from S3 and other connectors, automatic chunking, embedding, and vector storage
Agents: task decomposition, action groups (APIs, Lambda), memory
Guardrails: denied topics, content filters, sensitive-information (PII) filters, word filters, contextual grounding check; applied to input and output, across models
Model Evaluation: automatic metrics, human evaluation, or LLM-as-a-judge
Prompt Management: store, version, test, and reuse prompts
Custom models: supervised fine-tuning, reinforcement fine-tuning, distillation, continued pre-training
Provisioned Throughput: dedicated model units for predictable high-volume workloads; custom models can alternatively use on-demand custom-model deployment
Amazon Nova and Titan: Amazon's own models; Titan Image Generator and Nova Canvas embed an invisible watermark in every generated image, and a detection API confirms it

Domain 4: Guidelines for responsible AI (14%)

Mostly vocabulary: AWS's responsible-AI dimensions plus three or four services.

Concepts

Features of responsible AI: bias, fairness, inclusivity, robustness, safety, veracity; AWS's eight dimensions add explainability, privacy and security, controllability, governance, and transparency
Bias types: sampling and selection, measurement, confirmation, algorithmic; class imbalance; effects of bias and variance on demographic groups, overfitting, and underfitting
Dataset characteristics: inclusive, diverse, balanced, curated sources
Tools to detect and monitor bias: label-quality analysis, human audits, subgroup analysis
Interpretability vs explainability, and the trade-off with performance (a linear model vs a deep neural network); when to prefer the transparent model
Responsible model selection, including environmental and sustainability considerations
Human-in-the-loop and human oversight for high-stakes decisions; human-centred design: user feedback mechanisms, decision transparency
Legal risks: intellectual property infringement, biased outputs, hallucinations, end-user risk, loss of customer trust
Transparency artefacts: model cards, AI service cards, documented limitations, open-source models, data and licensing

AWS services

SageMaker Clarify: bias detection before and after training, plus explainability (SHAP)
SageMaker Model Cards: model documentation; AWS AI Service Cards do the same for AWS's own services
Bedrock Guardrails: toxicity, denied topics, PII; Bedrock Model Evaluations as a transparency tool
SageMaker Model Monitor: drift and bias drift in production
Amazon Augmented AI (A2I): human review of low-confidence predictions; the canonical human-in-the-loop example, though it is not on the current in-scope list

Remember: Trigger words — "detect bias" → Clarify. "Route low-confidence results to a human" → human review, A2I. "Document the model" → Model Cards. "Block harmful content" → Guardrails. "Explain a decision feature by feature" → Clarify (SHAP).

Domain 5: Security, compliance, and governance for AI solutions (14%)

General AWS security applied to AI workloads. If you are new to AWS, give this an extra 20 minutes: IAM, KMS, and VPC will be unfamiliar.

Concepts

Shared responsibility model: AWS secures the cloud, you secure what you put in it (IAM, data, guardrails, application)
IAM: users, roles, policies, least privilege; service roles (for example Bedrock reading from S3)
Encryption at rest (KMS, customer-managed keys) and in transit (TLS)
Private networking: VPC, interface endpoints with AWS PrivateLink to keep Bedrock traffic off the public internet
Bedrock data facts: your content is not used to train base models, is not shared with model providers, and stays in your Region
Source citation and data origins: data lineage and cataloguing (Glue Data Catalog, Lake Formation), Model Cards
Secure data engineering: data quality, privacy-enhancing technologies, access control, integrity
Security and privacy considerations: threat detection, vulnerability management, prompt injection, data leakage prevention, output filtering and validation, audit trail and logging for AI interactions, toxicity
Hallucination detection and grounding: RAG grounding, output validation, confidence scoring; Guardrails' contextual grounding check
AgentCore Identity and Policy in AgentCore for agent authentication and per-call authorisation
Data governance: lifecycles, logging, residency, retention, monitoring
Governance processes: policies, review cadence, the Generative AI Security Scoping Matrix (five scopes from consuming a public app to training your own model), transparency standards, team training
Compliance frameworks: ISO, SOC, GDPR, HIPAA; algorithmic accountability laws

AWS services

IAM, KMS, Secrets Manager
CloudTrail (who called which API, when), Bedrock model invocation logging (prompt and response content to S3 or CloudWatch Logs), CloudWatch (metrics and logs)
AWS Config (resource compliance rules), AWS Trusted Advisor (best-practice checks), AWS Well-Architected Tool
Macie (find PII in S3), Inspector (software vulnerabilities), AWS Artifact (download compliance reports)
Glue Data Catalog, Lake Formation (data governance and permissions)
VPC endpoints / PrivateLink
AWS Budgets, Cost Explorer (cost governance)

Caution: Amazon GuardDuty and AWS IAM Identity Center are explicitly out of scope in the 2026 guide, and AWS Audit Manager is not on the in-scope list. Older material lists all three under this domain.

Services cheat sheet

Around 40% of the exam is "which service?". Read the trigger phrase, recall the service. Read this table twice on day 9 and twice on day 13.

When the question says…	Service	In one line
Access foundation models via API, no infrastructure	Amazon Bedrock	Serverless FM access, pay per token
Chatbot over company documents (RAG) with least effort	Bedrock Knowledge Bases	Managed RAG from S3 and other sources
Multi-step tasks, call APIs, book or order things	Bedrock Agents / AgentCore	Orchestration plus action groups or tools
Run agents in production: runtime, memory, identity, tool gateway	Amazon Bedrock AgentCore	Managed agent platform
Build an agent in code with an open-source SDK	Strands Agents	Model-driven agent framework
Connect an agent to many tools through one standard	Model Context Protocol (MCP)	Open tool-integration standard
Block harmful content, deny topics, redact PII, check grounding	Bedrock Guardrails	Safety filters, model-agnostic
Version, test, and reuse prompts across teams	Bedrock Prompt Management	Prompt governance
Compare or score models, including LLM-as-a-judge	Bedrock Model Evaluation	Automatic, human, or judge-model evaluation
Smaller, cheaper model with a big model's accuracy	Bedrock distillation	Teacher trains student
Amazon's own text, image, and video models	Amazon Nova	Nova family in Bedrock
Enterprise assistant over employees' company data	Amazon Q Business	Permission-aware assistant
Agentic IDE that plans and writes code	Kiro	AWS's AI development environment
Ask natural-language questions of dashboards	Amazon Quick	BI, formerly QuickSight
Modernise mainframe or .NET workloads with agents	AWS Transform	Agentic migration
Build, train, deploy a custom ML model	Amazon SageMaker AI	Full ML platform
No-code ML for business analysts	SageMaker Canvas	Point-and-click models
Label training data with humans	SageMaker Ground Truth	Labelling workforce
Deploy an open-source FM with full control	SageMaker JumpStart	Pre-trained model hub
Detect bias, explain predictions	SageMaker Clarify	Bias metrics plus SHAP
Detect model or data drift in production	SageMaker Model Monitor	Continuous monitoring
Document a model for auditors	SageMaker Model Cards	Model documentation
Human review of low-confidence predictions	Amazon A2I	Human-in-the-loop workflows
Faces, objects, unsafe images in photos or video	Rekognition	Computer vision API
Extract text, tables, forms from scanned documents	Textract	OCR and more
Sentiment, entities, key phrases, PII in text	Comprehend	NLP API
Speech → text (call transcripts, subtitles)	Transcribe	Speech recognition
Text → lifelike speech	Polly	Text-to-speech
Translate between languages	Translate	Neural machine translation
Voice or text chatbot with intents and slots	Lex	The technology behind Alexa
Product recommendations like Amazon.com	Personalize	Real-time recommendations
Store embeddings for vector search	OpenSearch Service, Aurora PostgreSQL, Neptune Analytics, S3 Vectors	Vector stores for RAG
Find sensitive data (PII) in S3	Macie	Data security
Who called which API, and when	CloudTrail	Audit log
Keep prompts and responses for review	Bedrock model invocation logging	To S3 or CloudWatch Logs
Download SOC or ISO compliance reports	AWS Artifact	Compliance documents
Check resource configuration against rules	AWS Config	Configuration compliance
Best-practice recommendations for the account	Trusted Advisor	Checks and recommendations
Find software vulnerabilities in workloads	Amazon Inspector	Vulnerability management
Keep Bedrock traffic off the public internet	VPC endpoint (PrivateLink)	Private connectivity
Encrypt data, manage keys	KMS	Key management
Decide which security controls a GenAI project needs	Generative AI Security Scoping Matrix	Five scopes

Glossary: the terms that appear again and again

Term	Meaning
Token	A small piece of text (roughly four characters, three quarters of a word). Pricing and context limits are based on tokens.
Context window	How many tokens the model can see in one request, input plus output.
Embedding	A numeric vector for text or images; similar meaning → nearby vectors. The backbone of RAG.
Vector database	Stores embeddings and performs similarity search.
Chunking	Splitting large documents into smaller pieces before embedding.
RAG	Retrieval Augmented Generation: retrieve relevant data first, then insert it into the prompt before generating.
Context engineering	Choosing and structuring what enters the context window so the model sees what matters and nothing that misleads it.
Hallucination	The model states something false with confidence. Fixes: RAG grounding, guardrails, lower temperature, output validation, human review.
Temperature	The randomness knob. Near 0 is deterministic; higher is creative.
Top-p / top-k	Limit next-token candidates by probability mass (p) or count (k).
Zero-shot / few-shot	Prompting with no examples / with a few examples.
Chain-of-thought	Asking the model to reason step by step; improves accuracy on multi-step problems.
Prompt injection	User input that tries to override the model's instructions. Fixes: guardrails, input validation, least-privilege tools.
Prompt caching	Reusing the processed form of a repeated prompt prefix to cut cost and latency.
Fine-tuning	Adjusting model weights with labelled data (or a reward signal) for a specific task or style.
Distillation	Training a smaller student model to match a larger teacher model on a task.
RLHF	Reinforcement learning from human feedback: aligning a model to human preferences.
Transfer learning	Reusing a pre-trained model for a new, related task.
Agentic AI	Systems in which a model plans, uses tools, keeps memory, and acts toward a goal, often as several cooperating agents.
MCP	Model Context Protocol: an open standard that connects agents to tools and data sources.
Overfitting	Perfect on training data, poor on new data. Fixes: more data, regularisation, early stopping.
Precision vs recall	Precision: of everything flagged, how much was correct. Recall: of everything correct, how much was caught. Fraud and medical screening → recall matters more.
ROUGE / BLEU / BERTScore	Quality metrics for summarisation / translation / semantic similarity against a reference.
LLM-as-a-judge	Using a capable model to grade another model's outputs against a rubric, validated by human spot checks.
Model drift	Real-world data changes over time and accuracy drops. Model Monitor detects it.
Explainability	Why the model made a decision; required in regulated industries. SHAP via Clarify.

Study material

Few resources, fully used. One course, one question bank, and the free official material. Everything else is optional.

Resource	Cost	Notes
Stephane Maarek: Ultimate AWS Certified AI Practitioner (Udemy), the main course	About USD 10–15 on sale	About 10 hours, exam-focused, beginner-friendly, quiz after each section. Udemy is almost always on sale; never pay full price. Confirm the version you buy covers the 2026 additions. udemy.com/course/aws-ai-practitioner-certified
OpsCanopy free AIF-C01 mock, your day 8 baseline	Free	65 original questions in the real domain weighting, every answer explained, written against exam guide v1.1. Use it timed on day 8 and again on day 12. Take the mock
Tutorials Dojo (Jon Bonso): AIF-C01 Practice Exams, the question bank	About USD 15	Closest to real exam difficulty, with detailed explanations for every option. Those explanations are your revision. Timed mode and review mode. portal.tutorialsdojo.com
Official exam guide (v1.1)	Free	Skim on day 1, re-read on day 13. It contains every task statement and the in-scope services list; this is the syllabus. docs.aws.amazon.com exam guide
AWS Skill Builder: exam-prep plan and official practice question set	Free (account required)	AWS's own prep course and 20 official practice questions. The best way to learn the question style. Do it on day 10 or 11. skillbuilder.aws
Andrew Brown / freeCodeCamp course (YouTube), the free alternative to Maarek	Free	About 15 hours covering the full syllabus. Watch at 1.5× and skip hands-on demos if short on time. Search "AWS Certified AI Practitioner freeCodeCamp".
PartyRock	Free, no AWS account	A playground built on Bedrock. 20 minutes on day 4: build a tiny app, change temperature, change the prompt. Concepts click. partyrock.aws
Tutorials Dojo cheat sheets and AWS service FAQs	Free	Service-by-service one-pagers for revision. Bedrock, SageMaker AI, and the Domain 5 security services are enough. tutorialsdojo.com/aws-cheat-sheets

Total budget: roughly USD 25–30 for course and practice tests, plus the exam fee of USD 100, or USD 50 with the AIF2CLOUD promotion before 30 September 2026.

Exam day and question strategy

Keywords decide the answer. "Least operational overhead", "most cost-effective", "minimal ML expertise" → a managed service (Bedrock, Amazon Q, Rekognition). "Full control", "custom model" → SageMaker AI. "Needs current or private documents" → RAG. "Needs a consistent tone or format" → fine-tuning. "Must act, not just answer" → an agent.

Eliminate first. Two of the four options are almost always clearly wrong (wrong service, wrong domain). Remove them, then look for the trigger word in the remaining two.

Time management. 83 seconds per question. If you do not understand a question in 30 seconds, flag it and move on. Keep the last 15 minutes for flagged questions. Never leave a blank: unanswered questions score as incorrect and there is no penalty for guessing.

Multiple response. The question states how many to select. Choose exactly that many; there is no partial credit.

Ordering and matching. The FM lifecycle, the ML pipeline, and the RAG pipeline appear as ordering items. Memorise the sequences. Matching items pair services with use cases; the cheat sheet above is the preparation.

If testing online. Run the Pearson system test the day before. One screen only, clean desk, phone out of reach, no notes, government ID ready. AWS exams do not offer breaks; leaving the webcam view ends the exam.

Understand the score. 700 out of 1,000 is a scaled score and AWS does not publish a raw-percentage cutoff. Scoring 80% or more consistently in practice is safe. Results appear in your AWS Certification Account within five business days.

Confidence. This is a foundational exam. If you understand the concepts and service selection, you will pass. If in doubt, push the exam back three days, but do not over-prepare.

Take the free 65-question mock

The OpsCanopy AIF-C01 Full Mock Exam has 65 original questions in the real domain weighting (13 / 16 / 18 / 9 / 9), with about a dozen on the 2026 additions: agentic AI, MCP, AgentCore, Kiro, Strands Agents, distillation, prompt caching, context engineering, LLM-as-a-judge, and the Generative AI Security Scoping Matrix. Every answer explains why each wrong option fails. Take it timed on day 8, read every explanation, and retake it on day 12 to measure the difference. If you prefer to read rather than click, the answers-explained page lays out all 65 questions as prose.

The runner uses multiple-choice and multiple-response items only; the real exam adds ordering and matching, whose content is covered here in multiple-choice form.

Exam facts (65 questions, 90 minutes, USD 100, 700/1,000, domain weights, question types) are per AWS exam guide v1.1 dated 30 April 2026 and the AWS certification page as of September 2026. Promotion terms are per Pearson VUE's AIF2CLOUD page. Prices for third-party courses are approximate and change frequently.

x509: certificate signed by unknown authority — why it works in your browser but not in Go, curl or Docker

opscanopy — Wed, 02 Sep 2026 17:20:20 +0000

You deploy a service, it calls an HTTPS endpoint, and the request dies:

x509: certificate signed by unknown authority

Then you open the same URL in Chrome and it loads with a padlock. Nothing is wrong with the certificate as far as your browser is concerned, so the natural conclusion is that your code, your container or your language is broken.

It usually isn't. In the large majority of cases the server really is misconfigured, and your browser is hiding it. This post explains what the error actually means, why browsers disagree with Go and curl, and the four causes that between them account for almost every occurrence.

Which one is you — 20 seconds

The right cause depends almost entirely on what you were connecting to, so start here rather than reading straight through:

What you were doing	Almost always	Jump to
`docker login` / `docker pull` from a private registry	Self-signed cert the daemon was never given	Cause 4
Any HTTPS call from inside a container, failing against every host	The image has no CA bundle	Cause 2
A Go service, curl or CI job hitting a public endpoint that works in Chrome	The server is missing its intermediate	Cause 1
Anything at all, but only on the office network / VPN	A TLS-inspecting proxy re-signed it	Cause 3

If you are here from a Docker registry error specifically, cause 4 is your answer and the fix is three commands. The rest of the post explains why the error is worded so unhelpfully, which is worth knowing the next time it appears somewhere else.

What the error actually means

TLS verification is a walk. Your client takes the certificate the server presented (the leaf) and tries to build a path from it up to a certificate it already trusts (a root, shipped in the system trust store). Each step is proved by a signature: the leaf is signed by an intermediate, the intermediate by the root.

certificate signed by unknown authority means that walk ran out of road. The client found a certificate whose issuer it could not produce, so it never reached a trusted anchor.

The wording is misleading. It rarely means the authority is genuinely unknown — DigiCert and Let's Encrypt are in every trust store on earth. It almost always means a link in the middle is missing, so the client never got far enough to recognise the authority it does trust.

That distinction is the whole debugging strategy: don't start by suspecting the root. Start by counting the links.

Why your browser disagrees

This is the part that misleads people, and it is worth understanding precisely, because it explains the entire "works in Chrome, fails in Docker" class of report.

A correctly configured server sends the leaf and every intermediate needed to chain up to a root. It does not send the root — the client already has that.

When the server forgets the intermediate, clients diverge:

Browsers paper over it. They cache intermediates they have seen before, and they follow the Authority Information Access (AIA) extension in the leaf, which contains a URL for the issuing certificate. Chrome will quietly fetch the missing intermediate over HTTP and complete the chain. You see a padlock.
Go does not. crypto/x509 performs no AIA fetching, by design — a verification step that makes its own outbound HTTP request is an availability and privacy problem. If the intermediate is not presented, verification fails.
curl and OpenSSL generally do not either, so they fail the same way.

Because Docker, Kubernetes, Terraform, most CI runners and a large share of backend services are written in Go, "it works in the browser" is not evidence the server is fine. It is frequently evidence of exactly this bug. The browser is being generous; Go is telling you the truth.

Diagnose it in one command

Before changing anything, count the certificates the server actually sends:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null \
  | grep -c "BEGIN CERTIFICATE"

Read the number:

1 — only the leaf. The intermediate is missing. This is your bug, and it is the single most common cause.
2 or more — a chain is being sent. The problem is elsewhere; keep reading past cause 1.

-servername matters. Without it OpenSSL omits SNI, and a host serving multiple sites will hand you its default certificate rather than the one you are debugging — which sends you chasing a mismatch that does not exist in production.

To see what each certificate in that chain actually is — who issued it, what it covers, whether the order is right — paste the whole -showcerts output into the Certificate Decoder & Chain Checker. It reads every X.509 field, reports the chain order with the reason it was wrong, and gives a per-link signature verdict, so you can see which link fails to connect rather than inferring it. It runs entirely in your browser, which matters when the chain belongs to an internal host.

The four causes, ranked

1. The server is missing its intermediate

By a wide margin the most common. The admin installed only the leaf, or concatenated the files in the wrong order, or a renewal replaced a full-chain file with a leaf-only one.

Fix it on the server, not in your client. Nearly every issuer ships a "fullchain" file for exactly this reason:

# nginx — ssl_certificate must be leaf + intermediates, in that order
ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;   # not cert.pem
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

The ordering rule is: leaf first, then each issuer in turn, root optional and usually omitted. A chain assembled in reverse order fails on strict clients even though every certificate present is valid — which is precisely the failure the chain checker names for you rather than leaving you to eyeball PEM blocks.

After reloading, re-run the grep -c above and confirm the count went up.

2. Your container has no CA bundle at all

If the failure only happens inside a container, suspect the image before the network. Minimal base images ship no trust store, so every public certificate is signed by an unknown authority — the client has no roots whatsoever.

# Alpine — no CA bundle by default
RUN apk add --no-cache ca-certificates

# Debian/Ubuntu slim — usually present, but reinstall if it was stripped
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*

For scratch or distroless images, copy the bundle in from a build stage:

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

The tell for this cause: the same binary works on the host and fails in the container, against any HTTPS host — not just one.

3. A corporate proxy is re-signing traffic

On a corporate network, an inspecting proxy terminates TLS and re-signs it with an internal CA. That CA is legitimately unknown to your container, which never received it.

The tell: the leaf's issuer is your employer, not a public CA. Decode the certificate and read the issuer — if it says something like Issuer: CN=Acme Corp Root CA, this is your cause.

The fix is to install the corporate root into the image's trust store, not to disable verification:

COPY corp-root.crt /usr/local/share/ca-certificates/corp-root.crt
RUN update-ca-certificates

4. The certificate really is self-signed

Internal services, a local dev stack, a private registry. Here the error is correct: the certificate genuinely is not trusted by anyone, and nothing is misconfigured except your client's trust store.

This is by far the most common way people meet this error in practice, because it is what docker login and docker pull against a private registry produce on day one.

The Docker daemon does not use the system trust store for registries. It reads a per-registry directory, and the path must match the registry hostname and port exactly:

# note: directory name = exactly what you type after docker login,
# including the port. registry.internal and registry.internal:5000
# are two different directories.
sudo mkdir -p /etc/docker/certs.d/registry.internal:5000
sudo cp ca.crt /etc/docker/certs.d/registry.internal:5000/ca.crt
sudo systemctl restart docker

Three details that account for most of the "I did that and it still fails" follow-ups:

The file must be named ca.crt. Not ca.pem, not registry.crt. The daemon looks for that name (plus *.cert / *.key for client auth).
The port is part of the directory name. A registry on :5000 needs registry.internal:5000, and a registry on 443 needs the bare hostname.
Restart the daemon. It reads that directory at startup, so the file appearing is not enough.

For Kubernetes pulling the same registry, this path does not apply — the kubelet uses the node's system trust store, so install the CA there with update-ca-certificates and restart the kubelet. This is why an image can pull fine on your laptop and fail with ImagePullBackOff on the cluster.

The fix that is not a fix

Every one of these has a tempting one-liner: InsecureSkipVerify: true in Go, -k in curl, insecure_skip_verify in Terraform.

These do not fix anything. They disable the check that was correctly telling you the chain is broken, and they disable it for every connection that code path makes, permanently, including against an attacker. The failure you are silencing is the one case where verification did its job.

If you need an escape hatch during local development, scope it to a specific CA rather than switching verification off:

pool, _ := x509.SystemCertPool()
pool.AppendCertsFromPEM(internalCA)          // trust one extra CA
cfg := &tls.Config{RootCAs: pool}            // still verifies everything

That trusts exactly what you decided to trust, and nothing else.

A short checklist

Count the certificates the server sends with openssl s_client -showcerts. One means a missing intermediate — fix the server.
If it fails only in a container, and against every host, install ca-certificates.
Read the leaf's issuer. If it is your employer, install the corporate root.
If it is genuinely self-signed, add that CA — do not skip verification.

The through-line: certificate signed by unknown authority is nearly always a chain-assembly problem, and your browser's willingness to repair it silently is what makes it look like a client bug. Count the links first, and the cause usually names itself.

OOMKilled and exit code 137: why Kubernetes killed your pod, and what to actually change

opscanopy — Mon, 31 Aug 2026 17:20:22 +0000

A pod is restarting and you can't see why. The logs end mid-request with nothing that looks like a crash. Then you describe it:

    State:          Running
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 29 Aug 2026 09:14:02 +0000
      Finished:     Mon, 29 Aug 2026 09:41:37 +0000
    Restart Count:  6

Two things are worth knowing immediately. Your application did not crash — it was killed from outside, by the Linux kernel, with SIGKILL. And there is no graceful shutdown and no stack trace, because SIGKILL cannot be caught. That silence in the logs is the expected symptom, not a missing clue.

What 137 actually means

137 is 128 + 9. The 128 marks "terminated by signal", and 9 is SIGKILL. So exit code 137 means something sent SIGKILL — most often the kernel's OOM killer, but not always.

The distinction that matters:

Reason: OOMKilled — the container exceeded its memory limit. The kernel's cgroup OOM killer terminated it.
Reason: Error, exit code 137 — something else sent SIGKILL. Usually a terminationGracePeriodSeconds that expired: Kubernetes sent SIGTERM, the process ignored it or shut down too slowly, and Kubernetes escalated to SIGKILL.

Read the Reason field before assuming memory. The second case is a shutdown-handling bug and no amount of extra memory will fix it.

kubectl describe pod <pod> | grep -A4 "Last State"

Confirm it, don't infer it

Restart Count going up with OOMKilled in Last State is the confirmation. To see the pattern rather than a single instance:

# Which containers have OOMed, across the namespace
kubectl get pods -o json \
  | jq -r '.items[] | .metadata.name as $p
      | .status.containerStatuses[]?
      | select(.lastState.terminated.reason=="OOMKilled")
      | "\($p)\t\(.name)\trestarts=\(.restartCount)"'

Also check events, which sometimes name the node-level pressure that triggered it:

kubectl get events --field-selector reason=OOMKilling -A

Why raising the limit is often the wrong first move

The reflex is to double limits.memory and move on. Sometimes that is correct — the limit was simply set below what the workload legitimately needs. Often it isn't, and it converts a fast, obvious failure into a slow, expensive one.

Ask which of these you have:

A limit that was always too low. The container OOMs quickly and consistently, often within seconds of starting, at roughly the same memory figure every time. Raising the limit is the right fix.

A leak. Memory climbs steadily over minutes or hours until it hits the ceiling, and the time-to-OOM is proportional to the limit. Doubling the limit doubles the time between restarts and fixes nothing. The tell is that Started and Finished in Last State are far apart, and the interval grows when you raise the limit.

A runtime that ignores the cgroup. The JVM before it was container-aware, Node's default old-space, Go with no GOMEMLIMIT — the runtime sizes its heap against the node's total memory, not the container's limit, then confidently allocates past the ceiling. This is extremely common and looks exactly like a leak from the outside.

For the JVM, use the container-aware flags rather than a fixed -Xmx:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75.0"

For Go 1.19+, tell the runtime the ceiling so the garbage collector works against the real budget:

env:
  - name: GOMEMLIMIT
    valueFrom:
      resourceFieldRef:
        resource: limits.memory

resourceFieldRef reads the container's own limit, so the two can never drift apart.

The requests-versus-limits mistake behind most of it

Memory and CPU behave completely differently under pressure, and conflating them causes a lot of unnecessary OOMKills.

CPU is compressible. Exceed the CPU limit and you get throttled — slower, but alive.
Memory is not. Exceed the memory limit and you are killed. There is no "a bit slower" state.

That asymmetry has a direct consequence: CPU limits are optional and often harmful; memory limits are essential.

The second half is the Quality of Service class, which decides who gets killed first when the node runs out of memory:

requests vs limits	QoS class	Eviction order
Not set	`BestEffort`	Killed first
requests < limits	`Burstable`	Middle
requests == limits	`Guaranteed`	Killed last

A pod with no memory request is BestEffort and is the first thing evicted under node pressure — even if it was using very little. If a workload matters, give it a request. For anything latency-sensitive, set request equal to limit and get Guaranteed.

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"     # equal to request -> Guaranteed

Sizing it, and checking what it costs

Base the number on observation, not a guess. Use the working set over a representative window — including whatever your peak actually is:

max_over_time(
  container_memory_working_set_bytes{pod=~"my-app-.*", container!=""}[7d]
)

Take that peak and add headroom — commonly 20–30%, more if traffic is spiky. Note that container_memory_working_set_bytes is the metric the OOM killer effectively acts on, which is why it is the right one here rather than container_memory_usage_bytes (which includes reclaimable cache and will overstate your need).

Once you have a per-container number, the arithmetic across replicas is where mistakes creep back in — millicores and Mi/Gi units, multiplied by replica counts, summed across containers including sidecars. The Kubernetes Resource Calculator totals CPU and memory requests and limits across containers and replicas and converts the units, so you can see what a deployment actually reserves on the cluster before you apply it.

To be clear about what that does and does not do: it will not tell you why a container OOMed — that comes from the metric above and the runtime's own behaviour. What it answers is the question that follows, which is whether the limit you just chose, multiplied by every replica, still fits the nodes you have.

A working order

kubectl describe pod and read Reason. OOMKilled is memory; Error with 137 is a shutdown-grace problem instead.
Check whether time-to-OOM scales with the limit. If it does, you have a leak or a runtime ignoring the cgroup — fix that, don't raise the ceiling.
If the runtime is the JVM, Node or Go, make it container-aware before touching limits.
Size from max_over_time(container_memory_working_set_bytes[7d]) plus headroom.
Set a memory request as well as a limit, so the pod isn't BestEffort and first to be evicted.
Total it across replicas and confirm it still fits the node pool.

The recurring theme: exit code 137 is a report that something exceeded a boundary, not a description of the bug. The useful question is never "how much memory should I add" but "does this workload's memory grow with time, and does its runtime know what its ceiling is".

failed to solve: process \"/bin/sh -c ...\" did not complete successfully: exit code: 1

opscanopy — Mon, 31 Aug 2026 17:19:48 +0000

Your build stops with a wall of text ending in this:

------
 > [4/8] RUN apt-get install -y curl:
------
Dockerfile:12
--------------------
  10 |     WORKDIR /app
  11 |     COPY . .
  12 | >>> RUN apt-get install -y curl
  13 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get install -y curl" did not complete successfully: exit code: 1

The first thing to understand is that this error is not the problem. It is BuildKit reporting that a command you asked it to run exited non-zero. The actual cause is in that command's output, which BuildKit has usually just truncated away.

So there are two jobs here: get the real error back, then fix it. This post covers both, plus the four structural causes that produce this message far more often than anything else.

Reading the error properly

Three parts matter:

[4/8] — which step failed, and how far the build got. Steps 1–3 succeeded.
Dockerfile:12 with the >>> marker — the exact line. BuildKit is precise about this; trust it.
exit code: 1 — the status your command returned. This is the command's exit code, not Docker's.

That last one is worth reading carefully, because some exit codes are diagnostic on their own:

Exit code	Usually means
`1`	Generic failure — the command itself will say why
`100`	`apt-get` could not install (unresolved package, stale index)
`126`	Found the file but could not execute it — permissions, or CRLF line endings
`127`	Command not found — not installed, or not on `PATH` yet
`137`	Killed by SIGKILL — almost always the builder running out of memory

127 and 137 in particular save you a lot of time. 127 means you are calling something that does not exist in that layer. 137 means nothing is wrong with your command at all — the build ran out of RAM, and you should raise the memory available to the builder rather than edit the Dockerfile.

Get the real output back

BuildKit collapses successful output and truncates failed output. To see the whole thing:

docker build --progress=plain --no-cache .

--progress=plain disables the collapsing TUI and streams every line. --no-cache matters more than it looks: with a warm cache, BuildKit may not re-run the earlier steps that produced the actual error message, so you get the failure without the context.

If the output is long, capture it:

docker build --progress=plain --no-cache . 2>&1 | tee build.log

Nine times out of ten the answer is now sitting in build.log, several lines above the failed to solve line, in the failing command's own words.

Debug the layer interactively

When the output still isn't enough, get a shell in the last layer that did build. Every successful step leaves an image behind:

# grab the image id of the last good step from --progress=plain output
docker run --rm -it <last-good-image-id> sh

Then run the failing command by hand. This resolves most "but it works on my machine" cases immediately, because you are now standing inside the exact filesystem the command was given — not your shell, with your PATH, your installed packages and your environment variables.

The four structural causes

Most instances of this error are not exotic. They are the same handful of Dockerfile mistakes.

1. `apt-get install` without a fresh index

The single most common cause, and the most confusing, because it works one day and fails the next.

# Broken — and it will pass CI for weeks before it doesn't
RUN apt-get update
RUN apt-get install -y curl

These are two layers. Docker caches the apt-get update layer, and keeps using it long after the package index it downloaded has gone stale upstream. Eventually install asks for a package version that no longer exists at the URL the cached index recorded, and returns exit code 100.

# Correct — one layer, so the index is always as fresh as the install
RUN apt-get update && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

The rm -rf is not just hygiene: leaving the lists in the image guarantees a future reader will copy this pattern with a stale cache again.

2. The command needs a shell feature it doesn't have

Exec form does not run a shell:

RUN ["apt-get", "install", "-y", "curl && echo done"]   # && is a literal argument

There is no shell here to interpret &&, |, >, $VAR or *. If you need any of those, use shell form (RUN apt-get install -y curl && echo done).

The same trap bites CMD and ENTRYPOINT in exec form, and on Alpine there is a second layer to it: the default shell is /bin/sh, not bash. A script using [[, arrays or source will fail with exit code 2 or 127 even though it is a perfectly good bash script.

3. `COPY` did not copy what you think

COPY . .
RUN ./scripts/build.sh      # exit code 127: not found

Two usual reasons. First, .dockerignore excluded the file — it applies to the build context before COPY ever sees it, so the file silently is not there. Second, the file is there but not executable, which gives you 126 rather than 127.

Check what actually landed:

RUN ls -la ./scripts    # temporary, but it settles the question in one build

On Windows there is a third: a script committed with CRLF line endings fails with exec format error or a confusing not found, because the kernel reads the shebang as /bin/sh\r.

4. Architecture mismatch

On Apple Silicon building for a linux/amd64 target, a binary downloaded at build time may be the wrong architecture, producing exec format error and a non-zero exit.

docker build --platform linux/amd64 .

Make the platform explicit rather than relying on the daemon's default, especially in a CI pipeline whose runners may not match your laptop.

Catching these before the build runs

Causes 1–3 are all visible in the Dockerfile itself, without running anything. That is what a linter is for, and it is worth being precise about the division of labour:

A linter cannot tell you why your specific apt-get install returned 1 — that depends on the package index at that moment. What it can do is flag the structural patterns that make this error likely and recurrent: the split apt-get update / install cache trap, broken JSON exec forms, COPY ordering that busts the cache on every build, and secrets baked into image history.

The Dockerfile Linter runs seventeen such rules over a real parse of the file and returns line-numbered findings with a fix for each, plus an explicit list of what it deliberately does not flag. Paste the Dockerfile, fix what it names, and the recurring instances of this error stop happening. The one-off ones you still debug with --progress=plain.

The order to work in

Re-run with --progress=plain --no-cache and read the failing command's own output. This alone resolves most cases.
Check the exit code against the table above — 127 and 137 in particular mean something specific.
Still stuck? Shell into the last good layer and run the command by hand.
Then lint the Dockerfile, so the structural version of this failure does not come back next month.

The message itself is generic by design: BuildKit does not know anything about apt-get, only that a process it started returned 1. Once you stop reading failed to solve as the error and start treating it as a pointer to a line number and an exit code, it becomes one of the easier Docker failures to work through.

Linux for DevOps Engineers

opscanopy — Tue, 25 Aug 2026 16:03:02 +0000

Linux is the operating system under every Docker container, every Kubernetes node, and the majority of cloud VMs. Before you can deploy, debug, or automate anything in a modern infrastructure stack, you need to be comfortable at the shell. This guide takes you from the absolute fundamentals — how Linux thinks about files and processes — through the tools DevOps engineers reach for daily: permissions, grep/awk/sed, systemd, SSH, package managers, network commands, and production-grade bash scripting. Follow the Linux roadmap to see how these topics connect into a full learning path.

Introduction to Linux

Imagine you bought a brand new car. You see the dashboard, the steering wheel, the leather seats, the touchscreen infotainment system. But what actually moves the car? The engine. You rarely open the hood, you never see it during your daily drive, yet without it, nothing works. Linux is exactly that engine for the entire modern internet. Every time you scroll Instagram, order from Uber Eats, watch Netflix, or tap your phone to pay — somewhere in a datacenter, a Linux server is doing the heavy lifting. You never see it. You never think about it. But it's powering 96% of the world's digital infrastructure in 2026.

As an experienced frontend developer, you've been writing code that runs on top of this engine. You've been the person designing the dashboard. Now you're about to learn what's under the hood — and that's exactly what the highest-paid DevOps engineers know cold. The transition from frontend to DevOps is one of the most valuable career moves in tech today, because you already understand how applications behave; now you'll learn how they live, breathe, scale, and recover at the infrastructure level.

Real world: Think of Linux like the global air-traffic network. You don't see the radar, the control towers, or the routing systems when you fly — but they're the reason tens of thousands of flights move millions of passengers safely every single day. Linux is the air-traffic network of the internet. AWS, Azure, GCP, Netflix, Google, Facebook, WhatsApp, Spotify, Amazon — they all run on Linux.

Why Linux Dominates DevOps in 2026

If you walked into any tech company in a major tech hub today and asked their SRE team what operating system runs production, the answer is almost universally Linux. Here's why this domination is now absolute:

~96% of public cloud servers run Linux — AWS EC2, Azure VMs, Google Compute Engine. Even Microsoft, which makes Windows, runs the majority of Azure on Linux.
100% of containers use the Linux kernel — Docker, Podman, containerd all rely on Linux namespaces and cgroups. Even "Windows containers" running on Docker Desktop spin up a hidden Linux VM underneath.
Every single Kubernetes node is Linux — kubelet, kube-proxy, CNI plugins, the entire control plane is designed for Linux first.
All major CI/CD runners are Linux — GitHub Actions, GitLab Runners, Jenkins agents, ArgoCD, CircleCI default to Linux.
Every observability stack runs on Linux — Prometheus, Grafana, Loki, Elastic, Datadog agents, OpenTelemetry collectors.
Edge computing and IoT — Raspberry Pi, AWS Greengrass, Azure IoT Edge, all Linux.

For a React developer transitioning to DevOps, this means one undeniable truth: you cannot avoid Linux. It's not optional anymore. The good news? You already have Ubuntu 24.04 installed, which is the gold standard for DevOps learning in 2026.

Linux vs Windows vs macOS: The Honest Comparison

Feature	Linux	Windows	macOS
Cost	Free (most distros)	~$140 per license	Bundled with Apple hardware (premium)
Customization	Unlimited — change anything	Limited to UI/registry	Restricted — Apple's way only
CLI Power	Native, world-class (bash, zsh)	Catching up (PowerShell, WSL2)	Strong (Unix-based, zsh default)
Server Use	~96% of internet servers	~3% (legacy enterprise AD)	Negligible (not a server OS)
Security	Excellent (permissions, SELinux)	Improving but still targeted	Strong (sandboxed by default)
Package Mgmt	apt, dnf, pacman, snap, flatpak	winget, choco, MSI installers	brew (community), App Store
Source Code	100% open source (GPL)	Proprietary (closed)	XNU kernel partially open, OS closed
DevOps Fit	The de-facto standard	Niche (Windows containers)	Good for local dev only

The Linux Distro Universe

One of the most confusing things for beginners is the sheer number of Linux distributions ("distros"). They're all Linux at the core (same kernel), but they package the system differently — different installers, different package managers, different defaults, different philosophies. Here's the map you need:

Distro	Base	Package Mgr	Best For	DevOps Use
Ubuntu	Debian	apt	Development, learning, desktops	Dev laptops, CI runners, EC2 default
RHEL	Fedora	dnf/yum	Enterprise production	Banks, telco, govt, regulated industries
CentOS Stream	RHEL upstream	dnf	Testing RHEL features early	Rolling preview of RHEL
Debian	Independent	apt	Stable servers, rock-solid uptime	Long-running production servers
Alpine	Independent (musl)	apk	Containers — only 5MB base!	99% of Docker base images
Rocky/AlmaLinux	RHEL clones	dnf	Free RHEL alternative	Post-CentOS production replacement
Arch Linux	Independent	pacman	Advanced users, customization	Personal workstations (rarely prod)
Amazon Linux 2023	Fedora-based	dnf	AWS-optimized workloads	Default EC2/ECS/Lambda runtime

Note: Which distro do real DevOps engineers actually use?

Dev laptop: Ubuntu 24.04 LTS or macOS

Production servers: RHEL, Amazon Linux 2023, Rocky Linux, or Ubuntu Server LTS

Docker base images: Alpine (tiny, fast), or distroless/Debian-slim

Kubernetes nodes: Ubuntu, Bottlerocket (AWS), Flatcar, or Talos Linux

Master Ubuntu first because apt-based skills transfer 80% to Debian and Kali. Then learn dnf for RHEL/Rocky/Amazon Linux. That covers ~95% of real-world DevOps scenarios.

Kernel vs Operating System

This is the single most common interview blunder. Junior candidates confuse "kernel" and "OS" all the time.

The Linux Kernel is the core program — roughly 30+ million lines of C code — written originally by Linus Torvalds in 1991 when he was a 21-year-old student in Helsinki, Finland. The kernel is just the engine — it talks to the CPU, RAM, disks, network cards, and so on. By itself, the kernel cannot give you a usable computer.

An Operating System is the kernel plus all the surrounding software needed to make it usable: the GNU utilities (ls, cat, grep, sed, awk), the shell (bash, zsh), system libraries (glibc), package managers (apt, dnf), and applications. So when you say "Linux," what you usually mean is "GNU/Linux."

$ uname -r
6.8.0-31-generic
$ cat /etc/os-release | head -3
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
$ uname -a
Linux dev-laptop 6.8.0-31-generic #31-Ubuntu SMP x86_64 GNU/Linux

Interview tip: If an interviewer asks "What is Linux?" — never just say "an operating system." Say: "Linux is technically the kernel — a monolithic Unix-like kernel created by Linus Torvalds in 1991. When people say 'Linux,' they usually mean GNU/Linux, which is the kernel plus the GNU userland utilities, a shell, system libraries like glibc, and applications, all packaged by a distribution like Ubuntu or RHEL."

Open Source and the GPL

Linux is released under the GNU General Public License version 2 (GPLv2). The GPL guarantees four fundamental freedoms (numbered 0 through 3, because programmers count from zero):

Freedom 0: Run the program for any purpose, without restriction.
Freedom 1: Study how the program works and modify it. Source code access is required.
Freedom 2: Redistribute copies to help others.
Freedom 3: Distribute your modified versions so the community benefits.

Why do massive corporations like Google, Meta, Amazon, and Microsoft pour billions into a "free" project? Because open-source Linux gives them zero licensing cost at scale, total customizability, security through transparency, vendor independence, and faster innovation.

The DevOps Salary Reality (2026)

Level	Experience	US (USD)	Europe (EUR)	India (INR)	Typical Skills
Junior DevOps	0-2 years	$65k–100k	€45k–70k	₹6–12 LPA	Linux basics, Git, Docker, basic CI/CD, one cloud
Mid DevOps / SRE	2-5 years	$100k–150k	€70k–105k	₹15–25 LPA	Kubernetes, Terraform, Prometheus, scripting, multi-cloud
Senior DevOps / SRE	5-8 years	$150k–200k	€105k–145k	₹30–50 LPA	Architecture, security, cost optimization, on-call leadership
Lead / Staff / Principal	8-12+ years	$190k–270k	€140k–190k	₹50–80+ LPA	Platform engineering, multi-team coordination, strategy
DevOps Manager / Director	10+ years	$200k–350k+	€150k–230k	₹70 LPA–1.5 Cr+	People management, business alignment, P&L

Figures are rough 2026 market signals, not guarantees. They swing widely with company size, city, cost of living, and equity — treat them as the relative gaps between levels, not absolute targets.

Linux Architecture

Now that you know what Linux is, let's open the hood and look at how it actually works. Linux follows a layered architecture — each layer has a specific job, and they communicate through well-defined interfaces. If you understand this architecture clearly, you'll debug production issues 10x faster than engineers who only know commands by rote.

The Kernel: The Brain of the System

The Linux kernel is the lowest software layer above the hardware. It has six core responsibilities that you must internalize:

Process Management: Creating, scheduling, pausing, killing processes. The kernel decides which process gets CPU time. Tools like ps, top, htop, kill all interact with this subsystem.
Memory Management: Allocating RAM to processes, managing virtual memory, paging, swapping. The OOM (Out of Memory) killer lives here.
Device Drivers: Translating generic OS calls into hardware-specific signals — GPU, NIC, SSD, USB, Bluetooth all speak through drivers loaded as kernel modules.
Filesystem Management: Reading, writing, organizing data on disk through filesystems like ext4, xfs, btrfs, zfs, overlayfs (used by Docker).
Networking: The entire TCP/IP stack, sockets, packet routing, iptables/nftables — all kernel code.
Security & Permissions: User IDs, group IDs, file permissions, capabilities, SELinux/AppArmor, seccomp filters (used by Docker for sandboxing).

Shells: Your Window to the Kernel

You don't talk to the kernel directly — you talk to a shell, and the shell talks to the kernel. A shell is just a program that reads commands, interprets them, and asks the kernel to do work via system calls.

Shell	Default On	Strengths	Weaknesses	DevOps Verdict
Bash	Ubuntu, RHEL, Debian, Alpine (ash)	Universal, scripting, POSIX compliance	Plain UX, no autosuggestions	Must master — every server has it
Zsh	macOS Catalina+	Themes (oh-my-zsh), autosuggestions, plugins	Slightly heavier, not on servers	Great for daily dev laptop use
Fish	None default	Friendly UX, autosuggest out of the box	Non-POSIX — scripts don't transfer	Personal use only, never scripts
Dash	Ubuntu's /bin/sh	Tiny, fast, POSIX-strict	Minimal features	Used by Ubuntu boot scripts
PowerShell	Windows	Object pipeline, cross-platform now	Heavy, alien to Unix folk	Skip unless Windows-shop

Tip: Use Zsh + oh-my-zsh on your Ubuntu dev laptop for daily comfort, but always write your shell scripts in Bash (with #!/usr/bin/env bash) so they run on every server you'll ever touch. Production servers don't have your fancy zsh setup.

System Libraries and Utilities

Between user programs and the kernel sit two crucial layers. System Libraries — the most important is glibc (GNU C Library). When your Node.js process needs to open a file, it calls a C function like fopen() in glibc, which translates to the kernel's open() system call. Alpine Linux uses musl libc instead, which is why Alpine images are smaller but sometimes have subtle compatibility quirks.

System Utilities — the GNU coreutils package gives you ls, cp, mv, rm, cat, echo, mkdir, chmod, chown, and ~100 more. Then there's util-linux, findutils, grep, sed, awk, tar, gzip — collectively known as "the Unix toolbox."

User Space vs Kernel Space

Modern CPUs have two privilege modes: kernel mode (ring 0 on x86) where code can do anything, and user mode (ring 3) where code is sandboxed and must ask the kernel (via system calls) to do anything dangerous. Your React app, Node.js, Python scripts, even databases like PostgreSQL — all run in user space. Only the kernel and its drivers run in kernel space. You can literally see system calls with strace:

$ strace -c ls /tmp
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 24.31    0.000312          26        12           openat
 18.13    0.000233          19        12           read
 11.45    0.000147          12        12           close

The Linux Boot Process

When you press the power button on a Linux machine (or click "Start instance" on an EC2 console), a tightly choreographed sequence brings the system to life. Understanding this is critical because most production outages happen during boot.

BIOS / UEFI: The first code that runs lives in firmware on the motherboard. It initializes the CPU and looks for a bootable device. On EC2, this is the Nitro hypervisor's firmware.
POST (Power-On Self-Test): Hardware checks — RAM, keyboard, basic devices.
MBR / GPT Boot Sector: Firmware reads the first sector of the boot disk to find the bootloader.
GRUB (GRand Unified Bootloader): Displays the boot menu, lets you pick a kernel, and loads the kernel image and initramfs into memory. Config lives in /boot/grub/grub.cfg.
Kernel + initramfs: The kernel unpacks itself, the initramfs provides essential drivers to mount the real root filesystem. Kernel mounts / and starts PID 1.
systemd (PID 1): Modern init system. Reads /etc/systemd/system/, starts services in dependency order, mounts filesystems from /etc/fstab, brings up the network.
multi-user.target / graphical.target: systemd's "runlevel" — on servers it stops at multi-user.target (text login).
Login prompt: getty on TTYs, sshd for remote, GDM for graphical.

Interview tip: "Walk me through what happens when you press the power button on a Linux server until you get a login prompt." This is asked in literally every senior DevOps interview. Memorize the 8-step sequence above. Bonus points if you mention systemd targets, initramfs, and the role of /etc/fstab.

DevOps Tie-Ins: Why Boot Knowledge Pays

EC2 boot failures: When an EC2 instance won't start, AWS shows you the serial console output — you see exactly which boot step failed.
GRUB recovery: Kernel update broke boot? You boot from a rescue ISO, chroot into the broken system, run grub-install and update-grub.
systemd unit debugging: A service won't start at boot? systemctl status, journalctl -xeu service-name, check dependencies with systemctl list-dependencies.
Custom AMI building: Packer images, custom EKS node AMIs, Bottlerocket — all require understanding what boots and when.
Container startup: Containers skip BIOS/GRUB/kernel — they start at "PID 1 in a namespace." tini exists exactly because containers need a proper PID 1.

Linux Filesystem Hierarchy

Linux organises everything into a single tree rooted at /. Unlike Windows with drive letters, there is no C:\ — everything is a file or a directory beneath /. Understanding where things live stops you from wasting time searching for config files.

The Linux filesystem is a single tree rooted at /; each top-level directory has a defined purpose per the FHS standard.

Key Directories

Path	What lives here
`/bin`, `/usr/bin`	Essential user binaries (`ls`, `cp`, `bash`)
`/sbin`, `/usr/sbin`	System administration binaries (`fdisk`, `iptables`)
`/etc`	System-wide configuration files — everything in plain text
`/home`	User home directories (`/home/pushkar`)
`/root`	Root user's home
`/var`	Variable data: logs (`/var/log`), mail, caches, spool
`/tmp`	Temporary files; cleared on reboot
`/proc`	Virtual filesystem exposing kernel and process state
`/sys`	Virtual filesystem for kernel subsystems (devices, drivers)
`/dev`	Device files (`/dev/sda`, `/dev/null`, `/dev/random`)
`/boot`	Kernel image, initrd, GRUB config
`/opt`	Optional third-party software
`/usr/local`	Locally compiled software (takes precedence over `/usr`)
`/run`	Runtime data (PIDs, sockets) — tmpfs, cleared on boot
`/mnt`, `/media`	Mount points for temporary / removable filesystems

Navigating the Tree

pwd                    # print working directory
ls -la /etc            # long listing including hidden files
ls -lh /var/log        # human-readable sizes
cd /var/log            # change directory
cd -                   # go back to previous directory
cd ~                   # go home
tree -L 2 /etc         # visual tree, depth 2 (install with apt)

Inodes and Hard Links

Every file is represented by an inode — a data structure storing metadata (owner, permissions, timestamps, block pointers) but not the filename. Directory entries are simply (name → inode number) mappings. This is why:

Hard links are just extra directory entries pointing to the same inode. Delete one, the data survives until the last link is removed.
Symbolic links (ln -s) store a path string. They can cross filesystem boundaries; hard links cannot.
df -i shows inode usage — a filesystem can run out of inodes before running out of blocks if you create millions of tiny files.

ls -li /etc/hostname        # shows inode number in first column
stat /etc/hostname          # full inode metadata dump
ln /etc/hostname /tmp/hn    # hard link (same inode)
ln -s /etc/hostname /tmp/hn_sym   # symbolic link

Tip: /proc and /sys contain no data on disk — they are windows into kernel memory. cat /proc/cpuinfo reads CPU registers; cat /proc/$(pgrep nginx)/status reads a live process's memory map. Learn to explore them freely.

File Directory Operations

Confident file manipulation — copying, moving, finding, archiving — is the foundation of every shell workflow.

Essential Commands

# Create
touch notes.txt              # create empty file / update mtime
mkdir -p projects/web/src    # create directory tree in one shot

# Copy, Move, Delete
cp -r /etc/nginx /tmp/nginx-bak    # recursive copy
cp -p file dest/                   # preserve permissions + timestamps
mv old_name new_name               # rename (or move)
rm -rf /tmp/scratch                # recursive force delete — no undo!

# View
cat /etc/hostname           # dump whole file
less /var/log/syslog        # paged viewer (q to quit, / to search)
head -20 /var/log/auth.log  # first 20 lines
tail -50 /var/log/syslog    # last 50 lines
tail -f /var/log/nginx/access.log  # follow (real-time stream)

Finding Files

# find — the workhorse
find /var/log -name "*.log" -mtime -1          # modified in last 24 h
find /home -type f -size +100M                 # regular files over 100 MB
find /etc -name "*.conf" -exec grep -l "port" {} \;   # grep inside results
find /tmp -type f -newer /etc/hostname         # newer than reference file
find . -name "*.pyc" -delete                   # find and delete in one pass

# locate — index-based, faster but stale
sudo updatedb            # rebuild index
locate nginx.conf        # instant search

# which / type — find an executable
which python3
type ls                  # shows if alias, function, or binary

Archives and Compression

# tar — tape archive (most common)
tar -czf backup.tar.gz /etc/nginx      # create gzipped archive
tar -tzf backup.tar.gz                 # list contents without extracting
tar -xzf backup.tar.gz -C /tmp/       # extract to /tmp

# gzip / bzip2 / xz
gzip largefile.log         # compresses in-place → largefile.log.gz
gunzip largefile.log.gz    # decompress
zcat largefile.log.gz      # read without decompressing

# zip (cross-platform)
zip -r archive.zip dir/
unzip archive.zip -d /tmp/out

Tip: Remember tar flags as Create/eX*tract **zgzip **f*ile: czf to create, xzf to extract. For bzip2 use j instead of z. For xz use J.

File Viewing and Editing

As a DevOps engineer, 80% of your day will be spent reading logs, editing config files, and piecing together what went wrong on a server at 2 AM. Coming from React, you've used VS Code — but on a remote production server with no GUI, you have only the terminal. This section makes you fast at viewing files, slicing log data, and editing config files directly on Linux boxes.

Real world: On a production Ubuntu server, when nginx is throwing 502s, you SSH in and immediately run tail -f /var/log/nginx/error.log. Then you edit /etc/nginx/nginx.conf with vim, run nginx -t to validate, and reload. No VS Code, no GUI — just your terminal skills.

cat — Concatenate and View

cat dumps file contents to stdout. Great for small files; avoid on huge logs (use less instead).

$ cat /etc/hostname
$ cat -n /etc/nginx/nginx.conf       # -n prefixes every line with its line number
$ cat -A script.sh                   # -A shows hidden chars: $ for newline, ^I for tab, ^M for CRLF
$ cat file1.txt file2.txt > combined.txt
$ cat ~/.ssh/id_rsa.pub | ssh user@server 'cat >> ~/.ssh/authorized_keys'

Tip: If a bash script fails with weird "command not found" errors and you wrote it on Windows, run cat -A script.sh. If you see ^M$ at end of lines, that's CRLF. Fix with tr -d '\r' < script.sh > clean.sh or dos2unix script.sh.

tac — Reverse cat

tac prints lines in reverse order (last line first). Useful for logs when you want the newest entries on top.

$ tac /var/log/syslog | head -20     # 20 most recent syslog entries, newest first
$ tac access.log | grep "POST /api/login" | head -5

less and more — The Pagers

less is the modern, more powerful pager. Rule: always use less.

$ less /var/log/nginx/access.log     # file is NOT loaded fully into memory — works on huge logs
$ less +F /var/log/syslog            # open directly in follow mode
$ journalctl -u nginx | less

Key	Action
`q`	Quit
`/pattern`	Search forward
`?pattern`	Search backward
`n` / `N`	Next / previous match
`g` / `G`	First / last line
`Space` / `b`	Next / previous page
`F`	Follow mode (like tail -f) — Ctrl+C to exit
`&pattern`	Show ONLY lines matching pattern (filter)

head — First N Lines

$ head /etc/passwd                   # first 10 lines (default)
$ head -n 20 access.log
$ head -c 100 /dev/urandom | base64  # grab first 100 BYTES of random data
$ head -n -5 file.txt                # everything EXCEPT last 5 lines

tail — Last N Lines and Follow Mode

If there's ONE command you must master for DevOps, it's tail -f.

$ tail /var/log/syslog               # last 10 lines
$ tail -n 50 /var/log/nginx/error.log
$ tail -f /var/log/nginx/access.log  # FOLLOW: stream new lines live. Ctrl+C to stop.
$ tail -F /var/log/app/app.log       # -F = follow + REOPEN on rotation
$ tail -f /var/log/nginx/access.log | grep "POST"
$ tail -f app.log error.log          # follow multiple files

Caution: tail -f vs tail -F: Production logs are rotated daily by logrotate. If you use tail -f and the file gets rotated, you'll silently stop seeing new logs. Always use tail -F for long-running tails on production. Capital F = forever.

wc — Word/Line/Char Count

$ wc -l /var/log/syslog              # count lines
$ wc -w README.md                    # count words
$ wc -c image.jpg                    # count bytes
$ grep "ERROR" app.log | wc -l       # how many errors today?

sort — Sort Lines

$ sort names.txt                     # alphabetical
$ sort -n sizes.txt                  # NUMERIC sort
$ sort -r names.txt                  # reverse
$ sort -h                            # human-readable: handles 1K, 2M, 3G
$ du -h /var/log/* | sort -h         # find which logs eat disk
$ sort -t: -k3 -n /etc/passwd        # sort users by numeric UID
$ sort -u file.txt                   # sort + dedupe

uniq — Remove/Count Duplicates

Critical: uniq only removes adjacent duplicates. You MUST sort first.

$ sort ips.txt | uniq -c             # -c prefixes each unique line with its count
$ sort ips.txt | uniq -d             # -d only shows lines appearing MORE than once
$ awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
# THE classic DevOps one-liner: top 10 IPs hitting your server

Real world: Spot a DDoS in 5 seconds: awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20. If one IP has 50,000 requests and the next has 200, you've got a bot. Block it with iptables or ufw.

cut, paste, tr

$ cut -d: -f1 /etc/passwd            # extract usernames (colon delimiter, field 1)
$ cut -d: -f1,3,7 /etc/passwd        # username, UID, shell
$ echo "2026-05-26T14:23:11" | cut -c1-10   # extract just the date

$ paste names.txt ages.txt           # merge line-by-line with TAB separator
$ paste -d, names.txt ages.txt > combined.csv

$ echo "Hello World" | tr A-Z a-z    # lowercase
$ tr -d '\r' < windows_script.sh > linux_script.sh   # fix CRLF line endings
$ echo "a   b     c" | tr -s ' '     # squeeze repeated spaces

tee — Write to File AND stdout

$ ls -la | tee listing.txt           # see output AND save it
$ echo "new line" | tee -a log.txt   # -a append
$ echo "127.0.0.1 myapp.local" | sudo tee -a /etc/hosts
$ make 2>&1 | tee build.log

Tip: sudo tee: Bash redirection (>) happens in the calling shell BEFORE sudo runs. So sudo echo "x" > /etc/protected fails with "Permission denied." Use echo "x" | sudo tee /etc/protected — tee runs under sudo and CAN write.

Text Editor: nano (beginner-friendly)

All shortcuts are visible at the bottom of the screen. Good for quick edits to /etc/hosts or a config file.

Shortcut	Action
`Ctrl+O`	Save (Write Out)
`Ctrl+X`	Exit
`Ctrl+W`	Where-is — search forward
`Ctrl+\`	Search & replace
`Ctrl+K` / `Ctrl+U`	Cut line / paste
`Alt+/` or `Ctrl+_`	Go to line number

Text Editor: vim — Deep Dive

Vim is on every Linux server by default. Mastering it makes you 10x faster than someone fumbling with nano.

The Four Modes: Normal (navigate, delete, copy — keys are COMMANDS), Insert (type text — enter via i/a/o, exit via Esc), Visual (select text via v/V/Ctrl+V), Command (run commands via :).

Interview tip: Most Asked Vim Question: "What are the vim modes and how do you exit vim?" Answer: 4 modes — Normal, Insert, Visual, Command. To exit: press Esc to ensure you're in Normal mode, then type :q (quit), :wq (write and quit), or :q! (force quit without saving).

Command	Action
`:w`	Save (write)
`:q`	Quit (errors if unsaved)
`:q!`	Force quit, discard changes
`:wq` or `ZZ`	Save and quit
`i` / `a`	Insert before / append after cursor
`o` / `O`	Open new line below / above
`h j k l`	Left, Down, Up, Right
`w` / `b`	Next / previous word
`0` / `$`	Start / end of line
`gg` / `G`	Top / bottom of file
`dd` / `yy` / `p`	Delete line / yank line / paste
`u` / `Ctrl+r`	Undo / redo
`/pattern` / `n`	Search / next match
`:%s/old/new/g`	Replace ALL "old" with "new" in file
`:%s/old/new/gc`	Same, but confirm each

A solid starter ~/.vimrc for DevOps work:

syntax on
set number
set relativenumber
set expandtab          " convert tabs to spaces (critical for YAML/Python)
set tabstop=4
set shiftwidth=4
set hlsearch
set incsearch
set ignorecase
set smartcase
filetype plugin indent on
" YAML-specific (2-space indent — important for k8s/Ansible)
autocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab

Tip: Run vimtutor in your terminal right now. It's a 30-minute interactive tutorial that ships with vim. By the end, you'll know enough to edit any file confidently.

Quick Reference Cheatsheet

Task	Command
View small file	`cat file`
View large file	`less file`
Stream live log	`tail -F /var/log/app.log`
First/last N lines	`head -n 20` / `tail -n 20`
Count lines	`wc -l file`
Sort numerically	`sort -n`
Unique with counts	`sort \
Extract column N	{% raw %}`cut -d: -f N` or `awk '{print $N}'`
Write to protected file	`... \
Power edit	{% raw %}`vim file`
Exit vim (most asked)	`Esc` then `:wq` (save) or `:q!` (discard)

File Permissions Ownership

Linux permissions control who can read, write, or execute every file. Getting this wrong is a security vulnerability; getting it right is foundational operations knowledge.

See also: SSH Secure Shell Critical for how permissions protect SSH keys.

Each file has nine permission bits split into three triplets — owner, group, others — each summed to an octal digit for use with chmod.

The Permission Bits

Every file has three permission triplets: owner (u), group (g), others (o). Each triplet has three bits: r (read=4), w (write=2), x (execute=1).

-rwxr-xr--  1 pushkar devops 4096 Jan 1 12:00 script.sh
│└┬┘└┬┘└┬┘
│ │  │  └── others: r-- = 4 (read only)
│ │  └───── group:  r-x = 5 (read + execute)
│ └──────── owner:  rwx = 7 (full)
└────────── file type: - = regular, d = directory, l = symlink

chmod and chown

# Symbolic mode
chmod u+x script.sh          # add execute for owner
chmod go-w sensitive.conf    # remove write for group and others
chmod a+r public.html        # add read for everyone
chmod u=rwx,g=rx,o= priv/   # set exactly

# Octal mode (faster once memorised)
chmod 755 script.sh    # rwxr-xr-x
chmod 644 config.yml   # rw-r--r--
chmod 600 ~/.ssh/id_rsa       # rw------- (private key requirement)
chmod 700 ~/.ssh/             # rwx------ (SSH dir requirement)
chmod 777 /tmp/shared  # rwxrwxrwx — almost always wrong

# Ownership
chown pushkar:devops file.txt          # change owner and group
chown -R www-data:www-data /var/www/   # recursive
chgrp docker /var/run/docker.sock      # change group only

Special Bits

Bit	On files	On directories
setuid (4xxx)	Run as owner's UID (`-rwsr-xr-x`)	No effect
setgid (2xxx)	Run as owner's GID	New files inherit directory's group
sticky (1xxx)	No effect	Only owner/root can delete files (`/tmp`)

chmod u+s /usr/bin/passwd    # setuid — passwd needs root to modify /etc/shadow
chmod g+s /var/www/uploads   # setgid — all new files owned by www-data group
chmod +t /tmp                # sticky — standard on /tmp
ls -la /usr/bin/passwd       # -rwsr-xr-x (note the 's')

umask

The umask is subtracted from the default permissions when new files are created (default 666 for files, 777 for dirs).

umask          # 0022 → files get 644, dirs get 755
umask 027      # more restrictive: files 640, dirs 750

Tip: ACLs (setfacl/getfacl) extend beyond the three-owner model. Use them when you need fine-grained per-user access to a shared directory without changing ownership.

Grep Pattern Searching

grep filters lines matching a regular expression. It's your first tool for searching logs, configs, and code at the command line. See also Piping Redirection for how grep composes with other tools. Build and test patterns first with the Regex Log Tester.

Basic Usage

grep "error" /var/log/syslog               # case-sensitive match
grep -i "error" /var/log/syslog            # case-insensitive
grep -n "timeout" app.log                  # show line numbers
grep -c "WARN" app.log                     # count matching lines
grep -v "DEBUG" app.log                    # invert — lines NOT matching
grep -r "SECRET_KEY" /etc/                 # recursive directory search
grep -l "nginx" /etc/logrotate.d/*         # only print filenames
grep -w "root" /etc/passwd                 # whole-word match only

Extended Regex (ERE)

grep -E "error|warn|crit" syslog           # alternation
grep -E "^[0-9]{4}-[0-9]{2}" app.log      # lines starting with ISO date
grep -E "failed (password|publickey)" /var/log/auth.log  # auth failures
grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" access.log  # IPs

Context and Output

grep -A 3 "FATAL" app.log    # 3 lines after each match
grep -B 2 "FATAL" app.log    # 2 lines before
grep -C 5 "FATAL" app.log    # 5 lines before and after
grep --color=auto "error" syslog    # highlight matches
grep -o "[0-9]\+\.[0-9]\+" file     # print only the matching portion

Real-World Patterns

# Find all failed SSH logins
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn

# Count HTTP 5xx errors in nginx log
grep -E '" 5[0-9]{2} ' /var/log/nginx/access.log | wc -l

# Find config files containing a port number
grep -rn "^Listen\|^Port\|^port" /etc/nginx/ /etc/apache2/ 2>/dev/null

Awk Text Processing Powerhouse

awk is a field-oriented text processor. Think of it as a per-line program: for each line, split into fields, run your script. It handles 80% of the log-parsing and report-generation tasks you'd otherwise reach for Python for.

Basic Syntax

awk '{print $1}' file.txt          # print first field (whitespace-delimited)
awk '{print $1, $3}' file.txt      # fields 1 and 3
awk '{print NR, $0}' file.txt      # NR = record (line) number, $0 = whole line
awk 'NR==5' file.txt               # print only line 5
awk 'NF > 3' file.txt              # lines with more than 3 fields

Field Separators

awk -F: '{print $1, $3}' /etc/passwd         # colon-delimited
awk -F'\t' '{print $2}' data.tsv             # tab-delimited
awk 'BEGIN{FS=","} {print $1}' data.csv      # set FS in BEGIN block
awk -F'[,;]' '{print $1}' mixed.txt          # regex separator

Built-in Variables

Variable	Meaning
`$0`	Entire current line
`$1`, `$2`, ...	Fields 1, 2, ...
`NR`	Current record (line) number
`NF`	Number of fields in current record
`FS`	Input field separator (default: whitespace)
`OFS`	Output field separator (default: space)
`RS`	Record separator (default: newline)
`FILENAME`	Current filename

BEGIN and END Blocks

# Sum a column
awk '{sum += $3} END {print "Total:", sum}' sales.csv

# Count occurrences
awk '{count[$1]++} END {for (k in count) print count[k], k}' access.log

# Print header + filtered rows
awk 'BEGIN{print "Name,UID"} -F: $3 >= 1000 {print $1","$3}' /etc/passwd

Real-World Awk

# Extract top 5 IPs from nginx access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -5

# Calculate average response time (field 7 = response_time in custom log)
awk '{sum+=$7; n++} END {printf "avg: %.2fms\n", sum/n}' app.log

# Report disk usage by filesystem (parse df output)
df -hP | awk 'NR>1 {printf "%-30s %s used\n", $6, $5}'

# Reformat CSV — swap columns 1 and 2
awk -F, 'BEGIN{OFS=","} {print $2,$1,$3}' data.csv

sed — Stream Editor

sed is a non-interactive text editor that reads input line-by-line and applies editing commands. While vim/nano are interactive, sed runs in a CI/CD pipeline, in a deployment script, in a Dockerfile — anywhere you need to programmatically modify files without a human pressing keys.

Note: Why DevOps engineers can't live without sed:

CI/CD config injection: swap localhost for prod URL during deploy.

Templating: render config files from templates with environment-specific values.

Batch find-and-replace: across hundreds of files in a single command.

Log scrubbing: remove PII, mask passwords before sharing logs.

Basic Syntax

$ sed 's/old/new/' file           # substitute (first occurrence per line)
$ sed 's/old/new/g' file          # substitute ALL occurrences (global)
$ echo "hello world" | sed 's/world/devops/'
hello devops

By default sed prints to stdout and does NOT modify the file. Use -i to edit in-place.

Key Operations

Command	What it does
`s/old/new/`	Substitute first match per line
`s/old/new/g`	Substitute all matches (global)
`s/old/new/gi`	Global, case-insensitive
`/pattern/d`	Delete lines matching pattern
`3d`	Delete line 3
`1,5d`	Delete lines 1 through 5
`/pattern/p`	Print matching line (usually with `-n`)
`/pattern/i\text`	Insert text BEFORE matching line
`/pattern/a\text`	Append text AFTER matching line
`y/abc/xyz/`	Transliterate: a→x, b→y, c→z
`/start/,/end/d`	Delete from /start/ through /end/

In-Place Editing

$ sed -i 's/DEBUG=true/DEBUG=false/' .env          # DANGEROUS — no undo!
$ sed -i.bak 's/DEBUG=true/DEBUG=false/' .env       # creates .env.bak first — RECOMMENDED

Caution: Always use -i.bak in production. A bad sed command can corrupt thousands of files instantly. The .bak safety net costs nothing.

Multiple Commands and Regex

$ sed -e 's/foo/bar/g' -e 's/baz/qux/g' file       # chain with -e
$ sed 's/foo/bar/g; s/baz/qux/g' file              # or semicolons
$ sed -E 's/([0-9]+)/NUMBER/' file                  # -E for Extended Regex
$ sed -E 's/(.+)@(.+)/User: \1, Domain: \2/' email.txt   # backreferences

Real DevOps Examples

$ sed 's/localhost/prod-server.com/g' config.yml    # swap hostname (preview)
$ sed -i 's/DEBUG=true/DEBUG=false/' .env           # disable debug in prod
$ sed -n '5,10p' logfile                            # print lines 5-10
$ sed '/^#/d' config.conf                           # remove comment lines
$ sed '/^$/d' file                                  # strip blank lines
$ sed -i '1i\#!/bin/bash' script.sh                 # add shebang at top
$ sed 's/^[[:space:]]*//' file                      # trim leading whitespace
$ sed -i 's|/old/path|/new/path|g' config           # use | delimiter for paths
$ sed -i '/^#PermitRootLogin/s/^#//' /etc/ssh/sshd_config   # uncomment a line
$ sed -i 's/\r$//' file.txt                         # convert CRLF to LF
$ sed -E 's/(password=)[^&]+/\1******/g' app.log    # mask passwords

sed vs awk — When to Use Which

Task	Use	Why
Find and replace text	`sed`	Built for substitution
Delete lines by pattern	`sed`	One-liner with /pattern/d
Column-based processing	`awk`	Field-aware out of the box
Arithmetic / aggregation	`awk`	Full programming language
Edit config files in CI/CD	`sed`	Quick, simple substitutions
Parse log columns and count	`awk`	Native associative arrays

Real world: Kubernetes ConfigMap templating with sed:
$ sed -e "s/__ENV__/production/g" \
    -e "s/__VERSION__/v2.4.1/g" \
    -e "s|__DB_URL__|${DATABASE_URL}|g" \
    configmap.template.yaml > configmap.yaml
$ kubectl apply -f configmap.yaml
Every modern Helm chart and Kustomize patch fundamentally does this — sed is what every templating engine wraps under the hood.

Piping Redirection

The Unix philosophy: small tools that do one thing well, composed via pipes. Mastering redirection turns individual commands into data pipelines.

Data flows left-to-right through pipes; > redirects final stdout to a file, while 2>&1 merges stderr into the same stream.

Redirection Operators

# stdout redirection
command > file.txt          # redirect stdout (overwrite)
command >> file.txt         # append stdout
command 2> errors.txt       # redirect stderr
command 2>&1                # merge stderr into stdout
command > file.txt 2>&1     # both stdout and stderr to file
command &> file.txt         # bash shorthand for above

# stdin redirection
command < input.txt         # feed file as stdin
command <<< "string"        # here-string (bash)
command << EOF              # heredoc
line1
line2
EOF

# /dev/null — discard
command > /dev/null 2>&1   # suppress all output
command 2>/dev/null        # suppress errors only

Pipes

# Basic pipe: stdout of left → stdin of right
ls -la | grep ".log"
cat /etc/passwd | awk -F: '{print $1}'     # inefficient, use awk directly
ps aux | grep nginx | grep -v grep

# xargs — build commands from stdin
find /var/log -name "*.log" | xargs wc -l          # count lines in all logs
find /tmp -mtime +7 | xargs rm -f                  # delete old files
cat servers.txt | xargs -I{} ssh {} "uptime"       # run on each host

# tee — split: stdout goes to both pipe and file
command | tee output.txt | next_command             # log and continue

Process Substitution

# <(cmd) — treat command's output as a file (bash)
diff <(sort file1.txt) <(sort file2.txt)
comm -23 <(sort list1.txt) <(sort list2.txt)   # lines only in list1

# >(cmd) — feed output to command as if it were a file
tee >(wc -l > line_count.txt) < big_file.txt

Practical Pipelines

# Top 10 largest files in /var
find /var -type f -printf '%s %p\n' | sort -rn | head -10

# Count unique IPs in nginx log for today
grep "$(date +%d/%b/%Y)" /var/log/nginx/access.log | awk '{print $1}' | sort -u | wc -l

# Show only failed systemd units
systemctl list-units --state=failed --no-legend | awk '{print $1}'

# Monitor a log in real time, highlight errors
tail -f /var/log/syslog | grep --line-buffered -E "error|warn|crit"

Tip: grep --line-buffered forces grep to flush after every matched line when reading from a pipe — essential for real-time monitoring. Without it, grep buffers output and you get it in chunks.

Process Basics

In Linux, everything is a process. When you run ls, that's a process. When a Docker container starts, that's an (isolated) process. When nginx serves a request, that's a process too. To become a DevOps engineer you must understand processes deeply.

What is a Process?

A process is a running instance of a program. The program (executable file like /usr/bin/node) sits on disk — when you execute it, the kernel loads it into memory, gives it a unique PID (Process ID), assigns CPU time, memory, and file descriptors.

PID — Process ID. Unique integer. PID 1 is always init / systemd.
PPID — Parent Process ID. Every process (except PID 1) has a parent.
UID / GID — User and Group ID the process runs as. Determines permissions.
TTY — The terminal it's attached to (or ? for daemons).
NI — Nice value (priority, -20 highest to 19 lowest).

Foreground vs Background

A foreground process owns the terminal and blocks your shell until it exits. A background process detaches (append &) so the prompt returns immediately while it keeps running.

$ sleep 300        # foreground — terminal blocked
$ sleep 300 &      # background — prompt returns immediately
[1] 24817
$ jobs
[1]+  Running                 sleep 300 &

fork() and exec()

Linux creates new processes through two system calls — the famous fork() + exec() pair.

fork() — The parent process duplicates itself. The child gets a new PID; parent gets the child's PID, child gets 0.
exec() — The child replaces its memory image with a new program. Same PID, different code.
wait() — Parent waits for child to finish and reaps its exit status.

When you type ls in bash: bash forks itself, the child execs /usr/bin/ls, bash waits for it to exit.

Process States

State	Code	Meaning	When you see it
Running	R	Currently executing or runnable	Active workloads
Sleeping (Interruptible)	S	Waiting for an event (I/O, signal, timer)	Most idle daemons
Disk Sleep (Uninterruptible)	D	Waiting for I/O — cannot be killed even with SIGKILL	NFS hang, broken disk
Stopped	T	Suspended by signal (SIGSTOP / Ctrl+Z)	After you hit Ctrl+Z
Zombie	Z	Dead but parent hasn't reaped exit status	Bad parent process
Idle	I	Idle kernel thread	Kernel worker threads

Caution: D state is dangerous: Disk Sleep processes ignore all signals including SIGKILL. If you see lots of D state processes, your disk / NFS mount is likely broken. kill -9 won't help.

Exploring /proc/[pid]/

The kernel exposes every process as a directory under /proc/.

$ pgrep -f nginx | head -1
1842
$ cat /proc/1842/cmdline | tr '\0' ' '
nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
$ cat /proc/1842/status | head -6
Name:   nginx
State:  S (sleeping)
Pid:    1842
PPid:   1
$ readlink /proc/1842/exe       # path to binary
/usr/sbin/nginx

Why Processes Matter in DevOps

Containers ARE processes. A Docker container is just a regular Linux process with extra isolation provided by the kernel: namespaces (separate views of PIDs, network, mounts), cgroups (CPU, memory, I/O limits), chroot/pivot_root (restricted filesystem), and seccomp/capabilities (restricted syscalls).

Interview tip: "What's the difference between a VM and a container?" — A VM virtualizes hardware and runs a full guest kernel. A container is a regular process on the host kernel with namespaces and cgroups for isolation. Container = process. VM = virtual machine.

Process Commands

Understanding how to inspect and manage processes is essential for debugging hung services, investigating CPU/memory spikes, and safely stopping background work.

See also: systemd Services for managing processes as services.

Viewing Processes

ps aux                        # all processes, BSD syntax (a=all, u=user, x=no-tty)
ps -ef                        # all processes, POSIX syntax (-e=all, -f=full)
ps aux | grep nginx           # filter by name
ps -p 1234 -o pid,ppid,comm,etime,rss   # specific PID, custom columns

# pgrep / pkill — search/signal by name
pgrep -l nginx                # list PIDs matching "nginx"
pgrep -u www-data             # PIDs owned by www-data
pkill -HUP nginx              # send SIGHUP to all nginx processes (graceful reload)

top and htop

top              # interactive process viewer
                 # Keys: k=kill, r=renice, u=filter by user, 1=per-CPU, q=quit
                 # Press 'c' to show full command line

htop             # enhanced viewer (apt install htop)
                 # Mouse clicks work; F5=tree view, F6=sort column

Signals

kill -l                  # list all signals
kill PID                 # send SIGTERM (15) — polite shutdown request
kill -9 PID              # SIGKILL — immediate termination (no cleanup)
kill -HUP PID            # SIGHUP (1) — reload config (nginx, rsyslog)
kill -STOP PID           # pause a process
kill -CONT PID           # resume a paused process

# Sending to all instances
killall nginx            # by name
pkill -f "python worker" # by full command line regex

Signal	Number	Meaning
SIGHUP	1	Hangup / reload config
SIGINT	2	Keyboard interrupt (Ctrl-C)
SIGQUIT	3	Quit with core dump
SIGKILL	9	Immediate kill (uncatchable)
SIGTERM	15	Graceful shutdown (default for `kill`)
SIGUSR1/2	10/12	User-defined (app-specific)
SIGSTOP	19	Pause (uncatchable)
SIGCONT	18	Resume

Background Jobs and nice

# Background/foreground
command &          # start in background
jobs               # list background jobs in current shell
fg %1              # bring job 1 to foreground
bg %2              # resume stopped job 2 in background
nohup ./script.sh &  # immune to SIGHUP (survives shell logout)
disown %1          # detach job from shell (no SIGHUP on exit)

# nice / renice — CPU scheduling priority (-20=highest, 19=lowest)
nice -n 10 ./heavy_script.sh    # start with lower priority
renice 5 -p 1234                # change running process

lsof and fuser

lsof -i :80                   # what's listening on port 80
lsof -p 1234                  # all files opened by PID
lsof -u www-data              # files opened by a user
lsof +D /var/log              # all processes using files under /var/log

fuser 80/tcp                  # PID using port 80
fuser -k 80/tcp               # kill it

Tip: When a deleted file is still consuming disk space, lsof | grep deleted shows which process still holds the file descriptor open. Restart that process to reclaim the space.

systemd Services

systemd is the init system on every modern Linux distro. It manages services, mounts, timers, sockets, and devices as a dependency graph. Everything that was once a shell script in /etc/init.d is now a unit file.

A systemd service cycles through states driven by systemctl start/stop/restart; enable/disable controls boot-time auto-start independently.

Essential systemctl Commands

# Service lifecycle
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx            # graceful config reload (if supported)
systemctl status nginx            # status + last 10 log lines

# Enable/disable at boot
systemctl enable nginx            # create symlink in wants/
systemctl disable nginx
systemctl enable --now nginx      # enable + start in one command
systemctl is-enabled nginx        # returns enabled/disabled/static

# Inspection
systemctl list-units --type=service --state=running
systemctl list-units --state=failed
systemctl cat nginx               # print unit file
systemctl show nginx              # all properties
systemctl list-dependencies nginx # tree of dependencies

Unit File Structure

Unit files live in /etc/systemd/system/ (your files) and /lib/systemd/system/ (package defaults). Your files take precedence.

# /etc/systemd/system/myapp.service
[Unit]
Description=My Node Application
Documentation=https://github.com/yourorg/myapp
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp /var/log/myapp
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

# After creating/editing a unit file
systemctl daemon-reload          # reload all unit files
systemctl enable --now myapp

journalctl — Reading Logs

journalctl -u nginx                    # all logs for nginx unit
journalctl -u nginx -f                 # follow in real time
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since "2024-01-01" --until "2024-01-02"
journalctl -u nginx -n 50             # last 50 lines
journalctl -p err -u nginx            # only priority error and above
journalctl -b                         # since last boot
journalctl -b -1                      # previous boot
journalctl --disk-usage               # how much disk the journal uses
journalctl --vacuum-size=200M         # trim journal to 200 MB

systemd Timers

Timers replace cron with dependency tracking, logging via journal, and precise time expressions.

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=*-*-* 02:00:00   # every day at 02:00
Persistent=true              # run if missed while system was off
RandomizedDelaySec=10min     # spread load across fleet

[Install]
WantedBy=timers.target

systemctl enable --now backup.timer
systemctl list-timers                 # all timers + next trigger time

Tip: Always run systemctl daemon-reload after editing any unit file. Without it, systemd uses the cached version from memory and your changes are silently ignored.

User Management

User management is the foundation of Linux security. As a DevOps engineer, you'll create service accounts for CI/CD pipelines, manage developer access, lock down compromised users, and configure sudo policies. Think of this like managing IAM roles in AWS — except it's text files all the way down, and a single typo in /etc/sudoers can lock you out of your own server.

/etc/passwd and /etc/shadow

Every user account is defined in /etc/passwd. Despite the name, passwords are NOT stored here — they moved to /etc/shadow because /etc/passwd must be world-readable.

/etc/passwd has 7 colon-separated fields: username:x:UID:GID:GECOS:home_dir:login_shell

$ grep deploy /etc/passwd
deploy:x:1001:1001:Deploy User,,,:/home/deploy:/bin/bash

deploy — username
x — placeholder (the x means "look in /etc/shadow")
1001 — UID. 0 is root. 1-999 system users. 1000+ regular humans
1001 — primary GID
/home/deploy — home directory
/bin/bash — login shell (set to /usr/sbin/nologin to disable interactive login)

/etc/shadow (root-only) stores password hashes plus aging policy: username:hash:last_change:min:max:warn:inactive:expire:reserved. The hash prefix $6$ = SHA-512, $y$ = yescrypt (Ubuntu 24.04 default), ! or * = locked.

Creating Users: useradd vs adduser

useradd — low-level, scriptable, doesn't create home dir or set password by default. Everywhere.
adduser — Debian/Ubuntu's friendly wrapper. Interactive, creates home, prompts for password. NOT on RHEL.

$ sudo useradd -m -s /bin/bash -G sudo,docker deploy
$ sudo passwd deploy

The -m flag creates the home directory, -s sets the shell, -G adds supplementary groups. Without -m, the user has no home.

usermod — Modify Existing Users

$ sudo usermod -aG docker $USER        # APPEND to group
$ sudo usermod -L olduser              # lock password
$ sudo usermod -s /usr/sbin/nologin ci-runner   # change shell
$ sudo usermod -e 2026-12-31 contractor-jay     # set expiry

Caution: The -a in -aG is non-negotiable. Running sudo usermod -G docker alice (without -a) REPLACES all of Alice's supplementary groups with just docker. She loses sudo, adm, dialout — everything. Always: usermod -aG. The -a stands for "append".

passwd and chage — Password and Aging

$ passwd                    # change YOUR own password
$ sudo passwd alice         # change alice's password as root
$ sudo passwd -l olduser    # lock account
$ sudo passwd -e alice      # force password change on next login

$ sudo chage -l deploy      # list aging policy
$ sudo chage -M 90 -W 7 deploy      # expire in 90 days, warn 7 days before
$ sudo chage -d 0 alice     # force password change at next login

su vs su -

su username — switches user, keeps your current environment.
su - username — switches AND runs a login shell, loading ~/.bashrc and resetting environment as if you logged in fresh.

Interview tip: "Difference between sudo su and su -?" su - requires the target user's password and gives a full login shell. sudo su requires YOUR password (sudo authenticates the invoker), then runs su as root without a password prompt — but env vars may leak. Best practice: use sudo -i for a clean interactive root login shell.

sudo and /etc/sudoers

When you type sudo apt update: the setuid-root sudo binary reads /etc/sudoers, verifies your user is allowed, prompts for YOUR password (cached ~15 min), logs the action to /var/log/auth.log, then executes as the target user.

Sudoers rules: user_or_%group host=(runas_user) commands

root    ALL=(ALL:ALL) ALL                        # root can do anything
%sudo   ALL=(ALL:ALL) ALL                         # sudo group (Ubuntu default)
%wheel  ALL=(ALL) ALL                             # wheel group (RHEL)
deploy  ALL=(root) NOPASSWD: /usr/bin/systemctl restart nginx

Caution: NEVER edit /etc/sudoers with vim or nano directly. visudo validates syntax BEFORE saving. A single typo renders the file unparseable and breaks sudo for everyone — including root. Recovering requires single-user mode. Use sudo visudo or drop-in files in /etc/sudoers.d/ (chmod 0440).

Real DevOps Examples

# Create a developer account with sudo + docker
$ sudo useradd -m -s /bin/bash -G sudo,docker deploy && sudo passwd deploy

# Lock a former employee's account immediately
$ sudo passwd -l olduser && sudo usermod -L olduser && sudo usermod -s /usr/sbin/nologin olduser

# Service account for CI/CD — no shell, no home login
$ sudo useradd --system --shell /usr/sbin/nologin --home-dir /var/lib/jenkins jenkins

# Audit who has sudo right now
$ getent group sudo

# Run a single command as another user
$ sudo -u postgres psql

Real world: CI/CD Service Account — you NEVER want interactive login:
$ sudo useradd --system --shell /usr/sbin/nologin --home-dir /var/lib/runner \
       --create-home --comment "GitLab CI Runner" gitlab-runner
$ sudo usermod -aG docker gitlab-runner
$ sudo passwd -l gitlab-runner     # lock password — only SSH key auth
--system assigns a UID below 1000. nologin means even if SSH keys are stolen, no shell is granted. The locked password prevents su attacks. Defense-in-depth.

Group Management

Groups are how Linux scales permissions beyond individual users. Instead of granting access to 20 developers one-by-one, you create a developers group, grant the group access, and add people to the group.

/etc/group Structure

Four colon-separated fields: groupname:x:GID:member1,member2,member3

$ grep -E '^(sudo|docker|developers)' /etc/group
sudo:x:27:pushkar,alice,deploy
docker:x:998:pushkar,deploy,gitlab-runner
developers:x:1100:alice,bob,charlie

System groups: 0–999. User groups: 1000+. The member list shows users whose supplementary group this is — a user's primary group is set by the GID in /etc/passwd.

Primary vs Supplementary Groups

Primary group — exactly ONE per user, defined by GID in /etc/passwd. Files the user creates are owned by this group by default.
Supplementary groups — zero or many, listed in /etc/group. Grant additional permissions.

$ id pushkar
uid=1000(pushkar) gid=1000(pushkar) groups=1000(pushkar),27(sudo),998(docker),1100(developers)

Managing Groups

$ sudo groupadd developers
$ sudo groupadd -g 1500 deploy          # specific GID
$ sudo groupmod -n devops developers     # rename
$ sudo groupdel old-team                 # delete (refuses if it's anyone's primary group)

$ sudo gpasswd -a alice developers       # add member
$ sudo gpasswd -d alice developers       # remove member
$ sudo gpasswd -A bob developers         # make bob a group admin
$ sudo usermod -aG docker pushkar        # the standard "add to group" command

Caution: usermod -G without -a is destructive — it REPLACES all supplementary groups. Always usermod -aG. Mnemonic: "a is for ADD, without a it ANNIHILATES."

newgrp — Switch Active Primary Group

If you were just added to a group, your current shell doesn't know yet. newgrp spawns a subshell with the named group as primary:

$ sudo usermod -aG docker pushkar
$ docker ps        # permission denied — current session predates the change
$ newgrp docker    # spawn subshell with docker group active
$ docker ps        # works!

Common System Groups

Group	Purpose
`sudo`	Debian/Ubuntu: members can use `sudo`
`wheel`	RHEL/CentOS equivalent of `sudo`
`docker`	Run docker without sudo (effectively root — be careful)
`www-data`	Apache/Nginx run as this user
`adm`	Read access to log files in `/var/log/`
`systemd-journal`	Read journalctl logs without sudo
`disk`	Raw disk access — DANGEROUS

Real world: Shared Deployment Directory with setgid — the setgid bit makes new files inherit the directory's group:
$ sudo chown -R root:deploy /opt/app
$ sudo chmod -R 2775 /opt/app       # the 2 = setgid bit
$ sudo chmod g+s /opt/app
Now any file created inside /opt/app is owned by the deploy group, not the creator's primary group — so all team members can read/write each other's files. This is how /srv/web/, /opt/app/, and shared CI artifact directories should be configured.

Package Managers

Package managers are your software distribution layer. Each distro family has its own; as a DevOps engineer you'll touch all of them.

apt (Debian / Ubuntu)

# Update package lists
sudo apt update

# Install / remove
sudo apt install nginx git curl
sudo apt install -y nginx         # non-interactive (scripts/CI)
sudo apt remove nginx             # remove but keep config
sudo apt purge nginx              # remove + delete config files
sudo apt autoremove               # remove orphaned dependencies

# Upgrade
sudo apt upgrade                  # upgrade installed packages
sudo apt full-upgrade             # upgrade + handle dependency changes
sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y   # in scripts

# Search and inspect
apt search "text editor"
apt show nginx
apt list --installed | grep nginx
dpkg -L nginx                     # list files installed by package
dpkg -S /usr/sbin/nginx           # which package owns a file

# Pinning — hold a package at current version
sudo apt-mark hold nginx
sudo apt-mark showhold

dnf / yum (RHEL / Rocky / Fedora)

sudo dnf install nginx
sudo dnf remove nginx
sudo dnf update
sudo dnf search nginx
sudo dnf info nginx
sudo rpm -ql nginx                # list files in installed rpm
sudo rpm -qf /usr/sbin/nginx      # which rpm owns a file

# Repos
sudo dnf repolist
sudo dnf config-manager --enable epel

Alpine apk (containers / musl-based)

apk update
apk add nginx curl bash
apk del nginx
apk search nginx
apk info nginx

# Minimal Dockerfile example
FROM alpine:3.19
RUN apk add --no-cache nginx \
    && rm -rf /var/cache/apk/*

Tip: In Dockerfiles always apt-get update && apt-get install -y pkg in a single RUN layer. Splitting them means the cached update layer can have stale package lists when your install layer is rebuilt days later — causing "package not found" errors.

Disk Management

Disk management is one of the most frequent on-call problems in DevOps. "Server is down" usually means CPU pegged, OOM killer killing processes, or — most often — disk full. You must be fluent with df, du, partitioning tools, and /etc/fstab.

df — Disk Free

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   25G   23G  53% /
/dev/sda2       100G   89G   11G  90% /var

$ df -hT          # show filesystem TYPE — critical for choosing resize tool
$ df -i           # INODES — the silent killer
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2     52428800 52428799      1  100% /var   <- INODE EXHAUSTED!

du — Disk Usage (find WHO is eating space)

$ du -sh /var/log                          # total size of one directory
$ du -h --max-depth=1 /var | sort -h       # top-level breakdown, sorted
$ du -hac --max-depth=2 . | sort -rh | head -20    # "what's eating my disk"
$ sudo du -ah / 2>/dev/null | sort -rh | head -20  # whole system scan (slow)

Partitioning: fdisk and parted

$ sudo fdisk -l                  # list all disks and partitions
$ sudo fdisk /dev/sdb            # interactive (n=new, p=print, d=delete, w=WRITE)

$ sudo parted -l
$ sudo parted /dev/sdb mklabel gpt
$ sudo parted /dev/sdb mkpart primary ext4 0% 100%

gdisk is the GPT-only cousin of fdisk. Use it when fdisk complains about a GPT disk.

lsblk and blkid — Inspection

$ lsblk            # tree of block devices and mountpoints
$ lsblk -f         # with FS type and UUID — best "what's on this disk?" command
$ sudo blkid       # UUIDs for fstab entries

mount / umount

$ sudo mount /dev/sdb1 /mnt/data
$ sudo mount -t ext4 /dev/sdb1 /mnt/data        # specify FS type
$ sudo mount -o ro /dev/sdb1 /mnt/data          # read-only
$ sudo mount -o remount,rw /                    # remount root rw (rescue trick)
$ sudo umount /mnt/data
$ sudo umount -l /mnt/data                      # lazy unmount if "device is busy"
$ findmnt                                        # pretty tree view of all mounts

/etc/fstab — Persistent Mounts

Format: device mountpoint fstype options dump pass

$ cat /etc/fstab
UUID=a3f2e1b4-...  /        ext4    defaults,noatime                0  1
UUID=b1c2d3e4-...  /data    xfs     defaults,nofail                 0  2
tmpfs              /tmp     tmpfs   defaults,size=2G,noexec         0  0
/swapfile          none     swap    sw                              0  0
nfs.internal:/exports/shared  /mnt/nfs  nfs  defaults,_netdev,soft,timeo=30  0  0

$ sudo mount -a                # mount everything in fstab — TEST before reboot!
$ sudo findmnt --verify         # sanity-check fstab without mounting

The pass column controls fsck order at boot: 0=skip, 1=root only, 2=other filesystems.

Caution: A typo in /etc/fstab (wrong UUID, missing mountpoint, bad option) drops the system into emergency mode on next reboot. Always test with sudo mount -a after editing. For non-critical mounts add nofail; for network filesystems add _netdev. To recover from a boot hang: single-user mode, mount -o remount,rw /, fix fstab, reboot.

Mounting by UUID (Best Practice)

Device names like /dev/sdb can change across reboots. UUIDs are baked into the filesystem and never change. Always use UUIDs in fstab.

$ sudo blkid /dev/sdb1
/dev/sdb1: UUID="b1c2d3e4-..." TYPE="xfs"
$ echo 'UUID=b1c2d3e4-... /data xfs defaults,nofail 0 2' | sudo tee -a /etc/fstab

Swap

$ sudo fallocate -l 2G /swapfile
$ sudo chmod 600 /swapfile          # required — else mkswap refuses
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ swapon --show
$ free -h                            # confirm swap appears
$ echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab   # persist

LVM intro

LVM (Logical Volume Manager) adds a flexible layer between partitions and filesystems: PV (Physical Volume), VG (Volume Group), LV (Logical Volume). You can resize LVs without unmounting, take snapshots, and span multiple disks. Ubuntu Server defaults to LVM.

Real world: "No space left on device" but df shows free space? You're out of inodes, not bytes. Every file/dir consumes one inode regardless of size. Diagnose with df -i — if any IUse% is 100%, that's it. Common culprits: session files, mail queues, cache dirs with millions of tiny files. Fix: delete the small files; xfs allocates inodes dynamically and avoids this entirely.

Interview tip: "Disk is full, what do you do?" (1) df -h — which FS, how full. (2) df -i — rule out inode exhaustion. (3) du -h --max-depth=1 / | sort -h — find the heavy directory. (4) Check usual suspects: /var/log, /var/lib/docker (docker system prune), journal (journalctl --vacuum-size=500M). (5) Check deleted-but-open files: sudo lsof | grep deleted. (6) Extend the volume (cloud: resize EBS → growpart → resize2fs/xfs_growfs).

Filesystems

A filesystem is the contract between raw blocks on a disk and the files you see. Choosing the right one matters: ext4 for general Linux, xfs for big files and parallel I/O, btrfs/zfs for snapshots, tmpfs for ephemeral RAM-backed storage, overlayfs for Docker layers.

The Main Linux Filesystems

ext4 — Default on Ubuntu/Debian. Mature, journaled, max file 16 TB. Resize online with resize2fs.
xfs — Default on RHEL/Rocky/Amazon Linux. Excellent for large files and parallel I/O. Allocates inodes dynamically. Can grow online (xfs_growfs) but cannot shrink.
btrfs — Copy-on-write, snapshots, subvolumes, built-in RAID, compression. Default on Fedora Workstation.
zfs — Enterprise-grade. Pools, end-to-end checksums, snapshots, send/receive replication. Not in mainline kernel (license clash) — install zfsutils-linux.
NTFS / FAT32 / exFAT — Windows/removable media. FAT32 has 4 GB per-file limit; exFAT removes it.
tmpfs — RAM-backed. Files vanish on reboot. Used for /tmp, /run, /dev/shm.
overlayfs — The magic behind Docker. Stacks a writable upper layer over read-only lower layers under /var/lib/docker/overlay2/.
proc, sysfs, devtmpfs — Virtual filesystems exposing kernel state.

Comparison Table

Feature	ext4	xfs	btrfs	zfs
Max file size	16 TB	8 EB	16 EB	16 EB
Journaling	Yes	Yes	CoW	CoW + ZIL
Copy-on-Write	No	Reflinks	Yes	Yes
Snapshots	No (need LVM)	No (need LVM)	Native	Native
Online grow	Yes (resize2fs)	Yes (xfs_growfs)	Yes	Yes
Online shrink	No (offline)	No (never)	Yes	No
Built-in RAID	No	No	Yes	Yes (RAID-Z)
Compression	No	No	Yes	Yes
Checksums	Metadata only	Metadata only	Data + metadata	Data + metadata
Inode allocation	Static at mkfs	Dynamic	Dynamic	Dynamic
Default on	Ubuntu, Debian	RHEL, Amazon Linux	Fedora WS, SUSE	TrueNAS, Proxmox

Creating and Resizing Filesystems

$ sudo mkfs.ext4 /dev/sdb1
$ sudo mkfs.ext4 -L data -m 1 /dev/sdb1       # label "data", 1% reserved for root
$ sudo mkfs.xfs -L data -f /dev/sdb1
$ sudo mkfs.vfat -F 32 -n USB /dev/sdb1

# ext4 grow (online or offline)
$ sudo resize2fs /dev/sdb1            # grow to fill partition
# ext4 shrink (MUST unmount + fsck first)
$ sudo umount /data && sudo e2fsck -f /dev/sdb1 && sudo resize2fs /dev/sdb1 20G
# xfs grow ONLY (online, pass MOUNTPOINT not device)
$ sudo xfs_growfs /data

fsck — Filesystem Integrity

Caution: NEVER run fsck on a mounted filesystem — it corrupts it. Always unmount first, or boot from rescue media for the root filesystem.

$ sudo umount /dev/sdb1
$ sudo fsck.ext4 -f /dev/sdb1          # -f forces check even if "clean"
$ sudo fsck -y /dev/sdb1               # auto-answer "yes" to repairs
$ sudo xfs_repair /dev/sdb1            # xfs uses xfs_repair, not fsck
$ sudo xfs_repair -n /dev/sdb1         # dry run — report only

Inodes — Deep Dive

An inode is a fixed-size record storing all metadata for a file except its name: type, permissions, owner UID/GID, size, timestamps (atime/mtime/ctime), link count, and pointers to data blocks. The filename lives in the directory entry — a (name → inode number) mapping. This single design choice explains hard links, symlinks, rename, and "deleted but still open" files.

$ stat /etc/passwd      # full inode metadata
$ ls -i /etc/passwd     # show inode number
$ df -i /               # inode usage for a filesystem

File operations in terms of inodes:

Rename / mv (same FS): directory entry name changes; inode untouched — instant even for huge files.
Hard link (ln): a second directory entry pointing to the same inode. Link count increments. Cannot cross filesystems. Cannot link directories.
Symbolic link (ln -s): a tiny file with its own inode whose content is a path string. Can cross filesystems, can point to directories, can dangle.
Delete (rm): removes the directory entry and decrements link count. Inode and data freed only when link count reaches 0 and no process has the file open.

Interview tip: Hard link vs Symlink (inode perspective). Hard link: a second directory entry pointing to the same inode — both names are first-class, same FS only, can't link directories, survives deletion of one name while link count > 0. Symlink: a separate file whose data is a textual path, resolved at access time, can cross filesystems, can link directories, dangles if target removed. In ls -l: a hard link looks like a normal file with link count > 1; a symlink shows l and -> target.

Real world: Resizing an EBS volume on EC2 — three steps, miss one and df shows no change. (1) Modify the EBS volume size in the AWS console/CLI. (2) Grow the partition: sudo growpart /dev/nvme0n1 1. (3) Grow the filesystem: ext4 sudo resize2fs /dev/nvme0n1p1, xfs sudo xfs_growfs -d /. Entire process is online — no reboot, no unmount.

Network Commands

Networking knowledge separates engineers who can debug production incidents from those who can't. These tools let you inspect interfaces, test connectivity, trace traffic, and diagnose service failures.

ip — Modern Network Configuration

# Addresses
ip addr show                   # all interfaces and IPs
ip addr show eth0              # specific interface
ip addr add 192.168.1.10/24 dev eth0   # add IP (temporary)
ip addr del 192.168.1.10/24 dev eth0

# Routes
ip route show                  # routing table
ip route add default via 192.168.1.1   # add default gateway
ip route get 8.8.8.8           # which route is used to reach an IP

# Links
ip link show
ip link set eth0 up
ip link set eth0 down

ss — Socket Statistics (replaces netstat)

ss -tlnp          # TCP (-t) listening (-l) numeric (-n) with process (-p)
ss -ulnp          # UDP listening
ss -tnp           # all established TCP connections
ss -s             # summary statistics

# Common patterns
ss -tlnp | grep :80        # what's on port 80
ss -tnp state ESTABLISHED  # only established connections
ss -tnp dst 10.0.0.5       # connections to a specific host

dig — DNS Diagnostics

dig example.com              # A record (default)
dig example.com MX           # mail exchange records
dig example.com NS           # nameservers
dig example.com TXT          # TXT records (SPF, DKIM, etc.)
dig @8.8.8.8 example.com    # query specific DNS server
dig +short example.com       # just the IPs
dig -x 93.184.216.34         # reverse DNS lookup
dig +trace example.com       # trace full resolution chain

curl — HTTP Testing

curl https://api.example.com/health          # GET request
curl -I https://example.com                  # headers only (HEAD)
curl -X POST -H "Content-Type: application/json" \
     -d '{"key":"value"}' https://api.example.com/data
curl -o /tmp/file.tar.gz https://example.com/file.tar.gz  # download
curl -L https://short.url/abc               # follow redirects
curl -w "%{http_code} %{time_total}s\n" -o /dev/null -s https://example.com  # timing
curl --resolve example.com:443:10.0.0.5 https://example.com  # test before DNS propagation

Other Essential Network Tools

# ping / traceroute / mtr
ping -c 4 8.8.8.8
traceroute 8.8.8.8
mtr 8.8.8.8           # real-time traceroute (apt install mtr)

# nc (netcat) — the network Swiss Army knife
nc -zv host 22             # port scan / test connectivity
nc -l 9000                 # listen on port 9000
echo "PING" | nc -u host 514   # UDP test

# tcpdump — packet capture
sudo tcpdump -i eth0 port 80 -w capture.pcap     # capture to file
sudo tcpdump -i any host 10.0.0.5 and tcp        # filter by host+protocol
sudo tcpdump -r capture.pcap -A                  # read and print ASCII

# watch — repeat a command
watch -n 2 ss -tlnp        # refresh every 2 seconds

Tip: ss -tlnp is your first command when a service "can't start" — it shows exactly which ports are already in use and by which PID. Then systemctl status that PID to find the conflict.

Network Configuration

Commands you can run, but understanding the actual configuration files is the next level. This section covers Ubuntu 24.04's real config files — production servers, EC2 instances, on-prem bare metal all apply.

/etc/hosts — Local Hostname Resolution

Consulted before DNS (per nsswitch.conf order). The lifeline for local development and quick overrides.

$ cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       devops-laptop
10.0.5.20       db.internal    db
10.0.5.30       redis.internal redis
::1             localhost ip6-localhost ip6-loopback

Tip: Production cutover testing — add the new server's IP to /etc/hosts and test with curl/browser before changing DNS. Risk-free validation.

/etc/resolv.conf — DNS Resolvers

$ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 1.1.1.1
search prod.example.com internal
options timeout:2 attempts:2

Caution: On Ubuntu 24.04, /etc/resolv.conf is usually a symlink to /run/systemd/resolve/stub-resolv.conf managed by systemd-resolved. Direct edits get overwritten. Use Netplan or resolvectl instead:
$ resolvectl status
$ sudo resolvectl dns eth0 1.1.1.1 8.8.8.8

/etc/nsswitch.conf — Lookup Order

$ grep ^hosts /etc/nsswitch.conf
hosts: files mdns4_minimal [NOTFOUND=return] dns
# files → /etc/hosts first, mdns4 → .local addresses, dns → /etc/resolv.conf

Netplan — Modern Ubuntu (Primary)

Ubuntu 18.04+ default. YAML-based frontend for NetworkManager or systemd-networkd.

# /etc/netplan/01-static.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: false
      addresses:
        - 192.168.1.50/24
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]
        search: [prod.example.com]
      mtu: 1500

$ sudo netplan try          # test for 120s, auto-rollback if SSH dies
$ sudo netplan apply        # commit changes

Caution: YAML is strict — 2-space indentation, NO TABS. Wrong indent = network down. Always use netplan try first on remote boxes.

NetworkManager (nmcli) — Desktop/Laptop

$ nmcli device status
$ nmcli con show                              # all profiles
$ nmcli dev wifi connect "MySSID" password "secret"
$ nmcli con mod eth0 ipv4.addresses 10.0.1.50/24 ipv4.gateway 10.0.1.1 ipv4.method manual

systemd-networkd and Hostname

Cloud servers usually use systemd-networkd (lightweight). Netplan generates its config under /run/systemd/network/.

$ networkctl status
$ hostnamectl
$ sudo hostnamectl set-hostname prod-web-01

Tip: After a hostname change, also update /etc/hosts (especially the 127.0.1.1 line). Otherwise sudo warns "unable to resolve host".

Bonding and VLANs

# Netplan: active-backup bond for HA
network:
  version: 2
  bonds:
    bond0:
      interfaces: [eth0, eth1]
      addresses: [10.0.1.50/24]
      parameters:
        mode: active-backup
        primary: eth0
  ethernets:
    eth0: {}
    eth1: {}

Common bonding modes: active-backup (failover), balance-rr (round robin), 802.3ad (LACP, needs switch support). VLANs tag one physical NIC into multiple logical networks (802.1Q).

Note: Cloud reality check: 95% of production cloud instances (AWS EC2, GCP, Azure VM) take their IP via DHCP / cloud-init — static IPs are rarely set at the OS level because VPC subnets and ENIs handle assignment, and auto-scaling groups use dynamic IPs. But for on-prem, hybrid, edge devices, bastion hosts, and NFS servers, static IP is a must. Know both perspectives for interviews.

Interview tip: "Netplan vs NetworkManager?" — Netplan is a YAML frontend. The real backend is NetworkManager (desktop) or systemd-networkd (server). Ubuntu cloud images default to renderer: networkd.

SSH Secure Shell Critical

SSH is how you access every cloud VM, container host, and remote server. Beyond basic login, it gives you encrypted tunnels, file transfer, and agent forwarding. Misconfiguring it creates security holes; understanding it properly lets you build zero-trust access patterns.

See also: File Permissions Ownership — SSH enforces strict permission checks on key files. Going deeper on networking? See Networking for DevOps.

SSH key auth: the client signs a server-issued challenge with its private key; the server verifies using the stored public key — no password ever crosses the network.

Key-Based Authentication

# Generate a key pair (Ed25519 is faster and more secure than RSA)
ssh-keygen -t ed25519 -C "pushkar@work" -f ~/.ssh/id_ed25519

# Copy public key to server
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server
# or manually:
cat ~/.ssh/id_ed25519.pub | ssh user@server 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys'

# Permission requirements (SSH will refuse with wrong perms)
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519         # private key
chmod 644 ~/.ssh/id_ed25519.pub     # public key
chmod 600 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/known_hosts

# Start ssh-agent and load key
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
ssh-add -l                          # list loaded keys

Note: The public key body (the long string after ssh-ed25519) is base64-encoded binary. If you ever need to inspect or transform an encoded blob like this — or a base64 secret in a Kubernetes manifest — the Base64 Encoder/Decoder is handy.

~/.ssh/config

The client config file eliminates typing long hostnames and options every time:

# ~/.ssh/config

Host bastion
    HostName 203.0.113.10
    User ec2-user
    IdentityFile ~/.ssh/aws-key.pem
    ServerAliveInterval 60

Host prod-db
    HostName 10.0.1.50             # private IP
    User ubuntu
    ProxyJump bastion              # tunnel through bastion
    IdentityFile ~/.ssh/id_ed25519

Host dev-*
    User developer
    IdentityFile ~/.ssh/dev-key
    StrictHostKeyChecking no       # only for throwaway dev VMs
    UserKnownHostsFile /dev/null

ssh bastion            # uses config above
ssh prod-db            # automatically tunnels via bastion

SSH Tunnels

# Local port forwarding: access remote service on your local port
ssh -L 5432:db-host:5432 bastion     # connect to postgresql via bastion
# Now: psql -h localhost -p 5432

# Remote port forwarding: expose local port on the remote server
ssh -R 8080:localhost:3000 server    # server:8080 → your localhost:3000

# Dynamic (SOCKS proxy): route browser traffic through the tunnel
ssh -D 1080 bastion                  # configure browser to use SOCKS5 localhost:1080

# Background / non-interactive
ssh -fNL 5432:db:5432 bastion       # -f = background, -N = no command

scp and rsync

# scp — simple copy
scp file.txt user@server:/tmp/
scp -r user@server:/var/www/ ./backup/
scp -P 2222 file.txt user@server:/tmp/    # custom port

# rsync — incremental sync (preferred for large transfers)
rsync -avz /local/dir/ user@server:/remote/dir/    # sync to remote
rsync -avz --delete /src/ /dst/                     # mirror (delete extra)
rsync -avz --exclude '.git' --exclude 'node_modules' ./ server:/opt/app/
rsync -e "ssh -i ~/.ssh/key.pem" /file user@host:/dest/

sshd — Server Configuration Hardening

sudo nano /etc/ssh/sshd_config

# /etc/ssh/sshd_config (key hardening settings)
Port 2222                          # non-standard port (minor obscurity)
PermitRootLogin no                 # never allow root login
PasswordAuthentication no          # key-only (most important setting)
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
AllowUsers pushkar deploy
MaxAuthTries 3
LoginGraceTime 20
ClientAliveInterval 300
ClientAliveCountMax 2
X11Forwarding no
AllowTcpForwarding yes             # needed for tunnels

sudo sshd -t                        # test config syntax
sudo systemctl reload sshd          # apply without dropping connections

Tip: Always test your sshd config with sshd -t and keep an existing SSH session open while applying changes. If you lock yourself out of a cloud VM, you'll need console access or a rescue snapshot.

Bash Fundamentals

Bash (Bourne Again SHell) is the default interactive shell on Ubuntu, Debian, RHEL derivatives, and most Linux distros. As a DevOps engineer, bash is your daily driver — every CI pipeline, every Dockerfile RUN, every cron job, every systemd ExecStart is some flavour of shell glue.

Note: Ubuntu's /bin/sh is actually dash (Debian Almquist Shell) — a POSIX-compliant shell with no bash extensions. Scripts written assuming bash but invoked as sh script.sh will fail on [[ ]], arrays, or $(()) arithmetic. Always be explicit about your shebang.

The Shebang and Running Scripts

#!/bin/bash            # Absolute path — works on virtually every Linux box
#!/usr/bin/env bash    # Looks up bash via PATH — portable to macOS/BSD

Command	Shell	Needs +x?	Variables persist in parent?
`./script.sh`	Subshell (from shebang)	Yes	No
`bash script.sh`	New bash subshell	No	No
`source script.sh` or `. script.sh`	Current shell	No	Yes

Variables

The cardinal rule: no spaces around the = sign. Always quote when expanding to handle spaces and globs safely.

NAME="Pushkar"
GREETING="Hello, $NAME"
echo "${GREETING}!"                   # braces disambiguate variable name

# Parameter expansion
echo "${UNSET_VAR:-default}"         # use default if unset/empty
echo "${MAYBE:=set_now}"             # assign default if unset (sticky)
echo "${REQUIRED:?must be set}"      # error and exit if unset
echo "${NAME:+exists}"               # expand to 'exists' only if set

FILE="backup.tar.gz"
echo "${#FILE}"                      # 13  (length)
echo "${FILE%.gz}"                   # backup.tar  (strip shortest suffix)
echo "${FILE%%.*}"                   # backup       (strip longest suffix)
echo "${FILE/tar/zip}"               # backup.zip.gz (replace first match)

Positional Parameters

Variable	Meaning
`$0`	Script name
`$1` ... `$9`	First nine positional arguments
`$#`	Number of arguments
`"$@"`	All args, each preserved as one quoted word
`$$`	PID of current shell
`$?`	Exit code of last command (0 = success)
`$!`	PID of last background process

Bash Strict Mode

Put this at the top of every serious script:

set -euo pipefail
IFS=$'\n\t'

Flag	What it does
`-e` (errexit)	Exit immediately when any command returns non-zero
`-u` (nounset)	Treat references to unset variables as errors
`-o pipefail`	Pipeline exit code is the rightmost non-zero status
`IFS=$'\n\t'`	Restrict word-splitting to newlines/tabs

Arithmetic and Input

echo $((2 + 3 * 4))            # 14  — recommended form
x=5
((x++))                         # x is now 6
echo "scale=2; 7/3" | bc       # floating point: 2.33

read -p "Your name: " name
read -s -p "Password: " pass   # silent input
read -t 10 -p "Quick: " ans    # 10-second timeout

Tip: Use source only for shell configuration files (.bashrc, env loaders). Use ./script.sh for everything else — sourcing arbitrary scripts can pollute your interactive shell if they call exit or set -e.

Bash Control Flow

Control flow in bash looks unfamiliar at first — then, fi, do, done, double brackets. It's because the shell parses commands by whitespace, so keywords need clear separators. Once you internalize the patterns, complex automation becomes muscle memory.

if / elif / else

if [[ "$USER" == "root" ]]; then
    echo "Running as root"
elif [[ "$USER" == "pushkar" ]]; then
    echo "Hi Pushkar"
else
    echo "Unknown user: $USER"
fi

# if on any command's exit code
if grep -q "ERROR" /var/log/syslog; then
    echo "Errors found in syslog"
fi

if systemctl is-active --quiet nginx; then
    echo "nginx is running"
fi

Tip: Use [[ ]] in bash scripts always. Use [ ] only when writing a portable /bin/sh script. Inside [[ ]], do not quote the right side of =~ — quoting turns the regex into a literal string.

Comparison Operators

Integer (`[[ ]]` or `(( ))`)	String (`[[ ]]`)	Meaning
`-eq`	`==`	Equal
`-ne`	`!=`	Not equal
`-lt`	`<`	Less than
`-gt`	`>`	Greater than
—	`-z STR`	String is empty
—	`-n STR`	String is non-empty
—	`STR =~ REGEX`	String matches ERE regex

File Test Operators

Operator	True if...
`-e FILE`	File exists (any type)
`-f FILE`	Regular file
`-d DIR`	Directory
`-r FILE`	Readable
`-w FILE`	Writable
`-x FILE`	Executable
`-s FILE`	Exists and non-empty

case Statement

case "$action" in
    start)
        echo "Starting..."
        ;;
    stop|halt)
        echo "Stopping..."
        ;;
    *.log)
        echo "That looks like a log file"
        ;;
    *)
        echo "Unknown action: $action"
        exit 1
        ;;
esac

Loops

# for-in with a list
for env in dev staging prod; do
    echo "Deploying to $env"
done

# Brace expansion range
for i in {1..10}; do echo "loop $i"; done

# Glob iteration — always quote the variable
for f in /var/log/*.log; do
    [[ -f "$f" ]] || continue
    echo "Processing $f"
done

# C-style for
for ((i=0; i<5; i++)); do
    echo "index $i"
done

# while — read line by line (safest pattern)
while IFS= read -r line; do
    echo "> $line"
done < /etc/hosts

# until with retry
attempts=0
until curl -sf https://api.example.com/health > /dev/null; do
    ((attempts++))
    [[ $attempts -ge 5 ]] && { echo "giving up"; exit 1; }
    sleep 2
done

Warning: for f in $(ls *.log) is a classic anti-pattern. If filenames have spaces, they get split. Use for f in *.log (glob directly) instead.

Ten Practical Examples

1. Retry with exponential backoff:

URL="$1"
MAX_TRIES=5
DELAY=1
for ((try=1; try<=MAX_TRIES; try++)); do
    if curl -sfL "$URL" -o /tmp/out; then
        echo "Success on attempt $try"
        exit 0
    fi
    echo "Attempt $try failed, sleeping ${DELAY}s..."
    sleep "$DELAY"
    DELAY=$((DELAY * 2))
done
echo "All $MAX_TRIES attempts failed" >&2
exit 1

2. Wait for a port to be open (Docker Compose pattern):

HOST=db
PORT=5432
until (echo > /dev/tcp/"$HOST"/"$PORT") 2>/dev/null; do
    echo "Waiting for $HOST:$PORT..."
    sleep 1
done
echo "$HOST:$PORT is up"

3. Read a CSV line-by-line:

while IFS=, read -r name email role; do
    [[ "$name" == "name" ]] && continue   # skip header
    echo "User: $name <$email> ($role)"
done < users.csv

4. Interactive menu:

while true; do
    cat <<MENU
1) Show uptime
2) Show disk
q) Quit
MENU
    read -p "Choice: " ch
    case "$ch" in
        1) uptime ;;
        2) df -h ;;
        q|Q) break ;;
        *) echo "Invalid choice" ;;
    esac
done

Interview tip: A common screening question is "write a script to find the 5 largest files in a directory tree." Answer: find /path -type f -printf '%s %p\n' | sort -rn | head -5.

Bash Functions and Advanced

Once your scripts grow past 50 lines, you need functions, arrays, and disciplined error handling. This section covers the constructs that separate "shell script someone hacked together" from "production automation."

Function Definitions

# Style 1 — POSIX, works in /bin/sh too
greet() {
    echo "Hello, $1!"
}

# Style 2 — bash keyword form
function greet() {
    echo "Hello, $1!"
}

greet Pushkar     # Hello, Pushkar!

Function Arguments and Return Values

Inside a function, $1, $2, $@, $# refer to the function's arguments — not the script's. Bash return N only sets the exit status (0-255); to return data, echo it and capture with $():

get_kernel() { uname -r; }            # write to stdout
is_root()    { [[ $EUID -eq 0 ]]; }   # last command's exit status IS the return

KERNEL=$(get_kernel)
if is_root; then echo "Running as root"; fi

Local Variables

By default all bash variables are global. local (only valid inside functions) keeps a variable scoped:

counter=0
bump() {
    local counter=100    # shadows the global
    counter=$((counter + 1))
    echo "inside: $counter"
}
bump                # inside: 101
echo "outside: $counter"   # outside: 0

Caution: local var=$(some_cmd) masks the exit code of some_cmd because local itself succeeds. Under set -e, failures silently disappear. Safe idiom: declare first, assign second.
local var
var=$(some_cmd)   # now set -e catches a failure here

Arrays

servers=(web01 web02 db01 cache01)
echo "${servers[0]}"       # web01 (zero-indexed)
echo "${servers[@]}"       # all elements as separate words
echo "${#servers[@]}"      # 4 (length)
echo "${!servers[@]}"      # 0 1 2 3 (indices)
servers+=(db02 db03)         # append

for s in "${servers[@]}"; do echo "Server: $s"; done
echo "${servers[@]:1:2}"     # slice — from index 1, take 2
mapfile -t users < <(cut -d: -f1 /etc/passwd)   # read command output into array

Associative Arrays (Hash Maps)

Bash 4+ via declare -A. Ubuntu 24.04 ships bash 5.2:

declare -A env_url
env_url[dev]="https://dev.example.com"
env_url[prod]="https://example.com"
echo "${env_url[prod]}"
for key in "${!env_url[@]}"; do
    printf '%-10s -> %s\n' "$key" "${env_url[$key]}"
done
if [[ -v env_url[qa] ]]; then echo "qa defined"; else echo "qa missing"; fi

Exit Codes and Convention

Code	Convention
0	Success
1	General error (catch-all)
2	Misuse / invalid arguments
126	Command found but not executable
127	Command not found
128 + N	Killed by signal N (130 = Ctrl-C, 137 = SIGKILL)
255	Exit code out of range

Note: Follow the exit-code convention. CI systems (Jenkins, GitLab, GitHub Actions) only know "did the job succeed or fail" by exit code. If your script exit 0s on error, the pipeline shows green while the deploy actually failed. Always non-zero exit on any failure path.

trap — Cleanup on Exit/Signal

#!/usr/bin/env bash
set -euo pipefail
TMPDIR=$(mktemp -d)
cleanup() {
    local rc=$?
    echo "Cleaning up (exit=$rc)..."
    rm -rf "$TMPDIR"
    return "$rc"
}
trap cleanup EXIT                          # EXIT = whenever the script ends
trap 'echo "Interrupted"; exit 130' INT TERM   # INT = Ctrl-C, TERM = kill
echo "working in $TMPDIR"

getopts — Argument Parsing

while getopts "e:v:dh" opt; do
    case "$opt" in
        e) ENV="$OPTARG" ;;
        v) VERSION="$OPTARG" ;;
        d) DRY_RUN=1 ;;
        h) usage ;;
        *) usage ;;
    esac
done
shift $((OPTIND - 1))
[[ -z "$ENV" || -z "$VERSION" ]] && { echo "-e and -v required" >&2; exit 2; }

Heredocs and Here-Strings

NAME=Pushkar
cat <<EOF
Hello, $NAME
Today is $(date +%F)
EOF

cat <<'EOF'
Literal $NAME and $(date) — nothing expands here
EOF

grep -E '^[A-Z]' <<< "$LINE"     # here-string — feed a single string as stdin

The Source-Library Pattern

# File: lib/common.sh
log()  { printf '[%s] %s\n' "$(date +%FT%T)" "$*"; }
warn() { log "WARN: $*" >&2; }
die()  { log "FATAL: $*" >&2; exit 1; }
require_root() { [[ $EUID -eq 0 ]] || die "must run as root"; }

# File: deploy.sh
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/lib/common.sh"
require_root
log "Starting deploy"

Tip: Install shellcheck (sudo apt install shellcheck) and run it on every script before committing. It catches ~80% of bash bugs — quoting issues, unset variables, deprecated syntax, common SC2086 word-splitting traps. Hook it into your CI.

Real DevOps Bash Scripts

Below are complete, working scripts you would actually run in production. Each uses strict mode, functions, error handling, and proper logging. They cover the automation patterns you will encounter repeatedly as a DevOps engineer.

Real world: Every script here is based on patterns used in production: locking, retries, Slack alerts, S3 upload, rollback on health-check fail. Tweak the paths and service names for your environment.

Script 1: server_health_check.sh

One-shot dashboard of CPU, memory, disk, and top processes. Exits non-zero if any threshold is breached so it can drive alerting from cron.

#!/usr/bin/env bash
# server_health_check.sh — quick host health snapshot
set -euo pipefail
IFS=$'\n\t'

CPU_WARN=80
MEM_WARN=85
DISK_WARN=80
EXIT_CODE=0

hr() { printf '%s\n' "------------------------------------------------------------"; }

header() {
    hr
    printf '%s @ %s\n' "$(hostname -f)" "$(date +'%F %T %Z')"
    printf 'kernel: %s   uptime:%s\n' "$(uname -r)" "$(uptime -p)"
    hr
}

cpu_section() {
    local idle1 total1 idle2 total2 usage
    read -r _ a b c idle1 _ < /proc/stat
    total1=$((a + b + c + idle1))
    sleep 1
    read -r _ a b c idle2 _ < /proc/stat
    total2=$((a + b + c + idle2))
    usage=$(( 100 * ((total2 - total1) - (idle2 - idle1)) / (total2 - total1) ))
    printf 'CPU usage:  %d%%   load: %s\n' "$usage" "$(cut -d' ' -f1-3 /proc/loadavg)"
    (( usage > CPU_WARN )) && { echo "  WARN: cpu > ${CPU_WARN}%"; EXIT_CODE=1; }
}

mem_section() {
    local total used pct
    read -r _ total used _ < <(free -m | awk '/^Mem:/')
    pct=$(( 100 * used / total ))
    printf 'Memory:     %d / %d MiB  (%d%%)\n' "$used" "$total" "$pct"
    (( pct > MEM_WARN )) && { echo "  WARN: memory > ${MEM_WARN}%"; EXIT_CODE=1; }
}

disk_section() {
    printf 'Disks:\n'
    df -hP -x tmpfs -x devtmpfs | awk 'NR>1 { printf "  %-25s %5s used of %-5s on %s\n", $1, $5, $2, $6 }'
    while read -r usage mount; do
        (( usage > DISK_WARN )) && { echo "  WARN: $mount > ${DISK_WARN}% (${usage}%)"; EXIT_CODE=1; }
    done < <(df -P -x tmpfs -x devtmpfs | awk 'NR>1 { gsub("%","",$5); print $5, $6 }')
}

main() {
    header
    cpu_section
    mem_section
    disk_section
    hr
    exit "$EXIT_CODE"
}

main "$@"

Script 2: deploy_app.sh

Git pull, npm build, systemd restart, health check, and automatic rollback on failure. Uses a lockfile to prevent concurrent deploys.

#!/usr/bin/env bash
# deploy_app.sh — pull, build, restart, verify, rollback on fail
set -euo pipefail

APP_DIR="/opt/myapp"
SERVICE="myapp"
HEALTH_URL="http://127.0.0.1:3000/health"
LOCK="/var/run/deploy_${SERVICE}.lock"
HEALTH_TRIES=20

log() { printf '[%s] %s\n' "$(date +%FT%T)" "$*"; }
die() { log "FATAL: $*" >&2; exit 1; }

# Single-instance lock
exec 9> "$LOCK"
flock -n 9 || die "another deploy is in progress"

cd "$APP_DIR" || die "$APP_DIR not found"
OLD_SHA=$(git rev-parse HEAD)
log "current SHA: $OLD_SHA"

git fetch --quiet origin
git reset --hard origin/main
NEW_SHA=$(git rev-parse HEAD)
[[ "$OLD_SHA" == "$NEW_SHA" ]] && { log "already at latest, no-op"; exit 0; }
log "new SHA: $NEW_SHA"

rollback() {
    log "ROLLING BACK to $OLD_SHA"
    git reset --hard "$OLD_SHA"
    npm ci --omit=dev && npm run build || true
    systemctl restart "$SERVICE"
    die "deploy failed, rolled back"
}

npm ci --omit=dev || rollback
npm run build || rollback
systemctl restart "$SERVICE" || rollback

for ((i=1; i<=HEALTH_TRIES; i++)); do
    if curl -fsS -m 3 "$HEALTH_URL" > /dev/null; then
        log "healthy after $i probe(s)"
        log "deploy OK: $OLD_SHA -> $NEW_SHA"
        exit 0
    fi
    sleep 2
done

rollback

Script 3: service_monitor.sh

Watchdog for systemd units. Restarts down services and posts to Slack on incident. Designed for a cron-every-minute or systemd timer.

#!/usr/bin/env bash
# service_monitor.sh — auto-restart services and Slack on failure
set -euo pipefail

SERVICES=(nginx postgresql redis-server docker)
MAX_RESTART=3
SLACK_URL="${SLACK_WEBHOOK_URL:-}"
HOST=$(hostname -f)

log() { printf '[%s] %s\n' "$(date +%FT%T)" "$*"; }

slack() {
    [[ -z "$SLACK_URL" ]] && { log "SLACK_WEBHOOK_URL not set, skipping notify"; return; }
    local msg="$1"
    curl -fsS -m 5 -X POST -H 'Content-Type: application/json' \
        --data "{\"text\":\":rotating_light: [$HOST] $msg\"}" \
        "$SLACK_URL" > /dev/null || log "WARN: slack post failed"
}

check_service() {
    local svc="$1"
    if systemctl is-active --quiet "$svc"; then
        log "$svc: OK"; return 0
    fi
    log "$svc: DOWN — attempting restart"
    local attempt=1
    while ((attempt <= MAX_RESTART)); do
        systemctl restart "$svc" || true
        sleep $((attempt * 2))
        if systemctl is-active --quiet "$svc"; then
            log "$svc: recovered on attempt $attempt"
            slack "$svc was DOWN, restarted successfully on attempt $attempt"
            return 0
        fi
        ((attempt++))
    done
    log "$svc: FAILED after $MAX_RESTART attempts"
    slack "$svc is DOWN and failed to restart after $MAX_RESTART attempts"
    return 1
}

failures=0
for s in "${SERVICES[@]}"; do
    check_service "$s" || ((failures++))
done
exit "$failures"

Script 4: log_rotate.sh

Minimal logrotate alternative for ad-hoc log directories. Gzips logs older than N days, deletes anything older than M days. Safe for cron.

#!/usr/bin/env bash
# log_rotate.sh — compress old logs, delete very old ones
set -euo pipefail

LOG_DIR="${1:?Usage: $0 <dir> [compress_days] [delete_days]}"
COMPRESS_DAYS="${2:-7}"
DELETE_DAYS="${3:-30}"

log() { printf '[%s] %s\n' "$(date +%FT%T)" "$*"; }
[[ -d "$LOG_DIR" ]] || { log "FATAL: $LOG_DIR not found"; exit 1; }
log "Rotating in $LOG_DIR (gzip >${COMPRESS_DAYS}d, delete >${DELETE_DAYS}d)"

compressed=0
while IFS= read -r -d '' f; do
    gzip -- "$f"
    ((compressed++))
done < <(find "$LOG_DIR" -type f -name '*.log' -mtime +"$COMPRESS_DAYS" -print0)
log "Compressed $compressed file(s)"

deleted=$(find "$LOG_DIR" -type f \( -name '*.log' -o -name '*.log.gz' \) \
            -mtime +"$DELETE_DAYS" -print -delete | wc -l)
log "Deleted $deleted file(s)"

Script 5: log_analysis.sh

Parse nginx access logs for top IPs, top URLs, and 4xx/5xx counts. Drop-in incident-response tool.

#!/usr/bin/env bash
# log_analysis.sh — top IPs, URLs, status code breakdown from nginx logs
set -euo pipefail

LOG="${1:-/var/log/nginx/access.log}"
TOP_N="${2:-10}"

[[ -r "$LOG" ]] || { echo "cannot read $LOG" >&2; exit 1; }

if [[ "$LOG" == *.gz ]]; then READ="zcat"; else READ="cat"; fi

echo "=== Summary of $LOG ==="
TOTAL=$($READ "$LOG" | wc -l)
echo "Total requests: $TOTAL"
echo

echo "--- Top $TOP_N IPs ---"
$READ "$LOG" | awk '{ print $1 }' | sort | uniq -c | sort -rn | head -n "$TOP_N"

echo
echo "--- Top $TOP_N URLs ---"
$READ "$LOG" | awk '{ print $7 }' | sort | uniq -c | sort -rn | head -n "$TOP_N"

echo
echo "--- Status code breakdown ---"
$READ "$LOG" | awk '{
    code=$9
    counts[code]++
    if (code+0 >= 400 && code+0 < 500) c4xx++
    else if (code+0 >= 500) c5xx++
}
END {
    for (c in counts) printf "%5d %s\n", counts[c], c | "sort -rn"
    close("sort -rn")
    printf "\nTotal 4xx: %d\nTotal 5xx: %d\n", c4xx+0, c5xx+0
}'

Key Patterns to Internalize

Functions and libraries: Split helpers into lib/common.sh and source it. Define log(), warn(), and die() functions in every serious script.

trap for cleanup:

TMPDIR=$(mktemp -d)
cleanup() { rm -rf "$TMPDIR"; }
trap cleanup EXIT
trap 'echo "Interrupted"; exit 130' INT TERM

getopts for argument parsing:

while getopts "e:v:dh" opt; do
    case "$opt" in
        e) ENV="$OPTARG" ;;
        v) VERSION="$OPTARG" ;;
        d) DRY_RUN=1 ;;
        h) usage ;;
        *) usage ;;
    esac
done

Interview tip: When asked "design a deploy script," walk through: (1) idempotency, (2) lockfile to prevent concurrent runs, (3) capture old state for rollback, (4) health check after restart, (5) automatic rollback path, (6) logging and notifications. The deploy_app.sh above hits all six — internalize the pattern.

Warning: Never write rm -rf "$VAR/" without set -u and an explicit non-empty check. Empty variable + slash = root-level delete. Always: [[ -n "${DIR:-}" && -d "$DIR" ]] || die "bad DIR" before any destructive operation.

Linux Logs

For a DevOps engineer, logs are the single source of truth. When a production app misbehaves at 3 AM, you don't guess — you grep. On a Linux server you'll be living inside /var/log and journalctl. Master these and you can debug anything.

Note: Mental model — Traditional Linux services write plain-text files to /var/log. Modern systemd services write to the binary journal, queried with journalctl. On Ubuntu 24.04 you have both.

Key Log Files — Cheat Sheet

Path	Distro	What's in it
`/var/log/syslog`	Debian/Ubuntu	General system events — almost everything ends up here
`/var/log/messages`	RHEL/Rocky	RHEL equivalent of syslog
`/var/log/auth.log`	Debian/Ubuntu	SSH logins, sudo, PAM, su — every auth event
`/var/log/secure`	RHEL	RHEL equivalent of auth.log
`/var/log/kern.log`	Both	Kernel ring buffer messages (drivers, OOM kills)
`/var/log/apt/history.log`	Debian/Ubuntu	Every apt install/remove/upgrade
`/var/log/dpkg.log`	Debian/Ubuntu	Low-level package install/remove
`/var/log/nginx/access.log`	All	Nginx HTTP requests
`/var/log/nginx/error.log`	All	Nginx errors, upstream failures
`/var/log/postgresql/postgresql-*.log`	Ubuntu	PostgreSQL queries, errors, checkpoints
`/var/log/cloud-init.log`	Cloud VMs	EC2/GCE first-boot user-data output

journalctl — The systemd Journal

$ journalctl -e                       # jump to end (most recent)
$ journalctl -r                       # reverse — newest first
$ journalctl -n 50                    # last 50 lines
$ journalctl -f                       # follow (like tail -f)
$ journalctl -u nginx                 # only nginx.service
$ journalctl -u nginx -f              # follow nginx live
$ journalctl --since "1 hour ago"
$ journalctl --since today
$ journalctl -p err                   # priority err or worse
$ journalctl -b                       # this boot only
$ journalctl -b -1                    # previous boot
$ journalctl -k                       # kernel only (dmesg-style)
$ journalctl --disk-usage
$ sudo journalctl --vacuum-time=7d     # keep only last 7 days
$ sudo journalctl --vacuum-size=500M   # cap at 500 MB

Priority levels (numeric and named both work with -p): 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info, 7 debug.

logrotate — Stops Disks From Filling Up

Logs grow forever unless rotated. logrotate runs daily via /etc/cron.daily/logrotate and reads /etc/logrotate.conf plus per-service files in /etc/logrotate.d/.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14              # keep 14 old copies
    size 100M              # OR rotate when file > 100MB
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp adm
    sharedscripts
    postrotate
        systemctl reload myapp.service > /dev/null 2>&1 || true
    endscript
}

$ sudo logrotate -d /etc/logrotate.d/myapp      # dry run
$ sudo logrotate -f /etc/logrotate.d/myapp      # force run now

Tip: Without copytruncate or a postrotate reload, long-running daemons keep writing to the old (renamed) file descriptor — and you wonder why rotation did nothing. Always reload the service or use copytruncate.

rsyslog and Centralized Logging

rsyslog reads kernel and app messages and writes them to /var/log/* (or ships them to a remote server). Syntax: facility.priority destination.

auth,authpriv.*                 /var/log/auth.log
*.*  @@logserver.internal:514    # TCP to a central log server
$ logger -p local0.notice "Deploy v2.4.1 started by $USER"   # write to syslog

One server: grep is fine. Twenty servers: you need an aggregator.

Stack	Components	Best for
ELK / Elastic	Filebeat → Logstash → Elasticsearch → Kibana	Full-text search, rich dashboards
Grafana Loki	Promtail → Loki → Grafana	Cheap, label-based
AWS CloudWatch Logs	CW agent → CW Logs → Logs Insights	AWS-native, zero infra

Real DevOps Log Examples

$ awk '$9 ~ /^5/ {print $9}' /var/log/nginx/access.log | sort | uniq -c   # 5xx breakdown
$ awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head   # top IPs
$ sudo dmesg -T | grep -i "killed process"          # was the system OOM-killed?
$ systemctl --failed                                 # failed services
$ sudo du -sh /var/log/* | sort -h | tail -20        # disk usage per log dir

Real world: Tracing an SSH brute-force on a fresh VPS — auth.log balloons. Count failures and group by attacker IP:
$ sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
$ sudo grep "Invalid user" /var/log/auth.log | awk '{print $8}' | sort | uniq -c | sort -rn | head
Mitigation: install fail2ban, set PasswordAuthentication no in sshd_config, move SSH off port 22, or restrict via ufw allow from <your-IP> to any port 22.

System Monitoring

Monitoring is the second pillar after logs. When a customer says "the site is slow," you need to know within 30 seconds whether it's CPU, memory, disk I/O, or network.

uptime — Load Averages

$ uptime
 11:42:18 up 14 days,  3:22,  2 users,  load average: 1.42, 0.87, 0.63
$ nproc
4

The three numbers are the average run-queue length over 1, 5, and 15 minutes. Load 1.42 on a 4-core box ≈ 35% utilized. Load 8.0 on 4 cores = processes queueing. 1-min > 15-min means load is increasing.

free — Memory

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       2.1Gi       312Mi        85Mi       5.3Gi       5.3Gi
Swap:          2.0Gi          0B       2.0Gi

Caution: Trap for beginners: "free" looks scary low — that's not the number you want. Linux aggressively uses RAM for buff/cache (disk caches), reclaimable instantly. The number that matters is available.

vmstat and iostat

$ vmstat 2 5     # sample every 2s, 5 times
# watch: r (runnable, should be <= CPU count), si/so (swap, should be 0), wa (CPU% waiting on I/O)

$ sudo apt install -y sysstat
$ iostat -xz 2   # extended, skip idle devices
# %util near 100% = saturated. await > 20ms on SSD = problem.

sar — Historical Stats

sar (from sysstat) records metrics every 10 minutes, so you can look at yesterday's CPU.

$ sar -u 1 5          # live CPU
$ sar -r 1 5          # live memory
$ sar -n DEV 1 5      # live network per interface
$ sar -u -f /var/log/sysstat/sa25   # CPU on the 25th

dmesg and Hardware Inventory

$ sudo dmesg -T                     # human-readable timestamps
$ sudo dmesg -T | grep -i oom       # OOM killer events
$ lscpu                              # CPU model, cores, flags
$ lsblk                              # block devices & mounts
$ lspci -nnk | grep -A2 Ethernet    # NIC + driver in use

top, htop, and Friends

$ top         # press 1=per-core, M=sort by memory, P=sort by CPU, c=full command, k=kill
$ sudo apt install -y htop glances nload iftop iotop ncdu
$ htop        # color, scrollable, tree view (F5)
$ glances     # CPU+MEM+NET+DISK+containers in one screen
$ sudo iotop -o    # disk I/O per process; -o = only active
$ ncdu /var        # interactive du — navigate and delete junk

In top: RES (resident set size — actual physical RAM) is the number that matters; VIRT is mostly meaningless.

Real Monitoring Scenarios

$ watch -n 5 'ps -o pid,rss,vsz,cmd -p $(pgrep -f myapp)'   # watch for a memory leak
$ ps aux --sort=-%mem | head -11                            # top 10 by memory
$ ps aux --sort=-%cpu | head -11                            # top 10 by CPU
$ iostat -xz 2 5                                            # find I/O bottleneck
$ ss -tn state established | awk 'NR>1{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
$ ps aux | awk '$8 ~ /Z/ {print}'                           # find zombie processes

Tip: Interview answer for "server slow, what do you do?" — in this exact order: uptime (load), top/htop (process), free -h (memory), iostat -xz 2 (disk), ss -s (network), dmesg -T | tail (kernel). This is Brendan Gregg's USE Method: Utilization, Saturation, Errors.

Cron Jobs

Cron is the Unix scheduler — a daemon that wakes up every minute and runs jobs whose time has come. You'll use it for backups, certificate renewals, log shipping, health checks, and cleanup scripts. It's old (1975!) but still everywhere.

$ systemctl status cron           # Ubuntu/Debian
$ systemctl status crond          # RHEL

Crontab Syntax

# ┌───────────── minute        (0 - 59)
# │ ┌─────────── hour          (0 - 23)
# │ │ ┌───────── day of month  (1 - 31)
# │ │ │ ┌─────── month         (1 - 12)
# │ │ │ │ ┌───── day of week   (0 - 7, where 0 and 7 both = Sunday)
# │ │ │ │ │
# * * * * *  command-to-run

Symbol	Meaning	Example
`*`	Any value	`* * * * *` — every minute
`*/N`	Every N units	`/15 * * *` — every 15 minutes
`N-M`	Range	`0 9-17 * * *` — every hour, 9 AM to 5 PM
`N,M`	List	`0 8,12,18 * * *` — 8 AM, noon, 6 PM

Named shortcuts replace the whole 5-field expression: @reboot, @hourly, @daily, @weekly, @monthly, @yearly.

Tip: Cron expressions are easy to get subtly wrong. Validate them against a human-readable description and the next few run times with the Cron Expression Tester before committing a crontab.

Managing Crontabs

$ crontab -e                # edit YOUR crontab
$ crontab -l                # list YOUR crontab
$ crontab -ri               # delete with confirmation
$ sudo crontab -u deploy -e # edit deploy user's crontab

User crontabs live in /var/spool/cron/crontabs/<user> — always use crontab -e, never edit directly.

System Cron

System crontab files have an extra "user" field between the schedule and the command:

$ sudo cat /etc/crontab
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# m h dom mon dow user  command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || (cd / && run-parts --report /etc/cron.daily)

Path	Purpose
`/etc/crontab`	Main system schedule (with user field)
`/etc/cron.d/`	Drop-in files (packages install here)
`/etc/cron.hourly/` `.daily/` `.weekly/` `.monthly/`	Drop a script in, runs on schedule via `run-parts`

run-parts executes every executable file in a directory. Two gotchas: scripts must be executable (chmod +x), and filenames must not contain dots.

Cron Logs

$ sudo grep CRON /var/log/syslog | tail -20
$ sudo journalctl -u cron --since today

Common DevOps Use Cases

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=devops@example.com

# Daily Postgres backup at 02:00
0 2 * * * /usr/local/bin/backup-postgres.sh >> /var/log/backup.log 2>&1
# Renew Let's Encrypt certs twice a day
17 3,15 * * * /usr/bin/certbot renew --quiet --deploy-hook "systemctl reload nginx"
# Health check every 5 minutes
*/5 * * * * /usr/local/bin/healthcheck.sh || /usr/local/bin/slack-alert.sh "API down"
# Clean Docker dangling images weekly
30 4 * * 1 /usr/bin/docker system prune -af --filter "until=168h" >> /var/log/docker-prune.log 2>&1

Best Practices and Common Mistakes

PATH is minimal. Cron's default PATH is just /usr/bin:/bin. Your docker, aws, pg_dump, node may not be found. Set PATH= at the top of the crontab or use full paths.
No shell aliases / .bashrc. Cron runs /bin/sh non-interactively — no rc files are sourced.
Working directory is $HOME. Use absolute paths or cd /opt/app && first.
Always redirect output. Unredirected stdout/stderr becomes mail to the user. Use >> /var/log/myjob.log 2>&1.
Use scripts, not inline one-liners — testable and version-controllable.
Idempotency / overlap. Use a lockfile: flock -n /tmp/myjob.lock -c '/path/to/job.sh'.
% is special — a literal % must be escaped as \%.

Caution: The #1 cron PATH gotcha: your script works when you run ./backup.sh by hand, but cron says docker: command not found. That's because your shell's PATH includes /usr/local/bin but cron's doesn't. Always set PATH= at the top of the crontab OR call binaries by absolute path.

The at Command

Where cron is for recurring jobs, at is for one-shot jobs. "Restart this service at 3 AM tonight." "Run this cleanup once, ninety minutes from now." Then forget it.

$ sudo apt install -y at
$ sudo systemctl enable --now atd

Basic Usage

$ at now + 5 minutes
at> echo "Hello from the future" > /tmp/hello.txt
at> systemctl restart nginx
at> ^D
job 3 at Tue May 26 11:47:00 2026

Time formats at understands: now + 2 hours, now + 1 day, 14:30, 14:30 tomorrow, midnight, noon, teatime (16:00), 3pm Friday, 02:00 next month.

Piping Commands In

$ echo "/opt/scripts/restart-app.sh" | at now + 1 hour
$ echo "shutdown -h now" | sudo at 23:00
$ at -f /opt/scripts/cleanup.sh now + 30 minutes

Managing Jobs

$ atq                       # list pending jobs
$ at -c 7                   # show contents of job 7
$ atrm 7                    # cancel job 7

batch is at's sibling: it runs queued commands only when the system load average drops below 1.5 — great for heavy one-off jobs.

Need	Use
Run every day / hour / week	`cron` (or systemd timer)
Run exactly once, in the future	`at`
"Restart server in 5 min if I don't cancel"	`at` (great safety net during config changes)
Run a heavy job when the system is idle	`batch`
Retries, dependencies, complex orchestration	systemd timer, Airflow/Argo

Tip: The "dead man's switch" trick: before editing risky SSH/firewall config, schedule a rollback: echo "cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config && systemctl restart ssh" | sudo at now + 5 minutes. Make your changes and test. If you lock yourself out, the job revokes them in 5 minutes. If everything works: sudo atrm <jobnum>.

systemd Timers

systemd timers are the modern replacement for cron. More verbose to set up, but they integrate with the rest of systemd: structured logging in the journal, dependency management, resource limits, retries, and the ability to catch up after downtime.

Note: Key idea — a timer is a unit that triggers another unit. You always create TWO files: foo.timer (when to run) and foo.service (what to run).

A Complete Example: Daily DB Backup

# /etc/systemd/system/myapp-backup.service
[Unit]
Description=Daily backup of myapp Postgres database
Wants=network-online.target
After=network-online.target postgresql.service

[Service]
Type=oneshot
User=postgres
ExecStart=/usr/local/bin/backup-myapp.sh
StandardOutput=journal
StandardError=journal

# /etc/systemd/system/myapp-backup.timer
[Unit]
Description=Run myapp DB backup daily at 02:00

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=5min
Unit=myapp-backup.service

[Install]
WantedBy=timers.target

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now myapp-backup.timer   # enable the TIMER, not the service
$ systemctl list-timers myapp-backup.timer
$ sudo systemctl start myapp-backup.service        # manually trigger once to test
$ journalctl -u myapp-backup.service -n 50          # view output

OnCalendar Syntax

Format: DayOfWeek Year-Month-Day Hour:Minute:Second.

`OnCalendar=`	When
`hourly` / `daily` / `weekly` / `monthly`	Common shortcuts
`--* 02:00:00`	Every day at 02:00
`Mon-Fri 09:00`	Weekdays at 09:00
`--01 04:00:00`	1st of every month at 04:00
`--* *:0/15:00`	Every 15 minutes
`2026-12-31 23:59:00`	Once: New Year's Eve 2026

Always validate before deploying:

$ systemd-analyze calendar "Mon-Fri 09:00"
$ systemd-analyze calendar --iterations=5 "*-*-* *:0/15:00"

Other Trigger Types

Directive	Meaning
`OnBootSec=15min`	15 minutes after boot
`OnUnitActiveSec=1h`	1 hour after the service last activated
`Persistent=true`	If a run was missed (server off), run on next boot
`RandomizedDelaySec=5min`	Spread a fleet of identical jobs — prevents thundering herd

$ systemctl list-timers              # next/last run for every timer
$ systemctl list-timers --all
$ systemctl cat myapp-backup.timer

Cron vs systemd Timer

Feature	cron	systemd timer
Setup effort	One line	Two unit files
Logs	Cron log + your redirects	Centralized journal — `journalctl -u name`
Dependencies	None	`After=`, `Wants=`, `Requires=`
Retries on failure	Write your own	`Restart=on-failure`
Catch up missed runs	anacron only, blunt	`Persistent=true`
Resource limits	None	`MemoryMax=`, `CPUQuota=`, etc.
Test/dry-run	None	`systemd-analyze calendar "..."` + `systemctl start`
Familiarity	Universal	Linux-only

Tip: Migration tip: if you already have a cron job, you can convert it to a timer in 10 minutes. Bonus: journalctl -u myjob gives you every run, exit code, and stdout/stderr without setting up your own log redirection. That alone justifies the two extra files.

Linux Security

Security in Linux is not a feature you turn on, it's a posture you maintain. As a DevOps engineer you're responsible for hardening servers, protecting credentials, and reducing attack surface.

Note: Security is a habit, not an event. Check daily: who is logging in, what ports are open, are the latest patches applied. Hardening once and forgetting is the biggest mistake — attackers only need one small gap.

Strong Passwords (PAM Password Quality)

PAM is the framework Linux uses for authentication. The pam_pwquality module enforces strong passwords system-wide via /etc/security/pwquality.conf:

minlen = 14
minclass = 4           # uppercase + lowercase + digit + symbol
dcredit = -1           # require at least 1 digit
ucredit = -1           # require at least 1 uppercase
ocredit = -1           # require at least 1 symbol
retry = 3
enforce_for_root = 1

Account lockout after failed attempts via pam_faillock in /etc/pam.d/common-auth (deny=5, unlock_time=900).

Caution: Test before you log out! A bad PAM config can lock you out of your own server. Always keep a second SSH session open while editing PAM files.

SSH Hardening Checklist (Recap)

Edit /etc/ssh/sshd_config:

PermitRootLogin no                 # disable root login
PasswordAuthentication no          # keys only — most important setting
Port 2222                          # reduce noise
AllowUsers deploy ops-alice
MaxAuthTries 3
LoginGraceTime 30
ClientAliveInterval 300
X11Forwarding no
PermitEmptyPasswords no

$ ssh-keygen -t ed25519 -a 100 -C "pushkar@workstation"
$ sudo sshd -t                      # test config syntax
$ sudo systemctl reload ssh

Firewall

# ufw — Ubuntu's friendly front-end
$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow 2222/tcp comment 'SSH'
$ sudo ufw allow 80,443/tcp
$ sudo ufw allow from 192.168.1.0/24 to any port 5432   # Postgres LAN only
$ sudo ufw limit 2222/tcp           # rate-limit SSH
$ sudo ufw enable
$ sudo ufw status verbose

Firewall	Used By	Notes
ufw	Ubuntu, Debian	Simple front-end to iptables/nftables
iptables	Legacy everywhere	Powerful, being replaced by nftables
nftables	Modern kernel default	Unified, faster, atomic rules
firewalld	RHEL, Fedora	Zone-based, runtime + permanent

# firewalld (RHEL/Rocky)
$ sudo firewall-cmd --add-service=https --permanent
$ sudo firewall-cmd --add-port=2222/tcp --permanent
$ sudo firewall-cmd --reload

fail2ban — Intrusion Prevention

fail2ban scans log files and bans IPs that show malicious behaviour by updating firewall rules dynamically.

# /etc/fail2ban/jail.local
[sshd]
enabled  = true
port     = 2222
maxretry = 3
bantime  = 24h

$ sudo systemctl enable --now fail2ban
$ sudo fail2ban-client status sshd
$ sudo fail2ban-client set sshd unbanip 203.0.113.45

Real world: On any internet-facing VM you'll see SSH brute-force attempts within minutes of booting. fail2ban + non-standard port + key-only auth reduces auth.log from thousands of failure lines per day to near-zero. It's the single highest-leverage hardening step.

SELinux vs AppArmor — Mandatory Access Control

Standard Unix permissions are discretionary (the owner decides access). MAC adds a second layer: even root cannot violate kernel-enforced policies.

Feature	SELinux	AppArmor
Default on	RHEL, Fedora, CentOS	Ubuntu, Debian, SUSE
Policy style	Labels on inodes	Path-based profiles
Granularity	Very fine-grained	Easier to read & write

# SELinux
$ getenforce                       # Enforcing | Permissive | Disabled
$ sudo setenforce 0                # Permissive (logs only)
$ ls -Z /var/www/html              # show SELinux contexts
$ sudo restorecon -Rv /var/www/html

# AppArmor
$ sudo aa-status
$ sudo aa-complain /etc/apparmor.d/usr.bin.nginx   # learning mode
$ sudo aa-enforce  /etc/apparmor.d/usr.bin.nginx

Caution: Don't disable MAC because "the app won't work." Set it to Permissive/Complain mode, reproduce the issue, read the audit log, then write a targeted policy exception. Disabling SELinux/AppArmor in production is a major red flag in security audits.

File Integrity — Hashes

$ md5sum ubuntu-24.04.iso          # FAST but cryptographically broken — checksums only
$ sha256sum ubuntu-24.04.iso       # the standard for downloads
$ sha512sum kernel.tar.xz          # extra paranoid
$ sha256sum -c SHA256SUMS          # verify against a published checksum file

Tip: Need to hash a string or small file quickly, or compare a download against its published digest without dropping to a shell? The Hash Generator produces MD5/SHA-1/SHA-256/SHA-512 in the browser.

GPG Basics

GPG provides encryption, signing, and key management — used everywhere from APT repo signing to package release verification.

$ gpg --full-generate-key          # use ed25519 or RSA 4096
$ gpg --encrypt --armor -r alice@example.com secrets.txt
$ gpg --decrypt secrets.txt.asc > secrets.txt
$ gpg --detach-sign --armor release-1.4.0.tar.gz
$ gpg --verify kubectl.sha256.sig kubectl.sha256   # verify upstream releases

Finding Privilege-Escalation Vectors

$ sudo find / -perm -4000 -type f -xdev 2>/dev/null   # SUID binaries
$ sudo find / -perm -2000 -type f -xdev 2>/dev/null   # SGID binaries
$ sudo find / -perm -0002 -type f -xdev 2>/dev/null   # world-writable files (red flag)
$ sudo find / -nouser -o -nogroup 2>/dev/null         # files with no owner

Automated Updates

Unpatched systems are the #1 source of compromise. Automate it.

# Ubuntu
$ sudo apt install unattended-upgrades apt-listchanges
$ sudo dpkg-reconfigure --priority=low unattended-upgrades
# RHEL
$ sudo dnf install dnf-automatic
$ sudo systemctl enable --now dnf-automatic.timer

Real DevOps Security Checklist

Update OS and patches — unattended-upgrades / dnf-automatic; reboot for kernel CVEs.
Disable root SSH and use key-only authentication (ed25519).
Firewall default deny — open only required ports.
fail2ban on SSH and other internet-facing services.
Audit users — remove unused accounts, lock service accounts, rotate keys.
Disable unused services — systemctl list-unit-files --state=enabled, mask what you don't need.
Centralized logs — ship to ELK/Loki/CloudWatch; local logs can be wiped by attackers.
Time sync — chrony or systemd-timesyncd; broken clocks break TLS and audit timestamps.
SELinux/AppArmor enforcing — never permanently disable MAC.
Vulnerability scans — Lynis (sudo lynis audit system), OpenSCAP, Trivy for containers.
Backups + tested restore — an untested backup is a hope, not a backup.
Least privilege — sudo whitelists, no shared accounts, separate prod/staging credentials.
Container security — non-root USER, drop capabilities, read-only rootfs, scan images (Trivy/Grype), use distroless/minimal base images.
Secrets management — Vault, AWS Secrets Manager, sealed-secrets; never commit secrets to git.
Monitor auth events — alert on failed sudo, new SSH keys, sudoers changes.

Real world: When you join a new DevOps team, run Lynis on a staging box in week one. It generates a 100+ point hardening report. Even mature shops typically score 60-70/100 — the gap is your low-hanging-fruit roadmap and an instant credibility builder.

Troubleshooting Methodology

Troubleshooting is the skill that separates a junior DevOps from a senior. Tools you can google. Methodology you have to internalise.

The Systematic Approach

Define the problem clearly. "The website is slow" is not a definition. "p95 latency on /api/checkout went from 200ms to 3s at 14:02 UTC for all users" is.
Gather data. Logs, metrics, kernel messages (dmesg), recent changes (deploys, config), system state (top, df, ss).
Reproduce it. Either in staging, or get exact steps.
Form a hypothesis. One thing at a time.
Test with the smallest change. Don't change five things at once.
Document. Symptom, root cause, fix, prevention.

Note: The biggest mistake is skipping root cause in the rush to fix. A server restart gives 5 minutes of relief, but if you didn't catch the root cause the issue returns in 3 days — and by then the logs have rotated. Slow down to go fast.

Where to Start, by Symptom

Symptom Area	First Places to Look
Boot issues	`journalctl -b`, `journalctl -b -1`, `dmesg`, `systemd-analyze blame`
Auth / SSH	`/var/log/auth.log` (or `/var/log/secure`), `journalctl -u sshd`
Service problem	`systemctl status <svc>`, `journalctl -u <svc> -n 200`
Disk	`df -h`, `df -i`, `du -sh /*`, `lsblk -f`, `iostat -xz 1`
Network	`ip a`, `ip r`, `curl -v`, `ss -tulpn`, `dig`, `tcpdump`
Performance	`top`, `vmstat 1`, `iostat -xz 1`, `sar -u 1 5`, `pidstat 1`
Memory / OOM	`free -h`, `dmesg \

Common Troubleshooting Scenarios

Server slow / not responding:
{% raw %}

$ uptime; top -o %CPU; top -o %MEM; iostat -xz 1 5; vmstat 1 5; ss -s

Identify the bottleneck (CPU / IO / memory / network), then kill/throttle the offender, scale resources, or add caching. 80% of "server slow" tickets are: a runaway log filling disk, a query without an index, or memory pressure causing swap thrashing — check disk and swap first.

Service won't start:

$ sudo systemctl status nginx
$ sudo journalctl -u nginx -n 100 --no-pager
$ sudo nginx -t                  # config syntax check
$ sudo ss -tulpn | grep :80      # port already in use?

Common causes: config syntax error, port already bound, missing dependency in After=, wrong User=, SELinux/AppArmor denial.

SSH "Permission denied (publickey)":

$ ssh -vvv -i ~/.ssh/id_ed25519 deploy@server     # which key was tried?
$ stat -c '%a %U %G %n' ~/.ssh ~/.ssh/authorized_keys
# Required: ~ = 755/750, .ssh = 700, authorized_keys = 600, owned by the user

SSH "Connection refused" vs "timed out": Refused = something at the destination actively rejected (sshd down, or firewall REJECT). Timed out = packets never arrived (routing, cloud security group, or a DROP rule). AWS/GCP/Azure security groups are the most common culprit.

Out of memory (OOM killer):

$ dmesg -T | grep -i 'killed process'
$ journalctl -k | grep -i 'oom-killer'
$ free -h ; cat /proc/swaps

Fix: add swap (short term), increase memory, fix the leaking app, or set MemoryMax= in the systemd unit. Java/Node in containers often get OOM-killed because they read the host's memory, not the cgroup limit — set -Xmx or NODE_OPTIONS=--max-old-space-size to match the container limit.

Container not starting:

$ docker ps -a                           # find the dead container's exit code
$ docker logs --tail=200 myapp
$ docker run --rm -it --entrypoint sh myapp:latest   # poke inside

Common exit codes: 0 = clean exit, 1 = app error, 125 = docker daemon error, 126 = not executable, 127 = command not found, 137 = OOM-killed, 139 = segfault, 143 = SIGTERM. Volume permission mismatch (UID inside container vs host) is the #1 silent failure in dev environments.

Power Tools

Tool	Use For	Example
`strace`	Trace system calls	`strace -f -e openat,connect -p 1234`
`lsof`	List open files, sockets, FDs	`lsof -i :8080`, `lsof +L1`
`perf`	CPU profiling, flame graphs	`sudo perf top`
`tcpdump`	Packet capture	`sudo tcpdump -i any -w cap.pcap port 443`
`bpftrace`	eBPF tracing	`sudo execsnoop-bpfcc`

The USE Method

Brendan Gregg's USE method: for every resource (CPU, memory, disk IO, disk capacity, network IO, file descriptors), check Utilization, Saturation, and Errors.

Real world: In every senior DevOps interview you'll get an open-ended scenario: "Production API is returning 500s, walk me through what you do." They're testing whether you have a method: define → gather → hypothesise → test → document. Mention rollback as the first option before deep debugging in prod. That single framing puts you ahead of most candidates.

Common Interview Questions

These are the highest-frequency Linux questions asked in DevOps and SRE interviews, condensed to the answer that signals real understanding. Drill them out loud — if you can teach the answer in your own words, you own it.

1. Difference between Linux and Unix? Unix is a proprietary OS family from the 1970s (AT&T Bell Labs); Linux is a free, open-source, Unix-like kernel created by Linus Torvalds in 1991. Linux follows Unix design principles but shares no code with original Unix. macOS is certified Unix (BSD lineage); Linux is not.

2. Difference between the kernel and the operating system? The kernel is the core program that talks to hardware and manages CPU, memory, devices, and processes. The OS is the kernel plus userspace — shell, libraries, utilities — that makes it usable. Calling the kernel "the OS" is the classic blunder.

3. Difference between bash and sh? sh is the POSIX-standard minimal shell; bash is a superset with arrays, [[ ]], brace expansion, and the function keyword. On Debian/Ubuntu, /bin/sh is dash, not bash — so a #!/bin/sh script using bash-only features works on RHEL but breaks on Ubuntu. Match your shebang to the features you use.

4. How do Linux file permissions work? Three triplets (owner/group/other), each with read (4), write (2), execute (1). Set symbolically (chmod u+x) or octal (chmod 755). Special bits: setuid (run as owner), setgid (inherit group / run as group), sticky (only owner can delete, as on /tmp).

5. Hard link vs symbolic link? A hard link is a second directory entry pointing to the same inode — same FS only, can't link directories, survives deletion of any one name while link count > 0. A symlink is a separate file whose content is a path string, resolved at access time — can cross filesystems, can point to directories, dangles if the target is removed.

6. What happens from power-on to login prompt? BIOS/UEFI → POST → MBR/GPT boot sector → GRUB loads kernel + initramfs → kernel mounts / and starts PID 1 → systemd starts services in dependency order and mounts /etc/fstab → reaches multi-user.target → login (getty/sshd). Most production outages happen here.

7. What's the difference between a VM and a container? A VM virtualizes hardware and runs a full guest kernel. A container is a regular process on the host kernel isolated with namespaces (PID, net, mount, user) and cgroups (CPU, memory, I/O limits). Container = process; VM = virtual machine.

8. Difference between kill -9 and kill -15? -15 (SIGTERM) is a polite shutdown request the process can catch and clean up after. -9 (SIGKILL) is uncatchable and immediate — no cleanup, risk of corrupt state. Always try SIGTERM first.

9. Explain fork() and exec(). fork() duplicates the current process (child gets a new PID). exec() replaces the child's memory image with a new program (same PID). The shell uses fork+exec+wait to run every command.

10. Disk shows full but df reports free space — what's wrong? Either inode exhaustion (df -i shows 100% IUse) or a process holding a deleted-but-open file (lsof | grep deleted). For inodes, delete the millions of tiny files; for the open file, restart the holding process.

11. How do you debug a service that won't start? systemctl status <svc>, then journalctl -u <svc> -n 100, then validate config (nginx -t, sshd -t), then check the port isn't already bound (ss -tulpn). Common causes: config syntax, port conflict, wrong User=, missing dependency, MAC denial.

12. What's the difference between cron and a systemd timer? Both schedule recurring work. systemd timers add journal logging, dependency ordering (After=/Wants=), retries (Restart=), catch-up of missed runs (Persistent=true), and resource limits. Cron is simpler and universal. Use timers when you need observability and dependencies.

13. How do you harden SSH on a production server? PasswordAuthentication no (key-only), PermitRootLogin no, restrict with AllowUsers, lower MaxAuthTries, non-standard port, add fail2ban, and use ed25519 keys. Always sshd -t and keep a second session open before reloading.

14. A server is slow — walk me through it. Triage in order: uptime (load vs core count), top/htop (CPU/mem hogs), free -h (look at available, not free), iostat -xz 2 (disk %util/await), ss -s (connections), dmesg -T | tail (kernel/OOM). This is the USE method: Utilization, Saturation, Errors.

15. How do you find what's listening on a port / using a port? ss -tlnp | grep :80 or lsof -i :80 shows the listener and PID. To free it, identify the PID then stop that service (fuser -k 80/tcp as a blunt instrument).

Tip: In interviews, never just say "I don't know" — add "but here's how I'd find out." Confidence with humility beats fake expertise. For an experienced developer pivoting into DevOps, that maturity is a genuine differentiator.

Quick Reference Cheat Sheets

Bookmark this section — these are the reference sheets you will hit Ctrl+F on during your first 6 months on the job. Print them, paste them on your wall, whatever works. Real production-grade reference, no fluff.

1. Top 100 Linux Commands (Grouped)

Files & Directories

Command	Purpose	Example
`ls`	List directory	`ls -lah --color`
`cd`	Change directory	`cd -` (previous)
`pwd`	Print working dir	`pwd -P` (resolve symlinks)
`mkdir`	Make directory	`mkdir -p a/b/c`
`rmdir`	Remove empty dir	`rmdir empty_folder`
`rm`	Remove files	`rm -rf node_modules`
`cp`	Copy	`cp -av src/ dst/`
`mv`	Move/rename	`mv old.txt new.txt`
`ln`	Link (hard/soft)	`ln -s /opt/app /usr/local/app`
`touch`	Create empty / update mtime	`touch app.log`
`stat`	File metadata	`stat /etc/passwd`
`file`	Detect file type	`file binary.bin`
`find`	Find files	`find . -name "*.log" -mtime +7`
`locate`	Fast filename search	`locate sshd_config`
`tree`	Directory tree	`tree -L 2`
`basename`	Strip path	`basename /a/b/c.txt`
`dirname`	Strip filename	`dirname /a/b/c.txt`
`realpath`	Resolve absolute path	`realpath ./app`

Text Processing

Command	Purpose	Example
`cat`	Print file	`cat /etc/os-release`
`tac`	Reverse cat	`tac access.log`
`less`	Paged viewer	`less +F app.log` (follow)
`more`	Old pager	`more big.txt`
`head`	First N lines	`head -n 20 file`
`tail`	Last N lines	`tail -f -n 100 app.log`
`grep`	Pattern search	`grep -rEn "TODO
{% raw %}`egrep / fgrep`	ERE / fixed	`grep -E "a
{% raw %}`sed`	Stream editor	`sed -i 's/dev/prod/g' app.conf`
`awk`	Field processor	`awk '{print $1,$9}' access.log`
`cut`	Extract columns	`cut -d: -f1 /etc/passwd`
`sort`	Sort lines	`sort -u -k2,2n`
`uniq`	Dedupe (sorted)	`sort file
{% raw %}`wc`	Count	`wc -l *.py`
`tr`	Translate chars	`tr 'a-z' 'A-Z'`
`tee`	Write & pass through	`cmd
{% raw %}`diff`	Compare	`diff -u a b`
`patch`	Apply diff	`patch -p1 < fix.patch`
`jq`	JSON processor	`jq '.items[].name' data.json`
`yq`	YAML processor	`yq '.services' docker-compose.yml`
`xargs`	Build commands	`find . -name "*.bak"

Process

Command	Purpose	Example
{% raw %}`ps`	Process snapshot	`ps -ef` or `ps aux`
`top`	Live processes	`top -o %CPU`
`htop`	Better top	`htop` (F6 sort, F9 signal)
`kill`	Send signal	`kill -9 1234`
`pkill`	Kill by name	`pkill -f gunicorn`
`killall`	Kill all by name	`killall nginx`
`pgrep`	Find PIDs	`pgrep -af java`
`nice`	Run with priority	`nice -n 10 ./batch.sh`
`renice`	Change priority	`renice 5 -p 1234`
`nohup`	Survive logout	`nohup ./run.sh &`
`jobs`	Shell jobs	`jobs -l`
`bg / fg`	Background/foreground	`bg %1` / `fg %1`
`strace`	Trace syscalls	`strace -p 1234 -f -e network`
`lsof`	Open files	`lsof -i :8080`

Network

Command	Purpose	Example
`ip`	Modern net tool	`ip -br a` / `ip r`
`ifconfig`	Legacy	`ifconfig eth0`
`ping`	ICMP test	`ping -c 4 google.com`
`traceroute`	Hop trace	`traceroute -n 8.8.8.8`
`mtr`	Live traceroute	`mtr 1.1.1.1`
`dig`	DNS lookup	`dig +short A example.com`
`nslookup`	DNS query	`nslookup example.com 8.8.8.8`
`host`	Simple DNS	`host example.com`
`curl`	HTTP client	`curl -fsSL https://x.com`
`wget`	Download	`wget -c https://x.com/file`
`ss`	Sockets	`ss -tulnp`
`netstat`	Legacy ss	`netstat -tulnp`
`nc`	Netcat	`nc -zv host 443`
`nmap`	Port scanner	`nmap -sV -p 1-1000 host`
`tcpdump`	Packet capture	`tcpdump -i eth0 port 80`
`iptables`	Firewall (legacy)	`iptables -L -n -v`
`nft`	nftables	`nft list ruleset`

System

Command	Purpose	Example
`uname`	Kernel info	`uname -a`
`hostnamectl`	Hostname/OS info	`hostnamectl`
`uptime`	Load avg	`uptime`
`w / who`	Logged users	`w`
`last`	Login history	`last -n 20`
`date`	Date/time	`date -u +%Y-%m-%dT%H:%M:%SZ`
`timedatectl`	Timezone/NTP	`timedatectl set-timezone Asia/Kolkata`
`systemctl`	Service control	`systemctl status nginx`
`journalctl`	Logs	`journalctl -u nginx -f`
`dmesg`	Kernel ring buffer	`dmesg -T
{% raw %}`free`	Memory	`free -h`
`vmstat`	Virtual memory stats	`vmstat 1 5`
`iostat`	IO stats	`iostat -xz 1`
`sar`	Historical perf	`sar -u 1 5`
`env`	Env vars	`env
{% raw %}`which / type`	Locate binary	`type -a ls`

Security & Users

Command	Purpose	Example
`sudo`	Privilege escalation	`sudo -i`
`su`	Switch user	`su - deploy`
`passwd`	Change password	`sudo passwd alice`
`useradd`	Add user	`useradd -m -s /bin/bash alice`
`usermod`	Modify user	`usermod -aG docker alice`
`userdel`	Delete user	`userdel -r alice`
`groupadd`	Add group	`groupadd devops`
`id`	Show IDs	`id alice`
`chage`	Pass aging	`chage -l alice`
`getfacl/setfacl`	ACLs	`setfacl -m u:alice:rw file`
`fail2ban-client`	Ban abusers	`fail2ban-client status sshd`
`ufw / firewalld`	Firewall frontends	`ufw allow 22/tcp`
`openssl`	Crypto Swiss-army	`openssl s_client -connect host:443`

Compression

Command	Purpose	Example
`tar`	Archive	`tar -czvf a.tgz dir/`
`tar -x`	Extract	`tar -xzvf a.tgz`
`gzip / gunzip`	gzip files	`gzip -9 big.log`
`bzip2 / bunzip2`	bzip2	`bzip2 file`
`xz / unxz`	xz (best ratio)	`xz -T0 huge.tar`
`zip / unzip`	Zip format	`unzip -d out/ a.zip`
`zcat / zless`	View gzipped	`zcat app.log.gz

Permissions

Command	Purpose	Example
{% raw %}`chmod`	Change mode	`chmod 750 deploy.sh`
`chown`	Change owner	`chown -R deploy:deploy /opt/app`
`chgrp`	Change group	`chgrp www-data /var/www`
`umask`	Default mask	`umask 027`

Disk

Command	Purpose	Example
`df`	Filesystem usage	`df -hT`
`du`	Directory usage	`du -sh *
{% raw %}`lsblk`	Block devices	`lsblk -f`
`fdisk`	Partition tool	`sudo fdisk -l`
`parted`	GPT partitioning	`parted /dev/sdb print`
`mkfs`	Make filesystem	`mkfs.ext4 /dev/sdb1`
`mount / umount`	Mount FS	`mount /dev/sdb1 /mnt`
`blkid`	UUIDs/labels	`blkid /dev/sda1`
`ncdu`	TUI disk usage	`ncdu /var`
`fsck`	Filesystem check	`fsck -y /dev/sda1`
`lvm tools`	LVM	`pvs / vgs / lvs`

Miscellaneous

Command	Purpose	Example
`man / info`	Manuals	`man 5 crontab`
`tldr`	Practical examples	`tldr tar`
`history`	Shell history	`history
{% raw %}`alias`	Define alias	`alias ll='ls -lah'`
`watch`	Repeat command	`watch -n 2 'df -h'`
`time`	Time a command	`time ./build.sh`
`timeout`	Limit runtime	`timeout 30s ./probe`
`yes`	Repeat string	`yes
{% raw %}`screen / tmux`	Terminal multiplexer	`tmux new -s work`
`echo / printf`	Print	`printf "%s\n" "$VAR"`
`true / false`	Exit code helpers	`while true; do ...; done`

2. Vim Cheat Sheet

Modes

Mode	Enter	Purpose
Normal	`Esc`	Default; movement & commands
Insert	`i I a A o O`	Type text
Visual	`v`	Character selection
Visual line	`V`	Line selection
Visual block	`Ctrl+v`	Column / rectangular edit
Command	`:`	Ex commands (`:w`, `:q`)
Replace	`R`	Overwrite mode
Terminal	`:terminal`	Embedded shell

Navigation

Key	Action
`h j k l`	Left, down, up, right
`w / b`	Next / previous word
`e / ge`	End of word / previous word end
`0 / ^ / $`	Line start / first non-blank / line end
`gg / G`	File top / file bottom
`:42` or `42G`	Jump to line 42
`Ctrl+d / Ctrl+u`	Half page down / up
`Ctrl+f / Ctrl+b`	Full page down / up
`{ / }`	Previous / next blank line
`%`	Match bracket
`f x / F x`	Find char forward / backward
`* / #`	Search word under cursor forward / backward

Editing

Key	Action
`i / a`	Insert before / after cursor
`I / A`	Insert at line start / end
`o / O`	New line below / above
`x / X`	Delete char forward / backward
`dd / D`	Delete line / to end of line
`dw / d$ / dG`	Delete word / to EOL / to EOF
`yy / Y`	Yank line
`p / P`	Paste after / before
`cc / C`	Change line / to EOL
`cw / ciw / ci"`	Change word / inner word / inside quotes
`r / R`	Replace one char / replace mode
`u / Ctrl+r`	Undo / redo
`.`	Repeat last change
`>> / <<`	Indent / dedent line
`J`	Join line below

Search & Replace

/pattern        " Search forward
?pattern        " Search backward
n / N           " Next / previous match
:noh            " Clear highlight

:s/old/new/         " Replace first on line
:s/old/new/g        " Replace all on line
:%s/old/new/g       " Replace all in file
:%s/old/new/gc      " Confirm each
:%s/old/new/gI      " Case-sensitive (capital I)
:5,15s/old/new/g    " Replace in line range
:'<,'>s/old/new/g    " Replace in visual selection
:g/pattern/d        " Delete all lines matching
:v/pattern/d        " Delete lines NOT matching

Visual Mode

Key	Action
`v / V / Ctrl+v`	Char / line / block selection
`d / y / c`	Delete / yank / change selection
`> / <`	Indent / dedent block
`~`	Toggle case
`u / U`	Lower / upper case selection
`:'<,'>sort`	Sort selected lines
`Ctrl+v` then `I text Esc`	Insert column prefix on all lines

Advanced — Registers, Marks, Macros, Splits, Tabs

" Registers (named clipboards)
"ayy            " Yank line to register a
"ap            " Paste register a
:reg            " Show all registers
"+y             " Yank to system clipboard (needs +clipboard)
"+p             " Paste from system clipboard

" Marks
ma              " Set mark a at cursor
'a              " Jump to line of mark a
`a              " Jump to exact position of mark a
:marks          " List all marks

" Macros
qa              " Start recording macro into a
...do stuff...
q               " Stop recording
@a              " Play macro a
@@              " Replay last macro
10@a            " Play macro 10 times

" Splits
:split file     " Horizontal split  (or :sp)
:vsplit file    " Vertical split    (or :vsp)
Ctrl+w h/j/k/l  " Move between splits
Ctrl+w =        " Equalize sizes
Ctrl+w _        " Maximize height
Ctrl+w |        " Maximize width
Ctrl+w c        " Close split

" Tabs
:tabnew file    " Open in new tab
gt / gT         " Next / previous tab
:tabclose       " Close current tab
:tabs           " List tabs

3. File Permissions Reference

Octal Table (000–777)

Octal	Binary	rwx	Meaning
0	000	`---`	No access
1	001	`--x`	Execute only
2	010	`-w-`	Write only
3	011	`-wx`	Write + execute
4	100	`r--`	Read only
5	101	`r-x`	Read + execute
6	110	`rw-`	Read + write
7	111	`rwx`	Full access

Three digits = owner, group, others. So 750 = rwxr-x--- (owner full, group r-x, others nothing).

Special Bits

Bit	Octal	Symbol	Effect
setuid	4000	`s` in owner-x	Run binary as file owner (e.g. `passwd`)
setgid	2000	`s` in group-x	On dir: new files inherit group
sticky	1000	`t` in others-x	Only owner can delete (e.g. `/tmp`)

chmod 4755 /usr/local/bin/myapp      # setuid + 755
chmod 2775 /shared/uploads           # setgid + 775
chmod 1777 /tmp                      # sticky + 777

Common Permission Patterns

Mode	Use Case
`644`	Regular file (configs, code)
`600`	Sensitive file (SSH private key, secrets)
`640`	Owner rw, group read (shared config)
`755`	Executable script, directory
`750`	Private executable / private dir
`700`	Only owner (e.g. `~/.ssh`)
`444`	Read-only for everyone
`777`	Almost always wrong — security smell

chmod Symbolic Operators

Who	Op	Perm
`u` user	`+` add	`r` read
`g` group	`-` remove	`w` write
`o` others	`=` set	`x` execute
`a` all		`X` exec only on dirs/already-exec files

chmod u+x deploy.sh             # add execute for owner
chmod go-rwx secret.key         # remove all from group + others
chmod a=r README.md             # read-only for all
chmod -R u=rwX,g=rX,o= /app     # safe recursive (capital X)
chmod u+s /usr/bin/myapp        # setuid

chown / chgrp

chown alice file                 # change owner
chown alice:devs file            # owner + group
chown :devs file                 # only group
chown -R deploy:deploy /opt/app  # recursive
chown --reference=other.txt file # copy ownership from other.txt
chgrp www-data /var/www          # change group only

4. Bash Scripting Cheat Sheet

Variables & Parameters

name="alice"           # no spaces around =
echo "$name"           # always quote variable expansions
echo "${name}_log"     # disambiguate with braces
readonly PI=3.14       # constant
unset name             # delete variable
export PATH="$PATH:/opt/bin"   # export to children

# Positional params inside a script
$0       # script name
$1..$9   # args 1..9
${10}    # 10th arg (need braces)
$#       # number of args
$@       # all args (each quoted separately when "$@")
$*       # all args (one string when "$*")
$?       # exit code of last command
$$       # current PID
$!       # PID of last backgrounded command

# Parameter expansion
${var:-default}    # use default if var unset/empty
${var:=default}    # also assign default
${var:?error}      # error if unset
${var:+alt}        # use alt if var IS set
${#var}            # string length
${var:2:4}         # substring from pos 2, length 4
${var/foo/bar}     # replace first foo with bar
${var//foo/bar}    # replace ALL foo
${var#prefix}      # strip shortest prefix
${var##prefix}     # strip longest prefix
${var%.txt}        # strip shortest suffix
${var%%.*}         # strip longest suffix

Test Operators

Category	Operator	Meaning
Integer	`-eq`	equal
Integer	`-ne`	not equal
Integer	`-lt`	less than
Integer	`-le`	less or equal
Integer	`-gt`	greater than
Integer	`-ge`	greater or equal
String	`=` / `==`	equal
String	`!=`	not equal
String	`-z`	zero-length (empty)
String	`-n`	non-empty
String	`< / >`	lexicographic (inside `[[ ]]`)
File	`-e`	exists
File	`-f`	is regular file
File	`-d`	is directory
File	`-L`	is symlink
File	`-r / -w / -x`	readable / writable / executable
File	`-s`	non-empty (size > 0)
File	`-O`	owned by you
File	`-G`	owned by your group
File	`a -nt b`	a newer than b
File	`a -ot b`	a older than b

Control Structures

# if / elif / else
if [[ $count -gt 10 ]]; then
  echo "many"
elif [[ -z "$name" ]]; then
  echo "no name"
else
  echo "ok"
fi

# case
case "$env" in
  prod|production)  echo "prod" ;;
  dev|development)  echo "dev"  ;;
  *)                echo "unknown"; exit 1 ;;
esac

# for
for f in *.log; do
  gzip "$f"
done
for i in {1..5}; do echo "$i"; done
for ((i=0; i<10; i++)); do echo "$i"; done

# while / until
while read -r line; do
  echo "got: $line"
done < file.txt

count=0
until [[ $count -ge 5 ]]; do
  ((count++))
done

# functions
deploy() {
  local env="$1"      # 'local' avoids polluting global scope
  echo "deploying to $env"
  return 0
}
deploy prod

Common Idioms

# Strict mode (put at top of every serious script)
set -euo pipefail
IFS=$'\n\t'

# Check command exists
if ! command -v jq >/dev/null 2>&1; then
  echo "jq required" >&2; exit 1
fi

# Retry loop with backoff
for attempt in 1 2 3 4 5; do
  curl -fsSL "$URL" && break
  sleep $((2 ** attempt))
done

# Parse args (getopts)
while getopts "e:vh" opt; do
  case "$opt" in
    e) ENV="$OPTARG" ;;
    v) VERBOSE=1 ;;
    h) usage; exit 0 ;;
    *) usage; exit 1 ;;
  esac
done
shift $((OPTIND - 1))

# Trap cleanup
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT INT TERM

# Read file line by line (safe)
while IFS= read -r line || [[ -n "$line" ]]; do
  process "$line"
done < input.txt

# Heredoc
cat <<EOF > /etc/myapp/config
host=$(hostname)
env=$ENV
EOF

Arrays

# Indexed array
servers=(web1 web2 web3)
echo "${servers[0]}"          # web1
echo "${servers[@]}"          # all elements
echo "${#servers[@]}"         # length = 3
servers+=(web4)               # append
unset 'servers[1]'            # delete element

for s in "${servers[@]}"; do echo "$s"; done

# Associative array (bash 4+)
declare -A user_role
user_role[alice]=admin
user_role[bob]=dev
echo "${user_role[alice]}"
for k in "${!user_role[@]}"; do
  echo "$k => ${user_role[$k]}"
done

Brace Expansion

echo file{1,2,3}.txt          # file1.txt file2.txt file3.txt
echo {a..e}                   # a b c d e
echo {01..05}                 # 01 02 03 04 05
echo {1..10..2}               # 1 3 5 7 9
mkdir -p project/{src,test,docs}/{api,web}
cp app.conf{,.bak}            # quick backup -> app.conf, app.conf.bak

5. Regular Expressions

BRE vs ERE Side-by-Side

Feature	BRE (grep, sed)	ERE (egrep, grep -E, sed -E, awk)
Alternation	`\	`
Grouping	``	`( )`
One or more	`\+`	`+`
Zero or one	`\?`	`?`
Repetition	`\{m,n\}`	`{m,n}`
Zero or more	`*`	`*`
Backrefs	`\1 \2`	`\1 \2`

Rule of thumb: use grep -E (ERE) — fewer backslashes, fewer tears.

Character Classes

Class	Matches
`.`	Any single char (except newline)
`[abc]`	a, b, or c
`[^abc]`	NOT a, b, c
`[a-z]`	Lowercase letters
`[A-Za-z0-9_]`	"word" chars
`\d` / `\D`	Digit / non-digit (PCRE)
`\w` / `\W`	Word / non-word (PCRE)
`\s` / `\S`	Whitespace / non-ws (PCRE)
`[[:alpha:]]`	POSIX alpha
`[[:digit:]]`	POSIX digit
`[[:space:]]`	POSIX whitespace
`[[:alnum:]]`	POSIX alphanumeric
`[[:upper:]]` / `[[:lower:]]`	POSIX case
`[[:punct:]]`	POSIX punctuation

Anchors & Quantifiers

Pattern	Meaning
`^`	Start of line
`$`	End of line
`\b`	Word boundary
`\B`	Non-word-boundary
`\A` / `\Z`	Start / end of input (PCRE)
`*`	0 or more
`+`	1 or more
`?`	0 or 1
`{3}`	Exactly 3
`{3,}`	3 or more
`{3,5}`	Between 3 and 5
`*?` `+?`	Lazy / non-greedy (PCRE)

10 Example Patterns

#	Matches	Pattern (ERE)
1	Email (pragmatic)	`[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}`
2	IPv4 address (loose)	`([0-9]{1,3}\.){3}[0-9]{1,3}`
3	IPv4 strict (0–255)	`((25[0-5]
4	URL (http/https)	{% raw %}`https?://[A-Za-z0-9._~:/?#\[\]@!$&'()*+,;=%-]+`
5	Date YYYY-MM-DD	`[0-9]{4}-(0[1-9]
6	Time HH:MM:SS (24h)	{% raw %}`([01][0-9]
7	ISO timestamp	{% raw %}`[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+)?(Z
8	E.164 phone number	{% raw %}`\+[1-9][0-9]{7,14}`
9	UUIDv4	`[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}`
10	Nginx access log IP	`^([0-9]{1,3}\.){3}[0-9]{1,3}`

# Practical usage
grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log
sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/REDACTED/g' file
awk '/ERROR|FATAL/ {print}' app.log

6. SSH Commands Cheat Sheet

Key Generation & Distribution

# Generate modern keypair (ed25519 preferred)
ssh-keygen -t ed25519 -C "pushkar@1buy.ai"
ssh-keygen -t rsa -b 4096 -C "fallback"     # legacy systems

# View public key
cat ~/.ssh/id_ed25519.pub

# Copy public key to server (easiest)
ssh-copy-id -i ~/.ssh/id_ed25519.pub deploy@10.0.0.5

# Manually
cat ~/.ssh/id_ed25519.pub | ssh deploy@host \
  'mkdir -p ~/.ssh && chmod 700 ~/.ssh && \
   cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys'

# Fingerprint
ssh-keygen -lf ~/.ssh/id_ed25519.pub

~/.ssh/config Snippets

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 3
  HashKnownHosts yes

Host prod-web
  HostName 10.0.0.5
  User deploy
  Port 2222
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes

Host bastion
  HostName bastion.1buy.ai
  User pushkar

# Jump through bastion to private host
Host db-private
  HostName 10.0.5.20
  User dbadmin
  ProxyJump bastion

# Now just: ssh prod-web   /   ssh db-private

Tunneling (L / R / D)

Flag	Direction	Use Case
`-L`	Local → Remote	Access remote service via local port
`-R`	Remote → Local	Expose local service on remote host
`-D`	Dynamic (SOCKS)	Use server as SOCKS5 proxy

# Local forward: localhost:5433 -> db.internal:5432 via bastion
ssh -L 5433:db.internal:5432 bastion
psql -h localhost -p 5433 -U app

# Remote forward: expose local dev server on remote:9000
ssh -R 9000:localhost:3000 user@remote

# Dynamic / SOCKS proxy on localhost:1080
ssh -D 1080 -C -N bastion
# Configure browser SOCKS5 -> localhost:1080

# Run tunnel in background, no shell
ssh -fN -L 5433:db:5432 bastion

SSH Agent

eval "$(ssh-agent -s)"          # start agent
ssh-add ~/.ssh/id_ed25519       # load key
ssh-add -l                      # list loaded keys
ssh-add -D                      # remove all
ssh -A user@host                # forward agent (use cautiously)

scp / sftp / rsync

# scp (simple)
scp file.txt user@host:/tmp/
scp -r dir/ user@host:/opt/
scp -P 2222 -i key.pem ...      # custom port, custom key
scp user@host:/etc/nginx.conf .

# sftp (interactive)
sftp user@host
> put localfile
> get remotefile
> ls -la
> bye

# rsync (best for repeated syncs)
rsync -avz --progress src/ user@host:/dst/
rsync -avz --delete src/ user@host:/dst/        # mirror (deletes extras)
rsync -avz -e "ssh -p 2222 -i key.pem" src/ host:/dst/
rsync -avz --exclude '.git' --exclude 'node_modules' src/ host:/dst/
rsync -avzn src/ host:/dst/                     # -n = dry run

sshd_config Hardening (server side)

# /etc/ssh/sshd_config
Port 2222                          # change default
PermitRootLogin no                 # never allow root login
PasswordAuthentication no          # keys only
PubkeyAuthentication yes
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding no
MaxAuthTries 3
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2
AllowUsers deploy pushkar
AllowGroups sshusers
Protocol 2
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

# Apply
sudo sshd -t                       # test config first!
sudo systemctl reload sshd

7. System Monitoring

top

top                           # launch
# Interactive keys:
P    sort by CPU
M    sort by memory
T    sort by time
N    sort by PID
k    kill process (asks PID + signal)
r    renice
c    toggle full command line
1    toggle per-CPU view
H    toggle threads
W    save config to ~/.toprc
q    quit
top -b -n 1                   # batch mode (for scripts)
top -p 1234,5678              # only specific PIDs
top -u deploy                 # only user deploy

htop

htop                          # nicer top
# F1 help  F2 setup  F3 search  F4 filter  F5 tree
# F6 sort  F7/F8 nice +/-  F9 kill  F10 quit
# Space = tag a process; u = filter by user

ps

ps aux                              # BSD style, all processes
ps -ef                              # System V style
ps -eLf                             # threads too
ps -eo pid,user,%cpu,%mem,cmd --sort=-%cpu | head
ps --ppid 1234                      # children of PID 1234
ps -C nginx                         # find by command name
pstree -p                           # process tree with PIDs

Memory & CPU

free -h                             # memory in human units
free -m -s 2                        # refresh every 2s
cat /proc/meminfo
cat /proc/cpuinfo

vmstat 1 5                          # 5 samples, 1 sec apart
# columns: r b | swpd free buff cache | si so | bi bo | in cs | us sy id wa st

iostat -xz 1                        # extended IO stats, skip idle
iostat -mx 2 5                      # MB/s, 5 samples

mpstat -P ALL 1 3                   # per-CPU stats

sar (historical perf)

sar -u 1 5                          # CPU
sar -r 1 5                          # memory
sar -b 1 5                          # IO
sar -n DEV 1 5                      # network
sar -q 1 5                          # load avg + run queue
sar -f /var/log/sysstat/sa15        # read past data (day 15)

Kernel & Hardware

dmesg -T | tail -50                 # kernel messages w/ timestamps
dmesg -w                            # follow
lscpu                               # CPU info
lsblk -f                            # block devs + filesystems
lspci                               # PCI devices
lsusb                               # USB devices
lsmod                               # loaded modules
uname -r                            # kernel version

Disk & Sockets

df -hT                              # disk usage by FS
df -i                               # inode usage
du -sh /var/log/*                   # size per item
du -sh * 2>/dev/null | sort -h      # sorted
ncdu /                              # interactive TUI

ss -tulnp                           # listening TCP/UDP sockets + processes
ss -s                               # socket summary
ss -tan state established           # established TCP conns

8. Network Commands

ip (the modern tool)

ip a                          # show all addresses
ip -br a                      # brief view
ip -4 a show eth0             # only IPv4 on eth0
ip r                          # routing table
ip -br link                   # link state of interfaces
ip neigh                      # ARP table (neighbors)

# Modify
sudo ip addr add 10.0.0.10/24 dev eth1
sudo ip addr del 10.0.0.10/24 dev eth1
sudo ip link set eth1 up
sudo ip link set eth1 down
sudo ip route add default via 10.0.0.1

ping & traceroute

ping -c 4 google.com          # 4 packets then stop
ping -i 0.2 -c 20 host        # 20 packets, 0.2s apart
ping -s 1472 -M do host       # MTU test (don't fragment)

traceroute -n 8.8.8.8         # numeric, no DNS
traceroute -T -p 443 host     # TCP traceroute on port 443
mtr -rwn 1.1.1.1              # report mode, wide, numeric

dig (DNS)

dig example.com               # default A record
dig +short A example.com
dig +short MX example.com
dig +short NS example.com
dig +short TXT example.com
dig AAAA example.com          # IPv6
dig @8.8.8.8 example.com      # query specific resolver
dig +trace example.com        # full delegation chain
dig -x 8.8.8.8                # reverse lookup
dig +noall +answer example.com

curl (essential flags)

Flag	Meaning
`-X METHOD`	HTTP method (GET/POST/PUT/DELETE)
`-H "Header: val"`	Add request header
`-d 'data'`	POST body
`--data-binary @file`	POST file contents as body
`-F 'k=v'`	multipart/form-data
`-u user:pass`	Basic auth
`-o file`	Output to file
`-O`	Save to remote filename
`-L`	Follow redirects
`-I`	HEAD request, headers only
`-i`	Include response headers
`-s`	Silent (no progress)
`-f`	Fail on HTTP errors (4xx/5xx)
`-k`	Insecure (skip cert validation)
`-v / --trace-ascii -`	Verbose / full trace
`--resolve host:port:ip`	Override DNS
`-w "%{http_code}\n"`	Print response info

curl -fsSL https://api.example.com/v1/users | jq
curl -X POST -H 'Content-Type: application/json' \
     -d '{"name":"alice"}' https://api/x
curl -w "code=%{http_code} time=%{time_total}\n" -o /dev/null -s https://x.com

ss (sockets)

ss -tulnp                     # TCP/UDP listening + process
ss -tan                       # all TCP
ss -tan state established
ss -tan state time-wait | wc -l
ss -tnp '( dport = :443 )'    # outbound HTTPS
ss -lx                        # unix sockets

nmap

nmap -sn 10.0.0.0/24                  # ping sweep (host discovery)
nmap -p 22,80,443 host                # specific ports
nmap -p- host                         # all 65535 ports
nmap -sV -p 1-1000 host               # service version detect
nmap -sS -O host                      # SYN scan + OS detect (needs root)
nmap -A host                          # aggressive: version+OS+scripts+traceroute
nmap --script vuln host               # vulnerability scripts

tcpdump

sudo tcpdump -i eth0                          # capture on eth0
sudo tcpdump -i any -nn port 80               # HTTP traffic, numeric
sudo tcpdump -i eth0 host 10.0.0.5            # to/from a host
sudo tcpdump -i eth0 -w out.pcap port 443     # write capture
sudo tcpdump -r out.pcap -nn                  # read capture
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'   # SYN packets
sudo tcpdump -i eth0 -A port 80               # print ASCII payload

nc & telnet

nc -zv host 22 443 8080         # port check (z=scan, v=verbose)
nc -l -p 9000                   # listen on port 9000
nc host 9000                    # connect (chat)
nc -l 9000 > file.bin           # receive file
nc host 9000 < file.bin         # send file
echo "GET / HTTP/1.0" | nc host 80   # raw HTTP request
telnet host 25                  # legacy, still handy for SMTP debug

9. Process Management

ps Options You'll Actually Use

Command	Output
`ps`	Your shell's processes only
`ps aux`	All processes (BSD): USER PID %CPU %MEM ... COMMAND
`ps -ef`	All processes (SysV): UID PID PPID C STIME TTY TIME CMD
`ps -eLf`	Include threads (LWP)
`ps -fp 1234`	Full info for PID 1234
`ps -ef --forest`	ASCII tree view
`ps -eo pid,user,pri,ni,%cpu,%mem,stat,start,cmd --sort=-%mem	head`

top / htop Keys

Key	top	htop
`h` / `F1`	Help	Help
`P`	Sort by CPU	—
`M`	Sort by memory	—
`F6`	—	Choose sort column
`k` / `F9`	Kill (asks PID + signal)	Kill (signal menu)
`r` / `F7/F8`	Renice	Nice +/-
`F3`	—	Search
`F4`	—	Filter
`F5`	—	Tree view
`q` / `F10`	Quit	Quit

Signals Table

Signal	Num	Default	Meaning
`SIGHUP`	1	Terminate	Reload config / parent shell exited
`SIGINT`	2	Terminate	Ctrl+C from terminal
`SIGQUIT`	3	Core dump	Ctrl+\
`SIGILL`	4	Core dump	Illegal instruction
`SIGABRT`	6	Core dump	abort() called
`SIGKILL`	9	Terminate	Force kill (cannot be caught)
`SIGSEGV`	11	Core dump	Invalid memory access
`SIGPIPE`	13	Terminate	Write to closed pipe
`SIGTERM`	15	Terminate	Graceful kill (default for `kill`)
`SIGUSR1`	10	Terminate	App-defined (often "rotate logs")
`SIGUSR2`	12	Terminate	App-defined
`SIGCHLD`	17	Ignore	Child stopped/exited
`SIGCONT`	18	Continue	Resume a stopped process
`SIGSTOP`	19	Stop	Stop (cannot be caught)
`SIGTSTP`	20	Stop	Ctrl+Z

kill / pkill / killall / pgrep

kill 1234                      # SIGTERM (default)
kill -9 1234                   # SIGKILL by number
kill -SIGTERM 1234             # by name
kill -HUP $(pidof nginx)       # reload nginx config

pgrep -f gunicorn              # find PIDs by full cmdline
pgrep -af gunicorn             # also show cmdline
pkill -f gunicorn              # kill all matching
pkill -HUP -f nginx            # signal all matching

killall nginx                  # kill by exact process name
killall -u alice               # kill all of user alice's processes

nice / renice

# Nice value range: -20 (highest priority) to 19 (lowest)
nice -n 10 ./batch.sh                # start with niceness 10
renice 5 -p 1234                     # change running process
renice -n 15 -u alice                # all of alice's processes
ionice -c 3 -p 1234                  # IO scheduling class (3=idle)

nohup & Job Control

nohup ./long_job.sh > out.log 2>&1 &     # survives logout
disown %1                                # detach job from shell
./job.sh &                               # background
jobs                                     # list shell jobs
jobs -l                                  # with PIDs
fg %1                                    # foreground job 1
bg %1                                    # resume in background
Ctrl+Z                                   # suspend foreground job

systemctl

systemctl status nginx
systemctl start  nginx
systemctl stop   nginx
systemctl restart nginx
systemctl reload  nginx                  # signal reload, no restart
systemctl enable  --now nginx            # enable + start
systemctl disable --now nginx
systemctl is-active nginx
systemctl is-enabled nginx
systemctl list-units --type=service
systemctl list-units --failed
systemctl daemon-reload                  # after editing unit files
systemctl cat nginx                      # show unit file
systemctl edit nginx                     # override
systemctl mask nginx                     # prevent any start (strong disable)

journalctl

journalctl -u nginx                      # logs for nginx
journalctl -u nginx -f                   # follow (like tail -f)
journalctl -u nginx --since "10 min ago"
journalctl -u nginx --since "2026-05-26 09:00" --until "2026-05-26 10:00"
journalctl -u nginx -p err               # priority err and above
journalctl -k                            # kernel only (= dmesg)
journalctl -b                            # current boot
journalctl -b -1                         # previous boot
journalctl --disk-usage
journalctl --vacuum-time=7d              # keep only last 7 days

10. Cron Expressions

Field Syntax

┌───────────── minute        (0 - 59)
│ ┌─────────── hour          (0 - 23)
│ │ ┌───────── day of month  (1 - 31)
│ │ │ ┌─────── month         (1 - 12)  or JAN-DEC
│ │ │ │ ┌───── day of week   (0 - 6)   or SUN-SAT  (0 and 7 = Sun)
│ │ │ │ │
* * * * *  command_to_run

Special Characters

Char	Meaning	Example
`*`	Every value	`* * * * *` = every minute
`,`	List	`0 9,12,18 * * *` = 9am, 12pm, 6pm
`-`	Range	`0 9-17 * * 1-5` = 9–5 Mon–Fri
`/`	Step	`/15 * * *` = every 15 minutes
`L` (extended)	Last	`0 0 L * *` = last day of month (not standard cron)
`?` (extended)	No value	Used in Quartz/Spring

20+ Real Examples

Expression	Runs
`* * * * *`	Every minute
`/5 * * *`	Every 5 minutes
`/15 * * *`	Every 15 minutes
`0 * * * *`	Every hour on the hour
`0 /2 * *`	Every 2 hours
`30 3 * * *`	3:30 AM every day
`0 0 * * *`	Midnight every day
`0 9 * * 1-5`	9 AM weekdays
`0 9 * * 1`	9 AM every Monday
`0 9 * * 0`	9 AM every Sunday
`0 9-17 * * 1-5`	Every hour, 9–5, weekdays
`0 0 1 * *`	Midnight on 1st of month
`0 0 1 1 *`	Midnight on Jan 1 (yearly)
`0 0 1,15 * *`	1st and 15th at midnight
`0 2 * * 6`	2 AM every Saturday (weekly backup)
`15 14 1 * *`	2:15 PM on the 1st of each month
`0 22 * * 1-5`	10 PM weekdays
`5 4 * * SUN`	4:05 AM every Sunday
`0 4 1-7 * 1`	4 AM on first Monday of month (kind of)
`/10 9-17 * 1-5`	Every 10 min, business hours, weekdays
`0 0 /3 *`	Midnight every 3 days
`0 12 * * 1#1`	Noon on first Monday (extended/Quartz)

@ Shortcuts

Shortcut	Equivalent	Meaning
`@reboot`	—	Once at startup
`@yearly` / `@annually`	`0 0 1 1 *`	Once a year
`@monthly`	`0 0 1 * *`	Once a month
`@weekly`	`0 0 * * 0`	Once a week (Sunday)
`@daily` / `@midnight`	`0 0 * * *`	Once a day
`@hourly`	`0 * * * *`	Once an hour

Managing Crontabs

crontab -l                    # list current user's crontab
crontab -e                    # edit (uses $EDITOR)
crontab -r                    # remove (be careful!)
crontab -u alice -l           # list alice's crontab (need root)

# System-wide files:
/etc/crontab                  # has a 'user' column
/etc/cron.d/*                 # drop-in files (also need user column)
/etc/cron.{hourly,daily,weekly,monthly}/   # scripts dir (no schedule)

# Always set env + redirect output in cron jobs:
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=ops@1buy.ai

0 3 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

# Useful for verifying: test online at crontab.guru

Keyboard Shortcuts and Aliases

Speed comes from muscle memory, not from typing faster. Until these shortcuts live in your fingers, you won't feel the productivity boost. Pick 10 from this list, drill them this week, then come back for the next 10.

Terminal Keyboard Shortcuts (Readline / Bash)

Cursor & Editing

Shortcut	Action
`Ctrl+A`	Move to beginning of line
`Ctrl+E`	Move to end of line
`Ctrl+B` / `Ctrl+F`	Back / forward one character
`Alt+B` / `Alt+F`	Back / forward one word
`Ctrl+U`	Cut from cursor to beginning of line
`Ctrl+K`	Cut from cursor to end of line
`Ctrl+W`	Cut previous word
`Alt+D`	Cut next word
`Ctrl+Y`	Paste (yank) last cut
`Ctrl+T`	Swap current and previous character
`Alt+T`	Swap current and previous word
`Ctrl+_`	Undo last edit
`Ctrl+L`	Clear screen (keeps current line)

Process & Session

Shortcut	Action
`Ctrl+C`	Send SIGINT (interrupt) to foreground process
`Ctrl+Z`	Send SIGTSTP (suspend) — resume with `fg` or `bg`
`Ctrl+D`	EOF — exits shell if line is empty
`Ctrl+S`	Freeze terminal output (XOFF) — try not to hit this
`Ctrl+Q`	Resume frozen terminal (XON)
`Ctrl+\`	SIGQUIT — like Ctrl+C but with core dump

History & Search

Shortcut	Action
`Ctrl+R`	Reverse-i-search through history (press again for next match)
`Ctrl+G`	Cancel reverse search
`Ctrl+P` / `Ctrl+N`	Previous / next history entry
`Alt+.`	Insert last argument of previous command (press repeatedly)

History Expansion (the bang stuff)

Expansion	Meaning
`!!`	Previous command in full (e.g. `sudo !!`)
`!$`	Last argument of previous command
`!*`	All arguments of previous command
`!N`	Command number N from history
`!-N`	N commands back
`!abc`	Most recent command starting with `abc`
`!?abc?`	Most recent command containing `abc`
`^old^new`	Re-run previous command, replacing first `old` with `new`
`!!:gs/old/new/`	Re-run previous command, global substitution

# Forgot sudo?
$ apt update
E: Permission denied
$ sudo !!         # expands to:  sudo apt update

# Typo in long command?
$ systemctl restrat nginx
$ ^restrat^restart

# Reuse last argument
$ mkdir /opt/myapp
$ cd !$           # cd /opt/myapp

Bash History Management

# View / search
history                   # show full history
history 20                # last 20
history | grep ssh        # filter

# Re-run by number
!457                      # run history entry 457
!-2                       # run 2 commands ago

# Delete entries
history -d 457            # delete entry 457
history -c                # clear ALL history (session)
history -w                # write current session to file

# Tune behavior in ~/.bashrc
HISTSIZE=10000                       # in-memory entries
HISTFILESIZE=20000                   # on-disk entries
HISTCONTROL=ignoreboth:erasedups     # skip dupes & lines starting with space
HISTTIMEFORMAT="%F %T "              # add timestamps
HISTIGNORE="ls:ll:pwd:exit:clear:history"   # never record these
shopt -s histappend                  # append, don't overwrite
shopt -s cmdhist                     # multi-line commands as one entry

# Share history across sessions in real time:
PROMPT_COMMAND='history -a; history -n'

Tip: Prepending a command with a space (when HISTCONTROL includes ignorespace or ignoreboth) keeps it out of history — handy when typing one-off commands with secrets.

Tab Completion

Trigger	What it completes
`Tab` (once)	Complete if unambiguous
`Tab Tab` (double)	Show all possible completions
After `cd`	Directories only
After `ssh` / `scp`	Hostnames from `~/.ssh/config` and `~/.ssh/known_hosts`
After `$`	Environment variable names
After `~`	Usernames (for home dirs)
After `git`, `kubectl`, `docker`	Subcommands, flags, resources (with completion installed)

# Enable programmable completion (most distros do this by default)
# /etc/bash_completion or in ~/.bashrc:
[ -r /usr/share/bash-completion/bash_completion ] && \
  . /usr/share/bash-completion/bash_completion

# Add tool-specific completion
source <(kubectl completion bash)
source <(helm completion bash)
complete -F __start_kubectl k        # so 'k <tab>' also works

# Show completions for a command
complete -p git

20+ Useful Aliases for ~/.bashrc

# ---------- Navigation & Listing ----------
alias ll='ls -lah --color=auto'           # long, human, hidden, colored
alias la='ls -A --color=auto'             # show dotfiles (no . and ..)
alias l='ls -CF --color=auto'             # quick column listing
alias ..='cd ..'                          # up one
alias ...='cd ../..'                      # up two
alias ....='cd ../../..'                  # up three
alias -- -='cd -'                         # toggle to previous dir

# ---------- Safety ----------
alias rm='rm -i'                          # ask before remove
alias cp='cp -i'                          # ask before overwrite
alias mv='mv -i'                          # ask before overwrite
alias mkdir='mkdir -pv'                   # parents + verbose

# ---------- Search / Filters ----------
alias grep='grep --color=auto'            # colored matches
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'

# ---------- Disk / System ----------
alias df='df -hT'                         # human + filesystem type
alias du='du -h --max-depth=1'            # human, one level
alias free='free -h'                      # human memory
alias ports='ss -tulnp'                   # what's listening?
alias psg='ps aux | grep -v grep | grep -iE'  # process search

# ---------- Network ----------
alias myip='curl -s https://ifconfig.me && echo'   # public IP
alias localip="ip -4 -br a | grep -v '^lo'"        # local IPs
alias weather='curl -s wttr.in/?format=3'          # quick weather
alias headers='curl -sI'                           # response headers only

# ---------- Git ----------
alias g='git'
alias gs='git status -sb'                 # short branch + status
alias gco='git checkout'
alias gcm='git commit -m'
alias gp='git pull --rebase'
alias gpush='git push'
alias gl='git log --oneline --graph --decorate -20'
alias gd='git diff'

# ---------- DevOps Tools ----------
alias k='kubectl'                         # the universal short form
alias kgp='kubectl get pods'
alias kgs='kubectl get svc'
alias kdp='kubectl describe pod'
alias kx='kubectl config use-context'
alias kns='kubectl config set-context --current --namespace'
alias d='docker'
alias dc='docker compose'
alias dps='docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
alias tf='terraform'
alias jc='journalctl -u'                  # usage: jc nginx -f
alias sc='systemctl'                      # usage: sc status nginx

# ---------- Misc Quality of Life ----------
alias path='echo -e ${PATH//:/\\n}'       # print PATH one entry per line
alias now='date "+%Y-%m-%d %H:%M:%S"'     # quick timestamp
alias reload='source ~/.bashrc'           # re-source after edits
alias ports-listening='sudo lsof -i -P -n | grep LISTEN'
alias ..git='cd $(git rev-parse --show-toplevel)'  # jump to repo root

Rationale. Aliases like ll, .., k, d, gs shave 5–10 keystrokes off the dozens of times you type them daily. Safety aliases (rm -i, cp -i) save you from career-ending mistakes — yes, even seniors have done rm -rf / at 2 AM. The myip/localip/ports ones replace "wait, what's that command again?" with one word.

To make aliases work in non-interactive contexts (rare but useful for scripts), put them in ~/.bash_aliases and source that file from ~/.bashrc.

Useful Bash Functions

# --- mkcd: make a directory and cd into it in one step ---
mkcd() {
  if [[ -z "$1" ]]; then
    echo "Usage: mkcd <dir>" >&2
    return 1
  fi
  mkdir -p -- "$1" && cd -P -- "$1"
}
# Usage:  mkcd /opt/projects/new-app

# --- extract: smart extractor for any common archive ---
extract() {
  if [[ -z "$1" ]]; then
    echo "Usage: extract <archive>" >&2
    return 1
  fi
  if [[ ! -f "$1" ]]; then
    echo "extract: '$1' is not a regular file" >&2
    return 1
  fi
  case "$1" in
    *.tar.bz2|*.tbz2)   tar -xjvf "$1"   ;;
    *.tar.gz|*.tgz)     tar -xzvf "$1"   ;;
    *.tar.xz|*.txz)     tar -xJvf "$1"   ;;
    *.tar.zst)          tar --zstd -xvf "$1" ;;
    *.tar)              tar -xvf  "$1"   ;;
    *.bz2)              bunzip2   "$1"   ;;
    *.gz)               gunzip    "$1"   ;;
    *.xz)               unxz      "$1"   ;;
    *.zip)              unzip     "$1"   ;;
    *.rar)              unrar x   "$1"   ;;
    *.7z)               7z x      "$1"   ;;
    *.Z)                uncompress "$1"  ;;
    *) echo "extract: don't know how to handle '$1'" >&2; return 1 ;;
  esac
}
# Usage:  extract release-v2.4.1.tar.gz

# --- bak: quick timestamped backup of a file ---
bak() {
  if [[ -z "$1" ]]; then
    echo "Usage: bak <file>" >&2
    return 1
  fi
  if [[ ! -e "$1" ]]; then
    echo "bak: '$1' does not exist" >&2
    return 1
  fi
  local ts
  ts=$(date +%Y%m%d-%H%M%S)
  cp -a -- "$1" "$1.bak.$ts" && \
    echo "Backed up to $1.bak.$ts"
}
# Usage:  bak /etc/nginx/nginx.conf
#         -> /etc/nginx/nginx.conf.bak.20260526-101530

# --- Bonus: cdf -- cd into the directory of a file ---
cdf() {
  if [[ -z "$1" ]]; then
    echo "Usage: cdf <file>" >&2
    return 1
  fi
  cd -- "$(dirname -- "$1")"
}

# --- Bonus: fkill -- fuzzy kill (needs fzf) ---
fkill() {
  local pid
  pid=$(ps -ef | sed 1d | fzf -m | awk '{print $2}')
  [[ -n "$pid" ]] && echo "$pid" | xargs kill -"${1:-15}"
}

Drop these into ~/.bashrc (or a sourced ~/.bash_functions file), run source ~/.bashrc, and they become first-class commands. The extract function alone will save you "wait, was it xjvf or xzvf?" lookups for the rest of your career. Set it up once, and it pays off for life.

Advanced Linux for DevOps

This is where the real DevOps game begins. Containers, kernel tuning, boot internals, LVM, RAID — this is the layer that completes the transition from frontend developer to six-figure (USD) DevOps engineer. An interviewer will ask you: "How does a Docker container work under the hood?" — if you can explain namespaces + cgroups + overlayfs + capabilities, you're already in the top 20% of candidates.

Containers Are Not Magic — They Are Linux Features

A container = namespaces (isolation) + cgroups (resource limits) + overlayfs (layered FS) + capabilities (privilege control). Docker, containerd, Podman — they all sit on this foundation. You don't need a full hypervisor like a VM; the kernel is shared with the host, but each process sees an isolated world as if it had its own OS.

Linux Namespaces — A Process Gets Its Own Universe

Namespaces show a process group only a subset of the system. Inside a container, a process thinks it's PID 1, with its own network stack and its own filesystem root.

Namespace	Flag	What It Isolates	Practical Effect
`pid`	`CLONE_NEWPID`	Process IDs	The container's PID 1 might be PID 14523 on the host
`mnt`	`CLONE_NEWNS`	Mount points / filesystem view	The container sees its own `/`, not the host's `/`
`net`	`CLONE_NEWNET`	Network interfaces, routes, iptables, sockets	Its own eth0, its own loopback, its own iptables rules
`ipc`	`CLONE_NEWIPC`	SysV IPC, POSIX message queues	Shared memory segments isolated
`uts`	`CLONE_NEWUTS`	Hostname, domain name	Each container gets its own hostname (`hostname` command)
`user`	`CLONE_NEWUSER`	UID/GID mappings	The container's root (UID 0) maps to unprivileged user 100000 on the host
`cgroup`	`CLONE_NEWCGROUP`	Cgroup root view	The container sees only a subset of the cgroup hierarchy

Manual Namespace Creation — Build a Container Without Docker

# Create a new PID + network + mount namespace
sudo unshare --pid --net --mount --uts --ipc --fork --mount-proc bash

# Go inside and check
echo $$                    # Shows PID 1 (it's something else on the host)
hostname container-demo    # Changes only inside this namespace
ps aux                     # Only the processes in this namespace
ip addr                    # Only loopback (network is isolated)

# Everything is cleaned up on exit
exit

This is exactly what Docker does — it just adds an image, overlayfs, and orchestration on top.

Inspect Namespaces — lsns and /proc

# List all namespaces
lsns

# Specific type
lsns -t net
lsns -t pid

# Look at your own shell's namespaces
ls -l /proc/$$/ns/
# Output:
# lrwxrwxrwx 1 user user 0 May 26 10:23 cgroup -> 'cgroup:[4026531835]'
# lrwxrwxrwx 1 user user 0 May 26 10:23 ipc    -> 'ipc:[4026531839]'
# lrwxrwxrwx 1 user user 0 May 26 10:23 mnt    -> 'mnt:[4026531840]'
# lrwxrwxrwx 1 user user 0 May 26 10:23 net    -> 'net:[4026531992]'
# lrwxrwxrwx 1 user user 0 May 26 10:23 pid    -> 'pid:[4026531836]'
# lrwxrwxrwx 1 user user 0 May 26 10:23 user   -> 'user:[4026531837]'
# lrwxrwxrwx 1 user user 0 May 26 10:23 uts    -> 'uts:[4026531838]'

# Enter a container's namespace (once you have the Docker PID)
sudo nsenter --target 14523 --pid --net --mount

cgroups v2 (Ubuntu 24.04 Default)

cgroups (control groups) limit and account for the resource usage of processes. Ubuntu 24.04 uses the unified cgroup v2 — a single hierarchy at /sys/fs/cgroup.

# Verify cgroup v2
mount | grep cgroup
# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

# Available controllers
cat /sys/fs/cgroup/cgroup.controllers
# cpu io memory pids hugetlb cpuset rdma misc

# Create a new cgroup and set limits
sudo mkdir /sys/fs/cgroup/myapp
echo "+cpu +memory +io +pids" | sudo tee /sys/fs/cgroup/cgroup.subtree_control

# CPU: 50% of one core (50000 us out of 100000 us period)
echo "50000 100000" | sudo tee /sys/fs/cgroup/myapp/cpu.max

# Memory: 512 MB hard limit
echo "536870912" | sudo tee /sys/fs/cgroup/myapp/memory.max

# PIDs: max 100 processes
echo "100" | sudo tee /sys/fs/cgroup/myapp/pids.max

# IO: weight (10-1000, default 100), per-device limits also possible
echo "default 50" | sudo tee /sys/fs/cgroup/myapp/io.weight

# Put a process into the cgroup
echo $$ | sudo tee /sys/fs/cgroup/myapp/cgroup.procs

# Check current usage
cat /sys/fs/cgroup/myapp/memory.current
cat /sys/fs/cgroup/myapp/cpu.stat

Tip: Docker's --memory 512m --cpus="0.5" writes to exactly these cgroup files internally. A Kubernetes pod's resources.limits also end up here via kubelet → containerd → runc.

How Docker Stitches It All Together

Layer	Technology	Role
Isolation	Namespaces (pid, net, mnt, uts, ipc, user)	Each process in its own universe
Resource limits	cgroups v2	CPU, memory, IO, PIDs cap
Filesystem	overlayfs (lowerdir + upperdir = merged)	Image layers + writable container layer
Privileges	Capabilities (drop most by default)	Container root != host root
Security	seccomp, AppArmor/SELinux	Syscall filtering, MAC
Networking	veth pair + bridge (docker0) + iptables	Container-to-host-to-internet

# Try overlayfs yourself
mkdir -p /tmp/ovl/{lower,upper,work,merged}
echo "from image" > /tmp/ovl/lower/file.txt

sudo mount -t overlay overlay \
  -o lowerdir=/tmp/ovl/lower,upperdir=/tmp/ovl/upper,workdir=/tmp/ovl/work \
  /tmp/ovl/merged

cat /tmp/ovl/merged/file.txt              # "from image"
echo "container write" > /tmp/ovl/merged/file.txt
cat /tmp/ovl/lower/file.txt               # Still "from image" — read-only
cat /tmp/ovl/upper/file.txt               # "container write" — copy-on-write

Linux Capabilities — The Modern Replacement for Setuid

In old Unix, a binary was either root (UID 0, all-powerful) or a normal user. Linux capabilities break root's powers into 40+ smaller chunks. ping needs a raw socket — it used to be setuid-root, now it's given only CAP_NET_RAW.

# Look at your shell's capabilities
capsh --print

# Look at a file's capabilities
getcap /usr/bin/ping
# /usr/bin/ping cap_net_raw=ep

# Give Nginx the capability to bind port 80 (a privileged port) without running as root
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/nginx
getcap /usr/local/bin/nginx
# /usr/local/bin/nginx cap_net_bind_service=ep

# Common capabilities
# CAP_NET_BIND_SERVICE  - bind ports < 1024
# CAP_NET_ADMIN         - network config (ip, iptables)
# CAP_SYS_ADMIN         - "almost root" — very powerful, be careful
# CAP_CHOWN             - change file ownership
# CAP_DAC_OVERRIDE      - bypass permissions
# CAP_KILL              - send a signal to any process
# CAP_SETUID/SETGID     - change UID/GID

# Remove a capability
sudo setcap -r /usr/local/bin/nginx

# Docker drops a lot of caps by default
docker run --rm alpine sh -c 'apk add -q libcap; capsh --print' | grep Current
# Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,...
# (NOT full root capabilities)

Performance Tuning — sysctl, ulimit, I/O Schedulers

Kernel Parameters via sysctl

# Check the current value
sysctl vm.swappiness
sysctl net.core.somaxconn

# Runtime change (gone after reboot)
sudo sysctl -w vm.swappiness=10
sudo sysctl -w net.core.somaxconn=4096

# Persist it — create a file in /etc/sysctl.d/
sudo tee /etc/sysctl.d/99-devops-tuning.conf <<'EOF'
# Memory: prefer dropping cache over swapping (DB/app servers)
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.overcommit_memory = 1

# Network: high-connection web server
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.ip_local_port_range = 1024 65535

# File descriptors
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288

# Connection tracking (for nat/firewalls)
net.netfilter.nf_conntrack_max = 1048576
EOF

# Apply it
sudo sysctl --system

# All current values
sysctl -a | less

ulimit — Per-Process Resource Limits

# Current limits
ulimit -a

# Open files limit (the most common bottleneck)
ulimit -n               # soft
ulimit -Hn              # hard

# Raise it temporarily (current shell only)
ulimit -n 65535

# Permanent — /etc/security/limits.conf or /etc/security/limits.d/
sudo tee /etc/security/limits.d/99-app.conf <<'EOF'
# domain  type  item   value
*         soft  nofile 65535
*         hard  nofile 1048576
*         soft  nproc  32768
*         hard  nproc  65535
www-data  soft  nofile 100000
www-data  hard  nofile 200000
EOF

# For systemd services — in the service unit file
# [Service]
# LimitNOFILE=1048576
# LimitNPROC=65535

I/O Schedulers

# Look at the current scheduler
cat /sys/block/sda/queue/scheduler
# [mq-deadline] kyber bfq none

# 'none' is best for NVMe — NVMe has its own queue management
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

# For SATA SSD / HDD use 'mq-deadline' or 'bfq'
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

# Persist via udev rule
sudo tee /etc/udev/rules.d/60-ioschedulers.rules <<'EOF'
# NVMe
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
# SSD
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
# HDD
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
EOF

Caution: The most common cause of production outages — ulimit -n set too low. Under load, Node.js / Java apps crash at 1024 file descriptors with "EMFILE: too many open files". Always set it to 65535+.

Boot Process Deep Dive

Press the power button → kernel running → systemd → login prompt. There are many layers in between:

Stage	UEFI Boot	Legacy BIOS
1. Firmware	UEFI firmware (modern)	BIOS (legacy)
2. Boot loader location	EFI System Partition (ESP), FAT32, mounted at `/boot/efi`	MBR (first 512 bytes of disk)
3. Boot loader	GRUB EFI binary (`/boot/efi/EFI/ubuntu/grubx64.efi`)	GRUB stage 1 → stage 2
4. Config	`/boot/grub/grub.cfg` (generated)	`/boot/grub/grub.cfg`
5. Kernel + initramfs	Loaded from `/boot/`	Loaded from `/boot/`
6. Secure Boot	Supported (shim + signed kernel)	Not supported
7. Disk size	> 2 TB OK (GPT)	2 TB limit (MBR)

# Check whether you booted in UEFI mode
ls /sys/firmware/efi && echo "UEFI" || echo "BIOS"

# Look at the ESP partition
df -h /boot/efi
mount | grep /boot/efi
# /dev/nvme0n1p1 on /boot/efi type vfat ...

# EFI boot entries
sudo efibootmgr -v
# BootCurrent: 0001
# Boot0001* ubuntu  HD(1,GPT,...)/File(\EFI\ubuntu\shimx64.efi)

GRUB2 Configuration

# Edit defaults
sudo nano /etc/default/grub

# Common settings:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

# Add a kernel parameter (example: disable IPv6, enable cgroup v2 — already default in 24.04)
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash ipv6.disable=1"

# Regenerate grub.cfg
sudo update-grub
# OR:  sudo grub-mkconfig -o /boot/grub/grub.cfg

# Rebuild initramfs (after a driver change or crypto change)
sudo update-initramfs -u -k all

# Kernel command line currently running
cat /proc/cmdline

initramfs (initial RAM filesystem)

Right after the kernel boots, it needs drivers to mount the real root filesystem (NVMe, RAID, LVM, encrypted disk). All of this happens in a temporary RAM-based filesystem — the initramfs. /boot/initrd.img-* is a compressed cpio archive.

# See what's inside
lsinitramfs /boot/initrd.img-$(uname -r) | head -50

# Rebuild
sudo update-initramfs -u -k all

systemd Boot Targets & Analysis

# Current target
systemctl get-default          # graphical.target or multi-user.target

# Set it
sudo systemctl set-default multi-user.target

# Total boot time for a server
systemd-analyze
# Startup finished in 3.124s (kernel) + 8.456s (userspace) = 11.580s

# Which service is slow?
systemd-analyze blame
# 4.123s snapd.service
# 2.456s cloud-init.service
# 1.234s apt-daily.service

# Critical path (the serial dependency chain of boot)
systemd-analyze critical-chain
systemd-analyze critical-chain nginx.service

# Visual SVG plot
systemd-analyze plot > /tmp/boot-plot.svg
# Open it in a browser — you get a beautiful timeline

LVM — Logical Volume Manager (Production Essential)

LVM abstracts your disks. You pool multiple physical disks (PVs) into a group (VG), and carve flexible chunks (LVs) out of it — runtime resize, snapshots, and online migration all come for free.

The LVM stack, top to bottom: Filesystems (/ ext4, /var/lib/docker, /data xfs) sit on LVs (lv-root 20G, lv-docker 50G, lv-data 200G), which are carved from the VG (vg_main — Volume Group, 500 GB total, 230 GB free), which pools the PVs (/dev/nvme0n1p3 200G, /dev/nvme1n1 200G, /dev/nvme2n1 100G — physical disks / partitions / raw block devices).

Full End-to-End LVM Example

# 1. Create Physical Volumes (turn raw disks into LVM-managed devices)
sudo pvcreate /dev/nvme1n1 /dev/nvme2n1
sudo pvs

# 2. Create a Volume Group (a pool of PVs)
sudo vgcreate vg_main /dev/nvme1n1 /dev/nvme2n1
sudo vgs
sudo vgdisplay vg_main

# 3. Create Logical Volumes
sudo lvcreate -L 50G -n lv-docker vg_main
sudo lvcreate -L 200G -n lv-data vg_main
# Or all the free space:
# sudo lvcreate -l 100%FREE -n lv-data vg_main
sudo lvs

# 4. Create filesystems
sudo mkfs.ext4 /dev/vg_main/lv-docker
sudo mkfs.xfs  /dev/vg_main/lv-data

# 5. Mount them
sudo mkdir -p /var/lib/docker /data
sudo mount /dev/vg_main/lv-docker /var/lib/docker
sudo mount /dev/vg_main/lv-data /data

# 6. Add to /etc/fstab (UUID is best practice)
echo "/dev/vg_main/lv-docker /var/lib/docker ext4 defaults,noatime 0 2" | sudo tee -a /etc/fstab
echo "/dev/vg_main/lv-data   /data           xfs  defaults,noatime 0 2" | sudo tee -a /etc/fstab

# ===== NOW THE LIVE EXTEND — this is the real power of LVM =====

# The disk is filling up. Grow lv-data by another 100G
sudo lvextend -L +100G /dev/vg_main/lv-data
sudo xfs_growfs /data                    # XFS online grow

# For ext4:
# sudo lvextend -L +50G /dev/vg_main/lv-docker
# sudo resize2fs /dev/vg_main/lv-docker

# Combined (lvextend + filesystem resize in a single command):
sudo lvextend -r -L +50G /dev/vg_main/lv-docker

# Out of space in the VG? Add a new disk
sudo pvcreate /dev/nvme3n1
sudo vgextend vg_main /dev/nvme3n1
sudo lvextend -r -L +500G /dev/vg_main/lv-data

# ===== SNAPSHOTS — for backups =====
sudo lvcreate -L 10G -s -n lv-data-snap /dev/vg_main/lv-data
sudo mount -o ro /dev/vg_main/lv-data-snap /mnt/snapshot
tar czf /backup/data-$(date +%F).tar.gz -C /mnt/snapshot .
sudo umount /mnt/snapshot
sudo lvremove -f /dev/vg_main/lv-data-snap

Tip: Cloud relevance — why LVM on AWS? To stripe or pool multiple EBS volumes. Take 4×500GB gp3 volumes on an instance and make a single 2TB volume with LVM — combined IOPS goes up and growing it is easy.

RAID — Redundancy and Performance

RAID Level	Min Disks	Redundancy	Capacity	Read/Write	Use Case
RAID 0 (stripe)	2	None — 1 fail = data gone	100%	Fastest R + W	Scratch / cache (cloud: avoid)
RAID 1 (mirror)	2	1 disk fail OK	50%	Fast R, normal W	OS disk, boot
RAID 5 (parity)	3	1 disk fail OK	(N-1)/N	Good R, slow W (parity calc)	Bulk storage (legacy)
RAID 6 (double parity)	4	2 disks fail OK	(N-2)/N	Good R, slower W	Large arrays, long rebuilds
RAID 10 (1+0)	4	1 per mirror pair OK	50%	Fastest R + W with redundancy	Databases, production

# Software RAID with mdadm
sudo apt install mdadm

# RAID 10 from 4 disks
sudo mdadm --create /dev/md0 --level=10 --raid-devices=4 \
  /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1

# Status
cat /proc/mdstat
sudo mdadm --detail /dev/md0

# Save config (otherwise it disappears on boot)
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u

# Use as an LVM PV or a direct filesystem
sudo mkfs.xfs /dev/md0

Caution: Cloud reality — AWS EBS / GCP PD / Azure Managed Disks are already replicated (3 copies cross-AZ). You rarely run RAID in the cloud — the provider already does it. Just attach multiple EBS volumes + LVM stripe for IOPS — that's the cloud pattern. mdadm is only critical on bare-metal / on-prem.

Filesystem Tuning

# ext4: lower reserved blocks (root-only) from 5% to 1% — save space on big data disks
sudo tune2fs -m 1 /dev/vg_main/lv-data

# Turn off forced fsck interval / mount count (production servers — avoid fsck on boot)
sudo tune2fs -c 0 -i 0 /dev/vg_main/lv-data

# Show all options
sudo tune2fs -l /dev/vg_main/lv-data

# Mount options that matter in production
# noatime    — don't update atime on every read (huge perf gain, esp. for DBs, web)
# nodiratime — skip directory atime
# discard    — SSD TRIM on delete (alternative: fstrim weekly via timer)
# nofail     — don't fail boot if the disk is missing (cloud detached EBS scenario)
# nosuid,nodev,noexec — security for /tmp, /var

# /etc/fstab example for production
# UUID=...  /data       ext4  defaults,noatime,nodiratime,nofail  0  2
# UUID=...  /tmp        tmpfs defaults,nosuid,nodev,noexec,size=2G 0  0

# tmpfs (RAM-backed) in production — fast scratch / build directories
sudo mount -t tmpfs -o size=4G,mode=1777 tmpfs /var/cache/build

Note: Quick recap — A Docker container = unshare + cgroups + overlayfs + capability drop. Boot = UEFI → GRUB → kernel → initramfs → systemd. LVM = flexible disks, runtime grow. RAID = redundancy. You rarely need RAID in the cloud, but LVM and tuning are useful everywhere. All of these show up in interviews as the answer to "how does X work internally?"

Linux in Cloud Context

So far we've looked at Linux on a single isolated machine. In the cloud the same Linux is running, but with different patterns — ephemeral instances, the metadata service, cloud-init bootstrapping, EBS attach/grow, replacing SSH with SSM. A developer-turned-DevOps engineer spends roughly 70% of each day on AWS Linux instances + Kubernetes nodes. This section covers that 70%.

AWS EC2 Linux Distributions

Distro	Default User	Package Mgr	Notes
Amazon Linux 2023 (AL2023)	`ec2-user`	`dnf`	AWS optimized, kernel 6.x, glibc 2.34, free, fast-boot. Default choice for AWS.
Amazon Linux 2 (legacy)	`ec2-user`	`yum`	EOL 2026. Migrate to AL2023.
Ubuntu Server 24.04 LTS	`ubuntu`	`apt`	Most popular, huge community, debs, snap. Best for general use.
Rocky Linux 9 / RHEL 9	`rocky` / `ec2-user`	`dnf`	Enterprise, RHEL-compatible, FIPS, certifications.
Debian 12	`admin`	`apt`	Minimal, stable, like Ubuntu's parent.
Custom AMI	Whatever you bake	Whatever	Pre-configured golden image via Packer.

Tip: Practical pick — Production Kubernetes nodes → Amazon Linux 2023 (also consider Bottlerocket). General app servers → Ubuntu 24.04. Enterprise compliance → RHEL/Rocky.

Instance Metadata Service (IMDS) — The Cloud's /proc

Every EC2 instance gets a special endpoint: http://169.254.169.254. From here the instance can learn everything about itself — region, AZ, instance type, IAM role credentials, user-data, public IP. SDKs (boto3, aws-cli) also pull their IAM creds from here.

# IMDSv1 (legacy, insecure — SSRF attacks possible)
curl http://169.254.169.254/latest/meta-data/

# IMDSv2 (token-based, mandatory in new accounts) — SECURE PATTERN
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Use token in all requests
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/

# Useful endpoints
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-type
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/availability-zone
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/public-ipv4
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/security-groups
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/

# Tags (if enabled on instance)
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/tags/instance/

# User-data (the bootstrap script)
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/user-data

Caution: IMDSv1 is vulnerable to SSRF attacks — the Capital One breach happened through it. Always enforce IMDSv2 (HttpTokens=required) in the instance metadata options.

EBS Volumes — Attach, Format, Mount, Grow

# After attaching a new volume, look at the device list
lsblk
# NAME         SIZE TYPE MOUNTPOINTS
# nvme0n1       30G disk
# `-nvme0n1p1   30G part /
# nvme1n1      100G disk                  <-- new EBS volume

# No filesystem on it yet
sudo file -s /dev/nvme1n1
# /dev/nvme1n1: data        (i.e. empty)

# Create a filesystem
sudo mkfs.ext4 /dev/nvme1n1
# Modern alternative: sudo mkfs.xfs /dev/nvme1n1

# Create a mountpoint and mount it
sudo mkdir -p /data
sudo mount /dev/nvme1n1 /data

# /etc/fstab — UUID best practice; nofail critical for cloud
# (if the instance reboots with the volume detached, boot won't fail)
UUID=$(sudo blkid -s UUID -o value /dev/nvme1n1)
echo "UUID=$UUID  /data  ext4  defaults,nofail,noatime  0  2" | sudo tee -a /etc/fstab

# Verify
sudo mount -a
df -h /data

Grow EBS Volume In-Place (Zero Downtime)

# Step 1: Increase the volume size from the AWS console or CLI
aws ec2 modify-volume --volume-id vol-0abc123 --size 200

# Wait for "optimizing" state
aws ec2 describe-volumes-modifications --volume-id vol-0abc123

# Step 2: Inside the instance, the kernel sees the new size
lsblk
# nvme1n1      200G disk
# `-nvme1n1p1  100G part /data         <-- partition is still 100G

# Step 3: Grow the partition (if you're using a partition)
sudo growpart /dev/nvme1n1 1

# Filesystem directly on the raw device (no partition) — skip this step

# Step 4: Grow the filesystem
# ext4:
sudo resize2fs /dev/nvme1n1p1
# OR direct:
sudo resize2fs /dev/nvme1n1

# xfs (must be mounted):
sudo xfs_growfs /data

# Verify
df -h /data
# /dev/nvme1n1   200G ...

Instance Store vs EBS

	EBS	Instance Store
Persistence	Persistent — survives stop/start	Ephemeral — gone on stop/terminate
Replication	3× within AZ	None
Speed	Network-attached (NVMe over fabric)	Physically attached NVMe SSD — fastest
Snapshots	Yes	No
Detach & reattach	Yes	No
Use case	OS root, databases, app data	Cache, scratch, temp big-data shuffle

SSH to Cloud Instances

# Classic: keypair-based SSH
chmod 400 ~/.ssh/my-key.pem
ssh -i ~/.ssh/my-key.pem ec2-user@54.221.10.45    # Amazon Linux
ssh -i ~/.ssh/my-key.pem ubuntu@54.221.10.45      # Ubuntu

# A convenient alias in ~/.ssh/config
cat >> ~/.ssh/config <<'EOF'
Host prod-web
  HostName 54.221.10.45
  User ubuntu
  IdentityFile ~/.ssh/my-key.pem
  ServerAliveInterval 60
EOF
ssh prod-web

# Bastion host pattern (private instance via public jump box)
ssh -i key.pem -J bastion.example.com ubuntu@10.0.5.20
# Or ProxyJump in your config
Host prod-db
  HostName 10.0.5.20
  User ubuntu
  IdentityFile ~/.ssh/my-key.pem
  ProxyJump bastion.example.com

# ===== MODERN: SSM Session Manager (NO PORT 22, NO KEYS) =====
# Pre-requisites: SSM agent on instance, IAM role with AmazonSSMManagedInstanceCore

aws ssm start-session --target i-0abc123def456

# Port forwarding via SSM (RDS access through private instance)
aws ssm start-session \
  --target i-0abc123def456 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["mydb.cluster-xyz.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["15432"]}'

# SSH over SSM (best of both)
# ~/.ssh/config:
Host i-* mi-*
  ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p"

ssh ec2-user@i-0abc123def456

Tip: Production gold-standard — never open port 22 in your security groups. Use SSM Session Manager — no public IP, no SSH key management, full audit log in CloudTrail, IAM-based access control. This is the modern DevOps practice.

cloud-init — First Boot Bootstrap

cloud-init is Linux's standard first-boot configuration tool. It runs on AWS, GCP, Azure, OpenStack — everywhere. The user-data field you provide at launch time is parsed and executed by cloud-init.

# Status check
cloud-init status
# status: done

cloud-init status --long
cloud-init analyze show              # boot time breakdown

# Config files
ls /etc/cloud/
# cloud.cfg               main config
# cloud.cfg.d/            drop-ins (vendor + user)
# templates/              hostname etc.

# Logs (THE MOST IMPORTANT thing for debugging)
sudo tail -f /var/log/cloud-init.log              # detailed
sudo tail -f /var/log/cloud-init-output.log       # stdout/stderr of user-data

# Re-run cloud-init (for testing)
sudo cloud-init clean --logs
sudo cloud-init init
sudo cloud-init modules --mode=config
sudo cloud-init modules --mode=final

Format 1: Bash Script User-Data

#!/bin/bash
# user-data.sh — simple bash, runs as root on first boot
set -euxo pipefail
exec > >(tee /var/log/user-data.log | logger -t user-data -s 2>/dev/console) 2>&1

echo "=== Starting bootstrap at $(date) ==="

# Idempotency guard
if [ -f /var/lib/bootstrap-done ]; then
  echo "Already bootstrapped, exiting"
  exit 0
fi

# Update + install
apt-get update -y
apt-get upgrade -y
apt-get install -y nginx awscli docker.io curl jq

# Enable services
systemctl enable --now nginx docker
usermod -aG docker ubuntu

# Pull config from S3 (IAM role attached)
aws s3 cp s3://my-config-bucket/nginx.conf /etc/nginx/nginx.conf
systemctl reload nginx

# Mark done
touch /var/lib/bootstrap-done
echo "=== Bootstrap complete at $(date) ==="

Format 2: #cloud-config YAML (Recommended)

#cloud-config
# Declarative — much cleaner than bash

hostname: web-prod-01
fqdn: web-prod-01.internal.example.com
manage_etc_hosts: true

# Users
users:
  - name: deploy
    groups: [docker, sudo]
    shell: /bin/bash
    sudo: 'ALL=(ALL) NOPASSWD:ALL'
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1... deploy@laptop

# Package install
package_update: true
package_upgrade: true
packages:
  - nginx
  - docker.io
  - awscli
  - jq
  - htop
  - curl

# Drop config files
write_files:
  - path: /etc/nginx/sites-available/default
    permissions: '0644'
    content: |
      server {
        listen 80 default_server;
        root /var/www/html;
        index index.html;
        location /health { return 200 'ok'; add_header Content-Type text/plain; }
      }
  - path: /etc/sysctl.d/99-tuning.conf
    content: |
      vm.swappiness = 10
      net.core.somaxconn = 65535
      fs.file-max = 2097152

# Run commands at end (after packages, files)
runcmd:
  - sysctl --system
  - systemctl enable --now docker nginx
  - usermod -aG docker ubuntu
  - docker pull nginx:alpine
  - [ sh, -c, 'echo "Bootstrap done at $(date)" >> /var/log/bootstrap.log' ]

# Reboot if kernel updated (optional)
power_state:
  mode: reboot
  condition: test -f /var/run/reboot-required

AMI Creation — Golden Images

In production, instead of doing a full user-data setup on every deploy, you use pre-baked AMIs — boot in 30 seconds instead of 5 minutes. But before creating an AMI, a "sysprep" (clean) is essential.

# Pre-AMI cleanup script (run before creating image)
#!/bin/bash
set -e

# Stop services that hold state
sudo systemctl stop docker nginx

# Remove SSH host keys (they'll regenerate on the instance's first boot)
sudo rm -f /etc/ssh/ssh_host_*

# Clear machine-id (cloud-init will regenerate)
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id

# Clear cloud-init state
sudo cloud-init clean --logs --seed

# Clear logs
sudo find /var/log -type f -exec truncate -s 0 {} \;
sudo rm -rf /var/log/journal/*

# Clear bash history
cat /dev/null > ~/.bash_history
history -c
sudo rm -f /root/.bash_history

# Clear authorized_keys for default user (cloud-init re-injects)
sudo rm -f /home/ubuntu/.ssh/authorized_keys

# Trim filesystem (smaller snapshot)
sudo fstrim -av

# Now from your laptop/CI:
aws ec2 create-image \
  --instance-id i-0abc123 \
  --name "web-base-$(date +%Y%m%d-%H%M)" \
  --description "Nginx + Docker baseline" \
  --no-reboot=false

# Packer (recommended for repeatable builds) — basic config
# packer build webserver.pkr.hcl
# Packer launches temp instance, runs provisioners, creates AMI, terminates.

Provider Differences

	AWS EC2	GCP Compute Engine	Azure VM
Metadata endpoint	`169.254.169.254`	`metadata.google.internal` / `169.254.169.254`	`169.254.169.254` (with header)
Default users	ec2-user, ubuntu, rocky, admin	username from SSH key (gcloud creates one)	azureuser (you pick at create)
Agent	SSM Agent + ec2-instance-connect	Google guest agent / OS Login	WALinuxAgent (waagent)
Agent config	`/etc/amazon/ssm/`	`/etc/default/instance_configs.cfg`	`/etc/waagent.conf`
Block storage	EBS	Persistent Disk (PD)	Managed Disks
Bootstrap	user-data + cloud-init	startup-script (metadata) + cloud-init	customData / cloud-init
Keyless SSH	SSM Session Manager	OS Login + IAP tunnel	Bastion / Just-In-Time

# GCP metadata example
curl -s "http://metadata.google.internal/computeMetadata/v1/instance/name" \
  -H "Metadata-Flavor: Google"

# Azure metadata example (note required header)
curl -s -H "Metadata:true" \
  "http://169.254.169.254/metadata/instance?api-version=2021-02-01" | jq

Real-World DevOps Patterns

1. Immutable Infrastructure

Don't patch the server — launch a new instance from a new AMI, shift traffic, terminate the old one. Configuration drift = zero. Rollback = re-launch the previous AMI. This is the modern way.

2. cloud-init + Configuration Management Combo

#cloud-config
# cloud-init only "bootstraps Ansible/Chef" — that tool does the rest
package_update: true
packages: [ansible, git]
runcmd:
  - git clone https://github.com/myorg/ansible-playbooks.git /opt/ansible
  - ansible-pull -U https://github.com/myorg/ansible-playbooks.git -i localhost, site.yml

3. Centralized Logging

Local logs on an instance (/var/log/*) are ephemeral — when the instance is terminated, the logs are gone. In production, always ship your logs:

# CloudWatch agent
sudo amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:cw-config.json -s

# Datadog agent
DD_API_KEY=xxx DD_SITE="datadoghq.com" \
  bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

# Promtail (Grafana Loki)
# /etc/promtail/config.yml → ships journald + /var/log/* to Loki

# rsyslog → centralized syslog server (old-school but reliable)
# /etc/rsyslog.d/50-remote.conf:
# *.* @@logserver.internal:514

4. EBS Snapshots — Automated Backup

# AWS Backup or Data Lifecycle Manager (DLM) — preferred
# Snapshot daily, retain 7 days, cross-region copy weekly

# Manual snapshot (one-off)
aws ec2 create-snapshot \
  --volume-id vol-0abc123 \
  --description "pre-migration backup $(date +%F)" \
  --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Backup,Value=manual}]'

# Restore: snapshot → new volume → attach to instance
aws ec2 create-volume \
  --snapshot-id snap-0xyz \
  --availability-zone us-east-1a \
  --volume-type gp3

5. SSH via SSM — No Port 22 Open

# Security group inbound rules: (none — yes, ZERO)
# Instances are in private subnet
# Access only via:
aws ssm start-session --target i-0abc123

# And if you need the SSH protocol (scp, rsync, git):
ssh ec2-user@i-0abc123    # ProxyCommand via SSM (config shown earlier)
scp -r ./code/ ec2-user@i-0abc123:/home/ec2-user/

6. Putting It All Together — A Realistic Stack

# Terraform spins up:
#   - VPC with private subnets
#   - Auto Scaling Group with Launch Template
#   - Launch Template references golden AMI (built by Packer)
#   - user-data does minimal "ansible-pull" for last-mile config
#   - IAM role attached with SSM + CloudWatch + S3 read
#   - Application Load Balancer in public subnet
#   - Security groups: ALB → instances on app port only, port 22 NEVER open
#   - EBS volumes with daily DLM snapshots
#   - CloudWatch agent ships /var/log + custom metrics
#   - Logs to CloudWatch Logs → subscription filter → Datadog/Splunk

# Deploys happen by:
#   1. Packer builds new AMI on git push
#   2. Terraform updates Launch Template's AMI ID
#   3. ASG instance refresh rolls instances one-by-one
#   4. Old AMI kept for 7 days (instant rollback)

# Day-2 ops:
#   - SSH via `aws ssm start-session`
#   - Logs in CloudWatch / Datadog
#   - Metrics in CloudWatch / Grafana
#   - Disk grow: modify-volume + growpart + resize2fs (zero downtime)
#   - Bigger instance: ASG rolling update with new instance type

Note: Wrap-up — Five things you'll use daily with cloud Linux: (1) fetching instance data via IMDSv2, (2) EBS volume attach + format + mount + fstab nofail, (3) bootstrapping via cloud-init user-data, (4) keyless SSH access via SSM Session Manager, (5) the golden AMI + immutable infra pattern. You'll repeat these so often they become muscle memory.

Tip: The interview question that lands you a senior-level (USD six-figure) offer — "Describe what happens from aws ec2 run-instances until an application is serving traffic." Answer: an EBS root volume is created from the AMI snapshot, an ENI is attached, the instance boots (UEFI → GRUB → kernel → initramfs → systemd) → cloud-init pulls user-data from IMDS → packages install, config files are written, services start → the application binds to its port → the health check passes → the ALB target group marks it healthy → traffic routes. If you can explain that whole chain in 60 seconds, the job is yours.

Further Learning

You've built the foundation. Now you need to keep the muscle warm. This section is a curated map of where to go next — channels, platforms, books, and certifications that actually move the needle in the global DevOps market. Spend 30 minutes a day here for the next three months and you will be unrecognizable.

Tip: Study advice #1 — Pick ONE channel and finish it. Don't channel-hop. Bookmark 50 videos and watch zero. Pick Abhishek Veeramalla's Linux + DevOps playlist OR NetworkChuck's Linux for hackers, and finish it end-to-end before you touch anything else.

YouTube channels worth your subscription

Concept-first creators (great for building intuition):

Abhishek Veeramalla — a gold standard for DevOps aspirants. His "DevOps Zero to Hero" playlist is one of the most recommended free resources in DevOps communities. Real interview questions, real scenarios.
Telusko (Navin Reddy) — clean Linux fundamentals, scripting basics, Docker primers. Great when you want a relaxed explanation.
Cyber Platter — strong on shell scripting, system administration, and Red Hat workflows. Useful for RHCSA prep.

Production-polished creators (depth + polish):

NetworkChuck — high-energy, hands-on. His "Linux for Hackers" series is the most fun way to fall in love with the terminal.
Learn Linux TV (Jay LaCroix) — calm, thorough, no-fluff. The Ubuntu Server and Bash scripting series are excellent.
TechWorld with Nana — best DevOps overview channel on YouTube. Her Docker, Kubernetes, and CI/CD crash courses are essentially free certification prep.
DistroTube — pure Linux culture, window managers, dotfiles, tooling. Watch when you want to fall deeper down the rabbit hole.

Free practice platforms (this is where skill actually grows)

OverTheWire — Bandit (overthewire.org/wargames/bandit) — 34 levels of SSH-driven Linux puzzles. If you can clear Bandit 0–20, you are operationally literate. This is non-negotiable.
SadServers (sadservers.com) — "broken Linux server" scenarios. You SSH in, diagnose, and fix. This is the closest thing to a real on-call shift you can practice for free.
Linux Journey (linuxjourney.com) — beautifully structured beginner-to-intermediate text path. Perfect for revision.
KillerCoda (killercoda.com) — browser-based Linux, Docker, Kubernetes labs. Free tier is generous.
HackTheBox — privilege escalation and CTF-style Linux. Sharpens your understanding of permissions, processes, and networking.
TryHackMe — gentler than HTB. Their "Linux Fundamentals" and "Linux PrivEsc" rooms are interview-grade prep.
LeetCode Shell — small but mighty. Bash one-liners that show up in screening rounds (word frequency, transpose file, valid phone numbers).

Caution: Study advice #2 — Tutorial hell is real. For every 1 hour of video you watch, spend 2 hours in a terminal breaking things. Recruiters do not care that you watched 200 hours of content. They care that you can fix a full disk in 90 seconds on a shared screen.

Documentation — the most underrated tier

man — the original. man bash, man 5 crontab, man 7 signal. Learn to navigate with / and n.
tldr — community-maintained "just the examples" pages. Install with npm i -g tldr or apt install tldr. tldr tar beats reading the man page when you just need the syntax.
info — GNU's hyperlinked docs. Heavier than man, but the coreutils info pages are gold.
DigitalOcean tutorials (digitalocean.com/community/tutorials) — the cleanest English Linux/server tutorials on the internet. Tagged by Ubuntu version. Bookmark this.
ArchWiki (wiki.archlinux.org) — even if you never touch Arch, the ArchWiki is the most accurate, distro-agnostic Linux reference that exists.
tldp.org — The Linux Documentation Project. Older but the "Advanced Bash-Scripting Guide" is a classic.

Books that earn shelf space

"How Linux Works" by Brian Ward (No Starch Press) — the single best book to understand why Linux behaves the way it does. Bootloaders, init, devices, networking. Read it twice.
"The Linux Command Line" by William Shotts — legally free PDF at linuxcommand.org/tlcl.php. The reference for shell mastery.
"Linux Bible" by Christopher Negus — fat, slightly dry, but covers RHEL/Fedora workflows thoroughly. Perfect companion for RHCSA.
"Site Reliability Engineering" (Google SRE Book) — free at sre.google/books. Not Linux-specific but it teaches you how to think like the engineers earning top-of-market (USD six-figure) salaries.
Bonus: "UNIX and Linux System Administration Handbook" by Nemeth et al. — the "purple book." If you go senior sysadmin, this is the bible.

Certifications — what actually matters in the DevOps market

Ranked by ROI for someone at your stage (an experienced developer pivoting to DevOps, targeting six-figure USD roles):

RHCSA (Red Hat Certified System Administrator) — EX200. The most respected Linux cert in the industry. Hands-on exam, no MCQs. Recruiters at enterprises, Red Hat partners, and most product companies recognize it instantly. It isn't cheap, but it's worth it if your goal is Linux-heavy DevOps roles.
LFCS (Linux Foundation Certified Sysadmin). Vendor-neutral, performance-based, cheaper (~$300, often 50% off). Less brand recognition than RHCSA but globally respected.
CompTIA Linux+. Decent foundational cert. MCQ-based. Good for resumes scanned by ATS bots, weaker signal to senior engineers.
LPIC-1. Vendor-neutral, two exams. Solid content but lower brand pull than RHCSA.
Honest take: AWS Solutions Architect Associate (SAA-C03) + provable Linux skills (Bandit + a GitHub of scripts) beats any pure-Linux cert for six-figure (USD) DevOps roles. Hiring managers want cloud + Linux, not Linux alone. If you can only afford one cert this year, do SAA-C03 first, then RHCSA.

Online sandboxes (practice without burning your laptop)

Killercoda — instant Ubuntu/CentOS/K8s playgrounds in the browser. No signup needed for basic labs.
JSLinux (bellard.org/jslinux) — a full Linux kernel running in your browser via JavaScript. Useful for quick "what does this command do" checks on a locked-down machine.
DigitalOcean — $200 free credit for new accounts, droplets from ~$6/month after. Cleanest UX for spinning up real VMs.
Linode (Akamai) — similar to DO, $100 credit, slightly cheaper at the low end.
AWS EC2 free tier — t2.micro / t3.micro free for 12 months. This should be your daily-driver sandbox because it also teaches you AWS.
GCP free tier — e2-micro is free forever in select regions. Great for a persistent personal Linux server.

Note: Study advice #3 — Build in public. Push every script, every dotfile, every broken-then-fixed config to a GitHub repo called linux-journey-2026. Tweet/LinkedIn one thing you learned each week. In six months that repo + timeline is worth more than any certificate, because it's proof of consistent practice — which is exactly what DevOps hiring managers screen for.

Next Steps After Linux

Linux is the floor, not the ceiling. Every DevOps tool you'll touch — Docker, Kubernetes, Ansible, Terraform, AWS, GCP, CI/CD runners — is just Linux wearing a costume. Here's how each next step builds directly on what you already know.

Linux → Bash (automate everything you just learned)

You already know cp, grep, find, awk, sed. Bash scripting is just gluing those into reusable files. The leap is small but career-changing: a sysadmin who writes scripts becomes a DevOps engineer.

Goal: be able to write a 50-line script with functions, argument parsing, error handling, and logging without Googling syntax.
Resource: "Bash Scripting Cheatsheet" (devhints.io/bash) + ShellCheck (shellcheck.net) for every script you write.
Project: write a backup script that tars a directory, uploads to S3, rotates old backups, and emails on failure. That one script demonstrates 80% of what mid-level DevOps does daily.

Linux → Docker (namespaces + cgroups + overlayfs, nothing magical)

Docker is not a separate technology. It's three Linux kernel features in a trench coat: namespaces (process/network/mount isolation), cgroups (CPU/memory limits), and overlayfs (layered filesystems). You already learned about processes, mounts, and users — Docker is just those concepts with sharper edges.

7-day Docker plan:

Day 1: Install Docker. Run docker run -it ubuntu bash. Explore. Notice it feels exactly like a Linux box because it is one.
Day 2: Images vs containers. docker pull, docker ps -a, docker exec, docker logs. Build mental model.
Day 3: Write your first Dockerfile. Containerize a simple Node/React app (use your existing skills).
Day 4: Volumes and bind mounts. Networks (bridge, host, none). Port publishing.
Day 5: docker-compose. Spin up a 3-service stack (React + Node API + Postgres).
Day 6: Image optimization — multi-stage builds, alpine base images, .dockerignore. Cut a 1.2GB image to 80MB.
Day 7: Push to Docker Hub. Pull on a cloud VM. Run in production-like mode with restart policies and healthchecks.

Linux → Kubernetes (nodes are Linux, pods are processes)

A Kubernetes cluster is a fleet of Linux machines (nodes) running containers (pods, which are groups of Linux processes) coordinated by an API server. Everything you debug in K8s ultimately resolves to kubectl exec -it pod -- sh followed by the same Linux commands you already know.

Path:

Install minikube or kind locally. One-node cluster on your laptop.
Learn pods → deployments → services → ingress in that order. Don't skip ahead.
Do the free Kubernetes Basics tutorial on kubernetes.io.
KillerCoda K8s scenarios — 30 minutes/day for 3 weeks.
Aim for CKAD (Certified Kubernetes Application Developer). It's hands-on, performance-based, and can add a meaningful bump to your offer. CKA comes later when you're ops-focused.

Linux → AWS (EC2 is Linux, Lambda is Linux, Fargate is Linux)

You will be shocked how much of "learning AWS" is actually "applying Linux on rented hardware." An EC2 instance is a Linux VM. A Lambda function runs on Amazon Linux 2. ECS Fargate runs containers on Linux. RDS is Postgres/MySQL on Linux. The cloud is just someone else's Linux box with an API in front.

Path:

AWS Solutions Architect Associate (SAA-C03) — your gateway cert. ~$150, three months of evening study. Stephane Maarek's Udemy course is the standard.
Hands-on: build a 3-tier app (VPC + EC2 + RDS + S3 + CloudFront) and tear it down with Terraform.
AWS DevOps Engineer Professional — once you have 1+ year of real AWS experience. This is the cert that anchors senior-level (USD six-figure) conversations.

30-day Linux-to-DevOps ramp (one focused task per day)

Ready to apply what you've learned? Work through the Hands-on DevOps Projects to build real systems with Linux, Docker, and CI/CD.

Install Ubuntu 24.04 in a VM. Customize .bashrc, install zsh + oh-my-zsh.
Complete OverTheWire Bandit levels 0–10.
Complete Bandit levels 11–20.
Write a shell script that audits your home directory: file count, largest files, oldest files.
Master grep, awk, sed with the GNU "Mastering Text Processing" cheatsheet.
Set up SSH key auth between your laptop and a free-tier EC2 instance.
Write a systemd service unit for a Node.js app. Enable, start, check status, view logs with journalctl.
Configure UFW or iptables on your VM. Allow only SSH + HTTP/HTTPS.
Install nginx, serve a static site, configure a reverse proxy to a Node app on :3000.
Add Let's Encrypt (certbot) for HTTPS on a free domain (DuckDNS or Freenom).
Write a cron job that backs up /etc nightly to a tarball.
Learn rsync. Sync the backups to an S3 bucket using the AWS CLI.
Read /proc/cpuinfo, /proc/meminfo. Write a script that summarizes system health.
Use strace on a misbehaving process. Identify which syscall is failing.
Install Docker. Run hello-world, then Ubuntu, then nginx.
Containerize your favorite React side project. Write the Dockerfile from scratch.
Multi-stage build: cut your image size by 80%.
Write a docker-compose.yml for React + Express + Postgres + Redis.
Push your image to Docker Hub. Pull and run it on the EC2 instance.
Install minikube locally. Deploy your app as a Kubernetes Deployment + Service.
Add a Kubernetes ConfigMap and Secret. Wire them to your pod.
Add Horizontal Pod Autoscaler. Generate load with hey or ab and watch it scale.
Write a Bash script that deploys your app: build image → push → kubectl apply → check rollout.
Set up GitHub Actions: on push to main, build Docker image and push to Docker Hub.
Extend GitHub Actions: SSH to EC2 and pull the new image (a tiny CD pipeline).
Sign up for AWS free tier. Launch an EC2 via the console, then via AWS CLI.
Create an S3 bucket. Upload, download, set lifecycle policy to delete after 30 days.
Write a Terraform file that creates a VPC + 1 EC2 + 1 S3 bucket. Apply, destroy, repeat.
Install Prometheus + Grafana on minikube. Scrape node metrics. Build one dashboard.
Final boss: deploy your React + Express + Postgres app to EC2, behind nginx with HTTPS, with a GitHub Actions pipeline that auto-deploys on push, and a cron-based DB backup to S3. Take a screenshot. Post it on LinkedIn with the GitHub repo. You are now a junior DevOps engineer.

30-60-90 day vision (and the 6-month payoff)

Day 30 — Linux comfort. You can SSH anywhere, debug a stuck process, write a 100-line bash script, and edit any system config without panic. You speak fluent terminal.
Day 60 — Docker + AWS basics. You've containerized 3+ apps, you understand VPC/EC2/S3/IAM, and you've deployed something real to production with HTTPS. You can answer "explain Docker to a junior" in an interview.
Day 90 — K8s + CI/CD interview-ready. You've passed (or can pass) CKAD-style scenarios, you have a working GitHub Actions pipeline in a public repo, and you've cleared 3-5 mock interviews. Target: junior/mid DevOps roles at strong product companies. With your background, recruiters will fast-track you because you bring product sense most freshers lack.
Month 6 — Mid-level inflection. One real production system on your resume, SAA-C03 passed, CKAD passed, contributing to an open-source DevOps tool. Target: mid-level DevOps / SRE / Platform Engineer roles. The frontend-to-DevOps story becomes your superpower — you understand what developers actually need from a platform, which most pure-ops engineers don't.

Note: Prefer a day-by-day path? This is covered in Mission 90 Days 1–20 — a free 90-day guided DevOps program with browser terminal missions.

Real world: The pep talk you came here for. You are not late. The global DevOps market in 2026 is desperate for engineers who can talk to developers and ops — because most candidates can only do one. Your years of shipping real features is not a liability; it is a wedge. Recruiters will pay a premium for someone who has actually pushed code to production and now understands the pipeline that delivers it.

Six months from now, a hiring manager is going to look at your GitHub, see 180 days of consistent commits, see one solid end-to-end project, see SAA-C03 and maybe CKAD on the resume, and make you a strong offer without blinking. The only thing standing between you and that offer is showing up to the terminal every day between now and then.

Close this tab. Open one. Type ssh. Begin.

Kubernetes for DevOps: Pods, Deployments, Services & More

opscanopy — Wed, 19 Aug 2026 15:31:42 +0000

Kubernetes (K8s) has become the standard runtime fabric for containerised workloads. Whether you are deploying a single microservice or coordinating hundreds of them across multiple availability zones, K8s gives you a consistent, declarative API to describe what should run, where it should run, and how it should behave. This guide walks through the core concepts a DevOps engineer needs day-to-day — cluster architecture, workload primitives, networking, configuration, observability, and troubleshooting — with working YAML and kubectl commands throughout. To see where Kubernetes fits in the broader learning journey, check the Kubernetes roadmap.

Why Kubernetes?

Running containers in production with plain Docker quickly exposes a set of operational problems: who restarts a crashed container, how do you roll out a new image version without downtime, how do you balance traffic across multiple replicas, and how do you ensure a container on a node with 64 GB RAM does not starve a neighbouring container of memory?

Kubernetes solves these problems through the desired-state model: you tell the cluster what the end state should look like (a YAML manifest), and controllers continuously reconcile actual state toward that goal. This gives you:

Self-healing — crashed containers are restarted; failed nodes trigger Pod rescheduling.
Horizontal scaling — add replicas with one command or automatically via HPA.
Rolling updates and rollbacks — new versions roll out gradually; one command reverts them.
Service discovery and load balancing — Pods get stable DNS names without static IPs.
Declarative configuration — your entire infrastructure lives in version-controlled YAML.

Note: Kubernetes does not replace a CI/CD pipeline. It is the runtime target; tools like Argo CD, Flux, or Jenkins deploy to it.

Cluster Architecture

Understanding the architecture prevents a lot of confusion when things go wrong.

Control Plane

The control plane is the brain of the cluster. In managed services (EKS, GKE, AKS) it is operated for you; in self-managed setups you are responsible for its reliability.

Component	Role
kube-apiserver	Single entry point for all cluster operations. Validates and persists objects to etcd. Every `kubectl` command talks to this.
etcd	Distributed key-value store. Holds all cluster state. Losing etcd without a backup loses your cluster.
kube-scheduler	Watches for new Pods with no assigned node and picks the best node based on resource requests, taints, affinity rules, and topology spread.
kube-controller-manager	Runs core controllers: ReplicaSet, Deployment, Node, Endpoints, ServiceAccount, and others. Each controller reconciles its resource type.
cloud-controller-manager	(Cloud clusters only) Interfaces with cloud provider APIs — creates load balancers, routes, storage volumes.

Worker Nodes

Every node that runs workloads must have three components:

Component	Role
kubelet	Agent on each node. Receives PodSpec from the API server and instructs the container runtime to start/stop containers. Reports node and Pod status.
kube-proxy	Maintains iptables/ipvs rules that implement Service networking — forwarding traffic to the correct Pod IPs.
Container runtime	Actually runs containers. Kubernetes supports any CRI-compatible runtime: containerd (most common), CRI-O, or Docker (via cri-dockerd shim).

Tip: In production, the control plane nodes are typically tainted so user workloads do not schedule on them. Managed services do this automatically.

How a Pod Gets Scheduled

You run kubectl apply -f pod.yaml.
kube-apiserver validates the manifest and stores it in etcd.
kube-scheduler notices a Pod with no nodeName and selects a node.
kubelet on that node watches the API server, sees the Pod assigned to it, pulls the image, and starts the container.
kube-proxy updates network rules so the Pod is reachable via its Service.

Kubernetes cluster architecture: control plane (API server, etcd, scheduler, controller-manager) communicates with worker nodes (kubelet, kube-proxy, runtime) that host Pods.

Pods

A Pod is the smallest schedulable unit in Kubernetes. It wraps one or more containers (built from Docker for DevOps images) that share:

Network namespace — all containers in a Pod share the same IP address and port space.
Storage — volumes are declared at the Pod level and mounted into individual containers.
Lifecycle — all containers start and stop together with the Pod.

Minimal Pod Manifest

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
  labels:
    app: web
spec:
  containers:
    - name: nginx
      image: nginx:1.27
      ports:
        - containerPort: 80

kubectl apply -f pod.yaml
kubectl get pods
kubectl describe pod web-pod
kubectl delete pod web-pod

Pod Lifecycle

A Pod moves through these phases:

Phase	Meaning
`Pending`	Pod accepted by the cluster; containers not yet started (pulling images, waiting for node).
`Running`	At least one container is running or starting.
`Succeeded`	All containers exited with code 0 (typical for Jobs).
`Failed`	At least one container exited with non-zero code and will not restart.
`Unknown`	Node communication lost; state cannot be determined.

Multi-Container Patterns

Pods can host multiple containers. The three canonical patterns are:

Sidecar — a helper container that augments the main one (e.g., a log-shipper or a service mesh proxy like Envoy/Istio).
Init container — runs to completion before the main containers start (e.g., database migration, config rendering from a vault).
Ambassador — a proxy container that abstracts external services (e.g., a Redis proxy in front of a remote cluster).

apiVersion: v1
kind: Pod
metadata:
  name: app-with-sidecar
spec:
  initContainers:
    - name: init-migrate
      image: migrate-tool:v1
      command: ["./migrate", "--up"]
  containers:
    - name: app
      image: myapp:v2
      ports:
        - containerPort: 8080
    - name: log-shipper
      image: fluent/fluent-bit:3.0
      volumeMounts:
        - name: app-logs
          mountPath: /var/log/app
  volumes:
    - name: app-logs
      emptyDir: {}

Why You Rarely Create Bare Pods

Bare Pods are not rescheduled if the node they are on fails. You should almost always use a higher-level workload controller (Deployment, StatefulSet, DaemonSet) that manages Pods on your behalf.

ReplicaSets and Deployments

ReplicaSet

A ReplicaSet ensures a specified number of identical Pod replicas are running at any time. It uses a label selector to identify the Pods it owns. If a Pod dies, the ReplicaSet creates a replacement.

Note: You almost never create ReplicaSets directly. Deployments create and manage ReplicaSets for you, adding rollout history.

Deployments

A Deployment wraps a ReplicaSet and adds rollout logic: you update the Pod template and the Deployment gradually replaces old Pods with new ones according to the configured strategy.

A Deployment owns a ReplicaSet, which owns and maintains the desired number of identical Pods via a label selector.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: myapp:v2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"

Key fields:

selector.matchLabels — the Deployment manages any Pod with these labels. Must match template.metadata.labels.
strategy.rollingUpdate.maxUnavailable — how many Pods can be unavailable during a rollout (absolute or percentage).
strategy.rollingUpdate.maxSurge — how many extra Pods can be created above the desired count during a rollout.

# Apply or update
kubectl apply -f deployment.yaml

# Watch rollout progress
kubectl rollout status deployment/web-deployment

# Scale manually
kubectl scale deployment web-deployment --replicas=5

# View rollout history
kubectl rollout history deployment/web-deployment

# Rollback one version
kubectl rollout undo deployment/web-deployment

Tip: Always set resources.requests and resources.limits. Without them, the scheduler cannot make good placement decisions and your Pods get the lowest QoS class.

Services

A Service gives a stable DNS name and IP to a dynamic set of Pods selected by a label selector. Even as Pods are created and destroyed, the Service's IP (ClusterIP) and DNS name stay constant.

Service Types

Type	Accessibility	Typical Use
`ClusterIP` (default)	Inside the cluster only	Internal microservice communication
`NodePort`	Cluster nodes' IPs on a static port (30000–32767)	Dev/test external access without a cloud LB
`LoadBalancer`	External IP via cloud provider's load balancer	Production external traffic
`ExternalName`	DNS alias for an external service	Pointing in-cluster workloads at external endpoints

Service types and Ingress: Ingress routes HTTP traffic by host/path to Services, while LoadBalancer and NodePort expose Services externally; ClusterIP is internal-only.

ClusterIP Example

apiVersion: v1
kind: Service
metadata:
  name: web-service
  namespace: production
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: ClusterIP

Pods in the same namespace can reach this service at web-service:80; pods in other namespaces use web-service.production.svc.cluster.local:80.

LoadBalancer Example

apiVersion: v1
kind: Service
metadata:
  name: web-lb
  namespace: production
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 443
      targetPort: 8080
  type: LoadBalancer

How kube-proxy Implements Services

kube-proxy watches the API server for Service and Endpoints objects. When a Service is created, it programs iptables rules (or IPVS rules in IPVS mode) on every node so that traffic to the ClusterIP is DNAT'd to one of the healthy Pod IPs in the Endpoints list.

kubectl get svc -n production
kubectl describe svc web-service -n production
kubectl get endpoints web-service -n production

Caution: A Service with no matching Pods will have an empty Endpoints list and traffic will be silently dropped. Always check kubectl get endpoints when debugging connectivity.

Ingress

A Service of type LoadBalancer creates a separate cloud load balancer per service — expensive at scale. An Ingress is a cluster-wide HTTP(S) router that multiplexes many Services behind a single load balancer using hostname and path rules.

Ingress Controller

An Ingress object is just a configuration resource. You also need an Ingress controller — a Pod that reads Ingress objects and programs a reverse proxy. Common choices:

NGINX Ingress Controller (most widely deployed)
AWS Load Balancer Controller (for ALB on EKS — see AWS for DevOps Engineers)
Traefik (dynamic config, good for smaller clusters)
Istio Gateway (service mesh, advanced traffic management)

Ingress Manifest with TLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - app.example.com
      secretName: tls-app-example-com
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-service
                port:
                  number: 80

The tls.secretName field references a Kubernetes Secret of type kubernetes.io/tls that holds the certificate and key. Tools like cert-manager automate TLS certificate provisioning and renewal via Let's Encrypt.

kubectl get ingress -n production
kubectl describe ingress web-ingress -n production

Tip: Set ingressClassName explicitly. Older clusters used annotations (kubernetes.io/ingress.class), but the ingressClassName field is the current standard.

ConfigMaps

A ConfigMap stores non-sensitive configuration as key-value pairs or files. Pods consume ConfigMaps as environment variables or as mounted files.

Creating a ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"
  config.yaml: |
    server:
      host: 0.0.0.0
      port: 8080
    database:
      pool_size: 10

kubectl apply -f configmap.yaml
kubectl get configmap app-config -n production -o yaml

Consuming a ConfigMap as Environment Variables

spec:
  containers:
    - name: app
      image: myapp:v2
      envFrom:
        - configMapRef:
            name: app-config

Or inject individual keys:

      env:
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: LOG_LEVEL

Consuming a ConfigMap as a Mounted File

spec:
  containers:
    - name: app
      image: myapp:v2
      volumeMounts:
        - name: config-volume
          mountPath: /etc/app
  volumes:
    - name: config-volume
      configMap:
        name: app-config
        items:
          - key: config.yaml
            path: config.yaml

The file /etc/app/config.yaml inside the container will contain the YAML defined in the ConfigMap.

Note: When you update a ConfigMap that is mounted as a volume, Kubernetes eventually (within ~60 seconds) updates the file in the container. Environment variables injected via envFrom are not updated — the Pod must be restarted.

Secrets

Secrets store sensitive data — passwords, API tokens, TLS certificates, SSH keys. They are similar to ConfigMaps but:

Values are base64-encoded (not encrypted by default, but can be with encryption at rest).
Access is controlled by RBAC — separate from ConfigMap access.
They integrate with external secret stores (HashiCorp Vault, AWS Secrets Manager) via CSI drivers or operators.

Secret Types

Type	Use
`Opaque` (default)	Arbitrary key-value data
`kubernetes.io/tls`	TLS certificate and key
`kubernetes.io/dockerconfigjson`	Docker registry credentials
`kubernetes.io/service-account-token`	SA tokens (auto-managed)
`kubernetes.io/ssh-auth`	SSH private keys
`kubernetes.io/basic-auth`	Username/password pairs

ConfigMaps and Secrets inject configuration into Pods as environment variables (envFrom / secretKeyRef) or as mounted files in volumes.

Creating an Opaque Secret

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
data:
  username: YWRtaW4=        # base64("admin")
  password: c3VwZXJzZWNyZXQ= # base64("supersecret")

# Encode a value
echo -n "supersecret" | base64

# Create from literal (easier)
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=supersecret \
  -n production

# View (values are base64)
kubectl get secret db-credentials -n production -o yaml

# Decode a value
kubectl get secret db-credentials -n production \
  -o jsonpath='{.data.password}' | base64 --decode

Consuming a Secret in a Pod

spec:
  containers:
    - name: app
      image: myapp:v2
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Encryption at Rest and RBAC

By default, Secrets in etcd are stored base64-encoded but not encrypted. Enable encryption at rest via the API server's --encryption-provider-config flag using an aescbc or kms provider.

Use RBAC to restrict which service accounts can get, list, or watch Secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials"]
    verbs: ["get"]

Caution: Never commit Secret manifests with real values to version control. Use sealed secrets (Bitnami Sealed Secrets), External Secrets Operator, or a Vault CSI driver to inject secrets at runtime.

Namespaces

Namespaces provide a mechanism for isolating groups of resources within a single cluster. They are soft multi-tenancy: they scope names, RBAC, and resource quotas, but do not provide strong network isolation by themselves (for that, use NetworkPolicies).

Built-in Namespaces

Namespace	Purpose
`default`	Default for resources with no namespace specified
`kube-system`	Kubernetes control plane components (CoreDNS, kube-proxy, etc.)
`kube-public`	Publicly readable resources (ConfigMap with cluster info)
`kube-node-lease`	Node heartbeat lease objects

Common Namespace Operations

# List all namespaces
kubectl get namespaces

# Create a namespace
kubectl create namespace staging

# Run commands in a specific namespace
kubectl get pods -n staging

# Set a default namespace for your context
kubectl config set-context --current --namespace=staging

# View resources across all namespaces
kubectl get pods --all-namespaces
# or
kubectl get pods -A

ResourceQuota and LimitRange

Namespaces support ResourceQuota (cap total resource usage per namespace) and LimitRange (set default and max limits per container).

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: staging
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    count/pods: "20"

Resource Requests and Limits

Getting resource configuration right is one of the highest-leverage tuning tasks in Kubernetes.

The Concepts

Request: the amount of CPU/memory the scheduler reserves on a node for this container. The node must have at least this much available for the Pod to be scheduled there.
Limit: the maximum CPU/memory the container is allowed to use. Exceeding the CPU limit causes throttling; exceeding the memory limit causes the container to be OOM-killed.

resources:
  requests:
    cpu: "250m"      # 250 millicores = 0.25 CPU core
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

CPU is expressed in cores (1, 0.5) or millicores (500m). Memory uses binary suffixes (Ki, Mi, Gi).

QoS Classes

Kubernetes assigns one of three Quality of Service classes to every Pod based on its resource configuration:

QoS Class	Condition	Eviction Priority
Guaranteed	Every container has `requests == limits` for both CPU and memory	Last to be evicted
Burstable	At least one container has a request or limit, but not Guaranteed	Middle priority
BestEffort	No container has any request or limit	First to be evicted under pressure

CPU/memory request (scheduler reservation) versus limit (hard ceiling), and the three QoS classes that determine eviction order under node pressure.

Tip: For production workloads, aim for Guaranteed or Burstable QoS. Use kubectl describe pod <name> and look for QoS Class: in the output.

Setting the Right Numbers

Right-sizing requests and limits requires profiling your actual workload under load. Size CPU/memory requests and limits with the Kubernetes Resource Calculator.

Common mistakes:

Setting limits much higher than requests creates "noisy neighbour" risk.
Setting limits equal to very low requests may cause OOMKills under normal spikes.
Not setting requests at all results in BestEffort Pods that are first to go under node pressure.

Vertical Pod Autoscaler (VPA) and Horizontal Pod Autoscaler (HPA)

HPA scales the number of replicas based on CPU utilisation, memory, or custom metrics.
VPA adjusts the requests/limits of existing containers (requires Pod restarts).

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-deployment
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Health Probes

Kubernetes uses three probe types to determine container health. Without probes, a container is considered healthy the moment it starts — even if the application inside is not ready yet.

Probe Types

Probe	Kubernetes Action on Failure
Liveness	Restart the container. Use for detecting deadlocks — processes that are running but stuck.
Readiness	Remove the Pod from Service endpoints. Traffic stops flowing to it, but the container is not restarted.
Startup	Disables liveness and readiness until it succeeds. Use for slow-starting apps to prevent premature liveness restarts.

Probe Mechanisms

Each probe type can use one of three mechanisms:

httpGet — HTTP GET request; success = 2xx/3xx response.
tcpSocket — TCP connection; success = port is open.
exec — run a command inside the container; success = exit code 0.

Full Probe Example

spec:
  containers:
    - name: app
      image: myapp:v2
      ports:
        - containerPort: 8080
      startupProbe:
        httpGet:
          path: /healthz
          port: 8080
        failureThreshold: 30
        periodSeconds: 10
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 10
        failureThreshold: 3
        successThreshold: 1
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 30
        periodSeconds: 15
        failureThreshold: 3
        timeoutSeconds: 5

Key parameters:

Parameter	Default	Meaning
`initialDelaySeconds`	0	Seconds after container start before first probe
`periodSeconds`	10	How often to probe
`failureThreshold`	3	Failures before action is taken
`successThreshold`	1	Successes needed to transition back to healthy
`timeoutSeconds`	1	Seconds to wait for probe response

Tip: Always implement separate /healthz (liveness) and /ready (readiness) endpoints in your application. The readiness endpoint should check downstream dependencies (database reachability, cache connection); the liveness endpoint should only check the application itself.

Rollouts and Rollbacks

Rolling Update Strategy

The RollingUpdate strategy (default for Deployments) replaces old Pods with new ones incrementally. The Recreate strategy terminates all old Pods before starting new ones — useful when your app cannot run two versions simultaneously.

# Trigger a rollout by updating the image
kubectl set image deployment/web-deployment \
  web=myapp:v3 -n production

# Or edit the manifest and apply
kubectl apply -f deployment.yaml

# Watch the rollout in real time
kubectl rollout status deployment/web-deployment -n production

# Pause a rollout mid-way (canary-style)
kubectl rollout pause deployment/web-deployment -n production

# Resume
kubectl rollout resume deployment/web-deployment -n production

Rollback

# View rollout history with change-cause annotations
kubectl rollout history deployment/web-deployment -n production

# View a specific revision
kubectl rollout history deployment/web-deployment \
  --revision=2 -n production

# Roll back to previous version
kubectl rollout undo deployment/web-deployment -n production

# Roll back to a specific revision
kubectl rollout undo deployment/web-deployment \
  --to-revision=2 -n production

Tip: Annotate your rollouts with a change-cause so history is readable: add kubernetes.io/change-cause: "bump to v3, adds auth middleware" to the Deployment's metadata.annotations before applying.

Deployment Revision History

By default, Kubernetes retains 10 revisions. Control this with spec.revisionHistoryLimit.

spec:
  revisionHistoryLimit: 5

Observability

A Kubernetes cluster generates a rich stream of metrics, logs, and events. Building good visibility requires three layers:

Logs

kubectl logs reads from the container's stdout/stderr (captured by the container runtime and available via the API server):

# Follow logs from a running Pod
kubectl logs -f deployment/web-deployment -n production

# Show last 100 lines
kubectl logs web-pod --tail=100 -n production

# Logs from a specific container in a multi-container Pod
kubectl logs web-pod -c log-shipper -n production

# Logs from a previous (crashed) container instance
kubectl logs web-pod --previous -n production

For persistent log aggregation, deploy a log collector DaemonSet (Fluent Bit, Fluentd) that ships container logs to a backend (Elasticsearch, Loki, CloudWatch Logs).

Metrics and Prometheus

The Kubernetes ecosystem is deeply integrated with Prometheus. The metrics-server provides basic CPU/memory metrics for HPA and kubectl top. For full observability, deploy the kube-prometheus-stack (Prometheus + Alertmanager + Grafana) or a managed equivalent.

Key scrape targets:

kube-state-metrics — Deployment replicas, Pod status, resource requests/limits, PVC state.
node-exporter — node-level CPU, memory, disk, network.
cadvisor — per-container CPU and memory (built into kubelet).
Your application's /metrics endpoint (expose Prometheus format).

Understanding Prometheus query syntax is essential for writing alerts and dashboards. Use the PromQL Explainer to translate complex queries into plain English. When building scrape configurations and relabelling rules, the Prometheus Relabel Tester lets you test rules against live label sets without touching production.

Events

Kubernetes Events are the first place to look when a Pod is not starting correctly:

kubectl get events -n production --sort-by='.lastTimestamp'
kubectl get events -n production --field-selector reason=OOMKilled

kubectl Essentials

Command	What It Does
`kubectl get pods -n <ns>`	List Pods in a namespace
`kubectl get pods -A`	List Pods in all namespaces
`kubectl describe pod <name> -n <ns>`	Full details, events, conditions
`kubectl logs <pod> -n <ns>`	Stdout/stderr from the main container
`kubectl logs <pod> -c <container>`	Logs from a specific container
`kubectl logs <pod> --previous`	Logs from the last crashed container instance
`kubectl exec -it <pod> -- bash`	Interactive shell in a running container
`kubectl exec <pod> -- <cmd>`	Run a one-off command
`kubectl apply -f <file>`	Declaratively create or update resources
`kubectl delete -f <file>`	Delete resources defined in a manifest
`kubectl delete pod <name>`	Force-delete a single Pod
`kubectl scale deployment <name> --replicas=N`	Scale a Deployment
`kubectl rollout status deployment/<name>`	Watch rollout progress
`kubectl rollout undo deployment/<name>`	Roll back to previous version
`kubectl rollout history deployment/<name>`	View revision history
`kubectl set image deployment/<name> <c>=<img>`	Update a container image
`kubectl port-forward svc/<name> 8080:80`	Forward a local port to a Service
`kubectl port-forward pod/<name> 8080:8080`	Forward a local port to a Pod
`kubectl top pods -n <ns>`	Current CPU/memory usage (requires metrics-server)
`kubectl top nodes`	Node CPU/memory usage
`kubectl get events -n <ns> --sort-by=.lastTimestamp`	Recent cluster events
`kubectl explain deployment.spec.strategy`	Inline API documentation
`kubectl diff -f <file>`	Preview what `apply` would change
`kubectl get all -n <ns>`	All common resources in a namespace
`kubectl label pod <name> key=value`	Add or update a label
`kubectl annotate pod <name> key=value`	Add or update an annotation
`kubectl config get-contexts`	List available kubeconfig contexts
`kubectl config use-context <context>`	Switch cluster context

Tip: Install kubectl plugins via krew (the kubectl plugin manager). Useful plugins include kubectl-neat (clean up get -o yaml output), kubectl-ctx and kubectl-ns (fast context/namespace switching), and kubectl-tree (visualise owner references).

Troubleshooting

CrashLoopBackOff

A container keeps crashing and Kubernetes keeps restarting it with exponential backoff.

Diagnose:

# See the current state and restart count
kubectl get pod <name> -n <ns>

# Check events for the Pod
kubectl describe pod <name> -n <ns>

# Check logs from the current or previous instance
kubectl logs <name> -n <ns>
kubectl logs <name> -n <ns> --previous

Common causes: application throws an unhandled exception at startup; missing environment variable or config file; entrypoint command is wrong; dependency (database, cache) not reachable.

ImagePullBackOff / ErrImagePull

The container image cannot be pulled.

Diagnose:

kubectl describe pod <name> -n <ns>
# Look for events like: Failed to pull image ... unauthorized

Common causes: image name or tag is wrong; registry is private and no imagePullSecret is configured; credentials in the pull secret have expired; rate-limit on Docker Hub.

Fix a missing imagePullSecret:

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=<user> \
  --docker-password=<token> \
  -n production

Then reference it in the Pod spec:

spec:
  imagePullSecrets:
    - name: regcred

Pending Pods

A Pod stays in Pending and never starts.

Diagnose:

kubectl describe pod <name> -n <ns>
# Look for events under "Events:" — scheduler messages explain why

Common causes:

Insufficient resources — no node has enough CPU or memory to satisfy the requests. Check kubectl top nodes and kubectl describe nodes.
Taint/toleration mismatch — the target nodes are tainted but the Pod has no matching toleration.
NodeSelector or affinity — the Pod requires specific labels on the node that no current node has.
PVC not bound — a PersistentVolumeClaim is still Pending (waiting for a PV).

OOMKilled

A container was killed by the kernel OOM killer because it exceeded its memory limit.

Diagnose:

kubectl get pod <name> -n <ns>
# Look for: OOMKilled in the REASON column

kubectl describe pod <name> -n <ns>
# Look for: Last State: Terminated Reason: OOMKilled

Fix: Increase the memory limit (or request, to get a Guaranteed QoS). Profile the application's actual memory use under load. Use the Kubernetes Resource Calculator to size requests and limits based on observed usage.

Debugging Toolkit

When kubectl logs is not enough:

# Open a shell in the running container
kubectl exec -it <pod> -n <ns> -- sh

# Run a temporary debug Pod on a specific node
kubectl debug node/<node-name> -it --image=ubuntu:22.04

# Attach an ephemeral debug container to a running Pod (K8s 1.23+)
kubectl debug -it <pod> -n <ns> \
  --image=busybox:1.36 \
  --target=<container-name>

# Copy a file from a Pod to local
kubectl cp <ns>/<pod>:/path/to/file ./local-file

Common Kubernetes Interview Questions

Q: What is the role of the etcd in a Kubernetes cluster?
A: etcd is the cluster's source of truth — a distributed, consistent key-value store where all cluster state (objects, configs, secrets) is persisted. If etcd is lost without a backup, the cluster's configuration is lost. The API server is the only component that reads and writes etcd directly.

Q: How does a Kubernetes Service discover which Pods to send traffic to?
A: Services use label selectors to identify target Pods. The Endpoints controller (or EndpointSlice controller in newer clusters) watches for Pods matching the selector and maintains an Endpoints/EndpointSlice object. kube-proxy reads these to program iptables/IPVS forwarding rules.

Q: What is the difference between a StatefulSet and a Deployment?
A: A Deployment is for stateless workloads — Pods are interchangeable. A StatefulSet is for stateful applications (databases, queues): it gives each Pod a stable, ordered hostname (pod-0, pod-1), stable storage via PVCs that survive Pod rescheduling, and ordered, graceful start/stop.

Q: What is a DaemonSet?
A: A DaemonSet ensures that one copy of a Pod runs on every node (or a subset via nodeSelector/tolerations). Common uses: log collectors (Fluent Bit), monitoring agents (node-exporter), CNI plugins, security agents.

Q: How does Kubernetes handle persistent storage?
A: Via the PersistentVolume (PV) / PersistentVolumeClaim (PVC) abstraction. A PVC is a user's request for storage (size, access mode). Kubernetes binds it to a matching PV (a piece of actual storage — EBS volume, NFS share, etc.) either statically or dynamically via a StorageClass and a CSI driver.

Q: What is a Kubernetes Namespace and when should you use multiple namespaces?
A: A Namespace is a virtual cluster within a cluster. Use separate namespaces for teams, environments (dev/staging/production on the same cluster), or application boundaries. Apply ResourceQuotas and RBAC per namespace to enforce isolation.

Q: What is a Taint and Toleration?
A: Taints are set on nodes to repel Pods. kubectl taint nodes node1 key=value:NoSchedule prevents any Pod without a matching Toleration from being scheduled on that node. Used to dedicate nodes for specific workloads (GPU nodes, spot nodes).

Q: What happens when you run kubectl apply -f deployment.yaml?
A: kubectl sends the manifest to the API server, which validates it and merges it with the existing object (three-way merge using the last-applied-configuration annotation). If the Deployment's Pod template changed, the Deployment controller creates a new ReplicaSet and performs a rolling update.

Q: What is the difference between kubectl apply and kubectl create?
A: create is imperative — it fails if the object already exists. apply is declarative — it creates the object if missing or updates it if it exists, using a three-way merge. Prefer apply for GitOps and CI/CD pipelines.

Q: What is a Kubernetes Job vs a CronJob?
A: A Job creates one or more Pods and tracks their completion. It retries failed Pods until the desired completions count is reached, then the Job is marked complete. A CronJob schedules Jobs on a cron expression, creating a new Job object on each trigger.

Note: Prefer a day-by-day path? This is covered in Mission 90 Days 66–74 — a free 90-day guided DevOps program with browser terminal missions.

What's Next

Coming from containers? See Docker for DevOps.

Running on AWS (EKS)? See AWS for DevOps Engineers.

Docker Interview Prep: Scenario-Based Questions & Answers for DevOps

opscanopy — Mon, 17 Aug 2026 15:36:01 +0000

This guide is built for DevOps interview prep. It covers the ten core Docker topics that come up repeatedly — from why containers exist through to a production-grade troubleshooting playbook — using scenario-based questions, clear answers, worked examples, and the gotchas interviewers love to probe. Every command is real and runnable on Ubuntu 24.04 with a current Docker Engine. Before diving in, bookmark the Docker roadmap to see how these topics fit into a full learning path.

Study the concepts first in Docker for DevOps.

Why Containers Exist

Interview scenario: "You shipped a React app and it works on your machine but breaks in production. How do you explain what went wrong, and what's the architectural fix?"

Your laptop has Node 20, a specific OpenSSL version, certain environment variables, and a particular Ubuntu setup. The production server has Node 18, a different libc, and missing env vars. Same code, different behaviour. The bug is not in your code — it is in the environment around your code.

Traditionally, teams fixed this with long setup documents, install scripts, and "please match these exact versions" checklists. It never fully worked because the operating system, system libraries, and dependencies drifted between every developer's machine and every server. Containers solve this by packaging your app together with its entire runtime environment — the right Node version, libraries, files, and config — into one portable unit that runs identically everywhere.

Note: Think of a shipping container. Before standardised containers, dock workers loaded ships piece by piece — barrels, boxes, sacks — and every port handled it differently, slowly, and unreliably. The steel shipping container changed the world: one standard box, the same shape everywhere, that any crane, truck, or ship can handle without caring what's inside. Docker does the same for software.

Virtual Machines vs Containers

Before containers, the common way to isolate apps was the Virtual Machine (VM). A VM emulates an entire computer — including a full guest operating system — on top of a hypervisor. That gives strong isolation, but it is heavy: every VM carries its own multi-gigabyte OS.

Containers take a smarter approach. They share the host's OS kernel and only package the app plus its dependencies, making them tiny and fast.

Architecture comparison: In a VM setup, each application runs inside its own Guest OS on a Hypervisor, all sitting on Host OS + Hardware. Every VM is gigabytes in size. In a container setup, applications each have their own bins/libs but share a single Docker Engine on top of a shared Host OS Kernel. No guest OS means containers are megabytes, not gigabytes.

Aspect	Virtual Machine	Container
Boot time	Minutes (full OS boot)	Seconds or less
Size	Gigabytes (includes guest OS)	Megabytes (app + deps only)
Isolation	Strong — full hardware-level isolation via hypervisor	Process-level isolation via kernel namespaces & cgroups
Resource use	Heavy — each VM runs its own OS	Lightweight — shares host kernel
OS kernel	Each VM has its own kernel	All containers share the host kernel
Portability	Portable but bulky to move	Highly portable — small images, run anywhere Docker runs

Note: A team running 10 microservices as VMs would need 10 guest operating systems eating ~20 GB of RAM with minutes to spin up. As containers on a single host, those same 10 services share one kernel, boot in seconds, and fit comfortably in a fraction of the memory.

What Docker Actually Solves

Docker is not the same thing as "containers" — Linux had the underlying technology (namespaces and cgroups) for years. What Docker did was make containers easy: a simple way to define an environment (the Dockerfile), a standard format to package it (the image), a registry to share it (Docker Hub), and a clean CLI to run it. Consistency from laptop to CI to production, fast startup, and efficient resource use — that is the Docker promise.

Tip: "What's the difference between a VM and a container?" gets asked in almost every DevOps interview. Nail the one-liner: A VM virtualizes hardware and runs a full guest OS on a hypervisor; a container virtualizes the OS and shares the host kernel, packaging only the app and its dependencies. Then add the consequences — containers are smaller, boot faster, and use fewer resources, while VMs give stronger isolation. Bonus points: they are complementary — you often run containers inside VMs in the cloud.

Core Concepts: Images, Containers, Registry, and the Engine

Images vs Containers

This is the single most important distinction in Docker, and it maps perfectly onto something any developer already knows: class vs object. An image is a read-only blueprint — like a class definition. A container is a running instance of that image — like an object created with new. From one image you can spin up many containers, just like one class can produce many objects, each with its own state.

Note: An image is the recipe; a container is the actual dish you cooked from it. One recipe, many plates. The recipe never changes when you cook — and your image stays read-only while each container runs its own live copy.

Aspect	Image	Container
What it is	Read-only template / blueprint	Running instance of an image
State	Immutable, static	Live, has writable layer & runtime state
Analogy	Class / recipe	Object / cooked dish
Lifecycle	Built once, stored, reused	Created, started, stopped, removed
Quantity	One image	Many containers from that one image

One immutable image produces many independent containers, each adding its own ephemeral writable layer on top.

The Docker Engine Model

"Docker" is actually several pieces working together. Understanding this chain is valuable in interviews.

The flow is: CLI Client → (REST API) → Docker Daemon (dockerd) → (pull/push) → Registry (Docker Hub / ECR)

CLI Client — the docker command you type. It is a thin client that sends requests to the daemon over a REST API (Unix socket). It does no container work itself.
Docker Daemon (dockerd) — the background service that does the real work: building images, managing containers, networks, and volumes. It delegates container execution to containerd/runc.
Images — read-only templates stored locally by the daemon. If a requested image is absent, the daemon pulls it from the registry. Each image is a stack of layers.
Containers — running instances the daemon creates from images by adding a writable layer, then launching the process via containerd → runc with kernel namespaces and cgroups.
Registry — remote image store (Docker Hub, ECR, GHCR). The daemon pulls images down when missing and pushes built images up to share.

Image Layers and the Union Filesystem

A Docker image is not one solid blob — it is built from stacked layers. Each instruction in a Dockerfile (install a package, copy files, set config) creates a new read-only layer on top of the previous one. Docker uses a union filesystem to merge these layers into a single view that the container sees as one normal filesystem.

Two big wins come from this:

Caching — if a layer hasn't changed, Docker reuses the cached version instead of rebuilding it, so builds are fast.
Sharing — if ten images all start FROM ubuntu:24.04, that base layer is stored on disk only once and shared between them.

When a container runs, Docker adds a thin writable layer on top — all runtime changes live there, and the underlying image layers stay untouched.

Layer stack for a Node app (bottom to top):

FROM node:20-alpine        ← base layer (read-only, cached)
COPY package.json ./       ← layer: package.json
RUN npm ci                 ← layer: node_modules (slow, cached when deps unchanged)
COPY . .                   ← layer: app source (changes often)
CMD ["node","server.js"]   ← metadata: start command
                           ← writable container layer (top, ephemeral)

When you change a layer, Docker must rebuild that layer and every layer above it. Layers below it stay cached. This is why dependencies go early and source goes last.

Tip: Order your Dockerfile from least-changing to most-changing. Put COPY package.json and npm install before COPY . . — that way, changing your source code doesn't bust the cached dependency layer, and rebuilds stay fast.

Registry and Docker Hub

A registry is a storage and distribution system for images — think of it as "GitHub for Docker images." Docker Hub is the default public registry, hosting official images like nginx, node, postgres, and ubuntu. You pull images down to run them and push your own images up to share them. Companies usually run private registries too (AWS ECR, GitHub Container Registry, Harbor) for internal images.

Note: When you run docker pull nginx, Docker contacts Docker Hub, finds the official nginx image, and downloads it layer by layer. Each layer is pulled separately — and any layers already present from a previous pull are skipped. That is the layer-sharing system saving bandwidth and disk in real time.

docker run nginx
# CLI (client) ── REST API ──> dockerd (daemon)
#                                 │
#                  pulls image from registry if missing
#                                 │
#                              containerd ──> runc ──> container process

Tip: "Walk me through what happens when you run docker run nginx." Strong answer: (1) the CLI sends the request to the Docker daemon via the API; (2) the daemon checks for the nginx image locally; (3) if missing, it pulls the image layers from the registry; (4) it creates a new container by stacking a writable layer on the image; (5) it hands off to containerd/runc, which sets up namespaces and cgroups; (6) the container's main process starts. Mentioning the client–daemon split and containerd/runc shows real depth.

Essential Docker CLI

These are the commands you will type every single day. Each one below has a real example and a clear use case.

Container lifecycle: Created → Running → Stopped → Removed, driven by docker create → docker start/run → docker stop → docker rm.

docker run — create and start a container

The workhorse. Key flags: -d (detached/background), -p host:container (publish a port), --name (friendly name), -it (interactive terminal), --rm (auto-delete on exit), -e (environment variable), -v (mount a volume).

docker run -d -p 8080:80 --name web nginx

Run nginx in the background, mapping host port 8080 to container port 80. Open http://localhost:8080 to see it.

docker run -it --rm ubuntu:24.04 bash

Get an interactive shell inside a throwaway Ubuntu container that auto-deletes the moment you exit — perfect for quick experiments.

docker run -d --name db -e POSTGRES_PASSWORD=secret -v pgdata:/var/lib/postgresql/data -p 5432:5432 postgres:16

Run Postgres with an env var for the password and a named volume pgdata so your database survives even after the container is removed.

docker ps — list containers

docker ps
docker ps -a

docker ps shows only running containers; docker ps -a shows all containers including stopped/exited ones — essential for debugging why something died.

docker images — list local images

docker images

See every image stored locally, with its repository, tag, image ID, and size.

docker pull — download an image

docker pull node:20-alpine

Grab a specific image from the registry ahead of time, so the first docker run doesn't have to wait on the download.

docker exec — run a command in a running container

docker exec -it web bash

Jump inside a running container to inspect files, check config, or debug live. -it gives you an interactive shell. (If the image lacks bash, use sh.)

docker logs — view container output

docker logs -f web

Stream the logs (stdout/stderr) of the web container live with -f (follow) — your go-to for watching what an app is doing in real time.

docker stop / start — control lifecycle

docker stop web
docker start web

Gracefully stop a running container (it gets a chance to shut down cleanly) and later start it back up without recreating it.

docker rm / rmi — remove containers and images

docker rm web
docker rmi nginx

docker rm deletes a stopped container; docker rmi deletes an image you no longer need to reclaim disk space.

docker build — build an image from a Dockerfile

docker build -t myapp:1.0 .

Build an image named myapp tagged 1.0 from the Dockerfile in the current directory (the . is the build context).

docker tag — give an image another name/tag

docker tag myapp:1.0 pushkar/myapp:1.0

Re-tag your local image with your Docker Hub username (and/or registry) so it's ready to push to the right repository.

docker push — upload an image to a registry

docker login
docker push pushkar/myapp:1.0

After docker login, push your tagged image to Docker Hub so teammates or your servers can pull and run it.

Quick Reference

Command	What it does	Most-used form
run	Create + start a container	`docker run -d -p 8080:80 --name web nginx`
ps	List containers	`docker ps -a`
images	List local images	`docker images`
pull	Download an image	`docker pull node:20-alpine`
exec	Run a command in a live container	`docker exec -it web bash`
logs	View container output	`docker logs -f web`
stop / start	Control a container's lifecycle	`docker stop web`
rm / rmi	Remove container / image	`docker rm web` · `docker rmi nginx`
build	Build image from Dockerfile	`docker build -t myapp:1.0 .`
tag	Re-name/tag an image	`docker tag myapp:1.0 pushkar/myapp:1.0`
push	Upload image to registry	`docker push pushkar/myapp:1.0`

Tip: docker run is really three steps in one: pull (if the image isn't local) + create (build the container from the image) + start (launch its main process). docker create + docker start do separately what run does together.

Caution: docker rm -f kills and removes a running container with no graceful shutdown — fine for test junk, dangerous for anything with unsaved state. docker system prune -a deletes all stopped containers, unused networks, and all images not currently in use. Read what it's about to delete before you confirm.

Tip: A classic gotcha: "What's the difference between docker stop and docker kill?" — stop sends SIGTERM first (graceful, with a grace period) then SIGKILL; kill sends SIGKILL immediately. Another favourite: "Difference between EXPOSE in a Dockerfile and -p at runtime?" — EXPOSE only documents the port; -p actually publishes it to the host.

Dockerfile Deep-Dive

A Dockerfile is a recipe. Each line is one instruction that tells Docker how to assemble your image, layer by layer. Think of it like your build pipeline written as plain text: install the runtime, copy your code, install deps, define how the app starts.

Note: A Dockerfile is a recipe card and the image is the prepped meal kit. The container is when you actually cook and eat it. You can build the kit once and cook it on a thousand machines, and it tastes identical every time.

FROM — the base layer

Every Dockerfile starts with FROM. It sets the base image you build on top of. Pin a specific version tag; never rely on latest in production because it silently changes under you.

FROM node:20-alpine

node:20 is the version and -alpine is the variant (a tiny Linux distro, ~5 MB base).

RUN — execute commands at build time

RUN runs a command while building the image and bakes the result into a new layer. Use it to install packages, create directories, or run build steps. Each RUN creates a layer, so chain related commands with && to keep layer count down.

RUN apk add --no-cache python3 make g++ \
 && npm install -g some-cli

COPY — bring files into the image

COPY copies files/folders from your build context (your project directory) into the image. This is how your source code gets in.

COPY package.json package-lock.json ./
COPY . .

ADD — COPY's older, fancier cousin

ADD does everything COPY does, plus two extras: it can fetch a remote URL, and it auto-extracts local tar archives. Those extras make it less predictable, so the rule of thumb is: use COPY unless you specifically need ADD's tar extraction.

Behaviour	COPY	ADD
Copy local files/dirs	Yes	Yes
Auto-extract local `.tar`	No	Yes
Download from URL	No	Yes (discouraged)
Predictable / explicit	Yes	No (magic behaviour)
Recommended default	Use this	Only for tar extraction

WORKDIR — set the working directory

WORKDIR sets the current directory for every following instruction (RUN, CMD, COPY, etc.) and creates it if it doesn't exist. Always use it instead of RUN cd /app — cd doesn't persist across layers.

WORKDIR /app

CMD vs ENTRYPOINT — how the container starts

Interview scenario: "What's the difference between CMD and ENTRYPOINT?" This is a guaranteed question.

Both define what runs when the container starts, but they play different roles:

ENTRYPOINT is the fixed command — the thing this container is. It needs --entrypoint to override at runtime.
CMD provides default arguments (or a default command) that are easily overridden at docker run time. When both are present, CMD's values are appended as arguments to ENTRYPOINT.

Exec form vs shell form:

Exec form (JSON array): CMD ["node", "server.js"] — runs the binary directly, no shell. This is the correct form: signals like SIGTERM reach your process so it can shut down cleanly.
Shell form: CMD node server.js — wraps the command in /bin/sh -c. The shell becomes PID 1 and may swallow signals, so your app doesn't get the stop signal. Avoid for the main process.

# They combine: final command = ENTRYPOINT + CMD
ENTRYPOINT ["node"]
CMD ["server.js"]
# docker run myimg            -> node server.js
# docker run myimg other.js   -> node other.js   (CMD overridden)

Aspect	CMD	ENTRYPOINT
Purpose	Default command / args	Fixed executable
Overridable at `docker run`	Yes — just pass new args	No (needs `--entrypoint` flag)
When both exist	Becomes arguments to ENTRYPOINT	Stays as the executable
Typical use	App that users may re-run with different args	Wrapper / tool where the binary is fixed
Can appear multiple times	Only last one wins	Only last one wins

ENTRYPOINT locks in the executable; CMD provides the default arguments that a caller can replace without touching the entrypoint.

EXPOSE — document the port

EXPOSE documents which port the container listens on. It does not publish the port — it is metadata for humans and tools. You still need docker run -p 3000:3000 to actually map it to the host.

EXPOSE 3000

ENV vs ARG — variables

ENV sets environment variables that persist inside the running container (your app can read them). ARG sets build-time variables that exist only during the build and are gone at runtime.

ARG NODE_VERSION=20
ENV NODE_ENV=production
# Pass ARG at build: docker build --build-arg NODE_VERSION=20 .

Aspect	ARG	ENV
Available during build	Yes	Yes
Available in running container	No	Yes
Set from CLI	`--build-arg`	`-e` at run / Dockerfile
Good for	Versions, build flags	App config, NODE_ENV, ports
Visible in image history	Yes (don't put secrets)	Yes (don't put secrets)

Layer caching — why instruction order matters

Every instruction creates a cached layer. On rebuild, Docker reuses a layer if nothing it depends on changed. The moment one layer is invalidated, every layer after it is rebuilt too. The classic Node mistake is copying everything first and then installing dependencies — every source change forces npm install to re-run.

# GOOD: deps cached separately from source
COPY package.json package-lock.json ./
RUN npm ci
COPY . .          # changes often, but npm ci layer stays cached

Put rarely-changing things (dependencies) early, frequently-changing things (source code) late.

A realistic Node app Dockerfile

FROM node:20-alpine

# 1. Set the working directory
WORKDIR /app

# 2. Copy only manifests first (cache-friendly)
COPY package.json package-lock.json ./

# 3. Install exact, reproducible deps
RUN npm ci

# 4. Now copy the rest of the source
COPY . .

# 5. Build the app (e.g. React / Vite / Next build)
RUN npm run build

# 6. Runtime config
ENV NODE_ENV=production
EXPOSE 3000

# 7. Fixed binary + default arg
ENTRYPOINT ["node"]
CMD ["dist/server.js"]

Tip: Answer CMD vs ENTRYPOINT crisply: ENTRYPOINT is the fixed executable; CMD supplies default arguments the user can override at docker run. If both are set, CMD is appended to ENTRYPOINT. Mention that exec form (["node","app.js"]) is preferred so SIGTERM reaches your process for graceful shutdown. That last detail signals real experience.

Caution: Cache invalidation cascades. If you COPY . . before RUN npm ci, then changing any source file busts the cache for the install layer and everything after — your dependencies reinstall on every build. Order from least-changing to most-changing.

Tip: Use npm ci instead of npm install in Dockerfiles. It installs exactly what's in package-lock.json, fails if the lockfile is out of sync, and is faster and fully reproducible — exactly what you want for deterministic image builds.

Multi-Stage Builds and Image Optimization

A naive image bundles your entire build toolchain — compilers, dev dependencies, source code — into the thing you ship to production. That is bloated and insecure. Multi-stage builds let you build in one stage and copy only the finished artifact into a clean, tiny final stage.

Why image size matters in interviews:

Security: fewer packages = smaller attack surface. No compiler, no shell utilities, no dev deps for an attacker to exploit.
Speed: smaller images pull and start faster — faster deploys, faster autoscaling, faster CI.
Cost: less registry storage, less bandwidth, less disk on every node. At scale this is real money.

Note: Shipping a single-stage image is like mailing someone a cake along with the entire kitchen — oven, mixer, flour bags and all. Multi-stage builds bake the cake in the kitchen, then ship only the cake in a small box.

Multi-stage Dockerfile: React app served by nginx

Build the static bundle with Node, then throw away Node entirely and serve the files with a tiny nginx image.

# ---- Stage 1: build ----
FROM node:20-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build          # produces /app/dist (or /app/build)

# ---- Stage 2: serve ----
FROM nginx:1.27-alpine
COPY --from=build /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

The final image contains nginx and your static files only — no Node, no node_modules, no source. Typically tens of MB instead of hundreds.

Multi-stage Dockerfile: Go binary on distroless

Go compiles to a single static binary, so the final stage needs almost nothing — perfect for distroless.

# ---- Stage 1: build ----
FROM golang:1.22-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server .

# ---- Stage 2: minimal runtime ----
FROM gcr.io/distroless/static-debian12
COPY --from=build /app/server /server
EXPOSE 8080
ENTRYPOINT ["/server"]

The result is often under 10 MB: just your binary plus minimal certs. No shell, no package manager — nothing for an attacker to pivot into.

Optimization techniques

Small base images: prefer -alpine, -slim, or distroless over full ubuntu/debian.
Multi-stage builds: ship only the artifact, not the toolchain.
Combine RUN layers: chain with && and clean up in the same layer so the cleanup actually shrinks the image.
.dockerignore: keep node_modules, .git, logs out of the build context — faster builds and no accidental bloat.
Order instructions: least-changing first for cache hits.
Remove build deps: uninstall compilers in the same RUN, or just leave them in the discarded build stage.
--no-install-recommends: on Debian/Ubuntu, skip optional suggested packages.

# Debian/Ubuntu: clean up in the SAME layer
RUN apt-get update \
 && apt-get install -y --no-install-recommends curl ca-certificates \
 && rm -rf /var/lib/apt/lists/*

A typical .dockerignore:

node_modules
.git
dist
build
*.log
.env
Dockerfile
.dockerignore

Base image sizes

Base image	Approx size	Has shell / pkg manager?	Best for
`ubuntu`	~75 MB	Yes	Dev convenience, lots of tooling
`debian:slim`	~30 MB	Yes	Balanced general-purpose runtime
`alpine`	~5 MB	Yes (busybox + apk)	Small images, most languages
`distroless`	~2–20 MB	No	Most secure production runtime

Note: Alpine is small but can cause issues with native modules due to musl libc. In those cases a -slim variant or a proper multi-stage build is a better fit.

Before / after: the real win

$ docker images
REPOSITORY      TAG          SIZE
myapp           single       1.1GB     # node + source + node_modules
myapp           multistage   42MB      # nginx + static build only

Tip: Distroless images contain your app and its runtime — and literally nothing else. No shell, no apt, no ls. That kills a whole class of attacks. The trade-off: you can't docker exec in to poke around, so debug locally or use the :debug variant. For compiled languages (Go, Rust) and even Java/Python, distroless is the gold-standard production base.

Tip: "How would you reduce a Docker image's size?" Hit these in order: multi-stage build (biggest win), smaller base image (alpine/slim/distroless), .dockerignore, combine and clean up RUN layers in the same instruction, and remove dev/build dependencies from the final stage. Bonus: mention the security angle — smaller image = smaller attack surface.

Note: A team shipped a React app as a single-stage Node image at 1.1 GB. Pulls on every deploy took minutes and autoscaling lagged. Switching to a multi-stage build (Node to build, nginx-alpine to serve) dropped it to ~42 MB — a 25x reduction. Deploys went from minutes to seconds.

Data: Volumes vs Bind Mounts

Containers are ephemeral. The writable layer lives and dies with the container — docker rm the container and that data is gone forever. That is fine for a stateless frontend, but a disaster for a database. To persist data, or to share files between your host and a container, you mount storage from outside the container's lifecycle.

Note: A container's own filesystem is like a hotel room — when you check out, housekeeping wipes everything. A volume is your bank locker: it stays safe no matter how many times you check in and out. A bind mount is like leaving a door open straight into your house — convenient, but the container can now mess with your real stuff.

Visual model:

Named volume — container writes to /var/lib/... → Docker-managed storage at /var/lib/docker/volumes/pgdata. Portable, survives container removal, best for DB data.
Bind mount — container writes to /app → a specific host folder you pick (e.g. /home/user/project). Best for live dev where you want host edits visible instantly.

The three mount types

Named volume: Docker-managed storage living under /var/lib/docker/volumes/. Docker owns it; you reference it by name. Best for persistent app data like databases.
Bind mount: maps a specific host path straight into the container. You control the exact location. Best for local development (live code).
tmpfs: stored in host RAM only, never written to disk. Vanishes on stop. Best for secrets or scratch data you never want persisted.

Aspect	Named volume	Bind mount	tmpfs
Location	`/var/lib/docker/volumes/`	Any host path you pick	Host RAM
Managed by	Docker	You	Docker (memory)
Use case	DB / persistent data	Live dev, config files	Secrets, scratch
Portability	High (Docker-native)	Low (host-path dependent)	N/A (ephemeral)
Performance on Mac/Win	Fast (in VM)	Slower (host↔VM sync)	Fast (RAM)
Survives container rm	Yes	Yes (it's on host)	No

Named volumes are Docker-managed and portable (best for production data); bind mounts expose a specific host directory (best for local development).

Note: On Mac/Windows, Docker runs inside a Linux VM, so bind mounts involve syncing between host and VM on every file change — hence the slowdown. Named volumes live entirely inside the VM and are therefore fast.

Managing named volumes

docker volume create pgdata          # create
docker volume ls                     # list
docker volume inspect pgdata         # see mountpoint, driver
docker volume rm pgdata              # delete
docker volume prune                  # remove all unused volumes

Two mount syntaxes: -v vs --mount

The short -v form is compact; the verbose --mount form is explicit and self-documenting (preferred in scripts).

# -v form: source:target[:options]
docker run -v pgdata:/var/lib/postgresql/data postgres:16

# --mount form: explicit key=value
docker run --mount type=volume,source=pgdata,target=/var/lib/postgresql/data postgres:16

Bind mount for live development (hot reload)

Mount your source code into the container so changes on your host instantly reflect inside — the dev server hot-reloads as you edit.

docker run -it --rm \
  -p 5173:5173 \
  -v "$(pwd)":/app \
  -v /app/node_modules \
  -w /app \
  node:20-alpine npm run dev

The second -v /app/node_modules is an anonymous volume that shields the container's installed node_modules from being clobbered by the host bind mount — a very common gotcha.

Named volume for Postgres data

docker run -d \
  --name pg \
  -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data \
  postgres:16

docker rm -f pg destroys the container but pgdata survives. Start a new Postgres container against the same volume and all your tables are still there.

When to use which

Named volume → production data that must persist: databases, uploads, caches.
Bind mount → local dev where you want host edits live in the container, or injecting a config file.
tmpfs → sensitive or throwaway data that should never touch disk.

Caution: A bind mount overlays the target directory — it hides whatever the image had there. If you bind-mount your source over /app and the image baked node_modules into /app/node_modules, your mount hides them and the app breaks (hence the anonymous-volume trick above). Also watch file permissions: a process running as a non-root UID inside the container may not be able to write to a host directory owned by a different user.

Tip: "Where does a named volume's data actually live?" Answer: on the Docker host under /var/lib/docker/volumes/<name>/_data, managed by Docker — not inside the container. On Mac/Windows it's inside the Docker Desktop Linux VM, which is why you can't cd to it from your Mac terminal. Knowing this distinction (host vs container vs VM) shows you actually understand the storage model.

Docker Networking

Every container needs to talk to something — the host, the internet, or another container. Docker handles this through network drivers. When you install Docker on Ubuntu 24.04, three networks already exist out of the box: bridge, host, and none.

Note: Think of a Docker network like the wiring in an apartment building. bridge is the building's internal intercom — flats can call each other through a switchboard, and the building has one main door (the host) to the outside world. host means the flat has no walls at all and shares the building's street address directly. none is a sealed room — no phone, no door, total isolation.

Container-to-container on a user-defined bridge: On a user-defined network, Docker's embedded DNS lets containers reach each other by name. Only the API container needs a published port (-p 3000:3000) so the browser can reach it. The db and cache containers are never published to the host — secure by default.

The three default networks

Driver	Isolation	Use Case	Performance	Port Mapping
bridge (default)	Containers get private IPs on a virtual subnet; isolated from host network namespace.	The normal default for single-host apps.	Slight overhead from NAT + virtual bridge, negligible for most workloads.	Required. You must `-p` to reach a container from outside.
host	None. Container shares the host's network namespace directly.	Max network performance, or when you need the container on the host's real IP.	Fastest — no NAT, no bridge, no virtual layer.	Ignored. `-p` does nothing; the container binds host ports directly.
none	Total. Only a loopback interface; no external connectivity.	Security-sensitive batch jobs, or you wire up custom networking yourself.	N/A — no traffic flows.	N/A — nothing to publish.

Port mapping deep dive

A container on a bridge network lives on a private subnet (commonly 172.17.0.0/16). The flag -p hostPort:containerPort tells Docker to take traffic arriving on that host port and forward it (via iptables DNAT rules) into the container's port. Read it as outside:inside — the left number is what you type in the browser; the right number is what the app listens on inside the container.

# Map host port 8080 -> container port 80
docker run -d -p 8080:80 --name web nginx

# Now the host can reach the container:
curl http://localhost:8080

# Map only on loopback (NOT exposed to the LAN):
docker run -d -p 127.0.0.1:8080:80 --name web-local nginx

# Let Docker pick a random free host port:
docker run -d -p 80 --name web-rand nginx
docker port web-rand   # shows which host port was assigned

Container-to-container communication — the key part

Interview scenario: "Why can't my two containers talk by name?"

On the default bridge network, containers can reach each other by IP address, but there is no automatic DNS — you cannot ping or curl another container by its name. On a user-defined bridge network, Docker runs an embedded DNS server and containers resolve each other by container name automatically.

This is the root cause of 90% of "my app can't connect to my db" bugs.

Tip: Almost never use the default bridge for multi-container apps. Always docker network create a user-defined bridge (or let Docker Compose make one for you — it does this automatically). You get free DNS by container name, better isolation, and you can attach/detach containers on the fly.

# 1. Create a user-defined bridge network
docker network create appnet

# List networks (you'll see appnet alongside bridge/host/none)
docker network ls

# Inspect it: subnet, gateway, connected containers
docker network inspect appnet

# 2. Run a db container ON that network
docker run -d --name db --network appnet \
  -e POSTGRES_PASSWORD=secret postgres:16

# 3. Run a client container on the SAME network
docker run -it --rm --name client --network appnet alpine sh

# Inside the client shell — resolve 'db' by NAME, no IP needed:
#   apk add --no-cache curl
#   ping db
#   nc -zv db 5432          # TCP connect to postgres by name

# Attach an already-running container to a network later:
docker network connect appnet web

Tip: "bridge vs host?" — bridge isolates the container on its own subnet and needs -p port publishing + NAT to be reached; host removes that isolation, the container uses the host's network stack directly (no -p, fastest, but port conflicts and less isolation). "Why can't my two containers talk by name on the default bridge?" — because the default bridge has no embedded DNS; name resolution only works on user-defined networks. Fix: create a user-defined bridge and run both containers on it (or use Compose).

Caution: When you write -p 8080:80, Docker binds 0.0.0.0:8080 by default — meaning the port is exposed on every network interface, including your public/LAN IP, not just localhost. On a cloud VM with a public IP, that database or admin panel is now reachable from the internet. Bind to 127.0.0.1 (e.g. -p 127.0.0.1:8080:80) for local-only access. Docker's iptables rules can also bypass a UFW firewall — don't assume UFW alone protects published ports.

Docker Compose

Running one container with a long docker run command is fine. But a real app is a web frontend + an API + a database + a cache, each needing its own ports, volumes, env vars, and a shared network. Typing four docker run commands in the right order, every time, is painful and error-prone. Docker Compose solves this: you declare your entire multi-container stack in one YAML file and bring it all up with a single command. Use this with the Docker run → Compose converter to translate existing docker run commands into Compose services quickly.

Note: If docker run is cooking one dish, Compose is the full recipe card for a thali — it lists every dish (service), the ingredients (image, env, volumes), and the order to serve them (depends_on). One command and the whole meal arrives, plated the same way every single time.

Compose stack layout: A single docker compose up creates the network, the volume, and all three services. The api service builds from a local Dockerfile; db uses a named volume for persistence; all three are wired by service name via Compose's built-in DNS.

"docker compose" vs "docker-compose"

Old Compose was a separate Python tool invoked as docker-compose (with a hyphen). Modern Compose is a v2 plugin built into the Docker CLI, invoked as docker compose (a space, no hyphen). On Ubuntu 24.04 with current Docker, use docker compose. The hyphenated command is legacy and deprecated. Also note: the top-level version: key is now obsolete and can be omitted.

Anatomy of a Compose file

Key	What it does
`image`	Pull a prebuilt image (e.g. `postgres:16`).
`build`	Build from a local Dockerfile instead of pulling.
`ports`	Publish ports to the host — `"host:container"`.
`volumes`	Mount named volumes or host paths for persistent data.
`environment`	Set env vars inside the container.
`depends_on`	Control start order (and optionally wait for health).
`networks`	Attach the service to one or more networks.
`healthcheck`	Define how Docker tests if the service is actually ready.

A full, realistic compose.yaml

A Node API built from a local Dockerfile, a Postgres database with a named volume, and a Redis cache — all on a shared network. The API waits for Postgres to be healthy, not just started.

services:
  api:
    build: ./api                 # builds from ./api/Dockerfile
    ports:
      - "3000:3000"
    environment:
      DB_HOST: db                # resolves by service name (Compose DNS)
      DB_PORT: 5432
      DB_USER: appuser
      DB_PASSWORD: ${DB_PASSWORD}    # pulled from .env file
      REDIS_URL: redis://cache:6379
    depends_on:
      db:
        condition: service_healthy   # wait until db healthcheck passes
      cache:
        condition: service_started
    networks:
      - appnet
    restart: unless-stopped

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: appdb
    volumes:
      - pgdata:/var/lib/postgresql/data   # named volume = data survives restarts
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U appuser -d appdb"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s
    networks:
      - appnet
    restart: unless-stopped

  cache:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5
    networks:
      - appnet
    restart: unless-stopped

volumes:
  pgdata:                        # Docker-managed named volume

networks:
  appnet:                        # user-defined bridge -> DNS by service name

Because every service is on appnet, the API reaches the database at hostname db and Redis at cache — the service names are the hostnames, thanks to Compose's user-defined network (exactly the DNS behaviour covered in the networking section).

Key commands

docker compose up -d          # build (if needed) + start all, detached
docker compose ps             # list services in this project + their state
docker compose logs -f        # tail combined logs from all services
docker compose logs -f api    # tail just one service
docker compose build          # (re)build images that use 'build:'
docker compose exec api sh    # open a shell inside the running api container
docker compose down           # stop + remove containers and the network
docker compose down -v        # ALSO delete named volumes (wipes db data!)

depends_on vs healthcheck — the trap

Interview scenario: "Does depends_on wait for a service to be ready?"

A plain depends_on: [db] only controls start order. Compose will start db before api — but "started" just means the container process launched, not that Postgres is accepting connections. Databases take a few seconds to initialise, so your API can crash on boot trying to connect to a not-yet-ready DB.

For readiness, add a healthcheck to the dependency service and use condition: service_healthy in the dependent service — only then does Compose wait.

Tip: Keep secrets and environment-specific values out of the compose file. Put them in a .env file (auto-loaded by Compose for ${VAR} substitution) and add .env to .gitignore. For per-environment tweaks, use a compose.override.yaml (auto-merged in dev) or docker compose -f compose.yaml -f compose.prod.yaml up to layer a production config on top. Use the .env Example Checker to validate that your .env.example stays in sync with your actual .env keys.

Note: A new dev joining a team, instead of working through a half-day setup doc, can clone the repo, copy .env.example to .env, and run docker compose up -d. Sixty seconds later the API, database, and cache are running and wired together identically to every other machine — including CI.

Tip: Expect: "What does Compose actually do for you?" — declarative multi-container orchestration on a single host: networks, volumes, env, and start order in one file. Be ready for "Does depends_on wait for the service to be ready?" — No, only start order; for readiness add a healthcheck + condition: service_healthy. Note the scope: Compose is single-host; for multi-host/cluster orchestration you move to Kubernetes — see Kubernetes for DevOps for a full introduction. See also the Kubernetes Resource Calculator when you are ready to plan resource requests for a containerised workload running in a cluster. Mentioning this boundary shows you understand where Compose stops.

Caution: docker compose down -v deletes named volumes — that pgdata volume holding your database goes with it. Great for a clean dev reset, catastrophic on anything you care about. Plain docker compose down (no -v) keeps volumes. Build the muscle memory now, before you run it on the wrong project.

Best Practices and Security

A container that works and a container that is production-ready are different things. This section is the checklist that separates "it runs on my machine" from "it's safe to ship." Most of it shows up in DevOps interviews too — security questions are nearly guaranteed.

Run as non-root

By default, processes in a container run as root (UID 0). If an attacker breaks out of the app, they are root inside the container — and a container root maps to host root in several breakout scenarios. The fix is the USER instruction: create an unprivileged user and switch to it before the app runs. This is the single highest-impact security change you can make.

FROM node:20-slim

WORKDIR /app

# Install deps as root (needs write to /app)
COPY package*.json ./
RUN npm ci --omit=dev

COPY . .

# Create a non-root user and hand over ownership
RUN useradd --system --uid 10001 --no-create-home appuser \
 && chown -R appuser:appuser /app

# Everything from here runs as appuser, not root
USER appuser

EXPOSE 3000
CMD ["node", "server.js"]

.dockerignore

Just like .gitignore, a .dockerignore file tells the build to skip files when sending the build context to the daemon. This makes builds faster, images smaller, and — critically — stops secrets and junk from being copied in by a broad COPY . ..

# .dockerignore
node_modules
npm-debug.log
.git
.gitignore
.env
.env.*
*.md
Dockerfile
.dockerignore
dist
coverage
.vscode
.DS_Store

Excluding node_modules forces a clean npm ci inside the image (no host-OS native modules leaking in), and excluding .env / .git prevents secrets and history from ever entering the image.

Don't bake secrets into images

Interview scenario: "How do you handle secrets in Docker?"

Anything you put in an ENV instruction or pass as a build ARG is baked into an image layer — and image layers are readable by anyone with the image. docker history dumps them right out.

# An attacker (or teammate) with your image can read baked-in values:
docker history --no-trunc your-image:tag

# WRONG — secret is now permanently in a layer:
#   ENV API_KEY=sk_live_abcd1234
#   ARG DB_PASSWORD   (visible in history even if "removed" later)

Instead, inject secrets at runtime, never at build time:

# Runtime env (not in the image, but visible via 'docker inspect'):
docker run -e API_KEY="$API_KEY" myapp

# Better: file-based secrets (Compose/Swarm) mounted at /run/secrets/...
# Best: an external secrets manager (Vault, AWS Secrets Manager,
#       SSM Parameter Store) fetched by the app at startup.

Image scanning

Your base image and dependencies carry known vulnerabilities (CVEs). Scan before you ship, and re-scan regularly — new CVEs are disclosed against old images all the time.

# Docker Scout (built into modern Docker Desktop / CLI)
docker scout cves your-image:tag
docker scout quickview your-image:tag

# Trivy (popular open-source scanner) on Ubuntu 24.04
trivy image your-image:tag
trivy image --severity HIGH,CRITICAL your-image:tag

Hardening checklist

Do	Don't
Pin base images by version and digest: `node:20-slim@sha256:…`	Use `:latest` — it silently changes under you.
Use minimal images (`-slim`, `alpine`, `distroless`) — smaller attack surface.	Ship a full `ubuntu` base with compilers and shells you don't need.
Run as a non-root `USER`.	Run the app as root.
Drop Linux capabilities: `--cap-drop ALL`, add back only what's needed.	Run with default (or `--privileged`) capabilities.
Run read-only where possible: `--read-only` + `--tmpfs /tmp`.	Leave the whole filesystem writable.
Add a `HEALTHCHECK` so orchestrators know real readiness.	Assume "process running" means "service healthy."
Inject secrets at runtime via env files / secrets managers.	Bake secrets into `ENV`/`ARG` layers.
Use multi-stage builds to keep build tools out of the final image.	Ship the entire build toolchain in production.

# Least-privilege runtime: drop all caps, read-only root fs, writable /tmp
docker run -d \
  --cap-drop ALL \
  --read-only \
  --tmpfs /tmp \
  --name myapp myapp:1.0

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

Caution: Secrets in image layers are permanent and readable by anyone with the image. Deleting the ENV line in a later layer does NOT remove it — the earlier layer still contains it, and docker history --no-trunc reveals it. If you ever accidentally bake a real credential into an image and push it, treat that credential as compromised: rotate it immediately. You can't "edit it out" of a published image.

Tip: Container security questions cluster around a few themes: (1) run as non-root (USER); (2) minimal/pinned base images to shrink attack surface; (3) never bake secrets into layers (explain docker history); (4) scan images for CVEs (Scout/Trivy) in CI; (5) least privilege at runtime — drop capabilities, read-only fs, no --privileged. Bonus: the container shares the host kernel, so a kernel exploit can escape — keep the host patched and consider extra isolation (gVisor, rootless Docker).

Tip: Bake scanning into CI as a gate: fail the pipeline on HIGH/CRITICAL CVEs (trivy image --exit-code 1 --severity HIGH,CRITICAL). Combined with pinned base digests, this means an image can't silently drift into a vulnerable state between builds.

Note: A fintech team shipped a Node image with ENV STRIPE_KEY=sk_live_… baked in "temporarily." The image was pushed to a registry a contractor could pull. Months later, during an audit, docker history surfaced the live key sitting in a layer. They had to rotate the Stripe key, invalidate the image across every environment, and add a CI secret-scanner — a week of cleanup that one runtime -e flag (or a secrets manager) would have prevented entirely.

Troubleshooting Playbook

Think of this like a diagnostic chart: a container presents a symptom, you run a diagnose command to confirm the cause, then you apply the fix. You see a blank screen (symptom), open DevTools console (diagnose), spot the error (cause), patch the code (fix). Containers are no different — the tools just have different names.

90% of Docker problems are solved by three commands run in order: see what state the container is in (docker ps -a), read what it logged before dying (docker logs), then either inspect its config or jump inside it (docker inspect / docker exec -it). Master this loop and you will look like a wizard in interviews.

Symptom → Diagnose → Fix chart

Symptom	Diagnose (command)	Likely Cause	Fix
Container won't start / stuck in `Created` or `Restarting`	`docker ps -a` then `docker logs <name>`	Bad entrypoint, missing config/env, app crashes on boot, or a restart loop	Read logs, fix the underlying error, rebuild and re-run
Container exits immediately (`Exited (0)`)	`docker ps -a` + `docker logs <name>`	No long-running foreground process; `CMD` finished and nothing kept PID 1 alive	Run a process that stays in the foreground (e.g. `node server.js`); don't background your main process
Exits with non-zero code (`Exited (1)`, `Exited (127)`)	`docker logs <name>`	App threw on startup; `127` usually means "command not found"	Fix the app error; for `127`, verify the binary exists and the `CMD` path is correct
Image is huge (1GB+ for a small app)	`docker images` and `docker history <image>`	Full base image, build tools shipped, no `.dockerignore`, dev deps baked in	Multi-stage build, slim/alpine base, add `.dockerignore`, combine `RUN` layers
`port is already allocated`	`docker ps` then `sudo lsof -i :3000` (or `ss -ltnp \	grep 3000`)	Another container or host process is bound to that port
`permission denied` on a bind-mounted folder	`docker exec -it <name> ls -ln /path` and `id` inside	UID/GID inside container ≠ host file owner; or SELinux/AppArmor	`--user $(id -u):$(id -g)`, `chown` the host dir, or add `:z`/`:Z` on SELinux
Container A can't reach B by name	`docker network inspect bridge`; `docker exec -it A ping B`	Both on the default `bridge`, which has no DNS by name	Create a user-defined bridge and attach both: `docker network create appnet` + `--network appnet`
`no space left on device`	`docker system df` and `df -h`	Dangling images, stopped containers, unused volumes, build cache eating disk	`docker system prune -af` (add `--volumes` only if sure)
Cannot pull (`denied`, `unauthorized`, `toomanyrequests`)	Read the exact error from `docker pull <image>`	Not logged in, wrong tag, or Docker Hub anonymous rate limit	`docker login`; fix the tag; for rate limits, authenticate or wait
Code changes not reflecting after rebuild	`docker build` output (watch for `CACHED`)	Cached layer reused; or running an old image/container	Rebuild with `--no-cache`, re-tag and re-run; in dev, bind-mount source
Env var not set inside the container	`docker exec -it <name> env` or `docker inspect`	`-e`/`--env-file` not passed, or it's build-time `ARG` not runtime `ENV`	Pass `-e KEY=value` or `--env-file .env`; don't confuse `ARG` with `ENV`

The general debugging workflow

# 1. What is the state? (running, exited, restarting — and the exit code)
docker ps -a

# 2. What did it say before it died? (your single most useful command)
docker logs <container_name_or_id>
docker logs -f <container>          # follow live, like tail -f
docker logs --tail 50 <container>   # last 50 lines only

# 3. How is it configured? (env, mounts, network, IP, command)
docker inspect <container>
docker inspect -f '{{.State.ExitCode}}' <container>   # just the exit code

# 4. Get a shell inside a RUNNING container and poke around
docker exec -it <container> sh     # or 'bash' if the image has it

Four-step crash-debug loop: check state, read logs, reproduce interactively, then fix and rebuild.

Tip: A classic question is "a container starts and immediately stops, how do you debug it?" The winning answer names the loop: docker ps -a to confirm it exited and see the code, docker logs to read the error, then explain the most common cause — there was no long-running foreground process keeping PID 1 alive, or the app crashed on startup. If you can't exec into it (because it already exited), mention you can override the entrypoint to debug: docker run -it --entrypoint sh <image>.

Deep dive: container exits immediately

This is the #1 beginner trap and the #1 interview question. A container lives only as long as its main process (PID 1) runs. The moment that process exits, the container exits. It is like running npm run build and expecting a dev server to stay up — build finishes, process ends, done. You needed npm start (a long-running server) instead.

# Confirm it exited and check the code
docker ps -a
# "Exited (0)" = process finished cleanly, nothing to keep it alive
# "Exited (1)" or higher = app crashed; read the logs

docker logs <container>

# Can't exec because it already died? Override the entrypoint to get a shell:
docker run -it --entrypoint sh myimage
# Now run your CMD by hand and watch it fail in real time

# WRONG — the & backgrounds it, CMD returns instantly, container dies
CMD node server.js &

# RIGHT — runs in the foreground, holds the container open
CMD ["node", "server.js"]

Deep dive: port is already allocated

# Find who is holding host port 3000
docker ps                         # is another container mapping :3000?
sudo lsof -i :3000                # any host process on it?
ss -ltnp | grep :3000             # alternative on Ubuntu 24.04

# Fix A: stop the offender
docker stop <other_container>

# Fix B: just map to a free host port (left = host, right = container)
docker run -d -p 3001:3000 myimage
# App still listens on 3000 inside; you reach it at http://localhost:3001

Deep dive: image too big

# See sizes and which layers are fat
docker images
docker history <image>     # shows size added per layer

# Multi-stage: build stage has all the tooling, final stage is slim
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-slim AS runtime
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY package*.json ./
RUN npm ci --omit=dev
CMD ["node", "dist/server.js"]

Deep dive: no space left on device

# See the breakdown: images, containers, volumes, build cache
docker system df
docker system df -v        # verbose, per-item

# Reclaim everything unused (stopped containers, dangling/unused images, networks, build cache)
docker system prune -af    # -a = also unused images, -f = no prompt

# Target just one type if you prefer
docker image prune -af
docker container prune -f
docker builder prune -af   # build cache only

# Include volumes too — DANGER, can delete data (see warning)
docker system prune -af --volumes

Caution: docker system prune --volumes and docker volume prune delete anonymous and unused volumes — and that means real, persistent data (databases, uploads) can be wiped permanently. There is no undo. Never run prune --volumes on a production host without first confirming which volumes are in use (docker volume ls). When in doubt, prune images and build cache only.

Tip: Run docker system df before you blindly prune — it tells you whether the space is in images, volumes, or build cache, so you can reclaim surgically. Also, docker logs only works if the app writes to stdout/stderr; apps that log to a file inside the container will show nothing, so always configure your app to log to the console in containers.

Note: Prefer a day-by-day path? This is covered in Mission 90 Days 22–30 — a free 90-day guided DevOps program with browser terminal missions.

Note: A team's CI runner kept failing builds with no space left on device every Friday. docker system df revealed 40 GB of build cache. They added docker builder prune -af --filter "until=168h" (clear cache older than 7 days) to a nightly cron job. Builds stopped failing, and they never touched volumes, so no data was at risk. Diagnose with system df first, then prune the specific thing that grew.

Docker for DevOps: A Deep Dive into Images, Layers, and Compose

opscanopy — Sun, 16 Aug 2026 15:30:40 +0000

Docker is the tool that finally made "it works on my machine" a non-excuse. By packaging an application together with its exact dependencies into a portable image, Docker lets the same artifact run identically on a laptop, a CI runner, and a production server. This guide is a deep, practical tour for DevOps engineers: how images and layers actually work, how to write efficient and secure Dockerfiles, how containers talk to each other over networks, how to persist data with volumes, how to compose multi-service stacks, and how to take all of it to production. Work through it top to bottom, or jump to the part you need. For a structured learning path beyond this guide, see the Docker roadmap.

The Problem Docker Solves

The "Works on My Machine" Story

It's 2019. A developer at a startup has been grinding for 3 weeks on a Node.js app — a real-time dashboard for tracking delivery agents. The UI is slick, the APIs are fast, the PM is happy. He types npm start and everything just works. Perfectly.

He pushes the code to staging. The CI pipeline goes green. He Slacks his team: "Deployed! Please test."

3 minutes later:

Error: Cannot find module 'optional-chaining-polyfill'
Error: node: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.28' not found
npm ERR! peer dep missing: react@^17.0.0, required by react-query@3.39.0

His laptop: Node 14.18, npm 6.14, Ubuntu 22.04. The staging server: Node 12.22, npm 5.6, CentOS 7. Three days of debugging. Three days of "but it works on my machine!"

This problem was not unique — every developer at every company was facing it. Docker was the solution.

Dependency Hell — Real-World Examples

The "Works on My Machine" problem is a symptom of a deeper disease: dependency hell. Here are scenarios every developer has lived through:

Scenario	The Conflict	Pain Level
Two Python projects on same machine	Project A needs Python 2.7, Project B needs Python 3.9. `pip install` breaks both.	High
npm version conflict	App needs npm 6 for legacy scripts, another needs npm 8. Switching breaks lockfiles.	Medium
OpenSSL version mismatch	Dev has OpenSSL 3.0, prod server has 1.1. Node.js crypto breaks silently.	High
System library versions (glibc)	Compiled on Ubuntu 22, deployed on CentOS 7. Binary won't even load.	Critical
Ruby gem native extensions	Gem compiled against libxml2 on Mac, server has different libxml2. Segfault.	Critical

Note: Think of Docker like sharing a React component with node_modules — not just the code, but the exact node version, npm version, and OS libraries. Docker packages your application together with its entire environment.

Deployment Nightmares Before Docker

Before Docker, deploying an app to a new server looked like this:

# The legendary "deployment runbook" (50 pages of this)
ssh root@production-server-ip

# Step 1: Install the right Node version (pray it's compatible)
curl -fsSL https://deb.nodesource.com/setup_14.x | bash -
apt-get install -y nodejs

# Step 2: Hope npm is the right version
npm install -g npm@6.14.15

# Step 3: Clone app
git clone https://github.com/company/app.git /var/www/app
cd /var/www/app
npm install   # Often fails here due to native modules

# Step 4: Set up environment variables (done manually, often wrong)
export DB_HOST=192.168.1.100
export NODE_ENV=production
# (Someone always forgets one. App crashes at 2 AM)

# Step 5: Set up PM2 or systemd (manual, error-prone)
npm install -g pm2
pm2 start app.js --name "myapp"
pm2 startup  # Different command on every distro

# Step 6: Set up nginx (manual config, copy-paste errors)
# ... 20 more steps ...

# Total time: 4-8 hours per server
# Success rate: ~60% first try
# Consistency across servers: ZERO

And if you need to deploy to 10 servers? Do all of this 10 times — and something different will happen each time. This is called configuration drift — over time servers diverge from each other, and no one knows why production behaves differently.

The VM Tax — Resource Waste at Scale

Companies tried solving this with Virtual Machines — one VM per application. But VMs have a massive overhead:

Resource	Each VM Consumes	10 Apps = 10 VMs
RAM	2–4 GB minimum (OS alone)	20–40 GB wasted on OS
Disk	20–40 GB per VM image	200–400 GB
Boot time	1–5 minutes	Can't scale fast
CPU	Always-on OS processes	Significant idle overhead

A startup with 20 microservices using VMs wastes a minimum of 40 GB RAM just for operating systems. The same 20 services in Docker containers consume 2–5 GB of overhead — a dramatic difference that translates directly to cloud cost savings.

Docker's Shipping Container Analogy

Before the 1950s, international shipping was chaos. Every ship had a different design, every port used different equipment, and cargo had to be repacked repeatedly. A shipment from Europe to America could take months, and costs were astronomical.

In 1956, Malcolm McLean invented the standardized shipping container. One fixed size (20ft/40ft), standard corners, works on any ship, any port, any truck, any train. Load once in Shanghai — unload in Rotterdam without opening the box.

Result: Global trade exploded. Shipping costs dropped 90%.

Docker is the shipping container of software:

Shipping World	Software World
Different ships	Different servers/OS
Different ports	Different cloud providers
Cargo re-packing	Re-configuring for each environment
Standard container	Docker container
Load once, ship anywhere	Build once, run anywhere
McLean standardized trade	Docker standardized deployment

Tip: When asked "Why do you use Docker?" in an interview — use the shipping container analogy. Then give concrete numbers: "Our deployment time went from 4 hours to 15 minutes. Configuration drift eliminated. Dev-prod parity achieved."

Why Every Company Adopted Docker

By 2026, over 80% of cloud workloads run in containers. Here's why adoption was inevitable:

Dev-prod parity — Same container in dev, test, staging, production. No more surprises.
Faster onboarding — New developer joins? One command: docker compose up. Ready in 5 minutes, not 2 days.
Microservices enablement — Each service in its own container. Scale independently.
CI/CD becomes simple — Build a container, test it, push to registry, deploy. Repeatable every time.
Cost reduction — Pack more apps per server. AWS/GCP bills drop 40–60%.
Kubernetes compatibility — Kubernetes orchestrates containers. You cannot use Kubernetes without containers.

Containers vs Virtual Machines

Virtual Machines — The Apartment Building Analogy

Think of a VM like separate apartments in a large building. Each apartment has its own kitchen, its own bathroom, its own living room. What happens in one apartment is completely invisible to the other.

A Virtual Machine runs a complete operating system on top of a hypervisor — software that emulates physical hardware. Each VM thinks it owns the whole computer.

Hypervisor (VMware, VirtualBox, KVM, Hyper-V) sits between hardware and VMs
Each VM gets a full Guest OS (Windows, Ubuntu, etc.) — 1–4 GB just for the OS
Hardware is emulated — virtual CPU, virtual RAM, virtual disk
Strong isolation — a crash in VM1 doesn't affect VM2
Boot time: 1–5 minutes (full OS boot)

Containers — The Shared House Analogy

Think of containers like rooms in a shared house. You share the same kitchen and water supply, but each person has their own private room and lives independently.

A Container shares the host OS kernel but runs in an isolated process namespace. No hardware emulation, no duplicate OS.

Shared kernel — uses the host's Linux kernel directly
Each container gets an isolated view of processes, network, filesystem
No hypervisor overhead — near-native performance
Lightweight — MBs instead of GBs
Boot time: Milliseconds to seconds

Containers skip the Guest OS layer that VMs require, cutting overhead from gigabytes per app to near zero.

Detailed Comparison Table

Feature	Virtual Machine	Container
Startup Time	1–5 minutes (full OS boot)	Seconds (process start)
Size	2–40 GB per VM image	10 MB–1 GB (app + libs only)
OS	Full Guest OS per VM	Shares Host OS Kernel
Isolation Level	Hardware-level (very strong)	Process-level (strong, not absolute)
Performance Overhead	5–15% CPU/RAM overhead	<1–2% overhead (near-native)
Portability	Limited (hypervisor-specific)	Excellent (any Linux/Mac/Win with Docker)
Resource Usage	Heavy (duplicate OS per VM)	Light (shared kernel)
Density (per server)	5–20 VMs typical	50–500 containers typical
Security Isolation	Very strong (hardware boundary)	Good (namespaces + cgroups)
Best Use Case	Legacy apps, strong isolation, Windows workloads	Microservices, CI/CD, cloud-native apps

Tip: Real-world production numbers: Server utilization jumped from 30–40% (with VMs) to 80%+ (with containers). Deployment time dropped from 45 minutes to 8 minutes. Auto-scaling: containers spin up in 3 seconds vs 4 minutes for VMs — critical during peak load events.

When to Use VM vs Container

Use VM When...	Use Container When...
Running Windows workloads on Linux host	Running microservices/APIs
Need kernel-level isolation (security-critical)	CI/CD pipelines (build, test, deploy)
Legacy apps that need specific OS versions	Stateless applications
Running untrusted code (multi-tenant SaaS)	Development environments
Database servers needing dedicated resources	Auto-scaling workloads

Note: In enterprise environments, both are used together. VMs run Docker. In the cloud, an EC2 VM runs Docker containers on top. This is not a binary choice — they are complementary technologies.

What is Docker?

The Origin Story — From DotCloud to Changing the World

2008: Solomon Hykes co-founds DotCloud — a Platform-as-a-Service (PaaS) company in San Francisco. They're building tools to deploy customer apps on their platform, and internally they develop a container management tool.

March 21, 2013: PyCon, Santa Clara. Hykes gives a 5-minute lightning talk called "The Future of Linux Containers." He demos Docker — their internal tool they're open-sourcing. The audience is small. The impact: massive.

A 5-minute talk at a Python conference launched a technology that today runs in every data center in the world. That's what happens when the right solution meets the right problem at the right time.

2013: Docker open-sourced, goes viral on GitHub
2014: Google, Microsoft, Amazon begin supporting Docker
2015: Docker Inc. raises $95M, valuation $1B+
2016: Docker Swarm, Docker Compose GA. Enterprise adoption explodes.
2017: Kubernetes wins the orchestration wars, but Docker remains the build standard
2019: Docker Inc. sells its enterprise business to Mirantis, refocuses on developers
2023+: Docker Desktop 4.x, Docker Scout (security), AI-powered features
2026: Docker is as fundamental as Git — you cannot be a backend/DevOps engineer without it

Docker vs Other Container Runtimes

Docker is not just one tool — it is an ecosystem. But the container runtime market has other players:

Runtime	Used By	Key Feature	Should You Learn?
Docker	Developers, small teams, CI/CD	Best Developer Experience (DX), most popular, huge ecosystem	Yes, first priority
containerd	Kubernetes production clusters	Lightweight production runtime, CNCF graduated project	Learn after Docker
Podman	Red Hat / Fedora users, security-conscious teams	Rootless (no daemon), daemonless, Docker-compatible CLI	Be aware of it
CRI-O	OpenShift, Kubernetes	Minimal footprint, built for Kubernetes CRI interface	Be aware of it
runc	Low-level, all runtimes use it	OCI reference implementation, the actual container runner	Deep-dive later

Tip: When asked "What is the difference between Docker and containerd?" — say this: Docker is the full toolchain (build + run + push + compose). containerd is just the runtime that Kubernetes uses to actually run containers. Docker internally uses containerd. Think of it as: Docker = Full Car, containerd = The Engine.

Docker's Place in the DevOps Stack

Docker is the central connective tissue of the modern DevOps pipeline:

Code (React/Node/Python) → Build (docker build / Dockerfile) → Test (docker run test suite) → Registry (docker push / ECR / Docker Hub) → Deploy (ECS / Kubernetes / docker compose up)

Career Impact

Docker knowledge significantly expands your career options:

Skill Level	Typical Role	US (USD)	India (INR)
Without Docker	Junior/Mid Developer (Frontend/Backend)	$80k–120k	₹15–20 LPA
Docker + Kubernetes	DevOps / SRE (2–5 yrs exp)	$120k–170k	₹30–50 LPA
Docker + AWS + K8s + CI/CD	Senior DevOps / Staff SRE (5+ yrs exp)	$170k+	₹50+ LPA

Ranges are rough 2026 signals and vary widely by region, company size, and cost of living.

A React/frontend background is a unique advantage in DevOps — engineers who understand both frontend containerization and backend infrastructure are in high demand.

Docker By The Numbers — 2026

13M+ images on Docker Hub
13B+ pulls per month
20M+ Docker Desktop users
80% of cloud workloads containerized

Containerization Concepts — The Linux Magic

The Foundation — Why Linux?

Docker's magic comes from three specific Linux kernel features. Understanding them is what separates a senior engineer from a beginner — and this knowledge is invaluable in interviews when asked "How does Docker work internally?"

Docker containers are not magic — they're three Linux kernel features working together:

Namespaces — What a container can see (isolation of view)
Cgroups — What a container can use (resource limits)
Union Filesystem (OverlayFS) — How a container's filesystem is built

Linux Namespaces — The Isolation Magic

A namespace makes a container feel like it is the only process on the machine — like an actor on a film set who is made to feel they are actually in a royal palace, when they are actually standing on a soundstage.

Linux has 7 types of namespaces. Docker uses 6 of them:

PID Namespace — Process ID Isolation

The container believes it has its own process tree. Container's PID 1 = Host's PID 5678.

# Inside container: sees its own process tree
$ ps aux
PID   USER  CMD
1     root  node server.js   ← thinks it's PID 1!
12    root  /bin/sh

# On host: sees the real PID
$ ps aux | grep node
5678  root  node server.js   ← actually PID 5678

The container cannot see parent processes. Completely isolated process tree.

NET Namespace — Network Isolation

Each container gets its own network interfaces, IP address, routing table, and firewall rules.

# Container 1: has its own IP
$ docker exec container1 ip addr
172.17.0.2/16   ← container's IP

# Container 2: different IP
$ docker exec container2 ip addr
172.17.0.3/16   ← different IP, same host

# Host: sees docker0 bridge
$ ip addr show docker0
172.17.0.1/16   ← bridge connects them

MNT Namespace — Filesystem Isolation

The container has its own filesystem view. It appears to have a complete Linux filesystem — /etc, /usr, /var all separate.

# Container sees its own filesystem
$ docker exec mycontainer ls /
bin  dev  etc  home  lib  proc
root  sys  tmp  usr  var  app

# Host files are NOT visible in container
# (unless explicitly mounted with -v)

UTS Namespace — Hostname Isolation

Each container can have its own hostname, independent of the host machine.

# Set custom hostname for container
$ docker run --hostname=myapp-prod ubuntu hostname
myapp-prod

# Host machine sees its own hostname
$ hostname
prod-server-01

USER Namespace — User ID Mapping

Maps the container's root user to an unprivileged host user. Container "root" is NOT host root — important for security!

# Container root → host user 1000
# Even if container is "root",
# on host it's unprivileged UID 1000
# Security: container escape doesn't
# give attacker host root access

IPC Namespace — Inter-Process Communication Isolation

Containers cannot access each other's shared memory, semaphores, or message queues.

# Share IPC namespace between containers
# (for high-performance apps like Redis)
docker run --ipc=container:redis-container my-app

# Default: isolated IPC per container
# No shared memory leaks between apps

Cgroups — Resource Control

Control Groups (cgroups) control how much resource a container can use. If namespaces define what a container can see, cgroups define how much it can consume.

Think of cgroups as a resource budget per container:

# Memory limit: container cannot use more than 512MB
docker run --memory=512m nginx

# CPU limit: container gets max 50% of one CPU core
docker run --cpus=0.5 nginx

# Both together (production-ready)
docker run \
  --memory=512m \
  --memory-swap=512m \
  --cpus=0.5 \
  --cpu-shares=512 \
  nginx

# I/O limits (read/write speed)
docker run \
  --device-read-bps /dev/sda:100mb \
  --device-write-bps /dev/sda:50mb \
  nginx

# See current container resource usage
docker stats --no-stream

# Output:
# CONTAINER   CPU %   MEM USAGE / LIMIT    NET I/O
# nginx       0.1%    12MiB / 512MiB       1.2kB / 0B

Cgroup Subsystem	What It Controls	Docker Flag
`memory`	Max RAM usage, OOM killer behavior	`--memory=512m`
`cpu`	CPU time allocation	`--cpus=0.5`
`cpuset`	Which CPU cores to use	`--cpuset-cpus=0,1`
`blkio`	Block I/O limits (disk read/write)	`--device-read-bps`
`net_cls`	Network bandwidth tagging	(via tc/iptables)
`pids`	Max number of processes in container	`--pids-limit=100`

Caution: Always set memory and CPU limits on every production container. Without limits, a runaway container can crash the entire host server — the "noisy neighbour" problem. During peak traffic, if one container hogs CPU, all other containers will slow down. Limits protect all containers on the host.

Union Filesystem — OverlayFS

A Docker image is made of read-only layers. When you run a container, Docker adds a thin writable layer on top. This is OverlayFS (Overlay Filesystem).

# Example Dockerfile - creates layers
FROM ubuntu:22.04            # Layer 1: Base Ubuntu (~70MB)
RUN apt-get update           # Layer 2: Package index (~20MB)
RUN apt-get install -y curl  # Layer 3: curl binary (~5MB)
COPY app.js /app/            # Layer 4: Your app (~1MB)
CMD ["node", "app.js"]       # Layer 5: Metadata only

# See the layers of an image
$ docker history my-node-app
IMAGE         CREATED      SIZE     COMMENT
a1b2c3d4      2 hours ago  1.2MB    COPY app.js
f5e6d7c8      2 hours ago  5.1MB    apt-get install curl
g9h0i1j2      2 hours ago  22MB     apt-get update
ubuntu:22.04  2 weeks ago  72MB     base layer

Copy-on-Write (CoW) — The Secret Sauce

When a container modifies a file from a read-only layer:

Docker copies that file to the writable layer
Makes the modification in the writable layer
The original read-only layer is untouched
100 containers sharing the same image = base layers shared, only diffs stored

If 100 containers are using the same nginx image, there is still only one copy on disk. 100x efficiency!

# See OverlayFS in action on Linux host
$ docker inspect mycontainer | grep -A 10 GraphDriver
"GraphDriver": {
  "Data": {
    "LowerDir": "/var/lib/docker/overlay2/abc123/diff:
                 /var/lib/docker/overlay2/def456/diff",
    "MergedDir": "/var/lib/docker/overlay2/xyz789/merged",
    "UpperDir":  "/var/lib/docker/overlay2/xyz789/diff",
    "WorkDir":   "/var/lib/docker/overlay2/xyz789/work"
  }
}
# LowerDir = read-only image layers (stacked)
# UpperDir = writable container layer (your changes)
# MergedDir = what container sees (unified view)

Note (Senior Engineer Interview): "Both Docker containers and VMs provide isolation — so why are containers considered less secure?" Answer: VMs provide hardware-level isolation (hypervisor = hard boundary). Containers only use Linux kernel features — if there is a kernel vulnerability (like Dirty COW, runc CVE), container escape is possible. This is why multi-tenant environments running untrusted code prefer VMs or gVisor/Kata Containers.

Tip (Quick Summary): Docker container = Linux process + Namespaces (isolation of view: can't see others' processes, network, files) + Cgroups (resource limits: max CPU/RAM) + OverlayFS (layered filesystem: shared base + writable top). Everything else — Dockerfile, docker-compose, Kubernetes — is built on top of these three.

Docker Architecture Deep Dive

The Big Picture

Docker follows a client-server architecture. When you type docker run nginx, a lot is happening under the hood — think of it as an assembly line where each component performs a specific job.

The Docker CLI is a thin REST client; all heavy lifting happens inside the Docker daemon, which coordinates containerd and runc to run containers and communicates with registries for image storage.

1. Docker Client (CLI)

The Docker Client is the tool you use — the docker command. When you type docker run nginx, the client only translates CLI arguments into a REST API call and sends it to the Docker Daemon. The client itself does nothing else — it is just a translator.

# Docker CLI is just making REST API calls to the daemon
# You can curl the Docker socket directly

# List running containers (same as docker ps)
curl --unix-socket /var/run/docker.sock http://localhost/v1.43/containers/json

# Start a container via API
curl --unix-socket /var/run/docker.sock \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"Image": "nginx"}' \
  http://localhost/v1.43/containers/create

# Check Docker version via API
curl --unix-socket /var/run/docker.sock http://localhost/v1.43/version | jq

# See all API endpoints
curl --unix-socket /var/run/docker.sock http://localhost/v1.43/_ping

Tip: You can control a remote machine's Docker: export DOCKER_HOST=tcp://192.168.1.100:2376, then run normal docker commands. In production, always use TLS (port 2376), never plain port 2375!

2. Docker Daemon (dockerd)

The Docker Daemon (dockerd) is the background process that actually does the work. It accepts REST API requests and manages containers, images, networks, and volumes.

Feature	Details
Default Socket	`unix:///var/run/docker.sock`
TCP (TLS)	Port 2376 (production)
TCP (No TLS)	Port 2375 (NEVER in production!)
Config File	`/etc/docker/daemon.json`
Logs	`journalctl -u docker.service`
Restart	`sudo systemctl restart docker`

3. Docker Registry

Registry is where images are stored. Docker Hub is the default public registry, but enterprises use private registries:

Docker Hub — Default public registry. Free tier available. docker.io/library/nginx
AWS ECR — Amazon Elastic Container Registry. IAM integration.
GCR / GAR — Google Container/Artifact Registry. Seamless with GKE.
GHCR — GitHub Container Registry. Tight CI/CD integration.
Harbor — Open-source enterprise registry. Self-hosted. Vulnerability scanning built-in.
Artifactory — JFrog's universal artifact manager. Docker + Maven + npm in one place.

4. containerd — High-Level Runtime

containerd is a CNCF project that sits between the Docker Daemon and actual containers. It manages image pulling, storage, and container lifecycle. Kubernetes also uses containerd directly (without Docker).

# containerd has its own CLI: ctr
# Note: operates in a different namespace from Docker

# List images in containerd
sudo ctr images ls

# List containers
sudo ctr containers ls

# Pull image directly via containerd
sudo ctr images pull docker.io/library/nginx:latest

# Run container via containerd (low-level)
sudo ctr run docker.io/library/nginx:latest mynginx

# containerd socket
ls -la /run/containerd/containerd.sock

# Check containerd status
sudo systemctl status containerd

Note: Kubernetes v1.24+ removed Docker support. Kubernetes now talks directly to containerd via CRI (Container Runtime Interface). Docker's "removal" was only the removal of the Docker daemon — containers still run the same way because containerd and runc are the same underneath.

5. runc — Low-Level Runtime

runc is the reference implementation of the OCI Runtime Specification. It directly makes Linux kernel syscalls — creates namespaces, sets up cgroups, and starts the process. It is the actual tool that makes a "container" exist.

# runc version check
runc --version

# runc is not normally used directly
# But for understanding: containerd calls runc like this:
# runc create --bundle /path/to/bundle container_id
# runc start container_id
# runc delete container_id

Complete Flow: docker run nginx

You type docker run nginx
Docker CLI → REST API call → Docker daemon (via /var/run/docker.sock)
dockerd checks: is the image available locally? No → pull from Registry
dockerd → tells containerd to start the container
containerd → spawns a containerd-shim process
containerd-shim → calls runc
runc → makes Linux kernel syscalls (namespaces + cgroups setup)
Container process (nginx) starts
runc exits; containerd-shim stays alive (for I/O)
Container keeps running even on daemon restart

Docker Installation on Ubuntu 24.04 (Complete Guide)

Prerequisites Check

Caution: Check if Docker is already installed before starting. If an old version exists, remove it first: sudo apt remove docker docker-engine docker.io containerd runc

# System info check
uname -a
lsb_release -a
whoami

# Check if Docker already installed
docker --version 2>/dev/null || echo "Docker not installed"

# Remove old versions if they exist
sudo apt remove -y docker docker-engine docker.io containerd runc 2>/dev/null
sudo apt autoremove -y

Step 1: System Update

sudo apt update && sudo apt upgrade -y

# If the kernel was updated, reboot (optional but recommended)
# sudo reboot

Step 2: Install Prerequisites

sudo apt install -y \
  ca-certificates \
  curl \
  gnupg \
  lsb-release

# Verify install
curl --version
gpg --version

Step 3: Add Docker's Official GPG Key

# Create keyrings directory
sudo install -m 0755 -d /etc/apt/keyrings

# Download Docker's GPG key and convert to binary format
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Set read permissions
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Verify key
gpg --show-keys /etc/apt/keyrings/docker.gpg

Note: The GPG key ensures packages come from Docker's official server and have not been intercepted. This protects against man-in-the-middle attacks. Always verify packages before installing.

Step 4: Add Docker Repository

echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Verify repository was added
cat /etc/apt/sources.list.d/docker.list

# Expected output:
# deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg]
#   https://download.docker.com/linux/ubuntu noble stable

# Update apt cache with new repository
sudo apt update

Step 5: Install Docker Engine

sudo apt install -y \
  docker-ce \
  docker-ce-cli \
  containerd.io \
  docker-buildx-plugin \
  docker-compose-plugin

# Package breakdown:
# docker-ce              → Docker Community Edition (daemon)
# docker-ce-cli          → Docker CLI (the 'docker' command)
# containerd.io          → containerd runtime
# docker-buildx-plugin   → BuildKit for advanced image building
# docker-compose-plugin  → Docker Compose v2 (docker compose, not docker-compose)

# To install a specific version:
# List available versions
apt-cache madison docker-ce

# Install specific version
# sudo apt install docker-ce=5:26.0.0-1~ubuntu.24.04~noble \
#   docker-ce-cli=5:26.0.0-1~ubuntu.24.04~noble \
#   containerd.io docker-buildx-plugin docker-compose-plugin

Step 6: Verify Installation

sudo docker run hello-world

# Expected output:
# Hello from Docker!
# This message shows that your installation appears to be working correctly.

# Service status check
sudo systemctl status docker
sudo systemctl status containerd

Step 7: Post-Installation — Run Without sudo

# Create 'docker' group (usually already exists)
sudo groupadd docker 2>/dev/null || echo "Group already exists"

# Add current user to docker group
sudo usermod -aG docker $USER

# Apply group changes (no logout/login needed for this session)
newgrp docker

# Verify — run docker without sudo
docker run hello-world

# Permanent verification
groups $USER
# Output should include 'docker'

Caution: Adding a user to the docker group is practically equivalent to giving root access. Through the Docker socket (/var/run/docker.sock), anyone can mount a volume and access the host filesystem. Add users carefully on production servers. Rootless Docker is a more secure option.

Step 8: Enable Docker on Boot

sudo systemctl enable docker.service
sudo systemctl enable containerd.service

# Manual start/stop/restart
sudo systemctl start docker
sudo systemctl stop docker
sudo systemctl restart docker

# View logs
sudo journalctl -u docker.service -f      # Live logs
sudo journalctl -u docker.service --since "1 hour ago"

Post-Installation Verification

# Version info
docker --version
# Docker version 26.1.4, build 5650f9b

docker compose version
# Docker Compose version v2.27.1

# System info (detailed)
docker info

# Test container
docker run hello-world

# Interactive ubuntu container
docker run -it ubuntu bash

# Nginx web server test
docker run -d -p 8080:80 nginx
curl http://localhost:8080

# Cleanup
docker stop $(docker ps -q)
docker system prune -f

Common Installation Errors & Fixes

Error	Cause	Fix
`permission denied while trying to connect`	User not in docker group	`sudo usermod -aG docker $USER && newgrp docker`
`Cannot connect to the Docker daemon`	dockerd not running	`sudo systemctl start docker`
`dial unix /var/run/docker.sock: connect: no such file`	Docker not installed or not started	Install Docker OR `sudo systemctl start docker`
`Failed to connect to bus: No such file`	systemd not available (WSL/container)	Logout and login again, or `sudo service docker start`
`E: Package 'docker-ce' has no installation candidate`	Repository not added	Redo steps 3-4
`toomanyrequests` from Docker Hub	Rate limit hit (unauthenticated)	Run `docker login` or use a mirror

Docker Rootless Mode

# Install prerequisites
sudo apt install -y uidmap

# Install rootless Docker (as regular user, NOT sudo)
dockerd-rootless-setuptool.sh install

# Set environment variables
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock
export PATH=$HOME/bin:$PATH

# Add to ~/.bashrc for persistence
echo 'export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock' >> ~/.bashrc
echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc

# Enable service (user-level systemd)
systemctl --user enable docker
systemctl --user start docker

# Verify
docker info | grep -i rootless
# Should show: rootless: true

Note: In security-critical environments, use rootless Docker or Podman. In rootless mode, the impact of container breakout attacks is drastically reduced because the daemon is not root. In Kubernetes environments, configure securityContext with non-root users.

Installation on RHEL / CentOS / Rocky Linux

# Step 1: Install yum-utils
sudo yum install -y yum-utils

# Step 2: Add Docker repository
sudo yum-config-manager \
  --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo

# Step 3: Install Docker
sudo yum install -y \
  docker-ce \
  docker-ce-cli \
  containerd.io \
  docker-buildx-plugin \
  docker-compose-plugin

# Step 4: Start and enable
sudo systemctl start docker
sudo systemctl enable docker

# Step 5: Post-install
sudo usermod -aG docker $USER
newgrp docker

# Verify
docker run hello-world

# Note: On RHEL 9+, podman is preferred (Docker-compatible CLI)
# podman run hello-world  # Same commands, daemonless!

Docker Engine Components — Under the Hood

Component Stack

The Docker Engine follows this layered stack from top to bottom:

docker CLI — docker run / build / push / pull (makes REST API calls)
dockerd — Docker Daemon — REST API | /var/run/docker.sock | Port 2376 (TLS)
containerd — Image management | Container lifecycle | /run/containerd/containerd.sock
containerd-shim — Daemonless container I/O | stdout/stderr relay
runc (OCI Runtime) — syscalls to kernel: clone(), unshare(), setns()
Linux Kernel — namespaces (pid, net, mnt, uts, ipc) | cgroups (cpu, memory, io)
Container Process — Your app running as PID 1 inside container

dockerd — Docker Daemon Deep Dive

# View dockerd process
ps aux | grep dockerd
# root  1234  ... /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

# Socket files
ls -la /var/run/docker.sock             # Unix socket (default)
ls -la /run/containerd/containerd.sock  # containerd socket

# dockerd config file
cat /etc/docker/daemon.json

# Restart dockerd (containers will NOT stop!)
sudo systemctl restart docker

# View dockerd logs
sudo journalctl -u docker.service -n 50 --no-pager
sudo journalctl -u docker.service -f  # Live tail

# Docker events stream
docker events  # Real-time events: container start/stop/die, image pull, etc.

daemon.json — Production Configuration

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  },
  "default-address-pools": [
    {"base": "172.17.0.0/12", "size": 24}
  ],
  "storage-driver": "overlay2",
  "registry-mirrors": [
    "https://mirror.gcr.io"
  ],
  "insecure-registries": [],
  "live-restore": true,
  "max-concurrent-downloads": 10,
  "max-concurrent-uploads": 5,
  "metrics-addr": "0.0.0.0:9323",
  "experimental": false
}

Note (Production Must-Haves): log-driver + log-opts: By default Docker logs can consume unlimited disk. Set max-size: 100m and max-file: 3 — total 300MB max per container. live-restore: true: Containers will not stop when the Docker daemon restarts. Critical for zero-downtime daemon updates! storage-driver: overlay2: Best performance on modern kernels. Also the default on Ubuntu.

containerd — Detailed

# Inspect containerd directly
sudo ctr version
sudo ctr namespaces ls         # Docker uses 'moby' namespace
sudo ctr -n moby images ls     # Docker images in containerd
sudo ctr -n moby containers ls # Running containers

# containerd config
cat /etc/containerd/config.toml

# Restart containerd (Docker containers will temporarily stop!)
sudo systemctl restart containerd

# In Kubernetes clusters (containerd directly)
# crictl images         # CRI-compatible CLI
# crictl ps             # Running containers
# crictl inspect <id>   # Container details

Note (containerd-shim): containerd-shim is an intermediate process spawned for each container. Its job: (1) relay stdin/stdout/stderr, (2) report exit status, (3) keep the container alive even if the daemon crashes. This is why live-restore: true works!

OCI — Open Container Initiative

Specification	Defines	Implemented By
Image Spec	Image format: layers, manifest, config JSON	Docker, containerd, Buildah
Runtime Spec	How to run a container: config.json + rootfs	runc, crun, kata-containers
Distribution Spec	How images are pushed/pulled from registries	Docker Hub, ECR, GCR, Harbor

Tip (Interview Question): Q: "What is the difference between Docker and containerd?" A: Docker is a complete platform (CLI + daemon + build tools + compose). containerd is just a container runtime — lightweight, CRI-compatible, perfect for Kubernetes. Docker internally uses containerd. Kubernetes bypasses the Docker daemon and talks directly to containerd (since v1.24).

Understanding Docker Images

Image — The Blueprint

An image is an immutable blueprint. Think of it this way: a class Component extends React.Component — you don't directly "run" a class, you create an instance. Similarly, Image = Class, Container = Instance. An image is built once, then thousands of containers can be created from it. The image itself never changes.

Technically, a Docker image is a read-only filesystem snapshot + metadata (env vars, exposed ports, entry command), organized in layers — each layer representing an incremental change.

Image Layers — The Onion Model

┌─────────────────────────────────────────┐
│  Container Layer — WRITABLE             │  ← New files, modifications go here.
│  New files, modifications go here.      │    Deleted when container stops.
│  Deleted when container stops.          │
├─────────────────────────────────────────┤
│           ↑ WRITABLE / READ-ONLY ↓      │
├─────────────────────────────────────────┤
│  Layer 4: COPY . /app                   │  ← Your application source code (~5 MB)
├─────────────────────────────────────────┤
│  Layer 3: RUN npm install               │  ← node_modules directory (~80 MB)
├─────────────────────────────────────────┤
│  Layer 2: WORKDIR /app                  │  ← Creates /app directory (metadata only)
├─────────────────────────────────────────┤
│  Layer 1: FROM node:18-alpine           │  ← Base OS + Node.js runtime (~120 MB,
│                                         │    shared with other node images)
└─────────────────────────────────────────┘
  OverlayFS merges all layers → appears as single filesystem to container

Note: OverlayFS is a Linux kernel filesystem driver. It merges multiple read-only layers and one read-write layer into a single unified filesystem. The container sees a normal filesystem, but it is actually a stack of layers. Copy-on-Write is used — when a container modifies a file, it is first copied from the read-only layer to the read-write layer (upperdir), then modified.

Image Manifest and Digest

# View image manifest (JSON format)
docker manifest inspect nginx:latest

# Image digest — immutable SHA256 reference
docker images --digests nginx
# REPOSITORY   TAG       DIGEST                                                    IMAGE ID
# nginx        latest    sha256:a484819eb60...                                     e784f4560448

# Pull by digest (PRODUCTION BEST PRACTICE)
# Tags can change (latest is mutable), digests never change!
docker pull nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c

# Tags are mutable — this is dangerous:
docker pull nginx:latest   # Today pulls one version, tomorrow may pull another

# Safe production pull (pinned digest):
docker pull nginx@sha256:exact_digest_from_registry

# Get digest of a local image
docker inspect nginx --format='{{index .RepoDigests 0}}'

Caution: The :latest tag is a moving target. If you pull today you get 1.25; if CI/CD pulls tomorrow it might get 1.26 — with breaking changes. In production, always pin semantic versions: nginx:1.25.3-alpine or use digests. :latest is only acceptable for local development.

Image Storage on Disk

# Where Docker images are stored on disk
ls /var/lib/docker/

# Image layers (overlay2 storage driver)
sudo ls /var/lib/docker/overlay2/
# Each directory is a layer

# Image metadata
sudo ls /var/lib/docker/image/overlay2/imagedb/content/sha256/
# Each file is an image manifest/config

# Disk usage breakdown — important for DevOps!
docker system df
# TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
# Images          15        3         2.1GB     1.8GB (85%)
# Containers      5         2         45MB      23MB (51%)
# Local Volumes   8         3         560MB     300MB (53%)
# Build Cache     0         0         0B        0B

# Detailed breakdown
docker system df -v

# View layer sharing (images share base layers)
docker history nginx:latest --no-trunc
docker history nginx:alpine --no-trunc
# Common layers are shared — only one copy on disk!

Image Tags vs Digests vs IDs

Concept	Example	Mutable?	Use Case
Image ID	`e784f4560448`	No (content-addressed)	Local reference, scripting
Tag	`nginx:1.25.3`	Yes (can be moved)	Human-readable, dev/staging
Digest	`nginx@sha256:abc...`	No (cryptographic)	Production deployments, security
latest tag	`nginx:latest`	Yes (frequently moves)	Local dev ONLY, never production

Tip (Interview Question): Q: "What is the difference between a Docker image and a container?" A: An image is a read-only blueprint/template — a static filesystem snapshot. A container is a running instance of that image, with a writable layer added on top. You can create thousands of containers from one image. When a container is stopped, the writable layer is deleted (unless committed or a volume is used).

Working with Images — Command Deep Dive

docker pull — Downloading Images

Purpose: Download images from a registry to your local machine. Pull without docker run when you want the image without starting a container.

# Basic pull — pulls latest tag (avoid in production!)
docker pull nginx

# Specific tag pull (recommended)
docker pull nginx:1.25.3-alpine

# Specific digest pull (MOST RECOMMENDED for production)
docker pull nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c

# Pull from private registry
docker pull myregistry.company.com/myapp:v1.2.3

# Pull from AWS ECR (login first)
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin \
  123456789012.dkr.ecr.ap-south-1.amazonaws.com

docker pull 123456789012.dkr.ecr.ap-south-1.amazonaws.com/myapp:latest

# Pull from GCR
docker pull gcr.io/my-project/myapp:v2.0.0

# Pull from GitHub Container Registry
docker pull ghcr.io/myorg/myapp:main

# Specify platform (ARM/AMD64)
docker pull --platform linux/amd64 nginx:latest
docker pull --platform linux/arm64 nginx:latest  # For Apple Silicon

Caution: Always specify a tag when pulling. docker pull nginx = docker pull nginx:latest. If the latest tag is updated tomorrow, your CI/CD will pull a different image. Fix: docker pull nginx:1.25.3 or use a digest.

docker images — Listing Images

# Basic list
docker images
docker image ls  # Same command, new syntax

# Table format with size
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedAt}}"

# Filter dangling images (untagged, build artifacts)
docker images -f dangling=true

# Filter by label
docker images -f label=env=production

# Image IDs only (useful for scripting)
docker images -q

# Show images for a specific repository
docker images nginx

# Custom format
docker images --format "{{.ID}}: {{.Repository}}:{{.Tag}} — {{.Size}}"

# All images (including intermediates)
docker images -a

# Images with digests
docker images --digests

# Sort by size
docker images --format "{{.Size}}\t{{.Repository}}:{{.Tag}}" | sort -h

docker rmi — Deleting Images

# Remove by name
docker rmi nginx
docker image rm nginx

# Specify tag (when multiple tags exist)
docker rmi nginx:1.25.3-alpine

# Remove by image ID
docker rmi e784f4560448

# Remove multiple images at once
docker rmi nginx:latest ubuntu:20.04 redis:alpine

# Force remove — even if a container is using it (dangerous!)
docker rmi -f nginx:latest

# Remove dangling images (safe cleanup)
docker image prune

# Remove all unused images (aggressive cleanup)
docker image prune -a

# Remove without confirmation
docker image prune -f
docker image prune -a -f

# Nuclear option — delete all images (careful!)
docker rmi $(docker images -q)

# Delete by age filter
docker image prune -a --filter "until=24h"   # Older than 24 hours
docker image prune -a --filter "until=168h"  # Older than 7 days

Caution: An image used by a running container cannot be deleted — docker rmi will fail. Stop and remove the container first: docker stop <container> && docker rm <container>, then docker rmi <image>.

docker tag — Tagging Images

# Tag for Docker Hub
docker tag myapp:latest myusername/myapp:v1.0.0
docker tag myapp:latest myusername/myapp:latest

# Tag for AWS ECR
docker tag myapp:latest \
  123456789012.dkr.ecr.ap-south-1.amazonaws.com/myapp:v1.0.0

# Tag for GCR
docker tag myapp:latest gcr.io/my-gcp-project/myapp:v1.0.0

# Tag for GitHub Container Registry
docker tag myapp:latest ghcr.io/myorg/myapp:v1.0.0
docker tag myapp:latest ghcr.io/myorg/myapp:main-abc1234  # Git commit hash

# CI/CD common pattern (git hash + version)
GIT_HASH=$(git rev-parse --short HEAD)
VERSION="1.2.3"
docker tag myapp:latest myregistry.io/myapp:${VERSION}-${GIT_HASH}
docker tag myapp:latest myregistry.io/myapp:${VERSION}
docker tag myapp:latest myregistry.io/myapp:latest

Tip (Tagging Strategy for CI/CD): Good tagging strategy: (1) Git commit hash tag — always, immutable and traceable. (2) Semantic version tag. (3) Branch name tag (e.g., main, develop). (4) latest only on main/master branch. This strategy makes both rollback and debugging easy.

docker save & docker load — Transferring Images

# Save a single image
docker save -o myapp.tar myapp:latest

# Save multiple images into one tar
docker save -o all-images.tar nginx:latest redis:alpine postgres:15

# Compressed save (saves disk space)
docker save myapp:latest | gzip > myapp.tar.gz

# Load from tar file
docker load -i myapp.tar

# Load compressed
gunzip -c myapp.tar.gz | docker load

# Transfer via SSH (no file on disk!)
docker save myapp:latest | ssh user@remote-server docker load

# Verify loaded image
docker images | grep myapp

docker history — Viewing Image Layers

# View image history
docker history nginx

# Full commands (not truncated)
docker history --no-trunc nginx

# Custom format
docker history --format "table {{.ID}}\t{{.Size}}\t{{.CreatedBy}}" nginx

# Identify large layers
docker history nginx --format "{{.Size}}\t{{.CreatedBy}}" | \
  grep -v "^0B" | \
  sort -rh | head -5

docker inspect — Deep Inspection

# Full JSON inspection
docker inspect nginx
docker inspect nginx | jq  # Pretty printed (install jq: sudo apt install jq)

# Extract specific fields (Go template syntax)
docker inspect --format '{{.Config.Env}}' nginx
docker inspect --format '{{.Config.ExposedPorts}}' nginx
docker inspect --format '{{.Config.Cmd}}' nginx

# For containers: get IP address
docker inspect --format '{{.NetworkSettings.IPAddress}}' my_container

# Container mount points
docker inspect --format '{{json .Mounts}}' my_container | jq

# Container restart policy
docker inspect --format '{{.HostConfig.RestartPolicy}}' my_container

# Image layers (layer hashes)
docker inspect nginx | jq '.[0].RootFS.Layers'

# Container environment variables
docker inspect my_container | jq '.[0].Config.Env[]'

docker push — Uploading to Registry

# Login first
docker login                               # Docker Hub
docker login myregistry.company.com        # Private registry

# AWS ECR login
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin \
  123456789012.dkr.ecr.ap-south-1.amazonaws.com

# GitHub Container Registry login
echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_USERNAME --password-stdin

# Push
docker push myusername/myapp:v1.0.0
docker push myusername/myapp:latest

# Push multiple tags (loop)
for TAG in v1.0.0 v1.0 v1 latest; do
  docker push myregistry.io/myapp:${TAG}
done

# Logout (for security on shared machines)
docker logout
docker logout myregistry.company.com

Image Layers Deep Dive — Caching & Optimization

Place stable layers (base image, dependencies) below volatile ones (source code) so Docker's layer cache is reused on every incremental build.

OverlayFS — Technical Deep Dive

OverlayFS Component	Docker Equivalent	Description
`lowerdir`	Image layers (read-only)	Base image + all intermediate layers — never modified
`upperdir`	Container's writable layer	Container-specific changes go here
`workdir`	Internal OverlayFS use	Staging area for atomic operations
`merged`	Container's filesystem view	Unified mount — what the container sees

# View OverlayFS mount for a running container
docker run -d --name mynginx nginx

# Get mount info
docker inspect mynginx | jq '.[0].GraphDriver'
# {
#   "Data": {
#     "LowerDir": "/var/lib/docker/overlay2/abc.../diff:/var/lib/docker/overlay2/def.../diff",
#     "MergedDir": "/var/lib/docker/overlay2/xyz.../merged",
#     "UpperDir": "/var/lib/docker/overlay2/xyz.../diff",
#     "WorkDir": "/var/lib/docker/overlay2/xyz.../work"
#   },
#   "Name": "overlay2"
# }

# View actual filesystem layering on host
sudo ls /var/lib/docker/overlay2/

# View mount point on host
cat /proc/mounts | grep overlay

# Demonstrate Copy-on-Write
docker exec mynginx sh -c "echo 'hello' > /usr/share/nginx/html/test.html"
# The file went into the container's upperdir, NOT into the image.
# The image remains unchanged.

Layer Caching — The Secret of Build Speed

# First build — everything downloaded and executed (slow)
docker build -t myapp:v1 .
# Step 1/5 : FROM node:18-alpine      → DOWNLOADING (slow)
# Step 2/5 : WORKDIR /app             → Running
# Step 3/5 : COPY package*.json ./    → Running
# Step 4/5 : RUN npm install          → Installing (slow!)
# Step 5/5 : COPY . .                 → Running

# Second build (only source code changed)
docker build -t myapp:v2 .
# Step 1/5 : FROM node:18-alpine      → Using cache  ✓ (instant!)
# Step 2/5 : WORKDIR /app             → Using cache  ✓ (instant!)
# Step 3/5 : COPY package*.json ./    → Using cache  ✓ (package.json unchanged)
# Step 4/5 : RUN npm install          → Using cache  ✓ (instant!)
# Step 5/5 : COPY . .                 → Running      ← Only this re-runs!
# Total time: 2 seconds vs 3 minutes!

# Force bypass cache
docker build --no-cache -t myapp:fresh .

# Use cache from external source (CI/CD optimization)
docker build \
  --cache-from myregistry.io/myapp:latest \
  -t myapp:new .

Cache Invalidation — When Cache Breaks

Instruction	Cache Invalidates When	Note
`FROM`	Base image digest changes	Rarely — unless you explicitly pull new base
`RUN`	Command text changes	Even whitespace change = cache miss!
`COPY / ADD`	File content changes (checksum)	Even 1 byte change = cache miss for this layer + all subsequent
`ENV`	Value changes	All subsequent layers also invalidated
`ARG`	Build arg value changes	Be careful with VERSION args — they invalidate builds
`WORKDIR`	Path changes	Usually stable

Caution: A cache miss in one layer causes ALL subsequent layers to re-run. This is why layer ordering is so important — put frequently changing instructions last, stable instructions first.

Layer Ordering — Cache Optimization

# ❌ BAD — npm install runs EVERY TIME any file changes
FROM node:18-alpine
WORKDIR /app
COPY . .           # Source code AND package.json copied together
RUN npm install    # Re-runs even if only App.jsx changed!

# ✅ GOOD — npm install only when package.json changes
FROM node:18-alpine
WORKDIR /app

# Step 1: Copy ONLY package files first
COPY package.json package-lock.json ./

# Step 2: Install dependencies (CACHED as long as package.json is unchanged)
RUN npm install

# Step 3: Now copy source code (this layer changes with source changes)
COPY . .

# Build
RUN npm run build

# Serve
CMD ["node", "server.js"]

# ✅ BEST — Python/pip example follows the same principle
FROM python:3.11-slim
WORKDIR /app

# Dependencies first (stable layer)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Source code later (frequently changing)
COPY . .

CMD ["python", "app.py"]

# ✅ BEST — Java/Maven example
FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app

# Copy pom.xml first (dependencies are stable)
COPY pom.xml .
RUN mvn dependency:go-offline -q  # Download all dependencies (cached!)

# Source code later
COPY src ./src
RUN mvn package -q -DskipTests

FROM eclipse-temurin:17-jre-alpine
COPY --from=build /app/target/*.jar app.jar
CMD ["java", "-jar", "app.jar"]

Note (Golden Rule of Layer Ordering): Think of it this way: "What changes least goes first." Order of stability: Base image > System packages > Dependency files > Dependencies install > Config files > Source code > Build artifacts. Following this order will dramatically speed up CI/CD builds.

Layer Sharing — Memory & Disk Efficiency

When 50 Node.js microservices all use the same node:18-alpine base image:

Without sharing: 50 × 120MB = 6 GB
With Docker layer sharing: 120MB (base, stored once) + 50 × app_size

This means more containers fit per Kubernetes node, node startup is faster (layer already cached), and network bandwidth is saved during pulls.

Layer Analysis Tools

# Method 1: docker history (built-in)
docker history --no-trunc myapp:latest

# Method 2: docker inspect layers
docker inspect myapp:latest | jq '.[0].RootFS.Layers'

# Method 3: dive tool (BEST for optimization)
# Installation
wget https://github.com/wagoodman/dive/releases/download/v0.12.0/dive_0.12.0_linux_amd64.deb
sudo apt install ./dive_0.12.0_linux_amd64.deb

# Interactive layer analysis
dive myapp:latest
# Left panel: layer list + size
# Right panel: filesystem changes per layer
# Shows wasted space — e.g., files deleted in later layers
# Gives an efficiency score

# CI mode (automated check)
dive myapp:latest --ci   # Fails if efficiency is below threshold

# Method 4: BuildKit inline --progress=plain
DOCKER_BUILDKIT=1 docker build --progress=plain -t myapp:latest .
# Shows each step's timing and cache status

Advanced Layer Optimization Techniques

# Technique 1: Combine RUN commands (fewer layers)
# ❌ BAD — 3 layers for apt operations
RUN apt update
RUN apt install -y curl wget
RUN rm -rf /var/lib/apt/lists/*

# ✅ GOOD — Single layer, cleanup in SAME layer
RUN apt update && \
    apt install -y curl wget && \
    rm -rf /var/lib/apt/lists/*
# IMPORTANT: Cleanup must be in the SAME RUN instruction!
# Cleaning in a separate RUN does NOT reduce layer size
# (the layer with apt cache already exists)

# Technique 2: Multi-stage builds (drastically reduce SIZE)
# Build stage
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --production=false  # All deps including devDependencies
COPY . .
RUN npm run build               # Create production build

# Production stage (runtime only!)
FROM node:18-alpine             # Slim base (50MB vs 1GB!)
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
# Build tools, devDeps, source code do not go into the production image!
EXPOSE 3000
CMD ["node", "dist/server.js"]

# Result: 1.2GB → 180MB image!

# Technique 3: .dockerignore (like .gitignore for Docker)
# node_modules/      ← exclude from COPY
# .git/              ← unnecessary
# dist/              ← build output (will be rebuilt)
# *.log              ← log files
# .env               ← Secrets! NEVER in image
# coverage/          ← test coverage reports

# Technique 4: Specific COPY (avoid COPY . .)
# ❌ BAD
COPY . .              # Copies everything including .git, tests, docs

# ✅ GOOD
COPY src/ ./src/
COPY public/ ./public/
COPY package*.json ./

Tip (Senior Interview Question): Q: "A Docker image has grown to 2GB — how would you optimize it?" A: (1) Use docker history --no-trunc myimage to identify large layers. (2) Use multi-stage builds — don't include build tools in production. (3) Use Alpine base image (50MB vs 900MB). (4) Run apt-get clean && rm -rf /var/lib/apt/lists/* in the same RUN instruction. (5) Use .dockerignore to exclude unnecessary files. (6) Use the dive tool to find wasted space. (7) Combine RUN instructions. Typically, 2GB → 150-200MB is achievable!

Container Lifecycle

Container States

Every Docker container passes through multiple states during its life. Think of them like React component lifecycle (mount, update, unmount) — but more complex:

CREATED — Container created via docker create but not started
RUNNING — Container is active
PAUSED — Frozen, consuming no CPU
EXITED — Stopped or done (via docker stop, docker kill, exit, or crash)
RESTARTING — Auto-restart loop in progress
DEAD — Restart attempts failed, max retries exceeded
REMOVED — Container permanently deleted via docker rm

Image vs Container

Aspect	Image	Container
Nature	Blueprint / Template	Running instance
State	Immutable, static (never changes)	Mutable, stateful (can change)
Storage	Read-only layers	Image layers + writable layer on top
Multiple instances	Shared by all containers using it	Each container fully independent
Lifecycle	Build once, use forever	Create → Run → Stop → Delete
Disk space	Stored once even if 100 containers use it	Only writable layer extra per container
Analogy	Recipe	The actual dish
React analogy	React Component class/function	Component instance rendered in DOM

Container Lifecycle Commands — Quick Reference

# Full lifecycle — a container's complete journey
docker create --name myapp nginx          # Create but don't start
docker start myapp                        # Start it
docker pause myapp                        # Freeze it (stops CPU usage)
docker unpause myapp                      # Unfreeze it
docker stop myapp                         # Graceful stop (SIGTERM then SIGKILL)
docker start myapp                        # Start again
docker restart myapp                      # Stop + Start in one command
docker kill myapp                         # Immediate kill (SIGKILL)
docker rm myapp                           # Remove container (permanently delete)

# One-liner (create + start + remove when done — for temporary tasks)
docker run --rm --name temp-app nginx

# Check current state of all containers
docker ps -a

Tip: docker run = docker create + docker start + docker attach (in foreground mode). CI/CD pipelines almost always use docker run. docker create is useful when you want to set up a container first and start it later.

Caution: If a container enters the DEAD state, it cannot be restarted. You can only remove it with docker rm. This happens when restart policy max retries are exhausted. Monitor DEAD containers in production!

Essential Container Commands

docker run — The Master Command

docker run creates a container, starts it, and optionally attaches to it. Understanding its options is essential.

# Full syntax
docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

# -d: Detached mode (run in background)
docker run -d nginx
docker run -d --name my-nginx nginx

# -p: Port mapping (host_port:container_port)
docker run -d -p 8080:80 nginx             # Host 8080 → Container 80
docker run -d -p 127.0.0.1:8080:80 nginx  # Bind to specific IP
docker run -d -p 8080:80 -p 443:443 nginx # Multiple ports
docker run -d -P nginx                     # Auto-map ALL EXPOSED ports

# -v: Volume mount (old syntax)
docker run -d -v /host/path:/container/path nginx
docker run -d -v myvolume:/app/data nginx
docker run -d -v $(pwd):/app node:18       # Mount current directory

# --mount: Volume mount (new, preferred syntax)
docker run -d --mount type=bind,source=/host/path,target=/container/path nginx
docker run -d --mount type=volume,source=myvolume,target=/data nginx
docker run -d --mount type=tmpfs,target=/tmp nginx

# -e: Environment variables
docker run -d -e NODE_ENV=production node:18
docker run -d -e DB_HOST=localhost -e DB_PORT=5432 myapp
docker run -d --env-file .env myapp        # Load from .env file

# --name: Give the container a name
docker run -d --name my-api myapp:v1.0

# --rm: Automatically remove when it exits
docker run --rm ubuntu echo "hello"        # Perfect for one-off commands
docker run --rm -v $(pwd):/app node:18 npm test

# --network: Connect to a network
docker run -d --network mynetwork nginx
docker run -d --network host nginx         # Use host machine's network

# --restart: Restart policy
docker run -d --restart no nginx              # Never restart (default)
docker run -d --restart always nginx          # Always restart
docker run -d --restart on-failure nginx      # Restart only on failure
docker run -d --restart on-failure:3 nginx    # Max 3 retries
docker run -d --restart unless-stopped nginx  # Restart unless manually stopped

# -it: Interactive Terminal (enter the container)
docker run -it ubuntu bash               # Start bash session
docker run -it --rm ubuntu bash          # Session ends → container deleted
docker run -it node:18 node              # Node.js REPL

# Resource Limits (CRITICAL IN PRODUCTION!)
docker run -d --memory=512m nginx              # Max 512MB RAM
docker run -d --memory=512m --memory-swap=1g nginx  # RAM + swap limit
docker run -d --cpus=0.5 nginx                 # 50% of one CPU
docker run -d --cpus=2 nginx                   # 2 full CPUs

# --user: Run as a specific user (security)
docker run -d --user 1000:1000 nginx
docker run -d --user nobody nginx

# Real-world Production Example — all together
docker run -d \
  --name my-react-app \
  -p 3000:3000 \
  -e NODE_ENV=production \
  -e API_URL=https://api.myapp.com \
  -v /var/log/app:/app/logs \
  --memory=256m \
  --cpus=0.5 \
  --restart unless-stopped \
  --network app-network \
  myreactapp:v1.2.3

Note (Production): Without resource limits, a single buggy container can consume all server memory. A memory limit means the OOM killer will kill the container — not the entire server. CPU limits ensure fair sharing between containers.

docker ps — List Containers

# Running containers only
docker ps

# ALL containers (including stopped)
docker ps -a

# IDs only (useful in scripts)
docker ps -q

# Last 5 created containers
docker ps -n 5

# Custom format
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}"
docker ps --format "{{.Names}}: {{.Status}}"

# Filter by status
docker ps -f status=running
docker ps -f status=exited
docker ps -f name=myapp
docker ps -f ancestor=nginx       # Containers using nginx image

# Show container sizes
docker ps -s

docker stop / start / restart

# Graceful stop — sends SIGTERM, waits 10 sec, then SIGKILL
docker stop myapp
docker stop --time=30 myapp       # Wait 30 sec before SIGKILL

# Stop multiple containers at once
docker stop container1 container2 container3

# Stop all running containers
docker stop $(docker ps -q)

# Start a stopped container
docker start myapp
docker start -a myapp             # Start + attach to output

# Restart = stop + start
docker restart myapp
docker restart --time=5 myapp     # Wait 5 sec before SIGKILL

docker pause / unpause — Freeze a Container

# Freeze container (uses cgroups freezer)
# Container state is preserved, no CPU consumed
docker pause myapp

# Unfreeze — container continues from where it left off
docker unpause myapp

# When to use:
# - Load testing: pause one container, test another
# - Debug: capture exact state without losing data
# - Resource saving: temporarily free CPU

Note: docker pause = process freeze (still in memory, state preserved, no CPU). docker stop = process terminate (SIGTERM → SIGKILL). Unpause is very fast because the process doesn't restart — it just unfreezes.

docker kill — Immediate Termination

# Immediate kill (SIGKILL — no grace period)
docker kill myapp

# Send a specific signal
docker kill --signal=SIGTERM myapp    # Graceful shutdown request
docker kill --signal=SIGHUP myapp     # Nginx config reload (without restart!)
docker kill --signal=9 myapp          # SIGKILL (same as default kill)
docker kill --signal=SIGUSR1 myapp    # App-specific custom signal

docker rm — Remove Containers

# Remove a stopped container
docker rm myapp

# Force remove a running container (think before you do this!)
docker rm -f myapp

# Remove container and its anonymous volume
docker rm -v myapp

# Remove all stopped containers at once (recommended!)
docker container prune
docker rm $(docker ps -aq -f status=exited)

# Remove ALL containers (running + stopped) — nuclear option!
docker rm -f $(docker ps -aq)

docker logs — View Logs (First Debug Tool)

When something is not working, look at logs first!

# Basic logs
docker logs myapp

# Follow (like tail -f, real-time updates)
docker logs -f myapp

# Last 100 lines only
docker logs --tail 100 myapp

# Show with timestamps
docker logs -t myapp

# Logs since a specific time
docker logs --since 2024-01-01 myapp
docker logs --since 1h myapp              # Last 1 hour of logs

# Most useful combination — last 50 lines follow with timestamps
docker logs -f -t --tail 50 myapp

Tip: docker logs works when the application writes to stdout/stderr. If the app writes to a file (e.g., /var/log/app.log), use docker exec -it myapp tail -f /var/log/app.log, or mount a volume and read from the host.

docker exec — Enter a Running Container

This command is used daily for debugging. Enter a running container and investigate.

# Run a command inside the container
docker exec myapp ls /app
docker exec myapp cat /etc/nginx/nginx.conf

# Interactive bash session (most common use case)
docker exec -it myapp bash

# Alpine-based containers: use sh (bash not available)
docker exec -it myapp sh

# Login as root (when full access is needed)
docker exec -it --user root myapp bash

# With environment variable
docker exec -e DEBUG=true myapp node debug.js

# Real-world examples
docker exec -it postgres psql -U admin -d mydb    # Enter PostgreSQL
docker exec -it redis redis-cli                   # Open Redis CLI
docker exec -it nginx nginx -t                    # Test nginx config
docker exec -it nginx nginx -s reload             # Gracefully reload nginx
docker exec -it myapp cat /app/config.json        # Read config file
docker exec -it myapp env                         # View all env variables
docker exec -it myapp ps aux                      # View running processes

Caution: docker exec starts a new process inside the container. docker attach connects to the container's main process — pressing Ctrl+C will STOP the container! For debugging, always use exec.

docker inspect — Extract Full Details

# Full JSON info
docker inspect myapp

# Extract specific info with Go templates
docker inspect --format '{{.NetworkSettings.IPAddress}}' myapp
docker inspect --format '{{.Config.Env}}' myapp
docker inspect --format '{{.Config.Image}}' myapp
docker inspect --format '{{.State.StartedAt}}' myapp

# Port mappings
docker inspect --format '{{json .NetworkSettings.Ports}}' myapp | jq

# Volume/mount info
docker inspect --format '{{json .Mounts}}' myapp | jq

# Container exit code (for debugging)
docker inspect --format '{{.State.ExitCode}}' myapp

docker top, docker stats, docker cp, docker commit

# View processes inside container (from host, without entering container)
docker top myapp
docker top myapp aux
docker top myapp -ef

# Real-time resource usage (live dashboard)
docker stats
docker stats myapp
docker stats --no-stream          # One snapshot
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

# Copy files between container and host
docker cp myapp:/app/config.json ./config.json     # Container → host
docker cp ./config.json myapp:/app/config.json     # Host → container
docker cp myapp:/var/log/app.log ./app.log
# Works with stopped containers too!

# Create a new image from a modified container
docker commit myapp myapp-modified:v2
docker commit -m "Added custom nginx config" -a "Author" myapp myapp:custom
# NOTE: Use Dockerfiles in production — docker commit is not reproducible!

Container Operations — Advanced Concepts

Foreground vs Background — Attached vs Detached Mode

# Foreground (attached) — output visible, Ctrl+C stops the container
docker run nginx
# Terminal is blocked, nginx logs appear here

# Background (detached) — runs in background, terminal is free
docker run -d nginx
# Returns container ID, you can do other work

# Attach to a running container's output
docker attach myapp
# WARNING: Ctrl+C stops the container!
# To detach without stopping: Ctrl+P then Ctrl+Q

Detaching Without Stopping — Critical Shortcut

# Start an interactive session
docker run -it ubuntu bash

# To leave WITHOUT stopping the container:
# Press Ctrl+P then Ctrl+Q (both quickly)
# Container keeps running, you get your terminal back

# Return later
docker attach ubuntu_container_name

# exec vs attach — which to use?
# exec — starts a new process (SAFE, always use this)
docker exec -it myapp bash

# attach — connects to main process (RISKY for debugging)
docker attach myapp   # Ctrl+C = container STOP!

Caution: New users often use docker attach and then press Ctrl+C thinking they are only detaching from the terminal — but the container STOPS! In production, always use docker exec -it myapp bash. If you must use attach, detach with Ctrl+P Ctrl+Q.

Container Exit Codes

Exit codes tell you why a container stopped. Visible in docker ps -a in the STATUS column as "Exited (137)".

Exit Code	Meaning	Common Cause
0	Success	Container completed its task successfully
1	General Error	Application crash, unhandled exception, startup failure
125	Docker daemon error	Invalid `docker run` options, Docker internal error
126	Command can't execute	Permission error — command found but not executable
127	Command not found	Wrong command or PATH issue, typo in CMD/ENTRYPOINT
130	Ctrl+C (SIGINT)	Manual interruption
137	SIGKILL (128+9)	OOM killer or docker kill / docker stop timeout
139	Segmentation fault	Application memory error
143	SIGTERM (128+15)	docker stop — graceful shutdown signal

Tip: Signal-based exit codes = 128 + signal_number. SIGKILL = signal 9, exit code = 137. SIGTERM = signal 15, exit code = 143. Exit 137 almost always means OOM or force kill.

Restart Policies — Detailed

# no (default) — Never restart
docker run --restart no nginx
# Use case: one-off scripts, batch jobs, dev environment

# always — Always restart (ignores exit code)
docker run --restart always nginx
# Note: Also auto-starts on Docker daemon restart!
# Use case: critical services that must always be running

# on-failure — Restart only on failure (non-zero exit)
docker run --restart on-failure nginx         # Unlimited retries
docker run --restart on-failure:5 nginx       # Max 5 retries, then DEAD
# Use case: services that may occasionally crash

# unless-stopped — Like always, but respects manual stop
docker run --restart unless-stopped nginx
# docker stop → will NOT restart on daemon restart
# Use case: Production services (RECOMMENDED for most cases)

# Change policy on a running container (no restart required)
docker update --restart unless-stopped myapp
docker update --restart no myapp              # Disable restart

Policy	Restart on crash?	Restart on daemon restart?	Restart after docker stop?
`no`	Never	No	No
`always`	Yes	Yes	Yes (even after docker stop!)
`on-failure`	Yes (non-zero exit only)	Yes (if was running)	No
`unless-stopped`	Yes	Yes	No (respects manual stop)

Resource Monitoring

# Real-time stats dashboard (updates every 1 second)
watch -n 1 docker stats --no-stream

# Docker system overall usage
docker system df              # Disk usage — images, containers, volumes, build cache
docker system events          # Live event stream
docker system info            # Complete Docker system info

# Monitor events (powerful debugging tool)
docker events                              # All events real-time
docker events --filter 'type=container'   # Container events only
docker events --filter 'event=die'        # Container deaths
docker events --filter 'event=start'      # Container starts
docker events --since 1h                  # Events from last 1 hour

# Container health check status
docker inspect --format '{{.State.Health.Status}}' myapp
docker inspect --format '{{json .State.Health}}' myapp | jq

Cleanup Commands

# Check disk usage first
docker system df
# TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
# Images          25        3         12.5GB    10.2GB (82%)
# Containers      8         2         150MB     130MB (86%)
# Local Volumes   15        3         4.2GB     3.8GB (90%)
# Build Cache     -         -         2.1GB     2.1GB

# Individual cleanup (specific resources)
docker container prune                      # Remove stopped containers
docker image prune                          # Remove dangling images
docker image prune -a                       # Remove all unused images
docker volume prune                         # Remove unused volumes
docker network prune                        # Remove unused networks

# Nuclear options — use carefully!
docker system prune                         # Remove everything unused (no volumes)
docker system prune -a --volumes            # Remove everything including volumes

# Non-interactive (for CI/CD scripts)
docker system prune -f
docker system prune -af --volumes           # Full nuclear, no confirmation

# Smart cleanup — only remove old items
docker container prune --filter "until=24h"    # Containers older than 24 hours
docker image prune --filter "until=168h"       # Dangling images older than 1 week
docker system prune --filter "until=72h"       # Everything older than 3 days

Caution (Production Cleanup Strategy): On CI/CD servers, set up a daily cron job: 0 2 * * * docker system prune -f. On production servers, only docker container prune and docker image prune are safe — never run docker volume prune on production! An accidental docker volume prune has permanently deleted production data for multiple teams.

Tip (Interview — SIGTERM vs SIGKILL): docker stop first sends SIGTERM — a graceful shutdown request. The application has 10 seconds to clean up (close connections, complete in-flight requests, flush data). If it doesn't exit within 10 seconds, SIGKILL is sent — forced termination, no cleanup. docker kill sends SIGKILL directly — no grace period. Use docker stop in production so applications can shut down properly. In Node.js: process.on('SIGTERM', gracefulShutdown). Exit codes: SIGTERM = 143, SIGKILL = 137.

Dockerfile Basics

What is a Dockerfile?

A Dockerfile is a text file containing instructions for how to build an image. Think of it as a recipe card — ingredients and steps. Each instruction creates a layer, and layers together form the final Docker image.

Why build a custom image?

Your app requires a specific Node/Python version
Custom environment variables or config are needed
Pre-installed tools (curl, git, etc.) are required
You need a production-ready, minimal image
Security hardening — non-root user, read-only filesystem
Reproducible builds — every developer gets the same environment

Where to put the Dockerfile

Usually in the project root — the same directory as package.json or requirements.txt.

my-project/
├── Dockerfile          ← here
├── Dockerfile.dev      ← for dev environment
├── Dockerfile.prod     ← for prod environment
├── .dockerignore       ← CRITICAL! Never forget this
├── package.json
├── src/
└── dist/

docker build — Building an Image

# Build with current dir as context
docker build -t myapp:latest .

# Build with specific Dockerfile
docker build -f Dockerfile.prod -t myapp:prod .

# Build from a remote Git repo
docker build -t myapp https://github.com/user/repo.git

# Build with no cache (fresh build)
docker build --no-cache -t myapp .

# Build with build args
docker build --build-arg NODE_ENV=production -t myapp .

# Build with multiple tags
docker build -t myapp:latest -t myapp:v1.0.0 .

# Build with verbose output (useful for CI)
docker build --progress=plain -t myapp .

Tip: When you write docker build -t myapp ., the . (dot) is the build context — all files in this directory are sent to the Docker daemon. If your node_modules is 500MB and you have no .dockerignore, that entire directory gets sent. This is why .dockerignore is critical.

.dockerignore — Never Skip This

This file works exactly like .gitignore — it tells the Docker daemon which files to exclude from the build context.

Caution: Never let .env files into a Docker image. If AWS_SECRET_KEY or a database password ends up in an image, anyone can run docker history myapp and see it — even if you deleted it in a later layer!

# .dockerignore - put this in EVERY project!

# Dependencies (will be installed inside container)
node_modules/
vendor/
__pycache__/
*.pyc
.pytest_cache/
venv/
.venv/

# Build outputs
dist/
build/
.next/
out/
target/

# Environment files (SECURITY RISK!)
.env
.env.local
.env.*.local
.env.development
.env.production
*.pem
*.key
secrets/

# Version control
.git/
.gitignore

# Logs
*.log
logs/

# OS files
.DS_Store
Thumbs.db

# IDE/Editor files
.idea/
.vscode/

# CI/CD & Docker meta
.github/
.dockerignore
Dockerfile*

# Documentation
README.md
*.md
docs/

# Test & Coverage
coverage/
.nyc_output
__tests__/

Build Cache

Docker caches every layer. If a layer has not changed, Docker uses the cache — builds become very fast.

Scenario	Result	Speed
Layer unchanged	Cache HIT	Instant
Layer changed	Cache MISS	Slow (rebuild)
After a cache miss, all subsequent layers	Force rebuild	Slow
`--no-cache` flag	Skip all cache	Slowest

Tip: What changes least goes on top. What changes most (source code) goes at the bottom. If package.json hasn't changed, npm install won't re-run — only source code copy will.

Image Naming Conventions

# Tag formats
myapp:latest           # ❌ Avoid in production!
myapp:v1.0.0           # ✅ Semantic version (recommended)
myapp:1.0.0-alpine     # ✅ Version + variant
myapp:2024-01-15       # ✅ Date-based
myapp:abc1234          # ✅ Git commit hash (traceable)
myapp:prod-v1.2.3      # ✅ Environment-specific

# With registry
docker.io/myuser/myapp:v1.0.0          # Docker Hub
ghcr.io/myorg/myapp:v1.0.0            # GitHub Container Registry
123456.dkr.ecr.us-east-1.amazonaws.com/myapp:v1.0.0  # AWS ECR

# Build with git commit tag (common in CI/CD)
docker build -t myapp:$(git rev-parse --short HEAD) .

Caution: Using :latest in production means if you push a new image with :latest, any server that restarts will pull the new image — without any warning! Always use specific version tags for reproducible deployments.

Dockerfile Instructions — Deep Dive

FROM — Every Dockerfile Starts Here

# Basic — always specify a version
FROM ubuntu:22.04

# Specific digest (production best practice — immutable!)
FROM ubuntu@sha256:abc123def456...

# Multi-stage naming (used later with COPY --from)
FROM node:18-alpine AS builder
FROM nginx:alpine AS production

# ARG before FROM (dynamic base image version)
ARG NODE_VERSION=18
FROM node:${NODE_VERSION}-alpine

# ✅ BEST: Pinned version + variant
FROM node:18.19.0-alpine3.19

# ❌ BAD: Unpredictable, breaks builds
FROM node:latest

Tip (Alpine vs Slim vs Full): alpine: uses musl libc, 5MB, fastest — but some native modules may not compile. slim: Debian-based, ~75MB, more compatible. full: Most compatible, but large image. For most Node.js apps: start with node:18-alpine.

LABEL — Image Metadata

# ✅ BEST: All together in one layer
LABEL \
  maintainer="you@company.com" \
  version="1.0.0" \
  description="React DevOps Guide App" \
  org.opencontainers.image.source="https://github.com/user/repo" \
  org.opencontainers.image.created="2024-01-15" \
  org.opencontainers.image.revision="abc1234"

# Inspect labels
docker inspect myapp --format='{{json .Config.Labels}}'

RUN — Execute Commands

# Shell form (runs in /bin/sh -c)
RUN apt-get update

# Exec form (no shell — preferred for clarity)
RUN ["apt-get", "update"]

# ❌ BAD: Multiple RUN = multiple layers = big image
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y git

# ✅ BEST: All in one RUN + clean cache in SAME layer
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
      curl \
      git \
      build-essential && \
    rm -rf /var/lib/apt/lists/*   # CRITICAL: clean apt cache!

# Node.js
RUN npm ci --only=production && \
    npm cache clean --force

# Python
RUN pip install --no-cache-dir -r requirements.txt

# Alpine (apk package manager)
RUN apk add --no-cache \
    curl \
    git \
    bash

Caution: If you don't include rm -rf /var/lib/apt/lists/*, the apt cache (which can be several MB) stays in the image — size grows unnecessarily. Always clean in the SAME RUN instruction, not in a separate one (cleaning in a separate layer does not reduce the layer size).

CMD — Default Command

# ✅ Exec form (PREFERRED — no shell processing, better signal handling)
CMD ["nginx", "-g", "daemon off;"]
CMD ["node", "server.js"]
CMD ["python", "app.py"]
CMD ["uvicorn", "main:app", "--host", "0.0.0.0"]

# ❌ Shell form (avoid — shell processing, harder to handle signals)
CMD nginx -g "daemon off;"
CMD node server.js

# ⚠️ Multiple CMD — only the LAST one counts!
CMD ["node", "app.js"]
CMD ["node", "server.js"]   # ← This one runs; the above is ignored

# Override at runtime
docker run myapp node debug.js   # CMD overridden

ENTRYPOINT — Fixed Executable

# Exec form (preferred)
ENTRYPOINT ["node", "server.js"]

# ENTRYPOINT + CMD combination (powerful pattern!)
ENTRYPOINT ["node"]
CMD ["server.js"]    # docker run myapp          → node server.js
                     # docker run myapp debug.js  → node debug.js (CMD overridden)
                     # docker run myapp --version → node --version

# Common pattern: entrypoint script for setup logic
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["server"]

# entrypoint.sh example:
#!/bin/sh
set -e
echo "Starting container..."
# Run migrations, wait for DB, etc.
exec "$@"   # "$@" = pass CMD arguments through

CMD vs ENTRYPOINT — Most Important Comparison

Feature	CMD	ENTRYPOINT
Override easily?	Yes — `docker run myapp npm test`	No — requires `--entrypoint` flag
Purpose	Provide default arguments	Set a fixed executable
When used together	CMD = default args for ENTRYPOINT	ENTRYPOINT = command, CMD = args
Override example	`docker run myapp bash`	`docker run --entrypoint bash myapp`
Use case	Flexible containers (same image for dev/prod)	Single-purpose containers

Tip (Interview Question): "What is the difference between CMD and ENTRYPOINT?" — Short answer: CMD is easily overridden, ENTRYPOINT is not. Use them together — ENTRYPOINT as the executable, CMD as the default arguments.

COPY and ADD

# COPY — preferred for most cases
COPY package.json /app/
COPY . /app/
COPY package.json package-lock.json /app/

# With ownership (chown)
COPY --chown=node:node . /app/

# From a multi-stage build (POWERFUL!)
COPY --from=builder /app/dist /usr/share/nginx/html/
COPY --from=builder /app/.next/standalone ./

# ADD — supports URLs and auto-extracts tar files
ADD https://example.com/file.tar.gz /tmp/        # ❌ Prefer curl in RUN
ADD myarchive.tar.gz /app/                       # ✅ Auto-extracts tar

Feature	COPY	ADD
Local files	Yes	Yes
Download from URLs	No	Yes
Auto-extract tar	No	Yes
Caching	Predictable	Less predictable
Recommended	For most cases	Only for tar extraction

WORKDIR, ENV, ARG

# WORKDIR — sets working directory
WORKDIR /app
WORKDIR /usr/src/app

# ❌ BAD: Use cd command
RUN cd /app && npm install  # Only valid for this RUN

# ✅ GOOD: Use WORKDIR
WORKDIR /app
RUN npm install              # Runs in /app

# ENV — environment variables (build + runtime)
ENV NODE_ENV=production \
    PORT=3000 \
    LOG_LEVEL=info \
    TZ=Asia/Kolkata

# Override at runtime
docker run -e NODE_ENV=development myapp
# ⚠️ Never put secrets in ENV — visible in docker inspect!

# ARG — build-time arguments (NOT available at runtime)
ARG NODE_VERSION=18
FROM node:${NODE_VERSION}-alpine

ARG BUILD_DATE
ARG GIT_COMMIT
ARG APP_VERSION=1.0.0

# Pass at build time
docker build \
  --build-arg NODE_VERSION=20 \
  --build-arg BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --build-arg GIT_COMMIT=$(git rev-parse --short HEAD) \
  -t myapp .

# Convert ARG to ENV (to make it available at runtime)
ARG REACT_APP_API_URL
ENV REACT_APP_API_URL=$REACT_APP_API_URL

Feature	ARG	ENV
Available at	Build time only	Build + Runtime
Persists in container	No	Yes
Can use before FROM	Yes	No
Visibility	In layer history	In `docker inspect`

EXPOSE, VOLUME, USER

# EXPOSE — documentation only (does NOT actually open the port!)
EXPOSE 3000
EXPOSE 80 443
EXPOSE 8080/tcp
EXPOSE 53/udp

# Actual port publishing requires -p flag at runtime:
docker run -p 8080:3000 myapp   # host:container

# VOLUME — declare mount points
VOLUME /app/data
VOLUME /var/lib/postgresql/data
VOLUME ["/app/logs", "/app/uploads"]

# In production, use named volumes:
docker run -v postgres_data:/var/lib/postgresql/data postgres

# USER — run as non-root (security!)
# ❌ BAD: Running as root (default if USER not specified)

# ✅ Method 1: Named user
RUN groupadd -r appuser && useradd -r -g appuser appuser
USER appuser

# ✅ Method 2: Numeric UID (more secure)
RUN addgroup --system --gid 1001 nodejs && \
    adduser --system --uid 1001 nextjs
USER 1001

# ✅ Node.js: Use the built-in 'node' user
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY --chown=node:node . .
USER node
EXPOSE 3000
CMD ["node", "server.js"]

Note: Kubernetes and AWS ECS policies typically prohibit containers running as root. Always specify a non-root USER. In Kubernetes, set runAsNonRoot: true in PodSecurityPolicy.

HEALTHCHECK

# HTTP health check (common for web apps)
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

# Node.js health check (when curl is not available)
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', r => r.statusCode === 200 ? process.exit(0) : process.exit(1))"

# Disable inherited health check
HEALTHCHECK NONE

# Options:
# --interval=30s    → check every 30 seconds (default: 30s)
# --timeout=3s      → fail if no response within 3s (default: 30s)
# --start-period=5s → 5s grace period at start (container boot time)
# --retries=3       → 3 consecutive failures = unhealthy

# Check status
docker inspect --format='{{.State.Health.Status}}' container_name
docker ps   # STATUS column shows (healthy) or (unhealthy)

Shell Form vs Exec Form — Signal Handling

# Shell form: runs via /bin/sh -c
CMD nginx -g "daemon off;"
ENTRYPOINT node server.js

# Exec form: runs directly (no shell)
CMD ["nginx", "-g", "daemon off;"]
ENTRYPOINT ["node", "server.js"]

# KEY DIFFERENCE: Signal handling!
# Shell form:
#   /bin/sh (PID 1) → nginx (child process)
#   SIGTERM goes to /bin/sh, not nginx → dirty shutdown!

# Exec form:
#   nginx (PID 1) directly
#   SIGTERM goes directly to nginx → graceful shutdown!

# RULE: ALWAYS use exec form in CMD and ENTRYPOINT

Tip (Interview): "What is the difference between shell form and exec form?" — Shell form runs via /bin/sh -c (does not become PID 1), exec form executes directly (becomes PID 1). In production, exec form is preferred because signals are properly received.

Writing Efficient Dockerfiles

Before vs After — 1.5 GB to 180 MB

Before (1.5 GB image):

FROM ubuntu:22.04
RUN apt-get update
RUN apt-get install -y nodejs npm
RUN apt-get install -y git
COPY . /app
WORKDIR /app
RUN npm install
EXPOSE 3000
CMD node server.js

Problems: Ubuntu base (~80MB), Node+npm via apt (~200MB), multiple apt layers, all node_modules included, no cache optimization, running as root, shell form CMD.

After (180 MB image):

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
USER node
EXPOSE 3000
CMD ["node", "server.js"]

Benefits: Alpine base (~5MB), official Node image, layered cache optimization, production deps only, non-root user, exec CMD form.

Layer Ordering — Cache Optimization

# ❌ BAD ORDER: package.json copied with source code
FROM node:18-alpine
WORKDIR /app
COPY . .                    # Source code AND package.json together
RUN npm ci                  # Runs on every code change!
RUN npm run build
EXPOSE 3000
CMD ["node", "server.js"]

# ✅ OPTIMAL ORDER: What changes least goes first
FROM node:18-alpine          # Layer 1: base image (almost always cached)
WORKDIR /app                 # Layer 2: almost always cached
COPY package*.json ./        # Layer 3: rebuilt only when package.json changes
RUN npm ci                   # Layer 4: rebuilt only when package.json changes
COPY . .                     # Layer 5: rebuilt on any code change
RUN npm run build            # Layer 6: rebuilt on any code change
EXPOSE 3000
CMD ["node", "server.js"]

# RESULT:
# If only src/App.js changed:
# → Layers 1-4: CACHE HIT (npm ci skipped!)
# → Layers 5-6: rebuild
# Build time: 10s instead of 60s!

Tip: Thumb rule: "What changes least goes on top." Order: Base image → System deps → Package manifest (package.json/requirements.txt) → Dependencies install → Source code copy → Build → Configuration.

Base Image Choices

Base Image	Size	Use Case	Pros/Cons
`ubuntu:22.04`	~80MB	Legacy apps needing apt	Familiar, large
`debian:slim`	~75MB	Debian apps, smaller	Compatible, slim
`alpine:3.19`	~5MB	Smallest possible	musl libc issues sometimes
`node:18-alpine`	~50MB	Node.js apps	Best for Node
`python:3.11-slim`	~130MB	Python apps	Best for Python
`gcr.io/distroless/nodejs`	~30MB	Security-focused	No shell — very secure
`scratch`	0MB	Static Go/Rust binaries	Smallest possible

Security Best Practices

# 1. Pin specific versions
FROM node:18.19.0-alpine3.19   # ✅ Exact version
FROM node:latest                # ❌ Unpredictable

# 2. Non-root user (always!)
USER node

# 3. NEVER put secrets in ENV/ARG permanently
# ❌ WRONG — visible in docker history!
RUN export API_KEY=secret123 && ./setup.sh

# ✅ RIGHT — use runtime env vars
docker run -e API_KEY=secret123 myapp

# 4. Don't install unnecessary packages
RUN apt-get install -y --no-install-recommends \
    curl && \
    rm -rf /var/lib/apt/lists/*

# 5. Read-only filesystem (at runtime)
docker run --read-only \
    --tmpfs /tmp \
    --tmpfs /app/logs \
    myapp

# 6. Drop all capabilities (at runtime)
docker run --cap-drop ALL \
    --cap-add NET_BIND_SERVICE \
    myapp

# 7. Scan images for vulnerabilities
docker scout cves myapp:latest
trivy image myapp:latest

Note (Production Security Checklist): Non-root USER | .dockerignore with .env | No secrets in Dockerfile | Specific base image version | HEALTHCHECK defined | Image vulnerability scan in CI/CD | Read-only filesystem where possible.

npm install vs npm ci

Command	Use Case	In Dockerfile?
`npm install`	Development, install from package.json	Avoid
`npm ci`	Exact install from package-lock.json	Always use
`npm ci --only=production`	Exclude devDependencies	For production

Multi-Stage Builds

Multi-stage builds keep the build toolchain in a throw-away stage and ship only the compiled artifact in a minimal final image, cutting image size by 80–90%.

Why Multi-Stage Builds?

The problem: build tools are needed at build time (compiler, test runner, devDependencies) but should not be in the production image. Without multi-stage builds, everything ends up in the final image — size explodes.

Without Multi-Stage	With Multi-Stage
Build tools included (1.2 GB)	Only production artifacts (120 MB)
devDependencies included	Build tools discarded
Source code included	Source code discarded
Security risk (unnecessary tools)	Minimal attack surface
Slow to push/pull	Fast to push/pull

# BEFORE multi-stage: build tools in final image
FROM node:18
WORKDIR /app
COPY . .
RUN npm install      # devDependencies too — 500MB!
RUN npm run build
CMD ["node", "server.js"]
# Final image: ~1.2 GB

# AFTER multi-stage: only production artifacts
# ---- Stage 1: Builder ----
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci              # ALL deps (devDeps for build)
COPY . .
RUN npm run build       # Create production bundle

# ---- Stage 2: Runner (final image) ----
FROM node:18-alpine AS runner
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production   # Production deps only
COPY --from=builder /app/dist ./dist   # Only built files!
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]
# Final image: ~120 MB  (90% size reduction!)

Tip: COPY --from=builder only copies the files you specify — not the entire builder stage. Everything else (node_modules, devDeps, source code, compiler) is automatically discarded. The final image contains only the content from the LAST FROM onwards.

React App — Production-Ready Multi-Stage

# ============================================
# React App — Production Dockerfile
# Multi-stage: Node builder → Nginx server
# Final size: ~25-30MB
# ============================================

# ---- Stage 1: Build ----
FROM node:18-alpine AS builder

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .

ARG REACT_APP_API_URL=https://api.myapp.com
ARG REACT_APP_ENV=production
ARG REACT_APP_VERSION=1.0.0

ENV REACT_APP_API_URL=$REACT_APP_API_URL \
    REACT_APP_ENV=$REACT_APP_ENV \
    REACT_APP_VERSION=$REACT_APP_VERSION

RUN npm run build
# Output: /app/build/

# ---- Stage 2: Serve ----
FROM nginx:1.25-alpine AS production

LABEL maintainer="you@company.com" \
      version="1.0.0" \
      description="React App Production Image"

COPY --from=builder /app/build /usr/share/nginx/html

# Custom Nginx config (required for SPA routing!)
COPY nginx.conf /etc/nginx/conf.d/default.conf

RUN chown -R nginx:nginx /usr/share/nginx/html && \
    chown -R nginx:nginx /var/cache/nginx && \
    touch /var/run/nginx.pid && \
    chown -R nginx:nginx /var/run/nginx.pid

USER nginx

EXPOSE 80

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget -qO- http://localhost/health || exit 1

CMD ["nginx", "-g", "daemon off;"]

# nginx.conf
server {
    listen 80;
    server_name _;
    root /usr/share/nginx/html;
    index index.html;

    # Handle React Router (SPA routing!)
    # Without this, refreshing /about gives a 404!
    location / {
        try_files $uri $uri/ /index.html;
    }

    # Long-term cache for static assets
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }

    # Health check endpoint
    location /health {
        return 200 "healthy\n";
        add_header Content-Type text/plain;
    }

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;

    # Gzip compression
    gzip on;
    gzip_types text/plain text/css application/json
               application/javascript text/xml application/xml;
    gzip_min_length 1000;
}

Go Application — Scratch Image (10MB!)

# ---- Stage 1: Go Binary Build ----
FROM golang:1.21-alpine AS builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

# Static binary build (no external dependencies)
RUN CGO_ENABLED=0 \
    GOOS=linux \
    GOARCH=amd64 \
    go build \
    -ldflags="-w -s" \
    -a -installsuffix cgo \
    -o app .

# ---- Stage 2: Scratch (empty base!) ----
FROM scratch

# SSL certificates (for HTTPS calls)
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

# Only the binary
COPY --from=builder /app/app /app

EXPOSE 8080

ENTRYPOINT ["/app"]

# Final image: ~8-10MB (just the binary!)

Java Spring Boot Multi-Stage

# ---- Stage 1: Maven Build ----
FROM maven:3.9-openjdk-17-slim AS builder

WORKDIR /app

# Copy pom.xml first (dependency caching!)
COPY pom.xml .
RUN mvn dependency:go-offline -B   # Download all dependencies

COPY src ./src
RUN mvn clean package -DskipTests -B

# ---- Stage 2: JRE Only (no JDK!) ----
FROM eclipse-temurin:17-jre-alpine AS production

WORKDIR /app

RUN addgroup --system --gid 1001 spring && \
    adduser --system --uid 1001 spring
USER spring

# Only the JAR (not Maven, not JDK)
COPY --from=builder /app/target/*.jar app.jar

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
  CMD curl -f http://localhost:8080/actuator/health || exit 1

ENTRYPOINT ["java", \
  "-XX:+UseContainerSupport", \
  "-XX:MaxRAMPercentage=75.0", \
  "-jar", "app.jar"]

Building Specific Stages

# Build up to a specific stage (useful for debugging!)
docker build --target builder -t myapp:builder .
docker build --target production -t myapp:prod .

# Open a shell in the builder stage
docker run --rm -it myapp:builder sh

# Separate development vs production builds
docker build --target development -t myapp:dev .
docker build --target production -t myapp:prod .

Dockerfile Templates Library

These templates are copy-paste ready for production use. All are fully commented and follow security best practices. Customize the EXPOSE port, user names, and CMD for your project.

Template 1: React App (Production-Ready)

# ============================================
# React App — Production Dockerfile
# Multi-stage: Node builder → Nginx server
# Final size: ~25-30MB
# ============================================

# ---- Stage 1: Build ----
FROM node:18-alpine AS builder

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .

ARG REACT_APP_API_URL=https://api.myapp.com
ARG REACT_APP_ENV=production
ARG REACT_APP_VERSION=1.0.0

ENV REACT_APP_API_URL=$REACT_APP_API_URL \
    REACT_APP_ENV=$REACT_APP_ENV \
    REACT_APP_VERSION=$REACT_APP_VERSION

RUN npm run build
# Output: /app/build/

# ---- Stage 2: Serve ----
FROM nginx:1.25-alpine AS production

LABEL maintainer="you@company.com" \
      version="1.0.0" \
      description="React App Production Image"

COPY --from=builder /app/build /usr/share/nginx/html

COPY nginx.conf /etc/nginx/conf.d/default.conf

RUN chown -R nginx:nginx /usr/share/nginx/html && \
    chown -R nginx:nginx /var/cache/nginx && \
    touch /var/run/nginx.pid && \
    chown -R nginx:nginx /var/run/nginx.pid

USER nginx

EXPOSE 80

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget -qO- http://localhost/health || exit 1

CMD ["nginx", "-g", "daemon off;"]

# ============================================
# Build commands:
# docker build -t myreactapp:latest .
# docker build --build-arg REACT_APP_API_URL=https://prod.api.com -t myreactapp:prod .
# docker run -p 8080:80 myreactapp:latest
# ============================================

Template 2: Node.js Express API

# ============================================
# Node.js Express API — Production Dockerfile
# Features: dumb-init, TypeScript support,
#           non-root user, health check
# Final size: ~80-120MB
# ============================================

FROM node:18-alpine AS base

# dumb-init: proper PID 1 handling + signal forwarding
RUN apk add --no-cache dumb-init

WORKDIR /app

# ---- Stage 1: Dependencies ----
FROM base AS deps

COPY package*.json ./
RUN npm ci

# ---- Stage 2: Build (for TypeScript projects) ----
FROM deps AS builder

COPY . .
RUN npm run build 2>/dev/null || echo "No build step found, skipping..."

# ---- Stage 3: Production ----
FROM base AS production

ENV NODE_ENV=production

LABEL maintainer="you@company.com" \
      description="Express API Production Image"

COPY package*.json ./
RUN npm ci --only=production && \
    npm cache clean --force

COPY --from=builder /app/dist ./dist 2>/dev/null || true
COPY --from=deps /app/src ./src 2>/dev/null || true
COPY . .

RUN addgroup --system --gid 1001 nodejs && \
    adduser --system --uid 1001 nodeuser
RUN chown -R nodeuser:nodejs /app

USER nodeuser

EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', (r) => { \
    r.statusCode === 200 ? process.exit(0) : process.exit(1); \
  }).on('error', () => process.exit(1))"

ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "src/server.js"]

# ============================================
# Build commands:
# docker build -t myapi:latest .
# docker run -p 3000:3000 -e DB_URL=mongodb://... myapi:latest
#
# For TypeScript projects, change CMD to:
# CMD ["node", "dist/server.js"]
# ============================================

Template 3: Python Flask App

# ============================================
# Python Flask App — Production Dockerfile
# Features: Gunicorn WSGI, non-root user,
#           multi-stage, health check
# Final size: ~130-160MB
# ============================================

FROM python:3.11-slim AS base

RUN apt-get update && \
    apt-get upgrade -y && \
    rm -rf /var/lib/apt/lists/*

RUN addgroup --system --gid 1001 appgroup && \
    adduser --system --uid 1001 --gid 1001 appuser

WORKDIR /app

# ---- Stage 2: Dependencies ----
FROM base AS deps

COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# ---- Stage 3: Production ----
FROM base AS production

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    FLASK_ENV=production \
    PORT=5000

LABEL maintainer="you@company.com" \
      description="Flask API Production Image"

COPY --from=deps /root/.local /home/appuser/.local
ENV PATH=/home/appuser/.local/bin:$PATH

COPY --chown=appuser:appgroup . .

USER appuser

EXPOSE 5000

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:5000/health')" \
      || exit 1

CMD ["python", "-m", "gunicorn", \
     "--bind", "0.0.0.0:5000", \
     "--workers", "4", \
     "--timeout", "120", \
     "--access-logfile", "-", \
     "--error-logfile", "-", \
     "app:app"]

# ============================================
# requirements.txt must include:
# Flask==3.0.0
# gunicorn==21.2.0
# ============================================

Template 4: Python FastAPI

# ============================================
# FastAPI App — Production Dockerfile
# Features: Uvicorn ASGI, non-root user,
#           async support, health check
# Final size: ~130-150MB
# ============================================

FROM python:3.11-slim AS base

RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

RUN addgroup --system --gid 1001 appgroup && \
    adduser --system --uid 1001 --gid 1001 appuser

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

LABEL maintainer="you@company.com" \
      description="FastAPI Production Image"

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY --chown=appuser:appgroup . .

USER appuser

EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:8000/health || exit 1

CMD ["uvicorn", "main:app", \
     "--host", "0.0.0.0", \
     "--port", "8000", \
     "--workers", "4", \
     "--loop", "uvloop", \
     "--access-log"]

# ============================================
# requirements.txt must include:
# fastapi==0.109.0
# uvicorn[standard]==0.27.0
# uvloop==0.19.0
#
# main.py health endpoint:
# @app.get("/health")
# async def health(): return {"status": "healthy"}
# ============================================

Template 5: Next.js Production

# ============================================
# Next.js App — Production Dockerfile
# Features: Standalone output, optimized,
#           non-root, telemetry disabled
# Final size: ~100-150MB
# ============================================

FROM node:18-alpine AS base
RUN apk add --no-cache libc6-compat
WORKDIR /app

# ---- Stage 2: Dependencies ----
FROM base AS deps

COPY package*.json ./
RUN npm ci

# ---- Stage 3: Build ----
FROM base AS builder

COPY --from=deps /app/node_modules ./node_modules
COPY . .

ENV NEXT_TELEMETRY_DISABLED=1

# REQUIRED: next.config.js must have output: 'standalone'
RUN npm run build

# ---- Stage 4: Runner ----
FROM base AS runner

ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
ENV PORT=3000

LABEL maintainer="you@company.com" \
      description="Next.js Production Image"

RUN addgroup --system --gid 1001 nodejs && \
    adduser --system --uid 1001 nextjs

COPY --from=builder /app/public ./public

COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static

USER nextjs

EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/api/health', r => \
    r.statusCode === 200 ? process.exit(0) : process.exit(1))"

CMD ["node", "server.js"]

# ============================================
# REQUIRED: next.config.js must include:
# const nextConfig = {
#   output: 'standalone',
# }
# module.exports = nextConfig
# ============================================

Caution: For Next.js Docker images, output: 'standalone' in next.config.js is required. Without it, the .next/standalone folder will not be created and COPY will fail.

Quick Reference: Which Template to Use

Project Type	Template	Final Size	Server
React (CRA, Vite)	Template 1	~25-30MB	Nginx
Node.js / Express / Fastify	Template 2	~80-120MB	Node + dumb-init
Python Flask (sync)	Template 3	~130-160MB	Gunicorn
Python FastAPI (async)	Template 4	~130-150MB	Uvicorn
Next.js (full-stack)	Template 5	~100-150MB	Node standalone

Note (Common Patterns in Every Template):

Non-root USER — mandatory for security

HEALTHCHECK — so Docker/Kubernetes knows container health

Multi-stage builds — build tools not in final image

Cache optimization — package.json first, source code second

Exec form CMD — proper signal handling

Specific versions — reproducible builds

LABEL metadata — for traceability

Networking Fundamentals

Why Containers Need Networking

By default, every container runs in an isolated environment — its own filesystem, its own process space, and its own network namespace. However, real-world applications require containers to communicate with each other, connect to external databases, and serve clients. Understanding Docker's networking model is therefore essential.

Scenarios that require networking:

Container-to-Container: A frontend container needs to communicate with a backend API
Container to External Services: An application container needs to connect to AWS RDS or a third-party API
External Clients to Containers: A browser sends a request and a container serves it
Multi-host Communication: Containers on different servers still communicate with each other (Overlay networks)

Container Networking Model (CNM)

Docker defines a standard networking architecture called the Container Networking Model (CNM). It has three core concepts:

Concept	Description	Real-World Analogy
Network	Isolated communication group where containers can talk to each other	An office floor — only people on that floor can communicate directly
Endpoint	A virtual NIC (network interface card) for a container on a network	An employee's desk phone extension
Sandbox	Container's isolated network namespace — its own IP, routing table, and DNS	An employee's complete communication profile

Note: libnetwork — Docker's Networking Brain. CNM is implemented by libnetwork — Docker's networking subsystem. It is written in Go and supports a pluggable driver architecture. This means you can use bridge, overlay, macvlan — any driver — on top of libnetwork.

Network Driver Architecture: libnetwork uses a plugin system. Built-in drivers include: bridge, host, none, overlay, and macvlan. Third-party drivers can also be installed (e.g., Weave, Calico).

Docker Networking Model — Architecture Overview

Internet traffic arrives at the host → is forwarded to the docker0 bridge → passes through to Container A on port 80 via host port 8080. Container A and Container B can communicate directly with each other using their 172.17.x.x IP addresses through the bridge.

Tip: veth Pair — The Magic Behind Container Networking. When Docker starts a container, it creates a virtual ethernet pair (veth) — like two ends of a virtual cable. One end lives inside the container (eth0) and the other end attaches to the docker0 bridge. Data flows seamlessly from one end to the other.

Network Types — Bridge, Host, None, Overlay, Macvlan

The docker0 bridge connects containers on the same host using private IPs; a published port (-p 8080:80) uses iptables NAT to forward external traffic from the host into a specific container.

Bridge Network — The Most Common

When you start a container without the --network flag, it automatically joins the default bridge network (docker0). However, the default bridge has a significant limitation — DNS resolution does not work!

Caution: Common Beginner Mistake — No DNS on Default Bridge! On the default bridge, containers cannot find each other by name — only by IP address. The IP address can change on every restart. Always use a user-defined bridge!

# Default bridge — no DNS!
docker run -d --name db mysql:8        # joins default bridge
docker run -d --name myapp myimage     # joins default bridge
# myapp CANNOT connect to "db" by hostname!
# Must use IP: mysql -h 172.17.0.2 (fragile!)

# User-defined bridge — DNS built-in!
docker network create mynetwork

docker run -d --name db --network mynetwork mysql:8
docker run -d --name myapp --network mynetwork myimage
# Now myapp can connect directly using the "db" hostname!
# MONGODB_URI=mongodb://db:27017/mydb  ← works!

# Create a user-defined bridge with options
docker network create \
  --driver bridge \
  --subnet 172.20.0.0/16 \
  --gateway 172.20.0.1 \
  my-custom-network

Feature	Default Bridge	User-Defined Bridge
DNS Resolution (by name)	No	Yes (container name!)
Automatic Discovery	No	Yes
Recommended?	Legacy only	Always use this
Network Isolation	All containers share it	Per-network isolation
Connect/disconnect from running container	No	Yes

Host Network — Maximum Performance

# Container shares the host's network namespace
docker run --network host nginx
# nginx now listens directly on host port 80
# -p flag is not needed (and does not work)

# Benefits:
# - Best performance: no NAT overhead
# - Low latency: direct kernel network stack access

# Drawbacks:
# - No isolation: container can see all host interfaces
# - Port conflicts: container and host services cannot share the same port

# Use cases:
# - High-performance networking applications
# - Monitoring agents (Prometheus node_exporter)
# - Network packet inspection tools

None Network — Complete Isolation

# No networking at all — completely isolated
docker run --network none myapp

# Inside the container:
# - No internet access
# - No communication with other containers
# - Only loopback (127.0.0.1) is available

# Use cases:
# - Batch data processing jobs (file input/output only)
# - Security-sensitive computations
# - Testing network-agnostic code
# - Malware analysis environments

Overlay Network — Multi-Host Communication (Swarm)

# Multi-host networking — containers on different servers communicate with each other
# Requires: Docker Swarm mode (or Kubernetes)

# Initialize Swarm mode
docker swarm init

# Create an overlay network
docker network create --driver overlay myoverlay

# Attachable overlay (standalone containers can join too)
docker network create --driver overlay --attachable myoverlay

# Uses VXLAN tunneling automatically
# Containers are unaware they are on different hosts!
# 10.0.0.2 (Server 1) <-> 10.0.0.3 (Server 2) — transparent!

Note: Overlay = VXLAN Tunneling. Inside an overlay network, Docker uses VXLAN (Virtual Extensible LAN) — one packet is wrapped inside another. A packet from a container on Server A is wrapped in UDP and sent to Server B, where it is unwrapped. Containers have no idea they are running on different machines.

Macvlan — Container as a Physical Device

# Container gets its own MAC address
# Appears on the network like a real physical device

docker network create \
  --driver macvlan \
  --subnet=192.168.1.0/24 \
  --gateway=192.168.1.1 \
  -o parent=eth0 \
  mymacvlan

docker run -d \
  --name legacy-app \
  --network mymacvlan \
  --ip 192.168.1.50 \
  my-legacy-app

# The router treats 192.168.1.50 as a physical machine!

# Use cases:
# - Legacy applications that expect direct network access
# - Migrating VMs to containers
# - Network monitoring tools

Note: Bridge vs Host Network. Bridge: Container has its own network namespace, communicates with the host via NAT, and requires port mapping. Host: Container shares the host's network namespace — no NAT, no port mapping, maximum performance. Use Bridge when isolation is required; use Host when performance is critical and isolation can be sacrificed.

Tip: Always Use User-Defined Bridges. Never use the default bridge network in production. Create a dedicated user-defined network for each application stack. This provides DNS resolution, isolation, and security all in one. Docker Compose automatically creates a user-defined network for each project.

Network Commands — Complete Reference

docker network ls — Listing Networks

# List all networks
docker network ls
# NETWORK ID     NAME      DRIVER    SCOPE
# a1b2c3d4e5f6   bridge    bridge    local
# f6e5d4c3b2a1   host      host      local
# 0000000000000   none      null      local

# Show full IDs
docker network ls --no-trunc

# Filter by driver
docker network ls --filter driver=bridge

# Filter by name
docker network ls --filter name=mynetwork

# Custom format
docker network ls --format "table {{.ID}}\t{{.Name}}\t{{.Driver}}\t{{.Scope}}"

docker network create — Creating a Network

# Simple bridge network
docker network create mynetwork

# Bridge with full options
docker network create \
  --driver bridge \
  --subnet 172.20.0.0/16 \
  --ip-range 172.20.240.0/20 \
  --gateway 172.20.0.1 \
  --label env=production \
  mynetwork

# Internal network (no external access)
# Containers cannot connect to the internet
docker network create --internal private-db-net

# Overlay network (in Swarm mode)
docker network create --driver overlay --attachable myoverlay

# Macvlan network
docker network create \
  --driver macvlan \
  --subnet=192.168.1.0/24 \
  --gateway=192.168.1.1 \
  -o parent=eth0 \
  mymacvlan

docker network inspect — Viewing Details

# Full JSON details
docker network inspect mynetwork

# Specific field — IPAM config
docker network inspect --format '{{.IPAM.Config}}' mynetwork

# Find the gateway
docker network inspect --format '{{range .IPAM.Config}}{{.Gateway}}{{end}}' mynetwork

# List connected containers (requires jq)
docker network inspect mynetwork | jq '.[0].Containers'

# Most useful: which containers are on this network
docker network inspect mynetwork \
  --format '{{range .Containers}}{{.Name}}: {{.IPv4Address}}{{"\n"}}{{end}}'

docker network connect / disconnect — Runtime Changes

# Connect a running container to a new network
docker network connect mynetwork mycontainer

# Assign a specific IP
docker network connect --ip 172.20.0.100 mynetwork mycontainer

# Add an alias (other containers can reach this container via the alias)
docker network connect --alias db-primary mynetwork mycontainer

# Disconnect (container keeps running)
docker network disconnect mynetwork mycontainer

# Force disconnect
docker network disconnect --force mynetwork mycontainer

# Real use case: zero-downtime maintenance
# 1. Start new container on the new network
# 2. Gradually migrate old containers
# 3. Disconnect from the old network

Tip: Multiple Networks — One Container on Many Networks! A container can be connected to more than one network simultaneously. For example, a "gateway" container can be connected to the frontend network on one side and the backend network on the other. This is useful in DMZ architectures where you want to control traffic flow.

docker network rm / prune — Cleanup

# Remove a network (disconnect all containers first!)
docker network rm mynetwork
# Error: network mynetwork has active endpoints — if containers are attached

# Remove containers first, then the network
docker rm -f $(docker ps -q --filter network=mynetwork)
docker network rm mynetwork

# Remove multiple networks at once
docker network rm net1 net2 net3

# Remove all unused networks (none with connected containers)
docker network prune

# Skip confirmation prompt
docker network prune -f

# Filter by label before pruning
docker network prune --filter label=env=staging

Caution: docker network prune — Be Careful! This command deletes all "unused" networks — meaning any network not connected to at least one container. If you created a network for future use, it will be deleted too. Always run docker network ls before running prune in production.

Container-to-Container Communication

Real-World Example: MERN Stack Communication

Consider deploying a MERN (MongoDB, Express, React, Node) application. All three services need to communicate with each other. Here is how to set this up in Docker:

# Step 1: Create a dedicated network
docker network create mern-network

# Step 2: Start MongoDB first (others depend on it)
docker run -d \
  --name mongodb \
  --network mern-network \
  -v mongo_data:/data/db \
  mongo:7

# Step 3: Start the Node.js API
# Use "mongodb" as the hostname — Docker DNS resolves it!
docker run -d \
  --name api \
  --network mern-network \
  -e MONGODB_URI=mongodb://mongodb:27017/mydb \
  -e PORT=5000 \
  -p 5000:5000 \
  myapi:latest

# Step 4: Start the React frontend
# Use "api" as the hostname
docker run -d \
  --name frontend \
  --network mern-network \
  -e REACT_APP_API_URL=http://api:5000 \
  -p 3000:3000 \
  myfrontend:latest

# Communication flow:
# Browser → Host:3000 → frontend container
# frontend → api:5000 (DNS resolution via Docker) → api container
# api → mongodb:27017 (DNS resolution via Docker) → mongodb container

Testing DNS Resolution

# Test connectivity
docker exec frontend ping api -c 4
# PING api (172.20.0.3): 56 data bytes
# 64 bytes from 172.20.0.3: icmp_seq=0 ttl=64 time=0.089 ms

# DNS lookup
docker exec api nslookup mongodb
# Server:    127.0.0.11  <- Docker's built-in DNS server!
# Address 1: 127.0.0.11
# Name:      mongodb
# Address 1: 172.20.0.2 mongodb.mern-network

# Test HTTP connection
docker exec api wget -qO- http://mongodb:27017
# It looks like you are trying to access MongoDB over HTTP...
# (Connection succeeded — MongoDB responded)

# Test the API from the frontend using curl
docker exec frontend curl http://api:5000/health
# {"status":"ok","db":"connected"}

Note: 127.0.0.11 — Docker's Secret DNS Server. Whenever you use a user-defined network, Docker provides an embedded DNS server at 127.0.0.11 inside every container. It resolves container names to IP addresses. This happens automatically — no configuration required.

Service Discovery Patterns — Best Practices

Pattern	Example	When to Use
Container Name as Hostname	`mongodb://mongodb:27017`	Simple setups, same host
Docker Compose Service Name	`http://api:5000`	Compose-based deployments
Network Alias	`--network-alias db`	When the container name may change
Environment Variables	`DB_HOST=mongodb`	Externalizing configuration

Caution: Never Hardcode IPs! A very common mistake: MONGODB_URI=mongodb://172.17.0.2:27017. The IP can change when the container restarts. Always use the container name as the hostname: mongodb://mongodb:27017. The container name is stable; the IP is not.

Note: Production Pattern: Network Segmentation. In real production environments, create separate networks: frontend-net (frontend + API) and backend-net (API + database). The database will not be directly accessible from the frontend — only through the API. This is the Zero Trust networking principle. In Docker Compose, this is easily configured with the networks: key.

Port Management — EXPOSE vs -p vs -P

EXPOSE vs -p — Clearing the Confusion

This is a classic point of confusion for beginners. Both EXPOSE and -p deal with ports, but they serve different purposes.

# EXPOSE in Dockerfile — documentation only!
EXPOSE 3000
# This only says "this container listens on port 3000"
# It does NOT publish anything to the HOST
# Other containers can read this information via inspect

# -p flag: ACTUALLY publishes the port on the host
docker run -p 8080:3000 myapp
# host:8080 → container:3000
# Requests arriving at host port 8080 are forwarded to container port 3000

# -P flag (capital P): publish ALL EXPOSEd ports to random host ports
docker run -P myapp
# If Dockerfile has EXPOSE 3000 3306:
# Docker allocates: 32768:3000, 32769:3306 (random!)

# Check which port was assigned
docker port myapp
# 3000/tcp -> 0.0.0.0:32768

docker port myapp 3000
# 0.0.0.0:32768

Tip: Mnemonic: EXPOSE = Advertisement, -p = Actually Open the Door. EXPOSE is a "menu card" — it only describes what is available. The -p flag actually opens the door to the outside world. Without -p, services inside the container are not accessible from the host (though other containers on a custom network can still access them without port mapping).

Port Mapping Patterns

# 1. Specific mapping (most common)
docker run -p 8080:80 nginx          # Host 8080 → Container 80

# 2. Multiple ports
docker run -p 80:80 -p 443:443 nginx

# 3. Default binding: 0.0.0.0 (all interfaces)
docker run -p 8080:80 nginx
# Any IP can reach port 8080 and hit the container
# curl localhost:8080 works
# curl 192.168.1.100:8080 works (accessible on LAN too)

# 4. Bind to localhost only (security)
docker run -p 127.0.0.1:8080:80 nginx
# Only accessible from the same machine
# curl localhost:8080 works
# curl 192.168.1.100:8080 does not work

# 5. Random port on all interfaces
docker run -p 0.0.0.0::80 nginx
# Docker chooses a random high port
docker port <container> 80
# 0.0.0.0:49153

# 6. UDP port
docker run -p 53:53/udp my-dns-server

# 7. Range of ports
docker run -p 8000-8010:8000-8010 myapp

Handling Port Conflicts

# Error: Bind: address already in use
# docker: Error response from daemon: driver failed programming
# external connectivity on endpoint ...: Bind for 0.0.0.0:8080
# failed: port is already allocated

# Step 1: Find out who is using port 8080
sudo lsof -i :8080
sudo netstat -tulpn | grep 8080

# Step 2: Options:
# Option A: Use a different host port
docker run -p 9090:80 nginx

# Option B: Stop the conflicting process
sudo kill -9 <PID>

# Option C: Check if another Docker container is using the port
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep 8080
docker stop old-nginx
docker run -p 8080:80 nginx

# Useful: view all mapped ports
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"

Caution: Security: Bind to Localhost in Development. When you use -p 8080:80, Docker automatically adds an iptables rule — even if UFW or firewalld is disabled. This means the service may be accessible from the internet. In development, always use -p 127.0.0.1:8080:80 for sensitive services.

Note: EXPOSE vs -p — Interview Answer. EXPOSE is only metadata/documentation — it does not actually publish any port. The -p flag creates an actual port mapping at runtime between the host and container using iptables rules. EXPOSE without -p is useless for host access (though it does provide information to other containers within the network).

Storage Fundamentals — Why Persistence Matters

Containers Are Ephemeral — Data Is Lost!

The most important concept in storage: containers are ephemeral (temporary) by default. When a container is deleted, all its data is deleted too.

Caution: "The container was deleted and the data went with it!" — Classic Production Disaster. Imagine you started a MySQL container, added some data, and then accidentally ran docker rm -f mysql. The entire database is gone. This happens frequently in production. Solution: always use volumes for databases.

Container storage layers use a union filesystem (overlay2):

┌─────────────────────────────────────────┐
│  [Writable Layer]  ← Temporary!         │
│  The container's write area             │
│  Container deleted → THIS DATA IS GONE! │
├─────────────────────────────────────────┤
│  [App Layer]       ← Read-only          │
│  Your application code                  │
├─────────────────────────────────────────┤
│  [Base Image Layer] ← Read-only         │
│  ubuntu:22.04 or node:18 etc.           │
└─────────────────────────────────────────┘

# The bottom layers are part of the image — shared across containers!
# Only the writable layer is container-specific — and it is temporary

Storage Drivers — The Underlying Filesystem

Driver	Description	When to Use
overlay2	Default; uses Linux overlayfs	Almost always — best choice
devicemapper	Block device based	Legacy RHEL/CentOS (outdated)
btrfs	Btrfs filesystem	Only on Btrfs hosts
zfs	ZFS filesystem	Only on ZFS hosts
vfs	No layering, simple copy	Testing only — very slow

# Check the current storage driver
docker info | grep "Storage Driver"
# Storage Driver: overlay2

# Where does Docker store data?
/var/lib/docker/
  ├── containers/     # Per-container metadata, logs
  ├── image/          # Image metadata and manifests
  ├── overlay2/       # Image layers (most data is here!)
  ├── volumes/        # Named volumes
  └── network/        # Network configs

# Check sizes
sudo du -sh /var/lib/docker/*
# 1.2G    /var/lib/docker/overlay2   # Most data here!
# 200M    /var/lib/docker/volumes
# 50M     /var/lib/docker/containers

# Docker summary
docker system df
# TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
# Images          15        5         3.2GB     2.1GB (65%)
# Containers      8         3         120MB     90MB (75%)
# Local Volumes   5         2         1.1GB     500MB (45%)

# Detailed breakdown
docker system df -v

Tip: Copy-on-Write (CoW) — The Secret of Efficiency. overlay2 uses a Copy-on-Write strategy. If 10 containers use the same base image, the image layers are stored only once on disk. Only when a container writes something does it create its own copy of that file. This saves both disk space and startup time.

Note: Production Rule: Databases Always Need Volumes! MySQL, PostgreSQL, MongoDB, Redis — whenever you run any database in a container, volumes are MANDATORY. Without a volume, all data is permanently lost when the container crashes or restarts. This is the number one cause of production disasters.

Volume Types — Named, Bind Mount, tmpfs

Named Volumes — Production's Best Friend

Named volumes are Docker's managed volumes — Docker decides where to store the data (/var/lib/docker/volumes/). They survive container restarts and deletions.

# Create a volume
docker volume create mydata

# Use with -v syntax
docker run -v mydata:/app/data myapp

# Use with --mount syntax (recommended)
docker run --mount type=volume,source=mydata,target=/app/data myapp

# Use with a database
docker run -d \
  --name postgres \
  -v postgres_data:/var/lib/postgresql/data \
  postgres:16

# Delete the container — THE VOLUME STAYS SAFE!
docker rm -f postgres
docker volume ls
# postgres_data  ← Still there!

# Start a new container with the same volume — data is back!
docker run -d --name postgres-new -v postgres_data:/var/lib/postgresql/data postgres:16

Bind Mounts — Development's Superpower

With bind mounts, you specify the host path explicitly. This is extremely useful in development because your local changes are instantly reflected inside the container — without rebuilding!

# Specify the host path explicitly
docker run -v /home/pushkar/myproject:/app node:18

# Mount the current directory (most common in development!)
docker run -v $(pwd):/app node:18
# Edit code locally — changes appear inside the container immediately!
# Hot reload works!

# Node.js development example
docker run -d \
  --name dev-server \
  -v $(pwd):/app \
  -w /app \
  -p 3000:3000 \
  node:18 \
  npm run dev

# Read-only bind mount (for security — container cannot modify)
docker run -v /etc/nginx/conf:/etc/nginx/conf:ro nginx

# --mount syntax
docker run --mount type=bind,source=$(pwd),target=/app node:18
docker run --mount type=bind,source=/config,target=/config,readonly nginx

Tip: For React Developers: Bind Mount = Live Reload! docker run -v $(pwd):/app -p 3000:3000 node:18 npm start — edit code in VS Code and React hot reload fires automatically inside the container. No rebuild needed for each change. This is a game-changer for development workflow.

tmpfs Mounts — Sensitive Data In-Memory

# Stored in memory, never written to disk!
docker run --tmpfs /tmp nginx

# With a size limit
docker run --mount type=tmpfs,target=/tmp,tmpfs-size=100m myapp

# Multiple tmpfs locations
docker run \
  --tmpfs /tmp \
  --tmpfs /run \
  myapp

# Use cases:
# - Session data (passwords, tokens stored temporarily)
# - Sensitive computation results
# - Cache data that should never touch disk
# - High-speed temporary storage

# Container stops → data gone immediately (from RAM)
# Container crashes → data gone immediately
# Never persists to disk = more secure

Comparison — Which Type to Use When?

Feature	Named Volume	Bind Mount	tmpfs
Storage Location	/var/lib/docker/volumes	Any host path	Memory
Docker Managed	Yes	No	N/A
Data Persists	Yes	Yes	No (memory only)
Dev Workflow	OK	Best (hot reload!)	N/A
Production DB	Best	OK (path management)	Never
Easy Backup	Yes	Yes	No
Performance	Good	Best (NFS will be slow)	Excellent
Security	Container isolated	Host exposed	Most secure

Note: Named Volume vs Bind Mount — Interview Answer. Named Volume: Docker manages the location; it is portable, easy to back up, and best for production. Bind Mount: You control the specific host path; perfect for development (host files accessible directly in container), but path is host-specific — less portable. Prefer named volumes in production; use bind mounts in development for code changes.

Volume Commands — Complete Reference

docker volume create — Creating Volumes

# Simple named volume
docker volume create mydb-data

# Explicitly use the local driver (this is already the default)
docker volume create --driver local mydb-data

# NFS-backed volume (remote storage!)
docker volume create \
  --driver local \
  --opt type=nfs \
  --opt o=addr=192.168.1.100,rw,vers=4 \
  --opt device=:/exports/mydata \
  nfs-volume

# With labels (for management)
docker volume create \
  --label env=production \
  --label app=myapp \
  --label team=backend \
  mydb-data

# Volumes are automatically created if referenced in a run command
docker run -v newvolume:/data myapp
# "newvolume" is created automatically if it does not exist

docker volume ls, inspect, rm, prune

# List all volumes
docker volume ls
# DRIVER    VOLUME NAME
# local     mydb-data
# local     postgres_data
# local     mongo_data

# Find dangling volumes (not connected to any container)
docker volume ls -f dangling=true

# Filter by label
docker volume ls --filter label=env=production

# Inspect — full details
docker volume inspect mydb-data
# [
#   {
#     "Name": "mydb-data",
#     "Driver": "local",
#     "Mountpoint": "/var/lib/docker/volumes/mydb-data/_data",
#     "Labels": {"env": "production"},
#     ...
#   }
# ]

# Get only the mount path
docker volume inspect --format '{{.Mountpoint}}' mydb-data
# /var/lib/docker/volumes/mydb-data/_data

# Remove — stop the container first!
docker volume rm mydb-data
# Error: volume is in use — if a container is running

# Correct approach: remove the container first
docker rm -f mycontainer
docker volume rm mydb-data

# Remove multiple at once
# (Remove containers first, then volumes)
docker volume prune

# Skip confirmation
docker volume prune -f

# Filter by label before pruning
docker volume prune --filter label=env=staging

--mount vs -v Syntax — The Modern Way

# ─── OLD -v SYNTAX ──────────────────────────────────────
# Older style, still works, but less readable

# Named volume
docker run -v myvolume:/app/data myapp

# Bind mount
docker run -v /host/path:/container/path myapp

# Read-only bind mount
docker run -v /host/path:/container/path:ro myapp

# ─── NEW --mount SYNTAX ──────────────────────────────────
# Newer, explicit, recommended

# Named volume
docker run --mount type=volume,source=myvolume,target=/app/data myapp

# Bind mount
docker run --mount type=bind,source=/host/path,target=/app/path myapp

# Read-only bind mount
docker run --mount type=bind,source=/host/path,target=/app/path,readonly myapp

# tmpfs
docker run --mount type=tmpfs,target=/tmp,tmpfs-size=100m myapp

# Why --mount is better:
# - More explicit — type is clearly stated
# - Better error messages
# - Supports all options (tmpfs-size, etc.)
# - Recommended format in Docker Docs
# - More readable in scripts
# - Named parameters catch typos

Note: -v vs --mount: One Important Difference. A subtle distinction: -v /nonexistent:/data will cause Docker to automatically create the host directory (sometimes with the wrong type). --mount type=bind,source=/nonexistent,target=/data returns an error if the source does not exist. --mount is stricter and more predictable — which makes it better for production.

Caution: docker volume prune — Be Very Careful in Production! This command deletes "dangling" (unused) volumes. However, if you have temporarily stopped a container (intending to restore its data), its volume may be deleted too. In production, first run docker volume ls -f dangling=true to see which volumes are actually unused, then selectively use docker volume rm.

Production Volume Patterns — Real-World Usage

Database Persistence Patterns

PostgreSQL with Named Volume

# Step 1: Create the volume
docker volume create postgres_data

# Step 2: Run PostgreSQL with a proper production setup
docker run -d \
  --name postgres \
  -e POSTGRES_USER=admin \
  -e POSTGRES_PASSWORD=securepassword123 \
  -e POSTGRES_DB=myapp \
  -v postgres_data:/var/lib/postgresql/data \
  -p 127.0.0.1:5432:5432 \
  --restart unless-stopped \
  --health-cmd="pg_isready -U admin" \
  --health-interval=10s \
  postgres:16-alpine

# Verify
docker ps                                         # Check status and health
docker exec postgres psql -U admin -c "\l"        # List databases

MongoDB with Multiple Volumes

# MongoDB has two important paths:
# /data/db       — actual database files
# /data/configdb — configuration data

docker run -d \
  --name mongodb \
  -v mongo_data:/data/db \
  -v mongo_config:/data/configdb \
  -e MONGO_INITDB_ROOT_USERNAME=admin \
  -e MONGO_INITDB_ROOT_PASSWORD=securepass \
  -p 127.0.0.1:27017:27017 \
  --restart unless-stopped \
  mongo:7

# Verify
docker exec mongodb mongosh -u admin -p securepass --eval "db.adminCommand('listDatabases')"

Backup Strategies

# ─── VOLUME BACKUP (any type of data) ─────────────────────
# Use a temporary container — read the volume, create a tar archive
docker run --rm \
  -v mydb_data:/data \
  -v $(pwd)/backups:/backup \
  alpine \
  tar czf /backup/mydb-$(date +%Y%m%d-%H%M%S).tar.gz /data

# Creates a file in /backup: mydb-20240115-142030.tar.gz

# ─── VOLUME RESTORE ────────────────────────────────────────
docker run --rm \
  -v mydb_data:/data \
  -v $(pwd)/backups:/backup \
  alpine \
  tar xzf /backup/mydb-20240115-142030.tar.gz -C /

# ─── POSTGRES DUMP (recommended for PostgreSQL) ────────────
# Logical backup — portable and version-aware
docker exec postgres pg_dump -U admin myapp > backup-$(date +%Y%m%d).sql

# Restore
docker exec -i postgres psql -U admin myapp < backup-20240115.sql

# Compressed backup
docker exec postgres pg_dump -U admin myapp | gzip > backup.sql.gz

# Restore compressed
gunzip -c backup.sql.gz | docker exec -i postgres psql -U admin myapp

# ─── MONGODB DUMP ──────────────────────────────────────────
docker exec mongodb mongodump -u admin -p securepass --out /tmp/backup
docker cp mongodb:/tmp/backup ./mongodb-backup

# Restore
docker cp ./mongodb-backup mongodb:/tmp/backup
docker exec mongodb mongorestore -u admin -p securepass /tmp/backup

Shared Volumes Between Containers

# Pattern: one container writes, another reads
# No network communication required between them!
docker volume create shared-logs

# App container — writes logs
docker run -d \
  --name app1 \
  -v shared-logs:/var/log/app \
  myapp:latest

# Log viewer — read-only access to the same volume
docker run -d \
  --name log-viewer \
  -v shared-logs:/logs:ro \
  log-viewer-app:latest

# log-viewer can now read all of app1's logs
# No network communication needed!

# Real use case: Nginx serving React static files
docker volume create static-files

# Build container: build React, copy to volume
docker run --rm \
  -v static-files:/app/build \
  react-builder:latest

# Nginx container: serve files from the same volume
docker run -d \
  --name nginx \
  -v static-files:/usr/share/nginx/html:ro \
  -p 80:80 \
  nginx:alpine

Cloud Volume Drivers — AWS EBS, Azure Disk

# AWS EBS-backed volumes (rexray plugin)
# If the host is replaced, data remains safe on AWS EBS!

# Install the plugin
docker plugin install rexray/ebs \
  EBS_ACCESSKEY=AKIAIOSFODNN7EXAMPLE \
  EBS_SECRETKEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Create an EBS-backed volume
docker volume create \
  --driver rexray/ebs \
  --opt size=20 \
  --opt type=gp3 \
  --opt iops=3000 \
  myapp-data

# Use it — exactly like a regular volume
docker run -d \
  --name postgres \
  -v myapp-data:/var/lib/postgresql/data \
  postgres:16

# If the host crashes, detach and reattach the EBS volume to a new host
# Data is fully safe on AWS!

# For Azure: the cloudstor plugin is available
# For GCP: use the gcepd driver

Note: Production Reality: Cloud Volumes = True Persistence. Local named volumes are tied to the Docker host — if the host machine is lost, the volumes are gone (EBS is not). In real production, use AWS EFS/EBS, Azure Managed Disks, or GCP Persistent Disk via volume drivers. Or better yet, use managed databases (RDS, Atlas) — that is the preferred approach.

Tip: Volume Best Practices Summary.

Production databases: Named volumes or cloud-backed volumes (prefer RDS)

Development: Bind mounts for code, named volumes for databases

Secrets/session data: tmpfs

Backups: Automated cron + pg_dump/mongodump stored on separate storage

Never: Store critical data only in the container's writable layer

Also make sure your .env files are correctly structured — you can validate them with the .env Example Checker before deploying. Once your stack is wired up, put these skills to work with real scenarios in Hands-on DevOps Projects.

Docker Compose Introduction

One Compose file declares the full application stack; docker compose up creates all services on a shared network with automatic DNS so containers resolve each other by service name.

Why Docker Compose Exists

Real-world applications do not fit in a single container. A MERN stack application requires:

MongoDB — database
Express/Node — backend API
React — frontend (served via Nginx)
Redis — caching / sessions

Without Compose, everything must be managed manually:

# Without Compose — run all of this by hand!
docker network create myapp-net

docker run -d \
  --name mongodb \
  --network myapp-net \
  -e MONGO_INITDB_ROOT_USERNAME=admin \
  -e MONGO_INITDB_ROOT_PASSWORD=secret \
  -v mongo_data:/data/db \
  mongo:7.0

docker run -d \
  --name api \
  --network myapp-net \
  -e MONGODB_URI=mongodb://admin:secret@mongodb:27017/myapp \
  -p 5000:5000 \
  myapp-api:latest

docker run -d \
  --name frontend \
  --network myapp-net \
  -p 3000:80 \
  myapp-frontend:latest

# Repeat all of this on every restart...

Tip: Just as package.json lists all npm dependencies and npm install sets everything up — docker-compose.yml defines all your containers and docker compose up starts everything in one command.

With Compose, the same setup becomes:

# With Compose — just this!
docker compose up -d

# To shut everything down:
docker compose down

Convert an existing command with the Docker run → Compose converter.

Compose v1 vs Compose v2 — What Is the Difference?

Feature	Compose v1 (Old)	Compose v2 (Current)
Command	`docker-compose` (with dash)	`docker compose` (no dash!)
Installation	Separate Python binary	Built-in Docker plugin
Status	Deprecated (2023)	Active, standard
Performance	Slower	Faster (Go-based)
Features	Limited	Full feature set
2026 standard	Do not use	Use this!

Caution: Use docker compose (no dash) — this is Compose v2 and the current standard. If an older tutorial uses docker-compose (with a dash), it is using v1 syntax. V2 is mostly backward compatible, but make it a habit to use v2.

# Check version — should show v2.x.x
docker compose version
# Output: Docker Compose version v2.24.5

# Old v1 (deprecated — do not use!)
docker-compose version

# If the Compose plugin is missing:
# Ubuntu/Debian
sudo apt update && sudo apt install docker-compose-plugin

# RHEL/CentOS/Fedora
sudo dnf install docker-compose-plugin

# macOS (auto-bundled with Docker Desktop)
# Windows (auto-bundled with Docker Desktop)

Compose vs Kubernetes — When to Use Which?

Use Case	Compose	Kubernetes
Local development	Perfect choice	Overkill
Small apps (< 5 containers)	Yes	Overkill
Single server deployment	Yes	Not needed
Auto-scaling	No	Yes
Multi-host clusters	No	Yes
Zero-downtime deploys	Limited	Yes
Production at scale	Limited	Yes
Learning curve	Easy (YAML)	Steep
Startup / small teams	Best fit	Overkill initially

Note: Start with Compose. Run a MERN stack locally using Compose; Compose is sufficient for a single VPS in production too. When traffic grows and auto-scaling becomes necessary, then invest in learning Kubernetes. The progression is: Docker → Compose → Kubernetes.

Compose File Structure

Annotated docker-compose.yml Structure

Understanding the full structure allows you to read any Compose file with ease:

# docker-compose.yml
# In Compose v2, the version field is optional/deprecated
# version: '3.8'  ← no longer required!

services:          # All containers are defined here
  web:             # Service name (also becomes the DNS hostname!)
    image: nginx   # Which image to use
    # --- OR ---
    build: .       # Build from a local Dockerfile

  api:             # Another service
    build:
      context: ./api               # Build context folder
      dockerfile: Dockerfile.prod  # Specific Dockerfile name

  db:              # Database service
    image: postgres:16

networks:          # Define custom Docker networks
  frontend:        # Frontend network (public-facing)
  backend:         # Backend network (private, for DB communication)

volumes:           # Named volumes (data persistence)
  db_data:         # Database files stored here
  redis_data:      # Redis data persistence

configs:           # Inject config files
  nginx_config:
    file: ./nginx.conf

secrets:           # Sensitive data (mainly for Swarm mode)
  db_password:
    file: ./secrets/db_password.txt

Tip: Service Name = DNS Hostname! In Compose, the service name automatically becomes the DNS hostname. If a service is named db, other containers can connect to it via db:5432 — not localhost or an IP. Compose handles this automatically via Docker networking.

YAML Best Practices in Compose

# Use quotes for strings containing special characters
environment:
  - DATABASE_URL="postgresql://user:pass@db:5432/mydb"

# Use literal block (|) for multiline commands
command: |
  sh -c "
    until pg_isready; do sleep 1; done
    python manage.py migrate
    python manage.py runserver 0.0.0.0:8000
  "

# YAML Anchors — avoid repetition (DRY principle)
x-common-env: &common-env      # &common-env = define the anchor
  NODE_ENV: production
  LOG_LEVEL: info
  TZ: Asia/Kolkata

services:
  api:
    environment:
      <<: *common-env          # *common-env = use the anchor (merge)
      PORT: 3000
      SERVICE_NAME: api

  worker:
    environment:
      <<: *common-env          # Reuse the same env vars
      WORKER_TYPE: email
      CONCURRENCY: "4"

Note: YAML Anchors — DRY Principle. YAML anchors (&name) and aliases (*name) let you write common configuration once and reuse it across multiple services. <<: *common-env means "merge all key-value pairs here." This is the same as the spread operator (...commonProps) in JavaScript.

Override Files Pattern — Multiple Environments

One codebase, different configurations per environment — this pattern is used in production-grade applications:

# File structure:
docker-compose.yml              # Base config (common settings)
docker-compose.override.yml     # Dev overrides — AUTOMATICALLY loaded!
docker-compose.dev.yml          # Development specifics
docker-compose.prod.yml         # Production specifics
docker-compose.test.yml         # Testing / CI specifics

# Development (auto-loads base + override):
docker compose up -d

# Production (specify files explicitly):
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Testing:
docker compose -f docker-compose.yml -f docker-compose.test.yml up -d

# Validate the final merged config:
docker compose -f docker-compose.yml -f docker-compose.prod.yml config

# docker-compose.yml (base)
services:
  api:
    image: myapp-api:latest
    environment:
      NODE_ENV: production

---
# docker-compose.override.yml (dev — auto-loaded!)
services:
  api:
    build: .                    # Build locally in dev
    volumes:
      - ./src:/app/src          # Bind mount for hot reload
    environment:
      NODE_ENV: development
      DEBUG: "true"
    ports:
      - "5000:5000"             # Expose port directly in dev

---
# docker-compose.prod.yml
services:
  api:
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 512M
    logging:
      driver: json-file
      options:
        max-size: "10m"

Tip: Best Practice: Override Files. Keep common configuration in the base docker-compose.yml. Put environment-specific settings in override files. docker-compose.override.yml loads automatically for development — place bind mounts, debug ports, and dev-only tools there. The production file should contain resource limits, proper restart policies, and logging configuration.

Service Configuration Deep Dive

Complete Service Configuration Reference

All the options available in a service, annotated:

services:
  myapp:
    # ──────────────────────────────────────────────
    # IMAGE vs BUILD — choose one
    # ──────────────────────────────────────────────
    # Option 1: Use a pre-built image
    image: nginx:1.25-alpine

    # Option 2: Build from a local Dockerfile
    build:
      context: .                    # Build context (folder with Dockerfile)
      dockerfile: Dockerfile        # Which Dockerfile? (default: Dockerfile)
      args:                         # Build-time arguments (ARG in Dockerfile)
        NODE_ENV: production
        APP_VERSION: "1.0.0"
      target: production            # Specific stage in a multi-stage build

    # ──────────────────────────────────────────────
    # CONTAINER NAME
    # ──────────────────────────────────────────────
    container_name: my-nginx
    # Warning: setting container_name prevents scaling!
    # docker compose up --scale myapp=3 will fail if container_name is set

    # ──────────────────────────────────────────────
    # PORTS — host:container
    # ──────────────────────────────────────────────
    ports:
      - "8080:80"                   # host 8080 → container 80
      - "443:443"                   # HTTPS
      - "127.0.0.1:9090:9090"       # Localhost only (secure!)

    # ──────────────────────────────────────────────
    # ENVIRONMENT VARIABLES
    # ──────────────────────────────────────────────
    environment:
      NODE_ENV: production
      DB_HOST: db                   # Use the service name — not localhost!
      DB_PORT: "5432"               # Use quotes for numeric values
      DB_NAME: mydb

    # Load from env_file
    env_file:
      - .env                        # Main .env file
      - .env.production             # Production-specific variables

    # ──────────────────────────────────────────────
    # VOLUMES — data persistence and bind mounts
    # ──────────────────────────────────────────────
    volumes:
      - ./src:/app/src:ro           # Bind mount, read-only (dev hot reload)
      - db_data:/var/lib/postgresql/data  # Named volume (production data)
      - /etc/localtime:/etc/localtime:ro  # Host timezone inside container

    # ──────────────────────────────────────────────
    # NETWORKS
    # ──────────────────────────────────────────────
    networks:
      - frontend                    # Public-facing network
      - backend                     # Private network (for DB communication)

    # ──────────────────────────────────────────────
    # DEPENDS_ON — startup order
    # ──────────────────────────────────────────────
    depends_on:
      db:
        condition: service_healthy  # Wait until db is healthy
      redis:
        condition: service_started  # Just started, not necessarily healthy

    # ──────────────────────────────────────────────
    # RESTART POLICY
    # ──────────────────────────────────────────────
    restart: unless-stopped
    # Options:
    # no           — do not restart (default)
    # always       — always restart
    # on-failure   — restart only on failure
    # unless-stopped — always restart unless manually stopped

    # ──────────────────────────────────────────────
    # COMMAND and ENTRYPOINT
    # ──────────────────────────────────────────────
    command: npm start              # String form
    command: ["node", "server.js"]  # Array form (preferred — shell injection safe)

    entrypoint: ["./docker-entrypoint.sh"]

    # ──────────────────────────────────────────────
    # HEALTHCHECK
    # ──────────────────────────────────────────────
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s       # Check every 30 seconds
      timeout: 10s        # Response must arrive within 10 seconds
      retries: 3          # Mark unhealthy after 3 failures
      start_period: 40s   # Wait this long after startup before checking

    # ──────────────────────────────────────────────
    # DEPLOY — resource limits
    # ──────────────────────────────────────────────
    deploy:
      resources:
        limits:
          cpus: "0.50"            # Max 50% of one CPU core
          memory: 512M            # Max 512 MB RAM
        reservations:
          cpus: "0.25"            # Minimum guaranteed CPU
          memory: 256M            # Minimum guaranteed RAM

    # ──────────────────────────────────────────────
    # PROFILES — optional services
    # ──────────────────────────────────────────────
    profiles:
      - debug   # Only starts with: docker compose --profile debug up

    # ──────────────────────────────────────────────
    # LOGGING
    # ──────────────────────────────────────────────
    logging:
      driver: json-file
      options:
        max-size: "10m"     # Log file max 10 MB
        max-file: "3"       # Max 3 log files (then rotate)

    # ──────────────────────────────────────────────
    # EXTRA HOSTS — /etc/hosts entries
    # ──────────────────────────────────────────────
    extra_hosts:
      - "host.docker.internal:host-gateway"  # Connect to host machine

    # ──────────────────────────────────────────────
    # DNS — custom DNS servers
    # ──────────────────────────────────────────────
    dns:
      - 8.8.8.8
      - 8.8.4.4

depends_on and Healthchecks — A Common Gotcha

Caution: The Default depends_on Behavior Is Misleading! A simple depends_on: [db] only guarantees that the db container has started — not that it is ready. PostgreSQL takes 5–10 seconds to initialize. If the api service tries to connect immediately, the connection will fail. Solution: use condition: service_healthy and define a healthcheck on the db service.

services:
  db:
    image: postgres:16-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myuser -d mydb"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s    # Give PostgreSQL time to initialize

  api:
    build: .
    depends_on:
      db:
        condition: service_healthy   # Wait until HEALTHY, not just STARTED!

Tip: Profiles — Make Dev Tools Optional. Using profiles, you can make certain services optional. For example, pgadmin (a DB GUI) is only needed in development — give it profiles: [debug]. It will not start during a normal docker compose up. It will only start with docker compose --profile debug up. This prevents debug tools from accidentally running in production.

Docker Compose Commands — Complete Reference

docker compose up — Start Everything

# Basic — start in foreground (Ctrl+C to stop)
docker compose up

# Detached — start in background (for production)
docker compose up -d

# Rebuild images and start (after code changes)
docker compose up --build

# Force recreate containers (even if nothing changed)
docker compose up --force-recreate

# Start only specific services
docker compose up -d api db

# Run multiple instances (for load balancing)
docker compose up -d --scale api=3
# Note: container_name must not be set when scaling!

# Start with a specific profile
docker compose --profile debug up -d

# Use a custom .env file
docker compose --env-file .env.prod up -d

# Remove orphan containers (not defined in the compose file)
docker compose up -d --remove-orphans

docker compose down — Stop and Clean Up

# Standard — remove containers and networks
docker compose down

# Also remove volumes (DATA LOSS — be careful!)
docker compose down -v

# Also remove images
docker compose down --rmi all     # All images
docker compose down --rmi local   # Only locally built images

# Stop but keep containers (do not remove)
docker compose stop

# Complete teardown — clean up everything
docker compose down -v --rmi all --remove-orphans

Caution: docker compose down -v — Data Will Be Deleted! The -v flag removes all named volumes. This means all database data is gone. This is useful for a clean restart in development, but never use it carelessly in production. Take a backup first.

docker compose logs — Debugging

# Logs from all services
docker compose logs

# Follow (live stream)
docker compose logs -f

# Logs from a specific service
docker compose logs -f api

# Last 100 lines
docker compose logs --tail 100

# With timestamps
docker compose logs -t

# Multiple services at once
docker compose logs -f api db

# Last 1 hour
docker compose logs --since 1h

# From a specific time
docker compose logs --since "2024-01-15T10:00:00"

docker compose exec vs run — What Is the Difference?

Feature	exec	run
Container	Runs inside an existing running container	Starts a new container
Use case	Debug a running service	One-off commands (migrations, etc.)
Service running?	Must be running	Service does not need to be running
Cleanup	Container remains as-is	Auto-cleanup with `--rm`

# ── docker compose exec ──────────────────────────────
# Run a command in a running container
docker compose exec api bash
docker compose exec db psql -U admin -d mydb
docker compose exec redis redis-cli

# Without TTY (for automation/scripts)
docker compose exec -T api npm test

# As a specific user
docker compose exec --user root api bash

# ── docker compose run ──────────────────────────────
# One-off command in a new container
docker compose run api npm run migrate
docker compose run api python manage.py createsuperuser

# Auto-cleanup after run
docker compose run --rm api npm test

# Override the entrypoint
docker compose run --entrypoint bash api

# With custom env vars
docker compose run -e DEBUG=true api npm start

Tip: exec vs run — Simple Rule. Need to do something in a running service (debug, inspect)? → exec. Need to run a one-off task alongside the service (DB migration, seed data, test run)? → run --rm. docker compose run --rm web python manage.py migrate is the standard pattern for Django/Rails migrations.

Other Useful Commands

# List services
docker compose ps
docker compose ps -a             # Include stopped services
docker compose ps --format json  # JSON output

# Build only (do not start)
docker compose build
docker compose build --no-cache api      # Ignore cache
docker compose build --parallel          # Build all in parallel

# Pull latest images
docker compose pull

# Push images to registry
docker compose push

# Validate the compose file (shows final merged config)
docker compose config

# Show running processes
docker compose top

# Restart services
docker compose restart
docker compose restart api

# Pause/Unpause (stop CPU usage, keep memory)
docker compose pause
docker compose unpause

# Create without starting
docker compose create

# Stream events
docker compose events

Tip: Validate Your Compose File. docker compose config is a very useful command that merges all override files and shows the final YAML. Catch typos, missing variables, and syntax errors before deploying. Use it in CI/CD too: docker compose -f docker-compose.yml -f docker-compose.prod.yml config --quiet.

Production Compose Examples — Real-World Stacks

Example 1: MERN Stack (React + Node + MongoDB)

Most relevant for React developers — a complete MERN stack with Compose:

# docker-compose.yml — MERN Stack
# Usage: docker compose up -d
# Variables are loaded from a .env file

services:
  # ────────────────────────────────
  # MongoDB Database
  # ────────────────────────────────
  mongodb:
    image: mongo:7.0-jammy
    container_name: mern-mongodb
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_USER:-admin}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD:-password}
      MONGO_INITDB_DATABASE: ${MONGO_DB:-myapp}
    volumes:
      - mongo_data:/data/db
      - mongo_config:/data/configdb
      - ./mongo-init.js:/docker-entrypoint-initdb.d/init.js:ro
    networks:
      - backend          # Backend network only — no public access!
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

  # ────────────────────────────────
  # Node.js API Server
  # ────────────────────────────────
  api:
    build:
      context: ./api
      dockerfile: Dockerfile
      target: production    # Use the production stage of a multi-stage build
    container_name: mern-api
    environment:
      NODE_ENV: production
      MONGODB_URI: mongodb://${MONGO_USER:-admin}:${MONGO_PASSWORD:-password}@mongodb:27017/${MONGO_DB:-myapp}?authSource=admin
      JWT_SECRET: ${JWT_SECRET:-changeme-in-production}
      PORT: 5000
    volumes:
      - api_logs:/app/logs
    networks:
      - backend              # For MongoDB communication
      - frontend             # For receiving requests from React
    depends_on:
      mongodb:
        condition: service_healthy
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "node", "-e", "require('http').get('http://localhost:5000/health', r => process.exit(r.statusCode === 200 ? 0 : 1))"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
    deploy:
      resources:
        limits:
          cpus: "0.50"
          memory: 512M
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

  # ────────────────────────────────
  # React Frontend (served via Nginx)
  # ────────────────────────────────
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
      args:
        REACT_APP_API_URL: ${API_URL:-http://localhost:5000}
    container_name: mern-frontend
    ports:
      - "${FRONTEND_PORT:-3000}:80"
    networks:
      - frontend
    depends_on:
      api:
        condition: service_healthy
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost/health"]
      interval: 30s
      timeout: 5s
      retries: 3

# ────────────────────────────────
# Volumes
# ────────────────────────────────
volumes:
  mongo_data:
    driver: local
    labels:
      - "com.myapp.description=MongoDB data volume"
  mongo_config:
  api_logs:

# ────────────────────────────────
# Networks
# ────────────────────────────────
networks:
  frontend:
    driver: bridge             # Public traffic comes here
  backend:
    driver: bridge
    internal: true             # No external access! Container-to-container only

Note: Production Pattern: Internal Networks. Apply internal: true to the backend network. This means MongoDB is never directly accessible from the internet — only the api service can reach it. This is a security best practice. The flow is: Frontend → API (via frontend network), API → MongoDB (via backend network). MongoDB has no public port exposed.

Example 2: Python Flask + PostgreSQL + Redis + Celery

Full-stack Python application with background task processing:

# docker-compose.yml — Flask + Postgres + Redis + Celery

services:
  # ────────────────────────────────
  # PostgreSQL Database
  # ────────────────────────────────
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: ${DB_USER:-myuser}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-mypassword}
      POSTGRES_DB: ${DB_NAME:-mydb}
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./sql/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    networks:
      - backend
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-myuser} -d ${DB_NAME:-mydb}"]
      interval: 10s
      timeout: 5s
      retries: 5

  # ────────────────────────────────
  # Redis — Cache + Message Broker
  # ────────────────────────────────
  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD:-redis_secret}
    # appendonly yes = persist data even after restart
    volumes:
      - redis_data:/data
    networks:
      - backend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "--pass", "${REDIS_PASSWORD:-redis_secret}", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3

  # ────────────────────────────────
  # Flask Web Application
  # ────────────────────────────────
  web:
    build: .
    environment:
      DATABASE_URL: postgresql://${DB_USER:-myuser}:${DB_PASSWORD:-mypassword}@postgres:5432/${DB_NAME:-mydb}
      REDIS_URL: redis://:${REDIS_PASSWORD:-redis_secret}@redis:6379/0
      FLASK_ENV: ${FLASK_ENV:-production}
      SECRET_KEY: ${SECRET_KEY:-change-in-production}
    ports:
      - "${APP_PORT:-5000}:5000"
    volumes:
      - app_logs:/app/logs
      - ./uploads:/app/uploads   # Persist user uploads
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - backend
      - frontend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s

  # ────────────────────────────────
  # Celery Worker — Background Tasks
  # ────────────────────────────────
  celery_worker:
    build: .    # Same image as web!
    command: celery -A app.celery worker --loglevel=info --concurrency=4
    environment:
      DATABASE_URL: postgresql://${DB_USER:-myuser}:${DB_PASSWORD:-mypassword}@postgres:5432/${DB_NAME:-mydb}
      REDIS_URL: redis://:${REDIS_PASSWORD:-redis_secret}@redis:6379/0
    depends_on:
      - web
      - redis
    networks:
      - backend
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M

  # ────────────────────────────────
  # Celery Beat — Scheduled Tasks (Cron-like)
  # ────────────────────────────────
  celery_beat:
    build: .
    command: celery -A app.celery beat --loglevel=info
    environment:
      DATABASE_URL: postgresql://${DB_USER:-myuser}:${DB_PASSWORD:-mypassword}@postgres:5432/${DB_NAME:-mydb}
      REDIS_URL: redis://:${REDIS_PASSWORD:-redis_secret}@redis:6379/0
    depends_on:
      - web
    networks:
      - backend
    restart: unless-stopped
    # Beat should have only ONE instance — do not scale it!

  # ────────────────────────────────
  # Nginx Reverse Proxy
  # ────────────────────────────────
  nginx:
    image: nginx:1.25-alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - ./nginx/ssl:/etc/nginx/ssl:ro
    depends_on:
      - web
    networks:
      - frontend
    restart: unless-stopped

volumes:
  postgres_data:
  redis_data:
  app_logs:

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true    # DB and Redis must not be directly accessible!

Tip: Same Image, Different Command! Flask web, Celery worker, and Celery beat all use the same build: . (same Docker image) but with different command values. This is efficient: build one image, run different processes. When code changes, only one image needs to be rebuilt.

Example 3: WordPress + MySQL

Classic CMS setup — useful for client projects:

# docker-compose.yml — WordPress + MySQL + Nginx

services:
  # ────────────────────────────────
  # MySQL Database
  # ────────────────────────────────
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-rootpassword}
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: ${MYSQL_PASSWORD:-wppassword}
    volumes:
      - mysql_data:/var/lib/mysql
    networks:
      - wp-network
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "wordpress", "-pwppassword"]
      interval: 30s
      timeout: 10s
      retries: 5

  # ────────────────────────────────
  # WordPress (PHP-FPM)
  # ────────────────────────────────
  wordpress:
    image: wordpress:6.4-php8.2-fpm-alpine
    environment:
      WORDPRESS_DB_HOST: mysql:3306      # Use the service name!
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: ${MYSQL_PASSWORD:-wppassword}
      WORDPRESS_DB_NAME: wordpress
    volumes:
      - wp_content:/var/www/html/wp-content  # Themes, plugins, uploads
    depends_on:
      mysql:
        condition: service_healthy
    networks:
      - wp-network
    restart: unless-stopped

  # ────────────────────────────────
  # Nginx — Serve static files + PHP-FPM proxy
  # ────────────────────────────────
  nginx:
    image: nginx:1.25-alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx/wordpress.conf:/etc/nginx/conf.d/default.conf:ro
      - wp_content:/var/www/html/wp-content:ro   # Serve static files directly
    depends_on:
      - wordpress
    networks:
      - wp-network
    restart: unless-stopped

volumes:
  mysql_data:    # Database files
  wp_content:    # WordPress uploads, themes, plugins

networks:
  wp-network:
    driver: bridge

Example 4: Monitoring Stack (Prometheus + Grafana + cAdvisor)

Monitoring production applications is essential. This stack tracks CPU, memory, and network usage of Docker containers:

# docker-compose.yml — Monitoring Stack
# Usage: docker compose up -d
# Access: Grafana → http://localhost:3000 (admin/admin)
#         Prometheus → http://localhost:9090

services:
  # ────────────────────────────────
  # Prometheus — Metrics Collector
  # ────────────────────────────────
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-lifecycle         # Enable config hot-reload
    networks:
      - monitoring
    restart: unless-stopped

  # ────────────────────────────────
  # Grafana — Visualization Dashboard
  # ────────────────────────────────
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_USER: ${GRAFANA_USER:-admin}
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-admin}
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./grafana/datasources:/etc/grafana/provisioning/datasources:ro
    depends_on:
      - prometheus
    networks:
      - monitoring
    restart: unless-stopped

  # ────────────────────────────────
  # cAdvisor — Container Metrics
  # ────────────────────────────────
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring
    restart: unless-stopped
    # cAdvisor communicates directly with the Docker daemon
    # which is why host paths need to be mounted

  # ────────────────────────────────
  # Node Exporter — Host Machine Metrics
  # ────────────────────────────────
  node_exporter:
    image: prom/node-exporter:latest
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - --path.procfs=/host/proc
      - --path.sysfs=/host/sys
    ports:
      - "9100:9100"
    networks:
      - monitoring
    restart: unless-stopped
    # Node Exporter exposes host CPU, RAM, disk, and network metrics

volumes:
  prometheus_data:   # Time-series metrics data
  grafana_data:      # Dashboards and settings

networks:
  monitoring:
    driver: bridge

Note: What the Monitoring Stack Tracks. cAdvisor — CPU/RAM/network/disk usage for every Docker container. Node Exporter — hardware metrics for the host machine. Prometheus — collects and stores all data. Grafana — visualizes everything in dashboards. This monitoring setup is essential in production: you cannot improve what you cannot measure.

Note: Production Checklist for Compose Apps.

Add restart: unless-stopped to all services

Configure healthcheck on database services

Use internal: true on networks for sensitive services

Store passwords in a .env file — never hardcode them in the compose file

Add the .env file to .gitignore

Set resource limits (deploy.resources.limits)

Configure log rotation (max-size, max-file)

Keep images updated with regular docker compose pull

Container Registries

Just as npm registry stores packages, a Docker registry stores images. You build an image, push it to a registry, and anyone in the world can pull it.

Note: A Docker registry is a storage and distribution system for Docker images. Docker Hub is the default public registry, but you can use private registries for proprietary images. Think of it as GitHub for code — but for images.

Image Naming Convention

Every Docker image has a fully qualified name. The format is:

[registry/][username/]image[:tag][@digest]

# Examples — from simple to fully qualified:
ubuntu                                                        # Docker Hub official (implicit library/)
nginx:1.25-alpine                                             # Docker Hub official with tag
library/ubuntu:22.04                                          # Explicit official image path
username/myapp:v1.0                                           # Docker Hub user image

# Cloud registries
ghcr.io/username/myapp:latest                                 # GitHub Container Registry
123456789.dkr.ecr.us-east-1.amazonaws.com/myapp:v1.0         # AWS ECR (region-scoped)
asia.gcr.io/myproject/myapp:latest                            # Google Container Registry
registry.gitlab.com/group/project/myapp:1.0                   # GitLab Registry

# Digest (immutable — safer than a tag!)
ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2

Image Tagging Strategy

Tag Type	Example	Use Case	Stable?
`latest`	`myapp:latest`	Development	No — changes!
Semantic Version	`myapp:v1.2.3`	Production releases	Yes
Git SHA	`myapp:a3f1b2c`	CI/CD traceability	Yes
Branch name	`myapp:main`	Branch-based testing	No
Date-based	`myapp:20240115`	Nightly builds	Yes
Digest	`myapp@sha256:abc...`	Pinned dependencies	Immutable

Caution: Avoid the latest Tag! Using the latest tag in production is dangerous. If a new image is pushed, your deployment may update unexpectedly. Always use specific version tags in production — myapp:v1.2.3 or a Git SHA.

Tip: Best Practice: Multiple Tags. Tag a single image with multiple tags: myapp:v1.2.3, myapp:v1.2, myapp:v1, and myapp:latest. This gives users flexibility while maintaining immutable specific versions.

Docker Hub — The Default Registry

Docker Hub is the default public registry — like npm for JavaScript packages. Start with a free account and explore private registries once you hit the limits.

Note: Create an account at hub.docker.com. Free accounts include 1 private repository and unlimited public repositories. Use access tokens rather than passwords — they are safer.

Complete Docker Hub Workflow

# Step 1: Login to Docker Hub
docker login
# Username: yourname
# Password: (enter password or access token)

# Better: login with an access token (Settings → Security → New Access Token)
docker login -u username --password-stdin <<< "your_access_token_here"

# Step 2: Tag your local image for Docker Hub
docker tag myapp username/myapp:v1.0.0
docker tag myapp username/myapp:latest

# Step 3: Push to Docker Hub
docker push username/myapp:v1.0.0
docker push username/myapp:latest

# Step 4: Pull from anywhere in the world
docker pull username/myapp:v1.0.0

# Step 5: Logout (important on shared machines!)
docker logout

Rate Limits — Important for CI/CD

Account Type	Pull Limit	Time Window
Anonymous (no login)	100 pulls	Per 6 hours / IP
Free account (authenticated)	200 pulls	Per 6 hours
Pro / Team / Business	Unlimited	—

# Authenticate pulls in CI/CD to avoid rate limits
echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin

# Check your current rate limit status
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl -s --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit

Rate Limit Workaround — Mirror Setup

# /etc/docker/daemon.json — configure a Docker Hub mirror
sudo nano /etc/docker/daemon.json

{
  "registry-mirrors": [
    "https://mirror.gcr.io",
    "https://registry.docker-cn.com"
  ]
}

# Restart Docker daemon after the config change
sudo systemctl restart docker

# Verify the mirror is active
docker info | grep -A2 "Registry Mirrors"

Caution: CI/CD Rate Limit Problem. If your CI/CD pipeline pulls images anonymously (no docker login), rate limits are hit quickly — especially in large teams. Solution: always authenticate in CI, or mirror images to your own private registry.

Tip: Use Access Tokens, Not Passwords. Docker Hub Settings → Security → New Access Token. Tokens can be given specific permissions (read-only, read-write). If a token is compromised, only that token needs to be revoked — not the full account. Always use access tokens in CI/CD.

Other Registries — AWS ECR, GitHub, GCR

Docker Hub is not the only option. If you deploy to AWS, ECR is the best choice. If you use GitHub Actions, ghcr.io is a perfect fit. Here is how to use each.

AWS Elastic Container Registry (ECR)

Note: When to use ECR: When your application is deployed to AWS (ECS, EKS, Lambda). ECR images integrate automatically with AWS services — no extra authentication is needed when running on AWS with proper IAM roles.

# Prerequisites: AWS CLI installed and configured (aws configure)

# Step 1: Authenticate with ECR (generates a 12-hour token)
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin \
  123456789.dkr.ecr.ap-south-1.amazonaws.com

# Step 2: Create a repository in ECR
aws ecr create-repository \
  --repository-name myapp \
  --region ap-south-1 \
  --image-scanning-configuration scanOnPush=true \
  --encryption-configuration encryptionType=AES256

# Step 3: Tag the image for ECR
ECR_URI="123456789.dkr.ecr.ap-south-1.amazonaws.com"
docker tag myapp:latest $ECR_URI/myapp:latest
docker tag myapp:latest $ECR_URI/myapp:v1.0.0

# Step 4: Push
docker push $ECR_URI/myapp:latest
docker push $ECR_URI/myapp:v1.0.0

# Step 5: Pull (from any authenticated source)
docker pull $ECR_URI/myapp:latest

# ECR Lifecycle Policy — auto-delete old images (save storage costs!)
aws ecr put-lifecycle-policy \
  --repository-name myapp \
  --lifecycle-policy '{
    "rules": [{
      "rulePriority": 1,
      "description": "Keep only last 10 images",
      "selection": {
        "tagStatus": "any",
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": {"type": "expire"}
    }]
  }'

# List images in ECR repository
aws ecr describe-images \
  --repository-name myapp \
  --region ap-south-1 \
  --query 'sort_by(imageDetails, &imagePushedAt)[-5:]'

GitHub Container Registry (ghcr.io)

GITHUB_TOKEN is automatically available in GitHub Actions — perfect integration:

# Step 1: Create a Personal Access Token (PAT)
# GitHub → Settings → Developer Settings → Personal Access Tokens
# Required scopes: read:packages, write:packages, delete:packages

# Step 2: Login to ghcr.io
echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

# Step 3: Tag and push
docker tag myapp ghcr.io/username/myapp:latest
docker push ghcr.io/username/myapp:latest

# Make the image public (default is private)
# GitHub → Package → Package Settings → Change Visibility

# GitHub Actions — push to ghcr.io (automatic authentication!)
name: Build and Push
on:
  push:
    branches: [main]

jobs:
  push:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write        # Required for ghcr.io!

    steps:
      - uses: actions/checkout@v4

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}  # Auto-available!

      - name: Build and Push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}/myapp:latest

Google Container Registry / Artifact Registry

# Authenticate gcloud with Docker
gcloud auth configure-docker asia.gcr.io

# New Artifact Registry (recommended over GCR)
gcloud auth configure-docker asia-south1-docker.pkg.dev

# Tag and push
docker tag myapp asia.gcr.io/my-project/myapp:v1.0
docker push asia.gcr.io/my-project/myapp:v1.0

# Artifact Registry (newer, more features)
docker tag myapp asia-south1-docker.pkg.dev/my-project/my-repo/myapp:v1.0
docker push asia-south1-docker.pkg.dev/my-project/my-repo/myapp:v1.0

Registry Comparison

Registry	Type	Cost	Best For
Docker Hub	Cloud	Free (limited) / Paid	Open source, small teams
AWS ECR	Cloud	Per storage + transfer	AWS deployments
Google GCR/AR	Cloud	Per storage + transfer	GCP deployments
GitHub Packages	Cloud	Free for public	Open source + CI/CD
Harbor	Self-hosted	Free (infra cost)	Enterprise, full control
Nexus Repository	Self-hosted	Free/Commercial	Enterprise artifact management

Tip: Deploying to AWS → use ECR (pick the Region closest to your users). Using GitHub Actions → use ghcr.io (free for public repos). Startup or open source → start with the Docker Hub free tier.

Pushing and Pulling Workflows — CI/CD Integration

Integrating the registry workflow with CI/CD is the core of production-ready DevOps. Here is a complete, real-world workflow for a React application.

Note: Workflow Overview: Build → Tag (with version + Git SHA) → Authenticate → Push to Registry → Verify. Every step matters. Tagging with the Git SHA lets you know exactly which commit is running in production.

Complete Registry Push Workflow

#!/bin/bash
# Full CI/CD workflow for a React app
set -e  # Exit on any error

IMAGE_NAME="myreactapp"
REGISTRY="123456789.dkr.ecr.ap-south-1.amazonaws.com"

# Generate meaningful version tags
VERSION=$(git describe --tags --always --dirty 2>/dev/null || echo "dev")
GIT_SHA=$(git rev-parse --short HEAD)
DATE=$(date +%Y%m%d)
BRANCH=$(git rev-parse --abbrev-ref HEAD | tr / -)

echo "Building version: $VERSION (SHA: $GIT_SHA)"

# Step 1: Build with build args for traceability
docker build \
  --build-arg BUILD_DATE=$DATE \
  --build-arg GIT_COMMIT=$GIT_SHA \
  --build-arg VERSION=$VERSION \
  -t $IMAGE_NAME:$VERSION \
  -t $IMAGE_NAME:$GIT_SHA \
  -t $IMAGE_NAME:latest \
  .

echo "Build complete"

# Step 2: Authenticate with ECR
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin $REGISTRY
echo "Authenticated with ECR"

# Step 3: Tag all versions for registry
docker tag $IMAGE_NAME:$VERSION $REGISTRY/$IMAGE_NAME:$VERSION
docker tag $IMAGE_NAME:$GIT_SHA $REGISTRY/$IMAGE_NAME:$GIT_SHA
docker tag $IMAGE_NAME:latest $REGISTRY/$IMAGE_NAME:latest

# Step 4: Push all tags
docker push $REGISTRY/$IMAGE_NAME:$VERSION
docker push $REGISTRY/$IMAGE_NAME:$GIT_SHA
docker push $REGISTRY/$IMAGE_NAME:latest
echo "All tags pushed"

# Step 5: Verify push was successful
echo "Recent images in ECR:"
aws ecr describe-images \
  --repository-name $IMAGE_NAME \
  --region ap-south-1 \
  --query 'sort_by(imageDetails, &imagePushedAt)[-5:].{Tags:imageTags,Size:imageSizeInBytes,Pushed:imagePushedAt}' \
  --output table

GitHub Actions — Full CI/CD Pipeline

name: Build, Scan, and Push
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  REGISTRY: 123456789.dkr.ecr.ap-south-1.amazonaws.com
  IMAGE_NAME: myreactapp

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for git describe

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-south-1

      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Generate image metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=sha-
            type=ref,event=branch
            type=semver,pattern={{version}}
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: ${{ github.event_name == 'push' }}  # Only push on merge to main
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha          # GitHub Actions cache
          cache-to: type=gha,mode=max

Caution: Do Not Push on Pull Requests! Pull request builds should only build and test — do not push images. Push only when merging to main/master. That is why push: ${{ github.event_name == 'push' }} is used.

Tip: Speed Up Builds with Cache. cache-from: type=gha and cache-to: type=gha,mode=max use GitHub Actions cache. The first build will be slow; subsequent builds are significantly faster — especially when dependencies have not changed.

Container Security Fundamentals — Why It Matters

Docker containers share the host kernel. This means — if a container escapes, an attacker can reach the host machine. Do not treat security as an afterthought; you must think about it while writing your Dockerfile.

Caution: Container escape is real. In 2019, a runc vulnerability (CVE-2019-5736) allowed attackers to access the host machine from inside a container. In 2020, a Docker Desktop vulnerability followed. This is not theoretical — production systems have been compromised. Always follow security fundamentals.

Common Container Vulnerabilities

#	Vulnerability	Risk Level	Impact
1	Running as root	Critical	Container escape, host access
2	Outdated base images (CVEs)	Critical	Known exploits in production
3	Hardcoded secrets in image	Critical	Credential theft via docker history
4	Privileged mode	Critical	Full host access
5	Excessive capabilities	High	Privilege escalation
6	No resource limits	High	Denial of Service (fork bomb)
7	Unscanned images	High	Unknown vulnerabilities
8	Docker socket exposed	Critical	Complete host compromise

Real Attack Scenarios

# Scenario 1: Secret leakage via docker history
docker history myapp:latest --no-trunc
# IMAGE     CREATED   CREATED BY
# abc123    ...       /bin/sh -c export API_KEY=sk-prod-secret123 && node setup.js
#                     ^^^^ VISIBLE TO ANYONE who pulls the image!

# Scenario 2: Crypto mining via compromised public image
# Attacker pushes malicious "ubuntu:latest" to a similarly-named registry
# Your CI/CD pulls it, runs a crypto miner in your cloud (you pay the bill!)

# Scenario 3: Docker socket exposure
docker run -v /var/run/docker.sock:/var/run/docker.sock myapp
# Attacker inside container can now:
docker run -v /:/host alpine chroot /host  # Read/write host filesystem!

# Scenario 4: Privileged container escape
docker run --privileged alpine
# Inside container:
mount /dev/sda1 /mnt  # Mount host disk
chroot /mnt           # Root access to host!

Caution: Never mount /var/run/docker.sock into a container in production unless absolutely necessary (such as for CI agents). Any container that can access the Docker socket effectively has root access to the entire machine.

Note: Container security is layered: secure Dockerfile → runtime security flags → network policies → image scanning → secrets management → monitoring. If one layer fails, the next catches it. This is the "defense in depth" model.

Secure Dockerfile Practices — 20+ Best Practices

Security starts at build time. An insecure Dockerfile cannot be fixed at runtime. Follow these best practices for production-ready Dockerfiles.

Complete Secure Dockerfile Template

# ============================================
# SECURE DOCKERFILE TEMPLATE — React/Node App
# ============================================

# Best Practice #1: Use specific versions — never "latest" or just "node"
# The "latest" tag can change, breaking your build
FROM node:18.19.0-alpine3.19

# Best Practice #2: Use minimal base images
# Security risk order (most to least): ubuntu > debian > debian-slim > alpine > distroless
# Alpine has 98% fewer packages = smaller attack surface

# Best Practice #3: Update packages in base image (apply security patches)
RUN apk update && \
    apk upgrade && \
    rm -rf /var/cache/apk/*
# Note: Alpine uses apk; Debian/Ubuntu uses apt-get

# Best Practice #4: Create a non-root user EARLY
# Never run the app as root!
RUN addgroup --system --gid 1001 appgroup && \
    adduser --system --uid 1001 --ingroup appgroup appuser

WORKDIR /app

# Best Practice #5: Copy dependency files FIRST (caching + security)
# Set proper ownership during COPY with --chown
COPY --chown=appuser:appgroup package*.json ./

# Best Practice #6: Install ONLY production dependencies
RUN npm ci --only=production && \
    npm cache clean --force
# npm ci = exact versions from package-lock.json (more reproducible)

# Best Practice #7: Copy app source with proper ownership
COPY --chown=appuser:appgroup . .

# Best Practice #8: Build (if needed) while still root for permissions
# Then switch to non-root for runtime
RUN npm run build

# Best Practice #9: Remove dev files and build artifacts
RUN rm -rf src/ tests/ *.test.js .env.* coverage/

# Best Practice #10: Switch to non-root user — everything below runs as appuser
USER appuser

# Best Practice #11: Expose ONLY required ports (documentation, not enforcement)
EXPOSE 3000

# Best Practice #12: Add HEALTHCHECK — K8s/Docker knows if app is actually working
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', r => process.exit(r.statusCode===200?0:1))" || exit 1

# Best Practice #13: Use exec form CMD (not shell form)
# Shell form: CMD "node server.js" → /bin/sh -c "node server.js" (signals don't propagate!)
# Exec form: CMD ["node", "server.js"] → direct process (signals work correctly)
CMD ["node", "server.js"]

Security Anti-Patterns — Never Do This

# ❌ DANGEROUS: Hardcoded secrets visible in docker history!
ENV DATABASE_PASSWORD=mysupersecretpassword
ENV AWS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# ❌ DANGEROUS: Even in RUN, it stays in layer history
RUN export API_KEY=abc123secret && npm run setup

# ❌ BAD: Copying entire directory without .dockerignore
COPY . .
# This might include: .env, .git/, node_modules/, passwords.txt, etc.

# ❌ BAD: Using latest tag
FROM node:latest  # Which version? Will break unexpectedly!

# ❌ BAD: Running as root (default if no USER instruction)
# root inside container → if compromised, easier to escalate

# ❌ BAD: Adding unnecessary packages
RUN apt-get install -y curl wget vim git build-essential
# More packages = larger attack surface

# ❌ BAD: Ignoring package-lock.json
RUN npm install  # Might install different versions each time!

.dockerignore — First Line of Defense

# Always maintain a comprehensive .dockerignore!

# Secrets and config
.env
.env.*
.env.local
.env.production
secrets/
*.pem
*.key
*.cert

# Development artifacts
node_modules/
npm-debug.log*
yarn-error.log

# Git history (can contain old secrets!)
.git/
.gitignore

# Tests and coverage (not needed in production image)
tests/
__tests__/
*.test.js
*.spec.js
coverage/
.nyc_output/

# Documentation
README.md
docs/
*.md

# IDE files
.vscode/
.idea/
*.swp

# CI/CD configs (not needed in image)
.github/
.gitlab-ci.yml
Jenkinsfile

# Build artifacts (will be regenerated)
dist/
build/

Caution: .env MUST be in .dockerignore. If you use COPY . . and .env is not excluded, all your secrets end up in the image. Anyone can see them via docker history myapp or image inspection. Real data breaches have happened this way.

Tip: Multi-stage builds give you both security and smaller image sizes. Build tools (gcc, make, npm, pip) do not belong in a production image. Copy only the runtime artifacts into the final stage — no build tools means a smaller attack surface and 60–80% smaller images.

Note: Google's distroless images contain no shell, no package manager, and no utilities — only the app runtime. An attacker cannot even open a shell. Use gcr.io/distroless/nodejs18-debian11. The trade-off is harder debugging — use the debug variant for that purpose.

Runtime Security — Sandboxing Containers

After building a secure Dockerfile, apply security flags at runtime. These flags keep the container in a tight sandbox — even if the container is compromised, the damage is contained.

Runtime Security Flags

# 1. Drop ALL capabilities, add back only what's needed
# Linux capabilities are specific privileges that root has
docker run --cap-drop ALL --cap-add NET_BIND_SERVICE nginx
# Drops 38 capabilities and adds back only port binding < 1024

# 2. Read-only filesystem — prevent tampering
docker run --read-only --tmpfs /tmp --tmpfs /var/run myapp
# --read-only: container filesystem is read-only
# --tmpfs /tmp: in-memory temp storage (data lost on restart)

# 3. No new privileges — prevent sudo/setuid escalation
docker run --security-opt no-new-privileges myapp
# Even if a setuid binary exists, it cannot escalate privileges

# 4. Custom seccomp profile — limit system calls
docker run --security-opt seccomp=/path/to/seccomp-profile.json myapp
# Default Docker seccomp blocks 44 of 300+ syscalls
# A custom profile can be more restrictive

# 5. AppArmor profile — Mandatory Access Control
docker run --security-opt apparmor=docker-default myapp

# 6. Resource limits — prevent Denial of Service
docker run \
  --memory=512m \           # Max RAM
  --memory-swap=512m \      # No swap (set equal to memory)
  --cpus=0.5 \              # Max 0.5 CPU cores
  --pids-limit=100 \        # Max 100 processes (prevents fork bombs!)
  --ulimit nofile=1024:1024 \  # Max open file descriptors
  myapp

# 7. NEVER use --privileged in production!
# docker run --privileged myapp  ← Gives container host-level access!

# Complete secure run command
docker run \
  --cap-drop ALL \
  --cap-add NET_BIND_SERVICE \
  --read-only \
  --tmpfs /tmp \
  --security-opt no-new-privileges \
  --memory=512m \
  --cpus=0.5 \
  --pids-limit=100 \
  --user 1001:1001 \
  myapp:v1.0.0

Linux Capabilities — What to Grant and What to Deny

Capability	Purpose	Use in Production?
`NET_BIND_SERVICE`	Bind ports < 1024 (80, 443)	Only if port < 1024
`NET_ADMIN`	Configure networking	Only for network tools
`CHOWN`	Change file ownership	Very rarely
`DAC_OVERRIDE`	Bypass file permissions	Never
`FOWNER`	Bypass owner restrictions	Never
`SYS_ADMIN`	Broad system admin access	NEVER (equivalent to root!)
`SYS_PTRACE`	Process tracing (debuggers)	Never in production
`KILL`	Send signals to processes	Rarely needed

User Namespace Remapping (Advanced)

// /etc/docker/daemon.json — User namespace remapping
// Container root (UID 0) becomes an unprivileged user on the host!
{
  "userns-remap": "default"
}
// After this: sudo systemctl restart docker
// Container root → host UID 100000 (non-privileged)

Caution: Never use --privileged in production. The --privileged flag effectively makes the container root on the host, bypassing the security context entirely. If any tutorial asks you to use --privileged without explaining why, that tutorial is wrong. Use it only for specific, well-understood cases such as certain CI/CD systems.

Tip: Docker's default seccomp profile already blocks 44+ dangerous syscalls. To tighten further, create a custom profile. Docker's documentation provides the default profile as a starting point.

Image Scanning — Finding CVEs Before Production

Your image is built and pushed — but does it contain known vulnerabilities? Image scanning tools automatically check for CVEs (Common Vulnerabilities and Exposures) in your base image and dependencies.

Note: A CVE (Common Vulnerability and Exposure) is the unique identifier for a publicly known security vulnerability. For example, CVE-2021-44228 was the Log4Shell vulnerability. A scanner compares your image's packages against a CVE database.

Trivy — Most Popular Free Scanner

# Install Trivy (Linux/macOS)
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | \
  sh -s -- -b /usr/local/bin

# macOS via Homebrew
brew install trivy

# Basic image scan
trivy image nginx:latest

# Scan with severity filter — only show HIGH and CRITICAL
trivy image --severity HIGH,CRITICAL nginx:latest

# Fail CI/CD if CRITICAL vulnerabilities found (exit code 1)
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan your local Dockerfile for misconfigurations
trivy config ./Dockerfile

# Scan docker-compose.yml for security issues
trivy config ./docker-compose.yml

# Output as JSON for automated processing
trivy image --format json --output results.json myapp:latest

# Scan a GitHub repository (without cloning)
trivy repo https://github.com/user/myrepo

# Generate SBOM (Software Bill of Materials)
trivy image --format cyclonedx --output sbom.json myapp:latest

# GitHub Actions — Trivy scan in CI/CD
- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: myapp:latest
    format: sarif
    output: trivy-results.sarif
    severity: CRITICAL,HIGH
    exit-code: 1  # Fail build on critical issues

- name: Upload Trivy results to GitHub Security tab
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: trivy-results.sarif

Docker Scout — Built-in Scanner

# Docker Scout (built-in since Docker Desktop 4.17)

# Quick vulnerability overview
docker scout quickview myapp:latest

# Detailed CVE list
docker scout cves myapp:latest

# Filter by severity
docker scout cves --only-severity critical myapp:latest

# Compare two versions (what changed?)
docker scout compare --to myapp:v1.0 myapp:v2.0

# Get recommendations (what base image to use?)
docker scout recommendations myapp:latest

# Local filesystem scan
docker scout cves fs://./  # Scan current directory

Snyk — Enterprise Scanner

# Install Snyk CLI
npm install -g snyk

# Authenticate (free account at snyk.io)
snyk auth

# Scan Docker image
snyk container test nginx:latest

# Scan with fail threshold
snyk container test --severity-threshold=high nginx:latest

# Monitor over time (track vulnerabilities as they are discovered)
snyk container monitor nginx:latest

# Snyk in CI/CD (GitHub Actions)
- name: Run Snyk to check Docker image for vulnerabilities
  uses: snyk/actions/docker@master
  env:
    SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
  with:
    image: myapp:latest
    args: --severity-threshold=high

Tip: For development: use Docker Scout (built-in, no setup). For CI/CD: use Trivy (free, fast, no registration). For enterprise: use Snyk or Aqua Security (paid, more features). ECR users: enable "Scan on Push" in repository settings — it is automatic and free.

Note: Achieving zero vulnerabilities is an unrealistic goal. Having LOW/MEDIUM vulnerabilities in production images is normal — fixing everything is not feasible. Focus on: always fix CRITICAL, fix or document exceptions for HIGH, address MEDIUM only if exploitable. False positives also occur — triage carefully.

Secrets Management — Never Bake Secrets Into Images

The most common mistake in container security is hardcoding secrets (passwords, API keys, tokens) into Docker images. Once an image is pushed to a registry, anyone who pulls it can inspect the secrets.

Caution: docker history is a secret leakage tool. Every Dockerfile instruction creates an image layer. ENV API_KEY=secret is permanently baked into that layer — even if you unset it later. Anyone can see it via docker history --no-trunc myapp. Real incident: a company put API keys in their Dockerfile, pushed the image as public to Docker Hub, and received a $50,000 AWS bill from crypto mining.

Never Do This

# ❌ EXTREMELY DANGEROUS — visible in docker history forever!
ENV DATABASE_PASSWORD=mysupersecretpassword123
ENV AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
ENV JWT_SECRET=my-jwt-secret-key

# ❌ ALSO DANGEROUS — even in RUN, it stays in the layer!
RUN export API_KEY=sk-prod-abc123 && ./configure.sh

# ❌ BAD — copying .env file into image
COPY .env /app/.env  # .env is now inside the image layers!

# Check what's visible (run this on your image)
docker history --no-trunc myapp | grep -i "password\|secret\|key\|token"

Approach 1: Runtime Environment Variables

# Pass at runtime — not baked into the image
docker run -e DATABASE_PASSWORD=secret myapp

# From a file — more manageable (file security is your responsibility)
docker run --env-file secrets.env myapp

# secrets.env format (add this file to .gitignore!)
# DATABASE_URL=postgresql://user:pass@host:5432/db
# API_KEY=sk-prod-abc123
# JWT_SECRET=super-secret-jwt-key

# CAVEAT: Still visible in 'docker inspect'!
docker inspect myapp | jq '.[0].Config.Env'
# [
#   "DATABASE_PASSWORD=secret",  ← Visible to anyone with docker inspect!
#   "PATH=/usr/local/sbin:..."
# ]

Approach 2: Docker Secrets (Swarm Mode)

# Create a Docker secret (stored encrypted in Swarm Raft)
echo "mysupersecretpassword" | docker secret create db_password -
printf "sk-prod-abc123secret" | docker secret create api_key -

# List secrets (values NOT shown)
docker secret ls

# Use in service (mounted as file at /run/secrets/secretname)
docker service create \
  --secret db_password \
  --secret api_key \
  --env DB_PASSWORD_FILE=/run/secrets/db_password \
  --env API_KEY_FILE=/run/secrets/api_key \
  myapp

# Inside container, secrets are files:
cat /run/secrets/db_password  # mysupersecretpassword
cat /run/secrets/api_key      # sk-prod-abc123secret

# docker-compose.yml with secrets (Swarm mode)
version: "3.8"

secrets:
  db_password:
    external: true    # Secret already created via docker secret create
  api_key:
    external: true

services:
  api:
    image: myapp:latest
    secrets:
      - db_password
      - api_key
    environment:
      # Point to secret files, not actual values
      - DB_PASSWORD_FILE=/run/secrets/db_password
      - API_KEY_FILE=/run/secrets/api_key

Approach 3: AWS Secrets Manager (Production Best Practice)

#!/bin/sh
# entrypoint.sh — Fetch secrets at startup from AWS Secrets Manager
# Container uses an IAM role (no hardcoded AWS credentials needed!)

set -e

echo "Fetching secrets from AWS Secrets Manager..."

# Fetch database password
export DB_PASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id prod/myapp/db_password \
  --query SecretString \
  --output text \
  --region ap-south-1)

# Fetch API key
export API_KEY=$(aws secretsmanager get-secret-value \
  --secret-id prod/myapp/api_key \
  --query SecretString \
  --output text \
  --region ap-south-1)

echo "Secrets loaded successfully"

# Execute the main command (e.g., node server.js)
exec "$@"

Approach 4: BuildKit Secret Mounts (Build-time Secrets)

# syntax=docker/dockerfile:1
FROM node:18-alpine

WORKDIR /app

# Secret mounted ONLY during this RUN — not stored in the image layer!
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm install  # Can access private npm registry using .npmrc secret

# Private repo clone during build
RUN --mount=type=ssh \
    git clone git@github.com:myorg/private-repo.git /app/lib

# Build with secret (secret never enters the image!)
docker build \
  --secret id=npmrc,src=$HOME/.npmrc \
  -t myapp .

# SSH forwarding for private repos
eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa
docker build --ssh default -t myapp .

Tip: Always add .env to both .gitignore and .dockerignore. Commit a .env.example template without values so team members know which variables are required. In production, never use .env files — use a proper secrets management solution.

Note: Secrets manager options include: AWS Secrets Manager (paid, excellent rotation support), HashiCorp Vault (free, powerful, complex), Kubernetes Secrets (base64-encoded — not truly secret!), Doppler (developer-friendly SaaS), Azure Key Vault, and GCP Secret Manager. Choose based on your cloud provider.

BuildKit — Modern Docker Build Engine

BuildKit is the default build engine in Docker 23+. It is much faster, safer, and more feature-rich than the older builder. Parallel builds, better caching, secret mounts — all of these come from BuildKit.

Note: Old builder: sequential layers, no cache import/export, no secret mounts. BuildKit: parallel stage builds, registry cache, secret mounts, SSH forwarding, multi-arch support. Master BuildKit once and CI/CD build times improve dramatically.

Enabling BuildKit

# Check if BuildKit is available
docker buildx version
# github.com/docker/buildx v0.12.0 docker-desktop

# Docker 23+ enables it automatically
# For older versions:

# Option 1: Per-build environment variable
DOCKER_BUILDKIT=1 docker build -t myapp .

# Option 2: Enable permanently in daemon.json
sudo nano /etc/docker/daemon.json
# Add: { "features": { "buildkit": true } }
sudo systemctl restart docker

# Option 3: Set DOCKER_BUILDKIT in shell profile
echo 'export DOCKER_BUILDKIT=1' >> ~/.bashrc

BuildKit Special Features — syntax=docker/dockerfile:1

# This comment enables the latest BuildKit Dockerfile syntax
# syntax=docker/dockerfile:1

FROM node:18-alpine

WORKDIR /app

# Feature 1: Secret mounts — never stored in image layers!
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm install --legacy-peer-deps

# Feature 2: Cache mounts — persist between builds!
RUN --mount=type=cache,target=/root/.npm \
    npm ci  # npm cache persists across builds = MUCH faster!

# Feature 3: Bind mounts — mount host directory without COPY
RUN --mount=type=bind,source=package.json,target=package.json \
    --mount=type=bind,source=package-lock.json,target=package-lock.json \
    npm ci

# Feature 4: SSH forwarding
RUN --mount=type=ssh \
    git clone git@github.com:private/repo.git

docker buildx Commands

# List all builders
docker buildx ls

# Create a new builder with docker-container driver (more features)
docker buildx create \
  --name mybuilder \
  --driver docker-container \
  --bootstrap
docker buildx use mybuilder

# Inspect builder (shows supported platforms)
docker buildx inspect mybuilder

# Build for current platform (like regular docker build)
docker buildx build -t myapp .

# Build for a specific platform
docker buildx build --platform linux/amd64 -t myapp:amd64 --load .
docker buildx build --platform linux/arm64 -t myapp:arm64 --load .

# Build and push multi-arch image in one command!
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t username/myapp:latest \
  --push \
  .

# Remove builder
docker buildx rm mybuilder

Registry Cache — CI/CD Game Changer

# Export cache to registry (persists between CI runs)
docker buildx build \
  --cache-to type=registry,ref=myapp:buildcache,mode=max \
  --cache-from type=registry,ref=myapp:buildcache \
  -t myapp:latest \
  --push \
  .
# mode=max: Cache all intermediate layers (maximum cache hits)
# mode=min: Cache only final layer (smaller cache size)

# GitHub Actions — Optimal BuildKit caching
- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3

- name: Build and push with cache
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:latest
    cache-from: type=gha          # GitHub Actions cache (free!)
    cache-to: type=gha,mode=max   # Save cache for next run
    # Result: 5-minute builds → 45-second builds on cache hit!

Tip: Without caching: a Node app build takes 4–5 minutes. With caching (dependencies unchanged): 30–60 seconds. For large enterprise apps: 20 minutes → 2 minutes. This difference directly affects production deployment frequency.

Note: With multi-arch builds, --load (local image) and --push (registry) cannot be used simultaneously in most cases. For single-arch builds use --load; for multi-arch, push directly to the registry with --push.

Multi-Architecture Images — Supporting ARM and x86

You develop on a MacBook M1/M2/M3 (ARM64) and deploy to AWS EC2 (amd64). Multi-arch images work on both platforms without issues — no need for separate Dockerfiles.

Note: Apple Silicon Macs (ARM64) have become standard dev machines. AWS Graviton instances (ARM64) offer the same performance at 40% lower cost than x86. If your image only supports amd64, ARM machines will produce a "wrong platform" error. Build multi-arch images to support everyone.

Setup and Build

# Step 1: Install QEMU (enables emulation of other architectures)
docker run --privileged --rm tonistiigi/binfmt --install all
# Installs QEMU user-space emulators for ARM, etc.

# Verify QEMU installation
ls /proc/sys/fs/binfmt_misc/
# Should show: qemu-aarch64, qemu-arm, etc.

# Step 2: Create a multi-arch capable builder
docker buildx create \
  --name multiarch-builder \
  --driver docker-container \
  --bootstrap
docker buildx use multiarch-builder

# Step 3: Inspect — verify which platforms are supported
docker buildx inspect multiarch-builder --bootstrap
# Platforms: linux/amd64, linux/arm64, linux/arm/v7, linux/386...

# Step 4: Build and push multi-arch image in one command
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag username/myapp:latest \
  --tag username/myapp:v1.0.0 \
  --push \
  .

# Step 5: Verify multi-arch manifest
docker manifest inspect username/myapp:latest
# Shows separate manifests for each architecture

Platform-Specific Handling in Dockerfile

# syntax=docker/dockerfile:1

# BuildKit injects these automatically
ARG TARGETPLATFORM
ARG TARGETARCH
ARG TARGETOS

FROM node:18-alpine

# Platform-specific commands (if needed)
RUN case "$TARGETARCH" in \
    amd64) echo "Building for x86_64" ;; \
    arm64) echo "Building for ARM64" ;; \
    arm)   echo "Building for ARM v7" ;; \
    esac

# Most Node.js apps need no platform-specific code
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .

USER node
EXPOSE 3000
CMD ["node", "server.js"]

GitHub Actions — Multi-Arch CI/CD

name: Multi-Arch Build
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up QEMU (ARM emulation)
        uses: docker/setup-qemu-action@v3
        with:
          platforms: linux/amd64,linux/arm64

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_TOKEN }}

      - name: Build and push multi-arch
        uses: docker/build-push-action@v5
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: username/myapp:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

Note: QEMU emulation makes ARM64 builds 5–10x slower on an amd64 machine. Solutions: use native ARM builders (GitHub has paid ARM64 runners), or run matrix builds (amd64 and arm64 in parallel). QEMU is fine for small images.

Tip: AWS Graviton3 (arm64) instances are 20–40% cheaper than equivalent x86 instances for the same workload. Build one multi-arch image, deploy on Graviton — significant cost savings at scale. t4g instances are cheaper than t3, c7g instances are cheaper than c6i.

Docker Context — Managing Multiple Docker Hosts

Docker contexts let you manage multiple Docker hosts from one machine — local, staging server, production server — all with the same docker command. Switch context and all commands go there.

Note: Contexts are useful when you want to run Docker commands on remote servers without SSH-ing in manually, or when managing multiple environments (dev, staging, production) from a CI/CD workflow.

Context Management Commands

# List all contexts (* = currently active)
docker context ls
# NAME           DESCRIPTION   DOCKER ENDPOINT               KUBERNETES ENDPOINT
# default *      ...           unix:///var/run/docker.sock
# remote-server  ...           ssh://deploy@203.0.113.10

# Create context for a remote server (uses SSH)
docker context create remote-server \
  --description "Production server" \
  --docker "host=ssh://deploy@203.0.113.10"

# Create context with a specific SSH key
docker context create staging \
  --docker "host=ssh://ubuntu@staging.myapp.com"

# Switch to remote context
docker context use remote-server
# All docker commands now go to the remote host!

docker ps      # Shows containers on remote server!
docker images  # Shows images on remote server!
docker run -d nginx  # Runs on remote server!

# Switch back to local
docker context use default

# One-off command in a specific context (without switching)
docker --context remote-server ps
docker --context remote-server logs myapp -f

# Inspect a context
docker context inspect remote-server

Export/Import Contexts

# Export context (share with team)
docker context export remote-server > production.dockercontext

# Import on another machine
docker context import production-server production.dockercontext

# Remove a context
docker context rm old-server

Context with AWS ECS

# Create ECS context (deploy directly to ECS!)
docker context create ecs myecscontext
# Prompts for AWS credentials/profile

# Switch to ECS context
docker context use myecscontext

# Now docker compose commands deploy to ECS!
docker compose up  # Deploys to ECS Fargate!

Tip: Define hosts in ~/.ssh/config, then reference them in a context: docker context create prod --docker "host=ssh://prod-server". SSH config can include jump hosts and identity files — complex infrastructure can be managed cleanly this way.

Caution: A common mistake is switching to a remote context, doing some work, and forgetting about it. The next local command runs on the remote host instead. Always check the current context with docker context ls. Consider showing the current context in your shell prompt using Starship or oh-my-zsh plugins.

Docker Plugins — Extending Functionality

Docker plugins extend Docker daemon capabilities with custom volume drivers, network drivers, and authorization plugins. They are useful in enterprise environments and for integrating cloud storage.

Note: Plugin types — Volume plugins: connect cloud storage (AWS EBS, NFS, GlusterFS) to Docker volumes. Network plugins: custom networking (Weave, Calico, Flannel). Authorization plugins: centralized access control defining what each container is allowed to do.

Volume Plugins

# List installed plugins
docker plugin ls

# Install NFS volume plugin (mount NFS shares as Docker volumes)
docker plugin install vieux/sshfs DEBUG=1

# Install local-persist plugin (volumes survive container removal)
docker plugin install cwspear/docker-local-persist-volume-plugin

# Enable/disable plugin
docker plugin enable pluginname
docker plugin disable pluginname

# Remove plugin
docker plugin rm pluginname

# Inspect plugin
docker plugin inspect vieux/sshfs

# SSHFS Plugin — Mount a remote directory as a Docker volume
# Useful for: shared config files, remote storage

# First install the plugin
docker plugin install vieux/sshfs

# Create volume using plugin
docker volume create \
  --driver vieux/sshfs \
  --opt sshcmd=user@remote-host:/path/to/dir \
  --opt password=mysshpassword \
  remote-data

# Use volume in container
docker run -v remote-data:/app/data myapp

# AWS EBS volume plugin (for persistent EBS volumes)
docker plugin install rexray/ebs \
  EBS_REGION=ap-south-1

docker volume create \
  --driver rexray/ebs \
  --opt size=20 \
  my-ebs-volume

Authorization Plugins — Enterprise Security

# OPA (Open Policy Agent) authorization plugin
# Define policies: what each container is allowed to do

# Install
docker plugin install openpolicyagent/opa-docker-authz-local:v0.2 \
  policy-file=/etc/docker/policies/authz.rego

# authz.rego policy example (Rego language)
# package docker.authz
# allow {
#   not deny
# }
# deny {
#   input.Body.HostConfig.Privileged == true  # Block privileged containers
# }

# Enable in daemon.json:
# {
#   "authorization-plugins": ["openpolicyagent/opa-docker-authz-local:v0.2"]
# }

Caution: Docker plugins run with daemon-level access. Only use trusted, well-maintained plugins. Review permissions carefully when installing — a malicious plugin can compromise the entire Docker daemon.

Tip: Many plugin use cases are now better handled by Kubernetes and CSI (Container Storage Interface) drivers. If you are using Kubernetes, explore CSI drivers instead of Docker plugins — the ecosystem is better and offers more options.

Resource Management — Limiting Container Resources

Without resource limits, a misbehaving container can consume all the memory and CPU on the host, crashing everything else. Apply limits to every container in production.

Caution: :(){ :|:& };: is a bash fork bomb. Without --pids-limit, this can crash an entire machine from a single container. In the real world, buggy code leaks memory or spawns infinite processes. Always set limits.

Memory Limits

# Hard memory limit — OOM killer triggers if exceeded
docker run --memory=512m myapp

# Memory + swap limit
docker run --memory=512m --memory-swap=1g myapp
# RAM: 512MB, Swap: 512MB (total 1GB, swap = total - ram)

# Disable swap for container
docker run --memory=512m --memory-swap=512m myapp

# Soft limit — suggestion, not enforced (for low-memory situations)
docker run --memory-reservation=256m myapp

# OOM kill disable (risky — lets container grow without being killed)
docker run --oom-kill-disable myapp  # Use with extreme caution!

CPU Limits

# Hard CPU limit — cannot exceed X cores
docker run --cpus=1.5 myapp       # Max 1.5 CPU cores
docker run --cpus=0.25 myapp      # Max 25% of one CPU core

# CPU shares — relative weight (soft limit, when contention exists)
docker run --cpu-shares=512 myapp  # Default is 1024; gets half the CPU when contested
docker run --cpu-shares=2048 myapp # Gets double the default CPU priority

# Pin to specific CPU cores (NUMA optimization, isolation)
docker run --cpuset-cpus=0,1 myapp  # Only use CPUs 0 and 1
docker run --cpuset-cpus=2-5 myapp  # Use CPUs 2, 3, 4, 5

I/O and Other Limits

# Disk I/O limits
docker run --device-read-bps /dev/sda:100mb myapp    # Max 100MB/s read
docker run --device-write-bps /dev/sda:50mb myapp    # Max 50MB/s write
docker run --device-read-iops /dev/sda:1000 myapp    # Max 1000 IOPS read

# PID limit (prevent fork bombs!)
docker run --pids-limit=100 myapp  # Max 100 processes/threads

# Open file descriptor limit
docker run --ulimit nofile=1024:1024 myapp
# Format: soft_limit:hard_limit

# Update limits on a RUNNING container (no restart needed!)
docker update --memory=1g --cpus=2 myapp

# Check resource usage (real-time)
docker stats                     # All containers
docker stats myapp               # Specific container
docker stats --no-stream myapp   # Snapshot, not live

Production Resource Sizing Guide

Service Type	CPU	Memory	Notes
Web API (Node.js)	0.5–1.0	256–512MB	Scale horizontally
Database (PostgreSQL)	2.0+	1–4GB	Depends on dataset size
Cache (Redis)	0.25	256–512MB	Set maxmemory in Redis config
Background Worker	1.0	512MB–1GB	Depends on task type
Message Queue	0.5	512MB	RabbitMQ/Kafka varies
Monitoring Agent	0.1	64–128MB	Should be minimal
Nginx Proxy	0.25	128MB	Very lightweight

# docker-compose.yml with resource limits
services:
  api:
    image: myapi:latest
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: 512M
        reservations:
          cpus: "0.25"
          memory: 256M

  db:
    image: postgres:15-alpine
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 2G
        reservations:
          cpus: "1.0"
          memory: 1G

Tip: Strategy for setting limits: first run without limits and observe actual usage with docker stats. Then set limit = actual_peak × 1.5 (with buffer). Test on staging to verify the app works correctly within limits. Deploy to production and track OOM kills and CPU throttling via monitoring.

Debugging Containers — When Nothing Works

"It works on my machine" — everyone has heard this. Containers have their own issues. Here is a systematic debugging methodology for quickly resolving production problems.

Note: Debugging order: Status → Logs → Resources → Exec into container → Processes → Network → Events. Follow this order — 80% of issues are resolved via logs or exec. Advanced tools are only needed when the basics fail.

Systematic Debugging Playbook

# ===== STEP 1: Container Status =====
docker ps -a  # -a shows stopped containers too
# STATUS: Up 2 hours (good) | Exited (1) 5 min ago (bad) | Restarting (crash loop!)

# Detailed status
docker inspect myapp | jq '.[0].State'
# {
#   "Status": "exited",
#   "Running": false,
#   "ExitCode": 137,  ← 137 = OOM kill! 1 = app error, 0 = clean exit
#   "OOMKilled": true,
#   "Error": ""
# }

# ===== STEP 2: Logs =====
docker logs myapp              # All logs
docker logs myapp --tail 100   # Last 100 lines
docker logs myapp -f           # Follow (like tail -f)
docker logs myapp --since 30m  # Last 30 minutes
docker logs myapp 2>&1 | grep -i error  # Filter errors

# ===== STEP 3: Resource Usage =====
docker stats myapp --no-stream
# CONTAINER  CPU %  MEM USAGE / LIMIT   MEM %   NET I/O    BLOCK I/O
# myapp      95.2%  490MiB / 512MiB    95.7%   1.2GB/500MB  0B/0B
# ← 95% memory → OOM kill is imminent!

# ===== STEP 4: Shell into Running Container =====
docker exec -it myapp bash    # bash shell (Debian/Ubuntu)
docker exec -it myapp sh      # sh shell (Alpine)
docker exec -it myapp /bin/sh # explicit path

# Run a specific command without interactive shell
docker exec myapp env          # Check environment variables
docker exec myapp cat /app/config.json

# ===== STEP 5: Check Processes =====
docker top myapp               # Processes running in container
docker exec myapp ps aux       # More detailed process list

# ===== STEP 6: Network Debugging =====
docker inspect myapp | jq '.[0].NetworkSettings.Networks'
docker exec myapp netstat -tulpn    # Listening ports
docker exec myapp curl -v http://othercontainer:8080/health

# Test DNS resolution in container
docker exec myapp nslookup db        # Can container resolve service name?
docker exec myapp ping -c3 db        # Can container reach db?

# ===== STEP 7: Docker Events =====
docker events --filter 'container=myapp'
docker events --filter 'container=myapp' --since 1h
# Shows: die, kill, oom, start, stop, restart events

Debugging Containers That Won't Start

# Container crashes immediately — override the entrypoint
docker run -it --entrypoint /bin/sh myapp:latest
# Or bash:
docker run -it --entrypoint /bin/bash myapp:latest

# Then manually run the CMD to see the exact error:
$ node server.js
# Error: Cannot find module 'express'  ← npm install was never run!

# Debug with a minimal command
docker run -it myapp:latest echo "container works"
docker run -it myapp:latest ls /app    # Check if files are present
docker run -it myapp:latest env        # Check env vars

# Check image layers/history for clues
docker history myapp:latest
docker inspect myapp:latest | jq '.[0].Config'

Advanced Network Debugging — netshoot

# nicolaka/netshoot — network debugging toolkit
# Contains: curl, wget, dig, nslookup, netstat, tcpdump, etc.

# Share network namespace with target container
docker run -it \
  --net container:myapp \
  nicolaka/netshoot

# Now you have full network tools in myapp's network namespace
$ netstat -tulpn      # See what myapp is listening on
$ curl localhost:3000 # Test myapp's API
$ tcpdump -i eth0     # Capture traffic

# Share both network AND pid namespace
docker run -it \
  --net container:myapp \
  --pid container:myapp \
  nicolaka/netshoot

# Check for OOM kills in system logs
dmesg | grep oom
journalctl -k | grep oom | tail -20

Tip: Decoding exit codes — Exited (0): clean exit (intentional). Exited (1): application error. Exited (137): killed by OOM killer (memory limit exceeded). Exited (139): segfault. Exited (143): SIGTERM received (graceful stop). Exit codes reveal a lot before you even read the logs.

Caution: Alpine base images do not include bash, curl, or netstat by default. Install them temporarily for debugging: docker exec myapp apk add --no-cache curl bash net-tools. For distroless images, use ephemeral debug containers (a Kubernetes feature).

Docker Health Checks — Container Self-Assessment

A running container does not mean the application is healthy. The process may be alive but unable to connect to the database, or not listening on the expected port. Health checks tell Docker whether the app is actually working.

Note: Health check states — starting: within the start_period, checks are ignored. healthy: the most recent check passed. unhealthy: retries exhausted, all checks failed. Docker Swarm and Kubernetes automatically restart unhealthy containers.

Dockerfile Health Check Options

# HTTP endpoint check (for REST APIs)
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1
# --interval: How often to check (default: 30s)
# --timeout: How long to wait for one check (default: 30s)
# --start-period: Startup grace period — failed checks don't count (default: 0s)
# --retries: How many times to retry before marking unhealthy (default: 3)
# curl -f: Fails silently on HTTP errors (4xx, 5xx)

# TCP port check (for databases or non-HTTP services)
HEALTHCHECK --interval=30s --timeout=3s \
  CMD nc -z localhost 5432 || exit 1
# nc -z: Check if port is open (no data sent)

# Node.js application check (when curl is not available)
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', (r) => { process.exit(r.statusCode === 200 ? 0 : 1) })" || exit 1

# Custom script (for complex checks)
COPY healthcheck.sh /usr/local/bin/healthcheck
RUN chmod +x /usr/local/bin/healthcheck
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD /usr/local/bin/healthcheck

# Disable inherited healthcheck (from base image)
HEALTHCHECK NONE

Complex Health Check Script

#!/bin/sh
# healthcheck.sh — Multiple checks in one script
set -e

# Check 1: HTTP endpoint response
http_status=$(curl -sL -w "%{http_code}" http://localhost:3000/health -o /dev/null --max-time 3)
if [ "$http_status" != "200" ]; then
  echo "UNHEALTHY: HTTP check failed with status $http_status"
  exit 1
fi

# Check 2: Database connectivity
if ! nc -z db 5432 2>/dev/null; then
  echo "UNHEALTHY: Cannot reach database on port 5432"
  exit 1
fi

# Check 3: Disk space (prevent crash due to full disk)
disk_usage=$(df /app | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$disk_usage" -gt 90 ]; then
  echo "UNHEALTHY: Disk usage too high: ${disk_usage}%"
  exit 1
fi

echo "HEALTHY: All checks passed"
exit 0

Health Check in Docker Compose

version: "3.8"

services:
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: secret
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s   # PostgreSQL takes time to start!

  api:
    image: myapi:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 40s
    depends_on:
      db:
        condition: service_healthy  # Wait for db to be HEALTHY, not just running!

  worker:
    image: myworker:latest
    depends_on:
      api:
        condition: service_healthy  # Start worker only when api is healthy
      db:
        condition: service_healthy

# depends_on with service_healthy = proper startup ordering!
# Without this: api starts before db is ready → connection error!

Checking Health Status

# docker ps shows health status
docker ps
# CONTAINER ID  IMAGE    STATUS
# abc123        myapp    Up 5 minutes (healthy)
# def456        mydb     Up 2 minutes (starting)
# ghi789        worker   Up 10 minutes (unhealthy)  ← Problem!

# Get health status
docker inspect --format '{{.State.Health.Status}}' myapp
# healthy

# Detailed health info with history
docker inspect --format '{{json .State.Health}}' myapp | jq
# {
#   "Status": "healthy",
#   "FailingStreak": 0,
#   "Log": [
#     {
#       "Start": "2024-01-15T...",
#       "End": "2024-01-15T...",
#       "ExitCode": 0,
#       "Output": "HEALTHY: All checks passed"
#     }
#   ]
# }

# Wait for container to be healthy (useful in scripts)
until [ "$(docker inspect --format '{{.State.Health.Status}}' myapp)" == "healthy" ]; do
  echo "Waiting for myapp to be healthy..."
  sleep 5
done
echo "myapp is healthy!"

Tip: Design a dedicated /health endpoint that returns HTTP 200 when healthy and 503 when unhealthy. It should check database connectivity, required external services, and application state. Response time should be under 100ms — the health check itself should not be slow.

Caution: If start_period is too short and your app starts slowly, the container will immediately be marked "unhealthy" and enter a restart loop. Java apps may take 30–60 seconds. Node.js typically takes 5–10 seconds. Set the value based on your app's actual startup time.

Note: Docker's HEALTHCHECK instruction does not work in Kubernetes — it is a Docker Compose feature. Kubernetes has its own livenessProbe, readinessProbe, and startupProbe. The concept is the same, but the syntax differs. When migrating to Kubernetes, convert Dockerfile HEALTHCHECKs to Kubernetes probes.

Development Workflows with Docker

Docker for Local Development — The Real Game Changer

Docker development environments ensure every developer — whether on Windows, Mac, or Linux — runs the exact same stack. New developer onboarding becomes: git clone && docker compose up — done in under 5 minutes.

Tip: Docker dev environments eliminate "works on my machine" issues permanently. Every developer gets an identical stack, regardless of operating system.

# docker-compose.dev.yml - Development setup
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.dev  # Dev dockerfile with hot reload
    volumes:
      - .:/app                    # Mount source code!
      - /app/node_modules         # Keep container's node_modules
    environment:
      NODE_ENV: development
      DEBUG: "true"
    ports:
      - "9229:9229"               # Node.js debug port
    command: npm run dev          # Hot reload command

  # Development tools
  mailhog:
    image: mailhog/mailhog        # Email testing
    ports:
      - "8025:8025"               # Web UI
      - "1025:1025"               # SMTP
    profiles:
      - dev-tools

  redis-commander:
    image: rediscommander/redis-commander
    environment:
      REDIS_HOSTS: "local:redis:6379"
    ports:
      - "8081:8081"
    profiles:
      - dev-tools

Use profiles to start dev tools only when needed: docker compose --profile dev-tools up

Dockerfile.dev — Node.js with Hot Reload

# Dockerfile.dev - Development only
FROM node:18-alpine

# Install nodemon for hot reload
RUN npm install -g nodemon

WORKDIR /app

# Copy package files
COPY package*.json ./
RUN npm install  # All deps including devDependencies

# Don't COPY source - it's mounted via volume!

EXPOSE 3000 9229

# Hot reload with nodemon
CMD ["nodemon", "--inspect=0.0.0.0:9229", "src/server.js"]

Development vs Production — Key Differences

Aspect	Development	Production
Base Image	node:18 (full)	node:18-alpine
devDependencies	Installed	Excluded
Source Code	Via bind mount	COPYed into image
Hot Reload	nodemon	Not needed
Debug Ports	9229 exposed	No debug
Image Size	~1GB acceptable	< 200MB target
Multi-stage Build	Not needed	Yes

Benefits of Docker for Development

Same Environment: Every developer runs an identical stack. "Works on my machine" becomes irrelevant.
Instant Onboarding: New developer joins: git clone && docker compose up = fully running stack in 5 minutes.
No Version Conflicts: Switch between Node 16, 18, 20 projects without juggling version managers. Each project is isolated.
Exact Production Parity: Test with the exact same database version (Postgres 16, MongoDB 7) as production.

Docker for React/Node Development

Note: This section covers CRA, Vite, Next.js, and full MERN stack setups with Docker, including real-world pitfalls that trip up beginners.

1. Dockerize Create React App (CRA) with Hot Reload

# Dockerfile.dev for React CRA
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install
# Don't copy source - mounted via volume
EXPOSE 3000
CMD ["npm", "start"]

# docker-compose.yml for React dev
services:
  react-app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - .:/app
      - /app/node_modules          # Prevent overwriting container's node_modules!
    ports:
      - "3000:3000"
    environment:
      - CHOKIDAR_USEPOLLING=true   # Required for file watching in Docker!
      - REACT_APP_API_URL=http://localhost:5000
    stdin_open: true               # Required for CRA
    tty: true

2. Dockerize Vite React App

Caution: Vite by default only listens on localhost inside the container. The --host 0.0.0.0 flag is required to access it from outside the container.

# Dockerfile.dev for Vite
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install
EXPOSE 5173
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0"]
# --host 0.0.0.0 is CRITICAL! Otherwise Vite only listens on localhost inside container

services:
  vite-app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - .:/app
      - /app/node_modules
    ports:
      - "5173:5173"
    environment:
      - VITE_API_URL=http://localhost:3001

3. Production React App with Nginx

# Dockerfile.prod - Complete production build
FROM node:18-alpine AS builder

WORKDIR /app

ARG REACT_APP_API_URL
ARG REACT_APP_ENV=production

ENV REACT_APP_API_URL=$REACT_APP_API_URL
ENV REACT_APP_ENV=$REACT_APP_ENV

COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build

# Production: Nginx serving
FROM nginx:1.25-alpine

# Custom nginx config for React SPA
COPY nginx.conf /etc/nginx/conf.d/default.conf

# Copy built React app
COPY --from=builder /app/build /usr/share/nginx/html

# Health endpoint
RUN echo "healthy" > /usr/share/nginx/html/health

EXPOSE 80

HEALTHCHECK --interval=30s --timeout=3s \
  CMD wget -qO- http://localhost/health || exit 1

CMD ["nginx", "-g", "daemon off;"]

server {
    listen 80;
    root /usr/share/nginx/html;
    index index.html;

    # React Router support — REQUIRED for SPAs!
    location / {
        try_files $uri $uri/ /index.html;
    }

    # API proxy (optional)
    location /api/ {
        proxy_pass http://api:5000/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Cache static assets aggressively
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff2)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }

    # Gzip compression
    gzip on;
    gzip_vary on;
    gzip_types text/plain text/css application/json application/javascript;
}

4. Next.js Production — Official Recommended Dockerfile

FROM node:18-alpine AS base
RUN apk add --no-cache libc6-compat
WORKDIR /app

FROM base AS deps
COPY package*.json ./
RUN npm ci

FROM base AS builder
COPY --from=deps /app/node_modules ./node_modules
COPY . .
ENV NEXT_TELEMETRY_DISABLED=1
ENV NODE_ENV=production
RUN npm run build

FROM base AS runner
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
RUN addgroup --system --gid 1001 nodejs && adduser --system --uid 1001 nextjs
COPY --from=builder /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
USER nextjs
EXPOSE 3000
CMD ["node", "server.js"]

5. Full MERN Stack Development Setup

# docker-compose.yml - Full MERN Dev Stack
services:
  # React Frontend
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile.dev
    volumes:
      - ./frontend:/app
      - /app/node_modules
    ports:
      - "3000:3000"
    environment:
      - REACT_APP_API_URL=http://localhost:5000
      - CHOKIDAR_USEPOLLING=true
    depends_on:
      - api

  # Node.js API
  api:
    build:
      context: ./api
      dockerfile: Dockerfile.dev
    volumes:
      - ./api:/app
      - /app/node_modules
    ports:
      - "5000:5000"
      - "9229:9229"   # Debug port
    environment:
      NODE_ENV: development
      MONGODB_URI: mongodb://mongodb:27017/myapp
      JWT_SECRET: dev-secret-key
    depends_on:
      - mongodb

  # MongoDB
  mongodb:
    image: mongo:7
    ports:
      - "27017:27017"
    volumes:
      - mongo_data:/data/db
    environment:
      MONGO_INITDB_DATABASE: myapp

  # Redis cache
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  # Mongo Express - GUI for MongoDB (localhost:8081)
  mongo-express:
    image: mongo-express
    ports:
      - "8081:8081"
    environment:
      ME_CONFIG_MONGODB_SERVER: mongodb
      ME_CONFIG_BASICAUTH_USERNAME: admin
      ME_CONFIG_BASICAUTH_PASSWORD: admin
    depends_on:
      - mongodb

volumes:
  mongo_data:

Common React + Docker Problems and Fixes

Problem	Cause	Fix
Hot reload not working	File watching broken in Docker	Set `CHOKIDAR_USEPOLLING=true`
Can't access Vite app	Vite binding to localhost only	Add `--host 0.0.0.0` to dev command
node_modules overwritten	Host volume masks container deps	Add `/app/node_modules` as anonymous volume
Build fails: env vars missing	REACT_APP_ prefix not passed at build time	Use `ARG` + `ENV` in Dockerfile
Very slow Docker builds	Large build context with node_modules	Add `node_modules` to `.dockerignore`

Debugging Dockerized Apps

Node.js Remote Debugging

You can debug a Node.js app running inside a container using Chrome DevTools or VS Code.

# Start Node.js container in debug mode
docker run -d \
  --name api-debug \
  -p 9229:9229 \
  -v $(pwd):/app \
  node:18 \
  node --inspect=0.0.0.0:9229 src/server.js

# Then open Chrome: chrome://inspect
# Your server will appear under "Remote Target"

// VS Code launch.json
{
  "configurations": [
    {
      "type": "node",
      "request": "attach",
      "name": "Docker: Attach to Node",
      "port": 9229,
      "address": "localhost",
      "localRoot": "${workspaceFolder}",
      "remoteRoot": "/app",
      "restart": true
    }
  ]
}

Python Debugging with debugpy

# Python debugging with debugpy
docker run -d \
  -p 5678:5678 \
  -v $(pwd):/app \
  python:3.11 \
  python -m debugpy --listen 0.0.0.0:5678 --wait-for-client app.py

# In VS Code, use the Python: Remote Attach configuration
# Port: 5678, Host: localhost

Network Debugging with netshoot

Tip: nicolaka/netshoot contains curl, tcpdump, netstat, nslookup, traceroute, dig — everything you need. Attach to any container's network namespace and debug immediately.

# Network debugging with netshoot
docker run -it --rm \
  --network container:myapp \
  nicolaka/netshoot

# Now inside myapp's network namespace with full tooling:
# curl, tcpdump, netstat, nslookup, traceroute, dig, etc.

# Check DNS resolution inside container network
docker run --rm --network mynetwork nicolaka/netshoot nslookup api

# Check if port is reachable between services
docker run --rm --network mynetwork nicolaka/netshoot nc -zv api 5000

# Capture TCP traffic
docker run --rm --network container:myapp nicolaka/netshoot tcpdump -i eth0 port 5000

General Container Debugging Commands

# Open a shell inside the container
docker exec -it myapp sh

# View logs with timestamps
docker logs -f --timestamps myapp

# Process list inside container
docker exec myapp ps aux

# Check environment variables
docker exec myapp env | grep -i api

# Full container config
docker inspect myapp | jq '.[0].NetworkSettings'

# Browse the filesystem
docker exec -it myapp find /app -name "*.log"

# Copy files out of container for inspection
docker cp myapp:/app/logs/error.log ./error.log

Production Readiness Checklist

Caution: Before going to production, verify all items below. Common mistakes in production Dockerfiles and compose files create security vulnerabilities, crashes, or performance issues. Bookmark this checklist.

Production Dockerfile Checklist

Check	Why It Matters	Status
Specific base image version (not `latest`)	Reproducible builds, no surprise updates	☐
Multi-stage build used	Smaller final image, no build tools in prod	☐
Non-root user configured	Security — limits blast radius of a breach	☐
.dockerignore present	Smaller build context, no secrets leaked	☐
HEALTHCHECK configured	Orchestrators can detect unhealthy containers	☐
WORKDIR explicitly set	Predictable file paths	☐
package*.json COPY before source COPY	Better layer caching	☐
No secrets in image/Dockerfile	Security — images can be inspected	☐
Image size < 500MB (ideally < 200MB)	Faster pulls, lower storage cost	☐
LABEL with version/maintainer info	Traceability in production	☐
Graceful shutdown handled (SIGTERM)	Zero-downtime deployments	☐

Graceful Shutdown in Node.js

When Docker stops a container, it sends SIGTERM first, then SIGKILL after 10 seconds. If you do not handle SIGTERM, in-flight requests are dropped.

// server.js - Handle SIGTERM for graceful shutdown
process.on('SIGTERM', () => {
  console.log('SIGTERM received, shutting down gracefully');
  server.close(() => {
    console.log('HTTP server closed');
    // Close DB connections
    mongoose.connection.close(false, () => {
      console.log('MongoDB disconnected');
      process.exit(0);
    });
  });

  // Force exit after 30 seconds
  setTimeout(() => {
    console.error('Could not close connections in time, forcing exit');
    process.exit(1);
  }, 30000);
});

Image Versioning Strategy

# Tagging strategy for production
IMAGE="myapp"
REGISTRY="123456789.dkr.ecr.ap-south-1.amazonaws.com"
GIT_TAG=$(git describe --tags --always)
GIT_SHA=$(git rev-parse --short HEAD)
DATE=$(date +%Y%m%d)

# Always push with multiple tags
docker build -t $IMAGE .
docker tag $IMAGE $REGISTRY/$IMAGE:$GIT_TAG       # v1.2.3
docker tag $IMAGE $REGISTRY/$IMAGE:$GIT_SHA       # abc1234
docker tag $IMAGE $REGISTRY/$IMAGE:$DATE          # 20240115
docker tag $IMAGE $REGISTRY/$IMAGE:latest         # latest (don't use in deploys!)

# Deploy using a specific version, not latest!
docker pull $REGISTRY/$IMAGE:v1.2.3

Caution: Never deploy using the :latest tag in production. If someone pushes and the service restarts, an unexpected version may be deployed. Always use a git SHA or semantic version tag for deployments.

Docker with CI/CD Pipelines

Complete GitHub Actions Workflow — Build, Test, Push

Before writing your pipeline, validate your workflow syntax with the GitHub Actions Validator to catch errors early.

# .github/workflows/docker-build-push.yml
name: Build and Push Docker Image

on:
  push:
    branches: [main]
    tags: ['v*']
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run tests
        run: |
          docker build --target test -t myapp:test .
          docker run --rm myapp:test npm test

  build-push:
    needs: test
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Setup Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata (tags, labels)
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=sha,prefix=sha-
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha          # Use GitHub Actions cache!
          cache-to: type=gha,mode=max
          platforms: linux/amd64,linux/arm64   # Multi-arch build

Deploy to AWS ECS via CI/CD

  deploy:
    needs: build-push
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-south-1

      - name: Download task definition
        run: |
          aws ecs describe-task-definition \
            --task-definition myapp \
            --query taskDefinition > task-definition.json

      - name: Update ECS task definition
        id: task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: task-definition.json
          container-name: myapp
          image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest

      - name: Deploy to ECS
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1
        with:
          task-definition: ${{ steps.task-def.outputs.task-definition }}
          service: myapp-service
          cluster: production
          wait-for-service-stability: true   # Wait for deploy to complete

If you are using GitLab CI instead, validate your .gitlab-ci.yml syntax with the GitLab CI Validator before pushing.

CI/CD Pipeline Flow

Code Push: Developer runs git push → GitHub Actions triggers automatically.
Test: Docker image builds with test target → tests run inside the container.
Build and Push: Multi-arch production image builds and pushes to registry with proper tags.
Deploy: ECS task definition updates → service performs a rolling deploy with zero downtime.

Note: "Our CI/CD pipeline runs on GitHub Actions. Every push to main automatically builds the Docker image, runs tests inside the container, then pushes to ECR and does an ECS rolling deploy — all automated. Manual production deployments never happen."

Monitoring Docker Containers

Docker Built-in Monitoring Commands

# Real-time stats — CPU, Memory, Network, IO
docker stats                    # All containers
docker stats myapp --no-stream  # One-time snapshot (no live update)

# Human-readable format
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

# Events stream — what is happening in the system
docker events --since 1h

# Filter events by type
docker events --filter type=container --filter event=die

# Logs with time range
docker logs --since 1h --until 30m myapp
docker logs --since "2024-01-15T10:00:00" myapp

# Formatted logs
docker logs --timestamps myapp 2>&1 | grep ERROR

Full Monitoring Stack: cAdvisor + Prometheus + Grafana

# monitoring/docker-compose.monitoring.yml
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=30d

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3001:3000"             # Port 3000 reserved for app
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-admin123}
    volumes:
      - grafana_data:/var/lib/grafana

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"

volumes:
  prometheus_data:
  grafana_data:

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'docker'
    static_configs:
      - targets: ['cadvisor:8080']    # Container metrics

  - job_name: 'myapp'
    static_configs:
      - targets: ['myapp:3000']
    metrics_path: '/metrics'          # App must expose a /metrics endpoint

What to Monitor

CPU Usage: Detect container CPU throttling. Consistently above 80% means scale out or raise resource limits.
Memory: Detect memory leaks. Check for OOM kills with docker events --filter event=oom.
Network I/O: Detect unusual traffic patterns. Track high egress costs in AWS.
Container Restarts: Frequent restarts indicate a crash loop. Check the RESTARTS column in docker ps.

Docker Orchestration Introduction

Docker Swarm vs Kubernetes

Feature	Docker Swarm	Kubernetes
Complexity	Simple	Complex
Setup Time	Minutes	Hours/Days
Auto-scaling	Basic	Advanced (HPA, VPA)
Ecosystem	Docker only	Massive (CNCF)
Rolling Updates	Yes	Yes
Self-healing	Yes	Yes
Multi-host	Yes	Yes
Industry Adoption (2026)	Declining/Legacy	Dominant Standard
Cloud Native Support	Limited	First-class
Learning Curve	Low	High

Note: Swarm is now mostly legacy. In 2026, Kubernetes is the industry standard. That said, mastering Docker Compose and core Docker concepts makes learning Kubernetes significantly easier — the concepts are the same, just with greater complexity.

When Do You Need Orchestration?

Scale: 10+ Containers: Docker Compose runs on a single machine. Deploying across multiple machines requires orchestration.
High Availability: If a node dies and the app automatically shifts to another node — that is orchestration.
Auto-scaling: Automatically scale containers up when traffic increases and scale down when it drops.
Zero-downtime Deploys: Rolling deployments where the old version keeps running until the new version is healthy.

Transition Path

Tip: Learning path: Docker basics → Docker Compose (multi-container) → Kubernetes (multi-host, production-grade). Each step builds on the previous one. Master Docker concepts and Kubernetes will feel natural. When you are ready to take the next step, the Kubernetes for DevOps guide picks up exactly where containers leave off.

# Docker Swarm quick demo (educational only)
docker swarm init                       # Initialize swarm on current node
docker service create --name web nginx  # Create a service
docker service scale web=5              # Scale to 5 replicas (auto-distributed!)
docker service ls                       # List services
docker service ps web                   # See where each replica runs
docker swarm leave --force              # Leave swarm

# Equivalent nginx deployment in Kubernetes
kubectl create deployment web --image=nginx
kubectl scale deployment web --replicas=5
kubectl get deployments
kubectl get pods                        # See 5 pods running

AWS Container Services

AWS Container Services Overview

Service	What It Is	When to Use
ECR	Elastic Container Registry — private image registry	Store Docker images for AWS deployments
ECS	Managed container orchestration service	Production workloads, teams familiar with Docker
Fargate	Serverless containers — no EC2 management	Simple deployments, variable load, no infrastructure operations
EKS	Managed Kubernetes service	Complex microservices, existing Kubernetes expertise
App Runner	Fully managed container platform	Simple web apps, fastest time-to-deploy

ECS Architecture

Understanding how ECS maps to Docker concepts:

Task Definition — Blueprint for container configuration, similar to a docker-compose.yml. Defines image, ports, environment variables, CPU and memory limits.
Task — A running instance of a task definition. Equivalent to a running container or group of containers.
Service — Maintains the desired task count, automatically replaces failed tasks, and integrates with a load balancer.
Cluster — A group of EC2 instances or Fargate compute resources on which tasks are scheduled.

ECS Task Definition — Complete Example

{
  "family": "myapp",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "executionRoleArn": "arn:aws:iam::123456789:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::123456789:role/ecsTaskRole",
  "containerDefinitions": [
    {
      "name": "myapp",
      "image": "123456789.dkr.ecr.ap-south-1.amazonaws.com/myapp:v1.0",
      "portMappings": [
        {
          "containerPort": 3000,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {
          "name": "NODE_ENV",
          "value": "production"
        }
      ],
      "secrets": [
        {
          "name": "DATABASE_URL",
          "valueFrom": "arn:aws:secretsmanager:ap-south-1:123456789:secret:myapp/db-url"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/myapp",
          "awslogs-region": "ap-south-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"],
        "interval": 30,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 60
      }
    }
  ]
}

Note: Never store secrets directly in environment variable definitions. Use AWS Secrets Manager or Parameter Store. ECS automatically injects secrets as environment variables at runtime — they are not stored in the image.

Practical AWS Deployment Workflows

Complete Deployment Script — React App to AWS Fargate

Tip: Copy this script into your project, set the variables to match your setup, and deploy your entire application with a single command: bash deploy.sh

#!/bin/bash
# deploy.sh - Complete deployment script

set -e   # Stop the script if any step fails

# Variables — set your own values here
APP_NAME="myreactapp"
AWS_ACCOUNT_ID="123456789012"
AWS_REGION="ap-south-1"
ECR_REGISTRY="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"
IMAGE_TAG=$(git rev-parse --short HEAD)   # Use Git SHA as tag

echo "Starting deployment of $APP_NAME:$IMAGE_TAG"

# Step 1: Build image
echo "Building Docker image..."
docker build \
  --build-arg REACT_APP_API_URL=https://api.myapp.com \
  -t $APP_NAME:$IMAGE_TAG \
  .

# Step 2: Authenticate with ECR
echo "Authenticating with ECR..."
aws ecr get-login-password --region $AWS_REGION | \
  docker login --username AWS --password-stdin $ECR_REGISTRY

# Step 3: Tag for ECR
docker tag $APP_NAME:$IMAGE_TAG $ECR_REGISTRY/$APP_NAME:$IMAGE_TAG
docker tag $APP_NAME:$IMAGE_TAG $ECR_REGISTRY/$APP_NAME:latest

# Step 4: Push to ECR
echo "Pushing to ECR..."
docker push $ECR_REGISTRY/$APP_NAME:$IMAGE_TAG
docker push $ECR_REGISTRY/$APP_NAME:latest

# Step 5: Update ECS service (force new deployment)
echo "Updating ECS service..."
aws ecs update-service \
  --cluster production \
  --service $APP_NAME-service \
  --force-new-deployment \
  --region $AWS_REGION

# Step 6: Wait for deployment to stabilize
echo "Waiting for deployment to stabilize..."
aws ecs wait services-stable \
  --cluster production \
  --services $APP_NAME-service \
  --region $AWS_REGION

echo "Deployment successful! Image: $ECR_REGISTRY/$APP_NAME:$IMAGE_TAG"

AWS Fargate with Application Load Balancer

# Create ECS cluster (Fargate mode)
aws ecs create-cluster --cluster-name production

# Create ALB target group
aws elbv2 create-target-group \
  --name myapp-tg \
  --protocol HTTP \
  --port 3000 \
  --vpc-id vpc-12345 \
  --target-type ip \              # Use 'ip' for Fargate, not 'instance'
  --health-check-path /health

# Create ECS service with Fargate — 2 replicas for high availability
aws ecs create-service \
  --cluster production \
  --service-name myapp-service \
  --task-definition myapp:1 \
  --desired-count 2 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-1,subnet-2],securityGroups=[sg-123],assignPublicIp=ENABLED}" \
  --load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:...,containerName=myapp,containerPort=3000"

ECR — Common Operations

# Create ECR repository
aws ecr create-repository \
  --repository-name myapp \
  --image-scanning-configuration scanOnPush=true \   # Enable vulnerability scanning
  --region ap-south-1

# List images in repository
aws ecr list-images --repository-name myapp --region ap-south-1

# Delete old images (cost management)
aws ecr batch-delete-image \
  --repository-name myapp \
  --image-ids imageTag=old-tag

# Lifecycle policy — automatically delete old images
aws ecr put-lifecycle-policy \
  --repository-name myapp \
  --lifecycle-policy-text '{
    "rules": [{
      "rulePriority": 1,
      "description": "Keep last 10 images",
      "selection": {
        "tagStatus": "any",
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": {"type": "expire"}
    }]
  }'

10 Hands-On Docker Projects

Tip: Build all of these projects and push them to GitHub. Each project can become a resume bullet point. Recruiters and interviewers want to see working code — that is far more impactful than simply claiming Docker knowledge.

Project 1: React Portfolio Site

Difficulty: Beginner | Stack: React, Nginx, Docker | Concepts: Multi-stage build, static file serving

Convert a React application into a production-ready Docker image using Nginx for serving and optimal caching.

# Stage 1: Build React app
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2: Serve with Nginx
FROM nginx:1.25-alpine
COPY --from=builder /app/build /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

docker build -t portfolio .
docker run -d -p 8080:80 --name portfolio portfolio
# Visit: http://localhost:8080

# Check image size
docker images portfolio

Tip: Resume bullet — "Containerized React portfolio using Docker multi-stage builds with Nginx, achieving 90%+ image size reduction vs non-optimized builds; deployed via CI/CD pipeline."

Project 2: Secure Node.js REST API

Difficulty: Beginner | Stack: Node.js, Express, Docker | Concepts: Non-root user, health checks, production dependencies only

FROM node:18-alpine

# Security: create a non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production    # Only production dependencies

COPY --chown=appuser:appgroup . .

USER appuser                    # Switch to non-root user

EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s \
  CMD node -e "require('http').get('http://localhost:3000/health', r => process.exit(r.statusCode === 200 ? 0 : 1))"

CMD ["node", "server.js"]

docker build -t node-api .
docker run -d -p 3000:3000 node-api

# Check health check status
docker inspect --format='{{.State.Health.Status}}' $(docker ps -q)

# Test the API
curl http://localhost:3000/health

Project 3: Python FastAPI Data App

Difficulty: Beginner | Stack: Python, FastAPI, Uvicorn | Concepts: Python containerization, pip caching

FROM python:3.11-slim           # slim = smaller than full python image

WORKDIR /app

# Copy requirements first for better layer caching
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

ENV PYTHONUNBUFFERED=1          # Python logs appear in Docker logs

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

docker build -t fastapi-app .
docker run -d -p 8000:8000 fastapi-app

# Interactive API docs
# http://localhost:8000/docs

Project 4: Full MERN Stack Application

Difficulty: Intermediate | Stack: React, Node.js, MongoDB, Redis | Concepts: Multi-service Compose, service discovery, volumes

# docker-compose.yml - Full MERN Stack
services:
  frontend:
    build:
      context: ./frontend
      args:
        - REACT_APP_API_URL=http://localhost:5000
    ports:
      - "3000:80"               # Production: nginx serving
    depends_on:
      - backend

  backend:
    build: ./backend
    ports:
      - "5000:5000"
    environment:
      MONGODB_URI: mongodb://mongodb:27017/mern_db
      REDIS_URL: redis://redis:6379
    depends_on:
      - mongodb
      - redis

  mongodb:
    image: mongo:7
    volumes:
      - mongo_data:/data/db

  redis:
    image: redis:7-alpine

  mongo-express:              # MongoDB GUI — development only
    image: mongo-express
    ports:
      - "8081:8081"
    environment:
      ME_CONFIG_MONGODB_SERVER: mongodb
    depends_on:
      - mongodb

volumes:
  mongo_data:

# Start the complete MERN stack
docker compose up -d --build

# Check all services
docker compose ps

# Follow API logs
docker compose logs -f backend

# Access points:
# Frontend: http://localhost:3000
# Backend API: http://localhost:5000
# MongoDB GUI: http://localhost:8081

# Stop the stack (data is preserved)
docker compose down

Tip: Resume bullet — "Containerized full MERN stack using Docker Compose with service discovery, named volumes for data persistence, and Mongo Express GUI for development; reduced environment setup time from 2 hours to 5 minutes."

Project 5: WordPress + MySQL — Classic Stack

Difficulty: Intermediate | Stack: WordPress, MySQL 8 | Concepts: Health checks with conditions, restart policies, named volumes

services:
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: rootpassword
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpresspassword
    volumes:
      - mysql_data:/var/lib/mysql
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 10s
      retries: 5
      start_period: 30s

  wordpress:
    image: wordpress:latest
    ports:
      - "8080:80"
    environment:
      WORDPRESS_DB_HOST: mysql:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpresspassword
      WORDPRESS_DB_NAME: wordpress
    volumes:
      - wordpress_data:/var/www/html
    depends_on:
      mysql:
        condition: service_healthy    # Wait until MySQL is healthy
    restart: unless-stopped

volumes:
  mysql_data:
  wordpress_data:

docker compose up -d
# MySQL health check will be satisfied before WordPress starts
docker compose ps

# Visit: http://localhost:8080
# Complete the WordPress setup wizard

Project 6: Microservices with Nginx API Gateway

Difficulty: Intermediate | Stack: Node.js microservices, Nginx gateway, PostgreSQL | Concepts: API gateway pattern, service isolation, multiple databases

services:
  # Single entry point — all traffic routes through here
  api-gateway:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx/gateway.conf:/etc/nginx/conf.d/default.conf
    depends_on:
      - user-service
      - product-service

  user-service:
    build: ./user-service
    environment:
      DB_URL: postgresql://postgres:password@user-db:5432/users
    # No ports exposed externally — only accessible through the gateway
    depends_on:
      - user-db

  product-service:
    build: ./product-service
    environment:
      DB_URL: postgresql://postgres:password@product-db:5432/products
    depends_on:
      - product-db

  # Each service has its own database
  user-db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: users
      POSTGRES_PASSWORD: password
    volumes:
      - user_db_data:/var/lib/postgresql/data

  product-db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: products
      POSTGRES_PASSWORD: password
    volumes:
      - product_db_data:/var/lib/postgresql/data

volumes:
  user_db_data:
  product_db_data:

Tip: Resume bullet — "Designed and containerized microservices architecture with Nginx API gateway, service isolation, and per-service databases using Docker Compose; demonstrating event-driven service communication patterns."

Project 7: Image Size Optimization Challenge

Difficulty: Intermediate | Goal: 1.2 GB → under 100 MB | Concepts: Multi-stage builds, Alpine, production deps, .dockerignore

# BEFORE: Naive approach — DO NOT USE in production
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y nodejs npm
COPY . /app
WORKDIR /app
RUN npm install              # All deps including devDependencies
# Final image size: ~1.2 GB

# AFTER: Optimized multi-stage build
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci                   # All deps for building
COPY . .
RUN npm run build            # Compile/transpile

FROM node:18-alpine          # Fresh slim base
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production # Production deps only
COPY --from=builder /app/dist ./dist
USER node                    # Non-root
CMD ["node", "dist/server.js"]
# Final image size: ~85 MB (93% reduction)

# Compare before and after sizes
docker build -f Dockerfile.before -t app:before .
docker build -f Dockerfile.after -t app:after .
docker images | grep app

# Detailed layer analysis
docker history app:after

# Layer-by-layer analysis with dive
docker run --rm -it \
  -v /var/run/docker.sock:/var/run/docker.sock \
  wagoodman/dive app:after

Project 8: Full Monitoring Stack

Difficulty: Advanced | Stack: App + Prometheus + Grafana + cAdvisor | Concepts: Metrics collection, dashboards, alerting

services:
  app:
    build: .
    ports:
      - "3000:3000"
    # App must expose a /metrics endpoint (use prom-client npm package)

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus

  grafana:
    image: grafana/grafana
    ports:
      - "3001:3000"            # Use 3001 to avoid conflict with the app
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus

  cadvisor:
    image: gcr.io/cadvisor/cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
    ports:
      - "8080:8080"

volumes:
  prometheus_data:
  grafana_data:

docker compose up -d

# Access points:
# App: http://localhost:3000
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3001 (admin/admin)
# cAdvisor: http://localhost:8080

# In Grafana:
# 1. Add data source: Prometheus, URL = http://prometheus:9090
# 2. Import dashboard: ID 893 (Docker monitoring)
# 3. Watch container metrics in real time

Project 9: Complete CI/CD Pipeline

Difficulty: Advanced | Stack: GitHub Actions, Docker Hub, SSH Deploy | Concepts: Automated build, test, push, deploy

name: CI/CD Pipeline

on:
  push:
    branches: [main]
    tags: ['v*']

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run tests in Docker
        run: |
          docker build --target test -t test-image .
          docker run --rm test-image npm test

  build-push:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/myapp:latest
            ${{ secrets.DOCKERHUB_USERNAME }}/myapp:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy:
    needs: build-push
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to server via SSH
        uses: appleboy/ssh-action@v1.0.0
        with:
          host: ${{ secrets.SERVER_HOST }}
          username: ubuntu
          key: ${{ secrets.SSH_KEY }}
          script: |
            docker pull ${{ secrets.DOCKERHUB_USERNAME }}/myapp:latest
            docker stop myapp || true
            docker rm myapp || true
            docker run -d \
              --name myapp \
              --restart unless-stopped \
              -p 80:3000 \
              ${{ secrets.DOCKERHUB_USERNAME }}/myapp:latest
            echo "Deployment complete!"

Tip: Resume bullet — "Implemented end-to-end CI/CD pipeline using GitHub Actions: automated Docker image build and test on every push, Docker Hub publish with semantic versioning, and zero-touch SSH deployment to production server."

Project 10: Full Observability Stack (ELK + Jaeger + Prometheus)

Difficulty: Advanced | Stack: OpenTelemetry, Jaeger, Prometheus, ELK Stack | Concepts: Metrics + Tracing + Logging — the three pillars of observability

services:
  # Your application
  app:
    build: .
    environment:
      OTEL_SERVICE_NAME: myapp
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
    depends_on:
      - otel-collector

  # OpenTelemetry Collector — central data routing hub
  otel-collector:
    image: otel/opentelemetry-collector-contrib
    volumes:
      - ./otel-config.yml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4318:4318"   # OTLP HTTP

  # Distributed tracing — request flows across services
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"   # Jaeger UI

  # Metrics — numeric time-series data
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  # Dashboards — metrics visualization
  grafana:
    image: grafana/grafana
    ports:
      - "3001:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin123

  # Logs — structured log storage
  elasticsearch:
    image: elasticsearch:8.11.0
    environment:
      discovery.type: single-node
      xpack.security.enabled: "false"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"

  # Log visualization
  kibana:
    image: kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    depends_on:
      - elasticsearch

volumes:
  es_data:

# Start full observability stack
docker compose up -d

# Wait for Elasticsearch to finish starting
docker compose logs -f elasticsearch | grep "started"

# Access points:
# App:          http://localhost:3000
# Jaeger UI:    http://localhost:16686  (request traces)
# Prometheus:   http://localhost:9090   (raw metrics)
# Grafana:      http://localhost:3001   (metric dashboards)
# Kibana:       http://localhost:5601   (log analysis)

# Generate some traffic
for i in {1..20}; do curl http://localhost:3000/api/users; done

Tip: Resume bullet — "Implemented complete observability stack using Docker Compose: OpenTelemetry instrumentation, Jaeger distributed tracing, Prometheus + Grafana metrics dashboards, and ELK stack log aggregation — full visibility into system behavior."

Projects Summary — Difficulty Progression

#	Project	Difficulty	Key Skill	Resume Impact
1	React Portfolio	Beginner	Multi-stage build	High
2	Node.js API	Beginner	Security, health checks	High
3	Python FastAPI	Beginner	Python containerization	Medium
4	MERN Stack	Intermediate	Multi-service Compose	Very High
5	WordPress + MySQL	Intermediate	Health check conditions	Medium
6	Microservices	Intermediate	API gateway pattern	Very High
7	Size Optimization	Intermediate	Layer optimization	High
8	Monitoring Stack	Advanced	Prometheus + Grafana	Very High
9	CI/CD Pipeline	Advanced	GitHub Actions	Very High
10	Observability Stack	Advanced	OTel + ELK + Jaeger	Exceptional

Common Errors and Solutions

The following errors are encountered by almost every Docker developer at some point. Each includes the root cause, solution, and prevention guidance.

Error: Cannot connect to the Docker daemon

Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Root Cause: The Docker daemon (dockerd) has not started or has crashed.

# Check daemon status
sudo systemctl status docker

# Start the daemon
sudo systemctl start docker

# Enable on boot
sudo systemctl enable docker

# Verify
docker info

Prevention: Always enable the Docker service on boot with systemctl enable docker. On Mac, ensure Docker Desktop is running before using the CLI.

Error: Permission denied on Docker socket

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

Root Cause: The current user is not a member of the docker group.

# Add user to docker group
sudo usermod -aG docker $USER

# Apply group change immediately (or log out and back in)
newgrp docker

# Verify
docker run hello-world

Prevention: Add the user to the docker group during initial Docker installation. In CI/CD pipelines, add the runner user to the docker group.

Error: Port is already allocated

docker: Error response from daemon: driver failed programming external connectivity on endpoint myapp: Bind for 0.0.0.0:8080 failed: port is already allocated.

Root Cause: Host port 8080 is already in use by another process or container.

# Find what is using the port
sudo lsof -i :8080
sudo ss -tulpn | grep :8080

# Kill the process (replace PID)
kill -9 <PID>

# OR use a different host port
docker run -p 8081:80 nginx

# Find running containers using the port
docker ps | grep 8080

Prevention: Use environment variables for port configuration. Run docker ps before starting new containers.

Error: No space left on device

ERROR: failed to solve: ... write /var/lib/docker/...: no space left on device

Root Cause: The Docker daemon's disk space is exhausted — old images, containers, and volumes have consumed all available space.

# Check disk usage
df -h
docker system df

# Full cleanup (removes everything unused)
docker system prune -a --volumes

# Selective cleanup
docker image prune -a    # Remove unused images
docker container prune   # Remove stopped containers
docker volume prune      # Remove unused volumes

Prevention: Set up a cron job for docker system prune. Monitor /var/lib/docker size. Use multi-stage builds to keep images small.

Error: Container exits immediately

Root Cause: The container's main process (CMD/ENTRYPOINT) completed, or it crashed with an error.

# Check logs first
docker logs container_id
docker logs container_id --tail 50

# Check exit code
docker inspect --format '{{.State.ExitCode}}' container_id

# Run interactively to debug
docker run -it myapp bash

# Bad: CMD nginx (exits immediately — starts as daemon)
# Good: CMD ["nginx", "-g", "daemon off;"]

# Override entrypoint for debugging
docker run -it --entrypoint bash myapp

Prevention: Use foreground commands in CMD. Test CMD locally first. Add proper health checks.

Error: Pull access denied

Error response from daemon: pull access denied for myapp, repository does not exist or may require 'docker login'

Root Cause: The image name is incorrect, authentication is missing for a private registry, or the image does not exist.

# Login to Docker Hub
docker login

# Login to a private registry
docker login registry.mycompany.com -u username -p password

# Check exact image name and tag
docker pull myapp:latest

# Full path required for private registries
docker pull registry.mycompany.com/team/myapp:v1.0

Prevention: Use full image paths. Store credentials in Docker credential stores, not plain text. Set up registry mirrors for CI/CD.

Error: Build context is extremely large

Sending build context to Docker daemon  2.5GB

Root Cause: A .dockerignore file is missing, so Docker sends the entire project directory (including node_modules, .git, etc.) to the daemon.

# Create .dockerignore file
cat > .dockerignore << 'EOF'
node_modules/
dist/
build/
.git/
.env
.env.local
*.log
npm-debug.log*
.DS_Store
coverage/
.nyc_output
EOF

# Verify build context size after
docker build --no-cache . 2>&1 | head -5

Prevention: Always create a .dockerignore when starting a new Docker project. Commit it to version control alongside your code.

Error: Docker Hub rate limit exceeded

Error response from daemon: toomanyrequests: You have reached your pull rate limit.

Root Cause: Docker Hub anonymous pull limit has been hit (100 pulls per 6 hours unauthenticated, 200 for free accounts).

# Authenticate first to increase the limit
docker login
docker pull nginx

# OR configure a registry mirror in /etc/docker/daemon.json:
{
  "registry-mirrors": ["https://mirror.gcr.io"]
}

# Restart Docker after configuration change
sudo systemctl restart docker

Prevention: Always authenticate in CI/CD. Use ECR or GCR as a pull-through cache.

Error: Container OOMKilled — exit code 137

Root Cause: The container exceeded its memory limit. The Linux OOM Killer forcefully terminated the process. Exit code 137 = 128 + 9 (SIGKILL).

# Check if OOM killed
docker inspect myapp | jq '.[0].State.OOMKilled'

# Check kernel logs for OOM
dmesg | grep -i "oom\|killed process"

# Increase memory limit
docker run --memory=1g --memory-swap=2g myapp

# Monitor memory usage
docker stats myapp

Prevention: Profile application memory in development. Set realistic limits based on actual usage plus a 20% buffer. Add memory monitoring alerts in production.

Error: ECONNREFUSED connecting to database

Error: connect ECONNREFUSED 127.0.0.1:5432

Root Cause: The application inside the container is using 127.0.0.1 or localhost to connect to the database. Inside a container, localhost refers to the container itself — not another container.

# WRONG - localhost inside container = the container itself
DB_HOST=localhost  # ❌
DB_HOST=127.0.0.1  # ❌

# CORRECT - use the container name or service name
DB_HOST=postgres   # ✅ (with docker compose)
DB_HOST=db         # ✅ (compose service name)

# Verify connectivity
docker exec myapp ping postgres
docker exec myapp nslookup postgres

Prevention: Never hardcode localhost for inter-container communication. Use environment variables for hostnames. Use user-defined networks for DNS resolution.

Error: Volume mount permission denied

Error: Permission denied: '/app/data'

Root Cause: The container process user does not own the host volume directory.

# Option 1: Match user/group
docker run -u $(id -u):$(id -g) -v $(pwd)/data:/app/data myapp

# Option 2: Fix host directory permissions
sudo chown -R 1000:1000 ./data

# Option 3: Fix permissions in Dockerfile
RUN mkdir -p /app/data && chown -R node:node /app/data
USER node

# Option 4: Use named volume (Docker manages permissions)
docker run -v myapp_data:/app/data myapp

Error: DNS resolution failure between containers

Error: getaddrinfo ENOTFOUND api-service

Root Cause: The default bridge network does not provide automatic DNS resolution between containers. Only user-defined networks include the embedded DNS server.

# Create a user-defined network
docker network create app-network

# Run both containers on the same network
docker run --name api-service --network app-network myapi
docker run --name frontend --network app-network myfrontend

# Now 'api-service' is resolvable by name
docker exec frontend curl http://api-service:3000

Prevention: Always use user-defined networks for multi-container applications. Docker Compose sets this up automatically.

Error: COPY file not found in build context

COPY failed: file not found in build context or excluded by .dockerignore: stat package.json: file not found

Root Cause: The file does not exist in the build context, .dockerignore has excluded it, or the Dockerfile is being run from the wrong directory.

# Check if file exists
ls -la package.json

# Check .dockerignore isn't excluding it
cat .dockerignore | grep package

# Build from the correct directory
docker build -t myapp .   # note the dot — build context = current directory

# To build from a parent directory
docker build -t myapp -f ./subdir/Dockerfile .

Error: exec format error

exec /docker-entrypoint.sh: exec format error

Root Cause: The image was built for a different CPU architecture (e.g., built on an M1 Mac for arm64, then run on a linux/amd64 server).

# Check image platform
docker inspect myapp | jq '.[0].Architecture'

# Build for a specific platform
docker build --platform linux/amd64 -t myapp .

# Build a multi-platform image using buildx
docker buildx create --use
docker buildx build --platform linux/amd64,linux/arm64 \
  -t myapp:latest --push .

Prevention: Always use --platform linux/amd64 when building for production servers from an M1/M2 Mac.

Error: Cache keeps invalidating — npm install runs every time

Root Cause: COPY . . is placed before RUN npm install, so any file change invalidates the cache and forces npm install to re-run.

# WRONG - cache breaks on every file change
FROM node:18
WORKDIR /app
COPY . .          # ❌ Any change = cache miss
RUN npm install   # Re-runs every time
CMD ["node", "server.js"]

# CORRECT - npm install only re-runs when package.json changes
FROM node:18
WORKDIR /app
COPY package*.json ./      # ✅ Only copy dependency manifest
RUN npm ci                 # Only runs when package.json changes
COPY . .                   # Source code (frequently changing)
CMD ["node", "server.js"]

Prevention: Always separate dependency installation from source code copying. This is one of the most important Dockerfile optimizations.

Error: No configuration file provided (Docker Compose)

no configuration file provided: not found

Root Cause: Docker Compose cannot find a docker-compose.yml or compose.yml in the current directory.

# Check the current directory
ls -la | grep -i compose

# Navigate to the correct directory and run again
cd /path/to/your/project
docker compose up

# OR specify the file explicitly
docker compose -f /path/to/docker-compose.yml up

# Compose searches for these files in order:
# 1. compose.yaml
# 2. compose.yml
# 3. docker-compose.yaml
# 4. docker-compose.yml

Error: Container status is unhealthy

Root Cause: The HEALTHCHECK command is failing. The application has not properly started, or the health endpoint is not responding.

# Inspect health check output
docker inspect --format '{{json .State.Health}}' myapp | jq .

# See the last 5 health check results
docker inspect myapp | jq '.[0].State.Health.Log'

# Manually run the health check command
docker exec myapp curl -f http://localhost:3000/health

# Temporarily disable to debug
docker run --no-healthcheck myapp

Prevention: Test health check commands manually before adding them to the Dockerfile. Use --start-period for applications that take time to start. Implement a /health endpoint in your application.

Error: OOMKilled on Python — ModuleNotFoundError

ModuleNotFoundError: No module named 'flask'

Root Cause: requirements.txt was not copied correctly, or pip install ran in the wrong directory.

# Correct Dockerfile order
FROM python:3.11-slim
WORKDIR /app                           # Set WORKDIR first
COPY requirements.txt .                # Copy requirements
RUN pip install --no-cache-dir -r requirements.txt  # Install
COPY . .                               # Then copy code
CMD ["python", "app.py"]

# Debug: check if packages are installed
docker exec -it myapp pip list | grep flask

# Rebuild without cache
docker build --no-cache -t myapp .

Error: nginx bind permission denied on port 80

nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

Root Cause: A non-root user cannot bind to privileged ports below 1024.

# Option 1: Use port 8080 (non-privileged)
# In nginx.conf: listen 8080;
docker run -p 80:8080 nginx-custom

# Option 2: Add NET_BIND_SERVICE capability
docker run --cap-add NET_BIND_SERVICE -p 80:80 nginx-custom

# Option 3: Use the nginx-unprivileged image (uses port 8080 by default)
FROM nginxinc/nginx-unprivileged

Tip: When using a non-root user with nginx, always configure nginx.conf to listen on port 8080 or higher. Use nginxinc/nginx-unprivileged as the base image for a secure, non-root nginx setup.

Tip: docker logs and docker inspect are your best debugging tools. 90% of Docker errors can be diagnosed with just these two commands.

Debugging Methodology

Systematic Debugging — Step by Step

When something stops working, follow this systematic checklist. Do not make random changes without first diagnosing the problem.

Note: Approach Docker debugging like a doctor: check symptoms first, then diagnose, then treat. Never make changes without a clear diagnosis.

Step 1: Is the Container Running?

# Show running containers
docker ps

# Show ALL containers (including stopped)
docker ps -a

# Pretty format with useful columns
docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Filter by container name
docker ps -a --filter name=myapp
# Check the STATUS column: "Exited (1) 2 minutes ago" = crashed
# "Exited (0) 2 minutes ago" = completed normally

Step 2: What Do the Logs Say?

# Last 100 lines of logs
docker logs --tail 100 myapp

# Follow logs in real time
docker logs -f myapp

# Logs from the last 1 hour
docker logs --since 1h myapp

# Logs with timestamps
docker logs -t myapp

# Filter for errors
docker logs myapp 2>&1 | grep -i "error\|fatal\|critical"

# Logs persist even for stopped containers until the container is removed
docker logs myapp

Step 3: What Is the Exit Code?

# Get exit code
docker inspect --format '{{.State.ExitCode}}' myapp

# Get full state information
docker inspect --format '{{json .State}}' myapp | jq .

# Exit code reference:
# 0   = Normal exit (success, or CMD completed)
# 1   = General error (application error)
# 2   = Misuse of shell command
# 125 = Docker daemon error
# 126 = Container command cannot be invoked
# 127 = Container command not found
# 130 = Container terminated by Ctrl+C (SIGINT)
# 137 = SIGKILL (OOM kill or docker kill)
# 139 = Segmentation fault
# 143 = SIGTERM (graceful shutdown)

Step 4: Can You Enter the Container?

# Enter a running container (bash for ubuntu/debian)
docker exec -it myapp bash

# For Alpine Linux (no bash by default)
docker exec -it myapp sh

# Run specific commands inside
docker exec myapp ls -la /app
docker exec myapp cat /app/config.json
docker exec myapp env | grep DB_

# If the container keeps crashing, override entrypoint to debug
docker run -it --entrypoint bash myapp
docker run -it --entrypoint sh myapp  # Alpine

Step 5: Check Resource Usage

# Live stats for a specific container
docker stats myapp

# Live stats for all containers
docker stats

# One-time snapshot (no streaming)
docker stats --no-stream

# Check process list inside container
docker top myapp

# Check resource limits set on container
docker inspect --format '{{json .HostConfig.Memory}}' myapp

Step 6: Network Inspection

# Inspect a network
docker network inspect app-network

# Check container's network settings
docker inspect --format '{{json .NetworkSettings}}' myapp | jq .

# Test DNS resolution from inside the container
docker exec myapp nslookup postgres
docker exec myapp nslookup google.com

# Test connectivity
docker exec myapp curl -v http://api-service:3000/health
docker exec myapp ping -c 3 postgres

# Check which networks the container is on
docker inspect --format '{{json .NetworkSettings.Networks}}' myapp | jq 'keys'

# Show port mappings
docker port myapp

Step 7: Check Events

# All events in the last hour
docker events --since 1h

# Filter by container
docker events --since 1h --filter container=myapp

# Filter by event type
docker events --filter type=container --filter event=oom

# Watch events live
docker events

# Events by image
docker events --filter image=myapp:latest

Step 8: Full Configuration Inspect

# Full JSON inspection
docker inspect myapp | jq .

# Specific fields
docker inspect myapp | jq '.[0].Config.Env'        # Environment variables
docker inspect myapp | jq '.[0].Mounts'             # Volume mounts
docker inspect myapp | jq '.[0].HostConfig'         # Host config (ports, limits)
docker inspect myapp | jq '.[0].NetworkSettings'    # Network information

# Image inspection
docker inspect myimage:tag | jq '.[0].Config.Cmd'
docker inspect myimage:tag | jq '.[0].Config.Entrypoint'

Note: Do not include debugging tools in production images — they increase image size and expand the attack surface. Instead, maintain a separate debug image built from your production image plus debugging tools.

Quick Diagnosis Reference

Problem Symptom	First Command	Common Fix
Container not running	`docker ps -a`	Check exit code, check logs
App not responding	`docker logs myapp`	Look for errors in startup output
Containers cannot communicate	`docker network inspect`	Add both to the same user-defined network
Data not persisting	`docker inspect` (Mounts section)	Add a volume mount
High memory usage	`docker stats`	Add `--memory` limit
Slow builds	`docker build --no-cache`	Fix Dockerfile layer order
Port not accessible	`docker port myapp`	Add `-p` flag, check binding
Container keeps restarting	`docker logs myapp --tail 20`	Fix application error, check exit code

Common Docker Interview Questions

The following 15 questions cover the most important Docker topics you will encounter in engineering interviews. Each includes a concise answer, a detailed explanation, and common pitfalls to avoid.

Q1. What is the difference between a Docker image and a container?

A Docker image is an immutable, read-only template made of layered filesystems containing everything needed to run an application. A container is a running instance of that image — it adds a thin writable layer on top. Multiple containers can run from the same image simultaneously.

Key point: Image = blueprint (immutable). Container = running instance (has a writable layer).
Red flag: Saying "image and container are the same thing" immediately signals a foundational misunderstanding.

Q2. How does Docker image layer caching work?

Docker caches each instruction's result as a layer. On rebuild, if an instruction and its inputs are unchanged, Docker reuses the cached layer and skips re-execution. Once any layer is invalidated, all subsequent layers are rebuilt. This is why Dockerfile instruction order matters — place rarely-changing instructions (install dependencies) before frequently-changing ones (copy source code).

# Good - npm install only re-runs when package.json changes
COPY package*.json ./
RUN npm ci
COPY . .   # source changes only invalidate this layer and below

Red flag: Not structuring Dockerfiles to exploit layer caching. A 3-minute npm install running on every CI build is an immediate code review concern.

Q3. What are multi-stage builds and why use them?

Multi-stage builds use multiple FROM statements in one Dockerfile. You compile/build in a full environment, then COPY --from=stage only the compiled artifacts into a minimal runtime image. This keeps build tools out of the final image, dramatically reducing size and attack surface.

FROM golang:1.21 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o server .

FROM scratch AS runtime
COPY --from=builder /app/server /server
CMD ["/server"]
# Result: ~10 MB image vs ~800 MB with full Go toolchain

Red flag: Not knowing that you can have more than two stages, or that you can target specific stages with docker build --target=stagename.

Q4. What is the difference between CMD and ENTRYPOINT?

ENTRYPOINT sets the fixed executable that always runs. CMD provides default arguments that can be overridden at runtime. Together: ENTRYPOINT is "what to run" and CMD is "with what default arguments." CMD alone is fully overridable by passing arguments to docker run.

ENTRYPOINT ["python", "app.py"]   # always runs python app.py
CMD ["--port", "8080"]             # default arg, overridable

# docker run myapp                  → python app.py --port 8080
# docker run myapp --port 9090      → python app.py --port 9090

Red flag: Using shell form for CMD/ENTRYPOINT (CMD node server.js). Shell form runs your command via /bin/sh -c, meaning the shell is PID 1 — not your application. SIGTERM is not forwarded, breaking graceful shutdown.

Q5. What is the difference between volumes and bind mounts?

Bind mounts link a specific host path into the container — you control the location. Named volumes are managed by Docker, stored at /var/lib/docker/volumes/. Named volumes are preferred in production for persistent data (databases). Bind mounts are preferred in development for live code reloading.

Type	Storage	Best For
Named Volume	Docker-managed	DB data, persistent app data
Bind Mount	Specific host path	Development hot reload, config files
tmpfs	Host RAM (never on disk)	Secrets, temporary cache

Red flag: Using bind mounts in production — they create host-container coupling and can expose host files if misconfigured.

Q6. Explain Docker networking — bridge, host, and overlay.

bridge (default): Isolated virtual network on the host. Containers communicate via container names on user-defined bridge networks (with automatic DNS). Requires -p to expose ports to the host.
host: Container shares the host's network stack directly. No NAT overhead, but no network isolation. Only works on Linux.
overlay: Spans multiple Docker hosts using VXLAN tunneling. Required for Docker Swarm multi-host communication.

The default bridge network does not provide automatic DNS — containers must use IPs. User-defined bridge networks resolve container names via Docker's embedded DNS at 127.0.0.11.

Red flag: Using the default bridge network in production. Default bridge has no DNS and all containers on it can communicate with each other — no isolation.

Q7. What are Dockerfile best practices?

Key best practices:

Use minimal base images (alpine, distroless)
Use multi-stage builds for compiled languages and build-step frameworks
Exploit layer caching — copy dependency manifests before source code
Use .dockerignore to exclude node_modules, .git, .env
Run as a non-root user (USER node)
Use exec form for CMD and ENTRYPOINT (["node", "server.js"])
Combine RUN commands to minimize layers, clean up in the same instruction
Add HEALTHCHECK for production images
Pin base image versions — avoid :latest
Never pass secrets via ARG or ENV

Q8. What does depends_on do in Docker Compose — does it guarantee the service is ready?

depends_on controls startup order — it ensures the dependent container starts before the dependent service. By default it only waits for the container to start (service_started), not for the application inside to be ready.

To actually wait for a service to be ready, combine depends_on with condition: service_healthy and a proper healthcheck:

api:
  depends_on:
    postgres:
      condition: service_healthy   # waits for healthcheck to pass

postgres:
  image: postgres:15
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U postgres"]
    interval: 5s
    retries: 5

Red flag: Using depends_on without condition: service_healthy and then being surprised when the application crashes on startup because the database was not ready.

Q9. How do you secure a Docker container in production?

A hardened container uses multiple layers of defense:

docker run \
  --user 1000:1000 \              # non-root user
  --cap-drop=ALL \                # drop all capabilities
  --cap-add=NET_BIND_SERVICE \    # add only what is needed
  --read-only \                   # read-only root filesystem
  --tmpfs /tmp:size=50m \         # writable RAM-backed temp
  --no-new-privileges \           # prevent privilege escalation
  --security-opt seccomp=/etc/docker/seccomp.json \
  myapp

Checklist: non-root USER, no --privileged, --read-only filesystem, drop all capabilities, minimal base image, no docker.sock mount, scan with Trivy, use BuildKit secret mounts (never ARG/ENV for secrets).

Q10. What is the difference between docker stop and docker kill?

docker stop sends SIGTERM to allow graceful shutdown, waits 10 seconds (configurable), then sends SIGKILL. docker kill immediately sends SIGKILL without waiting.

Graceful shutdown matters: SIGTERM lets applications finish active requests, close database connections, and flush logs. Applications should handle SIGTERM explicitly. Always use docker stop in production scripts — docker kill can cause data corruption if the app is mid-write.

Q11. What is BuildKit and what advantages does it provide?

BuildKit is Docker's next-generation build engine (default since Docker 23.0). Key advantages over the classic builder:

Parallel stage execution in multi-stage builds
Improved layer caching (including registry-based cache)
Secret mounts (RUN --mount=type=secret) — secrets are never stored in any layer
SSH agent forwarding for private repositories
Multi-platform builds with buildx
Significantly faster build times

# BuildKit secret mount (secret is never baked into image)
RUN --mount=type=secret,id=npmrc \
    cp /run/secrets/npmrc ~/.npmrc && \
    npm install && \
    rm ~/.npmrc

docker build --secret id=npmrc,src=$HOME/.npmrc .

Red flag: Not knowing BuildKit secret mounts exist, and instead passing secrets via ARG or ENV, which appear in docker history.

Q12. How do you implement zero-downtime deployments with Docker?

Use blue-green or rolling update strategies. The key principle: start the new container and verify it is healthy before stopping the old one — never the reverse.

# Blue-green pattern
# 1. Start new (green) container
docker run -d --name myapp-green --network app myapp:new

# 2. Health check green
docker exec myapp-green curl -sf http://localhost:3000/health

# 3. Switch traffic (update nginx upstream, then reload)
# nginx -s reload

# 4. Stop old (blue)
docker stop myapp-blue && docker rm myapp-blue

In Kubernetes, use rolling updates with maxUnavailable: 0 and maxSurge: 1 combined with readiness probes.

Q13. How do you handle secrets in Docker?

Never store secrets in ARG, ENV, or files copied into an image. The correct approaches are:

Build-time secrets: Use BuildKit --mount=type=secret — the secret is available during the build RUN instruction but is never stored in any image layer.
Runtime secrets: Inject via environment variables at docker run time (values are not in the image), Docker Compose secrets (mounted as files at /run/secrets/), or external secret managers (AWS Secrets Manager, HashiCorp Vault).

Verify: docker history myimage and docker inspect myimage should never reveal secret values.

Q14. What is the difference between docker exec and docker attach?

docker exec starts a new process inside a running container — it is safe for debugging. docker attach connects your terminal to the container's existing PID 1 process. If you press Ctrl+C while attached, you send SIGINT to PID 1 and stop the container.

Always use docker exec -it mycontainer /bin/sh for debugging. Use docker attach only when you specifically need to interact with PID 1, and detach with Ctrl+P, Ctrl+Q.

Q15. How do you optimize Docker builds in CI/CD for speed?

Five key techniques:

Registry-based layer caching — reuse layers from previous builds with --cache-from
Dockerfile layer order — dependency installation before source code copy
BuildKit inline cache — export cache to registry with --cache-to type=registry,mode=max
Multi-stage parallelism — BuildKit builds independent stages in parallel
GitHub Actions/GitLab cache — use type=gha cache for persistent caching between runs

# GitHub Actions example
- uses: docker/build-push-action@v5
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

A well-optimized pipeline can reduce build times from 10 minutes to under 2 minutes.

Prepping for interviews? See Docker Interview Questions.

Complete Docker Cheat Sheet

Container Commands

Command	Description
`docker run -d -p 8080:80 --name app nginx`	Run detached with port mapping and name
`docker run -it --rm ubuntu bash`	Interactive shell, auto-remove on exit
`docker run -e ENV=prod --env-file .env app`	With environment variables
`docker run -v vol:/data --memory=512m app`	With volume and memory limit
`docker ps`	List running containers
`docker ps -a`	List ALL containers
`docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"`	Custom format output
`docker stop myapp`	Graceful stop (SIGTERM then SIGKILL)
`docker kill myapp`	Immediate kill (SIGKILL)
`docker rm myapp`	Remove stopped container
`docker rm -f myapp`	Force remove running container
`docker logs -f --tail 100 myapp`	Follow last 100 log lines
`docker exec -it myapp bash`	Enter container shell
`docker exec myapp cat /app/config.json`	Run command in container
`docker cp myapp:/app/file.txt ./`	Copy file from container
`docker cp ./file.txt myapp:/app/`	Copy file to container
`docker stats --no-stream`	Snapshot resource usage
`docker top myapp`	Show processes in container
`docker inspect myapp`	Full container JSON information
`docker inspect --format '{{.NetworkSettings.IPAddress}}' myapp`	Get container IP
`docker update --restart=always --memory=1g myapp`	Update running container settings
`docker rename myapp newname`	Rename container
`docker pause myapp`	Freeze container
`docker unpause myapp`	Unfreeze container
`docker container prune`	Remove all stopped containers
`docker events --filter type=container`	Live container events

Image Commands

Command	Description
`docker pull nginx:1.25-alpine`	Pull specific tag
`docker pull ubuntu@sha256:abc123`	Pull by digest (immutable)
`docker images`	List local images
`docker build -t myapp:v1.0 .`	Build from Dockerfile in current directory
`docker build -f Dockerfile.prod -t myapp:prod .`	Use specific Dockerfile
`docker build --no-cache -t myapp .`	Force fresh build
`docker build --build-arg NODE_ENV=prod -t myapp .`	With build argument
`docker build --target production -t myapp .`	Build specific stage
`docker tag myapp:latest registry/myapp:v1.0`	Tag for registry
`docker push registry/myapp:v1.0`	Push to registry
`docker rmi myapp:latest`	Remove image
`docker history myapp`	Show image layers
`docker history --no-trunc myapp`	Full layer commands
`docker inspect myapp`	Image JSON details
`docker save -o myapp.tar myapp:latest`	Export to tar archive
`docker load -i myapp.tar`	Import from tar archive
`docker image prune`	Remove dangling images
`docker image prune -a`	Remove all unused images
`docker system df`	Disk usage summary
`docker system prune -af --volumes`	Full cleanup of all unused resources

Network Commands

Command	Description
`docker network ls`	List networks
`docker network create mynet`	Create bridge network
`docker network create --internal secure`	Internal network (no internet access)
`docker network inspect mynet`	Network details
`docker network connect mynet mycontainer`	Connect running container to network
`docker network disconnect mynet mycontainer`	Disconnect container from network
`docker network rm mynet`	Remove network
`docker network prune`	Remove unused networks
`docker run --network mynet myapp`	Run in specific network
`docker run --network host myapp`	Use host network
`docker run --network none myapp`	No networking
`docker run -p 8080:80 myapp`	Publish port
`docker run -p 127.0.0.1:8080:80 myapp`	Bind to localhost only
`docker port myapp`	Show port mappings

Volume Commands

Command	Description
`docker volume create mydata`	Create named volume
`docker volume ls`	List volumes
`docker volume inspect mydata`	Volume details
`docker volume rm mydata`	Remove volume
`docker volume prune`	Remove unused volumes
`docker run -v mydata:/app/data myapp`	Mount named volume
`docker run -v $(pwd):/app myapp`	Bind mount current directory
`docker run -v /app/node_modules myapp`	Anonymous volume
`docker run --mount type=volume,source=mydata,target=/data myapp`	Mount syntax
`docker run --mount type=bind,source=$(pwd),target=/app myapp`	Bind mount syntax
`docker run --mount type=tmpfs,target=/tmp myapp`	tmpfs mount
`docker run -v /host/path:/container/path:ro myapp`	Read-only mount

Docker Compose Commands

Command	Description
`docker compose up -d`	Start all services detached
`docker compose up -d --build`	Rebuild and start
`docker compose up -d --scale api=3`	Start with 3 api replicas
`docker compose down`	Stop and remove containers and networks
`docker compose down -v`	Also remove volumes (destructive — use with care)
`docker compose ps`	List services
`docker compose logs -f api`	Follow api logs
`docker compose exec api bash`	Shell into running service
`docker compose run --rm api npm test`	One-off command in new container
`docker compose build --no-cache`	Force rebuild
`docker compose pull`	Pull latest images
`docker compose config`	Validate and show resolved configuration
`docker compose restart api`	Restart specific service
`docker compose --profile dev up -d`	Start with profile
`docker compose -f base.yml -f prod.yml up -d`	Multiple compose files

Dockerfile Quick Reference

Instruction	Syntax	Purpose
`FROM`	`FROM node:18-alpine AS builder`	Base image
`LABEL`	`LABEL version="1.0" maintainer="x@y.com"`	Metadata
`RUN`	`RUN apt-get update && apt-get install -y curl`	Execute during build
`CMD`	`CMD ["node", "server.js"]`	Default command (overridable)
`ENTRYPOINT`	`ENTRYPOINT ["npm", "start"]`	Fixed executable
`COPY`	`COPY --chown=node:node . /app`	Copy files from build context
`ADD`	`ADD archive.tar.gz /app/`	Copy and auto-extract archives
`WORKDIR`	`WORKDIR /app`	Set working directory
`ENV`	`ENV NODE_ENV=production PORT=3000`	Runtime environment variable
`ARG`	`ARG NODE_VERSION=18`	Build-time variable only
`EXPOSE`	`EXPOSE 3000`	Document port (does not publish)
`VOLUME`	`VOLUME /app/data`	Declare mount point
`USER`	`USER node`	Switch to non-root user
`HEALTHCHECK`	`HEALTHCHECK --interval=30s CMD curl -f http://localhost/health`	Health monitoring
`STOPSIGNAL`	`STOPSIGNAL SIGQUIT`	Override stop signal

Power One-Liners

One-liner	Purpose
`docker system prune -af --volumes`	Complete cleanup
`docker stop $(docker ps -q)`	Stop all running containers
`docker rm $(docker ps -aq)`	Remove all containers
`docker rmi $(docker images -q)`	Remove all images
`docker rm $(docker ps -aq -f status=exited)`	Remove only exited containers
`docker inspect --format '{{.NetworkSettings.IPAddress}}' myapp`	Get container IP
`docker inspect --format '{{.State.ExitCode}}' myapp`	Get exit code
`docker logs myapp 2>&1 \	grep -i error`
`docker logs --since 1h --tail 100 myapp`	Recent logs
`docker events --filter 'event=die'`	Watch for container deaths
`docker images --format "{{.Size}}\t{{.Repository}}:{{.Tag}}" \	sort -h`
`docker run --rm -v $(pwd):/app node:18-alpine npm install`	Run npm install via Docker
`docker exec myapp env`	List container environment variables
`docker diff myapp`	Show filesystem changes in container

Production Workflows

Real-world Docker workflows used daily in production environments.

Daily DevOps Docker Checklist

# ===== Morning Health Check =====
docker ps                                    # Are all containers running?
docker stats --no-stream                     # Is resource usage normal?
docker logs myapp --since 8h | grep -iE 'error|fatal|panic'  # Overnight errors?
df -h /var/lib/docker                        # Is disk space OK?

# ===== Deployment Workflow =====
# 1. Build
VERSION=$(git describe --tags --always)
docker build -t myapp:$VERSION .

# 2. Test
docker run --rm myapp:$VERSION npm test

# 3. Tag and push
docker tag myapp:$VERSION registry/myapp:$VERSION
docker push registry/myapp:$VERSION

# 4. Deploy (rolling update on server)
docker pull registry/myapp:$VERSION
docker stop myapp-old || true
docker rename myapp myapp-old || true
docker run -d \
  --name myapp \
  --restart unless-stopped \
  -p 80:3000 \
  registry/myapp:$VERSION

# 5. Verify
sleep 5 && docker ps | grep myapp
curl -f http://localhost:80/health

# 6. Clean up old container
docker rm myapp-old

Rollback Workflow

PREV_VERSION="v1.2.2"
docker pull registry/myapp:$PREV_VERSION
docker stop myapp && docker rm myapp
docker run -d --name myapp registry/myapp:$PREV_VERSION

Caution: Never run docker rmi on the last known-good image until the new deployment has been stable for at least 30 minutes. Tag the last working image as :stable or :prev to enable instant rollback.

Monthly Maintenance

# Clean up unused resources older than 30 days
docker system prune -a --filter "until=720h"
docker volume prune -f

# Update base images and rebuild with latest security patches
docker pull node:18-alpine
docker build --no-cache .

Zero-Downtime Deployment Pattern

#!/bin/bash
# zero-downtime-deploy.sh
set -e

NEW_IMAGE="registry/myapp:$1"
CONTAINER_NAME="myapp"
BLUE="${CONTAINER_NAME}-blue"
GREEN="${CONTAINER_NAME}-green"

echo "Pulling new image..."
docker pull $NEW_IMAGE

# Determine which container is active
if docker ps | grep -q $BLUE; then
  ACTIVE=$BLUE; INACTIVE=$GREEN
else
  ACTIVE=$GREEN; INACTIVE=$BLUE
fi

echo "Starting $INACTIVE..."
docker run -d --name $INACTIVE --network app $NEW_IMAGE

# Health check loop
for i in {1..30}; do
  if docker exec $INACTIVE curl -sf http://localhost:3000/health; then
    echo "Health check passed!"
    break
  fi
  sleep 2
done

# Switch traffic (update nginx/load balancer upstream here)
echo "Switching traffic to $INACTIVE..."

# Stop old container
docker stop $ACTIVE && docker rm $ACTIVE
echo "Deployment complete! Active: $INACTIVE"

Incident Response Commands

# Container crashed — investigate
docker logs myapp --tail 200
docker inspect myapp | jq '.[0].State'  # Check exit code, OOMKilled flag
docker events --since 1h

# High CPU or Memory
docker stats --no-stream
docker exec myapp top

# Network issues
docker exec myapp curl -v http://db:5432
docker network inspect myapp_default

Docker Compose Patterns

Battle-tested Compose patterns used in real production environments.

Pattern 1: Base + Override (Development vs Production)

The most common pattern — one base file with environment-specific overrides.

# docker-compose.yml (base — committed to git)
services:
  api:
    image: ${REGISTRY:-myapp}/api:${TAG:-latest}
    ports: ["3000:3000"]
    networks: [app]
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: user
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes: [pgdata:/var/lib/postgresql/data]
    networks: [app]
  redis:
    image: redis:7-alpine
    networks: [app]

volumes:
  pgdata:

networks:
  app:

# docker-compose.override.yml (development overrides — add to .gitignore)
# Auto-loaded when you run: docker compose up
services:
  api:
    build: .                          # Build locally instead of pulling
    volumes:
      - .:/app                        # Live code reload
      - /app/node_modules             # Preserve installed node_modules
    environment:
      DEBUG: "true"
      NODE_ENV: development
      DATABASE_URL: postgres://user:pass@db:5432/myapp
    command: npm run dev              # Use dev server with hot reload
  # Development-only extras
  mailhog:
    image: mailhog/mailhog
    ports: ["8025:8025"]             # Fake email server for testing
  redis-ui:
    image: rediscommander/redis-commander
    ports: ["8081:8081"]
    profiles: [debug]                 # Only starts with --profile debug

# docker-compose.prod.yml (production overrides)
services:
  api:
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M
        reservations:
          cpus: "0.25"
          memory: 256M
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
  nginx:
    image: nginx:1.25-alpine
    ports: ["80:80", "443:443"]
    volumes:
      - ./nginx/prod.conf:/etc/nginx/conf.d/default.conf:ro
      - ./ssl:/etc/ssl:ro
    networks: [app]
    restart: unless-stopped

# Development (auto-loads override file)
docker compose up -d

# Production
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Validate the merged configuration
docker compose -f docker-compose.yml -f docker-compose.prod.yml config

Pattern 2: Microservices with Profiles

services:
  # Core services (always start)
  gateway:
    image: myapp/gateway:latest
    ports: ["80:80"]
    networks: [app]

  auth-service:
    image: myapp/auth:latest
    networks: [app]

  # Optional services — start only with a profile
  monitoring:
    image: prom/prometheus
    profiles: [monitoring]
    volumes: [./prometheus.yml:/etc/prometheus/prometheus.yml]
    ports: ["9090:9090"]
    networks: [app]

  grafana:
    image: grafana/grafana
    profiles: [monitoring]
    ports: ["3001:3000"]
    networks: [app]

  # Integration test dependencies
  test-db:
    image: postgres:15-alpine
    profiles: [testing]
    environment:
      POSTGRES_DB: testdb
      POSTGRES_PASSWORD: testpass

networks:
  app:

# Start core services only
docker compose up -d

# Start with monitoring stack
docker compose --profile monitoring up -d

# Start with testing dependencies
docker compose --profile testing up -d

# Start everything
docker compose --profile monitoring --profile testing up -d

Pattern 3: Dependency Management with Health Checks

services:
  db:
    image: postgres:15-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      retries: 3

  api:
    build: .
    depends_on:
      db:
        condition: service_healthy    # Wait for db health check to pass
      redis:
        condition: service_healthy    # Wait for redis health check to pass
    environment:
      DATABASE_URL: postgres://user:pass@db:5432/myapp
      REDIS_URL: redis://redis:6379

Note: Plain depends_on only waits for the container to start — not for the service inside to be ready. Always combine with condition: service_healthy and a proper healthcheck block to avoid startup race conditions.

Docker Tools Ecosystem

Essential tools that every Docker practitioner should know.

Portainer — Web GUI

Web-based interface for managing Docker containers, images, networks, and volumes. Ideal for teams and for environments where a GUI is preferred.

docker volume create portainer_data
docker run -d \
  -p 9000:9000 \
  --name portainer \
  --restart always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:latest
# Visit: http://localhost:9000

Lazydocker — Terminal UI

Terminal UI for Docker. View containers, logs, stats, and images interactively without typing commands.

curl https://raw.githubusercontent.com/jesseduffield/lazydocker/master/scripts/install_via_bash.sh | bash
lazydocker

dive — Image Layer Analyzer

Inspect Docker image layers interactively. Identify which files are added at each layer and find what is bloating image size.

# Ubuntu
sudo snap install dive
# macOS
brew install dive
# Analyze any image
dive myapp:latest

hadolint — Dockerfile Linter

Lint your Dockerfiles for best practice violations, security issues, and layer ordering problems.

# Run via Docker (no installation required)
docker run --rm -i hadolint/hadolint < Dockerfile

# Install locally on macOS
brew install hadolint

hadolint Dockerfile

Trivy — Security Scanner

Comprehensive vulnerability scanner for container images. Detects CVEs in OS packages, language dependencies, and misconfigurations.

curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh

# Scan image for HIGH and CRITICAL CVEs only
trivy image --severity HIGH,CRITICAL myapp:latest

# Fail CI on CRITICAL findings
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan a Dockerfile for misconfigurations
trivy config Dockerfile

Watchtower — Automatic Container Updates

Automatically updates running containers when new images are pushed to the registry. Recommended for development and staging environments only.

docker run -d \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --interval 86400   # Check daily

ctop — Container Resource Monitor

Real-time CPU, memory, network, and I/O metrics in a clean terminal interface — like htop for Docker.

# macOS
brew install ctop

# Any OS via Docker
docker run --rm -ti \
  --name ctop \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  quay.io/vektorlab/ctop

Tools Comparison

Tool	Type	Best For	Free?
Portainer	Web GUI	Team management, beginners	Yes (CE)
Lazydocker	Terminal UI	Developers, quick inspection	Yes
dive	CLI	Image size optimization	Yes
hadolint	Linter	Dockerfile quality in CI	Yes
trivy	Security	CVE scanning, compliance	Yes
watchtower	Daemon	Auto-updates (dev/staging)	Yes
ctop	TUI	Real-time monitoring	Yes
Docker Desktop	GUI	Mac/Windows developers	Freemium
VS Code Extension	IDE	Development workflow	Yes

Tip: Recommended starter stack — install Lazydocker for daily use, dive to check image size before every push, and integrate hadolint and trivy into your CI/CD pipeline. These four tools will catch most common issues automatically.

Further Learning Resources

Top Learning Resources

Resource	Type	URL	Why It Is Valuable
Docker Official Docs	Documentation	`docs.docker.com`	Authoritative source, updated with every release
Play with Docker	Interactive Lab	`labs.play-with-docker.com`	Free browser-based Docker playground — no installation needed
TechWorld with Nana	YouTube	`youtube.com/@TechWorldwithNana`	Best free Docker course on YouTube, extremely clear explanations
Bret Fisher	YouTube / Course	`youtube.com/@BretFisher`	Deep DevOps focus, Docker Swarm, real-world production tips
KodeKloud	Course Platform	`kodekloud.com`	Hands-on browser labs, great for beginners, Kubernetes learning path
Docker Deep Dive (Nigel Poulton)	Book	Amazon / Leanpub	Best Docker book — concise, practical, updated regularly
awesome-docker	GitHub List	`github.com/veggiemonk/awesome-docker`	Curated list of Docker tools, tutorials, and resources
Docker Official Blog	Blog	`docker.com/blog`	Latest Docker updates, best practices, case studies
Ivan Velichko's Blog	Blog	`iximiuz.com`	Deep internals: containers, namespaces, cgroups explained with clarity

4-Week Practice Plan

Week 1: Install Docker, run official images (nginx, postgres, redis, ubuntu)
        → docker run, docker ps, docker logs, docker exec
        → Understand port mapping, volume mounting, environment variables

Week 2: Write Dockerfiles for your existing React/Node projects
        → .dockerignore, layer caching, multi-stage builds
        → Optimize: get below 200 MB for your application image

Week 3: Docker Compose — build a full MERN/PERN stack locally
        → depends_on, healthchecks, named volumes
        → Development vs production compose files, profiles

Week 4: Security, multi-stage builds, push to ECR, deploy to ECS/Fargate
        → Non-root USER, trivy scan, hadolint check
        → GitHub Actions pipeline: build → push → deploy

Certification Paths

Docker Certified Associate (DCA)
Official Docker certification covering installation, configuration, networking, security, and orchestration. A solid credential for validating foundational knowledge.
Format: 55 MCQ, 90 minutes | Cost: approximately $195 USD

CKA — Certified Kubernetes Administrator
The natural next step after mastering Docker. Kubernetes is Docker at scale, and your Docker knowledge maps directly to Kubernetes concepts.
Format: Hands-on practical exam, 2 hours | Cost: approximately $395 USD

AWS Solutions Architect
Includes deep coverage of ECS, Fargate, ECR, and EKS. Highly valued in the job market for senior DevOps and cloud engineering roles.
Format: 65 MCQ, 130 minutes | Cost: approximately $150 USD

What's Next? Your Docker Mastery Roadmap

You now have the knowledge — the next step is execution. Here is a precise 30-day plan to go from Docker learner to production-ready DevOps engineer.

30-Day Docker Mastery Plan

Days	Topic	Task / Goal
1–2	Installation and First Steps	Install Docker on Ubuntu, run hello-world, nginx, postgres — understand what each does
3–5	Container Commands	Practice all docker run flags, docker logs, exec, inspect, stats — build muscle memory
6–8	Images Deep Dive	Pull images, run docker history, inspect layers with dive, understand image size
9–12	Dockerfile Mastery	Write Dockerfiles for React, Node.js, and Python apps from scratch
13–15	Multi-Stage Builds	Optimize your apps: push image size from 1 GB down to 100 MB or less
16–18	Networking	Create custom networks, wire up MERN stack containers, test DNS resolution
19–21	Volumes and Persistence	Database persistence across restarts, dev bind mounts, tmpfs for secrets
22–25	Docker Compose	Full stack with Compose: health checks, depends_on, dev/prod override files, profiles
26–27	Security Hardening	Non-root USER, trivy scan (zero HIGH/CRITICAL), hadolint (zero warnings), secrets management
28–29	CI/CD Pipeline	GitHub Actions: auto-build on push → push to Docker Hub or ECR → deploy to server
30	AWS Deployment	Push to ECR, create ECS task definition, deploy to Fargate — live in production

Docker to Kubernetes Transition Map

Every Docker concept has a direct Kubernetes equivalent. Your Docker knowledge is your Kubernetes head start.

Docker Concept	Kubernetes Equivalent	Notes
`Container`	`Pod` (wraps one or more containers)	Pods are the smallest deployable unit in Kubernetes
`docker run`	`kubectl run` / Deployment	Deployments add replicas, rollouts, rollbacks
`docker-compose.yml`	Deployment + Service YAML	Compose maps to multiple Kubernetes manifest files
`Docker network`	Service + NetworkPolicy	Kubernetes Services provide stable DNS endpoints for pods
`Docker volume`	PersistentVolume (PV/PVC)	PVCs allow dynamic storage provisioning
`docker compose up`	`kubectl apply -f`	Declarative — Kubernetes reconciles desired state
`Docker Swarm service`	Deployment with replicas	Kubernetes has much more sophisticated scheduling
`HEALTHCHECK`	readinessProbe + livenessProbe	Kubernetes has separate readiness, liveness, and startup probes
`docker secret`	Kubernetes Secret	Kubernetes Secrets can be mounted as files or environment variables
`--env-file`	ConfigMap + Secret	ConfigMaps for non-sensitive config, Secrets for sensitive values

Need to plan resource requests for Kubernetes? Use the Kubernetes Resource Calculator.

Portfolio Projects for Senior DevOps Roles

Build these five projects to make your GitHub profile stand out to technical recruiters:

Dockerized MERN Stack — Full MERN (MongoDB, Express, React, Node) stack with Docker Compose. Proper health checks, named volumes, multi-stage build for React. Include a detailed README with an architecture diagram.
CI/CD Pipeline — GitHub Actions workflow that automatically builds an image on push to main, runs tests, pushes to ECR with a version tag, and deploys to ECS/Fargate. Include a rollback workflow.
Microservices Application — Three or more services (auth, products, orders) each in separate containers with an API gateway. Service discovery via Docker DNS, isolated databases, event-driven communication.
Security-Hardened Image — Dockerfile that passes a trivy scan with zero HIGH/CRITICAL CVEs, passes hadolint with zero warnings, runs as a non-root user, and uses multi-stage builds. Include scan reports in the README.
Monitoring Stack — Application with Prometheus, Grafana, and Alertmanager in Docker Compose. Custom dashboards showing container metrics, application metrics, and alerting rules.

Moving Forward

Completing this guide means you can confidently write production-grade Dockerfiles, deploy multi-container applications with Compose, use AWS ECR and ECS with Fargate, follow Docker security best practices, and tackle DevOps interviews with confidence.

Your Docker knowledge is the foundation for everything in modern DevOps. The concepts of containers, immutable images, service isolation, and declarative configuration carry directly into the Kubernetes ecosystem.

Note: Prefer a day-by-day path? This is covered in Mission 90 Days 22–30 — a free 90-day guided DevOps program with browser terminal missions.

Ready to orchestrate at scale? Continue with Kubernetes for DevOps.

AWS for DevOps Engineers: The Core Services You Actually Need

opscanopy — Sat, 15 Aug 2026 13:45:34 +0000

A focused, interview-ready guide built for a developer moving into DevOps. Depth on the services that actually show up in DevOps work and interviews — not all 200+ AWS offerings. This guide covers Regions and AZs, IAM, EC2, Lambda, containers, VPC networking, S3/EBS/EFS storage, RDS and DynamoDB, the AWS CLI, and cost and security guardrails. See the AWS roadmap for a structured learning path through certifications and deeper specialisations.

AWS Fundamentals

Before you touch a single EC2 instance, you need the mental map: where your stuff runs (Regions & AZs), who is responsible for what (Shared Responsibility), and how you pay for it (Billing & Free Tier). Get these three right and you avoid 90% of the surprises beginners hit.

Regions and Availability Zones

AWS physically runs in Regions (e.g. us-east-1 = N. Virginia, eu-west-1 = Ireland, ap-southeast-1 = Singapore). Each Region is a cluster of isolated datacenters grouped into Availability Zones (AZs). An AZ is one or more physical datacenters with independent power, cooling, and networking, but AZs in the same Region are linked by low-latency private fiber.

Region = a city (N. Virginia). A geographic area you pick.
AZ = separate neighborhoods in that city (us-east-1a, us-east-1b, us-east-1c), each with its own power grid, far enough apart that one flood/fire won't take down another.
Edge Location = a corner shop close to your house — used by CloudFront (CDN) and Route 53 to cache content near users for speed.

Note: Think of a Region like a big housing society in one city. Each AZ is a separate building (Tower A, B, C) with its own generator and water tank. If Tower A's power trips, Tower B is fine. So you spread your app across multiple towers — that's Multi-AZ. Edge Locations are the ATMs scattered all over the city so you don't have to drive to the main bank for cash.

Why Multi-AZ Matters for High Availability

If you run everything in one AZ and that AZ goes down, your app goes down. Period. Running across 2+ AZs means a failure in one is automatically absorbed by the others.

Multi-AZ RDS: a standby DB copy sits in another AZ; on failure AWS auto-promotes it (failover in ~60–120s).
EC2 + ELB Auto Scaling: spread instances across AZs; the load balancer routes around the dead AZ.
Multi-Region is the next level up — for disaster recovery (DR) and global latency, not just AZ failure. It costs more and is harder (data replication, DNS failover).

Note: A fintech runs its API on EC2 across us-east-1a and us-east-1b behind an Application Load Balancer, with RDS in Multi-AZ mode. When AWS had an AZ-level network issue, the ALB stopped sending traffic to the unhealthy AZ and RDS failed over to the standby. End users saw a 90-second blip instead of an outage. Single-AZ would have meant hours of downtime.

How to Choose a Region

Factor	What to check
Latency	Pick the Region closest to your users. Users in Southeast Asia → `ap-southeast-1`. Test with cloudping or RTT measurements.
Cost	Prices differ per Region. `us-east-1` is usually cheapest; some Regions (e.g. Sydney, São Paulo) can be pricier. Same instance, different bill.
Compliance / Data residency	Customer/financial data may legally need to stay in-country → pick the matching Region. GDPR data → an EU Region like `eu-west-1`.
Service availability	Newest services launch in `us-east-1` first. Some Regions lag. Check before committing.

Tip: Some services are global, not regional: IAM, Route 53, CloudFront, and WAF (for CloudFront). But the AWS console quietly defaults a lot of regional resources to us-east-1 — always confirm your selected Region (top-right of the console) before creating resources, or you'll "lose" an EC2 instance that's actually running (and billing) in the wrong Region.

Region vs AZ vs Edge Location

	Region	Availability Zone	Edge Location
What it is	Geographic area (a city)	1+ isolated datacenters in a Region	Small CDN/cache PoP near users
Count	30+ globally	Usually 3+ per Region	400+ globally
Used for	Choosing where to deploy	HA & fault tolerance within a Region	Caching content close to users (low latency)
Example service	Picked when launching EC2/RDS	Multi-AZ RDS, Auto Scaling groups	CloudFront, Route 53, Global Accelerator
Isolation	Fully isolated from other Regions	Independent power/cooling/network	Not for compute — caching only

Tip: Classic question: "What's the difference between Multi-AZ and Multi-Region?" Answer crisply: Multi-AZ = High Availability within one Region (handles datacenter/AZ failure, automatic, low cost). Multi-Region = Disaster Recovery + global reach across Regions (handles a whole-Region failure, more complex, costlier, needs data replication and DNS failover). Don't conflate them.

Shared Responsibility Model

AWS does NOT secure everything for you. Security and compliance are shared. The line is simple to remember:

AWS is responsible for the security OF the cloud — the hardware, the physical datacenters, the host hypervisor, networking backbone, and the managed-service infrastructure.
You are responsible for security IN the cloud — your data, your OS patches (on EC2), your IAM users/roles, your security groups, your application code, encryption settings.

Note: AWS is like a rental apartment building. The landlord (AWS) handles the building structure, the main gate security, the foundation, electricity supply. But inside your flat — locking your own door, what you keep inside, who you give a duplicate key to (IAM keys!) — that's on you. If you leave your flat door wide open, you can't blame the landlord.

AWS — "OF the cloud"	You (DevOps) — "IN the cloud"
Physical datacenter & hardware security	IAM users, roles, groups, MFA, least privilege
Hypervisor & host OS patching	Guest OS patching on EC2 (you own the AMI/instance)
Global network infrastructure	Security Groups, NACLs, VPC subnet design
Managed service uptime (S3, RDS engine, Lambda runtime)	Your data, client/server-side encryption, S3 bucket policies (don't make it public by accident!)
Decommissioning/wiping failed disks	Your application code & its vulnerabilities

Caution: The responsibility line shifts with the service type. On EC2 (IaaS) you patch the OS. On RDS (PaaS) AWS patches the DB engine but you still manage users, schemas, and encryption. On Lambda/S3 (serverless/managed) AWS handles even more, but data security and access policies are always yours. The most common breach in the news? A publicly-exposed S3 bucket — 100% the customer's fault, never AWS's.

Billing and Free Tier

AWS pricing is fundamentally pay-as-you-go: no upfront cost (unless you choose to commit), you pay only for what you actually use — per second/hour of compute, per GB of storage, per GB of data transfer out. No usage = (mostly) no bill. But "mostly" is where beginners get bitten.

Note: Pay-as-you-go is like an electricity meter, not a fixed monthly rent. You only pay for the units you burn. But some things keep the meter running even when you're "not using" them — like a reserved parking spot you booked but left empty (an idle Elastic IP), or a security guard you hired and forgot to send home (a running NAT Gateway). The meter doesn't care that you forgot.

The 3 Types of Free Tier

Type	How long	Examples
Always Free	Forever (within limits)	Lambda 1M requests/month, DynamoDB 25 GB, CloudWatch basic metrics
12-Month Free	12 months from signup	EC2 `t2.micro`/`t3.micro` 750 hrs/mo, S3 5 GB, RDS 750 hrs/mo
Trials	Short fixed window (e.g. 30/90 days)	Short-term service trials like certain SageMaker/Inspector features

Caution: What surprises beginners and quietly inflates the bill — even on "free tier": 1) Data transfer OUT: data going INTO AWS is free, but data going OUT to the internet costs money beyond the small free allowance. 2) Idle Elastic IPs (EIPs): an EIP is free only while attached to a running instance. Allocate one and leave it unattached and you pay hourly. 3) NAT Gateway: charges both an hourly rate AND per-GB processed — and it is NOT free tier. Spin one up in a tutorial, forget it, and it bleeds money 24/7. This is the #1 "why is my bill so high?" culprit.

Tip: On day one: set a Billing Alarm in CloudWatch (or AWS Budgets) for a low threshold like $5–$10. Enable Cost Explorer and check it weekly. Also stop/terminate tutorial resources when done — stop halts EC2 compute charges but you still pay for the EBS volume; terminate kills it fully. Tag resources by project so you can see exactly what's costing what.

IAM — Identity and Access Management

IAM is the front gate and key system of your entire AWS account: who can log in, and what they're allowed to do. It's the #1 DevOps interview area because almost every security incident traces back to bad IAM. IAM is global (not regional) and, importantly, free.

An IAM user or an EC2 instance role both attach to a policy, which defines which AWS resources and actions are permitted.

Root User vs IAM User

When you create an AWS account, you get a root user (the email you signed up with). The root user has unlimited, unrestricted power — it can do anything, including close the account and change billing. That's exactly why you should almost never use it.

Root user: full god-mode access. Use it only for a tiny set of tasks that require root (e.g. changing account settings, closing the account).
IAM user: an identity you create for a person or app, with only the permissions they need.

Note: The root user is the master key to the whole building — opens every door, the safe, the main switchboard, everything. You don't carry the master key around daily; you lock it in a vault, put a heavy alarm (MFA) on it, and hand out individual room keys (IAM users) for day-to-day work. If you lose the master key, the whole building is compromised.

Caution: Lock away root immediately: (1) enable MFA on root, (2) delete any root access keys — root should never have programmatic keys, (3) create an admin IAM user (or, better, use IAM Identity Center / SSO) for daily work, (4) never share or hardcode root credentials. A leaked root key = total account takeover, including running up massive bills and deleting all your data.

Users vs Groups

Attaching policies to each user individually doesn't scale. Groups are collections of users — you attach policies to the group, and every member inherits them.

User: a single identity (a person or a service account).
Group: e.g. Developers, Admins, ReadOnly. Add a user to the group and they get its permissions. Remove them and access goes away instantly.
A group is NOT an identity — you can't make a group the "principal" of an action, and groups can't be nested.

Policies — The Rules of Access

A policy is a JSON document defining permissions. Two big ways to categorize them:

	Identity-based policy	Resource-based policy
Attached to	An IAM user, group, or role	A resource (e.g. S3 bucket, SQS queue, KMS key)
Answers	"What can this identity do?"	"Who can access this resource?"
Has a Principal?	No (the identity is the principal)	Yes — specifies who (account/user/service)
Cross-account	Not directly	Yes — great for sharing across accounts
Example	"Devs can read S3 bucket X"	"S3 bucket policy: allow account B to read"

Managed vs inline:

AWS Managed: prebuilt by AWS (e.g. AmazonS3ReadOnlyAccess). Convenient, auto-updated, but often broader than you need.
Customer Managed: you create and reuse them across identities. Best for least privilege — version-controlled, reusable.
Inline: embedded directly into one user/group/role, 1:1, deleted with it. Use sparingly — for strict one-off relationships only.

Policy JSON Structure

Every statement has an Effect (Allow/Deny), Action (the API calls), Resource (the ARN it applies to), and optionally a Condition. This one lets a user list a specific bucket and read/write objects only under the app/ prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListTheBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-app-bucket",
      "Condition": {
        "StringLike": { "s3:prefix": "app/*" }
      }
    },
    {
      "Sid": "ReadWriteAppObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-app-bucket/app/*"
    }
  ]
}

Tip: Remember the evaluation logic: everything is implicitly denied by default. An explicit Allow grants access, but an explicit Deny always wins over any Allow. So you can broadly allow, then surgically deny sensitive actions. Interviewers love asking "Allow vs Deny — which wins?" Answer: explicit Deny always trumps Allow.

IAM Roles — Why Roles Beat Long-Lived Keys

A role is an identity with permissions but no permanent credentials. Instead, a trusted entity assumes the role and gets temporary credentials (via STS) that auto-expire. This is the single most important IAM best practice.

Note: An access key is like a permanent house key you cut and hand out — if it leaks, it works forever until someone notices and changes the lock. A role is like a hotel keycard: the front desk (STS) issues a fresh card that stops working after checkout. Even if someone copies it, it's useless in a few hours. This is why roles always beat long-lived keys.

Where roles shine:

EC2 Instance Profile: attach a role to an EC2 instance so your app calls AWS APIs without ever storing keys on the box. The SDK auto-fetches rotating temp creds from instance metadata.
AssumeRole: a user/service calls sts:AssumeRole to temporarily become the role.
Cross-account access: account A's role trusts account B; B's users assume it — no shared keys between accounts.
Service roles: e.g. Lambda execution role, ECS task role — the AWS service assumes the role to act on your behalf.

Caution: Never hardcode access keys in code, Dockerfiles, environment files committed to Git, or AMIs. Leaked keys on public GitHub are scraped by bots within minutes and used to spin up crypto-mining fleets — landing you a five-figure bill. On EC2/ECS/Lambda, always use a role instead.

IAM User vs IAM Role

	IAM User	IAM Role
Credentials	Long-lived (password / access keys)	Temporary, auto-expiring (via STS)
Who uses it	A specific person/app, persistently	Anyone/anything trusted who assumes it
Best for	Human console login	EC2/Lambda/ECS, cross-account, federation
Rotation	Manual, easy to forget	Automatic — handled by AWS
Leak risk	High (keys live forever)	Low (creds expire in minutes/hours)

IAM CLI Examples

Create a user, put them in a group, attach a managed policy:

# Create an IAM user
aws iam create-user --user-name riya-dev

# Create a group and attach an AWS managed policy to it
aws iam create-group --group-name Developers
aws iam attach-group-policy \
  --group-name Developers \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# Add the user to the group (inherits the group's permissions)
aws iam add-user-to-group --user-name riya-dev --group-name Developers

# Enforce MFA-friendly console access (set a login profile / password)
aws iam create-login-profile \
  --user-name riya-dev --password 'Temp#Pass123' --password-reset-required

Create a role with a trust policy so EC2 can assume it, then give the role permissions:

# 1) Trust policy: WHO is allowed to assume this role (here: the EC2 service)
cat > trust-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# 2) Create the role with that trust policy
aws iam create-role \
  --role-name app-s3-role \
  --assume-role-policy-document file://trust-policy.json

# 3) Attach a permissions policy (WHAT the role can do)
aws iam attach-role-policy \
  --role-name app-s3-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# 4) Wrap it in an instance profile and attach to EC2 (no keys needed!)
aws iam create-instance-profile --instance-profile-name app-s3-profile
aws iam add-role-to-instance-profile \
  --instance-profile-name app-s3-profile --role-name app-s3-role

Tip: Two policy types per role: the trust policy (the assume-role-policy-document) controls WHO can assume the role, and the permissions policy controls WHAT the role can do once assumed. Interviewers often probe this distinction — keep them separate in your head.

Least Privilege

Grant only the minimum permissions needed to do the job — nothing more. Start with zero, add permissions as required, and scope tightly (specific actions, specific resource ARNs, conditions).

Prefer specific actions (s3:GetObject) over wildcards (s3:* or *).
Scope Resource to exact ARNs, not "*".
Use IAM Access Analyzer to generate least-privilege policies from real CloudTrail usage.
Review with the IAM Policy Simulator before rollout. Audit with Access Analyzer / credential reports regularly.

MFA and Access Key Best Practices

MFA everywhere: mandatory on root and all human users. Add a virtual/hardware MFA device. You can even require MFA in policies via a Condition (aws:MultiFactorAuthPresent).
Access keys: only for programmatic CLI/SDK use where roles aren't possible. Rotate them regularly, never commit to Git, and delete unused keys.
Prefer roles over access keys whenever the workload runs inside AWS (EC2/ECS/Lambda) — zero keys to manage.
Audit: use IAM credential reports and CloudTrail to find stale keys and unused permissions.

Compute — EC2, Lambda, Containers

Compute is "where your code actually runs." On AWS you pick how much control you want versus how much AWS manages for you. More control = more responsibility (EC2). Less control = AWS handles the boring stuff (Lambda, Fargate). As a DevOps engineer your job is choosing the right one for the workload.

EC2 gives full OS control for long-running workloads; Lambda is event-driven and fully managed, scaling to zero between invocations.

Note: Think of renting a place. EC2 is renting an empty flat — you bring furniture, fix the plumbing, control everything. Lambda is a hotel room you only pay for the minutes you're inside. Fargate is a serviced apartment — your stuff, but housekeeping (servers) is handled. Same "shelter," very different effort.

EC2 — Elastic Compute Cloud

EC2 is a virtual server (a VM) in AWS's data center. You choose the OS, CPU, RAM, disk, and you SSH in like any Linux box. It's the foundational "rent a server" service.

Instance Families and Types

Instances are grouped into families optimized for different workloads. The naming follows a pattern: m5.large = family m, generation 5, size large. Bigger size = more vCPU + RAM (and roughly double the price each step up).

General purpose (t, m) — balanced CPU/RAM. t3.micro (burstable, cheap, great for dev/small web), m5.large (steady workloads, app servers).
Compute optimized (c) — high CPU per RAM. c5.xlarge for batch processing, gaming servers, CI build runners.
Memory optimized (r, x) — lots of RAM. r5.large for databases, in-memory caches (Redis), big data.
Storage optimized (i, d) — high disk throughput/IOPS. i3.large for NoSQL, data warehousing, large local datasets.

Tip: t-family instances are burstable — they earn CPU credits when idle and spend them during spikes. Perfect for spiky low-traffic apps, but a constantly-busy server will run out of credits and throttle. For steady load, use m/c.

AMIs (Amazon Machine Images)

An AMI is the template an instance boots from — OS + pre-installed software + config. Think of it as a snapshot/golden image. You launch from an AWS-provided AMI (Amazon Linux, Ubuntu) or bake your own with your app baked in (faster, repeatable deploys — a core DevOps practice with tools like Packer).

Key Pairs (SSH)

To log into a Linux EC2 instance you use SSH key-based auth — no passwords. AWS gives you the public key (placed on the instance), you keep the private .pem file. Lose the private key and you're locked out.

# Create a key pair and save the private key locally
aws ec2 create-key-pair \
  --key-name my-devops-key \
  --query 'KeyMaterial' \
  --output text > my-devops-key.pem

# Lock down permissions (SSH refuses world-readable keys)
chmod 400 my-devops-key.pem

# Later, connect to the instance
ssh -i my-devops-key.pem ec2-user@<PUBLIC_IP>

User Data (Bootstrap Script)

User data is a script that runs once, on first boot, as root. This is how you bootstrap a fresh instance — install packages, pull config, start your app — without manually SSHing in. Essential for automation and Auto Scaling.

#!/bin/bash
# user-data.sh — bootstrap a simple web server on first boot
yum update -y
yum install -y nginx
echo "<h1>Deployed via user data</h1>" > /usr/share/nginx/html/index.html
systemctl enable nginx
systemctl start nginx

Launching an Instance via CLI

# Launch a t3.micro with the bootstrap script and our key
aws ec2 run-instances \
  --image-id ami-0abcdef1234567890 \
  --instance-type t3.micro \
  --key-name my-devops-key \
  --security-group-ids sg-0123456789abcdef0 \
  --subnet-id subnet-0123456789abcdef0 \
  --user-data file://user-data.sh \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=web-01}]'

# See what's running (handy filtered view)
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,IP:PublicIpAddress}' \
  --output table

Pricing Models

Same instance, very different bill depending on how you buy it. Interviewers love this.

On-Demand — pay per second, no commitment. Most expensive per hour, max flexibility. Use for unpredictable/short workloads and testing.
Reserved Instances (RI) — commit to a specific instance type for 1 or 3 years. Up to ~72% off. Use for steady, predictable baseline load.
Savings Plans — commit to a $/hour spend for 1–3 years; more flexible than RIs (applies across families/regions). The modern preferred way to save on steady load.
Spot Instances — bid on spare AWS capacity, up to ~90% off, but AWS can reclaim them with a 2-minute warning. Use for fault-tolerant, stateless, interruptible work (CI builds, batch jobs, big data).

Note: A typical production setup mixes them: Reserved/Savings Plans for the always-on baseline (web servers running 24/7), Spot for the Auto Scaling group that handles traffic spikes and nightly batch jobs, and On-Demand as fallback when Spot capacity isn't available. This can cut a compute bill by half or more.

Lambda vs EC2 — Serverless or Server?

Lambda runs your code in response to events (an HTTP request via API Gateway, a file landing in S3, a message in a queue) with zero servers to manage. AWS provisions, scales, and bills only for execution time (per millisecond). You give it a function; it runs it. But it's short-lived (15-min max), stateless, and you don't control the OS.

Aspect	EC2	Lambda
Model	Long-running virtual server	Event-driven function (serverless)
You manage	OS, patching, scaling, capacity	Just your code
Scaling	Manual / Auto Scaling groups	Automatic, instant, per-request
Billing	Per second while running (even idle)	Per ms of execution; $0 when idle
Max runtime	Runs forever	15 minutes per invocation
State	Stateful (local disk, memory persist)	Stateless (cold starts possible)
Best for	Steady traffic, full control, legacy apps, databases	Spiky/event-driven work, APIs, glue code, cron jobs

Tip: "When would you NOT use Lambda?" Strong answer: long-running jobs (>15 min), workloads needing predictable low latency (cold starts hurt), apps requiring specific OS/runtime control, or steady high-volume traffic where always-on EC2 is actually cheaper. Lambda shines for bursty, event-driven, "scale-to-zero" workloads.

Containers: ECS vs EKS vs Fargate

First, separate two things: the orchestrator (decides where containers run, restarts them, scales them) from the compute (the actual machines they run on).

ECS (Elastic Container Service) — AWS's own orchestrator. Simpler, deeply AWS-integrated, no K8s knowledge needed.
EKS (Elastic Kubernetes Service) — managed Kubernetes. Industry-standard, portable across clouds, bigger learning curve.
Fargate — NOT an orchestrator; it's a serverless compute engine. It runs containers for ECS or EKS without you managing any EC2 nodes. "Containers without servers."

	ECS	EKS	Fargate
What it is	AWS-native orchestrator	Managed Kubernetes orchestrator	Serverless compute for containers
Runs on	EC2 or Fargate	EC2 or Fargate	N/A (it IS the compute)
Learning curve	Low (AWS-specific)	High (full K8s)	Low — no servers to manage
Portability	AWS-locked	Portable (standard K8s)	AWS-only billing model
You manage nodes?	Yes (if EC2 mode)	Yes (if EC2 mode)	No — AWS handles it
Best for	Simple AWS-only container apps	K8s-standard, multi-cloud, complex orchestration	Anyone who wants containers without managing servers

Tip: "ECS vs EKS" and "Fargate" are not three competitors — Fargate is a launch type for ECS or EKS. A common combo is "ECS on Fargate": AWS-native simplicity + zero server management. Pick EKS only when you genuinely need Kubernetes (portability, team already knows K8s, complex workloads).

Networking — VPC and Friends

Networking is where a lot of DevOps engineers get stuck — and where interviewers love to probe. The good news: you already know networking basics (IPs, CIDR, routing). AWS just gives you software-defined versions of the same physical concepts. For a deeper treatment of subnets, CIDR design, and routing, see Networking for DevOps.

A VPC with a public subnet (IGW + NAT Gateway) and a private subnet (EC2); public traffic enters via the IGW while EC2 reaches the internet outbound through the NAT Gateway.

Note: A VPC is your own private gated society (housing colony) inside AWS's huge city. Subnets are blocks within the society. The route table is the society's road map (which road leads where). The Internet Gateway is the main gate to the outside world. Security Groups are guards at each house's door; NACLs are guards at the block's entrance.

VPC (Virtual Private Cloud)

A VPC is your own isolated virtual network inside AWS. Nothing outside it can reach in unless you explicitly allow it. You define its IP range using a CIDR block, e.g. 10.0.0.0/16 — that gives you ~65,536 private IP addresses to carve up. Size your VPC subnets with the Subnet Calculator and validate CIDR ranges with the CIDR Checker.

CIDR Blocks — Quick Refresher

CIDR notation = IP + "how many bits are fixed." /16 fixes the first 16 bits → 10.0.x.x is yours (65,536 IPs). /24 fixes 24 bits → 10.0.1.x (256 IPs). Smaller number after the slash = bigger network. You give the VPC a large block (/16) and slice subnets out of it (/24).

Tip: AWS reserves 5 IPs in every subnet (network address, VPC router, DNS, future use, broadcast). So a /24 subnet gives you 256 − 5 = 251 usable IPs, not 256. This trips people up when sizing subnets.

Subnets: Public vs Private

A subnet is a slice of the VPC, and it lives in one Availability Zone. The big question: what actually makes a subnet "public"?

It's NOT a checkbox called "public." A subnet is public if and only if its route table has a route to an Internet Gateway (0.0.0.0/0 → igw-xxxx). A private subnet has no such route — its traffic can't reach the internet directly.

Tip: Classic question: "What makes a subnet public?" Wrong answer: "it has public IPs." Correct answer: its route table routes 0.0.0.0/0 to an Internet Gateway. An instance also needs a public IP to be reachable, but the defining factor is the route to the IGW.

Route Tables

A route table is a set of rules deciding where network traffic goes. Each subnet is associated with exactly one route table. Every table has a local route (so subnets within the VPC can talk to each other automatically). You add routes for outside traffic — to an IGW (public) or a NAT Gateway (private).

Internet Gateway (IGW)

The IGW is a horizontally-scaled, highly-available component attached to your VPC that allows two-way communication between the VPC and the internet. One IGW per VPC. Without it, nothing in your VPC can reach the public internet, period.

NAT Gateway — Why Private Subnets Need It

Your app servers sit in a private subnet (no IGW route — good, attackers can't reach them). But they still need to download OS updates, pull packages, call external APIs. How do they reach out without being reachable in?

Answer: a NAT Gateway. It lives in a public subnet and does Network Address Translation. Private instances route 0.0.0.0/0 → NAT Gateway; the NAT forwards their outbound requests to the internet using its own public IP, and returns the responses. Crucially, NAT only allows outbound-initiated traffic — the outside world cannot start a connection back in.

Note: NAT Gateway is like a hotel front desk. Guests (private servers) can call out for room service or order food from outside, but no outsider can directly walk to your room — they only reach the front desk. Calls go out, randos can't come in.

Caution: NAT Gateways are NOT free — you pay per hour and per GB processed. They're a common surprise on AWS bills. Also, a NAT Gateway lives in one AZ; for high availability you deploy one per AZ. Don't put it in a private subnet by mistake (it needs the IGW route to work).

Request Flow: User to App

Walking through a request to a web app makes it click:

User's browser sends a request to your app's public IP / load balancer.
Traffic hits the Internet Gateway (the VPC's front gate).
The public subnet's route table directs it to the resource (e.g. a load balancer or web server with a public IP).
The NACL (block-level guard) checks subnet rules, then the Security Group (door-level guard) checks if the port is allowed.
The web tier forwards to app servers in the private subnet over the VPC's internal local route.
Response travels back out the same path. Because SGs are stateful, the return traffic is automatically allowed.

Security Groups vs NACLs

This is one of the most asked AWS interview questions. Both are virtual firewalls, but they operate at different levels and behave differently.

Security Group (SG) — attached to the instance/ENI (the door). Stateful: if you allow inbound traffic, the response is automatically allowed out (and vice versa). Allow rules only — you can't write a "deny" rule; anything not allowed is implicitly denied.
NACL (Network ACL) — attached to the subnet (the block gate). Stateless: you must explicitly allow both inbound AND the return outbound traffic. Supports allow AND deny rules, evaluated in numbered order.

Feature	Security Group	NACL
Level	Instance / ENI	Subnet
State	Stateful (return traffic auto-allowed)	Stateless (must allow both directions)
Rules	Allow only	Allow AND Deny
Rule evaluation	All rules evaluated together	In numbered order, first match wins
Default behavior	Deny all inbound, allow all outbound	Default NACL allows all; custom denies all
Use case	Primary, everyday instance firewall	Coarse subnet-wide rules, blocking specific IPs

Tip: Memorize the one-liner: "SG = stateful, instance-level, allow-only. NACL = stateless, subnet-level, allow + deny." Follow-up: "Can you block a single malicious IP?" → Yes, with a NACL deny rule (SGs can't deny). In practice, 95% of your work is in Security Groups; NACLs are a secondary, coarse layer.

Building VPC with the CLI

# 1. Create the VPC (grab the returned VpcId)
aws ec2 create-vpc \
  --cidr-block 10.0.0.0/16 \
  --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=my-vpc}]'

# 2. Create a public subnet inside it
aws ec2 create-subnet \
  --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 10.0.1.0/24 \
  --availability-zone ap-south-1a \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=public-1a}]'

# 3. Create a security group in the VPC
aws ec2 create-security-group \
  --group-name web-sg \
  --description "Allow HTTP and SSH" \
  --vpc-id vpc-0123456789abcdef0

# 4. Allow inbound HTTP (80) from anywhere and SSH (22) from your IP
aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp --port 80 --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp --port 22 --cidr 203.0.113.25/32

Caution: Never open SSH (port 22) to 0.0.0.0/0 — that exposes your server to the entire internet's brute-force bots. Always restrict to your own IP (/32) or use a bastion host / AWS Systems Manager Session Manager (no open SSH port at all). This is a top finding in every security audit.

Storage — S3, EBS, EFS

AWS storage comes in three fundamental flavors, and knowing which to reach for is core DevOps knowledge: object storage (S3), block storage (EBS), and file storage (EFS).

S3 is object storage accessed over HTTPS; EBS is a block volume attached to a single EC2 instance; EFS is a shared NFS file system mountable by many instances simultaneously.

Note: EBS (block) = a hard drive plugged into one computer — fast, personal, one machine at a time. EFS (file) = a shared network drive in the office that everyone mounts at once. S3 (object) = a giant warehouse where you drop labelled boxes and fetch them by name via a web address — infinite shelves, but you can't "edit" a box, you replace it.

S3 — Simple Storage Service

S3 is object storage: you store objects (files + metadata) inside buckets (globally-unique named containers). It's accessed over HTTPS via an API/URL, scales infinitely, and is 99.999999999% (11 nines) durable. You don't mount it like a disk — you GET/PUT objects by key.

Caution: Bucket names are globally unique across all AWS accounts on Earth — like domain names. If someone took my-bucket, you can't. Use a prefix like mycompany-prod-logs-2026.

Versioning

Enable versioning and S3 keeps every version of an object. Overwrite or delete? The old version is retained (a "delete marker" is added instead of true deletion). This protects against accidental overwrites and ransomware. Once enabled, it can only be suspended, not fully disabled.

Lifecycle Policies

Rules that automatically move or expire objects over time to save money. Example: keep logs in Standard for 30 days, move to Infrequent Access, then to Glacier after 90 days, and delete after a year — all automatic. This is how you control storage costs at scale.

Storage Classes

S3 Standard — frequent access, lowest latency. Default for active data, website assets.
S3 Standard-IA (Infrequent Access) — cheaper storage, but you pay a retrieval fee. For data accessed rarely but needed instantly (backups, older logs).
S3 Glacier (Flexible / Deep Archive) — dirt cheap archival; retrieval takes minutes to hours. For compliance archives, long-term backups you rarely touch.
S3 Intelligent-Tiering — AWS auto-moves objects between tiers based on access patterns. Set-and-forget cost optimization.

Tip: Don't manually guess tiers for unpredictable access — use Intelligent-Tiering and let AWS optimize. Use explicit lifecycle policies when you KNOW the pattern (e.g., "logs are useless after 90 days").

Static Website Hosting

S3 can serve a static website (HTML/CSS/JS) directly — perfect for your React build output. Upload the build, enable website hosting, set the index document. Pair it with CloudFront (CDN) + an ACM certificate for HTTPS and global caching.

# Create a bucket
aws s3 mb s3://mycompany-react-app-2026 --region ap-south-1

# Upload your React production build
aws s3 sync ./build s3://mycompany-react-app-2026 --delete

# Enable static website hosting
aws s3 website s3://mycompany-react-app-2026 \
  --index-document index.html \
  --error-document index.html

Security: Block Public Access and Bucket Policies

Leaky S3 buckets are the most famous cause of data breaches. AWS now enables Block Public Access by default — a master switch that overrides any policy trying to make a bucket public. Access is otherwise controlled by bucket policies (JSON rules on the bucket) and IAM policies (rules on users/roles).

# Lock down public access (the safe default)
aws s3api put-public-access-block \
  --bucket mycompany-react-app-2026 \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# Enable versioning for safety
aws s3api put-bucket-versioning \
  --bucket mycompany-react-app-2026 \
  --versioning-configuration Status=Enabled

Caution: For a public website, the modern best practice is NOT to make the S3 bucket public. Instead keep Block Public Access ON and serve through CloudFront with an Origin Access Control (OAC), so only CloudFront can read the bucket. Public buckets should be a deliberate, rare exception.

EBS — Elastic Block Store

EBS is a block storage volume — a virtual hard disk you attach to a single EC2 instance. You format it, mount it, and it behaves like a local SSD. It's the root disk of most instances and where databases store data. It lives in one AZ and (traditionally) attaches to one instance at a time.

# Create a 50 GB gp3 SSD volume in a specific AZ
aws ec2 create-volume \
  --volume-type gp3 \
  --size 50 \
  --availability-zone ap-south-1a \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=db-data}]'

# Attach it to an instance (volume must be in the same AZ as the instance)
aws ec2 attach-volume \
  --volume-id vol-0123456789abcdef0 \
  --instance-id i-0123456789abcdef0 \
  --device /dev/sdf

Tip: EBS and EC2 must be in the same Availability Zone to attach — block storage is AZ-bound. To move data across AZs, take a snapshot (stored in S3 behind the scenes) and create a new volume from it in the target AZ.

EFS — Elastic File System

EFS is file storage — a managed NFS share that many EC2 instances mount simultaneously, across multiple AZs. It grows and shrinks automatically. Use it when a fleet of servers needs to read/write the same files (shared uploads, CMS content, web cluster assets).

Block vs File vs Object — When to Use Which

	S3 (Object)	EBS (Block)	EFS (File)
Type	Object store	Block volume (virtual disk)	Shared file system (NFS)
Access	HTTPS API / URL	Mounted to one EC2	Mounted to many EC2s
Attaches to	Anything over the internet	1 instance (1 AZ)	Many instances, multi-AZ
Scaling	Infinite, automatic	Fixed size (resize manually)	Automatic, elastic
Can edit in place?	No — replace whole object	Yes — random read/write	Yes — random read/write
Best for	Assets, backups, logs, static sites, big data	OS root disk, databases, single-server data	Shared storage across a server fleet

Tip: The killer interview line: "EBS = one disk for one server; EFS = one shared drive for many servers; S3 = object storage accessed over HTTP, not mounted." If asked "where do you store database files?" → EBS (low latency block). "Shared files across an Auto Scaling group?" → EFS. "Static assets / backups?" → S3.

Databases — RDS and DynamoDB

As a DevOps engineer, the database is the part nobody wants to babysit at 3am — so AWS gives you managed database services. You stop being a DBA and start being an operator who scripts the boring stuff away.

Note: Running a DB on a raw EC2 instance is like buying a car and also building your own service garage — you patch the engine, change the oil, handle backups. A managed database (RDS/DynamoDB) is like a chauffeur-driven car with a maintenance contract: you write business logic, AWS handles patching, backups, and failover.

Amazon RDS — Managed Relational

RDS = your familiar SQL world (tables, joins, foreign keys, transactions) but AWS handles provisioning, patching, backups, and failover. You pick an engine and AWS runs it for you.

Engines: PostgreSQL, MySQL, MariaDB, Oracle, SQL Server, and Amazon Aurora (AWS's high-performance MySQL/Postgres-compatible engine).
Multi-AZ (High Availability): AWS keeps a synchronous standby replica in a second Availability Zone. If the primary dies, RDS automatically fails over to the standby (DNS endpoint stays the same). This is about survival, not speed — the standby does not serve reads.
Read Replicas (Scale): Asynchronous copies you can read from to offload heavy SELECT traffic (reporting, dashboards). This is about scaling reads, not failover. You can promote a replica to standalone, but it isn't automatic HA.
Automated backups: Daily snapshot + transaction logs enable point-in-time recovery (PITR) within your retention window (up to 35 days). Manual snapshots live until you delete them — good for "before the migration" safety copies.
Parameter groups: tune engine settings (e.g. max_connections) without touching a config file. Subnet groups: tell RDS which subnets (across AZs) it may launch into — almost always your private subnets.

Caution: Multi-AZ and Read Replica are constantly confused in interviews. Multi-AZ = HA/failover (sync, same endpoint, no read offload). Read Replica = scaling reads (async, separate endpoint). You can have both at once.

Note: An e-commerce app runs RDS Postgres Multi-AZ for the orders database. The analytics team was hammering it with giant reports, slowing checkout. The fix: add two Read Replicas and point the reporting tool at the replica endpoint. Checkout latency dropped instantly — writes stayed on the primary, reads spread out.

Create a Multi-AZ Postgres instance in private subnets:

aws rds create-db-instance \
  --db-instance-identifier prod-orders-db \
  --db-instance-class db.t3.medium \
  --engine postgres \
  --engine-version 16.3 \
  --master-username appadmin \
  --master-user-password 'ChangeMe-UseSecretsManager!' \
  --allocated-storage 50 \
  --storage-type gp3 \
  --multi-az \
  --db-subnet-group-name prod-private-subnets \
  --vpc-security-group-ids sg-0a1b2c3d4e5f67890 \
  --backup-retention-period 7 \
  --storage-encrypted \
  --no-publicly-accessible

Tip: Never bake the DB password into CLI/Terraform. Use AWS Secrets Manager (RDS can even auto-rotate it) or --manage-master-user-password so AWS stores and rotates it for you.

Why Managed Beats Self-Hosted on EC2

Automated patching, backups, and PITR — no cron jobs you forget about.
One-flag HA (--multi-az) instead of hand-building replication and failover scripts.
Push-button read scaling and storage autoscaling.
You still own schema, queries, and indexes — i.e. the stuff that's actually your job.

Amazon DynamoDB — Managed NoSQL

DynamoDB is a fully managed key-value / document store. No servers, no engine versions, no connection pools — just an API. It delivers single-digit millisecond latency at any scale, which is why it powers shopping carts, session stores, and leaderboards.

Partition key (PK): determines which physical partition stores the item — choose a high-cardinality key so traffic spreads evenly (avoid "hot partitions").
Sort key (SK): optional; lets you store many related items under one PK and query ranges (e.g. PK=USER#42, SK=ORDER#2026-06-01). PK + SK together form the primary key.
On-demand capacity: pay per request, scales instantly — perfect for spiky or unknown traffic.
Provisioned capacity: you set read/write capacity units (with optional auto-scaling) — cheaper for steady, predictable load.

aws dynamodb create-table \
  --table-name UserOrders \
  --attribute-definitions \
    AttributeName=UserId,AttributeType=S \
    AttributeName=OrderId,AttributeType=S \
  --key-schema \
    AttributeName=UserId,KeyType=HASH \
    AttributeName=OrderId,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST \
  --tags Key=env,Value=prod

Tip: HASH = partition key, RANGE = sort key in DynamoDB CLI terminology. PAY_PER_REQUEST = on-demand; switch to PROVISIONED with --provisioned-throughput once your traffic is predictable.

RDS vs DynamoDB — When to Pick What

Pick RDS when you have relational data, complex joins, transactions, and ad-hoc queries (finance, orders, anything with strong consistency and relationships). Pick DynamoDB when you need massive scale, predictable access patterns by key, ultra-low latency, and zero ops (carts, sessions, IoT, gaming, event logs).

Aspect	Amazon RDS	DynamoDB
Data model	Relational (tables, joins)	NoSQL key-value / document
Query flexibility	Full SQL, ad-hoc queries	Key-based access; design schema around access patterns
Scaling	Vertical + read replicas	Horizontal, virtually unlimited, automatic
Latency	Low (ms), depends on instance	Single-digit ms at any scale
HA	Multi-AZ failover	Built-in, multi-AZ by default
Ops overhead	Low (you pick instance size, params)	Near zero (serverless)
Best for	Orders, finance, complex relationships	Carts, sessions, IoT, leaderboards

Tip: "How do you scale reads on RDS without hurting writes?" → Read Replicas, point read-only traffic at the replica endpoint. "How do you make RDS highly available?" → Multi-AZ. Saying the right one for the right reason instantly signals you understand the difference.

How It All Fits Together

You've met the building blocks one by one — IAM, VPC, subnets, EC2, S3, RDS, Security Groups, gateways, ALB. Now let's snap them together like Lego and deploy a typical web app. This is the mental model interviewers want to hear, and it's the diagram you'll draw on a whiteboard for the rest of your career.

Note: Think of your AWS account as a gated township. IAM = the security desk deciding who enters and what they can touch. VPC = the township's boundary wall. Subnets = individual lanes (some facing the main road = public, some tucked inside = private). IGW = the main gate to the public road. NAT = a one-way courier so inside residents can order online without strangers walking in. Security Groups = the lock on each house's door.

The 3-Tier Architecture (Web / App / Data)

Almost every classic web app splits into three tiers, and AWS maps cleanly onto them:

Web / presentation tier: the public entry point. An Application Load Balancer (ALB) sits in public subnets and accepts user traffic on 80/443. Static assets (images, JS bundles, your React build) often live in S3 and/or behind a CDN.
App / logic tier: your EC2 instances (or containers) running the backend, in private subnets. They never get a public IP — the ALB forwards traffic to them.
Data tier: RDS (and/or DynamoDB) in private subnets, reachable only from the app tier. Nothing on the internet can touch the database directly.

Each tier lives in its own subnet layer, spread across at least two Availability Zones for resilience, all inside one VPC.

The Deployment, Step by Step

IAM first. Create roles, not long-lived keys. An EC2 instance role lets the app read from S3 and pull DB credentials from Secrets Manager — no keys on disk.
Build the VPC. Carve a CIDR (e.g. 10.0.0.0/16) and create public + private subnets across two AZs.
Attach an Internet Gateway (IGW) to the VPC and route public subnets' 0.0.0.0/0 to it. Add a NAT Gateway in a public subnet so private instances can reach the internet outbound (for updates/patches) without being reachable inbound.
Launch EC2 app servers in the private subnets, behind an Auto Scaling Group ideally.
Create the ALB in the public subnets, with a target group pointing at the EC2 instances. Health checks decide which instances get traffic.
Provision RDS (Multi-AZ) in the private subnets via a DB subnet group.
Wire Security Groups (the real glue): ALB SG allows 443 from 0.0.0.0/0; EC2 SG allows the app port only from the ALB's SG; RDS SG allows 5432/3306 only from the EC2 SG. Referencing SGs by ID (not IP ranges) is the clean, least-privilege way.
Point your domain (Route 53) at the ALB and attach a TLS cert (ACM) for HTTPS.

Request Flow, End to End

User's browser resolves your domain via Route 53 → gets the ALB's address.
Request enters the VPC through the IGW and hits the ALB in a public subnet (TLS terminated by ACM cert).
ALB picks a healthy EC2 instance in a private subnet and forwards the request — allowed because the EC2 Security Group trusts the ALB's SG.
The app reads/writes RDS (private), pulls static files or uploads to S3, and uses its IAM instance role for those calls.
If the app needs to call an external API, traffic exits via the NAT Gateway — outbound only.
Response flows back through the ALB to the user. The database and app servers were never exposed to the internet.

Note: A SaaS dashboard runs exactly this shape: ALB in two public subnets, an Auto Scaling Group of EC2 in two private subnets, RDS Postgres Multi-AZ in the data tier, and the React build served from S3 + CloudFront. When a marketing email caused a traffic spike, the ASG added instances and the ALB spread load — zero manual intervention, no downtime.

Where CI/CD and IaC Fit

You'd never click all of the above by hand in production. Infrastructure as Code — Terraform or CloudFormation — defines the VPC, subnets, SGs, ALB, RDS, and IAM roles as version-controlled files. You plan and apply to create identical environments (dev/staging/prod) repeatably.

CI/CD (GitHub Actions, GitLab CI, CodePipeline) handles the application: on every git push it builds, tests, and deploys your code to the EC2/containers and pushes the React bundle to S3. IaC builds the house; CI/CD keeps moving the furniture in.

Tip: When asked "walk me through deploying a web app on AWS," structure your answer as tiers (web/app/data) + the security boundaries between them. Emphasize that databases and app servers live in private subnets, the ALB is the only public entry, and Security Groups reference each other by ID. That one paragraph signals senior-level thinking.

Hands-on AWS CLI

The console is great for learning, but DevOps lives in the terminal — scriptable, repeatable, automatable. The AWS CLI is your daily driver. Let's install it, configure credentials, and build a cheat sheet you'll actually reuse.

Install on Ubuntu 24.04

Use the official v2 bundle (the apt package is often outdated):

sudo apt update && sudo apt install -y unzip curl
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version   # confirm: aws-cli/2.x.x ...

Configure Credentials

aws configure walks you through four prompts: Access Key ID, Secret Access Key, default region, and output format (json, table, or text).

aws configure
# AWS Access Key ID [None]: AKIAEXAMPLE...
# AWS Secret Access Key [None]: wJalr...EXAMPLEKEY
# Default region name [None]: ap-south-1
# Default output format [None]: json

Named Profiles and Where Credentials Live

One laptop, many accounts (dev, prod, client). Named profiles keep them separate. Credentials and settings are stored in two plaintext files in your home directory:

~/.aws/credentials — the secret keys (per profile).
~/.aws/config — region, output format, and SSO/role settings (per profile).

# Create a named profile
aws configure --profile prod

# Use it per-command
aws s3 ls --profile prod

# Or set it for the whole shell session
export AWS_PROFILE=prod
aws sts get-caller-identity

# Inspect the files
cat ~/.aws/config
cat ~/.aws/credentials

Caution: ~/.aws/credentials stores your secret key in plaintext. Never commit this file, never paste keys into Slack, and on shared/CI machines prefer IAM roles or AWS SSO over long-lived keys.

Cheat Sheet — Identity and STS

Your "whoami" — always run this first to confirm which identity/account you're operating as:

aws sts get-caller-identity            # who am I? which account?
aws configure list                     # active profile, region, source
aws configure list-profiles            # all configured profiles

Cheat Sheet — S3

aws s3 ls                                      # list buckets
aws s3 ls s3://my-app-assets/                  # list objects
aws s3 cp ./build/app.js s3://my-app-assets/   # upload one file
aws s3 sync ./build/ s3://my-app-assets/ --delete   # mirror a dir (great for React builds)
aws s3 rb s3://old-bucket --force              # remove bucket + contents

Cheat Sheet — EC2

aws ec2 describe-instances \
  --query 'Reservations[].Instances[].{ID:InstanceId,State:State.Name,Type:InstanceType,IP:PrivateIpAddress}' \
  --output table
aws ec2 describe-security-groups --group-ids sg-0a1b2c3d
aws ec2 stop-instances  --instance-ids i-0123456789abcdef0
aws ec2 start-instances --instance-ids i-0123456789abcdef0

Cheat Sheet — IAM

aws iam list-users
aws iam list-roles --query 'Roles[].RoleName' --output text
aws iam list-attached-role-policies --role-name app-ec2-role

Cheat Sheet — CloudWatch Logs

aws logs describe-log-groups
aws logs tail /aws/lambda/my-func --follow         # live tail, like tail -f
aws logs tail /ecs/web-app --since 1h --format short

Tip: Two habits that save careers: (1) keep a separate profile per account and run a quick aws sts get-caller-identity before any destructive command — confirm you're not in prod. (2) For mutating EC2 actions, add --dry-run: it checks permissions and returns "would have succeeded" without actually doing anything. Combine with --output table and --query (JMESPath) to keep output readable.

Cost and Security Guardrails

Two ways to get fired fast in cloud: a surprise five-figure bill, or a leaked access key that lets attackers mine crypto on your account. Both are 100% preventable with guardrails you set up once.

Cost Guardrails

Billing alerts / AWS Budgets: set a monthly budget and get emailed at 50/80/100% — your early-warning system.
Cost Explorer: visualize spend by service, account, and tag to find what's actually eating money.
Tagging strategy: tag every resource with env, team, project, owner. Without tags you can't attribute cost or clean up safely.
Right-sizing: most instances run at <20% CPU. Use Compute Optimizer to downsize over-provisioned EC2/RDS.
Spot and Savings Plans: Spot Instances (up to ~90% off) for fault-tolerant/batch workloads; Savings Plans / Reserved Instances for steady baseline compute.
Kill idle resources — the silent money drains:
- Unattached Elastic IPs — an EIP not associated with a running instance is billed hourly.
- NAT Gateways — billed per hour AND per GB; idle ones in dev accounts are a classic leak.
- Unattached EBS volumes and old snapshots — keep getting billed after the instance is long gone.
- Idle load balancers and stopped-but-not-terminated dev instances.

Caution: Free-tier traps: the free tier is 12 months and capped. NAT Gateways, Elastic IPs, inter-AZ data transfer, and provisioned RDS storage are not free. A forgotten t2.micro beyond the cap, or one NAT Gateway left running, quietly bills you for months. Always set a Budget on day one.

Set a $50 monthly budget with an 80% email alert via CLI:

cat > budget.json <<'EOF'
{ "BudgetName": "monthly-50usd", "BudgetLimit": {"Amount":"50","Unit":"USD"},
  "TimeUnit": "MONTHLY", "BudgetType": "COST" }
EOF

cat > notify.json <<'EOF'
[ { "Notification": {"NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN",
      "Threshold":80,"ThresholdType":"PERCENTAGE"},
    "Subscribers":[{"SubscriptionType":"EMAIL","Address":"your@email.com"}] } ]
EOF

aws budgets create-budget \
  --account-id 123456789012 \
  --budget file://budget.json \
  --notifications-with-subscribers file://notify.json

Tip: Set your very first Budget before launching anything. Add a second alert with NotificationType: FORECASTED so AWS warns you when it predicts you'll blow the budget — not just after you already have.

Security Guardrails

MFA on the root account — then lock the root keys away and never use root for daily work.
Least privilege — grant only the permissions a role/user actually needs; start narrow and widen if required.
Never commit access keys — no keys in Git, Dockerfiles, or env files in repos. Use a .gitignore and a secrets scanner in CI.
Roles, not keys — EC2/Lambda/ECS use IAM roles for temporary, auto-rotating credentials. Long-lived keys are the #1 breach vector.
Encryption — at rest with KMS (enable on S3, EBS, RDS), in transit with TLS everywhere.
CloudTrail + GuardDuty — CloudTrail logs every API call (the audit trail); GuardDuty uses ML to flag suspicious behavior (e.g. crypto-mining, anomalous API calls).
Security Groups least-open — never 0.0.0.0/0 on SSH/RDP/DB ports. Reference SGs by ID and open only what's needed.
S3 Block Public Access — keep it ON at the account level unless a bucket is deliberately a public website.

Caution: The classic disaster: a developer commits an access key to a public GitHub repo. Bots scan GitHub within minutes and spin up expensive GPU instances for crypto mining. Use IAM roles instead of keys, and enable a secret scanner so a key never reaches the repo in the first place.

Enable a virtual MFA device for an IAM user:

# Create the virtual MFA device
aws iam create-virtual-mfa-device \
  --virtual-mfa-device-name my-mfa \
  --outfile /tmp/qr.png --bootstrap-method QRCodePNG

# Scan QR in your authenticator app, then enable with two consecutive codes
aws iam enable-mfa-device \
  --user-name myuser \
  --serial-number arn:aws:iam::123456789012:mfa/my-mfa \
  --authentication-code1 123456 \
  --authentication-code2 654321

# Turn ON account-wide S3 public access block
aws s3control put-public-access-block \
  --account-id 123456789012 \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Tip: A simple priority order for any new account: (1) MFA on root, (2) a Budget alert, (3) CloudTrail on, (4) S3 Block Public Access on, (5) IAM roles instead of keys. Five steps, fifteen minutes, and you've dodged the most common cost and security incidents.

Note: Prefer a day-by-day path? This is covered in Mission 90 Days 46–65 — a free 90-day guided DevOps program with browser terminal missions.

Running containers on AWS? Continue with Kubernetes for DevOps.

How to Validate .gitlab-ci.yml Before You Push

opscanopy — Tue, 11 Aug 2026 15:12:10 +0000

You change one line in .gitlab-ci.yml, push, and switch to something else. Two minutes later the pipeline goes red — not because the build broke, but because a job points at a stage you renamed last week. You fix the typo, push again, wait again. This is the loop, and the only way out is to validate .gitlab-ci.yml before the commit lands, not after the runner tells you.

The frustrating part is that GitLab already knows your config is broken the instant it parses it. It just doesn't tell you until you've pushed and burned a CI minute. The fix is to run that same check locally, in the browser, before you ever git push.

The push-and-pray loop

Here's the shape of the problem. You edit a job, push, and let GitLab be your linter:

git add .gitlab-ci.yml
git commit -m "split deploy into staging + prod"
git push
# wait for the runner to pick up the pipeline...
# pipeline failed: "chosen stage prod does not exist"
git commit -am "fix: declare prod stage"
git push
# wait again...

Each round trip is a commit you didn't want, a runner slot you didn't need, and a context switch that costs more than the typo did. The errors that cause this almost never need a runner to detect. They're visible the moment the YAML is parsed and the job graph is resolved — which is exactly what a validator does locally.

Two kinds of errors: YAML syntax vs structural

When GitLab rejects a pipeline, the failure is one of two categories, and they have completely different fixes.

The first is a YAML syntax error: the file isn't valid YAML at all, so nothing downstream can read it. The second is a structural error: the YAML parses fine, but the pipeline it describes is invalid — a job with no script, a stage that was never declared, a needs pointing at a job that doesn't exist.

# YAML error — the parser can't even build a document
build:
  script:
    - make
   - make test      # inconsistent indentation: parser bails here

# Structural error — valid YAML, invalid pipeline
deploy:
  stage: prod        # "prod" is not in stages: → GitLab refuses to run it
  script: ./deploy.sh

Valid YAML is only half the job. The GitLab CI Validator checks both in one pass: it parses the YAML first, and only if that succeeds does it run the structural checks against your jobs. If the parse fails, you get a single line-referenced error and nothing else — there's no point reporting "undefined stage" on a document that didn't parse.

YAML errors that bite: indentation, tabs, duplicate keys

YAML is whitespace-significant, and CI config is exactly the kind of nested structure where that bites. The classic GitLab error message — did not find expected key — is almost always one of these.

test:
  stage: test
    script:              # a literal TAB instead of spaces → parse error
    - npm test

variables:
  DEPLOY_ENV: staging
  DEPLOY_ENV: prod       # duplicate key — the first value is silently lost

deploy:
  script: &deploy_steps  # anchor defined...
    - ./deploy.sh
rollback:
  script: *deploy_step   # ...but referenced with a typo → "unknown alias"

A browser validator parses with a real YAML reader, so it reports the exact line where the structure broke. When you paste config and the result is Could not parse YAML: ... (line 4, column 2), that's the parser telling you precisely where to look — re-indent, swap the tab for spaces, or fix the anchor name, and re-validate.

Structural errors GitLab catches late: undefined stages, jobs with no script, bad needs/extends

These are the ones that make you wait for a runner only to be told the pipeline never started. They're the real reason to validate GitLab CI before push. The validator models the rules from GitLab's .gitlab-ci.yml keyword reference and flags each one with the offending job, the line, and the fix.

A job with no executable surface. Every visible job has to do something: run commands with script: (or the newer run:), start a downstream pipeline with trigger:, or inherit one of those via extends:. A job with none of them is rejected with the familiar "job config should implement a script: or a trigger: keyword."

# ERROR — empty-job defines no script, run, trigger, or extends
empty-job:
  stage: test
  # nothing here → GitLab won't run it

Note that an empty script: [] or script: "" counts as missing too — the validator treats only a non-empty command string or list as a real executable surface, the same way GitLab does.

A stage that isn't declared. If a job's stage: isn't in your stages: list (or one of the five defaults: .pre, build, test, deploy, .post), GitLab doesn't know when to run it.

stages:
  - build
  - test

release-job:
  stage: release      # ERROR — "release" is not in stages:
  script: make release

There's a subtle variant the validator also catches: a job that omits stage: defaults to the implicit test stage. If you declared a custom stages: list that doesn't include test, that job is now pointing at a stage you never declared — and GitLab fails with "chosen stage test does not exist."

needs / dependencies / extends pointing at a job that doesn't exist. Every name in needs:, dependencies:, or extends: has to resolve to a real job or hidden .template in the same file.

test:
  stage: test
  needs:
    - compile          # ERROR — no job named "compile"
  extends: .base       # ERROR — no template named ".base"
  script: make test

The validator builds the set of every job id and every .template, then checks each reference against it. Rename a template and forget to update an extends:, and it tells you which job broke before the runner does.

An invalid when: or a non-list rules:. The when: keyword only accepts on_success, on_failure, always, manual, delayed, or never. And rules: has to be a YAML list of rule objects — a bare mapping is a common mistake that silently changes when a job runs.

deploy:
  stage: deploy
  when: sometimes      # ERROR — not an allowed when value
  rules:
    if: '$CI_COMMIT_TAG'   # ERROR — rules must be a list, not a mapping
  script: ./deploy.sh

It also surfaces lower-severity advice: legacy only/except get an info note recommending rules: (the two can't be combined in one job), a top-level key that's one edit away from a reserved keyword — say varables: or beforescript: — gets a misspelling warning, and malformed image:/services: shapes are flagged as errors.

Validating before you push: GitLab CI Lint vs an in-browser validator

GitLab ships its own checker — CI Lint, inside the pipeline editor. It's authoritative: it resolves include: files and project-level CI/CD variables, which a client-side tool can't see. But it has a cost: it requires a project and a sign-in. You can't lint a snippet from a code review, a config you're drafting offline, or a proprietary pipeline you'd rather not paste into a hosted form.

So what does an in-browser validator actually check? Based on the engine, the flow is deterministic and entirely local:

Parse the YAML. Any failure returns a single line-referenced error and stops — no structural findings on an unparseable document.
Split the top level into global keywords (stages, default, variables, image, services…), visible jobs, and hidden .templates.
Resolve the stages — your declared stages: list, or the five defaults — into the set every job's stage: is checked against.
Check every job for an executable surface, a known stage, real needs/extends/dependencies targets, a valid when:, a list-shaped rules:, and sane image/services shapes.
Rank by severity — errors first, then warnings, then info — each with the line and a concrete remediation. It never throws; a parse failure is reported, not crashed.

The honest framing: a clean result in the browser is strong pre-push confidence on structure and syntax. It catches the entire class of mistakes that fail a pipeline before any job runs. For absolute certainty on a config that uses include: or project variables, confirm with GitLab's own CI Lint once you've pushed to a project — but use the in-browser pass to make that push count.

If you also run GitHub Actions, the same idea applies there: the GitHub Actions Validator finds YAML and security issues in your workflow files, and the GitHub Actions Expression Tester evaluates those ${{ … }} expressions before you push.

Wire it into your workflow

The validator is a paste-and-check tool, but the habit you want is "never push CI config you haven't validated." A pre-commit hook makes that automatic for the YAML half — catch the parse errors before the commit even forms:

#!/usr/bin/env bash
# .git/hooks/pre-commit — block a commit if .gitlab-ci.yml isn't valid YAML
set -euo pipefail

if git diff --cached --name-only | grep -q '^\.gitlab-ci\.yml$'; then
  # Fail fast on a syntax error before the commit lands.
  python -c "import sys, yaml; yaml.safe_load(open('.gitlab-ci.yml'))" \
    || { echo "✗ .gitlab-ci.yml is not valid YAML — commit blocked"; exit 1; }
  echo "✓ .gitlab-ci.yml parses — paste it into the validator for structural checks"
fi

A local YAML parse catches the indentation-and-tabs class instantly. For the structural class — undefined stages, broken needs, jobs with no script — paste the file into the browser validator before you push. The two together cover both error categories from the second section, and neither needs a runner.

# the loop you actually want
$ git add .gitlab-ci.yml          # pre-commit hook checks YAML
# paste .gitlab-ci.yml → validator → 0 errors
$ git commit -m "split deploy into staging + prod"
$ git push                        # green on the first try

Validate it now

The next time you touch .gitlab-ci.yml, don't let the runner be the first thing to read it. Paste the file into the GitLab CI Validator and you'll get the YAML errors and the structural mistakes — undefined stages, jobs with no script, broken needs/extends, invalid when: — in one pass, with the line and the fix for each. It runs entirely in your browser: no project, no login, and nothing uploaded, so it's safe for internal pipelines.

If you've ever pushed a CI change and hoped it worked, this is the step that was missing.

Unit Testing Loki Alert Rules: the gap promtool leaves

opscanopy — Tue, 11 Aug 2026 15:11:36 +0000

If you run Prometheus, you already have a safety net for your alerting logic: promtool test rules. You feed it a series of synthetic samples, declare what should fire and when, and CI tells you the moment a refactor breaks an alert. It’s the difference between catching a broken page rule in code review and discovering it during an incident.

Grafana Loki has no equivalent. You can write LogQL alerting and recording rules that look almost identical to their Prometheus cousins, load them into the ruler, and ship them — but there’s no first-class way to assert that a given stream of logs produces the alert you expect. The gap is real, it’s long-standing, and it’s exactly the kind of thing that bites you at 3 a.m.

Why promtool does not cover Loki

The instinctive move is to reach for promtool and point it at your Loki rules. It doesn’t work, and the reason is fundamental rather than cosmetic.

promtool test rules evaluates PromQL against a synthetic time series database. You describe metrics with the series/values syntax and the tool replays them through the rule engine. But a Loki alert rule doesn’t start from metrics — it starts from log lines. A rule like count_over_time({app="api"} |= "panic" [5m]) > 0 has to run a LogQL pipeline (stream selector, line filter, label extraction, then a metric aggregation) over raw log entries before there’s any series to evaluate. promtool has no concept of a log stream, no LogQL parser, and no way to materialise the intermediate metrics the way Loki’s query engine does. Feeding it Loki rules either errors out or silently tests the wrong thing.

So the test surface that matters for Loki — “given these log lines, does this LogQL rule fire?” — is precisely the surface promtool can’t reach.

Why this matters

LogQL alert rules are deceptively easy to get subtly wrong:

A line filter that matches more (or less) than you think because of an unescaped regex or a missing word boundary.
A label that you unwrap or label_format incorrectly, so the aggregation groups the wrong way.
A [5m] range and a for: 10m clause that interact so the alert never has enough data to fire, or fires far later than intended.
A recording rule whose output series silently changes labels after a pipeline edit, breaking every downstream alert that selects on it.

None of these are caught by YAML linting or a schema check. They’re behavioural bugs, and the only honest way to catch them is to run the rule against representative input and assert on the output. Without a test harness, that verification happens manually, infrequently, and usually after something has already paged the wrong team — or failed to page the right one.

What a Loki rule unit test should look like

The model promtool established is the right one; it just needs a log-shaped input. Instead of synthetic series, a Loki rule test should accept synthetic streams (a set of labels plus timestamped log lines), evaluate the rule at a chosen time, and assert on the alerts produced — something like this:

# loki-rule-tests.yaml
tests:
  - name: panic in api logs fires PanicDetected
    # Synthetic log streams replayed through the LogQL engine.
    input_streams:
      - labels: '{app="api", env="prod"}'
        entries:
          - { ts: "2026-06-08T10:00:30Z", line: "level=info msg=ok" }
          - { ts: "2026-06-08T10:01:10Z", line: "level=error msg=panic: nil map" }
          - { ts: "2026-06-08T10:02:40Z", line: "level=error msg=panic: nil map" }

    # Evaluate the rule group at this instant.
    eval_time: 2026-06-08T10:05:00Z

    alert_rule_test:
      - alertname: PanicDetected
        # What we expect the ruler to emit at eval_time.
        exp_alerts:
          - exp_labels:
              app: api
              env: prod
              severity: critical
            exp_annotations:
              summary: "Panic detected in api"

The rule under test is the same rule you ship to the ruler:

groups:
  - name: api-alerts
    rules:
      - alert: PanicDetected
        expr: |
          count_over_time({app="api", env="prod"} |= "panic" [5m]) > 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Panic detected in {{ $labels.app }}"

Read together, the test says: given two panic lines in the five-minute window before 10:05, the count_over_time(...) > 1 expression should be true, and the ruler should emit a PanicDetected alert carrying severity=critical and the app/env labels from the stream. Flip the input to a single panic line, or move one entry outside the [5m] window, and exp_alerts becomes empty — the test now guards both the firing and the not-firing case.

This is the shape every team that has asked for it on the Loki tracker keeps describing — see the long-running requests in Loki issues #7655 and #16659, where the community has repeatedly pointed out that a promtool-style unit test for LogQL rules simply doesn’t exist yet.

Closing the gap today

You don’t have to wait for upstream to ship this. AlertLint runs exactly this test loop in your browser: paste your Loki alerting and recording rules, define input_streams, declare your exp_alerts, and assert pass or fail before the rule ever reaches the ruler. Everything evaluates client-side — your rules and logs never leave the device — so you can wire it into review without touching infrastructure or sending data anywhere.

If you’ve ever shipped a Loki alert and hoped it worked, this is the missing step.

Try AlertLint — the Loki alert rule tester →

Feature	BRE (grep, sed)	ERE (egrep, grep -E, sed -E, awk)
Alternation	`\	`
Grouping	`\( \)`	`( )`
One or more	`\+`	`+`
Zero or one	`\?`	`?`
Repetition	`\{m,n\}`	`{m,n}`
Zero or more	`*`	`*`
Backrefs	`\1 \2`	`\1 \2`