DEV Community: Mo Rizal

Why Is Your Linux Server Slow? A Practical Guide to Performance Troubleshooting

Mo Rizal — Fri, 21 Aug 2026 03:21:50 +0000

A Linux server can be running without any obvious errors and still feel painfully slow.

Applications may take longer to respond, SSH sessions may become sluggish, users may suddenly experience high latency. The difficult part is that the root cause is not always obvious.

A slow Linux server does not necessarily mean that the CPU is overloaded.

It could be memory pressure, excessive disk I/O, network congestion, too many open file descriptors, or simply a process consuming resources unexpectedly.

The challenge for a DevOps Engineer is not just finding that the server is slow, but identifying why it is slow.

Instead of immediately restarting services or killing processes, we need to investigate the system, layer by layer:

Is the CPU under pressure?
Is the system running out of memory?
Is disk I/O becoming a bottleneck?
Are processes exhausting file descriptors?
Which process is actually responsible for the problem?

This article is a practical introduction to Linux performance troubleshooting using real reproducible scenarios.

Rather than looking at isolated commands, we will build a troubleshooting mindset: observe the symptom, collect evidence, identify the bottleneck, apply the appropriate solution, and verify the result.

To make the investigation reproducible, I created a companion lab repository containing scenarios that intentionally introduce problems:

github repository

High CPU

High CPU usage is one of the most common Linux performance problems.

In this scenario, we intentionally create a process that continuously executes an infinite loop.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./high-cpu/reproduce.sh
./high-cpu/reproduce.sh

The script will display a warning before starting the scenario.

While it is running, open another terminal to investigate the system.

Identify High CPU Usage

The first step is to confirm whether the system is actually experiencing high CPU utilization.

Use top:

top

Look at the CPU summary:

%Cpu(s)

Example result:

%Cpu(s): 95.2 us,  1.8 sy,  0.0 ni,  0.0 id,  2.5 wa, ...

The most important fields are:

us — CPU time spent running user-space processes
sy — CPU time spent running kernel-space processes
id — percentage of CPU time that is idle
wa — CPU time waiting for I/O

If id is consistently very low while us is high, the CPU is heavily utilized by processes.

Identify the Process

The next step is to find the process that consume the most CPU.

Use:

ps aux --sort=-%cpu | head

Example result:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       38690 90.5  0.0   7740  3512 pts/4   R+   02:39   0:16 bash ./reproduce.sh

The most important fields are:

PID — Process ID
%CPU — CPU consumption
%MEM — Memory consumption
STAT — Current process state
COMMAND — Command used to start the process

Here bash ./reproduce.sh process is clearly consuming a significant amount of CPU.

We now have a candidate process, but before terminating it, we should inspect it further.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 38690

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       38690   37782 96 02:39 pts/4    00:00:45 bash ./reproduce.sh

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

Understand the Cause

Inspect the process script:

cat reproduce.sh

The important part is:

while true; do
    :
done

This creates an infinite loop.

The loop continuously executes without any sleep, blocking operation, or other mechanism that would allow the process to stop consuming CPU therefore keeps the CPU busy.

This is the root cause of the high CPU usage in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 38690

Then check the CPU usage again:

top

The process should no longer appear in the process list, and CPU utilization should return to its previous level.

Disk I/O

High disk I/O is another common Linux performance problem.

In this scenario, we intentionally create a process that continuously writes data to disk.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./disk/reproduce.sh
./disk/reproduce.sh

Identify Disk I/O

The first step is to confirm whether the system is actually experiencing high Disk I/O activity.

Use iostat:

iostat -xz 1

Example result:

Device            r/s     w/s   rkB/s   wkB/s  %util
sda              0.00  125.00    0.00  51200.00  98.50

The most important fields are:

r/s — Number of read requests per second
w/s — Number of write requests per second
rkB/s — Data read per second
wkB/s — Data written per second
%util — Percentage of time the device was busy

If %util is consistently close to 100%, the storage device is heavily utilized.

Identify the Process

The next step is to find which process is generating the disk activity.

Use:

pidstat -d 1

Example result:

UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
0     41043      0.00 259548.51      0.00       0  bash

The most important fields are:

PID — Process ID
COMMAND — Command generating the I/O

Here the PID 41043 process is clearly generating significant disk write activity.

Inspect the Process

Use the PID identified from the previous step:

ps -fp 41043

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       41043   37782  0 03:03 pts/4    00:00:00 bash ./reproduce.sh

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is continuously writing data to testfile.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh

The important part is:

while true; do
    dd if=/dev/zero of=testfile bs=1M count=1024 conv=fsync
    rm -f testfile
done

The script continuously creates a file and writes data to it.

The dd command reads zero-filled data from /dev/zero and writes it to testfile.

The conv=fsync option forces dd to flush the written data to the storage device before completing the operation. This makes the scenario generate actual disk write activity instead of relying entirely on the filesystem page cache.

The file is then removed and the process repeats the operation.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 41043

Then check the disk activity again:

iostat -xz 1

You can also verify the processes performing I/O:

iotop

The dd process should no longer appear, disk write throughput should drop significantly, and %util should return to its previous level.

Finally, verify that the test file has been removed:

ls -lh testfile

Memory Pressure

High memory usage is another common Linux performance problem.

In this scenario, we create a process that allocates a large amount of memory and keeps it allocated.

The reproduction script is available in the github repository repository.

Run the reproduction script with a memory target:

chmod +x ./memory-pressure/reproduce.sh
./memory-pressure/reproduce.sh 2G

Replace 2G with the amount of memory you want to allocate.

Identify Memory Pressure

The first step is to confirm whether the system is actually experiencing high memory utilization.

Use:

free -h

Example result:

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       3.1Gi       141Mi       1.0Mi       822Mi       692Mi
Swap:             0B          0B          0B

The most important fields are:

total — Total physical memory available
used — Memory currently used by the system
free — Completely unused memory
buff/cache — Memory used by the kernel for buffers and filesystem cache
available — Memory available for new applications

The available value is particularly important when diagnosing memory pressure. If available memory is consistently very low while used memory is high, the system is under significant memory pressure.

If the system has swap enabled, also pay attention to swap usage. Significant swap usage can indicate that the system does not have enough physical memory to satisfy current workloads.

Identify the Process

The next step is to find which process is consuming the most memory.

Use:

ps aux --sort=-%mem | head

Example result:

USER         PID %CPU %MEM    VSZ    RSS TTY      STAT START   TIME COMMAND
root       39937  0.1 52.7 2123808 2116080 pts/4 S+   02:56   0:03 python3 ./memory_hog.py 2G

The most important fields are:

PID — Process ID
%MEM — Percentage of physical memory used by the process
VSZ — Virtual memory size
RSS — Resident memory currently held in RAM
STAT — Current process state
COMMAND — Command used to start the process

Here the PID 39937 process is clearly consuming a significant amount of memory.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 39937

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       39937   39910  0 02:56 pts/4    00:00:03 python3 ./memory_hog.py 2G

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is running memory_hog.py with a target allocation of 2G.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh

The shell script starts the Python memory allocation program:

python3 "$(dirname "$0")/memory_hog.py" "$MEMORY"

The Python script can then be inspected:

cat memory_hog.py

The important part is:

while allocated_bytes < amount:
    remaining = amount - allocated_bytes
    size = min(chunk_size, remaining)

    chunk = bytearray(size)

    chunk[0] = 1

    allocated.append(chunk)
    allocated_bytes += size

The script allocates memory in 1 MiB chunks until the requested memory amount is reached.

Each allocated chunk is stored in the allocated list:

allocated.append(chunk)

This keeps references to the allocated memory, preventing Python from releasing those objects.

The script then remains running:

while True:
    time.sleep(1)

This large memory allocation is the root cause of the memory pressure in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 39937

Then check the memory usage again:

free -h

You can also verify the processes sorted by memory usage:

ps aux --sort=-%mem | head

The memory_hog.py process should no longer appear in the process list.

The available memory should increase, and overall memory utilization should return closer to its previous level.

File Descriptor Exhaustion

In this scenario, we intentionally create a process that continuously opens file descriptors without closing them.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./file-descriptor/reproduce.sh
./file-descriptor/reproduce.sh

Identify File Descriptor Pressure

The first step is to check the system-wide file descriptor usage.

Use:

cat /proc/sys/fs/file-nr

Example result:

2656    0    9223372036854775807

The values represent:

allocated — Number of allocated file handles
unused — Number of unused allocated file handles
maximum — Maximum number of file handles allowed system-wide

The allocated value represents file handles currently allocated by the kernel.

A high number of allocated file handles can indicate file descriptor pressure, especially when applications are approaching their per-process or system-wide limits.

Identify the Process

The next step is to find the process running fd_exhaustion.py.

Use:

ps aux | grep '[f]d_exhaustion'

Example result:

root       42297  0.2  0.2  18488 10504 pts/4    S+   03:11   0:00 python3 ./fd_exhaustion.py

The most important fields are:

PID — Process ID
%CPU — CPU usage
%MEM — Memory usage
COMMAND — Command used to start the process

Here the python3 ./fd_exhaustion.py process is the candidate responsible for the file descriptor exhaustion.

Use the identified PID for the next step.

Inspect the Process

The next step is to determine how many file descriptors the process currently has open.

Use:

ls /proc/42297/fd | wc -l

Example result:

This indicates that the process currently has 1024 file descriptors open.

You can inspect the actual descriptors with:

ls -l /proc/<PID>/fd

Example:

lr-x------ 1 root root 64 Aug 21 03:11 0 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 1 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 2 -> /dev/pts/4
lr-x------ 1 root root 64 Aug 21 03:11 3 -> /dev/null
lr-x------ 1 root root 64 Aug 21 03:11 4 -> /dev/null

The large number of /dev/null entries indicates that the process is continuously opening new file descriptors.

Next, check the process limits:

cat /proc/42297/limits | grep "open files"

Output:

Max open files            1024                 1048576                files

The first value is the soft limit, while the second value is the hard limit.

The soft limit is the limit currently enforced for the process.

Once the process reaches this limit, attempts to open additional file descriptors will fail.

Understand the Cause

Inspect the reproduction script:

cat fd_exhaustion.py

The important part is:

while True:
    try:
        fd = os.open(
            "/dev/null",
            os.O_RDONLY
        )

        FILES.append(fd)

        counter += 1

The script continuously opens /dev/null using os.open().

Every returned file descriptor is stored in the FILES list:

FILES.append(fd)

The descriptors are intentionally not closed while the scenario is running. As a result, the number of open file descriptors continuously increases

Eventually, os.open() fails because the process has reached its file descriptor limit.

The script then reports the error:

Failed to open new file descriptor
[Errno 24] Too many open files

This is the root cause of the file descriptor exhaustion in this scenario.

File descriptor exhaustion can have broader consequences for real applications. Processes may fail to open files, accept new connections, create sockets, or perform other operations that require file descriptors.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 42297

Verify that the process has stopped:

ps -p 42297

If the process has stopped, the command should return no process entry.

You can also verify that the process no longer exists:

test -d /proc/<PID> && echo "Process still exists" || echo "Process stopped"

The file descriptors belonging to the process are automatically released by the kernel when the process terminates.

Therefore, /proc/<PID>/fd will no longer be available after the process has exited.

Finally, verify the system-wide file descriptor state again:

cat /proc/sys/fs/file-nr

The allocated file handle count should return closer to its previous level after the exhausted process has been terminated.

Network Connection Exhaustion

In this scenario, we intentionally create a process that continuously opens TCP connections to a local server and keeps those connections open.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./network-connection/reproduce.sh
./network-connection/reproduce.sh

The script will display a warning before starting the scenario.

The reproduction uses 127.0.0.1:9000 as the target, so all connections remain on the local machine.

Identify Abnormal Connections

The first step is to confirm whether the system is experiencing an unusually large number of TCP connections.

Use ss:

ss -s

Example result:

Total: 1252
TCP:   2051 (estab 1023, closed 1021, orphaned 1, timewait 1020)

Transport Total     IP        IPv6
RAW       1         0         1
UDP       3         3         0
TCP       1030      1028      2
INET      1034      1031      3
FRAG      0         0         0

The most important fields are:

estab — Established TCP connections
orphaned — Orphaned TCP connections
timewait — Connections waiting to be fully closed
TCP — Total TCP sockets

A significant increase in established connections can indicate abnormal connection usage.

In this scenario, the large number of estab connections is the main indicator that something is continuously creating and maintaining TCP connections.

Identify the Connection and Process

The next step is to identify which process owns the connections.

Use:

ss -tanp

Example result:

CLOSE-WAIT  1  0  127.0.0.1:46374  127.0.0.1:9000  users:(("python3",pid=46938,fd=880))
CLOSE-WAIT  1  0  127.0.0.1:39120  127.0.0.1:9000  users:(("python3",pid=46938,fd=86))
CLOSE-WAIT  1  0  127.0.0.1:44192  127.0.0.1:9000  users:(("python3",pid=46938,fd=634))

The most important information is:

State — Current TCP connection state
Local Address:Port — Local endpoint
Peer Address:Port — Remote endpoint
users — Process owning the socket
pid — Process ID
fd — File descriptor associated with the socket

Here we can identify the process responsible for the connections:

users:(("python3",pid=46938,fd=880))

The PID is:

The same PID appearing across many connections is a strong indication that one process is responsible for creating the connection buildup.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 46938

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       46938   46922  0 03:42 pts/1    00:00:00 python3 ./connection_exhaustion.py 127.0.0.1 9000

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is running connection_exhaustion.py and connecting to 127.0.0.1:9000.

Next, check how many file descriptors the process currently has open:

ls /proc/46938/fd | wc -l

A large number of file descriptors is expected because every TCP socket consumes a file descriptor.

You can inspect the descriptors directly:

ls -l /proc/46938/fd | head

You should see entries pointing to sockets, for example:

lrwx------ 1 root root 64 Aug 21 03:42 86 -> 'socket:[123456]'
lrwx------ 1 root root 64 Aug 21 03:42 87 -> 'socket:[123457]'
lrwx------ 1 root root 64 Aug 21 03:42 88 -> 'socket:[123458]'

This confirms that the process is holding a large number of socket file descriptors.

Understand the Cause

Inspect the reproduction script:

cat connection_exhaustion.py

The important part is:

while True:
    try:
        sock = socket.socket(
            socket.AF_INET,
            socket.SOCK_STREAM
        )

        sock.connect((host, port))

        CONNECTIONS.append(sock)

        counter += 1

The script continuously creates TCP sockets and connects them to the target server.

Every successful connection is stored in the CONNECTIONS list:

CONNECTIONS.append(sock)

The sockets are intentionally not closed while the scenario is running. As a result, the number of active connections continuously increases:

The TCP server is also intentionally designed to keep accepted connections open:

connection, address = server.accept()
connections.append(connection)

This means the server does not immediately close the connections created by the client. Eventually, the client process can reach its file descriptor limit and fail to create additional sockets.

This pattern can occur in real applications when connections are not properly closed, connection pools are misconfigured, or an application continuously creates new connections instead of reusing existing ones.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 46938

Then check the network connection summary again:

ss -s

You can also check the connections to port 9000 directly:

ss -tan | grep ':9000'

The number of established TCP connections should return to its previous level.

Conclusion

Linux performance troubleshooting is not about finding a single command that tells you why a server is slow.

It is about building a structured investigation from the symptoms you observe.

In the scenarios covered in this article, we investigated several different types of Linux performance problems:

High CPU caused by an infinite loop
High disk I/O caused by continuous writes
Memory pressure caused by excessive memory allocation
File descriptor exhaustion caused by descriptors that are never closed
Network connection exhaustion caused by continuously opened TCP connections

Although the symptoms are different, the troubleshooting approach remains consistent:

The important lesson is to avoid making assumptions based on a single metric.

High CPU does not immediately tell you which process is responsible. High memory usage does not necessarily mean the system is running out of usable memory. A large number of TCP connections does not automatically mean the network is slow.

Each symptom needs to be investigated with the right evidence.

When a production server becomes slow, the most valuable skill is not knowing how to restart it.

It is being able to answer:

What is happening, why is it happening, and how can I prove that my fix actually worked?

The scenarios in this article are intentionally simplified, but the troubleshooting methodology can be applied to much more complex production incidents.

You can reproduce all of these scenarios and investigate yourself by cloning my github repository bellow:

https://github.com/muhammadyulasfipahrizal/linux-performance-lab.git

7 Bash Scripts Every Devops Engineer Should Have

Mo Rizal — Tue, 18 Aug 2026 03:58:05 +0000

A production server rarely fails in a convenient way.

A disk suddenly fills up. A process consumes all available CPU. A network connection becomes unreliable, or an SSH account needs to be investigated after a suspicious login.

When this happens, a DevOps engineer usually does not start by writing a new monitoring system. The first response is often much simpler: connect to the server and investigate what is happening.

The problem is that the investigation itself can become repetitive.

Running df, du, ps, ss, journalctl, checking authentication logs, and collecting system information manually may seem trivial on one server. But during an incident, repeatedly remembering which commands to run, in which order, and how to collect the results wastes time and makes investigations inconsistent.

This is where Bash becomes useful.

Instead of treating Bash as a language for small one line commands, we can use it to turn recurring operational procedures into reusable tools. The goal is not to replace observability platforms or configuration management systems. The goal is to create lightweight scripts that can be executed directly on a server when an engineer needs fast, consistent information.

In this project, we will build a collection of 7 practical bash scripts for common DevOps operations:

Investigating disk usage
Collecting incident context
Inspecting log activity within a specific time window
Investigating network paths
Auditing running processes
Auditing SSH access
Cleaning up unnecessary system resources

Each script is designed around a real operational problem.

By the end of this project, we will have a small Bash toolkit that can turn common server investigations from a collection of ad-hoc commands into repeatable operational workflows.

The code snippets in this article focus on the important parts of the implementation. For the complete Bash Scripts you can find the full source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/terraform-setup

1. Disk Investigator

disk-investigator.sh is a filesystem investigation script designed to answer a common question during Linux server incidents:

What is consuming the disk, and where should I investigate next?

Instead of running several commands manually, the script combines filesystem, inode, directory, file, and modification-time information into a single investigation report.

The script accepts three parameters:

--path controls which directory or filesystem is investigated.
--top controls how many results are displayed.
--min-size defines what qualifies as a large file

The script starts with the filesystem itself. It reports the filesystem type and basic capacity information, then checks inode consumption. This distinction is important because a server can run out of inodes even when there is still available disk space.

It then moves from a broad view to more specific information:

Filesystem capacity and inode usage
Largest directories
Largest files
Large files modified within the last 24 hours

2. Incident Context

incident-context.sh collects the current state of a Linux server into a single incident report.

During an incident, the first challenge is often not finding the exact root cause immediately. It is establishing what the server looked like when the problem occurred.

Instead of manually running commands such as uptime, free, df, ps, ip, ss, systemctl, and journalctl, this script collects those signals together and presents them as one structured snapshot.

The report is organized around the main areas an engineer would typically investigate:

The script also introduces an important operational concept: time-bounded investigation.

By default, it collects journal and kernel information from the last 30 minutes, but the window can be changed with --since, for example:

sudo ./incident-context.sh --since "2 hours ago"

This prevents the report from becoming an uncontrolled dump of historical logs. During an incident, the most useful information is usually the information surrounding the period in which the failure occurred.

The script can also write the collected information to a report file:

sudo ./incident-context.sh --output incident-report.txt

The result is a reusable incident snapshot: instead of starting an investigation with an empty terminal and a list of commands to remember, the engineer can generate a consistent baseline of the server's state and use it as the starting point for deeper investigation.

3. Log Window

log-window.sh is a focused log investigation tool for extracting only the log entries that are relevant to an incident.

When an application produces thousands of log lines, searching the entire file manually can make an investigation unnecessarily difficult. Usually, an engineer already has some context about the problem:

when the problem happened,
what severity the event had, or
a phrase associated with the failure.

Instead of reading the entire log file, this script allows those clues to become filters.

The script supports four independent filters:

These filters can be combined like this:

./log-window.sh --file ../logs/dummy.log \
--level WARN \
--contains "Database response time increased" \
--start "2026-08-18 10:07:00"  --end "2026-08-19 10:09:00"

4. Network Path

network-path.sh is a network troubleshooting script that checks connectivity to a destination layer by layer, instead of treating connectivity as a single yes-or-no question.

A connection failure can happen at different points. DNS might fail to resolve the hostname, the routing table might not provide a valid path, the TCP port might be unreachable, or an application-layer problem might occur after the connection succeeds.

If DNS fails, there is no reason to continue testing TCP. If DNS and routing succeed but TCP fails, the investigation can focus on connectivity, firewall rules, security groups, or the destination service rather than the application itself.

The script therefore produces a more useful result than simply:

Connection failed

It can identify a failure boundary such as:

DNS      PASS
Route    PASS
TCP      FAIL
TLS      SKIP
HTTP     SKIP

Failure detected at: TCP

The script requires a destination host and TCP port:

./network-path.sh \
    --host google.com \
    --port 443 \
    --https

5. Process Audit

process-audit.sh provides a deeper view of how running processes are consuming system resources.

Commands such as top and ps are excellent for quickly identifying CPU or memory usage, but during troubleshooting we often need more context. A process consuming resources may also have an unusually high number of file descriptors, threads, context switches, or a large resident memory footprint.

This script collects those signals for every accessible process and turns them into a structured resource audit.

The script can rank processes by different resource dimensions:

sudo ./process-audit.sh --sort cpu
sudo ./process-audit.sh --sort memory
sudo ./process-audit.sh --sort fd
sudo ./process-audit.sh --sort threads

6. SSH Access Audit

ssh-access-audit.sh is a read-only security audit tool for examining how users can access a Linux server through SSH and what privileges they have after gaining access.

SSH is often the primary entry point for administrators and automation. Because of that, troubleshooting SSH access should not only ask:

Is SSH running?

It should also answer:

Who can access the server, how can they authenticate, what keys are authorized, and what privileges do those users have?

The script approaches SSH access from several layers:

The script evaluates the effective SSH configuration using sshd -T, covering authentication settings such as root login, password authentication, public-key authentication, and keyboard-interactive authentication.

Finally, findings are classified as HIGH, MEDIUM, or INFO so critical security issues can be prioritized.

7. System Cleanup

system-cleanup.sh is a safe filesystem cleanup tool that identifies files that may be consuming unnecessary disk space and removes them only after explicit confirmation.

Disk cleanup is inherently risky because deleting the wrong file can cause service failures or data loss. Instead of immediately running destructive commands such as rm -rf, the script separates cleanup into two phases:

It looks for cleanup candidates in common locations:

/tmp
/var/tmp
APT package cache
Old compressed logs
Systemd journal usage

With this script, engineers don't need to manually search for and delete files, reducing the risk of accidentally removing important data. The script only deletes discovered candidates from predefined safe locations after explicit confirmation.

Conclusion

These 7 bash scripts turn common Linux administration tasks into a practical operational toolkit.

Each script provides a focused capability:

Disk Investigator helps identify what is consuming storage.
Incident Context provides a structured view of the server state.
Log Window makes relevant events easier to isolate.
Network Path helps pinpoint where connectivity breaks.
Process Audit reveals which processes are consuming system resources.
SSH Access Audit exposes potential access and privilege risks.
System Cleanup makes disk cleanup safer and more controlled.

The biggest benefit comes from using them together. Instead of relying on individual commands and ad-hoc procedures, engineers have a consistent set of tools for investigation, diagnosis, security auditing, and cleanup.

More importantly, these scripts can become building blocks for larger operational practices. They can be extended with additional checks, integrated into automation, or adapted to the specific standards of an organization's infrastructure.

A command that you run once is useful.

A well designed script that consistently performs the same set of commands is an operational asset.

You can find the source code for this article in my github repository:
Github Repository

Building AWS Infrastructure The Right Way With Terraform

Mo Rizal — Sat, 15 Aug 2026 07:10:44 +0000

Provisioning AWS infrastructure is easy when there are only a few resources.

Open the AWS Console, create a VPC, add Subnets, configure Security Group, launch EC2 instance, etc.

The problem starts when the infrastructure grows.

A few manually created resources can quickly become dozens of resources with different configurations. One environment may have different network settings, another may contain a slightly different security rule.

Over time, the infrastructure becomes difficult to understand and even harder to reproduce.

The problem is not only how to create AWS resources, but how to structure them so the infrastructure does not become a mess.

This is where Terraform becomes valuable.

Instead of managing infrastructure through the AWS Console, we can define it as code, store it in Git, review changes, and reproduce the same architecture across environments.

In this project, we will build a small production style AWS setup using Terraform.

This project will include:

VPC
Public and private subnets
Internet gateway
Route tables
Security group
EC2 instance
IAM Role
S3 bucket
Terraform remote state

But the infrastructure itself is only part of the project.

The main goal is to apply several Terraform design principles that become increasingly important as infrastructure grows:

Modularization
Environment Separation
Variables Instead of Hardcoding
Outputs
State Management

The goal is simple:

Instead of managing AWS infrastructure manually, define it as reusable Terraform code that can be reviewed, reproduced, and maintained.

What We Are Building

Before writing Terraform code, let's define the architecture we want to build.

The final environment contains a VPC with public and private subnets. The EC2 instance runs in the public subnet so we can connect to it for validation.

The private subnet is included to demonstrate basic network separation and provide a foundation for expanding the architecture later if needed.

Final Architecture

The EC2 instance will be placed in the public subnet while the private subnet is included to demonstrate a basic network separation pattern

The network uses:

VPC              10.0.0.0/16
Public Subnet    10.0.1.0/24
Private Subnet   10.0.2.0/24

The EC2 instance is also associated with an IAM role so workloads can interact with AWS services without storing long lived AWS credentials on the server.

Terraform Architecture

The Terraform configuration is divided into three layers:

Bootstrap

The bootstrap directory is responsible for creating the resources required before the main Terraform environments can use remote state.

In this project, that means creating the S3 bucket used for Terraform state.

Environments

The environments directory contains environment specific configurations.

The environments are not responsible for implementing every AWS resource themselves. Instead, they will compose reusable modules.

For example, the dev environment can use:

The same modules can then be reused by the prod environment with different variables.

This allows the same infrastructure modules to be reused with different values for development and production.

Modules

The modules directory contains reusable infrastructure components.

Each module has a specific responsibility.

For example:

modules/vpc/

contains the logic for creating the VPC and its networking components.

Meanwhile:

modules/ec2/

contains the logic for creating the EC2 instance.

Project Structure

The module files follow Terraform conventional structure:

main.tf — resources
variables.tf — inputs
outputs.tf — values exposed for othe configurations

The environment directories additionally contain:

backend.tf — remote state configuration
terraform.tfvars— environment values

The rest of the article will use the dev environment to provision the infrastructure.

The code snippets in this article focus on the important parts of the implementation. For the complete Terraform configuration you can find the full source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/terraform-setup

Configure Remote State

Before provisioning the AWS infrastructure, we need to solve one problem

Where should Terraform store its state?

Terraform uses a state file to keep track of the infrastructure it manages. By default, this state is stored locally as:

terraform.tfstate

For a small personal project, local state may be enough.

However, once infrastructure is shared between environments, machines, and engineers, relying on a local state file becomes difficult to manage.

A better approach is to store the state remotely in AWS S3.

In this project, we use a separate bootstrap configuration to create the S3 bucket that will later be used as the Terraform backend.

Bootstrap

The purpose of this configuration is to create the S3 bucket required by the Terraform environments.

The bootstrap configuration can be initialized and applied independently:

cd bootstrap

terraform init

terraform plan

terraform apply

After the apply completes, the S3 bucket required for remote state exists.

We can now configure the dev environment to use it.

Configure the Dev Backend

The dev environment contains a backend.tf file:

The backend configuration tells Terraform where its state should be stored.

terraform {
  backend "s3" {
    bucket = "terraform-right-way"
    key    = "dev/terraform.tfstate"
    region = "ap-southeast-3"
  }
}

The important part is the key:

key = "dev/terraform.tfstate"

This allows different environments to maintain separate state locations within the same bucket.

Initialize the Dev Environment

Now move into the dev environment:

cd environments/dev

Then run:

terraform init

Terraform reads backend.tf and configures the S3 backend.

If Terraform was previously using local state, it may ask whether the existing state should be migrated to the new backend.

For a new environment with no existing infrastructure, there is normally no local state to migrate.

After initialization, Terraform is ready to use the S3 backend.

Why Bootstrap Is Separate

There is a small dependency problem when using S3 as the Terraform backend:

We solve this by creating the state bucket through a separate bootstrap configuration.

VPC

The first infrastructure component is the VPC, which provides the network boundary for our AWS resources.

VPC              10.0.0.0/16
Public Subnet    10.0.1.0/24
Private Subnet   10.0.2.0/24

The module creates the VPC, subnets, Internet Gateway, and route tables.

The public subnet is associated with a route table that sends Internet bound traffic through the Internet Gateway. The private subnet does not have a direct Internet route.

The module exposes the values required by other parts of the infrastructure:

output "vpc_id" {
  value = aws_vpc.this.id
}

output "public_subnet_id" {
  value = aws_subnet.public.id
}

output "private_subnet_id" {
  value = aws_subnet.private.id
}

The dev environment then consumes the module:

module "vpc" {
  source = "../../modules/vpc"

  vpc_cidr            = var.vpc_cidr
  public_subnet_cidr  = var.public_subnet_cidr
  private_subnet_cidr = var.private_subnet_cidr
  availability_zone   = var.availability_zone
}

The important design decision is that the CIDR ranges are variables, rather than being hardcoded directly into the environment's resources.

Create the Security Group

The Security Group will be attached to the EC2 instance and will define the allowed inbound traffic.

The main rule we need is SSH access to the EC2 instance.

Instead of hardcoding an IP address inside the module, the allowed SSH source is provided as a variable:

variable "allowed_ssh_cidr" {
  description = "CIDR block allowed to access SSH"
  type        = string
}

The Security Group can then use that value for its SSH rule:

ingress {
  description = "Allow SSH"
  protocol    = "tcp"
  from_port   = 22
  to_port     = 22
  cidr_blocks = [var.allowed_ssh_cidr]
}

This is an important design decision.

We don't want the module to assume that SSH should always be accessible from a particular IP address. The module defines the rule, while the environment decides who should be allowed to use it.

For example, the dev environment can provide:

allowed_ssh_cidr = "YOUR_IP/32"

Using /32 limits SSH access to a single public IP address.

The Security Group also needs to allow outbound traffic so the EC2 instance can communicate with external services when required.

Once created, the module exposes the Security Group ID:

output "security_group_id" {
  value = aws_security_group.this.id
}

The dev environment can then pass the output to the EC2 module:

module "ec2" {
  source = "../../modules/ec2"

  subnet_id         = module.vpc.public_subnet_id
  security_group_id = module.security_group.security_group_id
}

This creates another simple dependency between our modules without coupling their implementations.

Deploy EC2

The modules/ec2/ configuration provides the values required by the module, such as the AMI, instance type, subnet, Security Group, and SSH key.

The module then connects the EC2 instance to the infrastructure we created earlier:

module "ec2" {
  source = "../../modules/ec2"

  subnet_id         = module.vpc.public_subnet_id
  security_group_id = module.security_group.security_group_id

  ami_id            = var.ami_id
  instance_type     = var.instance_type
  key_name          = var.key_name
}

The important part here is how the dependencies are connected.

The EC2 instance does not need to know how the VPC or Security Group is implemented. It only consumes their outputs:

module.vpc.public_subnet_id
module.security_group.security_group_id

The actual EC2 resource remains inside the module:

resource "aws_instance" "this" {
  ami                    = var.ami_id
  instance_type          = var.instance_type
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [var.security_group_id]
  key_name               = var.key_name

  associate_public_ip_address = true

  tags = {
    Name = var.instance_name
  }
}

The instance is placed in the public subnet and receives a public IP so that we can connect to it and validate the infrastructure after deployment.

The values that can vary between environments are kept outside the module.

For example:

instance_type = "t3.micro"

The same EC2 module can later be used by production with a different instance type without changing the module itself.

The module also exposes useful information through outputs:

output "instance_id" {
  value = aws_instance.this.id
}

output "public_ip" {
  value = aws_instance.this.public_ip
}

This allows Terraform to display the instance information after deployment and gives other parts of the configuration a clean interface to consume it.

Create the IAM Role

The EC2 instance may need to interact with other AWS services. A common but unsafe approach would be to store AWS access keys directly on the server.

Instead, AWS provides IAM roles for EC2, allowing applications running on the instance to obtain temporary credentials automatically.

The module creates an IAM role with an EC2 trust policy:

resource "aws_iam_role" "this" {
  name = var.role_name

  assume_role_policy = jsonencode({
    Version = "2012-10-17"

    Statement = [
      {
        Effect = "Allow"

        Principal = {
          Service = "ec2.amazonaws.com"
        }

        Action = "sts:AssumeRole"
      }
    ]
  })
}

The important part is the trust relationship:

This allows the EC2 service to assume the role on behalf of the instance.

The role is then associated with an instance profile:

resource "aws_iam_instance_profile" "this" {
  name = var.instance_profile_name
  role = aws_iam_role.this.name
}

The instance profile is what allows the IAM role to be attached to the EC2 instance.

The EC2 module can then receive the instance profile:

resource "aws_instance" "this" {
  iam_instance_profile = var.iam_instance_profile
}

This keeps AWS credentials out of the server's configuration.

For this project, the role does not need broad permissions simply because the EC2 instance exists. Permissions should be added according to the actual AWS operations the workload requires.

Create the S3 Bucket

It is important to distinguish this bucket from the S3 bucket created during the bootstrap stage.

The bootstrap bucket is used by Terraform to store remote state, while this S3 bucket is part of the infrastructure managed by the dev environment.

resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name

  tags = {
    Name = var.bucket_name
  }
}

The bucket name is provided by the environment rather than being hardcoded inside the module.

bucket_name = "dev-server-bucket"

The module exposes the bucket information through outputs:

output "bucket_id" {
  value = aws_s3_bucket.this.id
}

output "bucket_arn" {
  value = aws_s3_bucket.this.arn
}

The dev environment can then consume the module:

module "s3" {
  source = "../../modules/s3"

  bucket_name = var.bucket_name
}

This follows the same pattern used by the other infrastructure components: the module contains the implementation, while the environment provides the configuration.

Validate and Deploy

We have now defined the infrastructure as Terraform code. Before creating anything in AWS, we should validate the configuration and review the changes Terraform intends to make.

For this project, we will deploy the dev environment from:

environments/dev/

Format the Configuration

cd environments/dev

terraform fmt -recursive

This keeps the Terraform configuration consistently formatted.

Validate the Configuration

terraform validate

Terraform checks whether the configuration is syntactically valid and whether the configuration can be successfully loaded.

A successful validation should return:

Success! The configuration is valid.

Review the Execution Plan

terraform plan

Terraform compares the desired configuration with the current state and shows which resources it intends to create, change, or destroy.

For a new development environment, we should expect Terraform to plan the creation of our infrastructure.

This step is important because terraform plan gives us an opportunity to review the changes before they are applied.

Apply the Infrastructure

Once the plan looks correct, apply it:

terraform apply

Terraform will display the execution plan again and ask for confirmation.

Enter:

yes

Terraform will then create the resources defined by the dev environment.

Why This Structure Matters

At first, this structure may look more complicated than putting everything into one main.tf.

For a small project, that may be true.

The benefit becomes clearer when the infrastructure grows.

A new environment can reuse the existing modules:

dev  → modules
prod → modules

Instead of duplicating the VPC, EC2, IAM, and S3 implementation, each environment provides its own configuration.

If the network implementation changes, we know where to look.
If production needs a larger EC2 instance, we change the production configuration rather than duplicating the EC2 module.
If another environment is introduced, it can reuse the existing modules.

Conclusion

Terraform is not difficult because creating an AWS resource is complicated.

The real challenge is managing infrastructure as it grows.

A single main.tf can work for a small experiment, but infrastructure becomes harder to maintain when environments, networking, compute, IAM, storage, and state management are all mixed together.

In this project, we applied five core concepts:

Modularization — infrastructure was divided into reusable modules for networking, security, compute, IAM, and storage instead of keeping everything in a single configuration.
Environment Separation — development and production have their own configurations while sharing the same reusable modules.
Variables Instead of Hardcoding — values such as CIDR ranges, instance types, AMIs, SSH access, and resource names are provided through variables.
Outputs — modules expose the values that other parts of the infrastructure need, allowing resources to be connected without tightly coupling their implementations.
State Management — Terraform state is stored remotely in S3, with separate state paths for each environment.

These concepts may seem unnecessary for a small infrastructure project. However, they become increasingly important as the number of resources and environments grows.

You can find the source code for this article in my github repository:

Github Repository

I built a zero touch ubuntu server provisioning workflow with ansible

Mo Rizal — Tue, 11 Aug 2026 04:09:47 +0000

Mo Rizal

Aug 11

Zero-Touch Provisioning With Ansible: From 0 To Secure Production Ready Server

16 min read

Zero-Touch Provisioning With Ansible: From 0 To Secure Production Ready Server

Mo Rizal — Tue, 11 Aug 2026 04:09:01 +0000

Provisioning a new server is often more complicated than simply installing a few packages.

A fresh ubuntu server may start with almost nothing configured. An engineer still needs to create users, configure SSH access, set the hostname configure the firewall, and make sure the required services are running correctly.

When these steps are performed manually, the process can become time-consuming and inconsistent.

One server might have slightly different configurations from another. A firewall rule might be forgotten, SSH password authentication might remain enabled, or a monitoring agent might not be installed at all.

The problem is not only how to configure a server, but how to configure every server consistently.

This is where infrastructure automation become valuable.

In this article, we will build a zero-touch server provisioning workflow using ansible, starting from a fresh Ubuntu server and automatically transforming it into a standardized and secured server environment.

The provisioning process covers:

Bootstrapping and configuring the server
Applying system security hardening
Installing and configuring docker
Setting up system monitoring
Restricting network access

The goal is simple:

Start with a fresh ubuntu server, run the ansible playbook, and let ansible handle the rest

By the end of the process, the server will be ready to run applications, protected with a basic security baseline, equipped with Docker, and connected to the monitoring infrastructure.

Provisioning Flow

Before looking at the ansible configuration, it is important to understand what we are actually building.

The goal of this project is not to create a collection of independent Ansible playbooks. The goal is to create a repeatable provisioning workflow that can take a fresh Ubuntu server through several configuration stages until it reaches standardized state.

The provisioning flow looks like this:

Project Structure

Instead of putting the entire provisioning process into a single playbook, the project separates configuration into several files, each with a specific responsibility.

The easiest way to understand the architecture is to follow how ansible processes a command.

When we run:

ansible-playbook site.yml

Ansible first loads its configuration from ansible.cfg

[defaults]
inventory = inventory/hosts.ini
interpreter_python = auto_silent

This configuration tells Ansible where to find the inventory

Then inventory/hosts.ini tells ansible which servers it should manage and how to connect to them.

[servers]
vm1 ansible_host=192.168.122.68

[servers:vars]
ansible_user=root
# ansible_user=devops
ansible_ssh_private_key_file=~/.ssh/ansible

The servers group is important because the playbooks target this group:

hosts: servers

Ansible can therefore apply the same configuration to every server belonging to that group.

group_vars/servers.yml

This file contains variables that apply to the entire servers group:

server_timezone: Asia/Makassar
admin_user: devops
server_hostname: ansible-demo
server_ssh_public_key: YOUR_SSH_PUBLIC_KEY
monitoring_server_ip: 192.168.122.1
node_exporter_version: 1.9.1
node_exporter_arch: linux-amd64

This allows the playbooks to focus on what should be configured, while server-specific values remain outside the playbooks.

For example, instead of writing:

name: devops

inside a playbook, we use:

name: "{{ admin_user }}"

The Main Entry Point

The entire provisioning process is orchestrated through site.yml:

- name: bootstrap server
  import_playbook: playbooks/bootstrap.yml

- name: configure base system
  import_playbook: playbooks/base.yml

- name: harden server
  import_playbook: playbooks/security.yml

- name: install docker
  import_playbook: playbooks/docker.yml

- name: configure monitoring
  import_playbook: playbooks/monitoring.yml

This file does not contain the actual server configuration tasks. Instead, it defines the order in which the provisioning stages are executed.

1. Bootstrap Playbook

At the start, the server does not have the administrative user that we want to use for day to day management. The initial connection therefore uses the existing root account.

The purpose of the bootstrap stage is to establish the foundation required for the rest of the provisioning process.

The bootstrap playbook performs four main tasks:

Creates the administrative user
Grants the user sudo privileges
Configures SSH key-based authentication
Sets the server hostname

The important part of this stage is that root access is only required for the initial bootstrap.

Once the devops user has been created and configured, subsequent playbooks connect to the server using that dedicated administrative account.

The playbook looks like this:

- name: bootstrap fresh ubuntu server
  hosts: servers

  tasks:
    - name: create admin user
      ansible.builtin.user:
        name: "{{ admin_user }}"
        shell: /bin/bash
        create_home: true
        groups: sudo
        append: true

    - name: configure passwordless sudo
      ansible.builtin.copy:
        content: "{{ admin_user }} ALL=(ALL) NOPASSWD:ALL\n"
        dest: "/etc/sudoers.d/{{ admin_user }}"
        owner: root
        group: root
        mode: '0440'

    - name: configure ssh access for admin user
      ansible.posix.authorized_key:
        user: "{{ admin_user }}"
        key: "{{ server_ssh_public_key }}"

    - name: set server hostname
      ansible.builtin.hostname:
        name: "{{ server_hostname }}"

Creating the Administrative User

- name: create admin user
      ansible.builtin.user:
        name: "{{ admin_user }}"
        shell: /bin/bash
        create_home: true
        groups: sudo
        append: true

The username comes from the variable defined in group_vars/servers.yml:

admin_user: devops

This means the playbook does not depend on a specific username.

Ansible creates the user's home directory and adds the user to the sudo group:

groups: sudo
append: true

The append: true option is important because it adds the user to the specified group without removing the user from other existing groups.

At the end of this task, the server has a dedicated administrative identity that can be used for subsequent configuration.

Configuring Passwordless Sudo

- name: configure passwordless sudo
      ansible.builtin.copy:
        content: "{{ admin_user }} ALL=(ALL) NOPASSWD:ALL\n"
        dest: "/etc/sudoers.d/{{ admin_user }}"
        owner: root
        group: root
        mode: '0440'

Instead of modifying the main /etc/sudoers file, the configuration is placed inside:

/etc/sudoers.d/

This keeps the custom sudo configuration separated from the system's main sudo configuration.

The resulting file is:

/etc/sudoers.d/devops

with permissions:

0440

The user can now execute administrative commands through sudo without being prompted for a password.

This is useful for automated provisioning because subsequent ansible tasks need to perform privileged operations.

Configuring SSH Key Authentication

- name: configure ssh access for admin user
      ansible.posix.authorized_key:
        user: "{{ admin_user }}"
        key: "{{ server_ssh_public_key }}"

The public key is provided through:

server_ssh_public_key: YOUR_SSH_PUBLIC_KEY

Ansible manages the user's authorized_keys file instead of requiring us to manually create it.

Setting the Hostname

- name: set server hostname
  ansible.builtin.hostname:
    name: "{{ server_hostname }}"

A predictable hostname is useful for identifying the server through monitoring systems, logs, and other infrastructure tooling.

2. Base Playbook

After the bootstrap stage, the server now has a dedicated administrative user and the basic access configuration required for ansible to continue provisioning it.

The next step is to establish a consistent baseline for the operating system.

A fresh Ubuntu installation may contain different package versions, system settings, or filesystem directories depending on how the server was created. Before installing infrastructure components, it is useful to bring the system into a predictable state.

The base.yml playbook is responsible for:

Updating the APT package cache
Upgrading installed packages
Configuring the server timezone
Creating required application directories

The playbook looks like this:

- name: configure base ubuntu server
  hosts: servers
  become: true

  tasks:
    - name: update apt package cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600

    - name: upgrade installed packages
      ansible.builtin.apt:
        upgrade: dist

    - name: configure server timezone
      community.general.timezone:
        name: "{{ server_timezone }}"

    - name: create standard application directories
      ansible.builtin.file:
        path: "/home/{{ admin_user }}/{{ item }}"
        state: directory
        owner: "{{ admin_user }}"
        group: "{{ admin_user }}"
        mode: '0755'
      loop:
        - apps
        - monitoring

Updating the APT Package Cache

- name: update apt package cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600

The update_cache option performs the equivalent of:

apt update

The cache_valid_time prevents ansible from unnecessarily updating the package cache on every execution when the cache is still considered valid.

This is useful when the playbook contains multiple package related tasks because it reduces unnecessary repository requests.

Upgrading the System

- name: upgrade installed packages
      ansible.builtin.apt:
        upgrade: dist

This is equivalent to performing a distribution-level package upgrade through APT.

The purpose is to ensure that the server starts the rest of the provisioning process with its existing packages brought up to date.

This also establishes a more predictable baseline before we install additional components such as docker and monitoring agents.

Configuring the Timezone

- name: configure server timezone
      community.general.timezone:
        name: "{{ server_timezone }}"

Consistent timezone configuration is particularly useful for infrastructure because timestamps appear throughout:

System logs
Application logs
Monitoring data
Scheduled jobs
Troubleshooting sessions

A consistent timezone makes these timestamps easier to correlate when investigating an issue.

Creating the Application Directory

- name: create standard application directories
      ansible.builtin.file:
        path: "/home/{{ admin_user }}/{{ item }}"
        state: directory
        owner: "{{ admin_user }}"
        group: "{{ admin_user }}"
        mode: '0755'
      loop:
        - apps
        - monitoring

Rather than allowing application files to be placed arbitrarily throughout the filesystem, the provisioning process establishes a predictable location:

/home/devops/apps
/home/devops/monitoring

The directory is owned by the administrative user defined in our variables:

admin_user: devops

This gives us a standardized location that can later be used when deploying applications or additional infrastructure components.

3. Security Playbook

The server now has a predictable operating system baseline.

However, a freshly provisioned server should not be considered secure simply because its packages are up to date. We still need to reduce unnecessary access, protect the SSH service, control network traffic, and add a mechanism to respond to repeated authentication attempts.

This is the responsibility of the security.yml playbook.

The security playbook focuses on four areas:

Hardening SSH access
Configuring UFW as the host firewall
Allowing only the required network traffic
Installing and enabling Fail2ban

The playbook looks like this:

- name: harden ubuntu server
  hosts: servers
  become: true

  tasks:
    - name: disable root SSH login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PermitRootLogin'
        line: 'PermitRootLogin no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

    - name: disable SSH password authentication
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PasswordAuthentication'
        line: 'PasswordAuthentication no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

    - name: allow SSH through firewall
      community.general.ufw:
        rule: allow
        port: '22'
        proto: tcp

    - name: enable UFW
      community.general.ufw:
        state: enabled
        policy: deny
        direction: incoming

    - name: install Fail2ban
      ansible.builtin.apt:
        name: fail2ban
        state: present

    - name: enable Fail2ban
      ansible.builtin.systemd_service:
        name: fail2ban
        enabled: true
        state: started

  handlers:
    - name: restart ssh
      ansible.builtin.systemd_service:
        name: ssh
        state: restarted

Hardening SSH

SSH is one of the primary entry points into a server, so securing it is an important part of the baseline.

The first task disables direct root login:

- name: disable root SSH login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PermitRootLogin'
        line: 'PermitRootLogin no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

The resulting configuration is:

PermitRootLogin no

This means an external SSH connection can no longer authenticate directly as root.

This is particularly important because the bootstrap stage has already created the devops administrative account.

The intended access model is therefore:

rather than:

Disable SSH Password Authentication

- name: disable SSH password authentication
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PasswordAuthentication'
        line: 'PasswordAuthentication no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

After this configuration, SSH authentication relies on ssh key configured during the bootstrap stage.

This removes password based SSH authentication from the server's remote access path.

Validating SSH Configuration

Notice that both SSH configuration tasks include:

validate: '/usr/sbin/sshd -t'

This is an important detail.

Ansible does not immediately apply an invalid SSH configuration. The sshd -t command validates the configuration before the file is accepted.

This reduces the risk of provisioning an invalid SSH configuration that could prevent future connections.

The tasks also use a handler:

notify: restart ssh

The SSH service is therefore restarted only when the configuration actually changes.

This is one of the advantages of using Ansible modules instead of simply executing shell commands.

Installing Security Packages

The playbook installs UFW and Fail2ban:

UFW provides a simple interface for managing the linux host firewall, while Fail2ban can monitor authentication related logs and temporarily block clients that repeatedly fail authentication.

These tools address different parts of the security baseline:

Configuring UFW

The first firewall rule allows SSH:

- name: allow ssh
  community.general.ufw:
    rule: allow
    name: OpenSSH

This rule is intentionally configured before enabling UFW.

The firewall is then enabled with a default-deny policy for incoming traffic:

- name: enable ufw
  community.general.ufw:
    state: enabled
    policy: deny
    direction: incoming

The resulting security model is essentially:

This follows the principle of default deny: services should not become reachable simply because they happen to be listening on a network port.

Additional ports can then be explicitly allowed when they are required by later components of the infrastructure.

For example, the monitoring stage will later add a specific rule for Node Exporter rather than exposing all ports on the server.

Enabling Fail2ban

The final security task ensures that Fail2ban is enabled and running:

- name: enable fail2ban
  ansible.builtin.systemd_service:
    name: fail2ban
    enabled: true
    state: started

There are two important states being configured here:

enabled: true

ensures that Fail2ban starts automatically when the server boots.

state: started

ensures that the service is running immediately after provisioning.

This means the server does not need to be manually configured after deployment to activate the security service.

4. Docker Playbook

With the operating system configured and the initial security baseline in place, the next step is to prepare the server to run containerized workloads.

For this project, docker is installed through the official docker APT repository rather than relying on the docker package provided by the default ubuntu repositories.

The docker.yml playbook is responsible for:

Installing the required repository prerequisites
Configuring Docker's official APT repository
Installing Docker Engine and related components
Enabling and starting the Docker service
Allowing the administrative user to manage Docker

The playbook looks like this:

- name: install docker on ubuntu server
  hosts: servers
  become: true

  tasks:
    - name: install docker repository prerequisites
      ansible.builtin.apt:
        name:
          - ca-certificates
          - curl
        state: present
        update_cache: true

    - name: create docker keyring directory
      ansible.builtin.file:
        path: /etc/apt/keyrings
        state: directory
        mode: '0755'

    - name: download docker GPG key
      ansible.builtin.get_url:
        url: https://download.docker.com/linux/ubuntu/gpg
        dest: /etc/apt/keyrings/docker.asc
        mode: '0644'

    - name: add docker APT repository
      ansible.builtin.deb822_repository:
        name: docker
        types: deb
        uris:
          - https://download.docker.com/linux/ubuntu
        suites:
          - "{{ ansible_facts['distribution_release'] }}"
        components:
          - stable
        architectures:
          - amd64
        signed_by: /etc/apt/keyrings/docker.asc
        state: present

    - name: install docker engine and compose plugin
      ansible.builtin.apt:
        name:
          - docker-ce
          - docker-ce-cli
          - containerd.io
          - docker-buildx-plugin
          - docker-compose-plugin
        state: present
        update_cache: true

    - name: ensure docker service is enabled and running
      ansible.builtin.systemd_service:
        name: docker
        enabled: true
        state: started

    - name: add admin user to docker group
      ansible.builtin.user:
        name: "{{ admin_user }}"
        groups: docker
        append: true

Installing Repository Prerequisites

- name: install docker repository prerequisites
      ansible.builtin.apt:
        name:
          - ca-certificates
          - curl
        state: present
        update_cache: true

ca-certificates allows the system to properly validate HTTPS certificates, while curl is used to retrieve the Docker repository signing key.

These packages provide the basic requirements for securely adding the external repository.

Creating the APT Keyring Directory

/etc/apt/keyrings

using:

- name: create docker keyring directory
      ansible.builtin.file:
        path: /etc/apt/keyrings
        state: directory
        mode: '0755'

APT repository signing keys are stored separately from the repository configuration.

This provides a clear location for repository specific signing keys instead of placing them in a global trusted key configuration.

Adding Docker's Repository Signing Key

- name: download docker GPG key
      ansible.builtin.get_url:
        url: https://download.docker.com/linux/ubuntu/gpg
        dest: /etc/apt/keyrings/docker.asc
        mode: '0644'

The signing key is then referenced by the docker repository configuration.

This allows APT to verify that packages retrieved from the docker repository are signed by the expected key.

The important part here is that the playbook does not simply add an arbitrary repository and trust it globally. The repository is explicitly associated with its signing key.

Configuring the Docker APT Repository

- name: add docker APT repository
      ansible.builtin.deb822_repository:
        name: docker
        types: deb
        uris:
          - https://download.docker.com/linux/ubuntu
        suites:
          - "{{ ansible_facts['distribution_release'] }}"
        components:
          - stable
        architectures:
          - amd64
        signed_by: /etc/apt/keyrings/docker.asc
        state: present

One useful detail here is:

suites:
  - "{{ ansible_facts['distribution_release'] }}"

Instead of hardcoding an ubuntu release name, ansible obtains the distribution release from the target system.

This makes the playbook less dependent on a specific ubuntu release.

The repository is also restricted to the stable component and amd64 architecture:

components:
  - stable

architectures:
  - amd64

Installing Docker

- name: install docker engine and compose plugin
  ansible.builtin.apt:
    name:
      - docker-ce
      - docker-ce-cli
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin
    state: present
    update_cache: true

Rather than installing only the docker engine package, the playbook installs the components needed for a practical container environment.

The packages include:

docker-ce — docker engine
docker-ce-cli — docker command line interface
containerd.io — container runtime
docker-buildx-plugin — docker image build functionality
docker-compose-plugin — docker compose

This gives the server the tooling required to build, run, and manage containerized applications.

Ensuring Docker Is Running

- name: ensure docker service is enabled and running
  ansible.builtin.systemd_service:
    name: docker
    enabled: true
    state: started

There are two desired states here.

enabled: true

ensures docker starts automatically when the server boots.

state: started

ensures docker is running immediately after the provisioning stage.

This means that once provisioning finishes, the server is already capable of running containers without requiring manual intervention.

Allowing the Administrative User to Manage Docker

The final task adds the administrative user to the docker group:

- name: add admin user to docker group
      ansible.builtin.user:
        name: "{{ admin_user }}"
        groups: docker
        append: true

This allows the administrative user to interact with the docker daemon without prefixing every docker command with sudo.

For example, the user can run:

docker ps

instead of:

sudo docker ps

5. Monitoring Playbook

The server is now configured, hardened, and ready to run containerized workloads.

There is one final piece missing: observability.

A server can be perfectly configured and still become difficult to operate if we cannot see its resource usage and system health.

For this project, Prometheus Node Exporter is used to expose hardware and operating system metrics that can be collected by Prometheus.

The monitoring.yml playbook is responsible for:

Creating a dedicated system user for Node Exporter
Downloading and installing Node Exporter
Creating a systemd service
Ensuring Node Exporter starts automatically
Restricting access to the metrics endpoint

The playbook looks like this:

- name: configure monitoring agent
  hosts: servers
  become: true

  tasks:
    - name: create node exporter user
      ansible.builtin.user:
        name: node_exporter
        system: true
        shell: /usr/sbin/nologin
        create_home: false

    - name: download node exporter
      ansible.builtin.get_url:
        url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}.tar.gz"
        dest: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        mode: '0644'

    - name: extract node exporter
      ansible.builtin.unarchive:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        dest: /tmp
        remote_src: true
        creates: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}"

    - name: install node exporter binary
      ansible.builtin.copy:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}/node_exporter"
        dest: /usr/local/bin/node_exporter
        owner: root
        group: root
        mode: '0755'
        remote_src: true

    - name: create node exporter systemd service
      ansible.builtin.copy:
        dest: /etc/systemd/system/node_exporter.service
        owner: root
        group: root
        mode: '0644'
        content: |
          [Unit]
          Description=Prometheus Node Exporter
          Wants=network-online.target
          After=network-online.target

          [Service]
          User=node_exporter
          Group=node_exporter
          Type=simple
          ExecStart=/usr/local/bin/node_exporter

          [Install]
          WantedBy=multi-user.target
      notify: restart node exporter

    - name: enable and start node exporter
      ansible.builtin.systemd_service:
        name: node_exporter
        enabled: true
        state: started
        daemon_reload: true

    - name: allow node exporter from monitoring server
      community.general.ufw:
        rule: allow
        from_ip: "{{ monitoring_server_ip }}"
        to_port: '9100'
        proto: tcp

  handlers:
    - name: restart node exporter
      ansible.builtin.systemd_service:
        name: node_exporter
        state: restarted

Creating a Dedicated System User

The first step is to create a dedicated system user:

- name: create node exporter user
      ansible.builtin.user:
        name: node_exporter
        system: true
        shell: /usr/sbin/nologin
        create_home: false

Node Exporter does not need an interactive shell or a home directory.

Therefore, instead of running the service as root, we create a dedicated user with:

shell: /usr/sbin/nologin

This follows the principle of least privilege.

The service only needs to perform its intended function, so there is no reason for it to have an interactive login account.

Downloading Node Exporter

The Node Exporter version is controlled through group_vars/servers.yml:

node_exporter_version: 1.9.1
node_exporter_arch: linux-amd64

The playbook uses these variables to construct the download url:

- name: download node exporter
      ansible.builtin.get_url:
        url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}.tar.gz"
        dest: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        mode: '0644'

If we need to upgrade Node Exporter later, the version can be changed in the variables file instead of modifying the playbook itself.

For example:

node_exporter_version: 1.10.2

Extracting the Binary

The downloaded archive is extracted using Ansible's unarchive module:

- name: extract node exporter
      ansible.builtin.unarchive:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        dest: /tmp
        remote_src: true
        creates: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}"

The important part here is:

remote_src: true

The archive already exists on the target server, so ansible does not need to transfer it from the machine running Ansible.

The creates parameter also provides an idempotency check.

If the extracted directory already exists, Ansible does not need to extract the archive again.

Installing the Node Exporter Binary

The Node Exporter binary is then copied into:

/usr/local/bin/node_exporter

using:

- name: install node exporter binary
      ansible.builtin.copy:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}/node_exporter"
        dest: /usr/local/bin/node_exporter
        owner: root
        group: root
        mode: '0755'
        remote_src: true

The binary is owned by root, while the service itself will run as the dedicated node_exporter user.

This creates a separation between the executable and the account that executes it.

Creating the systemd Service

Installing the binary alone does not make Node Exporter a managed system service.

The playbook therefore creates:

/etc/systemd/system/node_exporter.service

with:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Enabling and Starting Node Exporter

The service is then enabled and started:

- name: enable and start node exporter
  ansible.builtin.systemd_service:
    name: node_exporter
    enabled: true
    state: started
    daemon_reload: true

The daemon_reload option tells systemd to reload its service definitions after Ansible creates the new unit file.

The two important states are:

enabled: true
state: started

The first ensures Node Exporter starts automatically after a reboot.

The second ensures it is already running when provisioning finishes.

At this point, Node Exporter exposes its metrics endpoint on the standard port:

Restricting Monitoring Access

The metrics endpoint contains information about the server's operating system and resource usage, so there is no reason to expose it to every host that can reach the server.

The playbook therefore adds a specific UFW rule:

- name: allow node exporter from monitoring server
      community.general.ufw:
        rule: allow
        from_ip: "{{ monitoring_server_ip }}"
        to_port: '9100'
        proto: tcp

The monitoring server address is defined in:

monitoring_server_ip: 192.168.122.1

This results in a much narrower network rule:

This is an important continuation of the security model introduced in the Security Playbook.

We do not simply open port 9100 to the entire network:

9100 → 0.0.0.0/0

Instead, access is explicitly limited to the monitoring server.

Conclusion

Provisioning a server manually may seem simple when there is only one server to configure. But as the number of servers grows, manual configuration quickly becomes difficult to maintain and can lead to configuration drift.

In this project, we have built an ansible workflow that:

Bootstrapping and configuring the server
Applying system security hardening
Installing and configuring Docker
Setting up system monitoring
Restricting network access

Instead of relying on a sequence of manual commands, the desired server state is now defined as code that can be reviewed, version controlled, reproduced, and applied consistently.

Define the desired state. Automate it. Repeat it

You can find the source code for this article in my github repository: https://github.com/muhammadyulasfipahrizal/zero-touch-ansible.git

7 Docker Compose Patterns Every Devops Engineer Should Know

Mo Rizal — Wed, 05 Aug 2026 08:21:18 +0000

Docker compose has become one of the most widely used tools for running multi container applications, with a simple YAML file, engineers can spin up multiple containers at once

However, defining the proper configuration in your compose file is critical to ensuring seamless communication between containers, data persistence, and efficient system operation

This is where docker compose patterns become valuable

In this article, we will explore 7 docker compose patterns that you should know

1. Multi Environment Pattern

In a real world project, development and production environment usually have different requirements

In development, engineers need fast iteration, the ability to rebuild images, and exposed ports for easier access. While production requires stable image, automatic restart and minimal configuration changes.

The Multi Environment Pattern solves this problem by separating common service definitions from environment specific configurations

Instead of maintaining multiple completely different compose files, we create a base docker-compose.yml and extend it with environment specific overrides.

docker-compose.yml → base
docker-compose.dev.yml → development
docker-compose.prod.yml → production

Base Setup

services:
  backend:
    working_dir: /app
    environment:
      NODE_ENV: ${NODE_ENV}
    depends_on:
      - postgres
      - redis

  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}

  redis:
    image: redis:8-alpine

This file defines the application architecture without including any environment specific configuration

Development Setup

services:
  backend:
    build: .
    command: npm run dev
    volumes:
      - ./backend:/app
    ports:
      - "3000:3000"

  postgres:
    ports:
      - "5432:5432"

  redis:
    ports:
      - "6379:6379"

In this file we add source code mounting and accessible ports

Production Setup

services:
  backend:
    image: ghcr.io/multienv/backend:latest
    command: npm start
    restart: unless-stopped

  postgres:
    restart: unless-stopped

  redis:
    restart: unless-stopped

Production should use pre-built images and automatic restart

2. Health Check & Dependency Pattern

In a multi container applications, service startup order does not ensure service readiness

For example, if you have appication with backend container and database container, docker may start the backend immediately after starting the database. However, the database might still be initializing and unable to accept connections thus causing startup failure

Health Check & Dependency Pattern solves this problem by allowing containers to verify service availability before starting dependent services

Example

services:
  backend:
    image: node:22-alpine
    working_dir: /app
    volumes:
      - ./backend:/app
    command: node index.js
    depends_on:
      postgres:
        condition: service_healthy

  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: app
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "pg_isready -U postgres -d app"
        ]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 20s

The postgres container includes a health check using pg_isready:
pg_isready -U postgres -d app

The backend service uses:

depends_on:
      postgres:
        condition: service_healthy

This tell docker to start the backend only after PostgreSQL reports a healthy status. With this setup we ensure that the right container start at the right time

3. Network Isolation Pattern

For security reasons, not every container should be able to communicate with each other

For example, a database should not be directly accessible from the internet. Only the appropriate services should be allowed to communicate with it

The Network Isolation Pattern improves security by separating containers into different docker networks and controlling container communication

Example

services:
  frontend:
    image: nginx:alpine
    ports:
      - "8080:80"
    networks:
      - frontend-network

  backend:
    image: node:22-alpine
    networks:
      - frontend-network
      - backend-network

  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    networks:
      - backend-network

networks:
  frontend-network:
  backend-network:

This setup creates two isolated networks:

frontend-network
backend-network

The frontend container can communicate with the backend because both services share the same network:

frontend:
    networks:
      - frontend-network
backend:
    networks:
      - frontend-network

And the backend container can communicate with the database because both share the same network

backend:
    networks:
      - backend-network

  postgres:
    networks:
      - backend-network

However, the frontend can't be connected to the database because the database container does not include the frontend-network

4. Persistent Volume Pattern

By default containers are not persistent, this means that when a container is removed, all data inside the container filesystem is deleted

This becomes a problem for stateful services such as databases, where data must survive container restarts, updates, or redeployments

The Persistent Volume Pattern solves this problem by storing important data outside the container lifecycle using Docker volumes

Example

services:
  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - postgres-data:/var/lib/postgresql/data

volumes:
  postgres-data:

The PostgreSQL container stores its database files in:
/var/lib/postgresql/data

Instead of storing this data inside the container, in this setup docker will mount the data in

volumes:
  - postgres-data:/var/lib/postgresql/data

If the container is recreated:
docker compose up -d

Docker attaches the existing volume and postgres can continue using the existing data

5. Resource Limit Pattern

Without proper resource limitation, a single container might consume excessive cpu or memory that can affect other services running on the host

The Resource Limit Pattern prevents this by defining proper limits for containers.

Example

services:
  backend:
    image: node:22-alpine
    working_dir: /app
    volumes:
      - ./backend:/app
    command: node index.js
    mem_limit: 512m
    mem_reservation: 256m
    cpus: 1

In this example we define several config:

mem_limit: 512m - Defines the maximum memory a container can use
mem_reservation: 256m - Defines the minimum amount of memory that docker should reserve for the container
cpus: 1 - Limit the container to use one cpu core

6. Reverse Proxy Gateway Pattern

In production environment, applications are rarely exposed directly to the internet. Instead, traffic is handled by a reverse proxy that acts as a gateway between external users and internal services

The Reverse Proxy Pattern places a proxy server in front of application containers to handle incoming requests, routing, and load balancing

Example

services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - backend

  backend:
    image: node:22-alpine
    working_dir: /app
    volumes:
      - ./backend:/app
    command: node index.js

Here we use nginx as our reverse proxy, the nginx container is the only service exposed to the host:

ports:
  - "80:80"

The backend service does not expose any ports because it only needs to communicate internally with nginx.

nginx.conf

events {}

http {
    server {
        listen 80;

        location / {
            proxy_pass http://backend:3000;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

When a user accesses port 80, the request first reaches nginx which forwards the request internally to:
http://backend:3000

7. Secrets Management Pattern

Applications often require sensitive information such as database passwords, API keys, and authentication tokens

A common mistake is storing these credentials directly inside docker-compose.yml:

environment:
  POSTGRES_PASSWORD: password

This creates security risks because secrets can accidentally be exposed through source code repositories, logs, or shared configuration files

The Secrets Management Pattern separates sensitive data from application configuration by using Docker secrets

Example

services:
  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: app
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
    secrets:
      - postgres_password

secrets:
  postgres_password:
    file: ./postgres_password.txt

Instead of passing the password directly:
POSTGRES_PASSWORD: my-secure-password

Docker mounts the secret as a file inside the container:
/run/secrets/postgres_password

The PostgreSQL image automatically reads the password from this file:
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password

Docker secrets are useful for small deployments and single-host environments. However, for production systems it is recommended to use a dedicated secret management platform such as hashicorp vault, openbao, or Infisical

Conclusion

Docker compose patterns help devops engineers build container environments that are more secure, reliable, and easier to maintain

By applying these 7 patterns, you can create a more production ready setup

You can find the source code for this article in my github repository:
https://github.com/muhammadyulasfipahrizal/compose-patterns