DEV Community: Mo Rizal

From Static Passwords to Dynamic Database Credentials with HashiCorp Vault

Mo Rizal — Sat, 19 Sep 2026 04:24:44 +0000

Almost every production app need credentials to access their database.

A common approach is to create a database user with a password and provide that credential to the application through environment variable, configuration file, or secret management system.

The application then uses the same credential until it is manually changed.

This approach is simple, but the credential itself becomes a long lived security risk until someone rotates it.

Though credential rotation can also become a new problem.

A typical workflow might look like this:

The problem is not only how the credential is stored.

We also need to consider its entire lifecycle.

Who creates it?

How long should it remain valid?

How is it renewed while the application is running?

How do we replace it without interrupting the application?

What happens to the old credential after the replacement is ready?

We wanted to approach this differently.

Instead of giving the application a permanent database password, we wanted the database credential to be temporary by design.

This is where dynamic credentials become useful.

With HashiCorp Vault, an application can request a database credential when it needs one. Vault generates a temporary database user and returns the credential together with a lease.

The application can then use the credential while it is valid and renew its lease while the credential is still in use.

When the application needs to replace the credential, it can request another one instead of modifying a permanent password.

The workflow becomes:

This changes credential management from a manually maintained configuration value into a lifecycle that can be managed by the application.

Project Goals

We wanted an application whose database credentials are temporary by design, automatically renewed while in use, and replaced periodically without storing a static password.

The application should be able to:

Authenticate to Vault without using a Vault root token.
Obtain database credentials dynamically.
Connect to Database using those credentials.
Renew the credential lease while the application is running.
After three successful renewals, request a new credential.
Verify the new credential against Database.
Switch to the new credential.
Revoke the old lease.
Continue serving requests without restarting the application.

The three renewal threshold is intentionally used as a simple rotation trigger for this project. It gives us a deterministic lifecycle that can be observed and tested without introducing a more complex rotation policy.

The complete implementation will show how these pieces work together, from the initial authentication and credential request to lease renewal and the final zero downtime credential switch.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the full source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/vault-setup.git

Architecture

The project consists of three main components, each component has a specific responsibility in the credential lifecycle.

Vault

HashiCorp Vault is responsible for managing the database credentials used by the application.

Vault generates temporary credentials and manages the lease associated with each credential. The application communicates with Vault to authenticate, obtain credentials, and maintain the credential lease while it is running.

Application

The application is a small Go API that consumes credentials provided by Vault.

It authenticates to Vault, requests a dynamic PostgreSQL credential, and uses that credential to establish a connection to the database.

The application also manages the runtime credential lifecycle. While the credential is being used, it renews the lease. When the rotation condition is reached, the application requests a new credential and verifies the new connection before switching away from the existing one.

This allows the application to replace its database credential while continuing to run.

Database

PostgreSQL is the database accessed by the application.

The database is configured so that Vault can create temporary users for the application. These users are created with the permissions required by the application and are removed when their associated credentials are revoked.

The application does not communicate with PostgreSQL through Vault. Instead, Vault provides the application with the credentials, and the application connects directly to PostgreSQL.

Component Interaction

The interaction between the three components can be summarized as:

This architecture keeps credential management separate from database access. Vault provides the temporary identity needed to access the database, while the application remains responsible for using that identity throughout its lifecycle.

Project Structure

Each part of the repository has a specific responsibility.

app/ contains the Go application, including its source code, Go dependencies, Dockerfile, and Docker Compose configuration.
config/ contains the Vault server configuration.
db/ contains the PostgreSQL Docker Compose configuration and the SQL script used to initialize the database.
docker-compose.yml at the repository root defines the Vault service and connects it to the shared Docker network.

Implementation

PostgreSQL

The PostgreSQL service is defined in db/docker-compose.yml.

The configuration uses PostgreSQL 17 and persists its data through a Docker volume. The initialization directory is mounted into PostgreSQL's /docker-entrypoint-initdb.d directory so the database can be prepared when the container is initialized.

The service is also attached to the external Docker network used by the other components, allowing Vault and the application to communicate with PostgreSQL through its container name.

services:
  postgres:
    image: postgres:17
    container_name: db-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: ${YOUR_DB_NAME}
      POSTGRES_USER: ${YOUR_DB_USER}
      POSTGRES_PASSWORD: ${YOUR_DB_PASSWORD}
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./postgres/init:/docker-entrypoint-initdb.d:ro
    networks:
      - ${YOUR_DOCKER_NETWORK}

Database Initialization

The initialization script is located at:

db/postgres/init/01-init.sql

Its purpose is to create the administrative database user that Vault will use when managing dynamic PostgreSQL users.

The script grants this user permission to connect to the target database and create objects in the public schema.

The repository contains the following initialization:

Vault

Vault is defined in the root docker-compose.yml.

The service uses the official hashicorp/vault image, exposes ports 8200 and 8201, mounts the Vault data and configuration directories, and starts Vault using the repository's HCL configuration file.

services:
  vault:
    image: hashicorp/vault:latest
    container_name: vault
    restart: unless-stopped
    ports:
      - "8200:8200"
      - "8201:8201"
    volumes:
      - ./data:/vault/data
      - ./config:/vault/config:ro
    cap_add:
      - IPC_LOCK
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - ${YOUR_DOCKER_NETWORK}

Vault Configuration

The Vault server configuration is stored under config/.

ui = true

disable_mlock = true

api_addr     = "https://0.0.0.0:8200"
cluster_addr = "https://0.0.0.0:8201"

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable     = true
}

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-1"
}

The configuration enables the Vault UI and uses Raft storage for Vault's persistent data. The TCP listener accepts connections on ports 8200 and 8201.

Enable Database Secrets Engine

The Database Secrets Engine allows Vault to generate database credentials.

For this project, the database plugin is:

postgresql-database-plugin

The database connection is configured with the name:

vault-postgres

Vault connects to PostgreSQL through the Docker network using:

postgresql://{{username}}:{{password}}@vault-postgres:5432/appdb?sslmode=disable

The {{username}} and {{password}} values are provided by Vault's database plugin when it establishes the administrative connection.

Dynamic Database Role

When the application requests database/creds/app-role, Vault generates a new role and password.

The role is configured as a dynamic role with:

TTL:     1 hour
Max TTL: 1 day

When the application requests:

database/creds/app-role

Vault generates a new database username and password.

The creation statements are:

CREATE ROLE "{{name}}"
WITH LOGIN
PASSWORD '{{password}}'
VALID UNTIL '{{expiration}}';

GRANT CONNECT ON DATABASE YOUR_DATABASE_NAME
TO "{{name}}";

Vault replaces the template variables with generated values before executing the statements against PostgreSQL.

The role also defines a revocation statement:

DROP ROLE IF EXISTS "{{name}}";

This is what allows Vault to clean up the generated PostgreSQL user when the lease is revoked.

Policy

The application also needs permission to request them from Vault.

We can create ACL policy named app-policy:

This policy deliberately exposes only the credential endpoint required by the application.

AppRole Authentication

The application should not authenticate to Vault using the root token.

Instead, we enable the AppRole authentication method.

The configured authentication method is:

The application receives two values:

Role ID
Secret ID

These values are provided to the application through its runtime environment rather than embedding them directly. The application can therefore authenticate as a service identity without receiving the Vault root token.

App

The application is defined in app/docker-compose.yml.

The Compose configuration provides the Vault address, AppRole credentials, and PostgreSQL connection information through environment variables.

The application communicates with Vault using:

http://vault:8200

and database with:

db-postgres:5432

Both services are connected through the same external Docker network.

This keeps the application configuration independent from hard coded container IP addresses.

Application Lifecycle

Authenticate

When the application starts, it creates a Vault client and authenticates using AppRole.

The application sends its Role ID and Secret ID to:

auth/approle/login

Vault validates the credentials and returns a Vault token.

That token is then used for subsequent requests to the Vault API.

Request Dynamic Credentials

After authentication, the application requests:

database/creds/app-role

Vault creates a new Database role and returns:

username
password
lease ID
lease duration

The application stores these values in its runtime state.

Connect to Database

The generated username and password are then used to create a database connection.

The application verifies the credential by establishing the connection and performing a database ping.

Lease Renewal

The application obtains the lease duration directly from Vault when it requests the credentials.

Instead of using a fixed renewal interval, the renewal loop waits for approximately half of the current lease duration before attempting a renewal.

When the renewal succeeds, the application updates its lease information and increments its renewal counter.

Rotation

For this project, rotation occurs after three successful lease renewals.

The counter in this project is intentionally simple so the complete lifecycle can be observed during testing.

After the third successful renewal:

The application requests another credential from:

database/creds/app-role

Vault creates a second database user with a different username, password, and lease.

At this point, two credentials can temporarily exist:

The application does not immediately revoke the old credential.

First, it verifies that the new credential actually works.

Zero Downtime Credential Switch

The application creates a new PostgreSQL connection using the newly generated credential.

If the connection succeeds, the application switches its active database connection to the new one.

Only after the new connection is verified does the application revoke the old lease.

The old credential remains available until the replacement has been successfully verified. If the new credential cannot connect to database, the application keeps using the existing connection instead of disrupting the running application.

Observing the Rotation and Database Profile

The application exposes its current lifecycle state through an API handler, allowing us to observe information such as the current credential identity, lease information, expiration, and renewal count.

We can then compare that application state with the state visible from the Vault CLI.

The observation focuses on two events:

Lease renewal.
Credential rotation.

Lease Renewal

The application state should show that the lease is still associated with the same dynamic credential while the renewal count increases.

This demonstrates the distinction between renewal and rotation: the lease is extended, but the database credential itself has not yet been replaced.

Credential Rotation

After the third successful renewal, the application requests a new credential.

The final state should show that the application has successfully moved to the new credential while continuing to run.

Conclusion

Managing database credentials is not only about keeping passwords secret, but also about managing their entire lifecycle.

In this project, Vault provides temporary Database credentials through AppRole authentication, while the application handles lease renewal and credential rotation automatically.

After three renewals, the application requests a new credential, verifies the connection, switches to it, and revokes the old lease. This allows credentials to be rotated without restarting the application or interrupting database access.

The three renewal policy is only a demonstration. In production, the same pattern can be adapted to rotation intervals and security requirements.

The idea is to treat database credentials as temporary runtime rather than static configuration values.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the full source code in the repository below.

https://github.com/muhammadyulasfipahrizal/vault-setup.git

Building Highly Available PostgreSQL Cluster With Patroni, ETCD, HAProxy, And Keepalived

Mo Rizal — Sat, 12 Sep 2026 06:29:06 +0000

A database is one of the most critical components in an application.

When the database is running normally, the application can connect to it and continue serving requests. But when the database server becomes unavailable, the impact can quickly spread to the services that depend on it.

For a system that needs to remain available, relying on a single database server creates an obvious risk.

If that server fails, the application may lose its database connection until the server is recovered or another database server is manually brought into service.

This raises an important question:

What happens when the database server that our application depends on suddenly fails?

A common approach is to run multiple PostgreSQL servers. However, having multiple servers does not automatically make the database highly available.

A highly available database needs to continue serving applications even when part of the infrastructure fails. This means the system needs to handle failures without requiring the application to be manually reconfigured every time something goes wrong.

In this article, we will build a highly available database cluster using PostgreSQL, Patroni, etcd, HAProxy, and Keepalived.

What We Need

Before building the system, we first need to define what we expect from a highly available database.

The goal is not simply to have multiple PostgreSQL servers running at the same time. The system should remain available and usable even when individual components fail.

For this project, we define four main requirements:

Multiple PostgreSQL Nodes

We need multiple PostgreSQL nodes so that the database service has another node available when one of the servers becomes unavailable.

The nodes should maintain the same database state closely enough that one of them can take over when necessary.

This gives us redundancy at the database layer.

Automatic Failover

When the current primary PostgreSQL nodes becomes unavailable, the cluster should be able to detect the failure and recover by promoting an eligible node without requiring to manually reconfigure the entire database cluster.

Stable Connection Endpoint

We do not want application configuration to contain the address of a specific PostgreSQL server and require that address to be changed after every failover.

Instead, the application should connect to a stable endpoint while the infrastructure handles which PostgreSQL node currently receives the connection.

No Single Point of Failure

If database traffic passes through another component, that component can become a new single point of failure.

For example, having three PostgreSQL nodes does not provide end to end availability if all application traffic depends on one proxy server.

The connection layer therefore also needs redundancy so that the failure of a single proxy does not make the database unreachable.

Why Replication Is Not Enough

PostgreSQL provides replication mechanisms that allow data to be replicated from one instance to another.

This gives us an important foundation for high availability.

But having a replicated database does not automatically mean that the application can continue operating.

Consider a simple setup:

Under normal conditions, the application connects to the primary while the replica receives replicated data.

Now imagine that the primary suddenly fails:

The replica may still contain the database data, but it does not automatically solve the entire failure.

We still need a mechanism to determine that the primary is no longer available and decide what should happen next.

A Replica Is Not Automatically the New Primary

Replication gives us another copy of the data, but the cluster still needs to coordinate the transition from the failed primary to another node.

Without an automated mechanism, we need to:

Detect the failure
Determine whether the primary is really unavailable
Promote a suitable replica
Reconfigure clients to use the new primary
Re establish replication for the remaining nodes

The data may still be available, but the database service itself is not automatically recovered.

The Connection Problem

There is another problem after a failover.

Suppose the application connects directly to the primary's address:

After the primary fails and another node becomes primary:

The new primary may be ready to accept writes activity, but the application is still trying to connect to the old address.

This means that database failover and application connectivity are two different problems.

We need the database cluster to recover and provide a consistent way for clients to reach whichever node is currently serving as primary.

From Replication to High Availability

This shows why a highly available PostgreSQL system requires more than PostgreSQL replication alone.

We need additional components to coordinate the database nodes, manage the transition when a primary fails, and provide a reliable connection path for applications.

The architecture therefore needs to solve three different concerns:

Instead of treating the cluster as a collection of PostgreSQL servers, we separate it into two main layers:

Database layer, responsible for storing data and managing PostgreSQL availability
Connection layer, responsible for providing a stable path from applications to the active PostgreSQL node

The architecture consists of five components:

PostgreSQL
Patroni
etcd
HAProxy
Keepalived

Each component has a specific responsibility.

PostgreSQL

PostgreSQL is the database engine and the component that stores the application data.

We will run PostgreSQL on three separate virtual machines:

The three PostgreSQL instances form the database cluster.

One node acts as the current primary, while the other nodes act as replicas.

The primary handles write operations, while the replicas maintain copies of the database that can be used during failover.

Patroni

Patroni runs alongside PostgreSQL on each database node and manages the PostgreSQL instances as a high availability cluster.

Its responsibilities include:

Monitoring PostgreSQL health
Managing the PostgreSQL cluster state
Coordinating leader election
Promoting a replica when the primary fails
Managing PostgreSQL configuration required for the HA setup

etcd

etcd act as distributed system where cluster state can be stored and coordinated between the nodes.

Each database VM runs an etcd member:

The etcd members form a distributed consensus cluster.

Patroni uses etcd to store and coordinate information about the PostgreSQL cluster, including which node currently holds the leader role.

HAProxy

HAProxy monitors the PostgreSQL nodes and routes client connections to the node that is currently eligible to receive database traffic.

This means the application does not need to know which PostgreSQL VM is currently the primary.

In this project we will deploy two HAProxy instances:

Keepalived

Using two HAProxy instances introduces another problem.

If the application connects directly to one HAProxy server and that server fails, the PostgreSQL cluster may still be healthy but the application can no longer reach it.

We therefore need a way to make the HAProxy layer highly available as well.

Keepalived provides this through a Virtual IP (VIP).

Under normal conditions, one HAProxy node owns the VIP and handles client connections.

If that node fails, Keepalived can move the VIP to the other HAProxy node.

The application therefore continues using the same database endpoint.

Putting the Architecture Together

With each component assigned to a specific responsibility, the complete architecture for this project looks like this:

Each layer addresses a different failure scenario.

If a PostgreSQL primary fails, Patroni manages the database failover.
If the active HAProxy node fails, Keepalived moves the VIP to the other proxy.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

Github Repository: https://github.com/muhammadyulasfipahrizal/postgres-ha.git

Building the PostgreSQL HA Cluster

The database layer consists of three VM

Each VM has its own PostgreSQL instance, its own Patroni process, and one member of the etcd cluster. The three nodes then work together to provide a PostgreSQL cluster with automatic failover.

PostgreSQL Nodes

At any given time, one node is the primary and the other nodes are replicas.

The primary handles write operations, while the replicas continuously receive changes from the primary.

For example, the initial state might look like:

VM1 -> Primary
VM2 -> Replica
VM3 -> Replica

After a failure, the cluster may transition to:

VM1 -> Failed
VM2 -> Primary
VM3 -> Replica

The node roles are therefore part of the cluster's dynamic state rather than something we manually configure once and leave unchanged.

PostgreSQL Replication

The PostgreSQL instances use streaming replication to maintain copies of the database across the nodes.

This gives the cluster multiple copies of the database while allowing the primary to continue serving write traffic.

The replication settings are defined through Patroni's configuration. For example:

postgresql:
  listen: 0.0.0.0:5432
  connect_address: ${SERVER_IP}:5432

  parameters:
    hba_file: /etc/patroni/pg_hba.conf
    unix_socket_directories: '/var/run/postgresql'

The cluster is also configured with parameters required for streaming replication:

postgresql:
  parameters:
    wal_level: replica
    hot_standby: "on"
    wal_keep_size: 256MB
    max_wal_senders: 10
    max_replication_slots: 10

These settings are part of the Patroni configuration rather than being managed independently on each PostgreSQL instance.

etcd Cluster

Alongside PostgreSQL, each database VM runs an etcd member.

The three etcd members form a distributed consensus cluster.

Using three members is important because the cluster can tolerate the loss of one member while still maintaining a quorum.

Patroni uses this cluster to coordinate the state of the PostgreSQL nodes.

The etcd endpoints are configured in patroni.yml:

etcd3:
  hosts:
    - 192.168.122.1:2379
    - 192.168.122.2:2379
    - 192.168.122.3:2379

Patroni also defines the cluster scope and namespace used to store its distributed state:

scope: postgres-ha
namespace: /service/
name: postgres1

The name value is node specific, while the cluster scope and namespace identify the Patroni cluster. On the other nodes, the node name and server address change accordingly.

Patroni on Each Node

Each PostgreSQL node also runs a Patroni instance. Patroni connects the PostgreSQL instances with the distributed cluster state provided by etcd.

Each Patroni instance monitors its local PostgreSQL server and participates in managing the cluster.

An important part of the configuration is the timing used for leader management:

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10

These values control how Patroni interacts with the distributed configuration store and how frequently it checks the cluster state.

Cluster Configuration

The repository keeps the database configuration inside the db-server folder.

The Docker Compose configuration mounts the PostgreSQL data directory and the Patroni configuration into the database container:

services:
  db:
    build:
      context: .
    restart: unless-stopped

    ports:
      - "5432:5432"
      - "8008:8008"

    volumes:
      - ./data:/var/lib/postgresql/data
      - ./config/patroni.yml:/etc/patroni/patroni.yml:ro
      - ./config/pg_hba.conf:/etc/patroni/pg_hba.conf:ro

This keeps the database configuration version controlled while allowing the PostgreSQL data itself to remain in its persistent volume.

Starting the Cluster

Once the three database nodes are configured, we can start the services across the VMs. We can verify the resulting cluster state through Patroni before moving to the next layer.

The cluster should expose a state similar to:

The HAProxy Layer

The PostgreSQL cluster can now handle database level failover, but there is still a connection problem.

HAProxy receives PostgreSQL connections on port 5432 and forwards them to the PostgreSQL node that is currently serving as the primary.

This allows the application to use a single database endpoint while the cluster manages which PostgreSQL node is active.

HAProxy in TCP Mode

PostgreSQL uses its own protocol, so HAProxy is configured in TCP mode rather than HTTP mode.

The frontend listens for incoming PostgreSQL connections:

frontend postgres
    bind *:5432

    default_backend postgres_primary

The application therefore only needs to connect to the HAProxy endpoint using the normal PostgreSQL port.

Detecting the Current Primary

We cannot simply configure HAProxy to always send traffic to one server because that server may become unavailable.

Instead, HAProxy uses Patroni API to check the role of each PostgreSQL node.

backend postgres_primary
    mode tcp

    option httpchk GET /primary
    http-check expect status 200

    server postgres1 192.168.122.1:5432 check port 8008
    server postgres2 192.168.122.2:5432 check port 8008
    server postgres3 192.168.122.3:5432 check port 8008

The HTTP request is only used as a health check to ask Patroni whether the node is currently the primary.

Patroni /primary endpoint returns HTTP 200 when the node is the current primary. HAProxy uses that response to determine which backend server should receive write traffic.

The Routing Flow

HAProxy periodically checks the Patroni endpoint on each database node.

Only the node reporting itself as the current primary passes the /primary health check.

The result is that HAProxy can automatically direct new connections to the current primary without requiring configuration change.

Adding Keepalived

Keepalived uses VRRP (Virtual Router Redundancy Protocol) to manage VIP (virtual ip) between the two proxy servers.

For example:

Under normal conditions, HAProxy 1 owns the VIP.

The application therefore connects to:

192.168.122.200:5432

rather than:

192.168.122.10:5432

Choosing the Active Proxy

Keepalived assigns a priority to each node to determine which node should normally own the VIP.

The node with the higher priority becomes the active VRRP node and owns the VIP. If the active node becomes unavailable, the other node can take ownership of the VIP.

This allows the connection endpoint to remain unchanged even though the physical server handling the traffic changes.

Monitoring HAProxy

Simply checking whether the server itself is reachable is not enough.The proxy node could still be running while HAProxy itself has stopped.

In this situation, Keepalived should not continue advertising the VIP from that node.

The project therefore uses a health check script:

#!/bin/bash

if docker ps --format '{{.Names}}' | grep -qx haproxy; then
    exit 0
else
    exit 1
fi

The script checks whether the HAProxy container is running and returns a success or failure status accordingly.

Keepalived then tracks this script:

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    timeout 2
    fall 2
    rise 2
    weight -50
}

vrrp_instance VI_1 {
    state MASTER
    interface ens3

    virtual_router_id 51
    priority 110
    advert_int 1

    virtual_ipaddress {
        192.168.122.200/24
    }

    track_script {
        check_haproxy
    }
}

If the HAProxy check fails repeatedly, the configured negative weight reduces the node's VRRP priority.

This allows the other Keepalived node to become the owner of the VIP.

HAProxy Failure

Suppose the initial state is:

When HAProxy 1 fails, Keepalived detects the HAProxy failure through the tracking script.

The VIP can then move to HAProxy 2:

The application does not need to change its database configuration, It continues connecting to the same address

The physical proxy handling the connection has changed, but the application's endpoint has not.

Testing the Setup

With the complete architecture connected, we can now test whether the system actually behaves as expected.

We will test five scenarios:

Normal cluster state
PostgreSQL primary failure
HAProxy failure
VIP failover

Normal Cluster

Before introducing any failures, we first verify that all components are operating normally.

We can check the Patroni cluster state with:

patronictl -c /etc/patroni/patroni.yml list

We should see one node identified as the Leader and the remaining nodes as replicas, in my setup VM2 is currently the Leader.

We can also verify that HAProxy recognizes the current primary through its statistics page or by checking the backend state.

Finally, we verify that the VIP is currently owned by the active HAProxy node.

ip addr show <network-interface>

The expected result is that the VIP is present on the active proxy.

PostgreSQL Failure

Next, we intentionally stop the current PostgreSQL primary.

For example, if VM2 is currently the primary, we can stop the PostgreSQL container in VM2:

docker stop <postgres-container>

After the failure, VM2 should no longer be available as the primary:

Before:
VM1 -> REPLICA
VM2 -> PRIMARY
VM3 -> REPLICA


After:
VM1 -> PRIMARY
VM3 -> REPLICA

We can then check the Patroni cluster again:

patronictl -c /etc/patroni/patroni.yml list

At this point, the cluster should detect that the previous primary is unavailable and begin the leader transition.

Patroni should promote an eligible replica to become the new primary, here VM1 becomes the new primary.

We can also verify the Patroni REST API directly:

curl -i http://192.168.122.28:8008/primary

A successful primary check should return:

HTTP/1.1 200 OK

This confirms that the database layer has completed the failover.

HAProxy Detects the New Primary

Once Patroni has promoted the new primary, HAProxy should discover the role change through its health checks.

HAProxy Failure

First, we identify which HAProxy node currently owns the VIP.

ip addr show <network-interface>

In my setup VM4 currently owns the VIP:

We then stop HAProxy on VM4:

docker stop haproxy

Because the current Keepalived health check monitors the HAProxy container, the check should fail after the configured number of failed checks.

Keepalived should then reduce the active node's VRRP priority and allow the other proxy to take ownership of the VIP.

VIP Failover

After HAProxy on VM4 fails, the VIP is now on VM5.

The application still uses the same endpoint:

<VIP>:5432

This demonstrates that the proxy layer can fail independently from the PostgreSQL layer.

Conclusion

Building a highly available PostgreSQL environment is not about simply running multiple database servers. It is about designing the system so that individual failures do not automatically become application outages.

In the setup, we combined several components, each responsible for a different part of the availability problem:

PostgreSQL provides the database layer and streaming replication
Patroni manages cluster membership, leader election, and automatic failover
etcd provides the distributed state required by Patroni
HAProxy provides a stable database endpoint and routes connections to the current primary
Keepalived provides a virtual IP so that the HAProxy layer does not depend on a single server

The important lesson is that high availability is not achieved by adding redundancy alone. Every component needs to have a clear role in detecting failures, making decisions, and redirecting traffic when something goes wrong.

This project is intentionally built as a lab environment, but the concepts can be applied to more complex production deployments.

You can reproduce the entire setup and experiment with the failure scenarios yourself by cloning my GitHub repository below:

https://github.com/muhammadyulasfipahrizal/postgres-ha.git

How I Built End To End Observability Stack with OpenTelemetry, Prometheus, Grafana, Loki, and Jaeger

Mo Rizal — Sat, 05 Sep 2026 05:00:39 +0000

When an application is running, knowing that it is up does not necessarily mean knowing what it is doing.

For example, imagine an API request that takes several seconds to complete. We would want to answer questions such as:

Is the application receiving more requests than usual?
What happened during the request?
How long does the request take?
Did the application encounter an error?
Which part of the request should we investigate?

Instead of looking at behavior through a single source, we should collect different types of signal and make them available in one observability stack.

What We Need to Observe

To understand the application from different perspectives, we need three types of telemetry: metrics, logs, and traces.

Each signal provides a different view of the application, and each becomes useful at a different stage of an investigation.

Metrics

Metrics helps us to understand the overall behavior of the application.

How many requests are being processed?
Are request rates increasing?
Are errors becoming more frequent?
Is the application behaving differently from normal?

This gives us the high level view we need to detect if something might be wrong.

Logs

Logs give us details about individual events inside the application, such as a request starting, an operation completing, or an error occurring. They become especially useful when we need to understand the context behind an abnormal metric.

Traces

A trace allows us to follow a request through its execution and understand its timing. This becomes particularly useful when investigating slow requests or determining which part of an operation deserves further attention.

Centralize Collection

Instead of making the application communicate independently with several observability backends, we can place a telemetry collection layer between the application.

This provide a consistent way to receive telemetry from the application and route it to the systems responsible for storing and analyzing each signal.

Visualization

Jumping between different interfaces makes investigation slower, especially when we need to correlate information from multiple signals.

Therefore we need a visualization layer where metrics, logs, and traces can be explored together.

Architecture

Now that we know what we need to collect, the next question is how these signals should move through the system.

The architecture looks like this:

The application is the source of the telemetry. When the application generates metrics, logs, and traces, these signals are sent to the OpenTelemetry Collector using OTLP.

The Collector then routes each signal to the appropriate backend.

Metrics are exposed for Prometheus to collect, logs are sent to Loki, and traces are forwarded to Jaeger. Grafana sits on top of these systems and gives us a single place to explore the data.

This creates a clear separation between the application and the systems that store and analyze its telemetry.

Application to Collector

Consider a request entering the API:

GET /api/orders

The application processes the request and generates telemetry while doing so.

Instead of sending that telemetry directly to multiple backends, the application sends it to the Collector The application therefore only needs to know where the telemetry collection layer is located.

This becomes especially useful as the observability stack grows. Additional processing or exporters can be introduced at the Collector without requiring the application to establish a new connection to every backend.

Collector to Backends

Once telemetry reaches the Collector, the signals follow different paths depending on their type.

Each backend therefore has a specific responsibility.

Prometheus handles the metrics data, Loki handles application logs, and Jaeger handles traces.

From Backends to Grafana

Once the telemetry reaches its respective backends, we need a way to explore and make sense of the data.

Grafana provides this layer by connecting to Prometheus, Loki, and Jaeger, giving us a single interface to query and visualize the telemetry generated by our application.

The Application

Before sending any telemetry, we need an application that can generate it.

For this project, we use a small Go API as the application being observed. The API exposes several endpoints that represent different application behaviors:

GET /health
GET /api/orders
GET /api/transaction
GET /api/users

Each endpoint gives us a different scenario to observe.

/health represents a simple successful request
/api/orders returns application data
/api/transaction simulates a slow operation
/api/users intentionally returns an error

This gives us enough variety to see how metrics, logs, and traces behave when we test the observability stack later.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

Github Repository

Initializing Opentelemetry

The telemetry setup is separated from the HTTP handlers in telemetry.go.

The application first defines its service identity:

res, err := resource.New(
    ctx,
    resource.WithAttributes(
        semconv.ServiceName("observability-api"),
    ),
)

This gives the telemetry a consistent service name: observability-api.

More importantly, each telemetry provider is configured to send its data to the OpenTelemetry Collector:

otlptracegrpc.WithEndpoint("otel-collector:4317")

The same Collector endpoint is used for metrics and logs.

This means the application does not need to know where Prometheus, Loki, or Jaeger are running. It only needs to know the address of the Collector.

The trace, metric, and log providers are then registered globally in the application.

Instrumenting HTTP Requests

The next step is making the HTTP server itself observable.

mux.HandleFunc("/health", health)
mux.HandleFunc("/api/orders", orders)
mux.HandleFunc("/api/transaction", transaction)
mux.HandleFunc("/api/users", users)

Instead of passing this multiplexer directly to the HTTP server, we wrap it with OpenTelemetry's HTTP instrumentation:

handler := otelhttp.NewHandler(
    mux,
    "HTTP Server",
)

The resulting handler is then used by the HTTP server:

server := &http.Server{
    Addr:    ":8080",
    Handler: handler,
}

This instrumentation sits in front of all four endpoints:

As requests pass through this layer, OpenTelemetry can capture HTTP request telemetry and associate it with the request context.

Logging Inside Each Endpoint

For application specific events, we use a small helper called emitLog.

The helper receives the request context, severity, message, and additional attributes:

func emitLog(
    ctx context.Context,
    severity log.Severity,
    message string,
    attrs ...attribute.KeyValue,
) {
    ...
    logger.Emit(ctx, record)
}

The important part is the context:

logger.Emit(ctx, record)

Each handler passes r.Context() into emitLog, so the log is emitted using the context of the current HTTP request.

The endpoint therefore does not need to know how the log will eventually reach Loki. It only creates the log event through the OpenTelemetry logging API.

`/health`

The simplest example is the health endpoint.

When we call:

GET /health

the handler generates an informational log:

emitLog(
    r.Context(),
    log.SeverityInfo,
    "health check",
    attribute.String("endpoint", "/health"),
)

The application therefore produces telemetry as part of processing the request rather than requiring a separate logging pipeline inside the handler.

`/api/orders`

The orders endpoint follows the same pattern, but adds more context to the log.

After creating the orders, the handler emits:

emitLog(
    r.Context(),
    log.SeverityInfo,
    "orders fetched successfully",
    attribute.String("endpoint", "/api/orders"),
    attribute.Int("order_count", len(orders)),
)

Now the telemetry contains not only the event itself, but also structured attributes such as the endpoint and number of orders.

For example:

message:     orders fetched successfully
endpoint:    /api/orders
order_count: 3

These attributes become useful later when we query the logs with Loki.

`/api/transaction`

The transaction endpoint gives us a more interesting example because it intentionally takes three seconds to complete.

First, the handler records the start of the operation:

start := time.Now()

emitLog(
    r.Context(),
    log.SeverityInfo,
    "transaction started",
    attribute.String("endpoint", "/api/transaction"),
)

After the simulated operation:

time.Sleep(3 * time.Second)

the handler calculates the duration and emits another log:

duration := time.Since(start)

emitLog(
    r.Context(),
    log.SeverityInfo,
    "transaction completed",
    attribute.String("endpoint", "/api/transaction"),
    attribute.Int64("duration_ms", duration.Milliseconds()),
)

`/api/users`

Finally, we have an endpoint that intentionally returns an error.

emitLog(
    r.Context(),
    log.SeverityError,
    "failed to fetch users",
    attribute.String("endpoint", "/api/users"),
    attribute.Int("status_code", http.StatusInternalServerError),
)

The request therefore produces an error-level log before returning HTTP 500.

This gives us a different scenario from /api/transaction: instead of investigating latency, we can later use the observability stack to investigate an application error.

Building the Observability Pipeline

With the application ready, we can now configure how that telemetry is processed and delivered to each backend.

Configuring the OpenTelemetry Collector

The Collector accepts OTLP through gRPC:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

The Collector then uses the batch processor:

processors:
  batch:

Rather than defining one large pipeline for every signal, the configuration separates metrics, logs, and traces:

service:
  pipelines:
    metrics:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - prometheus

    traces:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - otlp/jaeger

    logs:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - otlphttp/loki

Collector to Prometheus

Metrics use the Prometheus exporter:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

The Collector exposes the metrics endpoint on port 8889.

This is slightly different from the way telemetry enters the Collector.

The application sends telemetry to the Collector, but Prometheus does not receive metrics through a push connection. Instead, Prometheus periodically scrapes the endpoint exposed by the Collector.

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "otel-collector"
    static_configs:
      - targets:
          - "otel-collector:8889"

Prometheus therefore scrapes the Collector every five seconds.

Collector to Loki

Logs use the otlphttp/loki exporter:

otlphttp/loki:
  endpoint: http://loki:3100/otlp

The Collector sends the processed logs to Loki through its OTLP HTTP endpoint.

This works particularly well with the structured information we added to the application logs.

For example, the /api/orders endpoint produces:

message:     orders fetched successfully
endpoint:    /api/orders
order_count: 3

Instead of treating the log as only a text message, these additional attributes provide context that can be queried later.

Loki is configured to support structured metadata:

limits_config:
  allow_structured_metadata: true
  volume_enabled: true

The project also uses filesystem storage for Loki, backed by the loki-data Docker volume.

Collector to Jaeger

Traces use a separate OTLP exporter:

otlp/jaeger:
  endpoint: jaeger:4317
  tls:
    insecure: true

The traces pipeline connects this exporter to the OTLP receiver and batch processor.

Jaeger is configured to receive OTLP through gRPC:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

For this project Jaeger uses an in memory backend:

extensions:
  jaeger_query:
    storage:
      traces: memory

  jaeger_storage:
    backends:
      memory:
        memory:
          max_traces: 100000

The trace exporter writes into that memory storage:

exporters:
  jaeger_storage_exporter:
    trace_storage: memory

Running the Pipeline

All of these components are defined as separate Docker Compose services.

The Collector mounts our configuration file into the container and exposes its OTLP and Prometheus endpoints:

otel-collector:
  image: otel/opentelemetry-collector-contrib:latest
  command:
    - "--config=/etc/otelcol-contrib/otel-collector.yml"
  volumes:
    - ./otel/otel-collector.yml:/etc/otelcol-contrib/otel-collector.yml:ro
  ports:
    - "4317:4317"
    - "8889:8889"

The services are connected through the research-observability Docker network, allowing the Collector to reach services such as loki and jaeger directly by their Docker service names.

Bringing Everything Together with Grafana

grafana:
  image: grafana/grafana:latest
  container_name: grafana
  environment:
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=admin
    - GF_AUTH_ANONYMOUS_ENABLED=false
  volumes:
    - grafana-data:/var/lib/grafana
  ports:
    - "3001:3000"

The container exposes Grafana on port 3001 of the host, while Grafana itself listens on port 3000 inside the container.

Because Grafana, Prometheus, Loki, and Jaeger are connected to the same research-observability Docker network, Grafana can communicate with the other services using their Docker service names.

This means we can configure the data sources using addresses such as:

http://prometheus:9090
http://loki:3100
http://jaeger:16686

Adding Prometheus as a Data Source

From the Grafana dashboard, open:

Connections → Data sources

Then select Add data source and choose Prometheus.

For the server URL, we use the Docker service name instead of localhost:

http://prometheus:9090

After entering the URL, select Save & test.

If the connection is successful, Grafana confirms that it can communicate with Prometheus.

Adding Loki

We repeat the same process for Loki.

Go to:

Connections → Data sources → Add data source

Then select Loki.

For the URL, use:

http://loki:3100

After entering the URL, select Save & test.

Once Grafana confirms the connection, Loki is available as a data source for exploring application logs.

Adding Jaeger

Connections → Data sources → Add data source

select Jaeger.

For the URL, use Jaeger's query endpoint:

http://jaeger:16686

This is different from the OTLP endpoint used by the Collector.

The Collector sends traces to Jaeger through its OTLP gRPC endpoint, while Grafana connects to Jaeger's query interface to retrieve traces.

After entering the URL, select Save & test.

At this point, Grafana has access to all three observability backends.

Verifying the Data Sources

Before building any dashboard, we can verify that Grafana can actually query the data.

Open:

Explore

From the data source selector, we can choose between:

Prometheus
Loki
Jaeger

For example, selecting Prometheus allows us to run a metrics query directly from Grafana. This gives us a simple way to verify each connection before creating dashboards.

Investigation

Now that Grafana is connected to our observability backends, we can use the stack to investigate actual application problems.

Instead of simply checking whether telemetry is being collected, we want to answer a more practical question:

What happens when something goes wrong?

For this example, we will investigate /api/transaction

A Slow Transaction

As mention before our /api/transaction endpoint intentionally waits for three seconds before returning a response:

time.Sleep(3 * time.Second)

From the users perspective, the problem is simple: the API feels slow.

However, knowing that a request is slow does not immediately tell us why.

We start by sending a request to:

GET /api/transaction

After generating several requests, we can open our Grafana dashboard and look at the request duration.

The metric tells us that the request is taking significantly longer than a normal API request.

But metrics alone do not provide much context about what happened inside the request.

This is where we move to the logs.

Looking at the Logs

In Grafana, we can open the Loki data source and search for logs generated by the transaction endpoint.

The application produces two important events:

transaction started
transaction completed

The completion log also contains the duration:

message:     transaction completed
endpoint:    /api/transaction
duration_ms: 3000

Now we know that the request really did spend approximately three seconds processing the transaction.

The logs give us more context than the metric alone, but we still do not have a detailed view of the request execution.

Following the Trace

The same request also generates a trace that is exported through the OpenTelemetry Collector and stored in Jaeger.

From Grafana, we can open the corresponding trace and inspect its timeline.

The trace gives us a view of how long the request took and where that time was spent.

From Symptom to Cause

This scenario demonstrate the main reason we built the observability stack.

Each signal gives us a different level of information.

Metrics help us detect that something unusual is happening.
Logs give us additional application context.
Traces allow us to follow the request and investigate its execution.

The important part is not simply having all three signals available. The real value comes from being able to use them together when investigating a problem.

Conclusion

Building this observability stack gave us a practical way to understand what is happening inside an application instead of only knowing whether it is running.

We started with a simple Go API and added OpenTelemetry to generate metrics, logs, and traces. The OpenTelemetry Collector then provided a central layer for processing and routing those signals to Prometheus, Loki, and Jaeger.

Grafana brought everything together into a single interface.

The /api/transaction example showed why having multiple signals matters. Metrics helped us identify that the request was slow. Logs provided additional context about the transaction, while the trace allowed us to inspect the request execution and its timing.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/E2E-observability-stack.git

[Boost]

Mo Rizal — Wed, 02 Sep 2026 06:44:45 +0000

Mo Rizal

Sep 2

A Practical Approach To Securing API With Cloudflare WAF

8 min read

A Practical Approach To Securing API With Cloudflare WAF

Mo Rizal — Wed, 02 Sep 2026 06:22:39 +0000

Building an API is relatively easy. The harder problem starts when the API becomes publicly accessible.

Once an API is exposed to the Internet, it can receive much more than the requests we expect. Attackers can send malicious requests, repeatedly hit endpoints, use unexpected HTTP methods, send suspicious query parameters, or simply generate enough traffic to consume application resources.

A common approach is to handle all of these concerns inside the application. The application receives the request, validates it, checks authentication, and finally decides whether the request should be processed.

But this raises another question:

What if we could reject unwanted requests before they reach the application?

This is where a Web Application Firewall (WAF) can help. A WAF is a security layer that inspects incoming web traffic and applies rules to determine whether a request should be allowed, challenged, or blocked.

Consider a simple request to an orders API:

GET /api/orders?id=1
Authorization: Bearer <token>

Without an edge security layer, the request travels directly to the application:

With an Web Application Firewall such as Cloudflare WAF, the request can be evaluated before it reaches the application:

If a request violates one of the configured policies, Cloudflare can reject it instead of forwarding it to the application.

This is the idea behind an edge security layer.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

Github Repository: https://github.com/muhammadyulasfipahrizal/WAF-API.git

What We Are Building

In this project, we will build a small Go API and place Cloudflare WAF in front of it to demonstrate several edge security controls.

The application is intentionally simple. It provides enough endpoints and request patterns to demonstrate different WAF policies without introducing unnecessary application complexity.

The application exposes three endpoints:

/api/login
/api/orders
/admin

API

The /api/login endpoint is used to authenticate a user. For this demo, valid credentials return a JWT that can then be used when accessing /api/orders.

The /api/orders endpoint represents a simple API that returns order data. It supports several query parameters:

id
item
price
status

These parameters give us different request patterns to test against the WAF.

The /admin endpoint represents an administrative page that should not be publicly accessible.

Security Controls

The WAF configuration demonstrates the following protections:

Security Control	Purpose
Main page challenge	Challenge requests when accessing the application
Authorization header enforcement	Reject protected API requests without an `Authorization` header
HTTP method filtering	Allow only expected methods for an endpoint
IP restriction	Restrict access to the restricted page
Suspicious request filtering	Block requests containing suspicious patterns
Rate limiting	Limit excessive request traffic
API schema validation	Ensure requests follow the defined API contract

Project Architecture

Cloudflare sits between the Internet and the application. This allows incoming requests to be evaluated against edge level security policies before they reach the origin.

For example, since /api/orders is a read only endpoint, requests using unsupported methods such as POST can be rejected by Cloudflare before they are forwarded to the application:

The responsibility is divided between two layers:

Layer	Responsibility
Cloudflare WAF	Challenge, rate limiting, method filtering, IP restrictions, suspicious request filtering, schema validation
Go API	JWT authentication, application validation, business logic

Passing the WAF does not mean that a request is trusted. It only means that the request passed the configured edge level policies.

Application Structure

cmd directory contains the application entry point, while internal contains the authentication, middleware, and HTTP handlers.
schema directory contains the OpenAPI definition used to describe the API contract and demonstrate schema validation with Cloudflare.
web directory contains the frontend used to interact with the API and test the WAF scenarios.
The application is containerized using the Dockerfile, and
.github/workflows is used to build and publish the Docker image with Github Action.

Cloudflare WAF

Cloudflare WAF provides several mechanisms that can be used to enforce policies at the edge. In this project, each rule is designed to address a specific type of unwanted request.

Challenge the Main Page

The first rule protects the main application page with an interactive challenge.

(http.request.uri.path eq "/")

When accessing /, the client must pass the Cloudflare challenge before the request continues to the application.

This rule prevent automated clients from accessing the application page without first passing a basic verification step. This is useful for reducing unwanted bot traffic before it reaches the origin.

Require Authorization Header for APIs

Protected API endpoints should not receive requests that do not contain an Authorization header.

The login endpoint is excluded because clients need to access it before they have a token.

starts_with(http.request.uri.path, "/api/")
and not any(http.request.headers["authorization"][*] ne "")
and http.request.uri.path ne "/api/login"

The behavior is:

This does not replace authentication.

The WAF checks for the presence of the Authorization header. The Go application validates the JWT itself.

For example, a request containing:

Authorization: Bearer <token>

may pass the WAF rule, but the application still needs to determine whether the token is valid, expired, and properly signed.

This provides a simple defense-in-depth mechanism: the edge performs an initial check, while the application performs the actual authentication.

Enforce HTTP Methods

The /api/orders endpoint is designed as a read-only API, so only GET requests are expected.

(http.request.uri.path eq "/api/orders" 
and http.request.method ne "GET")

The behavior is:

The reasoning is straightforward:

The /api/orders endpoint is designed as a read-only API, so unsupported write methods provide no value.

Protect Admin Page By IP

The /admin page represents an administrative interface and should not be publicly accessible.

For this project, access is restricted to a specific trusted IP address.

(http.request.uri.path eq "/admin" 
and ip.src ne ADMIN_IP)

The flow is:

This provides a simple way to restrict an administrative endpoint at the edge.

Block Suspicious Requests

This rule filters API requests containing patterns commonly associated with SQL injection attempts.

starts_with(http.request.uri.path, "/api/")
and (
  lower(http.request.uri.query) contains " or "
  or lower(http.request.uri.query) contains "' or "
  or lower(http.request.uri.query) contains "union select"
  or lower(http.request.uri.query) contains "select "
  or lower(http.request.uri.query) contains "drop table"
  or lower(http.request.uri.query) contains "insert into"
  or lower(http.request.uri.query) contains "delete from"
  or lower(http.request.uri.query) contains "--"
  or lower(http.request.uri.query) contains "/*"
  or lower(http.request.uri.query) contains "*/"
)

For example:

GET /api/orders?id=1' OR '1'='1

can be detected as suspicious and therefore blocked by Cloduflare WAF.

Application-level protection is still required, including input validation, parameterized queries, and secure database practices.

Rate Limiting

An API can also be abused through repeated requests.

Even when individual requests are valid, a large number of requests can consume CPU, memory, bandwidth, connection pools, and other application resources.

Cloudflare can apply rate limiting before this traffic reaches the origin.

Excessive requests can be stopped before they consume resources at the origin.

This is particularly useful when the origin has limited resources or when an endpoint is frequently targeted by bots.

API Schema Validation

An API schema describes what a valid API request should look like.

For example, the /api/orders endpoint define the id query parameter as an integer:

/api/orders:
  get:
    parameters:
      - name: id
        in: query
        schema:
          type: integer

A request such as:

GET /api/orders?id=10
matches the expected type.

The OpenAPI definition can be uploaded to Cloudflare WAF and used for API schema validation.

The flow is:

This is useful because the API contract can become part of the security boundary.

Instead of being used only as documentation, the OpenAPI definition can also help Cloudflare determine whether incoming requests conform to the expected API structure.

Results

After deploying the Go API behind Cloudflare WAF, the configured policies were tested against different request scenarios.

The following results show the behavior of each control.

Challenge the Main Page

Accessing / triggers the configured Cloudflare interactive challenge.

After successfully passing the challenge, the request continues to the application.

Require Authorization Header for APIs

A request to a protected API without an Authorization header is rejected:

GET /api/orders?id=1

A request containing the header can continue to the application, where the JWT is validated.

Enforce HTTP Methods

The /api/orders endpoint accepts GET, while unsupported methods are rejected by Cloudflare.

For example:

POST /api/orders
→ Blocked

Protect the Admin Page by IP

Requests to /admin from the configured administrator IP are allowed, while requests from other IP addresses are blocked.

Block Suspicious Requests

A request containing a suspicious SQL-like pattern is rejected by the WAF.

For example:

GET /api/orders?id=1' OR '1'='1

is detected and blocked by the configured rule.

Rate Limiting

Repeated requests eventually trigger the configured rate-limiting policy.

Normal traffic continues to the origin, while traffic exceeding the configured policy is stopped by Cloudflare.

API Schema Validation

Requests that conform to the OpenAPI contract can reach the origin, while requests that do not match the defined schema can be rejected.

For example:

GET /api/orders?id=10

matches the expected integer type for id.

Conclusion

Securing an API is not only about adding authentication to the application.

For an Internet-facing service, there is value in having an additional security layer between the client and the origin. Cloudflare WAF can inspect and control incoming traffic at the edge, helping protect the application from a wide range of unwanted or potentially malicious requests before they reach the origin.

In this project, we demonstrate several of these capabilities, including challenges, rate limiting, HTTP method restrictions, IP restrictions, suspicious request filtering, and API schema validation.

The important lesson from this project is that security responsibilities do not have to live entirely inside the application.

The edge can handle policies that are suitable for inspection before the request reaches the origin, while the application remains responsible for authentication, business logic, and application level validation.

The code snippets in this article focus on the important parts of the implementation. For the complete source code, you can find it through repository below.

Github Repository

How I Built A Blue Green Deployment Pipeline With Jenkins

Mo Rizal — Sat, 29 Aug 2026 07:26:43 +0000

Deploying a new application version can be simple when the application is small and downtime is acceptable.

An engineer can stop the current application, deploy the new version, and start it again. The problem appears when the application is expected to remain available while a new version is being released.

This creates a window where the application may become unavailable.

The problem becomes more visible when the new version also has an issue.

A deployment might complete successfully from the infrastructure's perspective, but the application itself could still be unhealthy. A container may start but fail its health check, the application may not respond correctly, or a configuration problem may only become visible after the new version receives traffic.

In this situation, simply replacing the old version is risky.

Deployment process needs a way to prepare the new version without immediately exposing it to users, verify that it is working, and only then switch traffic to it.

This is where blue-green deployment becomes useful.

Instead of replacing the running application directly, two environments are maintained:

Blue, the currently active version receiving production traffic
Green, the new version being prepared and validated

The new release is deployed to the inactive environment first. Once it passes the required checks, the reverse proxy switches traffic from the active environment to the new one.

The new application version can be started and tested while the existing version continues serving users. If the new version fails validation, traffic does not need to move at all. The existing environment can simply remain active.

Now Kubernetes is one way to implement this kind of deployment strategies. It provides powerful primitives for managing containers, service discovery, traffic routing, health checks, scaling, and automated rollouts.

However, Kubernetes also introduces additional infrastructure and operational complexity.

Not every application needs that level of orchestration, and not every company wants to operate a Kubernetes cluster.

For smaller applications with simpler operational requirements, a combination of Docker, reverse proxy, and a CI/CD tool can provide a much simpler alternative.

By the end of this project, we will have a Jenkins based deployment workflow that demonstrates:

Automated application testing and image building
Blue-green application environments
Health checks before traffic switching
Reverse proxy based traffic switching
SSH based remote deployment
Safe deployment path

What We Are Building

The complete implementation is available in repository bellow:

Github Repository

The goal of this project is to build a simple deployment environment where Jenkins can deploy a new application version alongside the currently running version, verify it, and then switch traffic to it.

The architecture consists of a Jenkins server and a separate deployment server.

The deployment server runs the application containers and the reverse proxy, while Jenkins acts as the automation layer responsible for executing the deployment workflow.

Architecture

There are four main pieces involved:

Jenkins, orchestrates the deployment process.
Blue environment, runs one version of the application.
Green environment, runs the other version of the application.
Nginx, receives user traffic and determines which environment should receive it.

Application

To demonstrate the deployment workflow, For this project, we use a small Go HTTP application.

The application does not contain any complex business logic because the main focus of this project is the deployment process, not the application itself.

The application exposes two endpoints:

GET /health
GET /version

Health Endpoint

The /health endpoint is used to determine whether the newly deployed application is ready to receive traffic.

A successful response returns HTTP 200 OK:

GET /health

HTTP/1.1 200 OK

OK

Jenkins uses this endpoint after starting the new environment.

Version Identification

The second endpoint /version, allows us to identify which application version is currently serving the request.

The application reads the version from the APP_VERSION environment variable:

APP_VERSION=v1
APP_VERSION=v2

The endpoint then returns the configured version:

GET /version

v1

Or after deploying v2 release:

GET /version

v2

This makes the deployment behavior easy to observe.

Instead of simply checking whether the application responds, we can verify exactly which version is receiving traffic.

Jenkins

Jenkins is the automation engine for the deployment process.

Instead of manually connecting to the deployment server and executing commands such as:

docker build
docker compose up
curl /health

Jenkins executes these steps as a repeatable pipeline.

The pipeline implemented in this project performs the following stages:

The repository contains separate Jenkinsfiles for the two deployment directions:

deploy/
├── Jenkinsfile.blue
└── Jenkinsfile.green

This is because the deployment target alternates between the two environments.

For example, Jenkinsfile.blue deploys the new image to the Blue environment and then switches Nginx to Blue. Jenkinsfile.green does the same thing for Green.

The pipeline also uses the Jenkins build number as the Docker image tag:

environment {
    APP_NAME   = 'zero-downtime-app'
    IMAGE_TAG  = "${BUILD_NUMBER}"
    DEPLOY_DIR = '/home/ubuntu/projects/zero-downtime-jenkins'
}

This gives every Jenkins build its own image tag instead of continuously overwriting a generic latest image.

For example:

Build #1 → zero-downtime-app:1
Build #2 → zero-downtime-app:2
Build #3 → zero-downtime-app:3

This makes it easier to identify exactly which build is running in an environment.

Blue Environment

The Blue environment represents one side of the deployment pair.

In this project, the application runs inside a Docker container named:

app-blue

The application itself listens on port 8080 inside the container, while Docker publishes it on port 8081 on the deployment server.

The corresponding Docker Compose configuration is kept in:

deploy/docker-compose.blue.yml

This allows the Blue environment to be started independently from the Green environment.

When Blue is the active environment, Nginx routes application traffic to:

app-blue:8080

and is deployed using Jenkinsfile.blue.

Green Environment

The Green environment provides the second application slot.

It runs in a separate Docker container:

app-green

Like Blue, the application listens on port 8080 inside the container. Docker publishes it on a different host port:

The Green environment is defined by:

deploy/docker-compose.green.yml

When Green is the active environment, Nginx routes application traffic to:

app-green:8080

and is deployed using Jenkinsfile.green.

Reverse Proxy

In this project we use Nginx, who acts as the entry point for application traffic.

Users do not need to know whether Blue or Green is currently active. They simply connect to the application through Nginx.

server {
    listen 80;

    location / {
        proxy_pass http://app;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For           $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Nginx uses an upstream configuration to determine which container should receive the traffic.

For example, when Blue is active:

upstream app {
    server app-blue:8080;
}

And when Green has been successfully deployed and verified, Jenkins changes the upstream configuration:

upstream app {
    server app-green:8080;
}

The pipeline does not simply reload Nginx after changing the configuration.

It first validates the configuration:

docker exec app-nginx nginx -t

and only then reloads Nginx:

docker exec app-nginx nginx -s reload

This is important because an invalid Nginx configuration should not be allowed to replace the currently working configuration.

The actual traffic switching logic is implemented directly in the Jenkins pipeline.

SSH Connection

Jenkins and the deployment environment are intentionally separated.

The Jenkins server does not run the application containers. Instead, Jenkins connects to the deployment server through SSH and executes the required deployment commands there.

The pipeline uses two Jenkins credentials:

deploy-host
ssh-vm1

The deploy-host credential provides the deployment server address, while ssh-vm1 is used by the Jenkins SSH agent for authentication.

For example, when Jenkins needs to deploy the Green environment, it connects to the deployment server and executes Docker Compose there:

APP_IMAGE_TAG=${IMAGE_TAG} \
docker compose \
    -f docker-compose.green.yml \
    up -d

The same SSH connection is also used for testing, health verification, and traffic switching.

This means Jenkins acts as the orchestrator, while the deployment server remains responsible for running the actual workload.

Result

The deployment workflow was tested through three scenarios: deploying the initial version (Blue), deploying the new version (Green), and handling a failed Green deployment.

Deploying v1

The first deployment starts the application in the Blue environment.

Jenkins runs the test, builds the Docker image, deploys app-blue, and verifies its health endpoint:

[Pipeline] { (Deploy Blue)

Container app-blue Creating
Container app-blue Created
Container app-blue Starting
Container app-blue Started

The health check then succeeds:

curl --fail http://127.0.0.1:8081/health

{"status":"ok"}

After the initial deployment, a request through Nginx confirms that v1 is serving traffic:

curl http://localhost

{
  "message": "Hello",
  "version": "v1",
  "hostname": "5f6a41482e36"
}

At this point, the application is running successfully and Blue is the active environment.

Deploying v2

The next deployment uses the Green environment.

Jenkins first runs the application tests:

ok      jenkins-zero-downtime    0.002s

It then builds a new image:

zero-downtime-app:6

and starts the Green environment:

Container app-green Creating
Container app-green Created
Container app-green Starting
Container app-green Started

At this point app-blue is still running. The new version is therefore deployed without first stopping the existing application.

Verifying v2

Before changing production traffic, Jenkins checks the new environment directly:

curl --fail http://127.0.0.1:8082/health

The check succeeds:

{"status":"ok"}

This confirms that the new container is responding successfully before it is exposed through Nginx.

Switching Traffic

After the health check succeeds, Jenkins updates the Nginx upstream configuration to point to Green:

upstream app {
    server app-green:8080;
}

The configuration is validated before Nginx is reloaded:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Nginx is then reloaded successfully.

A request through the normal application entry point now returns v2:

curl http://localhost

{
  "message": "Hello",
  "version": "v2",
  "hostname": "9a86f0f3486"
}

The deployment is now complete, and Jenkins removes the old Blue container:

docker rm -f app-blue

app-blue

What Happens When Deployment Fails?

The workflow was also tested with an intentionally broken application version.

In the failed branch, the /health behavior causes the existing Go unit test to fail:

--- FAIL: TestHealth (0.00s)
    main_test.go:16: expected status 200, got 500
FAIL
FAIL    jenkins-zero-downtime    0.002s

Because the test stage fails, Jenkins stops the deployment before creating or exposing the new version:

Build Image       → SKIPPED
Deploy Green      → SKIPPED
Verify Green      → SKIPPED
Switch Traffic    → SKIPPED
Stop Blue         → SKIPPED

The pipeline finishes with:

ERROR: script returned exit code 1
Finished: FAILURE

Most importantly, the Switch Traffic stage is never executed.

The existing version therefore remains active:

curl http://localhost

{
  "message": "Hello",
  "version": "v1",
  "hostname": "5f6a41482e36"
}

This demonstrates the failure behavior we wanted from the deployment workflow:

A release that fails validation does not automatically replace the currently running version.

The result is a simple deployment workflow that provides the core safety properties of blue-green deployment.

Conclusion

This project demonstrated how Jenkins, Docker, and Nginx can be combined to implement a simple blue-green deployment workflow without Kubernetes.

The new version is tested and verified before receiving traffic, while the existing version remains available during deployment. When validation fails, Jenkins stops the pipeline and leaves the current version untouched.

For smaller applications, this provides a simple and practical way to reduce deployment downtime and the risk of releasing a broken version.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the full source code in the repository below.

https://github.com/muhammadyulasfipahrizal/blue-green-jenkins.git

Building A Secure CI Pipeline With GitHub Actions, SonarQube, And Trivy

Mo Rizal — Thu, 27 Aug 2026 05:25:07 +0000

A CI pipeline can tell us that an application works, but can it tell us that the application is safe to ship?

A successful build does not necessarily mean that the software is secure.

An application can pass its tests while still containing insecure and bad code patterns. Other than that container image built from the application may inherit vulnerable packages.

The problem is that these security issues exist at different layers:

Checking only one of these layers is not enough.

For example, static code analysis can help identify security issues in the source code, but it does not tell us whether the packages inside our container image contain known vulnerabilities.

Likewise, scanning a container image does not tell us whether the application source itself contains problematic code.

A traditional CI workflow might look like this:

The application is tested and, if everything works, the resulting artifact is published. But there is no explicit security check between the code and the artifact.

That is what we are going to address in this article.

The complete implementation is available in repository bellow:

Github Repository

What We Are Building

We will build a small Go API and place it behind an automated CI pipeline.

The application itself is intentionally simple. There is no database, authentication system, or external service. The purpose of the application is to provide an artifact that we can test, analyze, containerize, and scan.

Application

The application is a small HTTP API written in Go.

It provides two endpoints:

GET /health
GET /api/users

The /health endpoint is used to verify that the application is running, while /api/users returns a small list of users.

This gives us enough application code to run unit tests and static analysis without introducing unnecessary application complexity.

The application is located under the cmd/ directory, with the HTTP handlers implemented in cmd/main.go.

The application is then packaged as a Docker image.

CI Pipeline

The application is connected to GitHub Actions that automate the entire process.

The final pipeline looks like this:

Each stage has a specific responsibility.

API Test, verifies that the application behaves as expected before any further analysis or packaging takes place.
SonarQube Scan, analyzes the source code for potential security, reliability, and maintainability issues.
Docker Build, packages the application and its runtime environment into a Docker image.
Trivy Scan, scans the resulting container image for known vulnerabilities in the image's operating-system packages, dependencies, and other components.
Push, publishes the container image to GitHub Container Registry after it has passed the preceding stages.

Project Structure

Each part of the repository has a specific responsibility.

.github/workflows/ci.yml defines the CI workflow and orchestrates the entire process.
cmd/main.go contains the Go API itself, including the HTTP server and API endpoints.
Dockerfile defines how the Go API is compiled and packaged into a Docker image.
sonar-project.properties contains the SonarQube project configuration, including which directories should be analyzed and how test files should be identified.

Implementation

The application in this project intentionally contains several code patterns that can be detected by static analysis.

The Docker image also uses an intentionally outdated base image so that the container scan has meaningful vulnerabilities to report.

The goal is not to demonstrate how to write perfect Go code. The goal is to demonstrate how this CI pipeline can identify problematic code and image automatically.

Go API

The API implementation itself is straightforward, but several functions in main.go contain intentionally problematic code.

For example the processUser function.

func processUser(user User) string {
    if user.Name == "" {
        if user.Email == "" {
            if user.ID > 0 {
                return "invalid-user"
            }
        } else if user.ID > 0 {
            return "invalid-user"
        }
    }

    if user.Email == "" {
        if user.ID > 0 {
            return "invalid-user"
        }
    }

    if user.ID > 0 {
        if user.Name == "Alice" {
            if user.Email == "alice@example.com" {
                return "valid-user"
            }
        }

        if user.Name == "Bob" {
            if user.Email == "bob@example.com" {
                return "valid-user"
            }
        }

        return "valid-user"
    }

    return "invalid-user"
}

The function intentionally contains deeply nested conditional statements and duplicated logic.

For example, the same user.ID > 0 condition is evaluated in multiple branches.

This makes the function harder to understand and maintain than necessary.

The same idea is used in getUserRole.

func getUserRole(role string) string {
    switch role {
    case "admin":
        return "Administrator"
    case "manager":
        return "Manager"
    case "developer":
        return "Developer"
    // ...
    default:
        return "Unknown"
    }
}

The function contains a long list of similar branches.

This is not necessarily a security vulnerability by itself. It is included to give the static analyzer code that can be evaluated for maintainability and code-quality issues.

Another example is unreachableCode:

func unreachableCode() string {
    return "valid"

    log.Println("this code is never reached")

    return "unreachable"
}

The statements after the first return can never be executed.

This is exactly the kind of implementation problem that static analysis can identify without executing the application.

We also intentionally introduce duplicated return values:

func duplicatedStrings(value int) string {
    if value == 1 {
        return "invalid-user"
    }

    if value == 2 {
        return "invalid-user"
    }

    if value == 3 {
        return "invalid-user"
    }

    return "valid-user"
}

The same string literal is repeated across multiple branches.

Why Introduce Problems Intentionally?

At this point, it may seem strange to deliberately write bad code.

In a normal development project, we would try to remove these problems before merging the code.

But this project has a different purpose.

If a developer introduces problematic code into the repository, will our CI pipeline detect it automatically?

That means the vulnerable or low quality code becomes part of our test scenario.

Container Image

The same approach is used for the container image.

The Dockerfile intentionally uses an older Go base image:

FROM golang:1.20

Once the application is built, the final image contains more than our own code. It also contains the software and packages provided by the base image.

This gives Trivy something different to analyze from what SonarQube sees.

SonarQube is primarily concerned with the source code, while Trivy examines the resulting container image and looks for known vulnerabilities in its components.

API Test

Before running the security analysis, the pipeline first runs the API test.

The workflow uses the standard Go test command:

go test ./...

The repository includes tests for the API handlers, giving the pipeline a basic verification that the application still behaves as expected.

If the tests fail, the following stages should not proceed.

SonarQube Scan

After the test pass, the pipeline moves to static analysis.

The SonarQube configuration is stored in:

sonar-project.properties

The important part is defining the application source and test locations:

sonar.sources=cmd
sonar.tests=cmd

Because we intentionally introduced problematic code into main.go, this stage gives us an opportunity to verify that the static analysis can actually identify those problems.

For example, the unnecessary nesting in processUser, unreachable statements in unreachableCode, and repeated string literals in duplicatedStrings provide concrete code quality problems for the analyzer to inspect.

Trivy Scan

After the Docker image has been built, Trivy scans the resulting image for known vulnerabilities.

The workflow specifically checks for:

HIGH
CRITICAL

severity vulnerabilities.

The scan is performed against the image that was just built, rather than against an unrelated image.

This is where the intentionally older base image becomes relevant.

Even though the Go application itself is small, the container inherits software from its base image. Trivy can therefore identify vulnerabilities that exist below the application layer.

SonarQube tells us about problems in the code we write. Trivy tells us about vulnerabilities in the artifact we are about to distribute.

There is one important configuration detail in this project.

The Trivy action currently uses:

exit-code: 0

This means the scan reports the detected vulnerabilities but does not fail the workflow because of them.

That is intentional for this demonstration.

We want to see the vulnerabilities produced by our intentionally vulnerable image while still allowing the rest of the workflow to demonstrate the complete delivery process.

This leads to an important distinction that we will revisit later:

A scanner can detect a vulnerability without necessarily enforcing a policy against it

Push the Image

The final stage publishes the container image to GitHub Container Registry.

The workflow is structured so that the push stage depends on the previous stages completing successfully.

The workflow also defines limited permissions for the GitHub Actions job:

permissions:
  contents: read
  packages: write

The workflow needs read access to the repository and write access to GitHub Packages in order to publish the image.

With the image push as the final stage, the pipeline now has a complete path from source code to a published container artifact.

The interesting part is that we can now observe whether the security checks actually detected the intentionally introduced problems.

Result

With everthing setup the scanners now can found the intentionally introduced problems.

SonarQube Result

The SonarQube analysis identified four issues in the Go application.

The most significant finding was in processUser:

Refactor this method to reduce its Cognitive Complexity from 21 to the 15 allowed.

SonarQube assigned this issue a High maintainability impact.

The function has a Cognitive Complexity score of 21, while the configured threshold is 15.

This is a good example of why static analysis is useful.

The application can still compile and the API can still work correctly, but the implementation is unnecessarily difficult to reason about. The CI analysis makes this problem visible without requiring a reviewer to manually inspect every conditional branch.

SonarQube also detected duplicated literals:

Define a constant instead of duplicating this literal "invalid-user" 7 times.

These findings map directly to the intentionally duplicated values in processUser and duplicatedStrings.

Finally, SonarQube identified the unreachable code in unreachableCode function:

Refactor this piece of code to not have any dead code after this "return".

This issue was classified as a Medium reliability issue.

Quality Gate Enforcement

There is another important detail in the workflow.

The SonarQube Quality Gate step is intentionally commented out to allow the pipeline to keep running, as this project is currently focused on detecting code quality issues rather than enforcing them.

# - name: SonarQube quality gate
#   uses: SonarSource/sonarqube-quality-gate-action@cf038b0e0cdecfa9e56c198bbb7d21d751d62c3b # v1.2.0
#   timeout-minutes: 5
#   env:
#     SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
#     SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}

Trivy Results

Trivy analyzed the container image:

secure-ci-pipeline:latest

The image was based on Debian 12.4 and Trivy reported:

1641 vulnerabilities

at the Debian package layer.

Trivy also identified vulnerabilities in the Go binaries contained in the image:

app/app       → 25 vulnerabilities
/usr/local/go/bin/go       → 25 vulnerabilities
/usr/local/go/bin/gofmt    → 25 vulnerabilities
/usr/local/go/pkg/tool/... → 25 vulnerabilities
---------------
---------------

The current Trivy configuration uses:

exit-code: 0

So although Trivy discovered a large number of vulnerabilities, the workflow does not stop because of those findings.

A mature pipeline could eventually evolve from:

Into:

The benefit of transitioning to this approach is that security findings become part of the release decision instead of being treated as information that developers may review later.

Without enforcement, a pipeline can report a critical vulnerability while still allowing the image to be published. This creates a gap between knowing about a problem and preventing the problem from reaching the next stage.

Conclusion

In this project, we addressed the gap described in the introduction by adding security checks directly into the CI workflow.

Instead of only asking whether the application works, the pipeline also checks whether the source code contains quality or reliability problems and whether the resulting container image contains known vulnerabilities.

The key benefit is moving security checks earlier in the delivery process, so problems can be identified before they become production problems.

The code snippets in this article focus on the important parts of the implementation. For the complete source code, you can find it through repository below.

Github Repository

Why Is Your Linux Server Slow? A Practical Guide to Performance Troubleshooting

Mo Rizal — Fri, 21 Aug 2026 03:21:50 +0000

A Linux server can be running without any obvious errors and still feel painfully slow.

Applications may take longer to respond, SSH sessions may become sluggish, users may suddenly experience high latency. The difficult part is that the root cause is not always obvious.

A slow Linux server does not necessarily mean that the CPU is overloaded.

It could be memory pressure, excessive disk I/O, network congestion, too many open file descriptors, or simply a process consuming resources unexpectedly.

The challenge for a DevOps Engineer is not just finding that the server is slow, but identifying why it is slow.

Instead of immediately restarting services or killing processes, we need to investigate the system, layer by layer:

Is the CPU under pressure?
Is the system running out of memory?
Is disk I/O becoming a bottleneck?
Are processes exhausting file descriptors?
Which process is actually responsible for the problem?

This article is a practical introduction to Linux performance troubleshooting using real reproducible scenarios.

Rather than looking at isolated commands, we will build a troubleshooting mindset: observe the symptom, collect evidence, identify the bottleneck, apply the appropriate solution, and verify the result.

To make the investigation reproducible, I created a companion lab repository containing scenarios that intentionally introduce problems:

github repository

High CPU

High CPU usage is one of the most common Linux performance problems.

In this scenario, we intentionally create a process that continuously executes an infinite loop.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./high-cpu/reproduce.sh
./high-cpu/reproduce.sh

The script will display a warning before starting the scenario.

While it is running, open another terminal to investigate the system.

Identify High CPU Usage

The first step is to confirm whether the system is actually experiencing high CPU utilization.

Use top:

top

Look at the CPU summary:

%Cpu(s)

Example result:

%Cpu(s): 95.2 us,  1.8 sy,  0.0 ni,  0.0 id,  2.5 wa, ...

The most important fields are:

us — CPU time spent running user-space processes
sy — CPU time spent running kernel-space processes
id — percentage of CPU time that is idle
wa — CPU time waiting for I/O

If id is consistently very low while us is high, the CPU is heavily utilized by processes.

Identify the Process

The next step is to find the process that consume the most CPU.

Use:

ps aux --sort=-%cpu | head

Example result:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       38690 90.5  0.0   7740  3512 pts/4   R+   02:39   0:16 bash ./reproduce.sh

The most important fields are:

PID — Process ID
%CPU — CPU consumption
%MEM — Memory consumption
STAT — Current process state
COMMAND — Command used to start the process

Here bash ./reproduce.sh process is clearly consuming a significant amount of CPU.

We now have a candidate process, but before terminating it, we should inspect it further.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 38690

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       38690   37782 96 02:39 pts/4    00:00:45 bash ./reproduce.sh

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

Understand the Cause

Inspect the process script:

cat reproduce.sh

The important part is:

while true; do
    :
done

This creates an infinite loop.

The loop continuously executes without any sleep, blocking operation, or other mechanism that would allow the process to stop consuming CPU therefore keeps the CPU busy.

This is the root cause of the high CPU usage in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 38690

Then check the CPU usage again:

top

The process should no longer appear in the process list, and CPU utilization should return to its previous level.

Disk I/O

High disk I/O is another common Linux performance problem.

In this scenario, we intentionally create a process that continuously writes data to disk.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./disk/reproduce.sh
./disk/reproduce.sh

Identify Disk I/O

The first step is to confirm whether the system is actually experiencing high Disk I/O activity.

Use iostat:

iostat -xz 1

Example result:

Device            r/s     w/s   rkB/s   wkB/s  %util
sda              0.00  125.00    0.00  51200.00  98.50

The most important fields are:

r/s — Number of read requests per second
w/s — Number of write requests per second
rkB/s — Data read per second
wkB/s — Data written per second
%util — Percentage of time the device was busy

If %util is consistently close to 100%, the storage device is heavily utilized.

Identify the Process

The next step is to find which process is generating the disk activity.

Use:

pidstat -d 1

Example result:

UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
0     41043      0.00 259548.51      0.00       0  bash

The most important fields are:

PID — Process ID
COMMAND — Command generating the I/O

Here the PID 41043 process is clearly generating significant disk write activity.

Inspect the Process

Use the PID identified from the previous step:

ps -fp 41043

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       41043   37782  0 03:03 pts/4    00:00:00 bash ./reproduce.sh

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is continuously writing data to testfile.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh

The important part is:

while true; do
    dd if=/dev/zero of=testfile bs=1M count=1024 conv=fsync
    rm -f testfile
done

The script continuously creates a file and writes data to it.

The dd command reads zero-filled data from /dev/zero and writes it to testfile.

The conv=fsync option forces dd to flush the written data to the storage device before completing the operation. This makes the scenario generate actual disk write activity instead of relying entirely on the filesystem page cache.

The file is then removed and the process repeats the operation.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 41043

Then check the disk activity again:

iostat -xz 1

You can also verify the processes performing I/O:

iotop

The dd process should no longer appear, disk write throughput should drop significantly, and %util should return to its previous level.

Finally, verify that the test file has been removed:

ls -lh testfile

Memory Pressure

High memory usage is another common Linux performance problem.

In this scenario, we create a process that allocates a large amount of memory and keeps it allocated.

The reproduction script is available in the github repository repository.

Run the reproduction script with a memory target:

chmod +x ./memory-pressure/reproduce.sh
./memory-pressure/reproduce.sh 2G

Replace 2G with the amount of memory you want to allocate.

Identify Memory Pressure

The first step is to confirm whether the system is actually experiencing high memory utilization.

Use:

free -h

Example result:

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       3.1Gi       141Mi       1.0Mi       822Mi       692Mi
Swap:             0B          0B          0B

The most important fields are:

total — Total physical memory available
used — Memory currently used by the system
free — Completely unused memory
buff/cache — Memory used by the kernel for buffers and filesystem cache
available — Memory available for new applications

The available value is particularly important when diagnosing memory pressure. If available memory is consistently very low while used memory is high, the system is under significant memory pressure.

If the system has swap enabled, also pay attention to swap usage. Significant swap usage can indicate that the system does not have enough physical memory to satisfy current workloads.

Identify the Process

The next step is to find which process is consuming the most memory.

Use:

ps aux --sort=-%mem | head

Example result:

USER         PID %CPU %MEM    VSZ    RSS TTY      STAT START   TIME COMMAND
root       39937  0.1 52.7 2123808 2116080 pts/4 S+   02:56   0:03 python3 ./memory_hog.py 2G

The most important fields are:

PID — Process ID
%MEM — Percentage of physical memory used by the process
VSZ — Virtual memory size
RSS — Resident memory currently held in RAM
STAT — Current process state
COMMAND — Command used to start the process

Here the PID 39937 process is clearly consuming a significant amount of memory.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 39937

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       39937   39910  0 02:56 pts/4    00:00:03 python3 ./memory_hog.py 2G

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is running memory_hog.py with a target allocation of 2G.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh

The shell script starts the Python memory allocation program:

python3 "$(dirname "$0")/memory_hog.py" "$MEMORY"

The Python script can then be inspected:

cat memory_hog.py

The important part is:

while allocated_bytes < amount:
    remaining = amount - allocated_bytes
    size = min(chunk_size, remaining)

    chunk = bytearray(size)

    chunk[0] = 1

    allocated.append(chunk)
    allocated_bytes += size

The script allocates memory in 1 MiB chunks until the requested memory amount is reached.

Each allocated chunk is stored in the allocated list:

allocated.append(chunk)

This keeps references to the allocated memory, preventing Python from releasing those objects.

The script then remains running:

while True:
    time.sleep(1)

This large memory allocation is the root cause of the memory pressure in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 39937

Then check the memory usage again:

free -h

You can also verify the processes sorted by memory usage:

ps aux --sort=-%mem | head

The memory_hog.py process should no longer appear in the process list.

The available memory should increase, and overall memory utilization should return closer to its previous level.

File Descriptor Exhaustion

In this scenario, we intentionally create a process that continuously opens file descriptors without closing them.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./file-descriptor/reproduce.sh
./file-descriptor/reproduce.sh

Identify File Descriptor Pressure

The first step is to check the system-wide file descriptor usage.

Use:

cat /proc/sys/fs/file-nr

Example result:

2656    0    9223372036854775807

The values represent:

allocated — Number of allocated file handles
unused — Number of unused allocated file handles
maximum — Maximum number of file handles allowed system-wide

The allocated value represents file handles currently allocated by the kernel.

A high number of allocated file handles can indicate file descriptor pressure, especially when applications are approaching their per-process or system-wide limits.

Identify the Process

The next step is to find the process running fd_exhaustion.py.

Use:

ps aux | grep '[f]d_exhaustion'

Example result:

root       42297  0.2  0.2  18488 10504 pts/4    S+   03:11   0:00 python3 ./fd_exhaustion.py

The most important fields are:

PID — Process ID
%CPU — CPU usage
%MEM — Memory usage
COMMAND — Command used to start the process

Here the python3 ./fd_exhaustion.py process is the candidate responsible for the file descriptor exhaustion.

Use the identified PID for the next step.

Inspect the Process

The next step is to determine how many file descriptors the process currently has open.

Use:

ls /proc/42297/fd | wc -l

Example result:

This indicates that the process currently has 1024 file descriptors open.

You can inspect the actual descriptors with:

ls -l /proc/<PID>/fd

Example:

lr-x------ 1 root root 64 Aug 21 03:11 0 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 1 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 2 -> /dev/pts/4
lr-x------ 1 root root 64 Aug 21 03:11 3 -> /dev/null
lr-x------ 1 root root 64 Aug 21 03:11 4 -> /dev/null

The large number of /dev/null entries indicates that the process is continuously opening new file descriptors.

Next, check the process limits:

cat /proc/42297/limits | grep "open files"

Output:

Max open files            1024                 1048576                files

The first value is the soft limit, while the second value is the hard limit.

The soft limit is the limit currently enforced for the process.

Once the process reaches this limit, attempts to open additional file descriptors will fail.

Understand the Cause

Inspect the reproduction script:

cat fd_exhaustion.py

The important part is:

while True:
    try:
        fd = os.open(
            "/dev/null",
            os.O_RDONLY
        )

        FILES.append(fd)

        counter += 1

The script continuously opens /dev/null using os.open().

Every returned file descriptor is stored in the FILES list:

FILES.append(fd)

The descriptors are intentionally not closed while the scenario is running. As a result, the number of open file descriptors continuously increases

Eventually, os.open() fails because the process has reached its file descriptor limit.

The script then reports the error:

Failed to open new file descriptor
[Errno 24] Too many open files

This is the root cause of the file descriptor exhaustion in this scenario.

File descriptor exhaustion can have broader consequences for real applications. Processes may fail to open files, accept new connections, create sockets, or perform other operations that require file descriptors.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 42297

Verify that the process has stopped:

ps -p 42297

If the process has stopped, the command should return no process entry.

You can also verify that the process no longer exists:

test -d /proc/<PID> && echo "Process still exists" || echo "Process stopped"

The file descriptors belonging to the process are automatically released by the kernel when the process terminates.

Therefore, /proc/<PID>/fd will no longer be available after the process has exited.

Finally, verify the system-wide file descriptor state again:

cat /proc/sys/fs/file-nr

The allocated file handle count should return closer to its previous level after the exhausted process has been terminated.

Network Connection Exhaustion

In this scenario, we intentionally create a process that continuously opens TCP connections to a local server and keeps those connections open.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./network-connection/reproduce.sh
./network-connection/reproduce.sh

The script will display a warning before starting the scenario.

The reproduction uses 127.0.0.1:9000 as the target, so all connections remain on the local machine.

Identify Abnormal Connections

The first step is to confirm whether the system is experiencing an unusually large number of TCP connections.

Use ss:

ss -s

Example result:

Total: 1252
TCP:   2051 (estab 1023, closed 1021, orphaned 1, timewait 1020)

Transport Total     IP        IPv6
RAW       1         0         1
UDP       3         3         0
TCP       1030      1028      2
INET      1034      1031      3
FRAG      0         0         0

The most important fields are:

estab — Established TCP connections
orphaned — Orphaned TCP connections
timewait — Connections waiting to be fully closed
TCP — Total TCP sockets

A significant increase in established connections can indicate abnormal connection usage.

In this scenario, the large number of estab connections is the main indicator that something is continuously creating and maintaining TCP connections.

Identify the Connection and Process

The next step is to identify which process owns the connections.

Use:

ss -tanp

Example result:

CLOSE-WAIT  1  0  127.0.0.1:46374  127.0.0.1:9000  users:(("python3",pid=46938,fd=880))
CLOSE-WAIT  1  0  127.0.0.1:39120  127.0.0.1:9000  users:(("python3",pid=46938,fd=86))
CLOSE-WAIT  1  0  127.0.0.1:44192  127.0.0.1:9000  users:(("python3",pid=46938,fd=634))

The most important information is:

State — Current TCP connection state
Local Address:Port — Local endpoint
Peer Address:Port — Remote endpoint
users — Process owning the socket
pid — Process ID
fd — File descriptor associated with the socket

Here we can identify the process responsible for the connections:

users:(("python3",pid=46938,fd=880))

The PID is:

The same PID appearing across many connections is a strong indication that one process is responsible for creating the connection buildup.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 46938

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       46938   46922  0 03:42 pts/1    00:00:00 python3 ./connection_exhaustion.py 127.0.0.1 9000

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is running connection_exhaustion.py and connecting to 127.0.0.1:9000.

Next, check how many file descriptors the process currently has open:

ls /proc/46938/fd | wc -l

A large number of file descriptors is expected because every TCP socket consumes a file descriptor.

You can inspect the descriptors directly:

ls -l /proc/46938/fd | head

You should see entries pointing to sockets, for example:

lrwx------ 1 root root 64 Aug 21 03:42 86 -> 'socket:[123456]'
lrwx------ 1 root root 64 Aug 21 03:42 87 -> 'socket:[123457]'
lrwx------ 1 root root 64 Aug 21 03:42 88 -> 'socket:[123458]'

This confirms that the process is holding a large number of socket file descriptors.

Understand the Cause

Inspect the reproduction script:

cat connection_exhaustion.py

The important part is:

while True:
    try:
        sock = socket.socket(
            socket.AF_INET,
            socket.SOCK_STREAM
        )

        sock.connect((host, port))

        CONNECTIONS.append(sock)

        counter += 1

The script continuously creates TCP sockets and connects them to the target server.

Every successful connection is stored in the CONNECTIONS list:

CONNECTIONS.append(sock)

The sockets are intentionally not closed while the scenario is running. As a result, the number of active connections continuously increases:

The TCP server is also intentionally designed to keep accepted connections open:

connection, address = server.accept()
connections.append(connection)

This means the server does not immediately close the connections created by the client. Eventually, the client process can reach its file descriptor limit and fail to create additional sockets.

This pattern can occur in real applications when connections are not properly closed, connection pools are misconfigured, or an application continuously creates new connections instead of reusing existing ones.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 46938

Then check the network connection summary again:

ss -s

You can also check the connections to port 9000 directly:

ss -tan | grep ':9000'

The number of established TCP connections should return to its previous level.

Conclusion

Linux performance troubleshooting is not about finding a single command that tells you why a server is slow.

It is about building a structured investigation from the symptoms you observe.

In the scenarios covered in this article, we investigated several different types of Linux performance problems:

High CPU caused by an infinite loop
High disk I/O caused by continuous writes
Memory pressure caused by excessive memory allocation
File descriptor exhaustion caused by descriptors that are never closed
Network connection exhaustion caused by continuously opened TCP connections

Although the symptoms are different, the troubleshooting approach remains consistent:

The important lesson is to avoid making assumptions based on a single metric.

High CPU does not immediately tell you which process is responsible. High memory usage does not necessarily mean the system is running out of usable memory. A large number of TCP connections does not automatically mean the network is slow.

Each symptom needs to be investigated with the right evidence.

When a production server becomes slow, the most valuable skill is not knowing how to restart it.

It is being able to answer:

What is happening, why is it happening, and how can I prove that my fix actually worked?

The scenarios in this article are intentionally simplified, but the troubleshooting methodology can be applied to much more complex production incidents.

You can reproduce all of these scenarios and investigate yourself by cloning my github repository bellow:

https://github.com/muhammadyulasfipahrizal/linux-performance-lab.git

7 Bash Scripts Every Devops Engineer Should Have

Mo Rizal — Tue, 18 Aug 2026 03:58:05 +0000

A production server rarely fails in a convenient way.

A disk suddenly fills up. A process consumes all available CPU. A network connection becomes unreliable, or an SSH account needs to be investigated after a suspicious login.

When this happens, a DevOps engineer usually does not start by writing a new monitoring system. The first response is often much simpler: connect to the server and investigate what is happening.

The problem is that the investigation itself can become repetitive.

Running df, du, ps, ss, journalctl, checking authentication logs, and collecting system information manually may seem trivial on one server. But during an incident, repeatedly remembering which commands to run, in which order, and how to collect the results wastes time and makes investigations inconsistent.

This is where Bash becomes useful.

Instead of treating Bash as a language for small one line commands, we can use it to turn recurring operational procedures into reusable tools. The goal is not to replace observability platforms or configuration management systems. The goal is to create lightweight scripts that can be executed directly on a server when an engineer needs fast, consistent information.

In this project, we will build a collection of 7 practical bash scripts for common DevOps operations:

Investigating disk usage
Collecting incident context
Inspecting log activity within a specific time window
Investigating network paths
Auditing running processes
Auditing SSH access
Cleaning up unnecessary system resources

Each script is designed around a real operational problem.

By the end of this project, we will have a small Bash toolkit that can turn common server investigations from a collection of ad-hoc commands into repeatable operational workflows.

The code snippets in this article focus on the important parts of the implementation. For the complete Bash Scripts you can find the full source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/terraform-setup

1. Disk Investigator

disk-investigator.sh is a filesystem investigation script designed to answer a common question during Linux server incidents:

What is consuming the disk, and where should I investigate next?

Instead of running several commands manually, the script combines filesystem, inode, directory, file, and modification-time information into a single investigation report.

The script accepts three parameters:

--path controls which directory or filesystem is investigated.
--top controls how many results are displayed.
--min-size defines what qualifies as a large file

The script starts with the filesystem itself. It reports the filesystem type and basic capacity information, then checks inode consumption. This distinction is important because a server can run out of inodes even when there is still available disk space.

It then moves from a broad view to more specific information:

Filesystem capacity and inode usage
Largest directories
Largest files
Large files modified within the last 24 hours

2. Incident Context

incident-context.sh collects the current state of a Linux server into a single incident report.

During an incident, the first challenge is often not finding the exact root cause immediately. It is establishing what the server looked like when the problem occurred.

Instead of manually running commands such as uptime, free, df, ps, ip, ss, systemctl, and journalctl, this script collects those signals together and presents them as one structured snapshot.

The report is organized around the main areas an engineer would typically investigate:

The script also introduces an important operational concept: time-bounded investigation.

By default, it collects journal and kernel information from the last 30 minutes, but the window can be changed with --since, for example:

sudo ./incident-context.sh --since "2 hours ago"

This prevents the report from becoming an uncontrolled dump of historical logs. During an incident, the most useful information is usually the information surrounding the period in which the failure occurred.

The script can also write the collected information to a report file:

sudo ./incident-context.sh --output incident-report.txt

The result is a reusable incident snapshot: instead of starting an investigation with an empty terminal and a list of commands to remember, the engineer can generate a consistent baseline of the server's state and use it as the starting point for deeper investigation.

3. Log Window

log-window.sh is a focused log investigation tool for extracting only the log entries that are relevant to an incident.

When an application produces thousands of log lines, searching the entire file manually can make an investigation unnecessarily difficult. Usually, an engineer already has some context about the problem:

when the problem happened,
what severity the event had, or
a phrase associated with the failure.

Instead of reading the entire log file, this script allows those clues to become filters.

The script supports four independent filters:

These filters can be combined like this:

./log-window.sh --file ../logs/dummy.log \
--level WARN \
--contains "Database response time increased" \
--start "2026-08-18 10:07:00"  --end "2026-08-19 10:09:00"

4. Network Path

network-path.sh is a network troubleshooting script that checks connectivity to a destination layer by layer, instead of treating connectivity as a single yes-or-no question.

A connection failure can happen at different points. DNS might fail to resolve the hostname, the routing table might not provide a valid path, the TCP port might be unreachable, or an application-layer problem might occur after the connection succeeds.

If DNS fails, there is no reason to continue testing TCP. If DNS and routing succeed but TCP fails, the investigation can focus on connectivity, firewall rules, security groups, or the destination service rather than the application itself.

The script therefore produces a more useful result than simply:

Connection failed

It can identify a failure boundary such as:

DNS      PASS
Route    PASS
TCP      FAIL
TLS      SKIP
HTTP     SKIP

Failure detected at: TCP

The script requires a destination host and TCP port:

./network-path.sh \
    --host google.com \
    --port 443 \
    --https

5. Process Audit

process-audit.sh provides a deeper view of how running processes are consuming system resources.

Commands such as top and ps are excellent for quickly identifying CPU or memory usage, but during troubleshooting we often need more context. A process consuming resources may also have an unusually high number of file descriptors, threads, context switches, or a large resident memory footprint.

This script collects those signals for every accessible process and turns them into a structured resource audit.

The script can rank processes by different resource dimensions:

sudo ./process-audit.sh --sort cpu
sudo ./process-audit.sh --sort memory
sudo ./process-audit.sh --sort fd
sudo ./process-audit.sh --sort threads

6. SSH Access Audit

ssh-access-audit.sh is a read-only security audit tool for examining how users can access a Linux server through SSH and what privileges they have after gaining access.

SSH is often the primary entry point for administrators and automation. Because of that, troubleshooting SSH access should not only ask:

Is SSH running?

It should also answer:

Who can access the server, how can they authenticate, what keys are authorized, and what privileges do those users have?

The script approaches SSH access from several layers:

The script evaluates the effective SSH configuration using sshd -T, covering authentication settings such as root login, password authentication, public-key authentication, and keyboard-interactive authentication.

Finally, findings are classified as HIGH, MEDIUM, or INFO so critical security issues can be prioritized.

7. System Cleanup

system-cleanup.sh is a safe filesystem cleanup tool that identifies files that may be consuming unnecessary disk space and removes them only after explicit confirmation.

Disk cleanup is inherently risky because deleting the wrong file can cause service failures or data loss. Instead of immediately running destructive commands such as rm -rf, the script separates cleanup into two phases:

It looks for cleanup candidates in common locations:

/tmp
/var/tmp
APT package cache
Old compressed logs
Systemd journal usage

With this script, engineers don't need to manually search for and delete files, reducing the risk of accidentally removing important data. The script only deletes discovered candidates from predefined safe locations after explicit confirmation.

Conclusion

These 7 bash scripts turn common Linux administration tasks into a practical operational toolkit.

Each script provides a focused capability:

Disk Investigator helps identify what is consuming storage.
Incident Context provides a structured view of the server state.
Log Window makes relevant events easier to isolate.
Network Path helps pinpoint where connectivity breaks.
Process Audit reveals which processes are consuming system resources.
SSH Access Audit exposes potential access and privilege risks.
System Cleanup makes disk cleanup safer and more controlled.

The biggest benefit comes from using them together. Instead of relying on individual commands and ad-hoc procedures, engineers have a consistent set of tools for investigation, diagnosis, security auditing, and cleanup.

More importantly, these scripts can become building blocks for larger operational practices. They can be extended with additional checks, integrated into automation, or adapted to the specific standards of an organization's infrastructure.

A command that you run once is useful.

A well designed script that consistently performs the same set of commands is an operational asset.

You can find the source code for this article in my github repository:
Github Repository

Building AWS Infrastructure The Right Way With Terraform

Mo Rizal — Sat, 15 Aug 2026 07:10:44 +0000

Provisioning AWS infrastructure is easy when there are only a few resources.

Open the AWS Console, create a VPC, add Subnets, configure Security Group, launch EC2 instance, etc.

The problem starts when the infrastructure grows.

A few manually created resources can quickly become dozens of resources with different configurations. One environment may have different network settings, another may contain a slightly different security rule.

Over time, the infrastructure becomes difficult to understand and even harder to reproduce.

The problem is not only how to create AWS resources, but how to structure them so the infrastructure does not become a mess.

This is where Terraform becomes valuable.

Instead of managing infrastructure through the AWS Console, we can define it as code, store it in Git, review changes, and reproduce the same architecture across environments.

In this project, we will build a small production style AWS setup using Terraform.

This project will include:

VPC
Public and private subnets
Internet gateway
Route tables
Security group
EC2 instance
IAM Role
S3 bucket
Terraform remote state

But the infrastructure itself is only part of the project.

The main goal is to apply several Terraform design principles that become increasingly important as infrastructure grows:

Modularization
Environment Separation
Variables Instead of Hardcoding
Outputs
State Management

The goal is simple:

Instead of managing AWS infrastructure manually, define it as reusable Terraform code that can be reviewed, reproduced, and maintained.

What We Are Building

Before writing Terraform code, let's define the architecture we want to build.

The final environment contains a VPC with public and private subnets. The EC2 instance runs in the public subnet so we can connect to it for validation.

The private subnet is included to demonstrate basic network separation and provide a foundation for expanding the architecture later if needed.

Final Architecture

The EC2 instance will be placed in the public subnet while the private subnet is included to demonstrate a basic network separation pattern

The network uses:

VPC              10.0.0.0/16
Public Subnet    10.0.1.0/24
Private Subnet   10.0.2.0/24

The EC2 instance is also associated with an IAM role so workloads can interact with AWS services without storing long lived AWS credentials on the server.

Terraform Architecture

The Terraform configuration is divided into three layers:

Bootstrap

The bootstrap directory is responsible for creating the resources required before the main Terraform environments can use remote state.

In this project, that means creating the S3 bucket used for Terraform state.

Environments

The environments directory contains environment specific configurations.

The environments are not responsible for implementing every AWS resource themselves. Instead, they will compose reusable modules.

For example, the dev environment can use:

The same modules can then be reused by the prod environment with different variables.

This allows the same infrastructure modules to be reused with different values for development and production.

Modules

The modules directory contains reusable infrastructure components.

Each module has a specific responsibility.

For example:

modules/vpc/

contains the logic for creating the VPC and its networking components.

Meanwhile:

modules/ec2/

contains the logic for creating the EC2 instance.

Project Structure

The module files follow Terraform conventional structure:

main.tf — resources
variables.tf — inputs
outputs.tf — values exposed for othe configurations

The environment directories additionally contain:

backend.tf — remote state configuration
terraform.tfvars— environment values

The rest of the article will use the dev environment to provision the infrastructure.

The code snippets in this article focus on the important parts of the implementation. For the complete Terraform configuration you can find the full source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/terraform-setup

Configure Remote State

Before provisioning the AWS infrastructure, we need to solve one problem

Where should Terraform store its state?

Terraform uses a state file to keep track of the infrastructure it manages. By default, this state is stored locally as:

terraform.tfstate

For a small personal project, local state may be enough.

However, once infrastructure is shared between environments, machines, and engineers, relying on a local state file becomes difficult to manage.

A better approach is to store the state remotely in AWS S3.

In this project, we use a separate bootstrap configuration to create the S3 bucket that will later be used as the Terraform backend.

Bootstrap

The purpose of this configuration is to create the S3 bucket required by the Terraform environments.

The bootstrap configuration can be initialized and applied independently:

cd bootstrap

terraform init

terraform plan

terraform apply

After the apply completes, the S3 bucket required for remote state exists.

We can now configure the dev environment to use it.

Configure the Dev Backend

The dev environment contains a backend.tf file:

The backend configuration tells Terraform where its state should be stored.

terraform {
  backend "s3" {
    bucket = "terraform-right-way"
    key    = "dev/terraform.tfstate"
    region = "ap-southeast-3"
  }
}

The important part is the key:

key = "dev/terraform.tfstate"

This allows different environments to maintain separate state locations within the same bucket.

Initialize the Dev Environment

Now move into the dev environment:

cd environments/dev

Then run:

terraform init

Terraform reads backend.tf and configures the S3 backend.

If Terraform was previously using local state, it may ask whether the existing state should be migrated to the new backend.

For a new environment with no existing infrastructure, there is normally no local state to migrate.

After initialization, Terraform is ready to use the S3 backend.

Why Bootstrap Is Separate

There is a small dependency problem when using S3 as the Terraform backend:

We solve this by creating the state bucket through a separate bootstrap configuration.

VPC

The first infrastructure component is the VPC, which provides the network boundary for our AWS resources.

VPC              10.0.0.0/16
Public Subnet    10.0.1.0/24
Private Subnet   10.0.2.0/24

The module creates the VPC, subnets, Internet Gateway, and route tables.

The public subnet is associated with a route table that sends Internet bound traffic through the Internet Gateway. The private subnet does not have a direct Internet route.

The module exposes the values required by other parts of the infrastructure:

output "vpc_id" {
  value = aws_vpc.this.id
}

output "public_subnet_id" {
  value = aws_subnet.public.id
}

output "private_subnet_id" {
  value = aws_subnet.private.id
}

The dev environment then consumes the module:

module "vpc" {
  source = "../../modules/vpc"

  vpc_cidr            = var.vpc_cidr
  public_subnet_cidr  = var.public_subnet_cidr
  private_subnet_cidr = var.private_subnet_cidr
  availability_zone   = var.availability_zone
}

The important design decision is that the CIDR ranges are variables, rather than being hardcoded directly into the environment's resources.

Create the Security Group

The Security Group will be attached to the EC2 instance and will define the allowed inbound traffic.

The main rule we need is SSH access to the EC2 instance.

Instead of hardcoding an IP address inside the module, the allowed SSH source is provided as a variable:

variable "allowed_ssh_cidr" {
  description = "CIDR block allowed to access SSH"
  type        = string
}

The Security Group can then use that value for its SSH rule:

ingress {
  description = "Allow SSH"
  protocol    = "tcp"
  from_port   = 22
  to_port     = 22
  cidr_blocks = [var.allowed_ssh_cidr]
}

This is an important design decision.

We don't want the module to assume that SSH should always be accessible from a particular IP address. The module defines the rule, while the environment decides who should be allowed to use it.

For example, the dev environment can provide:

allowed_ssh_cidr = "YOUR_IP/32"

Using /32 limits SSH access to a single public IP address.

The Security Group also needs to allow outbound traffic so the EC2 instance can communicate with external services when required.

Once created, the module exposes the Security Group ID:

output "security_group_id" {
  value = aws_security_group.this.id
}

The dev environment can then pass the output to the EC2 module:

module "ec2" {
  source = "../../modules/ec2"

  subnet_id         = module.vpc.public_subnet_id
  security_group_id = module.security_group.security_group_id
}

This creates another simple dependency between our modules without coupling their implementations.

Deploy EC2

The modules/ec2/ configuration provides the values required by the module, such as the AMI, instance type, subnet, Security Group, and SSH key.

The module then connects the EC2 instance to the infrastructure we created earlier:

module "ec2" {
  source = "../../modules/ec2"

  subnet_id         = module.vpc.public_subnet_id
  security_group_id = module.security_group.security_group_id

  ami_id            = var.ami_id
  instance_type     = var.instance_type
  key_name          = var.key_name
}

The important part here is how the dependencies are connected.

The EC2 instance does not need to know how the VPC or Security Group is implemented. It only consumes their outputs:

module.vpc.public_subnet_id
module.security_group.security_group_id

The actual EC2 resource remains inside the module:

resource "aws_instance" "this" {
  ami                    = var.ami_id
  instance_type          = var.instance_type
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [var.security_group_id]
  key_name               = var.key_name

  associate_public_ip_address = true

  tags = {
    Name = var.instance_name
  }
}

The instance is placed in the public subnet and receives a public IP so that we can connect to it and validate the infrastructure after deployment.

The values that can vary between environments are kept outside the module.

For example:

instance_type = "t3.micro"

The same EC2 module can later be used by production with a different instance type without changing the module itself.

The module also exposes useful information through outputs:

output "instance_id" {
  value = aws_instance.this.id
}

output "public_ip" {
  value = aws_instance.this.public_ip
}

This allows Terraform to display the instance information after deployment and gives other parts of the configuration a clean interface to consume it.

Create the IAM Role

The EC2 instance may need to interact with other AWS services. A common but unsafe approach would be to store AWS access keys directly on the server.

Instead, AWS provides IAM roles for EC2, allowing applications running on the instance to obtain temporary credentials automatically.

The module creates an IAM role with an EC2 trust policy:

resource "aws_iam_role" "this" {
  name = var.role_name

  assume_role_policy = jsonencode({
    Version = "2012-10-17"

    Statement = [
      {
        Effect = "Allow"

        Principal = {
          Service = "ec2.amazonaws.com"
        }

        Action = "sts:AssumeRole"
      }
    ]
  })
}

The important part is the trust relationship:

This allows the EC2 service to assume the role on behalf of the instance.

The role is then associated with an instance profile:

resource "aws_iam_instance_profile" "this" {
  name = var.instance_profile_name
  role = aws_iam_role.this.name
}

The instance profile is what allows the IAM role to be attached to the EC2 instance.

The EC2 module can then receive the instance profile:

resource "aws_instance" "this" {
  iam_instance_profile = var.iam_instance_profile
}

This keeps AWS credentials out of the server's configuration.

For this project, the role does not need broad permissions simply because the EC2 instance exists. Permissions should be added according to the actual AWS operations the workload requires.

Create the S3 Bucket

It is important to distinguish this bucket from the S3 bucket created during the bootstrap stage.

The bootstrap bucket is used by Terraform to store remote state, while this S3 bucket is part of the infrastructure managed by the dev environment.

resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name

  tags = {
    Name = var.bucket_name
  }
}

The bucket name is provided by the environment rather than being hardcoded inside the module.

bucket_name = "dev-server-bucket"

The module exposes the bucket information through outputs:

output "bucket_id" {
  value = aws_s3_bucket.this.id
}

output "bucket_arn" {
  value = aws_s3_bucket.this.arn
}

The dev environment can then consume the module:

module "s3" {
  source = "../../modules/s3"

  bucket_name = var.bucket_name
}

This follows the same pattern used by the other infrastructure components: the module contains the implementation, while the environment provides the configuration.

Validate and Deploy

We have now defined the infrastructure as Terraform code. Before creating anything in AWS, we should validate the configuration and review the changes Terraform intends to make.

For this project, we will deploy the dev environment from:

environments/dev/

Format the Configuration

cd environments/dev

terraform fmt -recursive

This keeps the Terraform configuration consistently formatted.

Validate the Configuration

terraform validate

Terraform checks whether the configuration is syntactically valid and whether the configuration can be successfully loaded.

A successful validation should return:

Success! The configuration is valid.

Review the Execution Plan

terraform plan

Terraform compares the desired configuration with the current state and shows which resources it intends to create, change, or destroy.

For a new development environment, we should expect Terraform to plan the creation of our infrastructure.

This step is important because terraform plan gives us an opportunity to review the changes before they are applied.

Apply the Infrastructure

Once the plan looks correct, apply it:

terraform apply

Terraform will display the execution plan again and ask for confirmation.

Enter:

yes

Terraform will then create the resources defined by the dev environment.

Why This Structure Matters

At first, this structure may look more complicated than putting everything into one main.tf.

For a small project, that may be true.

The benefit becomes clearer when the infrastructure grows.

A new environment can reuse the existing modules:

dev  → modules
prod → modules

Instead of duplicating the VPC, EC2, IAM, and S3 implementation, each environment provides its own configuration.

If the network implementation changes, we know where to look.
If production needs a larger EC2 instance, we change the production configuration rather than duplicating the EC2 module.
If another environment is introduced, it can reuse the existing modules.

Conclusion

Terraform is not difficult because creating an AWS resource is complicated.

The real challenge is managing infrastructure as it grows.

A single main.tf can work for a small experiment, but infrastructure becomes harder to maintain when environments, networking, compute, IAM, storage, and state management are all mixed together.

In this project, we applied five core concepts:

Modularization — infrastructure was divided into reusable modules for networking, security, compute, IAM, and storage instead of keeping everything in a single configuration.
Environment Separation — development and production have their own configurations while sharing the same reusable modules.
Variables Instead of Hardcoding — values such as CIDR ranges, instance types, AMIs, SSH access, and resource names are provided through variables.
Outputs — modules expose the values that other parts of the infrastructure need, allowing resources to be connected without tightly coupling their implementations.
State Management — Terraform state is stored remotely in S3, with separate state paths for each environment.

These concepts may seem unnecessary for a small infrastructure project. However, they become increasingly important as the number of resources and environments grows.

You can find the source code for this article in my github repository:

Github Repository

I built a zero touch ubuntu server provisioning workflow with ansible

Mo Rizal — Tue, 11 Aug 2026 04:09:47 +0000

Mo Rizal

Aug 11

Zero-Touch Provisioning With Ansible: From 0 To Secure Production Ready Server

16 min read

Zero-Touch Provisioning With Ansible: From 0 To Secure Production Ready Server

Mo Rizal — Tue, 11 Aug 2026 04:09:01 +0000

Provisioning a new server is often more complicated than simply installing a few packages.

A fresh ubuntu server may start with almost nothing configured. An engineer still needs to create users, configure SSH access, set the hostname configure the firewall, and make sure the required services are running correctly.

When these steps are performed manually, the process can become time-consuming and inconsistent.

One server might have slightly different configurations from another. A firewall rule might be forgotten, SSH password authentication might remain enabled, or a monitoring agent might not be installed at all.

The problem is not only how to configure a server, but how to configure every server consistently.

This is where infrastructure automation become valuable.

In this article, we will build a zero-touch server provisioning workflow using ansible, starting from a fresh Ubuntu server and automatically transforming it into a standardized and secured server environment.

The provisioning process covers:

Bootstrapping and configuring the server
Applying system security hardening
Installing and configuring docker
Setting up system monitoring
Restricting network access

The goal is simple:

Start with a fresh ubuntu server, run the ansible playbook, and let ansible handle the rest

By the end of the process, the server will be ready to run applications, protected with a basic security baseline, equipped with Docker, and connected to the monitoring infrastructure.

Provisioning Flow

Before looking at the ansible configuration, it is important to understand what we are actually building.

The goal of this project is not to create a collection of independent Ansible playbooks. The goal is to create a repeatable provisioning workflow that can take a fresh Ubuntu server through several configuration stages until it reaches standardized state.

The provisioning flow looks like this:

Project Structure

Instead of putting the entire provisioning process into a single playbook, the project separates configuration into several files, each with a specific responsibility.

The easiest way to understand the architecture is to follow how ansible processes a command.

When we run:

ansible-playbook site.yml

Ansible first loads its configuration from ansible.cfg

[defaults]
inventory = inventory/hosts.ini
interpreter_python = auto_silent

This configuration tells Ansible where to find the inventory

Then inventory/hosts.ini tells ansible which servers it should manage and how to connect to them.

[servers]
vm1 ansible_host=192.168.122.68

[servers:vars]
ansible_user=root
# ansible_user=devops
ansible_ssh_private_key_file=~/.ssh/ansible

The servers group is important because the playbooks target this group:

hosts: servers

Ansible can therefore apply the same configuration to every server belonging to that group.

group_vars/servers.yml

This file contains variables that apply to the entire servers group:

server_timezone: Asia/Makassar
admin_user: devops
server_hostname: ansible-demo
server_ssh_public_key: YOUR_SSH_PUBLIC_KEY
monitoring_server_ip: 192.168.122.1
node_exporter_version: 1.9.1
node_exporter_arch: linux-amd64

This allows the playbooks to focus on what should be configured, while server-specific values remain outside the playbooks.

For example, instead of writing:

name: devops

inside a playbook, we use:

name: "{{ admin_user }}"

The Main Entry Point

The entire provisioning process is orchestrated through site.yml:

- name: bootstrap server
  import_playbook: playbooks/bootstrap.yml

- name: configure base system
  import_playbook: playbooks/base.yml

- name: harden server
  import_playbook: playbooks/security.yml

- name: install docker
  import_playbook: playbooks/docker.yml

- name: configure monitoring
  import_playbook: playbooks/monitoring.yml

This file does not contain the actual server configuration tasks. Instead, it defines the order in which the provisioning stages are executed.

1. Bootstrap Playbook

At the start, the server does not have the administrative user that we want to use for day to day management. The initial connection therefore uses the existing root account.

The purpose of the bootstrap stage is to establish the foundation required for the rest of the provisioning process.

The bootstrap playbook performs four main tasks:

Creates the administrative user
Grants the user sudo privileges
Configures SSH key-based authentication
Sets the server hostname

The important part of this stage is that root access is only required for the initial bootstrap.

Once the devops user has been created and configured, subsequent playbooks connect to the server using that dedicated administrative account.

The playbook looks like this:

- name: bootstrap fresh ubuntu server
  hosts: servers

  tasks:
    - name: create admin user
      ansible.builtin.user:
        name: "{{ admin_user }}"
        shell: /bin/bash
        create_home: true
        groups: sudo
        append: true

    - name: configure passwordless sudo
      ansible.builtin.copy:
        content: "{{ admin_user }} ALL=(ALL) NOPASSWD:ALL\n"
        dest: "/etc/sudoers.d/{{ admin_user }}"
        owner: root
        group: root
        mode: '0440'

    - name: configure ssh access for admin user
      ansible.posix.authorized_key:
        user: "{{ admin_user }}"
        key: "{{ server_ssh_public_key }}"

    - name: set server hostname
      ansible.builtin.hostname:
        name: "{{ server_hostname }}"

Creating the Administrative User

- name: create admin user
      ansible.builtin.user:
        name: "{{ admin_user }}"
        shell: /bin/bash
        create_home: true
        groups: sudo
        append: true

The username comes from the variable defined in group_vars/servers.yml:

admin_user: devops

This means the playbook does not depend on a specific username.

Ansible creates the user's home directory and adds the user to the sudo group:

groups: sudo
append: true

The append: true option is important because it adds the user to the specified group without removing the user from other existing groups.

At the end of this task, the server has a dedicated administrative identity that can be used for subsequent configuration.

Configuring Passwordless Sudo

- name: configure passwordless sudo
      ansible.builtin.copy:
        content: "{{ admin_user }} ALL=(ALL) NOPASSWD:ALL\n"
        dest: "/etc/sudoers.d/{{ admin_user }}"
        owner: root
        group: root
        mode: '0440'

Instead of modifying the main /etc/sudoers file, the configuration is placed inside:

/etc/sudoers.d/

This keeps the custom sudo configuration separated from the system's main sudo configuration.

The resulting file is:

/etc/sudoers.d/devops

with permissions:

0440

The user can now execute administrative commands through sudo without being prompted for a password.

This is useful for automated provisioning because subsequent ansible tasks need to perform privileged operations.

Configuring SSH Key Authentication

- name: configure ssh access for admin user
      ansible.posix.authorized_key:
        user: "{{ admin_user }}"
        key: "{{ server_ssh_public_key }}"

The public key is provided through:

server_ssh_public_key: YOUR_SSH_PUBLIC_KEY

Ansible manages the user's authorized_keys file instead of requiring us to manually create it.

Setting the Hostname

- name: set server hostname
  ansible.builtin.hostname:
    name: "{{ server_hostname }}"

A predictable hostname is useful for identifying the server through monitoring systems, logs, and other infrastructure tooling.

2. Base Playbook

After the bootstrap stage, the server now has a dedicated administrative user and the basic access configuration required for ansible to continue provisioning it.

The next step is to establish a consistent baseline for the operating system.

A fresh Ubuntu installation may contain different package versions, system settings, or filesystem directories depending on how the server was created. Before installing infrastructure components, it is useful to bring the system into a predictable state.

The base.yml playbook is responsible for:

Updating the APT package cache
Upgrading installed packages
Configuring the server timezone
Creating required application directories

The playbook looks like this:

- name: configure base ubuntu server
  hosts: servers
  become: true

  tasks:
    - name: update apt package cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600

    - name: upgrade installed packages
      ansible.builtin.apt:
        upgrade: dist

    - name: configure server timezone
      community.general.timezone:
        name: "{{ server_timezone }}"

    - name: create standard application directories
      ansible.builtin.file:
        path: "/home/{{ admin_user }}/{{ item }}"
        state: directory
        owner: "{{ admin_user }}"
        group: "{{ admin_user }}"
        mode: '0755'
      loop:
        - apps
        - monitoring

Updating the APT Package Cache

- name: update apt package cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600

The update_cache option performs the equivalent of:

apt update

The cache_valid_time prevents ansible from unnecessarily updating the package cache on every execution when the cache is still considered valid.

This is useful when the playbook contains multiple package related tasks because it reduces unnecessary repository requests.

Upgrading the System

- name: upgrade installed packages
      ansible.builtin.apt:
        upgrade: dist

This is equivalent to performing a distribution-level package upgrade through APT.

The purpose is to ensure that the server starts the rest of the provisioning process with its existing packages brought up to date.

This also establishes a more predictable baseline before we install additional components such as docker and monitoring agents.

Configuring the Timezone

- name: configure server timezone
      community.general.timezone:
        name: "{{ server_timezone }}"

Consistent timezone configuration is particularly useful for infrastructure because timestamps appear throughout:

System logs
Application logs
Monitoring data
Scheduled jobs
Troubleshooting sessions

A consistent timezone makes these timestamps easier to correlate when investigating an issue.

Creating the Application Directory

- name: create standard application directories
      ansible.builtin.file:
        path: "/home/{{ admin_user }}/{{ item }}"
        state: directory
        owner: "{{ admin_user }}"
        group: "{{ admin_user }}"
        mode: '0755'
      loop:
        - apps
        - monitoring

Rather than allowing application files to be placed arbitrarily throughout the filesystem, the provisioning process establishes a predictable location:

/home/devops/apps
/home/devops/monitoring

The directory is owned by the administrative user defined in our variables:

admin_user: devops

This gives us a standardized location that can later be used when deploying applications or additional infrastructure components.

3. Security Playbook

The server now has a predictable operating system baseline.

However, a freshly provisioned server should not be considered secure simply because its packages are up to date. We still need to reduce unnecessary access, protect the SSH service, control network traffic, and add a mechanism to respond to repeated authentication attempts.

This is the responsibility of the security.yml playbook.

The security playbook focuses on four areas:

Hardening SSH access
Configuring UFW as the host firewall
Allowing only the required network traffic
Installing and enabling Fail2ban

The playbook looks like this:

- name: harden ubuntu server
  hosts: servers
  become: true

  tasks:
    - name: disable root SSH login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PermitRootLogin'
        line: 'PermitRootLogin no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

    - name: disable SSH password authentication
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PasswordAuthentication'
        line: 'PasswordAuthentication no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

    - name: allow SSH through firewall
      community.general.ufw:
        rule: allow
        port: '22'
        proto: tcp

    - name: enable UFW
      community.general.ufw:
        state: enabled
        policy: deny
        direction: incoming

    - name: install Fail2ban
      ansible.builtin.apt:
        name: fail2ban
        state: present

    - name: enable Fail2ban
      ansible.builtin.systemd_service:
        name: fail2ban
        enabled: true
        state: started

  handlers:
    - name: restart ssh
      ansible.builtin.systemd_service:
        name: ssh
        state: restarted

Hardening SSH

SSH is one of the primary entry points into a server, so securing it is an important part of the baseline.

The first task disables direct root login:

- name: disable root SSH login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PermitRootLogin'
        line: 'PermitRootLogin no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

The resulting configuration is:

PermitRootLogin no

This means an external SSH connection can no longer authenticate directly as root.

This is particularly important because the bootstrap stage has already created the devops administrative account.

The intended access model is therefore:

rather than:

Disable SSH Password Authentication

- name: disable SSH password authentication
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PasswordAuthentication'
        line: 'PasswordAuthentication no'
        validate: '/usr/sbin/sshd -t -f %s'
      notify: restart ssh

After this configuration, SSH authentication relies on ssh key configured during the bootstrap stage.

This removes password based SSH authentication from the server's remote access path.

Validating SSH Configuration

Notice that both SSH configuration tasks include:

validate: '/usr/sbin/sshd -t'

This is an important detail.

Ansible does not immediately apply an invalid SSH configuration. The sshd -t command validates the configuration before the file is accepted.

This reduces the risk of provisioning an invalid SSH configuration that could prevent future connections.

The tasks also use a handler:

notify: restart ssh

The SSH service is therefore restarted only when the configuration actually changes.

This is one of the advantages of using Ansible modules instead of simply executing shell commands.

Installing Security Packages

The playbook installs UFW and Fail2ban:

UFW provides a simple interface for managing the linux host firewall, while Fail2ban can monitor authentication related logs and temporarily block clients that repeatedly fail authentication.

These tools address different parts of the security baseline:

Configuring UFW

The first firewall rule allows SSH:

- name: allow ssh
  community.general.ufw:
    rule: allow
    name: OpenSSH

This rule is intentionally configured before enabling UFW.

The firewall is then enabled with a default-deny policy for incoming traffic:

- name: enable ufw
  community.general.ufw:
    state: enabled
    policy: deny
    direction: incoming

The resulting security model is essentially:

This follows the principle of default deny: services should not become reachable simply because they happen to be listening on a network port.

Additional ports can then be explicitly allowed when they are required by later components of the infrastructure.

For example, the monitoring stage will later add a specific rule for Node Exporter rather than exposing all ports on the server.

Enabling Fail2ban

The final security task ensures that Fail2ban is enabled and running:

- name: enable fail2ban
  ansible.builtin.systemd_service:
    name: fail2ban
    enabled: true
    state: started

There are two important states being configured here:

enabled: true

ensures that Fail2ban starts automatically when the server boots.

state: started

ensures that the service is running immediately after provisioning.

This means the server does not need to be manually configured after deployment to activate the security service.

4. Docker Playbook

With the operating system configured and the initial security baseline in place, the next step is to prepare the server to run containerized workloads.

For this project, docker is installed through the official docker APT repository rather than relying on the docker package provided by the default ubuntu repositories.

The docker.yml playbook is responsible for:

Installing the required repository prerequisites
Configuring Docker's official APT repository
Installing Docker Engine and related components
Enabling and starting the Docker service
Allowing the administrative user to manage Docker

The playbook looks like this:

- name: install docker on ubuntu server
  hosts: servers
  become: true

  tasks:
    - name: install docker repository prerequisites
      ansible.builtin.apt:
        name:
          - ca-certificates
          - curl
        state: present
        update_cache: true

    - name: create docker keyring directory
      ansible.builtin.file:
        path: /etc/apt/keyrings
        state: directory
        mode: '0755'

    - name: download docker GPG key
      ansible.builtin.get_url:
        url: https://download.docker.com/linux/ubuntu/gpg
        dest: /etc/apt/keyrings/docker.asc
        mode: '0644'

    - name: add docker APT repository
      ansible.builtin.deb822_repository:
        name: docker
        types: deb
        uris:
          - https://download.docker.com/linux/ubuntu
        suites:
          - "{{ ansible_facts['distribution_release'] }}"
        components:
          - stable
        architectures:
          - amd64
        signed_by: /etc/apt/keyrings/docker.asc
        state: present

    - name: install docker engine and compose plugin
      ansible.builtin.apt:
        name:
          - docker-ce
          - docker-ce-cli
          - containerd.io
          - docker-buildx-plugin
          - docker-compose-plugin
        state: present
        update_cache: true

    - name: ensure docker service is enabled and running
      ansible.builtin.systemd_service:
        name: docker
        enabled: true
        state: started

    - name: add admin user to docker group
      ansible.builtin.user:
        name: "{{ admin_user }}"
        groups: docker
        append: true

Installing Repository Prerequisites

- name: install docker repository prerequisites
      ansible.builtin.apt:
        name:
          - ca-certificates
          - curl
        state: present
        update_cache: true

ca-certificates allows the system to properly validate HTTPS certificates, while curl is used to retrieve the Docker repository signing key.

These packages provide the basic requirements for securely adding the external repository.

Creating the APT Keyring Directory

/etc/apt/keyrings

using:

- name: create docker keyring directory
      ansible.builtin.file:
        path: /etc/apt/keyrings
        state: directory
        mode: '0755'

APT repository signing keys are stored separately from the repository configuration.

This provides a clear location for repository specific signing keys instead of placing them in a global trusted key configuration.

Adding Docker's Repository Signing Key

- name: download docker GPG key
      ansible.builtin.get_url:
        url: https://download.docker.com/linux/ubuntu/gpg
        dest: /etc/apt/keyrings/docker.asc
        mode: '0644'

The signing key is then referenced by the docker repository configuration.

This allows APT to verify that packages retrieved from the docker repository are signed by the expected key.

The important part here is that the playbook does not simply add an arbitrary repository and trust it globally. The repository is explicitly associated with its signing key.

Configuring the Docker APT Repository

- name: add docker APT repository
      ansible.builtin.deb822_repository:
        name: docker
        types: deb
        uris:
          - https://download.docker.com/linux/ubuntu
        suites:
          - "{{ ansible_facts['distribution_release'] }}"
        components:
          - stable
        architectures:
          - amd64
        signed_by: /etc/apt/keyrings/docker.asc
        state: present

One useful detail here is:

suites:
  - "{{ ansible_facts['distribution_release'] }}"

Instead of hardcoding an ubuntu release name, ansible obtains the distribution release from the target system.

This makes the playbook less dependent on a specific ubuntu release.

The repository is also restricted to the stable component and amd64 architecture:

components:
  - stable

architectures:
  - amd64

Installing Docker

- name: install docker engine and compose plugin
  ansible.builtin.apt:
    name:
      - docker-ce
      - docker-ce-cli
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin
    state: present
    update_cache: true

Rather than installing only the docker engine package, the playbook installs the components needed for a practical container environment.

The packages include:

docker-ce — docker engine
docker-ce-cli — docker command line interface
containerd.io — container runtime
docker-buildx-plugin — docker image build functionality
docker-compose-plugin — docker compose

This gives the server the tooling required to build, run, and manage containerized applications.

Ensuring Docker Is Running

- name: ensure docker service is enabled and running
  ansible.builtin.systemd_service:
    name: docker
    enabled: true
    state: started

There are two desired states here.

enabled: true

ensures docker starts automatically when the server boots.

state: started

ensures docker is running immediately after the provisioning stage.

This means that once provisioning finishes, the server is already capable of running containers without requiring manual intervention.

Allowing the Administrative User to Manage Docker

The final task adds the administrative user to the docker group:

- name: add admin user to docker group
      ansible.builtin.user:
        name: "{{ admin_user }}"
        groups: docker
        append: true

This allows the administrative user to interact with the docker daemon without prefixing every docker command with sudo.

For example, the user can run:

docker ps

instead of:

sudo docker ps

5. Monitoring Playbook

The server is now configured, hardened, and ready to run containerized workloads.

There is one final piece missing: observability.

A server can be perfectly configured and still become difficult to operate if we cannot see its resource usage and system health.

For this project, Prometheus Node Exporter is used to expose hardware and operating system metrics that can be collected by Prometheus.

The monitoring.yml playbook is responsible for:

Creating a dedicated system user for Node Exporter
Downloading and installing Node Exporter
Creating a systemd service
Ensuring Node Exporter starts automatically
Restricting access to the metrics endpoint

The playbook looks like this:

- name: configure monitoring agent
  hosts: servers
  become: true

  tasks:
    - name: create node exporter user
      ansible.builtin.user:
        name: node_exporter
        system: true
        shell: /usr/sbin/nologin
        create_home: false

    - name: download node exporter
      ansible.builtin.get_url:
        url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}.tar.gz"
        dest: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        mode: '0644'

    - name: extract node exporter
      ansible.builtin.unarchive:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        dest: /tmp
        remote_src: true
        creates: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}"

    - name: install node exporter binary
      ansible.builtin.copy:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}/node_exporter"
        dest: /usr/local/bin/node_exporter
        owner: root
        group: root
        mode: '0755'
        remote_src: true

    - name: create node exporter systemd service
      ansible.builtin.copy:
        dest: /etc/systemd/system/node_exporter.service
        owner: root
        group: root
        mode: '0644'
        content: |
          [Unit]
          Description=Prometheus Node Exporter
          Wants=network-online.target
          After=network-online.target

          [Service]
          User=node_exporter
          Group=node_exporter
          Type=simple
          ExecStart=/usr/local/bin/node_exporter

          [Install]
          WantedBy=multi-user.target
      notify: restart node exporter

    - name: enable and start node exporter
      ansible.builtin.systemd_service:
        name: node_exporter
        enabled: true
        state: started
        daemon_reload: true

    - name: allow node exporter from monitoring server
      community.general.ufw:
        rule: allow
        from_ip: "{{ monitoring_server_ip }}"
        to_port: '9100'
        proto: tcp

  handlers:
    - name: restart node exporter
      ansible.builtin.systemd_service:
        name: node_exporter
        state: restarted

Creating a Dedicated System User

The first step is to create a dedicated system user:

- name: create node exporter user
      ansible.builtin.user:
        name: node_exporter
        system: true
        shell: /usr/sbin/nologin
        create_home: false

Node Exporter does not need an interactive shell or a home directory.

Therefore, instead of running the service as root, we create a dedicated user with:

shell: /usr/sbin/nologin

This follows the principle of least privilege.

The service only needs to perform its intended function, so there is no reason for it to have an interactive login account.

Downloading Node Exporter

The Node Exporter version is controlled through group_vars/servers.yml:

node_exporter_version: 1.9.1
node_exporter_arch: linux-amd64

The playbook uses these variables to construct the download url:

- name: download node exporter
      ansible.builtin.get_url:
        url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}.tar.gz"
        dest: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        mode: '0644'

If we need to upgrade Node Exporter later, the version can be changed in the variables file instead of modifying the playbook itself.

For example:

node_exporter_version: 1.10.2

Extracting the Binary

The downloaded archive is extracted using Ansible's unarchive module:

- name: extract node exporter
      ansible.builtin.unarchive:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.tar.gz"
        dest: /tmp
        remote_src: true
        creates: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}"

The important part here is:

remote_src: true

The archive already exists on the target server, so ansible does not need to transfer it from the machine running Ansible.

The creates parameter also provides an idempotency check.

If the extracted directory already exists, Ansible does not need to extract the archive again.

Installing the Node Exporter Binary

The Node Exporter binary is then copied into:

/usr/local/bin/node_exporter

using:

- name: install node exporter binary
      ansible.builtin.copy:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.{{ node_exporter_arch }}/node_exporter"
        dest: /usr/local/bin/node_exporter
        owner: root
        group: root
        mode: '0755'
        remote_src: true

The binary is owned by root, while the service itself will run as the dedicated node_exporter user.

This creates a separation between the executable and the account that executes it.

Creating the systemd Service

Installing the binary alone does not make Node Exporter a managed system service.

The playbook therefore creates:

/etc/systemd/system/node_exporter.service

with:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Enabling and Starting Node Exporter

The service is then enabled and started:

- name: enable and start node exporter
  ansible.builtin.systemd_service:
    name: node_exporter
    enabled: true
    state: started
    daemon_reload: true

The daemon_reload option tells systemd to reload its service definitions after Ansible creates the new unit file.

The two important states are:

enabled: true
state: started

The first ensures Node Exporter starts automatically after a reboot.

The second ensures it is already running when provisioning finishes.

At this point, Node Exporter exposes its metrics endpoint on the standard port:

Restricting Monitoring Access

The metrics endpoint contains information about the server's operating system and resource usage, so there is no reason to expose it to every host that can reach the server.

The playbook therefore adds a specific UFW rule:

- name: allow node exporter from monitoring server
      community.general.ufw:
        rule: allow
        from_ip: "{{ monitoring_server_ip }}"
        to_port: '9100'
        proto: tcp

The monitoring server address is defined in:

monitoring_server_ip: 192.168.122.1

This results in a much narrower network rule:

This is an important continuation of the security model introduced in the Security Playbook.

We do not simply open port 9100 to the entire network:

9100 → 0.0.0.0/0

Instead, access is explicitly limited to the monitoring server.

Conclusion

Provisioning a server manually may seem simple when there is only one server to configure. But as the number of servers grows, manual configuration quickly becomes difficult to maintain and can lead to configuration drift.

In this project, we have built an ansible workflow that:

Bootstrapping and configuring the server
Applying system security hardening
Installing and configuring Docker
Setting up system monitoring
Restricting network access

Instead of relying on a sequence of manual commands, the desired server state is now defined as code that can be reviewed, version controlled, reproduced, and applied consistently.

Define the desired state. Automate it. Repeat it

You can find the source code for this article in my github repository: https://github.com/muhammadyulasfipahrizal/zero-touch-ansible.git