DEV Community: Pahri

[Boost]

Pahri — Fri, 09 Oct 2026 04:36:45 +0000

Pahri

Oct 9

Designing a Reliable Kubernetes Deployment with GitOps

#automation #devops #kubernetes

8 min read

Designing a Reliable Kubernetes Deployment with GitOps

Pahri — Fri, 09 Oct 2026 04:36:34 +0000

Deploying an application to a Kubernetes cluster is not just about building a container image and running it. A reliable deployment workflow must define how artifacts are built, where they are stored, which version is selected for release, and how the desired configuration is applied to the cluster.

As these responsibilities grow, maintaining a clear relationship between application code, image versions, and deployment manifests becomes increasingly important.

Without an explicit workflow, deployments can become difficult to trace, inconsistent, and teams may lose visibility into which application version is actually running.

The problem becomes clearer when these responsibilities are not explicitly connected:

These challenges motivate a workflow that makes image versions, deployment configuration, and release decisions explicit.

This is where Continuous Integration (CI) and GitOps complement each other: CI automates building and publishing container images, while GitOps manages the desired deployment state through version-controlled configuration.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.
Application Repository
GItops Repository

What We Are Building

In this project, we build a deployment that establish a workflow in which application artifacts are versioned, deployment changes are traceable, and Kubernetes continuously converges toward an explicitly defined desired state.

This project uses two repositories with separate responsibilities:

Application repository: Go API source code, Dockerfile, and CircleCI configuration.
GitOps repository: Helm chart and production values that declare the desired Kubernetes configuration.

The supporting components are:

CircleCI builds the application image and pushes it to Harbor.
Harbor stores versioned container images.
Helm templates the Kubernetes resources from chart and values.
Argo CD compares the GitOps configuration with the cluster and synchronizes it.
Kubernetes schedules and runs the workload.

The pipeline automates image delivery, while release promotion remains explicit: a person updates the GitOps image tag to deliberately select the version to deploy, then manually starts synchronization in Argo CD to control when that version is applied to the cluster.

Automatically deploying every published image is not always desirable because a successful build does not necessarily mean the image is ready for release. New builds may still require validation, review, or coordination with the target environment.

Separating image publishing from deployment provides clear release checkpoints and reduces the risk of unreviewed changes reaching users.

Project Architecture

Artifact delivery and deployment

The CI path produces an image. The deployment path declares which image should run and applies that desired state to Kubernetes. The GitOps repository is the handoff between them.

The dashed connection is not an automated pipeline step. CircleCI does not commit to the GitOps repository. After an image is published, the image tag is updated manually in environments/production/values-prod.yaml. Argo CD then shows the configuration difference; a user manually selects Sync to apply it.

Application repository

Repository: https://github.com/muhammadyulasfipahrizal/circle

File	Responsibility
`app/main.go`	Implements the Go HTTP API, including `/health`.
`app/go.mod`	Defines the Go module and dependencies.
`Dockerfile`	Builds the binary and packages it into a minimal runtime image.
`.circleci/config.yml`	Defines the CircleCI build-and-push job.

GitOps repository

Repository: https://github.com/muhammadyulasfipahrizal/gitops

File	Responsibility
`charts/app/Chart.yaml`	Defines Helm chart metadata.
`charts/app/values.yaml`	Provides default chart values.
`charts/app/templates/_helpers.tpl`	Defines reusable template helpers.
`charts/app/templates/deployment.yaml`	Defines the application Deployment.
`charts/app/templates/service.yaml`	Defines the Service.
`charts/app/templates/ingress.yaml`	Defines the Ingress resource.
`environments/production/values-prod.yaml`	Supplies production-specific overrides, including image tag, replica count, service port, and resources.

The chart defines reusable resource templates; the production values supply the settings for this deployment. The separation makes it possible to review a release by examining a small Git diff instead of changing Kubernetes resources directly.

Implementation

The Go application exposes a /health endpoint so that we can check whether it responds after deployment.

The Dockerfile uses a multi-stage build: the first stage compiles the Go application, and the final image contains the binary rather than the Go toolchain.

FROM golang:1.22-alpine AS builder

WORKDIR /src

COPY app/go.mod ./
RUN go mod download

COPY app/ .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
    go build -ldflags="-s -w" -o /app .

FROM gcr.io/distroless/static-debian12:nonroot

WORKDIR /
COPY --from=builder /app /app

EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/app"]

This keeps build tools out of the runtime image and runs the application as a non-root user. The application and Kubernetes Service are configured to use port 8080.

We can test the container locally before involving the cluster:

docker build -t pahri-k8s:local .
docker run --rm -p 8080:8080 pahri-k8s:local

In another terminal:

curl http://localhost:8080/health

Expected response:

{"status":"ok"}

Automate image delivery with CircleCI

The CircleCI configuration lives in .circleci/config.yml. The configured job uses the custom runner resource class pahri/devops and an Ubuntu machine executor. It checks out the application repository, authenticates with Harbor, builds the image, and pushes it.

Registry credentials are supplied through environment variables rather than committed to Git:

- run:
    name: Login Harbor
    command: |
      echo "$HARBOR_PASSWORD" | docker login \
        "$HARBOR_REGISTRY" \
        -u "$HARBOR_USERNAME" \
        --password-stdin

The image is tagged with CircleCI's $CIRCLE_SHA1, linking the published artifact to the source commit:

- run:
    name: Build Image
    command: |
      IMAGE="$HARBOR_REGISTRY/production/pahri-k8s"
      docker build -t "$IMAGE:$CIRCLE_SHA1" .

- run:
    name: Push Image
    command: |
      IMAGE="$HARBOR_REGISTRY/production/pahri-k8s"
      docker push "$IMAGE:$CIRCLE_SHA1"

The registry address and credentials must be configured in the CircleCI environment. The snippets above show the relevant commands; the complete job also needs its normal CircleCI job and workflow structure.

The image name used by this project is:

harbor.k8s.local/production/pahri-k8s:<commit-sha>

Confirm the artifact in Harbor

Harbor is the private registry for the image that the Kubernetes workload will pull. After CircleCI finishes successfully, confirm that the repository contains the new commit-SHA tag.

At this stage, the artifact exists in the registry, but the running deployment has not necessarily changed. Kubernetes uses the image reference declared in the GitOps configuration, not simply the most recently pushed image.

Define the workload with Helm

The chart under charts/app/ defines the Kubernetes resources. The production values file declares the desired image, replica count, service port, and resource requests and limits.

replicaCount: 3

image:
  repository: harbor.k8s.local/production/pahri-k8s
  tag: <tag>

service:
  port: 8080

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

The replica count expresses the desired number of application replicas. Requests inform scheduling, while limits define the configured CPU and memory ceilings for each container. The Service provides a stable in-cluster endpoint, and the Ingress template defines external HTTP routing when that resource is enabled and configured.

Before synchronizing a change, inspect the rendered manifests from the root of the GitOps repository:

helm template pahri-k8s ./charts/app \
  -f environments/production/values-prod.yaml

Review the rendered image reference, replica count, Service, and resource settings. Rendering the chart lets us inspect the intended resources without applying them to the cluster.

Promote an image by updating the GitOps repository

This step is intentionally manual. CircleCI publishes the image, but it does not edit or commit to the GitOps repository. The operator selects the image version to promote by updating:

environments/production/values-prod.yaml

For example:

image:
  repository: harbor.k8s.local/production/pahri-k8s
  tag: <new-tag>

Use the exact tag that CircleCI pushed successfully. Then commit and push the change:

git add environments/production/values-prod.yaml
git commit -m "Update production application image"
git push

The Git diff records which version is being promoted and preserves that change in the repository history. This is a deliberate control point: a successful image build does not automatically mean that the new image should be deployed.

Synchronize in Argo CD

Argo CD reads the GitOps repository and compares the desired configuration with the live resources. In this project, synchronization is initiated manually rather than automatically.

After pushing the updated values file:

Open the application in Argo CD.
Review the application status and the pending differences.
Confirm that the intended image tag is shown in the desired configuration.
Select Sync to apply the desired state.
Wait for the workload rollout and check the application health.

This manual Sync action is separate from the manual Git edit. The Git commit changes the desired state; clicking Sync requests that Argo CD apply it. Keeping both actions explicit provides a clear release decision and avoids suggesting that image publication alone deploys a new version.

A Synced status indicates that the tracked resources match the desired state Argo CD is using. Health status provides additional information about managed resources, but neither one alone proves that an application endpoint responds correctly.

Verify the deployed release

Verify the deployment at three levels: Kubernetes workload readiness, the image reference, and the application endpoint.

Check the Deployment, Pods, and Service in the namespace:

kubectl get deployment,pods,svc -n pahri-k8s

The Deployment should show the expected number of available replicas, and the Pods should become ready. If they do not, inspect the Pod and recent events:

kubectl describe pod <pod-name> -n pahri-k8s

kubectl get events -n pahri-k8s \
  --sort-by=.metadata.creationTimestamp

Check the image reference on the Deployment:

kubectl get deployment -n pahri-k8s \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[*].image}{"\n"}{end}'

The image should match the commit-SHA tag in values-prod.yaml.

If the application does not have a reachable Ingress or external endpoint, use port forwarding to test it through the Kubernetes Service:

kubectl port-forward -n pahri-k8s \
  svc/pahri-k8s-app 8080:8080

In another terminal:

curl http://IP:8080/health

Expected response:

{"status":"ok"}

If the application is exposed through Ingress, test that route separately as well. Port-forwarding validates the Service-to-application path, not the full external access path.

The evidence should tell one consistent story: CircleCI published a specific image tag, the GitOps values file declares that tag, the operator manually synchronized it through Argo CD, and Kubernetes is running the declared image while the application responds successfully.

Conclusion

This project demonstrates how Continuous Integration and GitOps can work together to create a deployment workflow that is traceable, predictable, and controlled.

The key design decision is to separate artifact delivery from release promotion. Publishing an image does not automatically make it a production release. Instead, an operator explicitly selects the image version in Git and initiates synchronization through Argo CD. These two control points make release decisions visible, preserve configuration history, and reduce the risk of deploying unreviewed changes.

This approach prioritizes release control and traceability over fully automated deployment.

Ultimately, a reliable deployment workflow is not defined by how many steps it automates, but by how clearly it connects artifacts, configuration, and release decisions. By making these relationships explicit, this project provides a practical foundation for managing Kubernetes deployments with greater confidence and accountability.

[Boost]

Pahri — Tue, 29 Sep 2026 12:12:56 +0000

Pahri

Sep 29

Understanding Kubernetes Datapath With Cilium

#architecture #kubernetes #networking

17 min read

Understanding Kubernetes Datapath With Cilium

Pahri — Tue, 29 Sep 2026 12:06:56 +0000

Kubernetes network might look simple when a cluster is small.

A pod can communicate with another pod, an application can expose a service, and clients can connect to the application.

However as the cluster grows, networking becomes more than just connecting one pod to another.

For example, consider a simple application running inside Kubernetes. A request may start from one Pod and reach another Pod through a Kubernetes Service.

Now consider a request coming from outside the cluster.

The traffic needs to enter the cluster, reach a LoadBalancer address, be handled by the networking layer, and eventually reach the application Pod.

This raises an important question:

What actually happens to a packet as it travels from one Pod, through a Kubernetes Service, and eventually reaches an application from outside the cluster?

Answering this question requires looking beyond Kubernetes objects such as Pods and SFervices. We also need to understand the components responsible for turning those objects into actual network traffic.

Traditionally, Kubernetes networking relies on multiple components working together to provide service connectivity and traffic forwarding.

In this lab, we explore what changes when Cilium becomes the networking layer.

Rather than treating Cilium simply as a CNI plugin, we will look at how it participates in the Kubernetes networking datapath and how its components work together to handle different types of traffic.

This article will cover several topics:

Cilium's networking model
eBPF based service handling
kube proxy replacement
LoadBalancer IP allocation with LB-IPAM
Layer 2 advertisement
Envoy based ingress

Kubernetes networking is not just about assigning IP addresses to Pods. It is about building a datapath that determines how packets move between workloads, Services, nodes, and external networks.

The Environment

Before looking at the datapath, it is useful to establish the environment in which the experiments are performed.

The Kubernetes cluster runs on Ubuntu virtual machines hosted by a physical server using KVM/libvirt. The cluster is provisioned with kubeadm, while containerd is used as the container runtime.

The environment is intentionally kept close to a small on premises Kubernetes deployment rather than relying on a managed Kubernetes service. This makes the networking behavior, particularly LoadBalancer IP allocation and Layer 2 advertisement, visible from the underlying network.

Kubernetes Cluster

The cluster uses a multi control-plane architecture to provide a highly available Kubernetes control plane.

The Kubernetes API is exposed through HAProxy to provide a stable endpoint for cluster administration and for nodes joining the control plane.

The cluster consists of:

3 control plane nodes
3 worker nodes
Ubuntu virtual machines
kubeadm for cluster provisioning
containerd as the container runtime
HAProxy

Networking Environment

The cluster runs inside a libvirt network hosted on a physical server.

At a high level, the environment looks like this:

The Kubernetes nodes communicate through the node network, while Pods use a separate Pod CIDR managed by Cilium.

For the experiments in this article, the relevant network ranges are:

Component	Network
Node network	`192.168.122.0/24`
Pod CIDR	`10.244.0.0/16`
LoadBalancer IP range	`192.168.122.241–192.168.122.254`
Kubernetes	`v1.34.11`
Cilium	`v1.20.2`

The LoadBalancer addresses are allocated from the same Layer 2 network as the Kubernetes nodes.

This is important because the LoadBalancer IP is not provided by an external cloud provider. Cilium must allocate an address from the configured pool and make that address reachable from the local network.

Infrastructure Automation

The underlying Kubernetes environment is provisioned and configured through Ansible.

The automation covers the infrastructure and cluster setup, including the Kubernetes nodes, container runtime, control-plane configuration, HAProxy, and Cilium deployment.

The implementation is maintained in the accompanying GitHub repository:

https://github.com/muhammadyulasfipahrizal/cilium-k8s.git

The repository is used to make the underlying Kubernetes environment reproducible, while this article focuses on understanding the networking behavior implemented on top of it.

Kubernetes Networking Before Cilium

Before looking at what Cilium changes, we need to establish the networking model that Kubernetes presents to applications.

A Kubernetes application does not normally communicate using a single type of IP address.

A Pod has its own IP address. A Service provides a stable virtual address in front of a group of Pods. Nodes have their own addresses on the underlying network, and applications exposed outside the cluster may use a LoadBalancer address or an Ingress endpoint.

These addresses belong to different layers of the networking path.

Different Networking Layers

A simplified view of the path looks like this:

Pod network

Each Pod receives an IP address from the cluster's Pod CIDR. This address identifies the Pod within the Kubernetes networking environment.

For example, an application might have an address such as:

10.244.x.x

The important property is that the application does not need to know which physical or virtual machine is hosting the Pod. The networking layer is responsible for making Pod connectivity work across nodes.

Service network

A Service provides a stable virtual IP in front of one or more Pods. Instead of connecting directly to a changing Pod IP, clients can connect to the Service:

The Service therefore introduces an abstraction between the client and the actual workload.

The client does not need to know which Pod should receive the packet.

Something inside the networking datapath must make that decision.

A Service definition describes the desired connectivity:

Service
  name: nginx
  port: 80

But a packet arriving at the Service IP does not automatically know that it should be forwarded to one of the backend Pods.

The networking implementation must translate that logical Service into forwarding behavior.

Node network

Every Kubernetes node has an IP address on the underlying network. This address allows the node to communicate with other nodes and with networks outside the cluster.

At this point, the packet is no longer concerned only with Kubernetes abstractions. It has to interact with the actual network connecting the nodes.

External network

For externally exposed applications, Kubernetes can provide an address that is reachable outside the cluster. Depending on the environment, this may be a cloud provided LoadBalancer address, a manually assigned address, or an address advertised directly on the local network.

In this lab, the LoadBalancer addresses come from the local node network.

This creates a complete path across several networking layers:

Where kube-proxy Fits

One of the key components in the traditional Kubernetes networking model is kube proxy.

A Kubernetes Service can have multiple backend endpoints, while clients see only one stable Service IP.

The networking layer therefore needs to perform a service translation:

The exact implementation can vary, but the fundamental idea remains the same:

Kubernetes needs a datapath mechanism that turns Service definitions and endpoints into packet forwarding rules.

When a request is sent to a Service, kube proxy is responsible for programming the node's networking layer so that traffic can be redirected

Cilium's Role

Pods need connectivity, Services need to map stable virtual addresses to backend workloads, and externally exposed applications need a path between the Kubernetes network and the surrounding network.

Traditionally, these responsibilities are implemented by multiple networking components.

Cilium approaches the problem differently.

It starts as a CNI plugin, but its role does not end after a Pod receives an IP address. Cilium provides the datapath responsible for processing network traffic and can take responsibility for Service handling, network policy, load balancing, observability, and other networking functions.

CNI vs Networking Datapath

The Container Network Interface (CNI) defines how a container runtime invokes networking plugins when a Pod is created or removed.

This is an important responsibility, but it represents only one part of Kubernetes networking.

Once the Pod is running, packets continuously move through the networking datapath.

The CNI lifecycle answers questions such as:

How does this Pod become connected to the network?

The datapath answers a different question:

What should happen to each packet after the Pod is connected?

Cilium participates in both.

Cilium can provide the initial Pod networking, while its agents and datapath continue to manage how traffic is processed after the Pod has started.

This also allows several Kubernetes networking functions to be handled by the same networking system.

The result is a networking layer that understands more than just Pod interfaces.

It can observe Kubernetes resources and translate their desired state into actual packet processing behavior.

eBPF as the Datapath

Cilium uses Extended Berkeley Packet Filter (eBPF) to implement significant parts of its networking datapath.

eBPF allows programs to run inside the Linux kernel at specific networking and system execution points.

For Cilium, this provides a way to process packets close to where they enter and move through the kernel networking stack.

The high-level architecture can be simplified to:

The Cilium Agent runs in userspace and is responsible for observing the Kubernetes environment and configuring the datapath.

The eBPF programs then perform packet processing in the kernel.

The agent does not need to make every packet decision itself.

Instead, it configures the datapath so that packets can be processed directly in the kernel.

This becomes particularly important for Kubernetes Services.

With a traditional kube-proxy based architecture, Service definitions are translated into networking rules such as iptables or IPVS rules.

With Cilium, Service handling can instead be implemented using eBPF-based load-balancing and forwarding.

compared with:

The important difference is not simply that one uses iptables while the other uses eBPF.

The larger architectural difference is where networking decisions are implemented and how the datapath is programmed.

Cilium moves important packet-processing logic into the kernel, allowing the datapath to make decisions without requiring a userspace proxy to process every connection.

However, this does not mean that everything in Cilium runs inside the kernel.

Cilium still has userspace components.

For example, the Cilium Agent manages the datapath, while Envoy can handle application-layer traffic such as HTTP and TLS when Cilium's ingress functionality is used.

This distinction will become important later.

Not every networking problem should be solved at the same layer.

Some decisions can be made at the kernel level, such as Service load balancing and packet forwarding.

Other decisions require understanding application-level protocols, where a proxy such as Envoy becomes useful.

This gives us a more useful mental model of Cilium:

Cilium is not only responsible for connecting Pods to the network. It provides a programmable networking datapath that connects Kubernetes state with actual packet processing.

From here, the individual components we enabled in the lab start to make more sense.

Kube-proxy replacement, changes how Kubernetes Services are implemented.
LB-IPAM, determines how LoadBalancer addresses are allocated.
L2 Announcement, makes those addresses reachable from the local network.
Envoy, provides the application-layer entry point for Ingress traffic.

These are not isolated features. They are different parts of the same networking path.

Establishing the Baseline

Before changing the Kubernetes networking datapath, we first need to establish what the cluster looks like in its initial state.

Verify the Cilium Nodes

Cilium maintains its own representation of Kubernetes nodes through CiliumNode resources.

We can inspect them with:

kubectl get ciliumnodes

What this demonstrates:

Cilium recognizes the Kubernetes nodes.
Each node participates in the Cilium networking environment.
Kubernetes node networking state is represented through Cilium resources.

This gives us the first indication that Cilium is not simply providing connectivity during Pod creation.

It maintains networking state for the cluster and uses that state to construct its datapath.

Pod-to-Pod Communication

The next step is to verify basic workload connectivity.

We create two lightweight test Pods and use them to observe how traffic moves through the cluster.

The simplified communication path is:

We first verify that a Pod can resolve a Kubernetes Service:

nslookup kubernetes.default.svc.cluster.local

What this demonstrates:

The Pod can reach Kubernetes DNS.
Kubernetes Service discovery is functioning.
The Pod network can communicate with the cluster's DNS service.

We can then verify direct connectivity between the test Pods.

Here i create two pods using busybox:1.36 and testing their connectivity inside the cluster

What this demonstrates:

Pods can communicate across the Kubernetes network layer.
The CNI datapath is forwarding workload traffic successfully.
Pod connectivity works before introducing the additional Cilium features examined later.

Replacing kube-proxy with Cilium

The baseline confirms that the cluster can provide basic Pod networking and Kubernetes Service connectivity.

However, there is still another component involved in the Service datapath: kube-proxy.

In a traditional Kubernetes deployment, kube-proxy watches Services and their endpoints and programs the node networking layer so that traffic sent to a Service can reach one of its backend Pods.

This model works, but it introduces another component responsible for translating Kubernetes Service state into packet forwarding behavior.

Since Cilium already provides a programmable networking datapath, a natural question is:

What happens if Cilium takes responsibility for Kubernetes Service handling as well?

Why Replace kube-proxy?

With the traditional Kubernetes networking model, kube-proxy programs the node's networking rules to implement Service forwarding.

With Cilium's kube-proxy replacement, these Service networking functions can instead be handled directly by Cilium and eBPF, allowing traffic to be processed closer to the kernel networking layer.

Instead of:

we move toward a datapath where Cilium handles Service translation:

This changes the architecture rather than simply changing a configuration flag.

The main benefits are:

A more direct datapath, Service translation and forwarding can be handled by eBPF in the kernel instead of relying on kube-proxy-managed networking rules.
Lower networking overhead, Removing an additional Service-proxying layer can simplify packet processing, particularly in clusters with a large number of Services and endpoints.
Better integration with Cilium, Service handling becomes part of the same datapath used for networking, load balancing, security, and observability.
Scalable load balancing, Cilium can perform Service load balancing using eBPF-based mechanisms, including efficient handling of large endpoint sets.
More control over networking behavior, Cilium exposes features such as socket-level load balancing, direct routing, and other eBPF-based datapath capabilities that go beyond traditional kube-proxy behavior.

Enabling kube-proxy Replacement

Cilium exposes kube-proxy replacement through its configuration.

The relevant setting in this lab is:

kubeProxyReplacement=true

The configuration tells Cilium to take responsibility for Kubernetes Service handling through its own datapath.

The Cilium Agent observes Kubernetes Service and endpoint state and programs the eBPF datapath accordingly.

This means that the Service abstraction remains the same from the application's perspective.

Applications still communicate with a Kubernetes Service.

What changes is the mechanism underneath that abstraction.

Removing kube-proxy

Once Cilium's kube-proxy replacement is active, we can inspect the existing kube-proxy DaemonSet:

kubectl -n kube-system get ds kube-proxy

The DaemonSet can then be removed from the cluster.

kubectl -n kube-system delete ds kube-proxy

If Cilium is correctly providing the Service datapath, removing kube-proxy should not prevent applications from communicating through Kubernetes Services.

Verification

Now we repeat the same networking tests used during the baseline.

This is important because we want to compare behavior rather than introduce an entirely different test.

The tests include:

DNS resolution
Pod-to-Pod connectivity
Kubernetes Service connectivity

First, we verify that Kubernetes DNS is still reachable.

We then repeat the Pod-to-Pod connectivity test.

Finally, we test connectivity through a Kubernetes Service.

The important observation is that the Kubernetes abstraction has not changed. What changed is the datapath underneath it.

From ClusterIP to LoadBalancer

A Pod can communicate with another Pod, and a Kubernetes Service can provide a stable endpoint in front of those workloads.

But what happens when the client is outside the cluster?

A LoadBalancer Service introduces another requirement. The Service needs an IP address that external clients can actually reach.

In a cloud environment, this is usually handled by a cloud controller that provisions an external load balancer and assigns an address to the Service.

Our environment does not have a cloud provider.

Instead, we want Kubernetes to expose a Service using an address from the same Layer 2 network as the Kubernetes nodes:

This creates two separate networking problems:

Which IP address should the LoadBalancer Service use?
How does the local network learn that this IP is reachable through the Kubernetes cluster?

Cilium solves these problems with two different components:

LB-IPAM and L2 Announcement

LB-IPAM

Cilium LB-IPAM provides IP address management for Kubernetes LoadBalancer Services.

In this lab, the available addresses come from a dedicated pool:

The allocation process looks like this:

LB-IPAM can assign:

Service → 192.168.122.241

but that does not automatically make the address reachable from the LAN.

The network still needs to know where 192.168.122.241 exists.

L2 Announcement

The LoadBalancer IP belongs to the same Layer 2 network as the Kubernetes nodes.

When an external client wants to communicate with 192.168.122.241, it needs to resolve that IP to a MAC address using the local network's address-resolution mechanism.

Cilium's L2 Announcement functionality allows a Kubernetes node to announce that it can receive traffic for the LoadBalancer IP.

Expressing the L2 Policy

The L2 policy in this lab specifies which LoadBalancer addresses should be announced and which node interfaces may be used for that announcement.

The relevant configuration is:

loadBalancerIPs: true

interfaces:
  - ^ens3$

The policy expresses two decisions:

This allows Cilium to connect the Kubernetes LoadBalancer abstraction with the actual Layer 2 network interface through which the address becomes reachable.

Testing External Connectivity

At this point, the complete external path can be tested from outside the Service abstraction:

curl http://192.168.122.241

If the request succeeds, several components have successfully worked together:

Cilium Ingress and Envoy

Cilium can allocate a LoadBalancer IP through LB-IPAM, advertise that address on the local network using L2 Announcement, and forward traffic into the Kubernetes networking datapath.

But there is still a limitation.

A LoadBalancer address tells us where traffic enters the cluster. It does not by itself define how HTTP requests should be routed between multiple applications.

Consider a cluster running several applications:

Exposing every application independently would require a separate external entry point for each application.

Instead, we can introduce an HTTP aware entry point:

Now a single LoadBalancer address can act as the entry point for multiple applications.

The component responsible for understanding the HTTP request is Envoy.

Cilium provides an Ingress controller that translates Kubernetes Ingress configuration into the configuration required by Envoy.

Kubernetes Ingress and the Data Plane

The Kubernetes Ingress resource describes the desired HTTP routing behavior.

For example, our test application uses the hostname:

nginx.k8s.local

The Ingress resource describes the desired routing:

The Cilium Ingress Controller observes this Kubernetes configuration and configures Envoy accordingly.

Envoy then becomes the component that actually handles the HTTP request.

The important point is:

The Ingress resource describes the desired HTTP routing, while Envoy becomes the data-plane component handling the request.

This is different from the Service datapath we examined earlier.

At the Service layer, Cilium can make forwarding decisions based on network-level information such as IP addresses and ports.

At the Ingress layer, the system needs to understand application-level information such as the HTTP Host header and request path.

For example:

Host: nginx.k8s.local
Path: /

Envoy can inspect this information and determine which configured backend should receive the request.

Following One HTTP Request

Rather than looking at each component independently, we can now follow a single HTTP request through the entire system.

The request is generated with:

curl --resolve nginx.k8s.local:80:192.168.122.241 \
  http://nginx.k8s.local/

The --resolve option makes the client connect to 192.168.122.241 while still sending the HTTP request for:

Host: nginx.k8s.local

This is useful because the request now contains both pieces of information required by the architecture:

Destination IP
192.168.122.241

Host
nginx.k8s.local

We can trace the request conceptually as:

The Client Targets the LoadBalancer IP

The client starts with an HTTP request for:

http://nginx.k8s.local/

The --resolve option maps the hostname to the LoadBalancer address:

nginx.k8s.local: 192.168.122.241

The client therefore sends the packet toward the address allocated by Cilium LB-IPAM.

Local Network Resolves the Destination

Because 192.168.122.241 belongs to the local Layer 2 network, the client needs to determine which MAC address should receive the traffic.

The L2 Announcement mechanism allows Cilium to advertise the LoadBalancer address through an eligible Kubernetes node interface.

The important distinction from the previous section is now visible:

LB-IPAM provided the address. L2 Announcement made the address reachable on the network.

The Packet Enters the Kubernetes Datapath

Once the packet reaches the Kubernetes node, it enters the networking datapath managed by Cilium.

At this point, the request has crossed from the physical or virtual network into Kubernetes. The packet can now be processed according to the networking configuration established by Cilium.

Envoy Handles the HTTP Request

The request reaches the Envoy based Ingress path.

Unlike a simple L3/L4 forwarding decision, Envoy can inspect HTTP information such as:

Host: nginx.k8s.local
Path: /

The Ingress configuration tells Cilium which HTTP routes should exist, while Envoy implements those routes in the data plane.

Envoy therefore determines that the request should be forwarded to the Service associated with the Ingress rule.

The Service Resolves the Backend

The request is then forwarded toward the Kubernetes Service:

The Service provides the stable abstraction between the Ingress layer and the actual application Pods.

The client does not need to know which Pod is running the application.

Cilium Selects the Backend

With kube-proxy removed, Cilium's eBPF-based Service handling provides the forwarding mechanism for the Service.

The request can therefore reach one of the NGINX Pods behind the Service.

The Complete Path

This single request connects the different networking mechanisms explored throughout the article.

The request starts as ordinary HTTP traffic on the network.
Cilium L2 Announcement makes the LoadBalancer address reachable.
Cilium datapath brings the traffic into the cluster.
Envoy interprets the HTTP request and applies the Ingress routing configuration.
The request then reaches a Kubernetes Service, where Cilium's eBPF Service datapath selects the backend workload.

Conclusion

Kubernetes networking becomes easier to understand when we stop looking at Pods, Services, LoadBalancers, and Ingress as isolated Kubernetes objects and instead follow the packet moving between them.

In this lab, we started with basic Pod networking and Service connectivity, then progressively changed the networking datapath by introducing Cilium's kube-proxy replacement, LB-IPAM, L2 Announcement, and Envoy-based Ingress.

Each component solves a different part of the networking problem:

Together, these components form a layered networking architecture:

This also gives us a more useful mental model of Cilium.

Cilium is not simply the component that gives Pods IP addresses. It connects Kubernetes state with the Linux networking datapath and provides mechanisms for processing traffic at different layers. Some decisions can be handled in the kernel through eBPF, while application-layer traffic can be handled by Envoy in userspace.

Ultimately, the most useful way to understand Kubernetes networking is to ask a simple question whenever traffic needs to move:

What component is responsible for making the next networking decision?

Once that question can be answered, Kubernetes networking becomes less about memorizing individual components and more about understanding the datapath that connects them.

From Static Passwords to Dynamic Database Credentials with HashiCorp Vault

Pahri — Sat, 19 Sep 2026 04:24:44 +0000

Almost every production app need credentials to access their database.

A common approach is to create a database user with a password and provide that credential to the application through environment variable, configuration file, or secret management system.

The application then uses the same credential until it is manually changed.

This approach is simple, but the credential itself becomes a long lived security risk until someone rotates it.

Though credential rotation can also become a new problem.

A typical workflow might look like this:

The problem is not only how the credential is stored.

We also need to consider its entire lifecycle.

Who creates it?

How long should it remain valid?

How is it renewed while the application is running?

How do we replace it without interrupting the application?

What happens to the old credential after the replacement is ready?

We wanted to approach this differently.

Instead of giving the application a permanent database password, we wanted the database credential to be temporary by design.

This is where dynamic credentials become useful.

With HashiCorp Vault, an application can request a database credential when it needs one. Vault generates a temporary database user and returns the credential together with a lease.

The application can then use the credential while it is valid and renew its lease while the credential is still in use.

When the application needs to replace the credential, it can request another one instead of modifying a permanent password.

The workflow becomes:

This changes credential management from a manually maintained configuration value into a lifecycle that can be managed by the application.

Project Goals

We wanted an application whose database credentials are temporary by design, automatically renewed while in use, and replaced periodically without storing a static password.

The application should be able to:

Authenticate to Vault without using a Vault root token.
Obtain database credentials dynamically.
Connect to Database using those credentials.
Renew the credential lease while the application is running.
After three successful renewals, request a new credential.
Verify the new credential against Database.
Switch to the new credential.
Revoke the old lease.
Continue serving requests without restarting the application.

The three renewal threshold is intentionally used as a simple rotation trigger for this project. It gives us a deterministic lifecycle that can be observed and tested without introducing a more complex rotation policy.

The complete implementation will show how these pieces work together, from the initial authentication and credential request to lease renewal and the final zero downtime credential switch.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the full source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/vault-setup.git

Architecture

The project consists of three main components, each component has a specific responsibility in the credential lifecycle.

Vault

HashiCorp Vault is responsible for managing the database credentials used by the application.

Vault generates temporary credentials and manages the lease associated with each credential. The application communicates with Vault to authenticate, obtain credentials, and maintain the credential lease while it is running.

Application

The application is a small Go API that consumes credentials provided by Vault.

It authenticates to Vault, requests a dynamic PostgreSQL credential, and uses that credential to establish a connection to the database.

The application also manages the runtime credential lifecycle. While the credential is being used, it renews the lease. When the rotation condition is reached, the application requests a new credential and verifies the new connection before switching away from the existing one.

This allows the application to replace its database credential while continuing to run.

Database

PostgreSQL is the database accessed by the application.

The database is configured so that Vault can create temporary users for the application. These users are created with the permissions required by the application and are removed when their associated credentials are revoked.

The application does not communicate with PostgreSQL through Vault. Instead, Vault provides the application with the credentials, and the application connects directly to PostgreSQL.

Component Interaction

The interaction between the three components can be summarized as:

This architecture keeps credential management separate from database access. Vault provides the temporary identity needed to access the database, while the application remains responsible for using that identity throughout its lifecycle.

Project Structure

Each part of the repository has a specific responsibility.

app/ contains the Go application, including its source code, Go dependencies, Dockerfile, and Docker Compose configuration.
config/ contains the Vault server configuration.
db/ contains the PostgreSQL Docker Compose configuration and the SQL script used to initialize the database.
docker-compose.yml at the repository root defines the Vault service and connects it to the shared Docker network.

Implementation

PostgreSQL

The PostgreSQL service is defined in db/docker-compose.yml.

The configuration uses PostgreSQL 17 and persists its data through a Docker volume. The initialization directory is mounted into PostgreSQL's /docker-entrypoint-initdb.d directory so the database can be prepared when the container is initialized.

The service is also attached to the external Docker network used by the other components, allowing Vault and the application to communicate with PostgreSQL through its container name.

services:
  postgres:
    image: postgres:17
    container_name: db-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: ${YOUR_DB_NAME}
      POSTGRES_USER: ${YOUR_DB_USER}
      POSTGRES_PASSWORD: ${YOUR_DB_PASSWORD}
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./postgres/init:/docker-entrypoint-initdb.d:ro
    networks:
      - ${YOUR_DOCKER_NETWORK}

Database Initialization

The initialization script is located at:

db/postgres/init/01-init.sql

Its purpose is to create the administrative database user that Vault will use when managing dynamic PostgreSQL users.

The script grants this user permission to connect to the target database and create objects in the public schema.

The repository contains the following initialization:

Vault

Vault is defined in the root docker-compose.yml.

The service uses the official hashicorp/vault image, exposes ports 8200 and 8201, mounts the Vault data and configuration directories, and starts Vault using the repository's HCL configuration file.

services:
  vault:
    image: hashicorp/vault:latest
    container_name: vault
    restart: unless-stopped
    ports:
      - "8200:8200"
      - "8201:8201"
    volumes:
      - ./data:/vault/data
      - ./config:/vault/config:ro
    cap_add:
      - IPC_LOCK
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - ${YOUR_DOCKER_NETWORK}

Vault Configuration

The Vault server configuration is stored under config/.

ui = true

disable_mlock = true

api_addr     = "https://0.0.0.0:8200"
cluster_addr = "https://0.0.0.0:8201"

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable     = true
}

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-1"
}

The configuration enables the Vault UI and uses Raft storage for Vault's persistent data. The TCP listener accepts connections on ports 8200 and 8201.

Enable Database Secrets Engine

The Database Secrets Engine allows Vault to generate database credentials.

For this project, the database plugin is:

postgresql-database-plugin

The database connection is configured with the name:

vault-postgres

Vault connects to PostgreSQL through the Docker network using:

postgresql://{{username}}:{{password}}@vault-postgres:5432/appdb?sslmode=disable

The {{username}} and {{password}} values are provided by Vault's database plugin when it establishes the administrative connection.

Dynamic Database Role

When the application requests database/creds/app-role, Vault generates a new role and password.

The role is configured as a dynamic role with:

TTL:     1 hour
Max TTL: 1 day

When the application requests:

database/creds/app-role

Vault generates a new database username and password.

The creation statements are:

CREATE ROLE "{{name}}"
WITH LOGIN
PASSWORD '{{password}}'
VALID UNTIL '{{expiration}}';

GRANT CONNECT ON DATABASE YOUR_DATABASE_NAME
TO "{{name}}";

Vault replaces the template variables with generated values before executing the statements against PostgreSQL.

The role also defines a revocation statement:

DROP ROLE IF EXISTS "{{name}}";

This is what allows Vault to clean up the generated PostgreSQL user when the lease is revoked.

Policy

The application also needs permission to request them from Vault.

We can create ACL policy named app-policy:

This policy deliberately exposes only the credential endpoint required by the application.

AppRole Authentication

The application should not authenticate to Vault using the root token.

Instead, we enable the AppRole authentication method.

The configured authentication method is:

The application receives two values:

Role ID
Secret ID

These values are provided to the application through its runtime environment rather than embedding them directly. The application can therefore authenticate as a service identity without receiving the Vault root token.

App

The application is defined in app/docker-compose.yml.

The Compose configuration provides the Vault address, AppRole credentials, and PostgreSQL connection information through environment variables.

The application communicates with Vault using:

http://vault:8200

and database with:

db-postgres:5432

Both services are connected through the same external Docker network.

This keeps the application configuration independent from hard coded container IP addresses.

Application Lifecycle

Authenticate

When the application starts, it creates a Vault client and authenticates using AppRole.

The application sends its Role ID and Secret ID to:

auth/approle/login

Vault validates the credentials and returns a Vault token.

That token is then used for subsequent requests to the Vault API.

Request Dynamic Credentials

After authentication, the application requests:

database/creds/app-role

Vault creates a new Database role and returns:

username
password
lease ID
lease duration

The application stores these values in its runtime state.

Connect to Database

The generated username and password are then used to create a database connection.

The application verifies the credential by establishing the connection and performing a database ping.

Lease Renewal

The application obtains the lease duration directly from Vault when it requests the credentials.

Instead of using a fixed renewal interval, the renewal loop waits for approximately half of the current lease duration before attempting a renewal.

When the renewal succeeds, the application updates its lease information and increments its renewal counter.

Rotation

For this project, rotation occurs after three successful lease renewals.

The counter in this project is intentionally simple so the complete lifecycle can be observed during testing.

After the third successful renewal:

The application requests another credential from:

database/creds/app-role

Vault creates a second database user with a different username, password, and lease.

At this point, two credentials can temporarily exist:

The application does not immediately revoke the old credential.

First, it verifies that the new credential actually works.

Zero Downtime Credential Switch

The application creates a new PostgreSQL connection using the newly generated credential.

If the connection succeeds, the application switches its active database connection to the new one.

Only after the new connection is verified does the application revoke the old lease.

The old credential remains available until the replacement has been successfully verified. If the new credential cannot connect to database, the application keeps using the existing connection instead of disrupting the running application.

Observing the Rotation and Database Profile

The application exposes its current lifecycle state through an API handler, allowing us to observe information such as the current credential identity, lease information, expiration, and renewal count.

We can then compare that application state with the state visible from the Vault CLI.

The observation focuses on two events:

Lease renewal.
Credential rotation.

Lease Renewal

The application state should show that the lease is still associated with the same dynamic credential while the renewal count increases.

This demonstrates the distinction between renewal and rotation: the lease is extended, but the database credential itself has not yet been replaced.

Credential Rotation

After the third successful renewal, the application requests a new credential.

The final state should show that the application has successfully moved to the new credential while continuing to run.

Conclusion

Managing database credentials is not only about keeping passwords secret, but also about managing their entire lifecycle.

In this project, Vault provides temporary Database credentials through AppRole authentication, while the application handles lease renewal and credential rotation automatically.

After three renewals, the application requests a new credential, verifies the connection, switches to it, and revokes the old lease. This allows credentials to be rotated without restarting the application or interrupting database access.

The three renewal policy is only a demonstration. In production, the same pattern can be adapted to rotation intervals and security requirements.

The idea is to treat database credentials as temporary runtime rather than static configuration values.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the full source code in the repository below.

https://github.com/muhammadyulasfipahrizal/vault-setup.git

Building Highly Available PostgreSQL Cluster With Patroni, ETCD, HAProxy, And Keepalived

Pahri — Sat, 12 Sep 2026 06:29:06 +0000

A database is one of the most critical components in an application.

When the database is running normally, the application can connect to it and continue serving requests. But when the database server becomes unavailable, the impact can quickly spread to the services that depend on it.

For a system that needs to remain available, relying on a single database server creates an obvious risk.

If that server fails, the application may lose its database connection until the server is recovered or another database server is manually brought into service.

This raises an important question:

What happens when the database server that our application depends on suddenly fails?

A common approach is to run multiple PostgreSQL servers. However, having multiple servers does not automatically make the database highly available.

A highly available database needs to continue serving applications even when part of the infrastructure fails. This means the system needs to handle failures without requiring the application to be manually reconfigured every time something goes wrong.

In this article, we will build a highly available database cluster using PostgreSQL, Patroni, etcd, HAProxy, and Keepalived.

What We Need

Before building the system, we first need to define what we expect from a highly available database.

The goal is not simply to have multiple PostgreSQL servers running at the same time. The system should remain available and usable even when individual components fail.

For this project, we define four main requirements:

Multiple PostgreSQL Nodes

We need multiple PostgreSQL nodes so that the database service has another node available when one of the servers becomes unavailable.

The nodes should maintain the same database state closely enough that one of them can take over when necessary.

This gives us redundancy at the database layer.

Automatic Failover

When the current primary PostgreSQL nodes becomes unavailable, the cluster should be able to detect the failure and recover by promoting an eligible node without requiring to manually reconfigure the entire database cluster.

Stable Connection Endpoint

We do not want application configuration to contain the address of a specific PostgreSQL server and require that address to be changed after every failover.

Instead, the application should connect to a stable endpoint while the infrastructure handles which PostgreSQL node currently receives the connection.

No Single Point of Failure

If database traffic passes through another component, that component can become a new single point of failure.

For example, having three PostgreSQL nodes does not provide end to end availability if all application traffic depends on one proxy server.

The connection layer therefore also needs redundancy so that the failure of a single proxy does not make the database unreachable.

Why Replication Is Not Enough

PostgreSQL provides replication mechanisms that allow data to be replicated from one instance to another.

This gives us an important foundation for high availability.

But having a replicated database does not automatically mean that the application can continue operating.

Consider a simple setup:

Under normal conditions, the application connects to the primary while the replica receives replicated data.

Now imagine that the primary suddenly fails:

The replica may still contain the database data, but it does not automatically solve the entire failure.

We still need a mechanism to determine that the primary is no longer available and decide what should happen next.

A Replica Is Not Automatically the New Primary

Replication gives us another copy of the data, but the cluster still needs to coordinate the transition from the failed primary to another node.

Without an automated mechanism, we need to:

Detect the failure
Determine whether the primary is really unavailable
Promote a suitable replica
Reconfigure clients to use the new primary
Re establish replication for the remaining nodes

The data may still be available, but the database service itself is not automatically recovered.

The Connection Problem

There is another problem after a failover.

Suppose the application connects directly to the primary's address:

After the primary fails and another node becomes primary:

The new primary may be ready to accept writes activity, but the application is still trying to connect to the old address.

This means that database failover and application connectivity are two different problems.

We need the database cluster to recover and provide a consistent way for clients to reach whichever node is currently serving as primary.

From Replication to High Availability

This shows why a highly available PostgreSQL system requires more than PostgreSQL replication alone.

We need additional components to coordinate the database nodes, manage the transition when a primary fails, and provide a reliable connection path for applications.

The architecture therefore needs to solve three different concerns:

Instead of treating the cluster as a collection of PostgreSQL servers, we separate it into two main layers:

Database layer, responsible for storing data and managing PostgreSQL availability
Connection layer, responsible for providing a stable path from applications to the active PostgreSQL node

The architecture consists of five components:

PostgreSQL
Patroni
etcd
HAProxy
Keepalived

Each component has a specific responsibility.

PostgreSQL

PostgreSQL is the database engine and the component that stores the application data.

We will run PostgreSQL on three separate virtual machines:

The three PostgreSQL instances form the database cluster.

One node acts as the current primary, while the other nodes act as replicas.

The primary handles write operations, while the replicas maintain copies of the database that can be used during failover.

Patroni

Patroni runs alongside PostgreSQL on each database node and manages the PostgreSQL instances as a high availability cluster.

Its responsibilities include:

Monitoring PostgreSQL health
Managing the PostgreSQL cluster state
Coordinating leader election
Promoting a replica when the primary fails
Managing PostgreSQL configuration required for the HA setup

etcd

etcd act as distributed system where cluster state can be stored and coordinated between the nodes.

Each database VM runs an etcd member:

The etcd members form a distributed consensus cluster.

Patroni uses etcd to store and coordinate information about the PostgreSQL cluster, including which node currently holds the leader role.

HAProxy

HAProxy monitors the PostgreSQL nodes and routes client connections to the node that is currently eligible to receive database traffic.

This means the application does not need to know which PostgreSQL VM is currently the primary.

In this project we will deploy two HAProxy instances:

Keepalived

Using two HAProxy instances introduces another problem.

If the application connects directly to one HAProxy server and that server fails, the PostgreSQL cluster may still be healthy but the application can no longer reach it.

We therefore need a way to make the HAProxy layer highly available as well.

Keepalived provides this through a Virtual IP (VIP).

Under normal conditions, one HAProxy node owns the VIP and handles client connections.

If that node fails, Keepalived can move the VIP to the other HAProxy node.

The application therefore continues using the same database endpoint.

Putting the Architecture Together

With each component assigned to a specific responsibility, the complete architecture for this project looks like this:

Each layer addresses a different failure scenario.

If a PostgreSQL primary fails, Patroni manages the database failover.
If the active HAProxy node fails, Keepalived moves the VIP to the other proxy.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

Github Repository: https://github.com/muhammadyulasfipahrizal/postgres-ha.git

Building the PostgreSQL HA Cluster

The database layer consists of three VM

Each VM has its own PostgreSQL instance, its own Patroni process, and one member of the etcd cluster. The three nodes then work together to provide a PostgreSQL cluster with automatic failover.

PostgreSQL Nodes

At any given time, one node is the primary and the other nodes are replicas.

The primary handles write operations, while the replicas continuously receive changes from the primary.

For example, the initial state might look like:

VM1 -> Primary
VM2 -> Replica
VM3 -> Replica

After a failure, the cluster may transition to:

VM1 -> Failed
VM2 -> Primary
VM3 -> Replica

The node roles are therefore part of the cluster's dynamic state rather than something we manually configure once and leave unchanged.

PostgreSQL Replication

The PostgreSQL instances use streaming replication to maintain copies of the database across the nodes.

This gives the cluster multiple copies of the database while allowing the primary to continue serving write traffic.

The replication settings are defined through Patroni's configuration. For example:

postgresql:
  listen: 0.0.0.0:5432
  connect_address: ${SERVER_IP}:5432

  parameters:
    hba_file: /etc/patroni/pg_hba.conf
    unix_socket_directories: '/var/run/postgresql'

The cluster is also configured with parameters required for streaming replication:

postgresql:
  parameters:
    wal_level: replica
    hot_standby: "on"
    wal_keep_size: 256MB
    max_wal_senders: 10
    max_replication_slots: 10

These settings are part of the Patroni configuration rather than being managed independently on each PostgreSQL instance.

etcd Cluster

Alongside PostgreSQL, each database VM runs an etcd member.

The three etcd members form a distributed consensus cluster.

Using three members is important because the cluster can tolerate the loss of one member while still maintaining a quorum.

Patroni uses this cluster to coordinate the state of the PostgreSQL nodes.

The etcd endpoints are configured in patroni.yml:

etcd3:
  hosts:
    - 192.168.122.1:2379
    - 192.168.122.2:2379
    - 192.168.122.3:2379

Patroni also defines the cluster scope and namespace used to store its distributed state:

scope: postgres-ha
namespace: /service/
name: postgres1

The name value is node specific, while the cluster scope and namespace identify the Patroni cluster. On the other nodes, the node name and server address change accordingly.

Patroni on Each Node

Each PostgreSQL node also runs a Patroni instance. Patroni connects the PostgreSQL instances with the distributed cluster state provided by etcd.

Each Patroni instance monitors its local PostgreSQL server and participates in managing the cluster.

An important part of the configuration is the timing used for leader management:

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10

These values control how Patroni interacts with the distributed configuration store and how frequently it checks the cluster state.

Cluster Configuration

The repository keeps the database configuration inside the db-server folder.

The Docker Compose configuration mounts the PostgreSQL data directory and the Patroni configuration into the database container:

services:
  db:
    build:
      context: .
    restart: unless-stopped

    ports:
      - "5432:5432"
      - "8008:8008"

    volumes:
      - ./data:/var/lib/postgresql/data
      - ./config/patroni.yml:/etc/patroni/patroni.yml:ro
      - ./config/pg_hba.conf:/etc/patroni/pg_hba.conf:ro

This keeps the database configuration version controlled while allowing the PostgreSQL data itself to remain in its persistent volume.

Starting the Cluster

Once the three database nodes are configured, we can start the services across the VMs. We can verify the resulting cluster state through Patroni before moving to the next layer.

The cluster should expose a state similar to:

The HAProxy Layer

The PostgreSQL cluster can now handle database level failover, but there is still a connection problem.

HAProxy receives PostgreSQL connections on port 5432 and forwards them to the PostgreSQL node that is currently serving as the primary.

This allows the application to use a single database endpoint while the cluster manages which PostgreSQL node is active.

HAProxy in TCP Mode

PostgreSQL uses its own protocol, so HAProxy is configured in TCP mode rather than HTTP mode.

The frontend listens for incoming PostgreSQL connections:

frontend postgres
    bind *:5432

    default_backend postgres_primary

The application therefore only needs to connect to the HAProxy endpoint using the normal PostgreSQL port.

Detecting the Current Primary

We cannot simply configure HAProxy to always send traffic to one server because that server may become unavailable.

Instead, HAProxy uses Patroni API to check the role of each PostgreSQL node.

backend postgres_primary
    mode tcp

    option httpchk GET /primary
    http-check expect status 200

    server postgres1 192.168.122.1:5432 check port 8008
    server postgres2 192.168.122.2:5432 check port 8008
    server postgres3 192.168.122.3:5432 check port 8008

The HTTP request is only used as a health check to ask Patroni whether the node is currently the primary.

Patroni /primary endpoint returns HTTP 200 when the node is the current primary. HAProxy uses that response to determine which backend server should receive write traffic.

The Routing Flow

HAProxy periodically checks the Patroni endpoint on each database node.

Only the node reporting itself as the current primary passes the /primary health check.

The result is that HAProxy can automatically direct new connections to the current primary without requiring configuration change.

Adding Keepalived

Keepalived uses VRRP (Virtual Router Redundancy Protocol) to manage VIP (virtual ip) between the two proxy servers.

For example:

Under normal conditions, HAProxy 1 owns the VIP.

The application therefore connects to:

192.168.122.200:5432

rather than:

192.168.122.10:5432

Choosing the Active Proxy

Keepalived assigns a priority to each node to determine which node should normally own the VIP.

The node with the higher priority becomes the active VRRP node and owns the VIP. If the active node becomes unavailable, the other node can take ownership of the VIP.

This allows the connection endpoint to remain unchanged even though the physical server handling the traffic changes.

Monitoring HAProxy

Simply checking whether the server itself is reachable is not enough.The proxy node could still be running while HAProxy itself has stopped.

In this situation, Keepalived should not continue advertising the VIP from that node.

The project therefore uses a health check script:

#!/bin/bash

if docker ps --format '{{.Names}}' | grep -qx haproxy; then
    exit 0
else
    exit 1
fi

The script checks whether the HAProxy container is running and returns a success or failure status accordingly.

Keepalived then tracks this script:

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    timeout 2
    fall 2
    rise 2
    weight -50
}

vrrp_instance VI_1 {
    state MASTER
    interface ens3

    virtual_router_id 51
    priority 110
    advert_int 1

    virtual_ipaddress {
        192.168.122.200/24
    }

    track_script {
        check_haproxy
    }
}

If the HAProxy check fails repeatedly, the configured negative weight reduces the node's VRRP priority.

This allows the other Keepalived node to become the owner of the VIP.

HAProxy Failure

Suppose the initial state is:

When HAProxy 1 fails, Keepalived detects the HAProxy failure through the tracking script.

The VIP can then move to HAProxy 2:

The application does not need to change its database configuration, It continues connecting to the same address

The physical proxy handling the connection has changed, but the application's endpoint has not.

Testing the Setup

With the complete architecture connected, we can now test whether the system actually behaves as expected.

We will test five scenarios:

Normal cluster state
PostgreSQL primary failure
HAProxy failure
VIP failover

Normal Cluster

Before introducing any failures, we first verify that all components are operating normally.

We can check the Patroni cluster state with:

patronictl -c /etc/patroni/patroni.yml list

We should see one node identified as the Leader and the remaining nodes as replicas, in my setup VM2 is currently the Leader.

We can also verify that HAProxy recognizes the current primary through its statistics page or by checking the backend state.

Finally, we verify that the VIP is currently owned by the active HAProxy node.

ip addr show <network-interface>

The expected result is that the VIP is present on the active proxy.

PostgreSQL Failure

Next, we intentionally stop the current PostgreSQL primary.

For example, if VM2 is currently the primary, we can stop the PostgreSQL container in VM2:

docker stop <postgres-container>

After the failure, VM2 should no longer be available as the primary:

Before:
VM1 -> REPLICA
VM2 -> PRIMARY
VM3 -> REPLICA


After:
VM1 -> PRIMARY
VM3 -> REPLICA

We can then check the Patroni cluster again:

patronictl -c /etc/patroni/patroni.yml list

At this point, the cluster should detect that the previous primary is unavailable and begin the leader transition.

Patroni should promote an eligible replica to become the new primary, here VM1 becomes the new primary.

We can also verify the Patroni REST API directly:

curl -i http://192.168.122.28:8008/primary

A successful primary check should return:

HTTP/1.1 200 OK

This confirms that the database layer has completed the failover.

HAProxy Detects the New Primary

Once Patroni has promoted the new primary, HAProxy should discover the role change through its health checks.

HAProxy Failure

First, we identify which HAProxy node currently owns the VIP.

ip addr show <network-interface>

In my setup VM4 currently owns the VIP:

We then stop HAProxy on VM4:

docker stop haproxy

Because the current Keepalived health check monitors the HAProxy container, the check should fail after the configured number of failed checks.

Keepalived should then reduce the active node's VRRP priority and allow the other proxy to take ownership of the VIP.

VIP Failover

After HAProxy on VM4 fails, the VIP is now on VM5.

The application still uses the same endpoint:

<VIP>:5432

This demonstrates that the proxy layer can fail independently from the PostgreSQL layer.

Conclusion

Building a highly available PostgreSQL environment is not about simply running multiple database servers. It is about designing the system so that individual failures do not automatically become application outages.

In the setup, we combined several components, each responsible for a different part of the availability problem:

PostgreSQL provides the database layer and streaming replication
Patroni manages cluster membership, leader election, and automatic failover
etcd provides the distributed state required by Patroni
HAProxy provides a stable database endpoint and routes connections to the current primary
Keepalived provides a virtual IP so that the HAProxy layer does not depend on a single server

The important lesson is that high availability is not achieved by adding redundancy alone. Every component needs to have a clear role in detecting failures, making decisions, and redirecting traffic when something goes wrong.

This project is intentionally built as a lab environment, but the concepts can be applied to more complex production deployments.

You can reproduce the entire setup and experiment with the failure scenarios yourself by cloning my GitHub repository below:

https://github.com/muhammadyulasfipahrizal/postgres-ha.git

How I Built End To End Observability Stack with OpenTelemetry, Prometheus, Grafana, Loki, and Jaeger

Pahri — Sat, 05 Sep 2026 05:00:39 +0000

When an application is running, knowing that it is up does not necessarily mean knowing what it is doing.

For example, imagine an API request that takes several seconds to complete. We would want to answer questions such as:

Is the application receiving more requests than usual?
What happened during the request?
How long does the request take?
Did the application encounter an error?
Which part of the request should we investigate?

Instead of looking at behavior through a single source, we should collect different types of signal and make them available in one observability stack.

What We Need to Observe

To understand the application from different perspectives, we need three types of telemetry: metrics, logs, and traces.

Each signal provides a different view of the application, and each becomes useful at a different stage of an investigation.

Metrics

Metrics helps us to understand the overall behavior of the application.

How many requests are being processed?
Are request rates increasing?
Are errors becoming more frequent?
Is the application behaving differently from normal?

This gives us the high level view we need to detect if something might be wrong.

Logs

Logs give us details about individual events inside the application, such as a request starting, an operation completing, or an error occurring. They become especially useful when we need to understand the context behind an abnormal metric.

Traces

A trace allows us to follow a request through its execution and understand its timing. This becomes particularly useful when investigating slow requests or determining which part of an operation deserves further attention.

Centralize Collection

Instead of making the application communicate independently with several observability backends, we can place a telemetry collection layer between the application.

This provide a consistent way to receive telemetry from the application and route it to the systems responsible for storing and analyzing each signal.

Visualization

Jumping between different interfaces makes investigation slower, especially when we need to correlate information from multiple signals.

Therefore we need a visualization layer where metrics, logs, and traces can be explored together.

Architecture

Now that we know what we need to collect, the next question is how these signals should move through the system.

The architecture looks like this:

The application is the source of the telemetry. When the application generates metrics, logs, and traces, these signals are sent to the OpenTelemetry Collector using OTLP.

The Collector then routes each signal to the appropriate backend.

Metrics are exposed for Prometheus to collect, logs are sent to Loki, and traces are forwarded to Jaeger. Grafana sits on top of these systems and gives us a single place to explore the data.

This creates a clear separation between the application and the systems that store and analyze its telemetry.

Application to Collector

Consider a request entering the API:

GET /api/orders

The application processes the request and generates telemetry while doing so.

Instead of sending that telemetry directly to multiple backends, the application sends it to the Collector The application therefore only needs to know where the telemetry collection layer is located.

This becomes especially useful as the observability stack grows. Additional processing or exporters can be introduced at the Collector without requiring the application to establish a new connection to every backend.

Collector to Backends

Once telemetry reaches the Collector, the signals follow different paths depending on their type.

Each backend therefore has a specific responsibility.

Prometheus handles the metrics data, Loki handles application logs, and Jaeger handles traces.

From Backends to Grafana

Once the telemetry reaches its respective backends, we need a way to explore and make sense of the data.

Grafana provides this layer by connecting to Prometheus, Loki, and Jaeger, giving us a single interface to query and visualize the telemetry generated by our application.

The Application

Before sending any telemetry, we need an application that can generate it.

For this project, we use a small Go API as the application being observed. The API exposes several endpoints that represent different application behaviors:

GET /health
GET /api/orders
GET /api/transaction
GET /api/users

Each endpoint gives us a different scenario to observe.

/health represents a simple successful request
/api/orders returns application data
/api/transaction simulates a slow operation
/api/users intentionally returns an error

This gives us enough variety to see how metrics, logs, and traces behave when we test the observability stack later.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

Github Repository

Initializing Opentelemetry

The telemetry setup is separated from the HTTP handlers in telemetry.go.

The application first defines its service identity:

res, err := resource.New(
    ctx,
    resource.WithAttributes(
        semconv.ServiceName("observability-api"),
    ),
)

This gives the telemetry a consistent service name: observability-api.

More importantly, each telemetry provider is configured to send its data to the OpenTelemetry Collector:

otlptracegrpc.WithEndpoint("otel-collector:4317")

The same Collector endpoint is used for metrics and logs.

This means the application does not need to know where Prometheus, Loki, or Jaeger are running. It only needs to know the address of the Collector.

The trace, metric, and log providers are then registered globally in the application.

Instrumenting HTTP Requests

The next step is making the HTTP server itself observable.

mux.HandleFunc("/health", health)
mux.HandleFunc("/api/orders", orders)
mux.HandleFunc("/api/transaction", transaction)
mux.HandleFunc("/api/users", users)

Instead of passing this multiplexer directly to the HTTP server, we wrap it with OpenTelemetry's HTTP instrumentation:

handler := otelhttp.NewHandler(
    mux,
    "HTTP Server",
)

The resulting handler is then used by the HTTP server:

server := &http.Server{
    Addr:    ":8080",
    Handler: handler,
}

This instrumentation sits in front of all four endpoints:

As requests pass through this layer, OpenTelemetry can capture HTTP request telemetry and associate it with the request context.

Logging Inside Each Endpoint

For application specific events, we use a small helper called emitLog.

The helper receives the request context, severity, message, and additional attributes:

func emitLog(
    ctx context.Context,
    severity log.Severity,
    message string,
    attrs ...attribute.KeyValue,
) {
    ...
    logger.Emit(ctx, record)
}

The important part is the context:

logger.Emit(ctx, record)

Each handler passes r.Context() into emitLog, so the log is emitted using the context of the current HTTP request.

The endpoint therefore does not need to know how the log will eventually reach Loki. It only creates the log event through the OpenTelemetry logging API.

`/health`

The simplest example is the health endpoint.

When we call:

GET /health

the handler generates an informational log:

emitLog(
    r.Context(),
    log.SeverityInfo,
    "health check",
    attribute.String("endpoint", "/health"),
)

The application therefore produces telemetry as part of processing the request rather than requiring a separate logging pipeline inside the handler.

`/api/orders`

The orders endpoint follows the same pattern, but adds more context to the log.

After creating the orders, the handler emits:

emitLog(
    r.Context(),
    log.SeverityInfo,
    "orders fetched successfully",
    attribute.String("endpoint", "/api/orders"),
    attribute.Int("order_count", len(orders)),
)

Now the telemetry contains not only the event itself, but also structured attributes such as the endpoint and number of orders.

For example:

message:     orders fetched successfully
endpoint:    /api/orders
order_count: 3

These attributes become useful later when we query the logs with Loki.

`/api/transaction`

The transaction endpoint gives us a more interesting example because it intentionally takes three seconds to complete.

First, the handler records the start of the operation:

start := time.Now()

emitLog(
    r.Context(),
    log.SeverityInfo,
    "transaction started",
    attribute.String("endpoint", "/api/transaction"),
)

After the simulated operation:

time.Sleep(3 * time.Second)

the handler calculates the duration and emits another log:

duration := time.Since(start)

emitLog(
    r.Context(),
    log.SeverityInfo,
    "transaction completed",
    attribute.String("endpoint", "/api/transaction"),
    attribute.Int64("duration_ms", duration.Milliseconds()),
)

`/api/users`

Finally, we have an endpoint that intentionally returns an error.

emitLog(
    r.Context(),
    log.SeverityError,
    "failed to fetch users",
    attribute.String("endpoint", "/api/users"),
    attribute.Int("status_code", http.StatusInternalServerError),
)

The request therefore produces an error-level log before returning HTTP 500.

This gives us a different scenario from /api/transaction: instead of investigating latency, we can later use the observability stack to investigate an application error.

Building the Observability Pipeline

With the application ready, we can now configure how that telemetry is processed and delivered to each backend.

Configuring the OpenTelemetry Collector

The Collector accepts OTLP through gRPC:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

The Collector then uses the batch processor:

processors:
  batch:

Rather than defining one large pipeline for every signal, the configuration separates metrics, logs, and traces:

service:
  pipelines:
    metrics:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - prometheus

    traces:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - otlp/jaeger

    logs:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - otlphttp/loki

Collector to Prometheus

Metrics use the Prometheus exporter:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

The Collector exposes the metrics endpoint on port 8889.

This is slightly different from the way telemetry enters the Collector.

The application sends telemetry to the Collector, but Prometheus does not receive metrics through a push connection. Instead, Prometheus periodically scrapes the endpoint exposed by the Collector.

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "otel-collector"
    static_configs:
      - targets:
          - "otel-collector:8889"

Prometheus therefore scrapes the Collector every five seconds.

Collector to Loki

Logs use the otlphttp/loki exporter:

otlphttp/loki:
  endpoint: http://loki:3100/otlp

The Collector sends the processed logs to Loki through its OTLP HTTP endpoint.

This works particularly well with the structured information we added to the application logs.

For example, the /api/orders endpoint produces:

message:     orders fetched successfully
endpoint:    /api/orders
order_count: 3

Instead of treating the log as only a text message, these additional attributes provide context that can be queried later.

Loki is configured to support structured metadata:

limits_config:
  allow_structured_metadata: true
  volume_enabled: true

The project also uses filesystem storage for Loki, backed by the loki-data Docker volume.

Collector to Jaeger

Traces use a separate OTLP exporter:

otlp/jaeger:
  endpoint: jaeger:4317
  tls:
    insecure: true

The traces pipeline connects this exporter to the OTLP receiver and batch processor.

Jaeger is configured to receive OTLP through gRPC:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

For this project Jaeger uses an in memory backend:

extensions:
  jaeger_query:
    storage:
      traces: memory

  jaeger_storage:
    backends:
      memory:
        memory:
          max_traces: 100000

The trace exporter writes into that memory storage:

exporters:
  jaeger_storage_exporter:
    trace_storage: memory

Running the Pipeline

All of these components are defined as separate Docker Compose services.

The Collector mounts our configuration file into the container and exposes its OTLP and Prometheus endpoints:

otel-collector:
  image: otel/opentelemetry-collector-contrib:latest
  command:
    - "--config=/etc/otelcol-contrib/otel-collector.yml"
  volumes:
    - ./otel/otel-collector.yml:/etc/otelcol-contrib/otel-collector.yml:ro
  ports:
    - "4317:4317"
    - "8889:8889"

The services are connected through the research-observability Docker network, allowing the Collector to reach services such as loki and jaeger directly by their Docker service names.

Bringing Everything Together with Grafana

grafana:
  image: grafana/grafana:latest
  container_name: grafana
  environment:
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=admin
    - GF_AUTH_ANONYMOUS_ENABLED=false
  volumes:
    - grafana-data:/var/lib/grafana
  ports:
    - "3001:3000"

The container exposes Grafana on port 3001 of the host, while Grafana itself listens on port 3000 inside the container.

Because Grafana, Prometheus, Loki, and Jaeger are connected to the same research-observability Docker network, Grafana can communicate with the other services using their Docker service names.

This means we can configure the data sources using addresses such as:

http://prometheus:9090
http://loki:3100
http://jaeger:16686

Adding Prometheus as a Data Source

From the Grafana dashboard, open:

Connections → Data sources

Then select Add data source and choose Prometheus.

For the server URL, we use the Docker service name instead of localhost:

http://prometheus:9090

After entering the URL, select Save & test.

If the connection is successful, Grafana confirms that it can communicate with Prometheus.

Adding Loki

We repeat the same process for Loki.

Go to:

Connections → Data sources → Add data source

Then select Loki.

For the URL, use:

http://loki:3100

After entering the URL, select Save & test.

Once Grafana confirms the connection, Loki is available as a data source for exploring application logs.

Adding Jaeger

Connections → Data sources → Add data source

select Jaeger.

For the URL, use Jaeger's query endpoint:

http://jaeger:16686

This is different from the OTLP endpoint used by the Collector.

The Collector sends traces to Jaeger through its OTLP gRPC endpoint, while Grafana connects to Jaeger's query interface to retrieve traces.

After entering the URL, select Save & test.

At this point, Grafana has access to all three observability backends.

Verifying the Data Sources

Before building any dashboard, we can verify that Grafana can actually query the data.

Open:

Explore

From the data source selector, we can choose between:

Prometheus
Loki
Jaeger

For example, selecting Prometheus allows us to run a metrics query directly from Grafana. This gives us a simple way to verify each connection before creating dashboards.

Investigation

Now that Grafana is connected to our observability backends, we can use the stack to investigate actual application problems.

Instead of simply checking whether telemetry is being collected, we want to answer a more practical question:

What happens when something goes wrong?

For this example, we will investigate /api/transaction

A Slow Transaction

As mention before our /api/transaction endpoint intentionally waits for three seconds before returning a response:

time.Sleep(3 * time.Second)

From the users perspective, the problem is simple: the API feels slow.

However, knowing that a request is slow does not immediately tell us why.

We start by sending a request to:

GET /api/transaction

After generating several requests, we can open our Grafana dashboard and look at the request duration.

The metric tells us that the request is taking significantly longer than a normal API request.

But metrics alone do not provide much context about what happened inside the request.

This is where we move to the logs.

Looking at the Logs

In Grafana, we can open the Loki data source and search for logs generated by the transaction endpoint.

The application produces two important events:

transaction started
transaction completed

The completion log also contains the duration:

message:     transaction completed
endpoint:    /api/transaction
duration_ms: 3000

Now we know that the request really did spend approximately three seconds processing the transaction.

The logs give us more context than the metric alone, but we still do not have a detailed view of the request execution.

Following the Trace

The same request also generates a trace that is exported through the OpenTelemetry Collector and stored in Jaeger.

From Grafana, we can open the corresponding trace and inspect its timeline.

The trace gives us a view of how long the request took and where that time was spent.

From Symptom to Cause

This scenario demonstrate the main reason we built the observability stack.

Each signal gives us a different level of information.

Metrics help us detect that something unusual is happening.
Logs give us additional application context.
Traces allow us to follow the request and investigate its execution.

The important part is not simply having all three signals available. The real value comes from being able to use them together when investigating a problem.

Conclusion

Building this observability stack gave us a practical way to understand what is happening inside an application instead of only knowing whether it is running.

We started with a simple Go API and added OpenTelemetry to generate metrics, logs, and traces. The OpenTelemetry Collector then provided a central layer for processing and routing those signals to Prometheus, Loki, and Jaeger.

Grafana brought everything together into a single interface.

The /api/transaction example showed why having multiple signals matters. Metrics helped us identify that the request was slow. Logs provided additional context about the transaction, while the trace allowed us to inspect the request execution and its timing.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

GitHub Repository: https://github.com/muhammadyulasfipahrizal/E2E-observability-stack.git

[Boost]

Pahri — Wed, 02 Sep 2026 06:44:45 +0000

Pahri

Sep 2

A Practical Approach To Securing API With Cloudflare WAF

8 min read

A Practical Approach To Securing API With Cloudflare WAF

Pahri — Wed, 02 Sep 2026 06:22:39 +0000

Building an API is relatively easy. The harder problem starts when the API becomes publicly accessible.

Once an API is exposed to the Internet, it can receive much more than the requests we expect. Attackers can send malicious requests, repeatedly hit endpoints, use unexpected HTTP methods, send suspicious query parameters, or simply generate enough traffic to consume application resources.

A common approach is to handle all of these concerns inside the application. The application receives the request, validates it, checks authentication, and finally decides whether the request should be processed.

But this raises another question:

What if we could reject unwanted requests before they reach the application?

This is where a Web Application Firewall (WAF) can help. A WAF is a security layer that inspects incoming web traffic and applies rules to determine whether a request should be allowed, challenged, or blocked.

Consider a simple request to an orders API:

GET /api/orders?id=1
Authorization: Bearer <token>

Without an edge security layer, the request travels directly to the application:

With an Web Application Firewall such as Cloudflare WAF, the request can be evaluated before it reaches the application:

If a request violates one of the configured policies, Cloudflare can reject it instead of forwarding it to the application.

This is the idea behind an edge security layer.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below.

Github Repository: https://github.com/muhammadyulasfipahrizal/WAF-API.git

What We Are Building

In this project, we will build a small Go API and place Cloudflare WAF in front of it to demonstrate several edge security controls.

The application is intentionally simple. It provides enough endpoints and request patterns to demonstrate different WAF policies without introducing unnecessary application complexity.

The application exposes three endpoints:

/api/login
/api/orders
/admin

API

The /api/login endpoint is used to authenticate a user. For this demo, valid credentials return a JWT that can then be used when accessing /api/orders.

The /api/orders endpoint represents a simple API that returns order data. It supports several query parameters:

id
item
price
status

These parameters give us different request patterns to test against the WAF.

The /admin endpoint represents an administrative page that should not be publicly accessible.

Security Controls

The WAF configuration demonstrates the following protections:

Security Control	Purpose
Main page challenge	Challenge requests when accessing the application
Authorization header enforcement	Reject protected API requests without an `Authorization` header
HTTP method filtering	Allow only expected methods for an endpoint
IP restriction	Restrict access to the restricted page
Suspicious request filtering	Block requests containing suspicious patterns
Rate limiting	Limit excessive request traffic
API schema validation	Ensure requests follow the defined API contract

Project Architecture

Cloudflare sits between the Internet and the application. This allows incoming requests to be evaluated against edge level security policies before they reach the origin.

For example, since /api/orders is a read only endpoint, requests using unsupported methods such as POST can be rejected by Cloudflare before they are forwarded to the application:

The responsibility is divided between two layers:

Layer	Responsibility
Cloudflare WAF	Challenge, rate limiting, method filtering, IP restrictions, suspicious request filtering, schema validation
Go API	JWT authentication, application validation, business logic

Passing the WAF does not mean that a request is trusted. It only means that the request passed the configured edge level policies.

Application Structure

cmd directory contains the application entry point, while internal contains the authentication, middleware, and HTTP handlers.
schema directory contains the OpenAPI definition used to describe the API contract and demonstrate schema validation with Cloudflare.
web directory contains the frontend used to interact with the API and test the WAF scenarios.
The application is containerized using the Dockerfile, and
.github/workflows is used to build and publish the Docker image with Github Action.

Cloudflare WAF

Cloudflare WAF provides several mechanisms that can be used to enforce policies at the edge. In this project, each rule is designed to address a specific type of unwanted request.

Challenge the Main Page

The first rule protects the main application page with an interactive challenge.

(http.request.uri.path eq "/")

When accessing /, the client must pass the Cloudflare challenge before the request continues to the application.

This rule prevent automated clients from accessing the application page without first passing a basic verification step. This is useful for reducing unwanted bot traffic before it reaches the origin.

Require Authorization Header for APIs

Protected API endpoints should not receive requests that do not contain an Authorization header.

The login endpoint is excluded because clients need to access it before they have a token.

starts_with(http.request.uri.path, "/api/")
and not any(http.request.headers["authorization"][*] ne "")
and http.request.uri.path ne "/api/login"

The behavior is:

This does not replace authentication.

The WAF checks for the presence of the Authorization header. The Go application validates the JWT itself.

For example, a request containing:

Authorization: Bearer <token>

may pass the WAF rule, but the application still needs to determine whether the token is valid, expired, and properly signed.

This provides a simple defense-in-depth mechanism: the edge performs an initial check, while the application performs the actual authentication.

Enforce HTTP Methods

The /api/orders endpoint is designed as a read-only API, so only GET requests are expected.

(http.request.uri.path eq "/api/orders" 
and http.request.method ne "GET")

The behavior is:

The reasoning is straightforward:

The /api/orders endpoint is designed as a read-only API, so unsupported write methods provide no value.

Protect Admin Page By IP

The /admin page represents an administrative interface and should not be publicly accessible.

For this project, access is restricted to a specific trusted IP address.

(http.request.uri.path eq "/admin" 
and ip.src ne ADMIN_IP)

The flow is:

This provides a simple way to restrict an administrative endpoint at the edge.

Block Suspicious Requests

This rule filters API requests containing patterns commonly associated with SQL injection attempts.

starts_with(http.request.uri.path, "/api/")
and (
  lower(http.request.uri.query) contains " or "
  or lower(http.request.uri.query) contains "' or "
  or lower(http.request.uri.query) contains "union select"
  or lower(http.request.uri.query) contains "select "
  or lower(http.request.uri.query) contains "drop table"
  or lower(http.request.uri.query) contains "insert into"
  or lower(http.request.uri.query) contains "delete from"
  or lower(http.request.uri.query) contains "--"
  or lower(http.request.uri.query) contains "/*"
  or lower(http.request.uri.query) contains "*/"
)

For example:

GET /api/orders?id=1' OR '1'='1

can be detected as suspicious and therefore blocked by Cloduflare WAF.

Application-level protection is still required, including input validation, parameterized queries, and secure database practices.

Rate Limiting

An API can also be abused through repeated requests.

Even when individual requests are valid, a large number of requests can consume CPU, memory, bandwidth, connection pools, and other application resources.

Cloudflare can apply rate limiting before this traffic reaches the origin.

Excessive requests can be stopped before they consume resources at the origin.

This is particularly useful when the origin has limited resources or when an endpoint is frequently targeted by bots.

API Schema Validation

An API schema describes what a valid API request should look like.

For example, the /api/orders endpoint define the id query parameter as an integer:

/api/orders:
  get:
    parameters:
      - name: id
        in: query
        schema:
          type: integer

A request such as:

GET /api/orders?id=10
matches the expected type.

The OpenAPI definition can be uploaded to Cloudflare WAF and used for API schema validation.

The flow is:

This is useful because the API contract can become part of the security boundary.

Instead of being used only as documentation, the OpenAPI definition can also help Cloudflare determine whether incoming requests conform to the expected API structure.

Results

After deploying the Go API behind Cloudflare WAF, the configured policies were tested against different request scenarios.

The following results show the behavior of each control.

Challenge the Main Page

Accessing / triggers the configured Cloudflare interactive challenge.

After successfully passing the challenge, the request continues to the application.

Require Authorization Header for APIs

A request to a protected API without an Authorization header is rejected:

GET /api/orders?id=1

A request containing the header can continue to the application, where the JWT is validated.

Enforce HTTP Methods

The /api/orders endpoint accepts GET, while unsupported methods are rejected by Cloudflare.

For example:

POST /api/orders
→ Blocked

Protect the Admin Page by IP

Requests to /admin from the configured administrator IP are allowed, while requests from other IP addresses are blocked.

Block Suspicious Requests

A request containing a suspicious SQL-like pattern is rejected by the WAF.

For example:

GET /api/orders?id=1' OR '1'='1

is detected and blocked by the configured rule.

Rate Limiting

Repeated requests eventually trigger the configured rate-limiting policy.

Normal traffic continues to the origin, while traffic exceeding the configured policy is stopped by Cloudflare.

API Schema Validation

Requests that conform to the OpenAPI contract can reach the origin, while requests that do not match the defined schema can be rejected.

For example:

GET /api/orders?id=10

matches the expected integer type for id.

Conclusion

Securing an API is not only about adding authentication to the application.

For an Internet-facing service, there is value in having an additional security layer between the client and the origin. Cloudflare WAF can inspect and control incoming traffic at the edge, helping protect the application from a wide range of unwanted or potentially malicious requests before they reach the origin.

In this project, we demonstrate several of these capabilities, including challenges, rate limiting, HTTP method restrictions, IP restrictions, suspicious request filtering, and API schema validation.

The important lesson from this project is that security responsibilities do not have to live entirely inside the application.

The edge can handle policies that are suitable for inspection before the request reaches the origin, while the application remains responsible for authentication, business logic, and application level validation.

The code snippets in this article focus on the important parts of the implementation. For the complete source code, you can find it through repository below.

Github Repository

How I Built A Blue Green Deployment Pipeline With Jenkins

Pahri — Sat, 29 Aug 2026 07:26:43 +0000

Deploying a new application version can be simple when the application is small and downtime is acceptable.

An engineer can stop the current application, deploy the new version, and start it again. The problem appears when the application is expected to remain available while a new version is being released.

This creates a window where the application may become unavailable.

The problem becomes more visible when the new version also has an issue.

A deployment might complete successfully from the infrastructure's perspective, but the application itself could still be unhealthy. A container may start but fail its health check, the application may not respond correctly, or a configuration problem may only become visible after the new version receives traffic.

In this situation, simply replacing the old version is risky.

Deployment process needs a way to prepare the new version without immediately exposing it to users, verify that it is working, and only then switch traffic to it.

This is where blue-green deployment becomes useful.

Instead of replacing the running application directly, two environments are maintained:

Blue, the currently active version receiving production traffic
Green, the new version being prepared and validated

The new release is deployed to the inactive environment first. Once it passes the required checks, the reverse proxy switches traffic from the active environment to the new one.

The new application version can be started and tested while the existing version continues serving users. If the new version fails validation, traffic does not need to move at all. The existing environment can simply remain active.

Now Kubernetes is one way to implement this kind of deployment strategies. It provides powerful primitives for managing containers, service discovery, traffic routing, health checks, scaling, and automated rollouts.

However, Kubernetes also introduces additional infrastructure and operational complexity.

Not every application needs that level of orchestration, and not every company wants to operate a Kubernetes cluster.

For smaller applications with simpler operational requirements, a combination of Docker, reverse proxy, and a CI/CD tool can provide a much simpler alternative.

By the end of this project, we will have a Jenkins based deployment workflow that demonstrates:

Automated application testing and image building
Blue-green application environments
Health checks before traffic switching
Reverse proxy based traffic switching
SSH based remote deployment
Safe deployment path

What We Are Building

The complete implementation is available in repository bellow:

Github Repository

The goal of this project is to build a simple deployment environment where Jenkins can deploy a new application version alongside the currently running version, verify it, and then switch traffic to it.

The architecture consists of a Jenkins server and a separate deployment server.

The deployment server runs the application containers and the reverse proxy, while Jenkins acts as the automation layer responsible for executing the deployment workflow.

Architecture

There are four main pieces involved:

Jenkins, orchestrates the deployment process.
Blue environment, runs one version of the application.
Green environment, runs the other version of the application.
Nginx, receives user traffic and determines which environment should receive it.

Application

To demonstrate the deployment workflow, For this project, we use a small Go HTTP application.

The application does not contain any complex business logic because the main focus of this project is the deployment process, not the application itself.

The application exposes two endpoints:

GET /health
GET /version

Health Endpoint

The /health endpoint is used to determine whether the newly deployed application is ready to receive traffic.

A successful response returns HTTP 200 OK:

GET /health

HTTP/1.1 200 OK

OK

Jenkins uses this endpoint after starting the new environment.

Version Identification

The second endpoint /version, allows us to identify which application version is currently serving the request.

The application reads the version from the APP_VERSION environment variable:

APP_VERSION=v1
APP_VERSION=v2

The endpoint then returns the configured version:

GET /version

v1

Or after deploying v2 release:

GET /version

v2

This makes the deployment behavior easy to observe.

Instead of simply checking whether the application responds, we can verify exactly which version is receiving traffic.

Jenkins

Jenkins is the automation engine for the deployment process.

Instead of manually connecting to the deployment server and executing commands such as:

docker build
docker compose up
curl /health

Jenkins executes these steps as a repeatable pipeline.

The pipeline implemented in this project performs the following stages:

The repository contains separate Jenkinsfiles for the two deployment directions:

deploy/
├── Jenkinsfile.blue
└── Jenkinsfile.green

This is because the deployment target alternates between the two environments.

For example, Jenkinsfile.blue deploys the new image to the Blue environment and then switches Nginx to Blue. Jenkinsfile.green does the same thing for Green.

The pipeline also uses the Jenkins build number as the Docker image tag:

environment {
    APP_NAME   = 'zero-downtime-app'
    IMAGE_TAG  = "${BUILD_NUMBER}"
    DEPLOY_DIR = '/home/ubuntu/projects/zero-downtime-jenkins'
}

This gives every Jenkins build its own image tag instead of continuously overwriting a generic latest image.

For example:

Build #1 → zero-downtime-app:1
Build #2 → zero-downtime-app:2
Build #3 → zero-downtime-app:3

This makes it easier to identify exactly which build is running in an environment.

Blue Environment

The Blue environment represents one side of the deployment pair.

In this project, the application runs inside a Docker container named:

app-blue

The application itself listens on port 8080 inside the container, while Docker publishes it on port 8081 on the deployment server.

The corresponding Docker Compose configuration is kept in:

deploy/docker-compose.blue.yml

This allows the Blue environment to be started independently from the Green environment.

When Blue is the active environment, Nginx routes application traffic to:

app-blue:8080

and is deployed using Jenkinsfile.blue.

Green Environment

The Green environment provides the second application slot.

It runs in a separate Docker container:

app-green

Like Blue, the application listens on port 8080 inside the container. Docker publishes it on a different host port:

The Green environment is defined by:

deploy/docker-compose.green.yml

When Green is the active environment, Nginx routes application traffic to:

app-green:8080

and is deployed using Jenkinsfile.green.

Reverse Proxy

In this project we use Nginx, who acts as the entry point for application traffic.

Users do not need to know whether Blue or Green is currently active. They simply connect to the application through Nginx.

server {
    listen 80;

    location / {
        proxy_pass http://app;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For           $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Nginx uses an upstream configuration to determine which container should receive the traffic.

For example, when Blue is active:

upstream app {
    server app-blue:8080;
}

And when Green has been successfully deployed and verified, Jenkins changes the upstream configuration:

upstream app {
    server app-green:8080;
}

The pipeline does not simply reload Nginx after changing the configuration.

It first validates the configuration:

docker exec app-nginx nginx -t

and only then reloads Nginx:

docker exec app-nginx nginx -s reload

This is important because an invalid Nginx configuration should not be allowed to replace the currently working configuration.

The actual traffic switching logic is implemented directly in the Jenkins pipeline.

SSH Connection

Jenkins and the deployment environment are intentionally separated.

The Jenkins server does not run the application containers. Instead, Jenkins connects to the deployment server through SSH and executes the required deployment commands there.

The pipeline uses two Jenkins credentials:

deploy-host
ssh-vm1

The deploy-host credential provides the deployment server address, while ssh-vm1 is used by the Jenkins SSH agent for authentication.

For example, when Jenkins needs to deploy the Green environment, it connects to the deployment server and executes Docker Compose there:

APP_IMAGE_TAG=${IMAGE_TAG} \
docker compose \
    -f docker-compose.green.yml \
    up -d

The same SSH connection is also used for testing, health verification, and traffic switching.

This means Jenkins acts as the orchestrator, while the deployment server remains responsible for running the actual workload.

Result

The deployment workflow was tested through three scenarios: deploying the initial version (Blue), deploying the new version (Green), and handling a failed Green deployment.

Deploying v1

The first deployment starts the application in the Blue environment.

Jenkins runs the test, builds the Docker image, deploys app-blue, and verifies its health endpoint:

[Pipeline] { (Deploy Blue)

Container app-blue Creating
Container app-blue Created
Container app-blue Starting
Container app-blue Started

The health check then succeeds:

curl --fail http://127.0.0.1:8081/health

{"status":"ok"}

After the initial deployment, a request through Nginx confirms that v1 is serving traffic:

curl http://localhost

{
  "message": "Hello",
  "version": "v1",
  "hostname": "5f6a41482e36"
}

At this point, the application is running successfully and Blue is the active environment.

Deploying v2

The next deployment uses the Green environment.

Jenkins first runs the application tests:

ok      jenkins-zero-downtime    0.002s

It then builds a new image:

zero-downtime-app:6

and starts the Green environment:

Container app-green Creating
Container app-green Created
Container app-green Starting
Container app-green Started

At this point app-blue is still running. The new version is therefore deployed without first stopping the existing application.

Verifying v2

Before changing production traffic, Jenkins checks the new environment directly:

curl --fail http://127.0.0.1:8082/health

The check succeeds:

{"status":"ok"}

This confirms that the new container is responding successfully before it is exposed through Nginx.

Switching Traffic

After the health check succeeds, Jenkins updates the Nginx upstream configuration to point to Green:

upstream app {
    server app-green:8080;
}

The configuration is validated before Nginx is reloaded:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Nginx is then reloaded successfully.

A request through the normal application entry point now returns v2:

curl http://localhost

{
  "message": "Hello",
  "version": "v2",
  "hostname": "9a86f0f3486"
}

The deployment is now complete, and Jenkins removes the old Blue container:

docker rm -f app-blue

app-blue

What Happens When Deployment Fails?

The workflow was also tested with an intentionally broken application version.

In the failed branch, the /health behavior causes the existing Go unit test to fail:

--- FAIL: TestHealth (0.00s)
    main_test.go:16: expected status 200, got 500
FAIL
FAIL    jenkins-zero-downtime    0.002s

Because the test stage fails, Jenkins stops the deployment before creating or exposing the new version:

Build Image       → SKIPPED
Deploy Green      → SKIPPED
Verify Green      → SKIPPED
Switch Traffic    → SKIPPED
Stop Blue         → SKIPPED

The pipeline finishes with:

ERROR: script returned exit code 1
Finished: FAILURE

Most importantly, the Switch Traffic stage is never executed.

The existing version therefore remains active:

curl http://localhost

{
  "message": "Hello",
  "version": "v1",
  "hostname": "5f6a41482e36"
}

This demonstrates the failure behavior we wanted from the deployment workflow:

A release that fails validation does not automatically replace the currently running version.

The result is a simple deployment workflow that provides the core safety properties of blue-green deployment.

Conclusion

This project demonstrated how Jenkins, Docker, and Nginx can be combined to implement a simple blue-green deployment workflow without Kubernetes.

The new version is tested and verified before receiving traffic, while the existing version remains available during deployment. When validation fails, Jenkins stops the pipeline and leaves the current version untouched.

For smaller applications, this provides a simple and practical way to reduce deployment downtime and the risk of releasing a broken version.

The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the full source code in the repository below.

https://github.com/muhammadyulasfipahrizal/blue-green-jenkins.git

Building A Secure CI Pipeline With GitHub Actions, SonarQube, And Trivy

Pahri — Thu, 27 Aug 2026 05:25:07 +0000

A CI pipeline can tell us that an application works, but can it tell us that the application is safe to ship?

A successful build does not necessarily mean that the software is secure.

An application can pass its tests while still containing insecure and bad code patterns. Other than that container image built from the application may inherit vulnerable packages.

The problem is that these security issues exist at different layers:

Checking only one of these layers is not enough.

For example, static code analysis can help identify security issues in the source code, but it does not tell us whether the packages inside our container image contain known vulnerabilities.

Likewise, scanning a container image does not tell us whether the application source itself contains problematic code.

A traditional CI workflow might look like this:

The application is tested and, if everything works, the resulting artifact is published. But there is no explicit security check between the code and the artifact.

That is what we are going to address in this article.

The complete implementation is available in repository bellow:

Github Repository

What We Are Building

We will build a small Go API and place it behind an automated CI pipeline.

The application itself is intentionally simple. There is no database, authentication system, or external service. The purpose of the application is to provide an artifact that we can test, analyze, containerize, and scan.

Application

The application is a small HTTP API written in Go.

It provides two endpoints:

GET /health
GET /api/users

The /health endpoint is used to verify that the application is running, while /api/users returns a small list of users.

This gives us enough application code to run unit tests and static analysis without introducing unnecessary application complexity.

The application is located under the cmd/ directory, with the HTTP handlers implemented in cmd/main.go.

The application is then packaged as a Docker image.

CI Pipeline

The application is connected to GitHub Actions that automate the entire process.

The final pipeline looks like this:

Each stage has a specific responsibility.

API Test, verifies that the application behaves as expected before any further analysis or packaging takes place.
SonarQube Scan, analyzes the source code for potential security, reliability, and maintainability issues.
Docker Build, packages the application and its runtime environment into a Docker image.
Trivy Scan, scans the resulting container image for known vulnerabilities in the image's operating-system packages, dependencies, and other components.
Push, publishes the container image to GitHub Container Registry after it has passed the preceding stages.

Project Structure

Each part of the repository has a specific responsibility.

.github/workflows/ci.yml defines the CI workflow and orchestrates the entire process.
cmd/main.go contains the Go API itself, including the HTTP server and API endpoints.
Dockerfile defines how the Go API is compiled and packaged into a Docker image.
sonar-project.properties contains the SonarQube project configuration, including which directories should be analyzed and how test files should be identified.

Implementation

The application in this project intentionally contains several code patterns that can be detected by static analysis.

The Docker image also uses an intentionally outdated base image so that the container scan has meaningful vulnerabilities to report.

The goal is not to demonstrate how to write perfect Go code. The goal is to demonstrate how this CI pipeline can identify problematic code and image automatically.

Go API

The API implementation itself is straightforward, but several functions in main.go contain intentionally problematic code.

For example the processUser function.

func processUser(user User) string {
    if user.Name == "" {
        if user.Email == "" {
            if user.ID > 0 {
                return "invalid-user"
            }
        } else if user.ID > 0 {
            return "invalid-user"
        }
    }

    if user.Email == "" {
        if user.ID > 0 {
            return "invalid-user"
        }
    }

    if user.ID > 0 {
        if user.Name == "Alice" {
            if user.Email == "alice@example.com" {
                return "valid-user"
            }
        }

        if user.Name == "Bob" {
            if user.Email == "bob@example.com" {
                return "valid-user"
            }
        }

        return "valid-user"
    }

    return "invalid-user"
}

The function intentionally contains deeply nested conditional statements and duplicated logic.

For example, the same user.ID > 0 condition is evaluated in multiple branches.

This makes the function harder to understand and maintain than necessary.

The same idea is used in getUserRole.

func getUserRole(role string) string {
    switch role {
    case "admin":
        return "Administrator"
    case "manager":
        return "Manager"
    case "developer":
        return "Developer"
    // ...
    default:
        return "Unknown"
    }
}

The function contains a long list of similar branches.

This is not necessarily a security vulnerability by itself. It is included to give the static analyzer code that can be evaluated for maintainability and code-quality issues.

Another example is unreachableCode:

func unreachableCode() string {
    return "valid"

    log.Println("this code is never reached")

    return "unreachable"
}

The statements after the first return can never be executed.

This is exactly the kind of implementation problem that static analysis can identify without executing the application.

We also intentionally introduce duplicated return values:

func duplicatedStrings(value int) string {
    if value == 1 {
        return "invalid-user"
    }

    if value == 2 {
        return "invalid-user"
    }

    if value == 3 {
        return "invalid-user"
    }

    return "valid-user"
}

The same string literal is repeated across multiple branches.

Why Introduce Problems Intentionally?

At this point, it may seem strange to deliberately write bad code.

In a normal development project, we would try to remove these problems before merging the code.

But this project has a different purpose.

If a developer introduces problematic code into the repository, will our CI pipeline detect it automatically?

That means the vulnerable or low quality code becomes part of our test scenario.

Container Image

The same approach is used for the container image.

The Dockerfile intentionally uses an older Go base image:

FROM golang:1.20

Once the application is built, the final image contains more than our own code. It also contains the software and packages provided by the base image.

This gives Trivy something different to analyze from what SonarQube sees.

SonarQube is primarily concerned with the source code, while Trivy examines the resulting container image and looks for known vulnerabilities in its components.

API Test

Before running the security analysis, the pipeline first runs the API test.

The workflow uses the standard Go test command:

go test ./...

The repository includes tests for the API handlers, giving the pipeline a basic verification that the application still behaves as expected.

If the tests fail, the following stages should not proceed.

SonarQube Scan

After the test pass, the pipeline moves to static analysis.

The SonarQube configuration is stored in:

sonar-project.properties

The important part is defining the application source and test locations:

sonar.sources=cmd
sonar.tests=cmd

Because we intentionally introduced problematic code into main.go, this stage gives us an opportunity to verify that the static analysis can actually identify those problems.

For example, the unnecessary nesting in processUser, unreachable statements in unreachableCode, and repeated string literals in duplicatedStrings provide concrete code quality problems for the analyzer to inspect.

Trivy Scan

After the Docker image has been built, Trivy scans the resulting image for known vulnerabilities.

The workflow specifically checks for:

HIGH
CRITICAL

severity vulnerabilities.

The scan is performed against the image that was just built, rather than against an unrelated image.

This is where the intentionally older base image becomes relevant.

Even though the Go application itself is small, the container inherits software from its base image. Trivy can therefore identify vulnerabilities that exist below the application layer.

SonarQube tells us about problems in the code we write. Trivy tells us about vulnerabilities in the artifact we are about to distribute.

There is one important configuration detail in this project.

The Trivy action currently uses:

exit-code: 0

This means the scan reports the detected vulnerabilities but does not fail the workflow because of them.

That is intentional for this demonstration.

We want to see the vulnerabilities produced by our intentionally vulnerable image while still allowing the rest of the workflow to demonstrate the complete delivery process.

This leads to an important distinction that we will revisit later:

A scanner can detect a vulnerability without necessarily enforcing a policy against it

Push the Image

The final stage publishes the container image to GitHub Container Registry.

The workflow is structured so that the push stage depends on the previous stages completing successfully.

The workflow also defines limited permissions for the GitHub Actions job:

permissions:
  contents: read
  packages: write

The workflow needs read access to the repository and write access to GitHub Packages in order to publish the image.

With the image push as the final stage, the pipeline now has a complete path from source code to a published container artifact.

The interesting part is that we can now observe whether the security checks actually detected the intentionally introduced problems.

Result

With everthing setup the scanners now can found the intentionally introduced problems.

SonarQube Result

The SonarQube analysis identified four issues in the Go application.

The most significant finding was in processUser:

Refactor this method to reduce its Cognitive Complexity from 21 to the 15 allowed.

SonarQube assigned this issue a High maintainability impact.

The function has a Cognitive Complexity score of 21, while the configured threshold is 15.

This is a good example of why static analysis is useful.

The application can still compile and the API can still work correctly, but the implementation is unnecessarily difficult to reason about. The CI analysis makes this problem visible without requiring a reviewer to manually inspect every conditional branch.

SonarQube also detected duplicated literals:

Define a constant instead of duplicating this literal "invalid-user" 7 times.

These findings map directly to the intentionally duplicated values in processUser and duplicatedStrings.

Finally, SonarQube identified the unreachable code in unreachableCode function:

Refactor this piece of code to not have any dead code after this "return".

This issue was classified as a Medium reliability issue.

Quality Gate Enforcement

There is another important detail in the workflow.

The SonarQube Quality Gate step is intentionally commented out to allow the pipeline to keep running, as this project is currently focused on detecting code quality issues rather than enforcing them.

# - name: SonarQube quality gate
#   uses: SonarSource/sonarqube-quality-gate-action@cf038b0e0cdecfa9e56c198bbb7d21d751d62c3b # v1.2.0
#   timeout-minutes: 5
#   env:
#     SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
#     SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}

Trivy Results

Trivy analyzed the container image:

secure-ci-pipeline:latest

The image was based on Debian 12.4 and Trivy reported:

1641 vulnerabilities

at the Debian package layer.

Trivy also identified vulnerabilities in the Go binaries contained in the image:

app/app       → 25 vulnerabilities
/usr/local/go/bin/go       → 25 vulnerabilities
/usr/local/go/bin/gofmt    → 25 vulnerabilities
/usr/local/go/pkg/tool/... → 25 vulnerabilities
---------------
---------------

The current Trivy configuration uses:

exit-code: 0

So although Trivy discovered a large number of vulnerabilities, the workflow does not stop because of those findings.

A mature pipeline could eventually evolve from:

Into:

The benefit of transitioning to this approach is that security findings become part of the release decision instead of being treated as information that developers may review later.

Without enforcement, a pipeline can report a critical vulnerability while still allowing the image to be published. This creates a gap between knowing about a problem and preventing the problem from reaching the next stage.

Conclusion

In this project, we addressed the gap described in the introduction by adding security checks directly into the CI workflow.

Instead of only asking whether the application works, the pipeline also checks whether the source code contains quality or reliability problems and whether the resulting container image contains known vulnerabilities.

The key benefit is moving security checks earlier in the delivery process, so problems can be identified before they become production problems.

The code snippets in this article focus on the important parts of the implementation. For the complete source code, you can find it through repository below.

Github Repository

Why Is Your Linux Server Slow? A Practical Guide to Performance Troubleshooting

Pahri — Fri, 21 Aug 2026 03:21:50 +0000

A Linux server can be running without any obvious errors and still feel painfully slow.

Applications may take longer to respond, SSH sessions may become sluggish, users may suddenly experience high latency. The difficult part is that the root cause is not always obvious.

A slow Linux server does not necessarily mean that the CPU is overloaded.

It could be memory pressure, excessive disk I/O, network congestion, too many open file descriptors, or simply a process consuming resources unexpectedly.

The challenge for a DevOps Engineer is not just finding that the server is slow, but identifying why it is slow.

Instead of immediately restarting services or killing processes, we need to investigate the system, layer by layer:

Is the CPU under pressure?
Is the system running out of memory?
Is disk I/O becoming a bottleneck?
Are processes exhausting file descriptors?
Which process is actually responsible for the problem?

This article is a practical introduction to Linux performance troubleshooting using real reproducible scenarios.

Rather than looking at isolated commands, we will build a troubleshooting mindset: observe the symptom, collect evidence, identify the bottleneck, apply the appropriate solution, and verify the result.

To make the investigation reproducible, I created a companion lab repository containing scenarios that intentionally introduce problems:

github repository

High CPU

High CPU usage is one of the most common Linux performance problems.

In this scenario, we intentionally create a process that continuously executes an infinite loop.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./high-cpu/reproduce.sh
./high-cpu/reproduce.sh

The script will display a warning before starting the scenario.

While it is running, open another terminal to investigate the system.

Identify High CPU Usage

The first step is to confirm whether the system is actually experiencing high CPU utilization.

Use top:

top

Look at the CPU summary:

%Cpu(s)

Example result:

%Cpu(s): 95.2 us,  1.8 sy,  0.0 ni,  0.0 id,  2.5 wa, ...

The most important fields are:

us — CPU time spent running user-space processes
sy — CPU time spent running kernel-space processes
id — percentage of CPU time that is idle
wa — CPU time waiting for I/O

If id is consistently very low while us is high, the CPU is heavily utilized by processes.

Identify the Process

The next step is to find the process that consume the most CPU.

Use:

ps aux --sort=-%cpu | head

Example result:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       38690 90.5  0.0   7740  3512 pts/4   R+   02:39   0:16 bash ./reproduce.sh

The most important fields are:

PID — Process ID
%CPU — CPU consumption
%MEM — Memory consumption
STAT — Current process state
COMMAND — Command used to start the process

Here bash ./reproduce.sh process is clearly consuming a significant amount of CPU.

We now have a candidate process, but before terminating it, we should inspect it further.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 38690

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       38690   37782 96 02:39 pts/4    00:00:45 bash ./reproduce.sh

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

Understand the Cause

Inspect the process script:

cat reproduce.sh

The important part is:

while true; do
    :
done

This creates an infinite loop.

The loop continuously executes without any sleep, blocking operation, or other mechanism that would allow the process to stop consuming CPU therefore keeps the CPU busy.

This is the root cause of the high CPU usage in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 38690

Then check the CPU usage again:

top

The process should no longer appear in the process list, and CPU utilization should return to its previous level.

Disk I/O

High disk I/O is another common Linux performance problem.

In this scenario, we intentionally create a process that continuously writes data to disk.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./disk/reproduce.sh
./disk/reproduce.sh

Identify Disk I/O

The first step is to confirm whether the system is actually experiencing high Disk I/O activity.

Use iostat:

iostat -xz 1

Example result:

Device            r/s     w/s   rkB/s   wkB/s  %util
sda              0.00  125.00    0.00  51200.00  98.50

The most important fields are:

r/s — Number of read requests per second
w/s — Number of write requests per second
rkB/s — Data read per second
wkB/s — Data written per second
%util — Percentage of time the device was busy

If %util is consistently close to 100%, the storage device is heavily utilized.

Identify the Process

The next step is to find which process is generating the disk activity.

Use:

pidstat -d 1

Example result:

UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
0     41043      0.00 259548.51      0.00       0  bash

The most important fields are:

PID — Process ID
COMMAND — Command generating the I/O

Here the PID 41043 process is clearly generating significant disk write activity.

Inspect the Process

Use the PID identified from the previous step:

ps -fp 41043

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       41043   37782  0 03:03 pts/4    00:00:00 bash ./reproduce.sh

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is continuously writing data to testfile.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh

The important part is:

while true; do
    dd if=/dev/zero of=testfile bs=1M count=1024 conv=fsync
    rm -f testfile
done

The script continuously creates a file and writes data to it.

The dd command reads zero-filled data from /dev/zero and writes it to testfile.

The conv=fsync option forces dd to flush the written data to the storage device before completing the operation. This makes the scenario generate actual disk write activity instead of relying entirely on the filesystem page cache.

The file is then removed and the process repeats the operation.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 41043

Then check the disk activity again:

iostat -xz 1

You can also verify the processes performing I/O:

iotop

The dd process should no longer appear, disk write throughput should drop significantly, and %util should return to its previous level.

Finally, verify that the test file has been removed:

ls -lh testfile

Memory Pressure

High memory usage is another common Linux performance problem.

In this scenario, we create a process that allocates a large amount of memory and keeps it allocated.

The reproduction script is available in the github repository repository.

Run the reproduction script with a memory target:

chmod +x ./memory-pressure/reproduce.sh
./memory-pressure/reproduce.sh 2G

Replace 2G with the amount of memory you want to allocate.

Identify Memory Pressure

The first step is to confirm whether the system is actually experiencing high memory utilization.

Use:

free -h

Example result:

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       3.1Gi       141Mi       1.0Mi       822Mi       692Mi
Swap:             0B          0B          0B

The most important fields are:

total — Total physical memory available
used — Memory currently used by the system
free — Completely unused memory
buff/cache — Memory used by the kernel for buffers and filesystem cache
available — Memory available for new applications

The available value is particularly important when diagnosing memory pressure. If available memory is consistently very low while used memory is high, the system is under significant memory pressure.

If the system has swap enabled, also pay attention to swap usage. Significant swap usage can indicate that the system does not have enough physical memory to satisfy current workloads.

Identify the Process

The next step is to find which process is consuming the most memory.

Use:

ps aux --sort=-%mem | head

Example result:

USER         PID %CPU %MEM    VSZ    RSS TTY      STAT START   TIME COMMAND
root       39937  0.1 52.7 2123808 2116080 pts/4 S+   02:56   0:03 python3 ./memory_hog.py 2G

The most important fields are:

PID — Process ID
%MEM — Percentage of physical memory used by the process
VSZ — Virtual memory size
RSS — Resident memory currently held in RAM
STAT — Current process state
COMMAND — Command used to start the process

Here the PID 39937 process is clearly consuming a significant amount of memory.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 39937

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       39937   39910  0 02:56 pts/4    00:00:03 python3 ./memory_hog.py 2G

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is running memory_hog.py with a target allocation of 2G.

Understand the Cause

Inspect the reproduction script:

cat reproduce.sh

The shell script starts the Python memory allocation program:

python3 "$(dirname "$0")/memory_hog.py" "$MEMORY"

The Python script can then be inspected:

cat memory_hog.py

The important part is:

while allocated_bytes < amount:
    remaining = amount - allocated_bytes
    size = min(chunk_size, remaining)

    chunk = bytearray(size)

    chunk[0] = 1

    allocated.append(chunk)
    allocated_bytes += size

The script allocates memory in 1 MiB chunks until the requested memory amount is reached.

Each allocated chunk is stored in the allocated list:

allocated.append(chunk)

This keeps references to the allocated memory, preventing Python from releasing those objects.

The script then remains running:

while True:
    time.sleep(1)

This large memory allocation is the root cause of the memory pressure in this scenario.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 39937

Then check the memory usage again:

free -h

You can also verify the processes sorted by memory usage:

ps aux --sort=-%mem | head

The memory_hog.py process should no longer appear in the process list.

The available memory should increase, and overall memory utilization should return closer to its previous level.

File Descriptor Exhaustion

In this scenario, we intentionally create a process that continuously opens file descriptors without closing them.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./file-descriptor/reproduce.sh
./file-descriptor/reproduce.sh

Identify File Descriptor Pressure

The first step is to check the system-wide file descriptor usage.

Use:

cat /proc/sys/fs/file-nr

Example result:

2656    0    9223372036854775807

The values represent:

allocated — Number of allocated file handles
unused — Number of unused allocated file handles
maximum — Maximum number of file handles allowed system-wide

The allocated value represents file handles currently allocated by the kernel.

A high number of allocated file handles can indicate file descriptor pressure, especially when applications are approaching their per-process or system-wide limits.

Identify the Process

The next step is to find the process running fd_exhaustion.py.

Use:

ps aux | grep '[f]d_exhaustion'

Example result:

root       42297  0.2  0.2  18488 10504 pts/4    S+   03:11   0:00 python3 ./fd_exhaustion.py

The most important fields are:

PID — Process ID
%CPU — CPU usage
%MEM — Memory usage
COMMAND — Command used to start the process

Here the python3 ./fd_exhaustion.py process is the candidate responsible for the file descriptor exhaustion.

Use the identified PID for the next step.

Inspect the Process

The next step is to determine how many file descriptors the process currently has open.

Use:

ls /proc/42297/fd | wc -l

Example result:

This indicates that the process currently has 1024 file descriptors open.

You can inspect the actual descriptors with:

ls -l /proc/<PID>/fd

Example:

lr-x------ 1 root root 64 Aug 21 03:11 0 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 1 -> /dev/pts/4
lrwx------ 1 root root 64 Aug 21 03:11 2 -> /dev/pts/4
lr-x------ 1 root root 64 Aug 21 03:11 3 -> /dev/null
lr-x------ 1 root root 64 Aug 21 03:11 4 -> /dev/null

The large number of /dev/null entries indicates that the process is continuously opening new file descriptors.

Next, check the process limits:

cat /proc/42297/limits | grep "open files"

Output:

Max open files            1024                 1048576                files

The first value is the soft limit, while the second value is the hard limit.

The soft limit is the limit currently enforced for the process.

Once the process reaches this limit, attempts to open additional file descriptors will fail.

Understand the Cause

Inspect the reproduction script:

cat fd_exhaustion.py

The important part is:

while True:
    try:
        fd = os.open(
            "/dev/null",
            os.O_RDONLY
        )

        FILES.append(fd)

        counter += 1

The script continuously opens /dev/null using os.open().

Every returned file descriptor is stored in the FILES list:

FILES.append(fd)

The descriptors are intentionally not closed while the scenario is running. As a result, the number of open file descriptors continuously increases

Eventually, os.open() fails because the process has reached its file descriptor limit.

The script then reports the error:

Failed to open new file descriptor
[Errno 24] Too many open files

This is the root cause of the file descriptor exhaustion in this scenario.

File descriptor exhaustion can have broader consequences for real applications. Processes may fail to open files, accept new connections, create sockets, or perform other operations that require file descriptors.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 42297

Verify that the process has stopped:

ps -p 42297

If the process has stopped, the command should return no process entry.

You can also verify that the process no longer exists:

test -d /proc/<PID> && echo "Process still exists" || echo "Process stopped"

The file descriptors belonging to the process are automatically released by the kernel when the process terminates.

Therefore, /proc/<PID>/fd will no longer be available after the process has exited.

Finally, verify the system-wide file descriptor state again:

cat /proc/sys/fs/file-nr

The allocated file handle count should return closer to its previous level after the exhausted process has been terminated.

Network Connection Exhaustion

In this scenario, we intentionally create a process that continuously opens TCP connections to a local server and keeps those connections open.

The reproduction script is available in the github repository repository.

Run the reproduction script:

chmod +x ./network-connection/reproduce.sh
./network-connection/reproduce.sh

The script will display a warning before starting the scenario.

The reproduction uses 127.0.0.1:9000 as the target, so all connections remain on the local machine.

Identify Abnormal Connections

The first step is to confirm whether the system is experiencing an unusually large number of TCP connections.

Use ss:

ss -s

Example result:

Total: 1252
TCP:   2051 (estab 1023, closed 1021, orphaned 1, timewait 1020)

Transport Total     IP        IPv6
RAW       1         0         1
UDP       3         3         0
TCP       1030      1028      2
INET      1034      1031      3
FRAG      0         0         0

The most important fields are:

estab — Established TCP connections
orphaned — Orphaned TCP connections
timewait — Connections waiting to be fully closed
TCP — Total TCP sockets

A significant increase in established connections can indicate abnormal connection usage.

In this scenario, the large number of estab connections is the main indicator that something is continuously creating and maintaining TCP connections.

Identify the Connection and Process

The next step is to identify which process owns the connections.

Use:

ss -tanp

Example result:

CLOSE-WAIT  1  0  127.0.0.1:46374  127.0.0.1:9000  users:(("python3",pid=46938,fd=880))
CLOSE-WAIT  1  0  127.0.0.1:39120  127.0.0.1:9000  users:(("python3",pid=46938,fd=86))
CLOSE-WAIT  1  0  127.0.0.1:44192  127.0.0.1:9000  users:(("python3",pid=46938,fd=634))

The most important information is:

State — Current TCP connection state
Local Address:Port — Local endpoint
Peer Address:Port — Remote endpoint
users — Process owning the socket
pid — Process ID
fd — File descriptor associated with the socket

Here we can identify the process responsible for the connections:

users:(("python3",pid=46938,fd=880))

The PID is:

The same PID appearing across many connections is a strong indication that one process is responsible for creating the connection buildup.

Inspect the Process

Use the PID identified in the previous step:

ps -fp 46938

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root       46938   46922  0 03:42 pts/1    00:00:00 python3 ./connection_exhaustion.py 127.0.0.1 9000

This provides additional information about the process.

The most important fields are:

PID — Process ID
PPID — Parent Process ID
STIME — Process start time
TIME — Total CPU time consumed
CMD — Command used to start the process

The command shows that the process is running connection_exhaustion.py and connecting to 127.0.0.1:9000.

Next, check how many file descriptors the process currently has open:

ls /proc/46938/fd | wc -l

A large number of file descriptors is expected because every TCP socket consumes a file descriptor.

You can inspect the descriptors directly:

ls -l /proc/46938/fd | head

You should see entries pointing to sockets, for example:

lrwx------ 1 root root 64 Aug 21 03:42 86 -> 'socket:[123456]'
lrwx------ 1 root root 64 Aug 21 03:42 87 -> 'socket:[123457]'
lrwx------ 1 root root 64 Aug 21 03:42 88 -> 'socket:[123458]'

This confirms that the process is holding a large number of socket file descriptors.

Understand the Cause

Inspect the reproduction script:

cat connection_exhaustion.py

The important part is:

while True:
    try:
        sock = socket.socket(
            socket.AF_INET,
            socket.SOCK_STREAM
        )

        sock.connect((host, port))

        CONNECTIONS.append(sock)

        counter += 1

The script continuously creates TCP sockets and connects them to the target server.

Every successful connection is stored in the CONNECTIONS list:

CONNECTIONS.append(sock)

The sockets are intentionally not closed while the scenario is running. As a result, the number of active connections continuously increases:

The TCP server is also intentionally designed to keep accepted connections open:

connection, address = server.accept()
connections.append(connection)

This means the server does not immediately close the connections created by the client. Eventually, the client process can reach its file descriptor limit and fail to create additional sockets.

This pattern can occur in real applications when connections are not properly closed, connection pools are misconfigured, or an application continuously creates new connections instead of reusing existing ones.

Fix and Verify

Once the responsible process and its cause have been identified, terminate the process using its PID:

kill 46938

Then check the network connection summary again:

ss -s

You can also check the connections to port 9000 directly:

ss -tan | grep ':9000'

The number of established TCP connections should return to its previous level.

Conclusion

Linux performance troubleshooting is not about finding a single command that tells you why a server is slow.

It is about building a structured investigation from the symptoms you observe.

In the scenarios covered in this article, we investigated several different types of Linux performance problems:

High CPU caused by an infinite loop
High disk I/O caused by continuous writes
Memory pressure caused by excessive memory allocation
File descriptor exhaustion caused by descriptors that are never closed
Network connection exhaustion caused by continuously opened TCP connections

Although the symptoms are different, the troubleshooting approach remains consistent:

The important lesson is to avoid making assumptions based on a single metric.

High CPU does not immediately tell you which process is responsible. High memory usage does not necessarily mean the system is running out of usable memory. A large number of TCP connections does not automatically mean the network is slow.

Each symptom needs to be investigated with the right evidence.

When a production server becomes slow, the most valuable skill is not knowing how to restart it.

It is being able to answer:

What is happening, why is it happening, and how can I prove that my fix actually worked?

The scenarios in this article are intentionally simplified, but the troubleshooting methodology can be applied to much more complex production incidents.

You can reproduce all of these scenarios and investigate yourself by cloning my github repository bellow:

https://github.com/muhammadyulasfipahrizal/linux-performance-lab.git