DEV Community: Amira Abidi

Kubernetes HPA on Amazon EKS: Scaling from 2 to 6 Pods Under Load

Amira Abidi — Tue, 13 Oct 2026 13:53:32 +0000

What happens when your application receives more traffic than its current capacity can handle?

While building LibraryCorner, my AWS Cloud/DevOps project, I tested Kubernetes Horizontal Pod Autoscaler (HPA) on Amazon EKS to see how the application would respond to increasing CPU utilization.

The result was more interesting than simply watching pods appear: the test demonstrated automatic scaling, observable performance changes, and a clearly defined capacity limit.

Here's what happened.

1. The setup

I configured Kubernetes HPA to scale the application between 2 and 6 pods, targeting 55% average CPU utilization.

The test followed the complete lifecycle:

Start with two pods at rest.
Gradually increase the load.
Observe CPU utilization and application latency.
Watch HPA scale out.
Reach the configured capacity limit.
Stop the load and observe recovery.

The objective wasn't just to increase the replica count. It was to verify that the system reacted to real metrics and returned to its initial state afterward.

2. From 2 pods to 6

At rest, the application ran with two pods and low resource utilization.

As the load increased, CPU utilization rose significantly. At the peak, it reached 216%, and HPA increased the number of replicas to four, then six.

The scaling process looked like this:

       Increasing traffic
               |
               v
       CPU utilization rises
               |
               v
       Kubernetes HPA reacts
               |
               v
          2 → 4 → 6 pods
               |
               v
       More application capacity

HPA uses observed metrics to calculate the desired replica count. Kubernetes then schedules and starts additional pods, provided the cluster has enough capacity.

The important detail is that scaling isn't instantaneous. New pods need time to start and become ready, so application latency can temporarily increase while demand rises.

That was visible in my test: latency degraded around the traffic peak, then returned to normal as additional pods became available.

3. The results: 15,000+ requests, zero failures

The load test produced encouraging results:

15,000+ requests
Zero failed requests
CPU utilization reached 216% at the peak
Pod count increased from 2 to 6
Latency remained below the configured alert threshold at the 95th percentile

The dashboard captured the entire sequence: traffic increased, latency rose, additional pods started, and performance recovered as the system scaled.

This is why I consider observability essential to autoscaling. The dashboard helped connect the HPA decisions to actual application behavior rather than just showing a changing replica count.

4. Why stop at six pods?

Six wasn't an arbitrary number.

At the time of the experiment, the cluster's node networking capacity limited the number of pods that could be scheduled. Increasing the HPA maximum beyond the available capacity could have left new pods pending.

I therefore configured the maximum at six and added an alert to signal when the workload reached that limit.

At peak load, the alert fired and a notification was emitted.

One important distinction: the alert did not trigger scaling. CPU metrics drove HPA. The alert simply indicated that the configured maximum had been reached and that additional capacity might be needed.

This is a useful operational signal: autoscaling can work exactly as configured and still reach a point where it cannot scale further.

5. Scaling back down

When the load stopped, the alert resolved automatically, and HPA returned the workload to two pods after the stabilization window.

That final step matters.

Autoscaling isn't just about adding replicas under pressure. It also needs to remove unnecessary replicas without reacting too aggressively to short-lived changes.

The complete lifecycle was:

Rest → Load → Scale out → Capacity alert → Recovery → Scale down

This validated three aspects of the project together: elasticity, observability, and automatic reactions based on metrics.

What I learned

This experiment reinforced three lessons:

1. Autoscaling depends on capacity. HPA can request more pods, but the cluster must have enough networking and compute resources to run them.

2. Monitoring makes behavior explainable. CPU, replica count, latency and alerts helped me understand what happened throughout the test.

3. Scaling needs end-to-end validation. Reaching six pods is not enough. You also need to check application errors, latency, recovery, and what happens when the maximum is reached.

Coming from QA and validation, this last point feels natural to me: I want to verify the complete behavior, not just whether a Kubernetes resource has the expected status.

Final thoughts

My LibraryCorner load test demonstrated that HPA could scale the application from two to six pods, handle more than 15,000 requests without failures, and return to two pods after the load ended.

It also exposed an important constraint: application autoscaling cannot compensate indefinitely for limited cluster capacity.

Autoscaling isn't simply about adding pods. It's about responding to demand, observing the result, and knowing when you've reached your limits.

Disaster Recovery: Restoring PostgreSQL on Amazon RDS

Amira Abidi — Tue, 06 Oct 2026 14:53:50 +0000

A backup is easy to create.

The harder question is:

Can you actually recover your application when you need to?

While building LibraryCorner, my AWS project running on Amazon EKS with PostgreSQL on Amazon RDS, I wanted to look beyond simply deploying a database and enabling backups.

I wanted to understand what happens when things go wrong.

Because having a backup is not the same as having a recovery strategy.

Backup is only the starting point

For a database-backed application, disaster recovery is more than storing a copy of your data.

The real recovery flow looks more like:

Failure
   ↓
Choose recovery point
   ↓
Restore PostgreSQL
   ↓
Reconnect application
   ↓
Validate data
   ↓
Validate application
   ↓
Return to service

Every step matters.

A restored database that the application cannot reach is not a successful recovery.

My LibraryCorner architecture

In LibraryCorner, the application runs on Amazon EKS, while PostgreSQL is hosted on Amazon RDS.

At a high level:

                  Internet
                     │
                     ▼
                  ALB
                     │
                     ▼
                Amazon EKS
                     │
                     ▼
                Amazon RDS
                 PostgreSQL
                     │
                     ▼
              Backup / Snapshot

The production database uses a multi-AZ configuration to improve availability.

But this led me to an important distinction:

High availability is not the same as disaster recovery.

Multi-AZ helps protect against certain infrastructure failures.

Backups and snapshots provide another layer of protection when recovery is required.

What would recovery look like?

Imagine that the database needs to be replaced.

The first question isn't simply:

“Can I restore it?”

It is:

“Which recovery point should I restore?”

This is where two concepts become important.

RPO — Recovery Point Objective

How much data can I afford to lose?

A system that can tolerate an hour of data loss has very different requirements from one that needs almost zero data loss.

RTO — Recovery Time Objective

How long can the application remain unavailable?

A recovery strategy that takes several hours may be perfectly acceptable for one application and completely unacceptable for another.

RPO and RTO therefore aren't just theoretical concepts.

They influence the architecture you choose.

Restoring the database isn't the end

Suppose the PostgreSQL instance has been restored successfully.

The RDS console says:

Available ✓

Is the recovery finished?

Not yet.

The application still needs to communicate with the restored database.

The recovery process is therefore closer to:

Backup / Snapshot
        ↓
Restore RDS
        ↓
Database available
        ↓
Application reconnects
        ↓
Functional validation
        ↓
Recovery complete

This is where my QA background strongly influences how I approach recovery.

I don't want to validate only the infrastructure.

I want to validate the complete system.

What would I validate?

First, the database:

Can I connect?
Is the expected schema present?
Is the expected data available?

Then the application:

Can it connect to PostgreSQL?
Can it retrieve existing data?
Can it create new data?
Do the main API operations work?

And finally, the infrastructure:

Is network connectivity working?
Are access controls still correct?
Is monitoring operational?

This is essentially the same mindset I used in QA:

Don't just check that the component is running. Check that the system actually works.

The lesson I took from this project

Before working on Cloud and DevOps, I thought of backups mainly as a safety mechanism.

Now I see them as one part of a larger process:

Backup → Restore → Reconnect → Validate

A resilient architecture needs both availability and recoverability.

And recovery should not be something you discover for the first time during an incident.

Ideally, the restore process should be tested.

Because a backup you have never successfully restored is ultimately an assumption.

My QA background taught me to question those assumptions.

Cloud is teaching me how to turn them into repeatable processes.

And that's probably the most important lesson from this project:

A backup gives you a recovery point. A tested restore gives you confidence.

What QA Taught Me About DevOps

Amira Abidi — Tue, 29 Sep 2026 09:02:03 +0000

When I started moving from QA and validation toward Cloud and DevOps, I initially thought I had to leave my QA background behind.

I was wrong.

The more I worked with AWS, Terraform, Kubernetes and CI/CD, the more I realized that many habits I developed as a QA engineer were still useful — sometimes even more than I expected.

Here are the lessons from QA that I now carry into my Cloud and DevOps projects.

1. Think about failure before success

QA teaches you not to ask only:

“Does it work?”

You also ask:

What happens if the service is unavailable?
What happens if a dependency fails?
What happens with invalid input?
What happens under load?
What happens after a restart?

This mindset became very useful when working with Kubernetes and AWS.

For example, when building my LibraryCorner project on Amazon EKS, I didn't want to validate only that the application was running.

I also looked at scaling, health checks, database availability, monitoring and recovery.

The question changed from:

“Can I deploy it?”

to:

“How does it behave when something goes wrong?”

2. Automation is more than saving time

Automation was already part of my QA work.

Tests, validation processes and repetitive checks are much more reliable when they can be executed automatically.

Moving into DevOps made this principle even stronger.

Infrastructure can be automated with Terraform.

Application delivery can be automated with CI/CD.

Kubernetes can automatically maintain the desired state.

Monitoring can continuously check what is happening.

The goal isn't simply to eliminate manual work.

It is to make processes repeatable, predictable and easier to trust.

3. Reproducibility matters

One of the most frustrating situations in testing is:

“It worked on my machine.”

A good test environment should be reproducible.

The same principle applies to infrastructure.

With Terraform, I can describe infrastructure as code instead of manually creating resources in the AWS console.

With Kubernetes manifests and Kustomize, I can define application configurations consistently across environments.

This changed the way I think about infrastructure.

Instead of asking:

“How did I configure this?”

I can ask:

“Can I reproduce this configuration?”

That's a much better question.

4. Observability is part of validation

QA engineers spend a lot of time looking for evidence.

Logs, test results, error messages and system behavior help answer:

“What actually happened?”

Observability follows the same principle.

While working on LibraryCorner, I implemented Prometheus and Grafana monitoring for my EKS environment.

Metrics aren't just dashboards that look nice.

They provide evidence about what the system is doing.

CPU usage, pod count, resource consumption and application behavior can help explain why something happened.

For me, observability is therefore closely related to validation:

you cannot validate what you cannot see.

5. Testing doesn't disappear in DevOps

Before moving into Cloud, I sometimes associated DevOps with infrastructure, pipelines and deployments.

Today, I see it differently.

Testing is still there.

It simply becomes part of a larger feedback loop.

Code changes → tests → build → deployment → monitoring → feedback.

The objective is to detect problems earlier and shorten the time between a change and reliable feedback.

That is one of the reasons my QA background fits naturally into this environment.

6. My mindset changed more than my tools

The biggest lesson from this transition is that moving from QA to DevOps isn't only about learning new technologies.

Yes, I had to learn AWS, Terraform, Kubernetes, Linux, networking and CI/CD.

But the most valuable thing I brought with me wasn't a specific tool.

It was a way of thinking:

Challenge assumptions.
Look for failure scenarios.
Automate repetitive work.
Make environments reproducible.
Collect evidence.
Monitor what happens after deployment.

These principles remain useful regardless of whether the system is a test environment, a Kubernetes cluster or an AWS architecture.

From QA to Cloud

My transition from QA/validation to Cloud and DevOps didn't mean starting from zero.

It meant building on top of what I already knew.

QA taught me to question systems.

Cloud and DevOps are teaching me how to build, automate and operate them.

And I'm discovering that these two perspectives complement each other more than I initially expected.

That's probably the most important lesson I've learned so far in this transition.

Your previous experience doesn't necessarily disappear when you change careers. Sometimes, it becomes your advantage.

AWS EKS Networking: Understanding What Happens Behind the Cluster

Amira Abidi — Tue, 22 Sep 2026 07:53:40 +0000

When I started working with Amazon EKS, I initially focused on the Kubernetes layer:

Pods
Deployments
Services
Ingress
HPA

But running Kubernetes on AWS quickly raises another question:

What actually happens at the network level?

How does a pod communicate with another pod?
How does Internet traffic reach the application?
How does a pod connect to Amazon RDS?

While building my LibraryCorner platform, I spent time understanding how Kubernetes networking interacts with AWS networking.

Here are the main concepts I found important.

1. EKS runs inside an AWS VPC

The first thing to understand is that an EKS cluster is connected to an AWS VPC.

The VPC provides the underlying network for the cluster and its AWS resources.

A simplified architecture looks like this:

                    Internet
                       │
                       ▼
               AWS Load Balancer
                       │
                       ▼
                Kubernetes Service
                       │
                       ▼
                      Pods
                       │
              ┌────────┴────────┐
              ▼                 ▼
             EKS               RDS
           workloads         PostgreSQL

This means that understanding EKS also requires understanding basic AWS networking.

2. Public and private subnets

A common architecture separates public-facing resources from private workloads.

Internet
   │
   ▼
Public subnet
   │
   ▼
Load Balancer
   │
   ▼
Private subnet
   │
   ▼
EKS workloads

The application workloads don't need to be directly exposed to the Internet.

This separation reduces the attack surface and makes the architecture easier to control.

3. What does the NAT Gateway do?

Private workloads may still need outbound Internet access.

For example, a pod may need to reach an external API or download something.

A NAT Gateway provides this outbound path:

Private subnet
      │
      ▼
NAT Gateway
      │
      ▼
Internet Gateway
      │
      ▼
Internet

The important distinction is:

NAT Gateway enables outbound connectivity. It does not make the private workload directly reachable from the Internet.

4. How do Pods get network connectivity?

With the Amazon VPC CNI, Kubernetes pods can receive IP addresses from the VPC network.

Conceptually:

VPC
 │
 ├── EKS Node
 │     ├── Pod → IP
 │     ├── Pod → IP
 │     └── Pod → IP
 │
 └── RDS → IP

This creates a strong relationship between Kubernetes and the AWS network.

It also means that IP address availability matters when scaling an EKS cluster.

More pods can mean more demand for available IP addresses and subnet capacity.

5. How does Internet traffic reach a Pod?

Pods are ephemeral, so we don't want clients connecting directly to individual pod IPs.

Instead, Kubernetes and AWS provide abstractions for routing traffic.

A simplified request path is:

User
 │
 ▼
AWS Load Balancer
 │
 ▼
Kubernetes Service
 │
 ▼
Pod
 │
 ▼
Application

The Load Balancer provides an external entry point, while the Kubernetes Service provides a stable way to reach a group of pods.

If a pod is recreated, the Service can continue routing traffic to the available pods.

6. EKS and RDS

LibraryCorner uses Amazon RDS for PostgreSQL.

The application communicates with the database through the VPC:

EKS Pod
   │
   │ PostgreSQL
   ▼
Amazon RDS

The database does not need to be publicly accessible.

Security rules can restrict database access so that only the required application traffic is allowed.

For example:

Allowed:
EKS → RDS : 5432

Not allowed:
Internet → RDS : 5432

This is a simple but important security principle:

Only expose what needs to be exposed.

7. Multi-AZ and networking

For the production-oriented architecture of LibraryCorner, I used multiple Availability Zones.

              AWS Region
                  │
        ┌─────────┴─────────┐
        │                   │
       AZ-A                AZ-B
        │                   │
    EKS nodes           EKS nodes
        │                   │
        └─────────┬─────────┘
                  │
                 RDS

Multi-AZ architecture requires thinking about subnet placement, routing, IP capacity and resource availability across zones.

Networking is therefore part of the architecture, not just a configuration detail.

8. The lesson I took from this

Before working on EKS networking, I tended to think about Kubernetes mainly in terms of:

Pods
Services
Deployments
Ingress

Now I see a larger picture:

AWS VPC
   │
   ├── Subnets
   ├── Routes
   ├── Security Groups
   ├── NAT
   │
   └── EKS
        │
        ├── Nodes
        ├── Pods
        └── Services

Kubernetes networking on AWS cannot really be understood independently from AWS networking.

That was one of the most useful lessons from building my EKS platform.

Conclusion

Building an EKS platform is not only about deploying containers.

You also need to understand the network underneath them:

VPCs
Subnets
Route tables
NAT Gateways
Security Groups
Load Balancers
Kubernetes Services
Pod networking
Availability Zones
RDS connectivity

For me, understanding these layers was another step from "I can deploy an application" toward "I understand the infrastructure that makes the application work."

This article is part of my LibraryCorner Cloud/DevOps project, where I progressively build and document a containerized application platform on AWS.

Kubernetes Configuration Management with Kustomize: Dev vs Prod

Amira Abidi — Tue, 15 Sep 2026 14:48:06 +0000

Managing multiple Kubernetes environments looks simple at first.

You create a few YAML manifests, deploy them to development, then adapt them for production.

Then reality happens.

Development and production need different replica counts, resource limits, image tags, hostnames, configuration values, and sometimes completely different infrastructure settings.

Copying the manifests for each environment may work for a small project, but it quickly becomes difficult to maintain.

This is where Kustomize becomes useful.

In my LibraryCorner project, I used Kustomize to manage Kubernetes configurations across separate development and production environments while keeping a shared configuration base.

1. The Problem: Kubernetes Manifests Across Environments

Imagine that we have a simple application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: librarycorner-api
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: api
          image: librarycorner-api:latest
          ports:
            - containerPort: 8000

This might be perfectly fine for development.

But production may require:

more replicas
different resource requests and limits
a production image tag
different configuration
different ingress/hostname settings
stricter security settings

One tempting approach is to create:

deployment-dev.yaml
deployment-prod.yaml

and maintain both independently.

The problem is duplication.

If the base Deployment changes, we now have to remember to update multiple copies.

That's where Kustomize provides a cleaner approach.

2. What Is Kustomize?

Kustomize is a Kubernetes-native configuration management tool.

Instead of using templates, Kustomize lets us define a common base configuration and then apply environment-specific customizations through overlays.

The general structure looks like this:

k8s/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
│
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    │
    └── prod/
        └── kustomization.yaml

The important concept is:

                 Base
                  │
          ┌───────┴───────┐
          │               │
         Dev             Prod
       Overlay           Overlay

The base contains what is common.

The overlays contain what changes.

3. Building the Base

Let's start with the common configuration.

`base/deployment.yaml`

apiVersion: apps/v1
kind: Deployment
metadata:
  name: librarycorner-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: librarycorner-api
  template:
    metadata:
      labels:
        app: librarycorner-api
    spec:
      containers:
        - name: api
          image: librarycorner-api
          ports:
            - containerPort: 8000

`base/service.yaml`

apiVersion: v1
kind: Service
metadata:
  name: librarycorner-api
spec:
  selector:
    app: librarycorner-api
  ports:
    - port: 80
      targetPort: 8000

Then we define the resources in base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

At this point, the base describes the application without knowing whether it will run in development or production.

4. Creating the Development Overlay

Now let's create:

overlays/dev/kustomization.yaml

We reference the base:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

We can then apply development-specific changes.

For example, we might want only one replica:

replicas:
  - name: librarycorner-api
    count: 1

We can also modify the container image:

images:
  - name: librarycorner-api
    newName: librarycorner-api
    newTag: dev

The resulting development configuration is generated from the base plus these customizations.

5. Creating the Production Overlay

The production overlay follows the same principle.

overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

replicas:
  - name: librarycorner-api
    count: 3

images:
  - name: librarycorner-api
    newName: librarycorner-api
    newTag: prod

Now we have:

Base
 │
 ├── Dev → 1 replica + dev image
 │
 └── Prod → 3 replicas + prod image

The common Kubernetes configuration remains in one place.

6. Patches for More Specific Changes

Sometimes changing replicas or images isn't enough.

For example, production may require different resource limits.

We can use a patch.

`overlays/prod/resources-patch.yaml`

apiVersion: apps/v1
kind: Deployment
metadata:
  name: librarycorner-api
spec:
  template:
    spec:
      containers:
        - name: api
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"

Then reference the patch from the production overlay:

patches:
  - path: resources-patch.yaml

This keeps the production-specific configuration isolated from the shared base.

7. Environment-Specific Configuration

Another common requirement is changing configuration values between environments.

For example:

ENVIRONMENT=dev

versus:

ENVIRONMENT=prod

Kustomize can generate ConfigMaps from literals or files.

For example:

configMapGenerator:
  - name: librarycorner-config
    literals:
      - ENVIRONMENT=dev

The production overlay can define:

configMapGenerator:
  - name: librarycorner-config
    literals:
      - ENVIRONMENT=prod

This allows the same application manifest to be reused while keeping environment-specific configuration separate.

For sensitive information, however, I would not put credentials directly into Git-managed manifests. Secrets should be handled through an appropriate secret-management solution.

8. Previewing the Result

One of my favorite aspects of Kustomize is that we can render the final Kubernetes manifests before applying them.

For development:

kubectl kustomize overlays/dev

For production:

kubectl kustomize overlays/prod

This makes it possible to inspect exactly what Kubernetes will receive.

We can also apply the configuration directly:

kubectl apply -k overlays/dev

or:

kubectl apply -k overlays/prod

The -k option tells kubectl to use Kustomize.

9. Kustomize in a CI/CD Pipeline

Kustomize becomes particularly useful when combined with CI/CD.

For example, a simplified deployment pipeline can look like:

Git push
   │
   ▼
Build application image
   │
   ▼
Run tests
   │
   ▼
Push image
   │
   ▼
Update Kubernetes configuration
   │
   ▼
Kustomize overlay
   │
   ▼
Deploy to EKS

For example, a pipeline could deploy development using:

kubectl apply -k k8s/overlays/dev

while production uses:

kubectl apply -k k8s/overlays/prod

This makes the Kubernetes configuration part of the application delivery process instead of maintaining deployment settings manually.

10. How I Used It in LibraryCorner

In LibraryCorner, I used separate Kubernetes configurations for development and production.

The goal was not to create two independent Kubernetes applications.

Instead, I wanted:

One common application configuration

plus

Environment-specific differences.

The structure was conceptually:

k8s/
├── base/
│   ├── deployment
│   ├── service
│   ├── config
│   └── kustomization.yaml
│
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    │
    └── prod/
        ├── kustomization.yaml
        └── patches/

This approach helped keep the environments consistent while still allowing production to have different scaling and resource requirements.

It also fits naturally with the infrastructure architecture of the project:

AWS
│
├── Development
│   └── EKS
│       └── Kustomize → Dev configuration
│
└── Production
    └── EKS
        └── Kustomize → Prod configuration

11. Kustomize vs Helm

Kustomize and Helm are often compared because both can be used to manage Kubernetes configurations.

The difference is largely in their approach.

Kustomize

works directly with Kubernetes YAML
uses bases and overlays
doesn't require a templating language
is integrated into kubectl

Helm

uses charts and templates
provides packaging and versioning
has a larger ecosystem for distributing applications
is particularly useful when deploying reusable third-party applications

For my LibraryCorner project, Kustomize was a good fit because I wanted to maintain a relatively small number of environment-specific configurations without introducing a templating layer.

12. What I Learned

The biggest lesson for me was that environment separation is not only an infrastructure problem.

It also exists at the Kubernetes configuration level.

A clean structure such as:

Base + Dev Overlay + Prod Overlay

helps avoid configuration drift and duplicated manifests.

It also makes the deployment process easier to reason about:

Same application
      +
Different environment configuration
      =
Predictable deployments

Kustomize is therefore a relatively small tool, but it solves a very practical problem when Kubernetes configurations start becoming more complex.

Conclusion

Kubernetes manifests are easy to write.

Keeping them maintainable across multiple environments is the harder part.

Kustomize provides a simple Kubernetes-native approach:

Base → Overlays → Environment-specific configuration

For projects running across development and production environments, this can significantly reduce duplication while keeping the differences explicit.

In my case, it became an important part of the LibraryCorner deployment architecture on AWS EKS.

And the next question naturally follows:

How do we make sure these Kubernetes workloads are actually healthy and observable once they're running?

That's where monitoring and observability come in — and that's a topic I've already explored with Prometheus and Grafana in LibraryCorner.

Monitoring AWS EKS with Prometheus and Grafana

Amira Abidi — Tue, 08 Sep 2026 08:35:31 +0000

In my previous articles, I shared how I moved my application from Docker to AWS EKS and how I used Terraform to build the infrastructure.

Once everything was deployed and running, I had another question:

How do I know if my application and Kubernetes cluster are actually healthy?

Seeing pods in a Running state is useful, but it doesn't tell the whole story.

That's where observability came in.

For my project, I implemented a monitoring stack based on Prometheus, Grafana, and Alertmanager.

Here's what I built and, more importantly, what I learned from it.

The Monitoring Stack

I deployed kube-prometheus-stack on my EKS cluster using Helm.

It provides several components out of the box:

Prometheus to collect and store metrics
Grafana to visualize them
Alertmanager to manage alerts
node-exporter for node-level metrics
kube-state-metrics for Kubernetes object metrics

Instead of installing and configuring every component separately, this gave me a solid monitoring foundation for Kubernetes.

The simplified flow looks like this:

EKS Cluster
    │
    ├── Application Pods
    ├── Kubernetes Metrics
    └── Worker Nodes
           │
           ▼
      Prometheus
           │
      ┌────┴────┐
      ▼         ▼
   Grafana   Alertmanager

Prometheus collects the metrics, Grafana turns them into dashboards, and Alertmanager handles situations that require attention.

What Am I Actually Monitoring?

Installing Prometheus is easy.

Knowing what is worth monitoring is more interesting.

For my EKS environment, I wanted visibility at several levels.

Cluster and nodes

I can monitor things such as:

CPU usage
Memory consumption
Node availability
Kubernetes resource usage

This helps answer questions like:

Is the cluster running normally, or are resources becoming saturated?

Kubernetes workloads

At the Kubernetes level, I can observe:

Number of running pods
Pod restarts
Deployment status
Resource consumption per pod

This is particularly useful when troubleshooting.

A service may still be reachable while one of its pods is constantly restarting in the background.

Without monitoring, that kind of behavior can easily go unnoticed.

Grafana: Turning Metrics into Something Useful

Prometheus collecting thousands of metrics isn't very useful if I can't easily interpret them.

That's where Grafana comes in.

I use Grafana dashboards to get a visual overview of the cluster and quickly identify unusual behavior.

For example, instead of repeatedly running:

kubectl top pods

I can observe CPU and memory usage over time.

That “over time” part is important.

A command tells me what is happening now.

Monitoring helps me understand what happened before the problem appeared.

Accessing Grafana

I also wanted Grafana to be accessible without relying on local port forwarding.

I exposed it through an AWS Application Load Balancer, using a dedicated subdomain and its own ACM certificate for HTTPS.

Conceptually:

Browser
   │
   ▼
Grafana subdomain
   │
   ▼
AWS ALB + HTTPS
   │
   ▼
Grafana on EKS

This made the monitoring interface accessible in a way that was closer to how I would expose an internal platform service in a real environment.

Monitoring vs Alerting

Dashboards are useful when someone is looking at them.

But nobody should have to watch Grafana all day.

That's why the stack also includes Alertmanager.

Prometheus can evaluate alerting rules based on collected metrics. When a condition is triggered, Alertmanager is responsible for processing the alert and routing the notification.

This introduced an important distinction for me:

Monitoring tells me what's happening.
Alerting tells me when I need to care.

A good monitoring system shouldn't generate noise for every small variation. The challenge is deciding which conditions actually require action.

Observability Changed How I Look at Kubernetes

One of the biggest lessons from this project was that:

Running is not the same as healthy.

Kubernetes can tell me that a pod exists.

Prometheus can tell me that its memory consumption has been increasing for the last two hours.

Grafana can make that trend immediately visible.

Alertmanager can tell me when that trend crosses a threshold that requires attention.

These tools complement Kubernetes rather than replace its built-in health mechanisms.

What I Learned

Before implementing this stack, I mostly thought about monitoring as dashboards.

Now I see observability as part of the architecture itself.

When deploying an application, I should be able to answer questions like:

Is it available?

Is it performing normally?

Are resources becoming saturated?

Did something change before the incident happened?

And most importantly:

Will I know there's a problem before a user tells me?

Building the monitoring stack made the EKS environment feel much less like a collection of running resources and much more like a system I could actually operate.

What's Next?

Monitoring EKS also forced me to understand more about what happens underneath Kubernetes.

How does traffic reach my application?

Why are my worker nodes in private subnets?

Why do they need NAT Gateways?

And where exactly does the Application Load Balancer fit into all of this?

That's what I'll cover next:

Understanding EKS Networking: ALB, Private Subnets & NAT Gateway

If you're running Kubernetes, what are the first metrics or alerts you usually configure?

How I Built My AWS EKS Infrastructure with Terraform

Amira Abidi — Mon, 31 Aug 2026 11:55:00 +0000

In my previous article, I shared how I moved my application from a local Docker environment to AWS EKS.

This time, I want to focus on one of the most important parts of the project: Infrastructure as Code with Terraform.

My goal was simple: I didn't want my AWS infrastructure to depend on resources created manually from the AWS Console.

I wanted it to be reproducible, structured, and easier to maintain.

Why Terraform?

My architecture includes several AWS components:

VPC and subnets
Internet and NAT Gateways
EKS
RDS PostgreSQL
Application Load Balancer
Route 53
ACM certificates
IAM roles and policies
ECR repositories

Creating everything manually would work, but reproducing the same architecture consistently would quickly become difficult.

That's where Terraform came in.

Instead of describing the infrastructure in documentation, I could describe it directly as code.

Structuring the Infrastructure

One thing I quickly learned is that putting every resource into a single main.tf file doesn't scale very well.

So I organized the infrastructure into reusable Terraform modules.

The structure looks roughly like this:

terraform/
├── modules/
│   ├── vpc/
│   ├── eks/
│   ├── rds/
│   ├── ecr/
│   └── security/
│
└── environments/
    ├── dev/
    └── prod/

Each module has a specific responsibility.

The VPC module manages networking, the EKS module manages the Kubernetes infrastructure, and the RDS module manages the PostgreSQL database.

This made the code much easier to understand and maintain.

Networking First

Before deploying EKS, I needed the network.

I created a VPC distributed across three Availability Zones, with public and private subnets.

The public layer hosts internet-facing components such as the Application Load Balancer and NAT Gateways.

The EKS worker nodes run in private subnets, which means they aren't directly exposed to the internet.

The simplified architecture is:

Internet
   │
   ▼
Application Load Balancer
   │
   ▼
Private Subnets
   │
   ├── EKS Worker Nodes
   │
   └── RDS PostgreSQL

This was an important lesson for me:

Using Kubernetes doesn't remove the need to understand networking.

You still need to think about routing, security groups, public/private subnets, DNS, and internet access.

Creating the EKS Cluster

Once the network was ready, Terraform could create the EKS cluster and its node groups.

The worker nodes are distributed across the three Availability Zones.

From there, Kubernetes manages the application workloads.

My application itself isn't defined directly in Terraform. Kubernetes resources are managed separately, allowing Terraform to focus primarily on the AWS infrastructure.

I found this separation useful:

Terraform manages the infrastructure. Kubernetes manages the workloads.

Managing Multiple Environments

Another requirement was separating development and production.

Instead of maintaining two completely different Terraform codebases, both environments reuse the same modules with different configurations.

Conceptually:

              ┌── dev
Terraform ────┤
              └── prod
                  │
                  ▼
             Shared Modules

This means I can change the infrastructure logic once while keeping environment-specific values separate.

Each environment has its own EKS cluster and RDS database.

Remote Terraform State

Terraform state is critical because it maps the configuration to the resources that actually exist in AWS.

Keeping it only on my laptop wasn't something I wanted to rely on.

So I configured a remote backend using Amazon S3.

This makes the infrastructure state persistent and independent from my local machine.

It also reinforced an important point:

Terraform state is part of the infrastructure and needs to be protected accordingly.

What I Learned

Before this project, Terraform looked mostly like a way to automate AWS resource creation.

After building a complete environment with it, I see it differently.

The real value isn't just:

“I can create an EKS cluster with Terraform.”

It's being able to describe an architecture in a way that is repeatable, reviewable, and maintainable.

I also learned that the order of thinking matters.

Before writing Terraform code, I needed to understand the architecture:

networking → security → compute → database → application → observability

Terraform doesn't design the architecture for you.

It forces you to express the architecture you've designed.

What's Next?

Infrastructure is now reproducible and the application is running on Kubernetes.

But deploying a service isn't enough.

The next question is:

How do I know if my Kubernetes cluster and application are actually healthy?

In the next article, I'll look at how I added Prometheus and Grafana monitoring to AWS EKS and what I learned about observability along the way.

From a Local Docker App to AWS EKS: My DevOps Project

Amira Abidi — Sat, 22 Aug 2026 10:36:43 +0000

I recently worked on a DevOps project with a simple starting point: an application that worked perfectly fine on my local machine.

The application, LibraryCorner, consists of a frontend, a FastAPI backend, and a PostgreSQL database. It was already containerized with Docker, so technically, everything worked.

But “it works on my machine” is very different from running a service in a production-like environment.

So I gave myself a challenge:

How can I transform a local application into a service that is deployable, highly available, observable, and recoverable?

These four properties became the foundation of the project.

The Architecture

I deployed the application on AWS using EKS for container orchestration and RDS PostgreSQL for the database.

The request flow looks roughly like this:

User → Route 53 → Application Load Balancer → EKS → RDS PostgreSQL

The EKS nodes and the database run inside private subnets, while the Application Load Balancer acts as the public entry point.

Around this core architecture, I used:

Terraform for Infrastructure as Code
GitLab CI/CD for automation
ECR for Docker images
AWS Secrets Manager for secrets
Prometheus & Grafana for monitoring
CloudWatch for AWS monitoring
S3 for Terraform state

I also separated the development and production environments, each with its own infrastructure.

1. Deployable

My first goal was to avoid manual infrastructure changes as much as possible.

The AWS infrastructure is managed with Terraform, while the application deployment is handled through GitLab CI/CD.

The pipeline builds the Docker images, pushes them to ECR, and deploys the application to Kubernetes.

For AWS authentication, I used OIDC instead of storing long-lived AWS access keys in GitLab.

I actually wrote a separate article about this because I found the authentication flow particularly interesting.

2. Highly Available

Deploying something to Kubernetes doesn't automatically make it highly available.

I configured the EKS infrastructure across multiple Availability Zones and deployed multiple application replicas.

An Application Load Balancer distributes incoming traffic, while Kubernetes can replace unhealthy pods.

I also configured a Horizontal Pod Autoscaler (HPA) to scale the application between 2 and 6 pods depending on resource usage.

For the production database, RDS uses a Multi-AZ configuration.

The idea is simple: the application shouldn't depend on a single instance, pod, or Availability Zone.

3. Observable

A service can be running and still be unhealthy.

That's why monitoring was another important part of the project.

I used Prometheus to collect Kubernetes and application metrics and Grafana to visualize them.

This gave me visibility into things like resource usage, pod health, and application behavior.

I also configured alerts so that important problems don't require someone to constantly watch a dashboard.

This part of the project changed the way I think about infrastructure: deploying an application is only the beginning. You also need to understand what happens after deployment.

4. Recoverable

High availability doesn't replace backups.

I wanted to know what would happen if the database actually had to be restored.

So instead of simply enabling RDS snapshots and assuming recovery would work, I performed a real restoration test.

The measured recovery time was approximately 14 minutes.

For me, this was one of the most useful exercises in the project because it turned disaster recovery from a configuration checkbox into something measurable.

What I Learned

The biggest lesson wasn't about a specific AWS service.

It was understanding how all these pieces work together.

Terraform creates the infrastructure. GitLab automates delivery. Kubernetes manages the workloads. AWS provides the underlying services. Prometheus and Grafana tell me what's happening. Backups give me a recovery path when things go wrong.

And each layer introduces its own decisions and trade-offs.

This project also reminded me of something from my QA background:

A system isn't reliable because we expect it to work. It's reliable because we can verify what happens when it doesn't.

What's Next?

This article was an overview of the architecture, but several parts deserve their own deep dive.

Next, I'll share how I built the AWS EKS infrastructure with Terraform, including some of the problems I encountered and what I learned while fixing them.

If you're also learning AWS, Kubernetes, or DevOps by building projects, I'd love to hear about what you're working on.

How I Removed AWS Access Keys from GitLab CI/CD with OIDC

Amira Abidi — Wed, 12 Aug 2026 09:05:09 +0000

When I first connected my GitLab CI/CD pipelines to AWS, I used the simplest solution: an IAM user with an Access Key and Secret Access Key stored as GitLab CI/CD variables.

It worked.

But there was one problem: those credentials were permanent.

They had to be stored, protected and eventually rotated. If they were accidentally exposed in logs or compromised, they could remain valid until manually revoked.

I wanted a cleaner solution.

So I replaced permanent AWS credentials with OIDC federation between GitLab and AWS.

The result is simple: GitLab pipelines can access AWS without storing any permanent AWS credentials.

In this post, I'll explain how I implemented it, how the authentication flow works, and one important issue I faced when using it with EKS and Terraform.

The architecture

The authentication flow looks like this:

┌──────────────┐
│  GitLab CI   │
└──────┬───────┘
       │
       │ OIDC token
       ▼
┌──────────────┐
│   AWS STS    │
└──────┬───────┘
       │
       │ Temporary credentials
       ▼
┌──────────────┐
│   IAM Role   │
└──────┬───────┘
       │
       ├──────────► Terraform
       │
       ├──────────► ECR
       │
       └──────────► EKS

Instead of GitLab storing an AWS Access Key, it proves its identity to AWS using a short-lived OIDC token.

AWS verifies the token and returns temporary credentials.

How does OIDC authentication work?

The process can be summarized in five steps:

GitLab creates an OIDC token for the CI/CD job.
The pipeline sends this token to AWS.
AWS verifies that the token really comes from GitLab.
AWS checks that the project is allowed to assume the requested IAM role.
AWS STS returns temporary credentials.

These credentials expire automatically.

So there is nothing permanent to store or rotate inside GitLab.

Step 1 — Register GitLab as an OIDC provider

AWS first needs to trust GitLab as an identity provider.

I configured the OIDC provider using Terraform:

data "tls_certificate" "gitlab" {
  url = "${var.gitlab_url}/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "gitlab" {
  url             = var.gitlab_url
  client_id_list  = [var.gitlab_url]
  thumbprint_list = [
    data.tls_certificate.gitlab.certificates[0].sha1_fingerprint
  ]
}

This tells AWS that tokens issued by GitLab can be used for authentication.

But trusting GitLab itself is not enough.

AWS also needs to know which GitLab projects are allowed to access which IAM roles.

Step 2 — Restrict access with the IAM Trust Policy

This is one of the most important parts of the configuration.

We don't want any GitLab project to be able to assume our AWS roles.

The IAM Trust Policy checks information contained in the GitLab token.

For example:

condition {
  test     = "StringEquals"
  variable = "gitlab.com:aud"
  values   = [var.gitlab_url]
}

condition {
  test     = "StringLike"
  variable = "gitlab.com:sub"
  values   = [
    "project_path:${each.value}:ref_type:branch:ref:*"
  ]
}

condition {
  test     = "StringEquals"
  variable = "gitlab.com:namespace_id"
  values   = [var.gitlab_namespace_id]
}

The important values here are:

aud: checks the expected audience.
sub: identifies the GitLab project and branch.
namespace_id: identifies the GitLab namespace.

I especially like using namespace_id because it is a stable numeric identifier.

GitLab project and group names can change, but the namespace ID provides an additional way to make sure the token comes from the expected namespace.

Step 3 — One IAM role per repository

My project uses four GitLab repositories, so I created a separate IAM role for each one.

Repository	Main AWS permissions
Backend	Push Docker images to ECR
Frontend	Push Docker images to ECR
Infrastructure	Manage AWS infrastructure with Terraform
Deploy	Check ECR images and deploy to EKS

Why separate them?

Because the backend pipeline doesn't need permission to modify the EKS cluster.

And the deployment pipeline doesn't need permission to push Docker images.

Each pipeline gets only the permissions required for its job.

For example, the deployment role only needs to check whether an image exists:

actions = [
  "ecr:DescribeImages",
  "ecr:DescribeRepositories"
]

It cannot push a new image.

This gives me a much cleaner separation of responsibilities.

Step 4 — Request the OIDC token from GitLab

On the GitLab side, the configuration is surprisingly small.

GitLab can generate the OIDC token directly inside the CI/CD job:

id_tokens:
  GITLAB_OIDC_TOKEN:
    aud: https://gitlab.com

Then I define the AWS role and the location of the token:

variables:
  AWS_REGION: "us-east-1"
  AWS_ROLE_ARN: "arn:aws:iam::123456789012:role/gitlab-ci-book-infra"
  AWS_WEB_IDENTITY_TOKEN_FILE: "${CI_PROJECT_DIR}/.gitlab-oidc-token"

And write the token to the file:

before_script:
  - echo "$GITLAB_OIDC_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"

That's basically it.

Terraform and AWS SDKs understand the web identity authentication flow automatically.

There is no need to run:

aws configure

And more importantly, I don't need to store permanent values such as:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

in GitLab CI/CD variables anymore.

Step 5 — EKS needs an extra configuration

This was an important thing I learned while implementing the solution.

Successfully authenticating with AWS does not automatically mean that the pipeline can access Kubernetes.

For example:

aws eks update-kubeconfig

may work correctly while:

kubectl get pods

returns:

Unauthorized

Why?

Because there are two different authorization levels:

AWS IAM
   │
   │ Can this role access the AWS/EKS API?
   ▼
EKS Cluster
   │
   │ Is this IAM role allowed inside Kubernetes?
   ▼
Kubernetes resources

The IAM role therefore also needs access to the EKS cluster.

I configured this with an EKS access entry:

resource "aws_eks_access_entry" "ci_deploy" {
  cluster_name  = module.eks.cluster_name
  principal_arn = var.ci_deploy_role_arn
  type          = "STANDARD"
}

And associated the required EKS access policy with it.

Once this was configured, the deployment pipeline could authenticate correctly to the cluster.

The Terraform + EKS issue I ran into

This was probably the most interesting problem I encountered.

EKS authentication tokens are short-lived.

Initially, I used:

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}

This works when terraform plan and terraform apply happen immediately.

But my pipeline separates them.

The flow is:

terraform plan
      │
      ▼
Manual approval
      │
      ▼
terraform apply

The problem is that the EKS token can be generated during terraform plan.

If I wait before approving the apply, the saved token may already be expired.

Terraform then fails when trying to communicate with Kubernetes.

The solution: generate the token when it is needed

Instead of storing the token during the plan, I use an exec block:

provider "kubernetes" {
  host = module.eks.cluster_endpoint

  cluster_ca_certificate = base64decode(
    module.eks.cluster_certificate_authority_data
  )

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws-iam-authenticator"
    args        = [
      "token",
      "-i",
      module.eks.cluster_name
    ]
  }
}

Now the token is generated when Terraform actually needs to connect to Kubernetes.

So even if I wait before approving terraform apply, Terraform gets a fresh token.

Another small GitLab CI lesson

Manual GitLab jobs can allow failures by default depending on how they are configured.

For an infrastructure deployment, that's something I definitely don't want.

If terraform apply fails, the pipeline should fail too.

So I explicitly use:

apply:
  rules:
    - if: '$CI_COMMIT_BRANCH'
      when: manual
      allow_failure: false

It's a small configuration detail, but an important one for infrastructure pipelines.

Debugging OIDC authentication

When authentication fails, I usually check three things first:

aud
sub
namespace_id

Typical errors include:

Error	What I check
`AssumeRoleWithWebIdentity` denied	Project path / `sub`
`Incorrect token audience`	`aud` configuration
`InvalidIdentityToken`	Token expiration / OIDC provider
`Unauthorized` from `kubectl`	EKS access entry

I also keep this command in the pipeline:

aws sts get-caller-identity

It's very useful because it immediately shows which AWS identity the pipeline is actually using.

What I learned

The biggest lesson from this implementation is that OIDC is not as complicated as it first looks.

The basic idea is simply:

GitLab proves its identity
        ↓
AWS verifies the GitLab project
        ↓
AWS STS provides temporary credentials
        ↓
The pipeline accesses AWS

The main advantages for me are:

No permanent AWS credentials in GitLab.
No Access Keys to rotate.
Temporary credentials expire automatically.
Each repository has its own IAM role.
Permissions can be limited per pipeline.
AWS activity can be traced through CloudTrail.
GitLab can securely deploy to EKS.

There are a few extra details when EKS is involved, especially around cluster access and short-lived Kubernetes tokens, but once those pieces are understood, the overall architecture is quite simple.

For CI/CD pipelines, I now prefer this approach over storing permanent AWS credentials whenever OIDC federation is available.