DEV Community: Amira Abidi

AWS EKS Networking: Understanding What Happens Behind the Cluster

Amira Abidi — Tue, 22 Sep 2026 07:53:40 +0000

When I started working with Amazon EKS, I initially focused on the Kubernetes layer:

Pods
Deployments
Services
Ingress
HPA

But running Kubernetes on AWS quickly raises another question:

What actually happens at the network level?

How does a pod communicate with another pod?
How does Internet traffic reach the application?
How does a pod connect to Amazon RDS?

While building my LibraryCorner platform, I spent time understanding how Kubernetes networking interacts with AWS networking.

Here are the main concepts I found important.

1. EKS runs inside an AWS VPC

The first thing to understand is that an EKS cluster is connected to an AWS VPC.

The VPC provides the underlying network for the cluster and its AWS resources.

A simplified architecture looks like this:

                    Internet
                       │
                       ▼
               AWS Load Balancer
                       │
                       ▼
                Kubernetes Service
                       │
                       ▼
                      Pods
                       │
              ┌────────┴────────┐
              ▼                 ▼
             EKS               RDS
           workloads         PostgreSQL

This means that understanding EKS also requires understanding basic AWS networking.

2. Public and private subnets

A common architecture separates public-facing resources from private workloads.

Internet
   │
   ▼
Public subnet
   │
   ▼
Load Balancer
   │
   ▼
Private subnet
   │
   ▼
EKS workloads

The application workloads don't need to be directly exposed to the Internet.

This separation reduces the attack surface and makes the architecture easier to control.

3. What does the NAT Gateway do?

Private workloads may still need outbound Internet access.

For example, a pod may need to reach an external API or download something.

A NAT Gateway provides this outbound path:

Private subnet
      │
      ▼
NAT Gateway
      │
      ▼
Internet Gateway
      │
      ▼
Internet

The important distinction is:

NAT Gateway enables outbound connectivity. It does not make the private workload directly reachable from the Internet.

4. How do Pods get network connectivity?

With the Amazon VPC CNI, Kubernetes pods can receive IP addresses from the VPC network.

Conceptually:

VPC
 │
 ├── EKS Node
 │     ├── Pod → IP
 │     ├── Pod → IP
 │     └── Pod → IP
 │
 └── RDS → IP

This creates a strong relationship between Kubernetes and the AWS network.

It also means that IP address availability matters when scaling an EKS cluster.

More pods can mean more demand for available IP addresses and subnet capacity.

5. How does Internet traffic reach a Pod?

Pods are ephemeral, so we don't want clients connecting directly to individual pod IPs.

Instead, Kubernetes and AWS provide abstractions for routing traffic.

A simplified request path is:

User
 │
 ▼
AWS Load Balancer
 │
 ▼
Kubernetes Service
 │
 ▼
Pod
 │
 ▼
Application

The Load Balancer provides an external entry point, while the Kubernetes Service provides a stable way to reach a group of pods.

If a pod is recreated, the Service can continue routing traffic to the available pods.

6. EKS and RDS

LibraryCorner uses Amazon RDS for PostgreSQL.

The application communicates with the database through the VPC:

EKS Pod
   │
   │ PostgreSQL
   ▼
Amazon RDS

The database does not need to be publicly accessible.

Security rules can restrict database access so that only the required application traffic is allowed.

For example:

Allowed:
EKS → RDS : 5432

Not allowed:
Internet → RDS : 5432

This is a simple but important security principle:

Only expose what needs to be exposed.

7. Multi-AZ and networking

For the production-oriented architecture of LibraryCorner, I used multiple Availability Zones.

              AWS Region
                  │
        ┌─────────┴─────────┐
        │                   │
       AZ-A                AZ-B
        │                   │
    EKS nodes           EKS nodes
        │                   │
        └─────────┬─────────┘
                  │
                 RDS

Multi-AZ architecture requires thinking about subnet placement, routing, IP capacity and resource availability across zones.

Networking is therefore part of the architecture, not just a configuration detail.

8. The lesson I took from this

Before working on EKS networking, I tended to think about Kubernetes mainly in terms of:

Pods
Services
Deployments
Ingress

Now I see a larger picture:

AWS VPC
   │
   ├── Subnets
   ├── Routes
   ├── Security Groups
   ├── NAT
   │
   └── EKS
        │
        ├── Nodes
        ├── Pods
        └── Services

Kubernetes networking on AWS cannot really be understood independently from AWS networking.

That was one of the most useful lessons from building my EKS platform.

Conclusion

Building an EKS platform is not only about deploying containers.

You also need to understand the network underneath them:

VPCs
Subnets
Route tables
NAT Gateways
Security Groups
Load Balancers
Kubernetes Services
Pod networking
Availability Zones
RDS connectivity

For me, understanding these layers was another step from "I can deploy an application" toward "I understand the infrastructure that makes the application work."

This article is part of my LibraryCorner Cloud/DevOps project, where I progressively build and document a containerized application platform on AWS.

Kubernetes Configuration Management with Kustomize: Dev vs Prod

Amira Abidi — Tue, 15 Sep 2026 14:48:06 +0000

Managing multiple Kubernetes environments looks simple at first.

You create a few YAML manifests, deploy them to development, then adapt them for production.

Then reality happens.

Development and production need different replica counts, resource limits, image tags, hostnames, configuration values, and sometimes completely different infrastructure settings.

Copying the manifests for each environment may work for a small project, but it quickly becomes difficult to maintain.

This is where Kustomize becomes useful.

In my LibraryCorner project, I used Kustomize to manage Kubernetes configurations across separate development and production environments while keeping a shared configuration base.

1. The Problem: Kubernetes Manifests Across Environments

Imagine that we have a simple application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: librarycorner-api
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: api
          image: librarycorner-api:latest
          ports:
            - containerPort: 8000

This might be perfectly fine for development.

But production may require:

more replicas
different resource requests and limits
a production image tag
different configuration
different ingress/hostname settings
stricter security settings

One tempting approach is to create:

deployment-dev.yaml
deployment-prod.yaml

and maintain both independently.

The problem is duplication.

If the base Deployment changes, we now have to remember to update multiple copies.

That's where Kustomize provides a cleaner approach.

2. What Is Kustomize?

Kustomize is a Kubernetes-native configuration management tool.

Instead of using templates, Kustomize lets us define a common base configuration and then apply environment-specific customizations through overlays.

The general structure looks like this:

k8s/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
│
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    │
    └── prod/
        └── kustomization.yaml

The important concept is:

                 Base
                  │
          ┌───────┴───────┐
          │               │
         Dev             Prod
       Overlay           Overlay

The base contains what is common.

The overlays contain what changes.

3. Building the Base

Let's start with the common configuration.

`base/deployment.yaml`

apiVersion: apps/v1
kind: Deployment
metadata:
  name: librarycorner-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: librarycorner-api
  template:
    metadata:
      labels:
        app: librarycorner-api
    spec:
      containers:
        - name: api
          image: librarycorner-api
          ports:
            - containerPort: 8000

`base/service.yaml`

apiVersion: v1
kind: Service
metadata:
  name: librarycorner-api
spec:
  selector:
    app: librarycorner-api
  ports:
    - port: 80
      targetPort: 8000

Then we define the resources in base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

At this point, the base describes the application without knowing whether it will run in development or production.

4. Creating the Development Overlay

Now let's create:

overlays/dev/kustomization.yaml

We reference the base:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

We can then apply development-specific changes.

For example, we might want only one replica:

replicas:
  - name: librarycorner-api
    count: 1

We can also modify the container image:

images:
  - name: librarycorner-api
    newName: librarycorner-api
    newTag: dev

The resulting development configuration is generated from the base plus these customizations.

5. Creating the Production Overlay

The production overlay follows the same principle.

overlays/prod/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

replicas:
  - name: librarycorner-api
    count: 3

images:
  - name: librarycorner-api
    newName: librarycorner-api
    newTag: prod

Now we have:

Base
 │
 ├── Dev → 1 replica + dev image
 │
 └── Prod → 3 replicas + prod image

The common Kubernetes configuration remains in one place.

6. Patches for More Specific Changes

Sometimes changing replicas or images isn't enough.

For example, production may require different resource limits.

We can use a patch.

`overlays/prod/resources-patch.yaml`

apiVersion: apps/v1
kind: Deployment
metadata:
  name: librarycorner-api
spec:
  template:
    spec:
      containers:
        - name: api
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"

Then reference the patch from the production overlay:

patches:
  - path: resources-patch.yaml

This keeps the production-specific configuration isolated from the shared base.

7. Environment-Specific Configuration

Another common requirement is changing configuration values between environments.

For example:

ENVIRONMENT=dev

versus:

ENVIRONMENT=prod

Kustomize can generate ConfigMaps from literals or files.

For example:

configMapGenerator:
  - name: librarycorner-config
    literals:
      - ENVIRONMENT=dev

The production overlay can define:

configMapGenerator:
  - name: librarycorner-config
    literals:
      - ENVIRONMENT=prod

This allows the same application manifest to be reused while keeping environment-specific configuration separate.

For sensitive information, however, I would not put credentials directly into Git-managed manifests. Secrets should be handled through an appropriate secret-management solution.

8. Previewing the Result

One of my favorite aspects of Kustomize is that we can render the final Kubernetes manifests before applying them.

For development:

kubectl kustomize overlays/dev

For production:

kubectl kustomize overlays/prod

This makes it possible to inspect exactly what Kubernetes will receive.

We can also apply the configuration directly:

kubectl apply -k overlays/dev

or:

kubectl apply -k overlays/prod

The -k option tells kubectl to use Kustomize.

9. Kustomize in a CI/CD Pipeline

Kustomize becomes particularly useful when combined with CI/CD.

For example, a simplified deployment pipeline can look like:

Git push
   │
   ▼
Build application image
   │
   ▼
Run tests
   │
   ▼
Push image
   │
   ▼
Update Kubernetes configuration
   │
   ▼
Kustomize overlay
   │
   ▼
Deploy to EKS

For example, a pipeline could deploy development using:

kubectl apply -k k8s/overlays/dev

while production uses:

kubectl apply -k k8s/overlays/prod

This makes the Kubernetes configuration part of the application delivery process instead of maintaining deployment settings manually.

10. How I Used It in LibraryCorner

In LibraryCorner, I used separate Kubernetes configurations for development and production.

The goal was not to create two independent Kubernetes applications.

Instead, I wanted:

One common application configuration

plus

Environment-specific differences.

The structure was conceptually:

k8s/
├── base/
│   ├── deployment
│   ├── service
│   ├── config
│   └── kustomization.yaml
│
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    │
    └── prod/
        ├── kustomization.yaml
        └── patches/

This approach helped keep the environments consistent while still allowing production to have different scaling and resource requirements.

It also fits naturally with the infrastructure architecture of the project:

AWS
│
├── Development
│   └── EKS
│       └── Kustomize → Dev configuration
│
└── Production
    └── EKS
        └── Kustomize → Prod configuration

11. Kustomize vs Helm

Kustomize and Helm are often compared because both can be used to manage Kubernetes configurations.

The difference is largely in their approach.

Kustomize

works directly with Kubernetes YAML
uses bases and overlays
doesn't require a templating language
is integrated into kubectl

Helm

uses charts and templates
provides packaging and versioning
has a larger ecosystem for distributing applications
is particularly useful when deploying reusable third-party applications

For my LibraryCorner project, Kustomize was a good fit because I wanted to maintain a relatively small number of environment-specific configurations without introducing a templating layer.

12. What I Learned

The biggest lesson for me was that environment separation is not only an infrastructure problem.

It also exists at the Kubernetes configuration level.

A clean structure such as:

Base + Dev Overlay + Prod Overlay

helps avoid configuration drift and duplicated manifests.

It also makes the deployment process easier to reason about:

Same application
      +
Different environment configuration
      =
Predictable deployments

Kustomize is therefore a relatively small tool, but it solves a very practical problem when Kubernetes configurations start becoming more complex.

Conclusion

Kubernetes manifests are easy to write.

Keeping them maintainable across multiple environments is the harder part.

Kustomize provides a simple Kubernetes-native approach:

Base → Overlays → Environment-specific configuration

For projects running across development and production environments, this can significantly reduce duplication while keeping the differences explicit.

In my case, it became an important part of the LibraryCorner deployment architecture on AWS EKS.

And the next question naturally follows:

How do we make sure these Kubernetes workloads are actually healthy and observable once they're running?

That's where monitoring and observability come in — and that's a topic I've already explored with Prometheus and Grafana in LibraryCorner.

Monitoring AWS EKS with Prometheus and Grafana

Amira Abidi — Tue, 08 Sep 2026 08:35:31 +0000

In my previous articles, I shared how I moved my application from Docker to AWS EKS and how I used Terraform to build the infrastructure.

Once everything was deployed and running, I had another question:

How do I know if my application and Kubernetes cluster are actually healthy?

Seeing pods in a Running state is useful, but it doesn't tell the whole story.

That's where observability came in.

For my project, I implemented a monitoring stack based on Prometheus, Grafana, and Alertmanager.

Here's what I built and, more importantly, what I learned from it.

The Monitoring Stack

I deployed kube-prometheus-stack on my EKS cluster using Helm.

It provides several components out of the box:

Prometheus to collect and store metrics
Grafana to visualize them
Alertmanager to manage alerts
node-exporter for node-level metrics
kube-state-metrics for Kubernetes object metrics

Instead of installing and configuring every component separately, this gave me a solid monitoring foundation for Kubernetes.

The simplified flow looks like this:

EKS Cluster
    │
    ├── Application Pods
    ├── Kubernetes Metrics
    └── Worker Nodes
           │
           ▼
      Prometheus
           │
      ┌────┴────┐
      ▼         ▼
   Grafana   Alertmanager

Prometheus collects the metrics, Grafana turns them into dashboards, and Alertmanager handles situations that require attention.

What Am I Actually Monitoring?

Installing Prometheus is easy.

Knowing what is worth monitoring is more interesting.

For my EKS environment, I wanted visibility at several levels.

Cluster and nodes

I can monitor things such as:

CPU usage
Memory consumption
Node availability
Kubernetes resource usage

This helps answer questions like:

Is the cluster running normally, or are resources becoming saturated?

Kubernetes workloads

At the Kubernetes level, I can observe:

Number of running pods
Pod restarts
Deployment status
Resource consumption per pod

This is particularly useful when troubleshooting.

A service may still be reachable while one of its pods is constantly restarting in the background.

Without monitoring, that kind of behavior can easily go unnoticed.

Grafana: Turning Metrics into Something Useful

Prometheus collecting thousands of metrics isn't very useful if I can't easily interpret them.

That's where Grafana comes in.

I use Grafana dashboards to get a visual overview of the cluster and quickly identify unusual behavior.

For example, instead of repeatedly running:

kubectl top pods

I can observe CPU and memory usage over time.

That “over time” part is important.

A command tells me what is happening now.

Monitoring helps me understand what happened before the problem appeared.

Accessing Grafana

I also wanted Grafana to be accessible without relying on local port forwarding.

I exposed it through an AWS Application Load Balancer, using a dedicated subdomain and its own ACM certificate for HTTPS.

Conceptually:

Browser
   │
   ▼
Grafana subdomain
   │
   ▼
AWS ALB + HTTPS
   │
   ▼
Grafana on EKS

This made the monitoring interface accessible in a way that was closer to how I would expose an internal platform service in a real environment.

Monitoring vs Alerting

Dashboards are useful when someone is looking at them.

But nobody should have to watch Grafana all day.

That's why the stack also includes Alertmanager.

Prometheus can evaluate alerting rules based on collected metrics. When a condition is triggered, Alertmanager is responsible for processing the alert and routing the notification.

This introduced an important distinction for me:

Monitoring tells me what's happening.
Alerting tells me when I need to care.

A good monitoring system shouldn't generate noise for every small variation. The challenge is deciding which conditions actually require action.

Observability Changed How I Look at Kubernetes

One of the biggest lessons from this project was that:

Running is not the same as healthy.

Kubernetes can tell me that a pod exists.

Prometheus can tell me that its memory consumption has been increasing for the last two hours.

Grafana can make that trend immediately visible.

Alertmanager can tell me when that trend crosses a threshold that requires attention.

These tools complement Kubernetes rather than replace its built-in health mechanisms.

What I Learned

Before implementing this stack, I mostly thought about monitoring as dashboards.

Now I see observability as part of the architecture itself.

When deploying an application, I should be able to answer questions like:

Is it available?

Is it performing normally?

Are resources becoming saturated?

Did something change before the incident happened?

And most importantly:

Will I know there's a problem before a user tells me?

Building the monitoring stack made the EKS environment feel much less like a collection of running resources and much more like a system I could actually operate.

What's Next?

Monitoring EKS also forced me to understand more about what happens underneath Kubernetes.

How does traffic reach my application?

Why are my worker nodes in private subnets?

Why do they need NAT Gateways?

And where exactly does the Application Load Balancer fit into all of this?

That's what I'll cover next:

Understanding EKS Networking: ALB, Private Subnets & NAT Gateway

If you're running Kubernetes, what are the first metrics or alerts you usually configure?

How I Built My AWS EKS Infrastructure with Terraform

Amira Abidi — Mon, 31 Aug 2026 11:55:00 +0000

In my previous article, I shared how I moved my application from a local Docker environment to AWS EKS.

This time, I want to focus on one of the most important parts of the project: Infrastructure as Code with Terraform.

My goal was simple: I didn't want my AWS infrastructure to depend on resources created manually from the AWS Console.

I wanted it to be reproducible, structured, and easier to maintain.

Why Terraform?

My architecture includes several AWS components:

VPC and subnets
Internet and NAT Gateways
EKS
RDS PostgreSQL
Application Load Balancer
Route 53
ACM certificates
IAM roles and policies
ECR repositories

Creating everything manually would work, but reproducing the same architecture consistently would quickly become difficult.

That's where Terraform came in.

Instead of describing the infrastructure in documentation, I could describe it directly as code.

Structuring the Infrastructure

One thing I quickly learned is that putting every resource into a single main.tf file doesn't scale very well.

So I organized the infrastructure into reusable Terraform modules.

The structure looks roughly like this:

terraform/
├── modules/
│   ├── vpc/
│   ├── eks/
│   ├── rds/
│   ├── ecr/
│   └── security/
│
└── environments/
    ├── dev/
    └── prod/

Each module has a specific responsibility.

The VPC module manages networking, the EKS module manages the Kubernetes infrastructure, and the RDS module manages the PostgreSQL database.

This made the code much easier to understand and maintain.

Networking First

Before deploying EKS, I needed the network.

I created a VPC distributed across three Availability Zones, with public and private subnets.

The public layer hosts internet-facing components such as the Application Load Balancer and NAT Gateways.

The EKS worker nodes run in private subnets, which means they aren't directly exposed to the internet.

The simplified architecture is:

Internet
   │
   ▼
Application Load Balancer
   │
   ▼
Private Subnets
   │
   ├── EKS Worker Nodes
   │
   └── RDS PostgreSQL

This was an important lesson for me:

Using Kubernetes doesn't remove the need to understand networking.

You still need to think about routing, security groups, public/private subnets, DNS, and internet access.

Creating the EKS Cluster

Once the network was ready, Terraform could create the EKS cluster and its node groups.

The worker nodes are distributed across the three Availability Zones.

From there, Kubernetes manages the application workloads.

My application itself isn't defined directly in Terraform. Kubernetes resources are managed separately, allowing Terraform to focus primarily on the AWS infrastructure.

I found this separation useful:

Terraform manages the infrastructure. Kubernetes manages the workloads.

Managing Multiple Environments

Another requirement was separating development and production.

Instead of maintaining two completely different Terraform codebases, both environments reuse the same modules with different configurations.

Conceptually:

              ┌── dev
Terraform ────┤
              └── prod
                  │
                  ▼
             Shared Modules

This means I can change the infrastructure logic once while keeping environment-specific values separate.

Each environment has its own EKS cluster and RDS database.

Remote Terraform State

Terraform state is critical because it maps the configuration to the resources that actually exist in AWS.

Keeping it only on my laptop wasn't something I wanted to rely on.

So I configured a remote backend using Amazon S3.

This makes the infrastructure state persistent and independent from my local machine.

It also reinforced an important point:

Terraform state is part of the infrastructure and needs to be protected accordingly.

What I Learned

Before this project, Terraform looked mostly like a way to automate AWS resource creation.

After building a complete environment with it, I see it differently.

The real value isn't just:

“I can create an EKS cluster with Terraform.”

It's being able to describe an architecture in a way that is repeatable, reviewable, and maintainable.

I also learned that the order of thinking matters.

Before writing Terraform code, I needed to understand the architecture:

networking → security → compute → database → application → observability

Terraform doesn't design the architecture for you.

It forces you to express the architecture you've designed.

What's Next?

Infrastructure is now reproducible and the application is running on Kubernetes.

But deploying a service isn't enough.

The next question is:

How do I know if my Kubernetes cluster and application are actually healthy?

In the next article, I'll look at how I added Prometheus and Grafana monitoring to AWS EKS and what I learned about observability along the way.

From a Local Docker App to AWS EKS: My DevOps Project

Amira Abidi — Sat, 22 Aug 2026 10:36:43 +0000

I recently worked on a DevOps project with a simple starting point: an application that worked perfectly fine on my local machine.

The application, LibraryCorner, consists of a frontend, a FastAPI backend, and a PostgreSQL database. It was already containerized with Docker, so technically, everything worked.

But “it works on my machine” is very different from running a service in a production-like environment.

So I gave myself a challenge:

How can I transform a local application into a service that is deployable, highly available, observable, and recoverable?

These four properties became the foundation of the project.

The Architecture

I deployed the application on AWS using EKS for container orchestration and RDS PostgreSQL for the database.

The request flow looks roughly like this:

User → Route 53 → Application Load Balancer → EKS → RDS PostgreSQL

The EKS nodes and the database run inside private subnets, while the Application Load Balancer acts as the public entry point.

Around this core architecture, I used:

Terraform for Infrastructure as Code
GitLab CI/CD for automation
ECR for Docker images
AWS Secrets Manager for secrets
Prometheus & Grafana for monitoring
CloudWatch for AWS monitoring
S3 for Terraform state

I also separated the development and production environments, each with its own infrastructure.

1. Deployable

My first goal was to avoid manual infrastructure changes as much as possible.

The AWS infrastructure is managed with Terraform, while the application deployment is handled through GitLab CI/CD.

The pipeline builds the Docker images, pushes them to ECR, and deploys the application to Kubernetes.

For AWS authentication, I used OIDC instead of storing long-lived AWS access keys in GitLab.

I actually wrote a separate article about this because I found the authentication flow particularly interesting.

2. Highly Available

Deploying something to Kubernetes doesn't automatically make it highly available.

I configured the EKS infrastructure across multiple Availability Zones and deployed multiple application replicas.

An Application Load Balancer distributes incoming traffic, while Kubernetes can replace unhealthy pods.

I also configured a Horizontal Pod Autoscaler (HPA) to scale the application between 2 and 6 pods depending on resource usage.

For the production database, RDS uses a Multi-AZ configuration.

The idea is simple: the application shouldn't depend on a single instance, pod, or Availability Zone.

3. Observable

A service can be running and still be unhealthy.

That's why monitoring was another important part of the project.

I used Prometheus to collect Kubernetes and application metrics and Grafana to visualize them.

This gave me visibility into things like resource usage, pod health, and application behavior.

I also configured alerts so that important problems don't require someone to constantly watch a dashboard.

This part of the project changed the way I think about infrastructure: deploying an application is only the beginning. You also need to understand what happens after deployment.

4. Recoverable

High availability doesn't replace backups.

I wanted to know what would happen if the database actually had to be restored.

So instead of simply enabling RDS snapshots and assuming recovery would work, I performed a real restoration test.

The measured recovery time was approximately 14 minutes.

For me, this was one of the most useful exercises in the project because it turned disaster recovery from a configuration checkbox into something measurable.

What I Learned

The biggest lesson wasn't about a specific AWS service.

It was understanding how all these pieces work together.

Terraform creates the infrastructure. GitLab automates delivery. Kubernetes manages the workloads. AWS provides the underlying services. Prometheus and Grafana tell me what's happening. Backups give me a recovery path when things go wrong.

And each layer introduces its own decisions and trade-offs.

This project also reminded me of something from my QA background:

A system isn't reliable because we expect it to work. It's reliable because we can verify what happens when it doesn't.

What's Next?

This article was an overview of the architecture, but several parts deserve their own deep dive.

Next, I'll share how I built the AWS EKS infrastructure with Terraform, including some of the problems I encountered and what I learned while fixing them.

If you're also learning AWS, Kubernetes, or DevOps by building projects, I'd love to hear about what you're working on.

How I Removed AWS Access Keys from GitLab CI/CD with OIDC

Amira Abidi — Wed, 12 Aug 2026 09:05:09 +0000

When I first connected my GitLab CI/CD pipelines to AWS, I used the simplest solution: an IAM user with an Access Key and Secret Access Key stored as GitLab CI/CD variables.

It worked.

But there was one problem: those credentials were permanent.

They had to be stored, protected and eventually rotated. If they were accidentally exposed in logs or compromised, they could remain valid until manually revoked.

I wanted a cleaner solution.

So I replaced permanent AWS credentials with OIDC federation between GitLab and AWS.

The result is simple: GitLab pipelines can access AWS without storing any permanent AWS credentials.

In this post, I'll explain how I implemented it, how the authentication flow works, and one important issue I faced when using it with EKS and Terraform.

The architecture

The authentication flow looks like this:

┌──────────────┐
│  GitLab CI   │
└──────┬───────┘
       │
       │ OIDC token
       ▼
┌──────────────┐
│   AWS STS    │
└──────┬───────┘
       │
       │ Temporary credentials
       ▼
┌──────────────┐
│   IAM Role   │
└──────┬───────┘
       │
       ├──────────► Terraform
       │
       ├──────────► ECR
       │
       └──────────► EKS

Instead of GitLab storing an AWS Access Key, it proves its identity to AWS using a short-lived OIDC token.

AWS verifies the token and returns temporary credentials.

How does OIDC authentication work?

The process can be summarized in five steps:

GitLab creates an OIDC token for the CI/CD job.
The pipeline sends this token to AWS.
AWS verifies that the token really comes from GitLab.
AWS checks that the project is allowed to assume the requested IAM role.
AWS STS returns temporary credentials.

These credentials expire automatically.

So there is nothing permanent to store or rotate inside GitLab.

Step 1 — Register GitLab as an OIDC provider

AWS first needs to trust GitLab as an identity provider.

I configured the OIDC provider using Terraform:

data "tls_certificate" "gitlab" {
  url = "${var.gitlab_url}/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "gitlab" {
  url             = var.gitlab_url
  client_id_list  = [var.gitlab_url]
  thumbprint_list = [
    data.tls_certificate.gitlab.certificates[0].sha1_fingerprint
  ]
}

This tells AWS that tokens issued by GitLab can be used for authentication.

But trusting GitLab itself is not enough.

AWS also needs to know which GitLab projects are allowed to access which IAM roles.

Step 2 — Restrict access with the IAM Trust Policy

This is one of the most important parts of the configuration.

We don't want any GitLab project to be able to assume our AWS roles.

The IAM Trust Policy checks information contained in the GitLab token.

For example:

condition {
  test     = "StringEquals"
  variable = "gitlab.com:aud"
  values   = [var.gitlab_url]
}

condition {
  test     = "StringLike"
  variable = "gitlab.com:sub"
  values   = [
    "project_path:${each.value}:ref_type:branch:ref:*"
  ]
}

condition {
  test     = "StringEquals"
  variable = "gitlab.com:namespace_id"
  values   = [var.gitlab_namespace_id]
}

The important values here are:

aud: checks the expected audience.
sub: identifies the GitLab project and branch.
namespace_id: identifies the GitLab namespace.

I especially like using namespace_id because it is a stable numeric identifier.

GitLab project and group names can change, but the namespace ID provides an additional way to make sure the token comes from the expected namespace.

Step 3 — One IAM role per repository

My project uses four GitLab repositories, so I created a separate IAM role for each one.

Repository	Main AWS permissions
Backend	Push Docker images to ECR
Frontend	Push Docker images to ECR
Infrastructure	Manage AWS infrastructure with Terraform
Deploy	Check ECR images and deploy to EKS

Why separate them?

Because the backend pipeline doesn't need permission to modify the EKS cluster.

And the deployment pipeline doesn't need permission to push Docker images.

Each pipeline gets only the permissions required for its job.

For example, the deployment role only needs to check whether an image exists:

actions = [
  "ecr:DescribeImages",
  "ecr:DescribeRepositories"
]

It cannot push a new image.

This gives me a much cleaner separation of responsibilities.

Step 4 — Request the OIDC token from GitLab

On the GitLab side, the configuration is surprisingly small.

GitLab can generate the OIDC token directly inside the CI/CD job:

id_tokens:
  GITLAB_OIDC_TOKEN:
    aud: https://gitlab.com

Then I define the AWS role and the location of the token:

variables:
  AWS_REGION: "us-east-1"
  AWS_ROLE_ARN: "arn:aws:iam::123456789012:role/gitlab-ci-book-infra"
  AWS_WEB_IDENTITY_TOKEN_FILE: "${CI_PROJECT_DIR}/.gitlab-oidc-token"

And write the token to the file:

before_script:
  - echo "$GITLAB_OIDC_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"

That's basically it.

Terraform and AWS SDKs understand the web identity authentication flow automatically.

There is no need to run:

aws configure

And more importantly, I don't need to store permanent values such as:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

in GitLab CI/CD variables anymore.

Step 5 — EKS needs an extra configuration

This was an important thing I learned while implementing the solution.

Successfully authenticating with AWS does not automatically mean that the pipeline can access Kubernetes.

For example:

aws eks update-kubeconfig

may work correctly while:

kubectl get pods

returns:

Unauthorized

Why?

Because there are two different authorization levels:

AWS IAM
   │
   │ Can this role access the AWS/EKS API?
   ▼
EKS Cluster
   │
   │ Is this IAM role allowed inside Kubernetes?
   ▼
Kubernetes resources

The IAM role therefore also needs access to the EKS cluster.

I configured this with an EKS access entry:

resource "aws_eks_access_entry" "ci_deploy" {
  cluster_name  = module.eks.cluster_name
  principal_arn = var.ci_deploy_role_arn
  type          = "STANDARD"
}

And associated the required EKS access policy with it.

Once this was configured, the deployment pipeline could authenticate correctly to the cluster.

The Terraform + EKS issue I ran into

This was probably the most interesting problem I encountered.

EKS authentication tokens are short-lived.

Initially, I used:

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}

This works when terraform plan and terraform apply happen immediately.

But my pipeline separates them.

The flow is:

terraform plan
      │
      ▼
Manual approval
      │
      ▼
terraform apply

The problem is that the EKS token can be generated during terraform plan.

If I wait before approving the apply, the saved token may already be expired.

Terraform then fails when trying to communicate with Kubernetes.

The solution: generate the token when it is needed

Instead of storing the token during the plan, I use an exec block:

provider "kubernetes" {
  host = module.eks.cluster_endpoint

  cluster_ca_certificate = base64decode(
    module.eks.cluster_certificate_authority_data
  )

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws-iam-authenticator"
    args        = [
      "token",
      "-i",
      module.eks.cluster_name
    ]
  }
}

Now the token is generated when Terraform actually needs to connect to Kubernetes.

So even if I wait before approving terraform apply, Terraform gets a fresh token.

Another small GitLab CI lesson

Manual GitLab jobs can allow failures by default depending on how they are configured.

For an infrastructure deployment, that's something I definitely don't want.

If terraform apply fails, the pipeline should fail too.

So I explicitly use:

apply:
  rules:
    - if: '$CI_COMMIT_BRANCH'
      when: manual
      allow_failure: false

It's a small configuration detail, but an important one for infrastructure pipelines.

Debugging OIDC authentication

When authentication fails, I usually check three things first:

aud
sub
namespace_id

Typical errors include:

Error	What I check
`AssumeRoleWithWebIdentity` denied	Project path / `sub`
`Incorrect token audience`	`aud` configuration
`InvalidIdentityToken`	Token expiration / OIDC provider
`Unauthorized` from `kubectl`	EKS access entry

I also keep this command in the pipeline:

aws sts get-caller-identity

It's very useful because it immediately shows which AWS identity the pipeline is actually using.

What I learned

The biggest lesson from this implementation is that OIDC is not as complicated as it first looks.

The basic idea is simply:

GitLab proves its identity
        ↓
AWS verifies the GitLab project
        ↓
AWS STS provides temporary credentials
        ↓
The pipeline accesses AWS

The main advantages for me are:

No permanent AWS credentials in GitLab.
No Access Keys to rotate.
Temporary credentials expire automatically.
Each repository has its own IAM role.
Permissions can be limited per pipeline.
AWS activity can be traced through CloudTrail.
GitLab can securely deploy to EKS.

There are a few extra details when EKS is involved, especially around cluster access and short-lived Kubernetes tokens, but once those pieces are understood, the overall architecture is quite simple.

For CI/CD pipelines, I now prefer this approach over storing permanent AWS credentials whenever OIDC federation is available.