DEV Community: Lanre Awe

How I Built a Production-Style GitOps Platform on AWS EKS — Solo, From Scratch

Lanre Awe — Sat, 27 Jun 2026 12:13:10 +0000

Most DevOps portfolio projects follow the same pattern: deploy a "hello world" app to Kubernetes, write a README, call it done.

This isn't that.

I took the Spring PetClinic microservices — a real Java application with 7 independent services, service discovery, an API gateway, and distributed tracing — and built the entire platform around it on AWS. Infrastructure as code, a proper GitOps delivery pipeline, autoscaling at two layers, end-to-end observability, and a reproducible lifecycle that provisions or destroys the whole environment with a single command.

The live app is running right now at petclinic.ralphnetwork.online.

This post is a walkthrough of what I built, how I made the decisions I made, and — most importantly — what broke and why. Because that last part is what actually teaches you something.

Why I built this

I'm an infrastructure engineer with 18 years of hands-on experience — servers, networking, firewalls, backup and DR — making the transition into DevOps and cloud engineering. I've been building cloud-native projects and documenting the journey publicly.

My goal with this project was specific: demonstrate that I can operate at the platform layer, not just the tool layer. Anyone can follow a tutorial and get kubectl apply to work. What I wanted to prove was that I could make engineering decisions, build a reliable delivery pipeline, handle real failures, and articulate the trade-offs — the way a working engineer actually operates.

So I treated it like a real system, not a demo.

The architecture

At a high level: a push to main triggers a GitHub Actions pipeline that builds and pushes Docker images to ECR, then commits a tag bump to the Helm chart in Git. Argo CD detects the change and syncs the cluster. The CI pipeline never runs kubectl directly — git is the authoritative source of truth.

flowchart TB
    Dev[Push to GitHub main] --> GHA[GitHub Actions CI]
    GHA -->|OIDC role - no static keys| ECR[Amazon ECR]
    GHA -->|bump image tag + commit| Git[Helm chart in Git]
    Git --> Argo[Argo CD]
    Argo -->|sync| Cluster
    ECR --> Cluster

    subgraph Cluster["EKS cluster (petclinic-prod) — eu-central-1"]
      direction TB
      ALB[ALB Ingress - ACM TLS] --> GW[api-gateway]
      GW --> APP[customers / vets / visits]
      APP --- Platform[discovery + config server]
      HPA[HPA] -. scales pods .-> APP
      Karpenter[Karpenter] -. scales nodes .-> Nodes[EC2 nodes]
      APP -. traces .-> Zipkin
      APP -. metrics .-> Prometheus --> Grafana
    end

Cluster: petclinic-prod · Region: eu-central-1 · Kubernetes: 1.33

The full stack

Layer	Tooling
Cloud	AWS (EKS, ECR, VPC, IAM, ALB, ACM, SQS)
IaC	Terraform — remote state on S3 + DynamoDB, reusable modules
Containers	Docker, Amazon ECR (one repo per service, scan-on-push)
Orchestration	Kubernetes (EKS, managed node group + Karpenter)
Packaging	Helm (one values-driven chart for all 7 services)
GitOps	Argo CD
CI/CD	GitHub Actions (OIDC auth — no static AWS keys)
Autoscaling	HPA (pods) + Karpenter (nodes) + metrics-server
Observability	Prometheus, Grafana, Zipkin (distributed tracing)
App	Spring Boot microservices, Spring Cloud Config + Eureka

Layer by layer: what I built and why

Infrastructure as Code (Terraform)

Every AWS resource is defined in Terraform, split into reusable modules and wired together in a single prod environment. The first thing I provisioned — before anything else — was the remote state backend: an S3 bucket (versioned, encrypted, public access blocked) and a DynamoDB lock table. If you lose your state file, you lose control of your infrastructure. That comes first, always.

The modules:

vpc — 2 availability zones, public and private subnets, with the specific subnet tags the AWS Load Balancer Controller and Karpenter need to discover them.
eks — built on the official terraform-aws-modules/eks module, EKS 1.33, managed node group, IRSA and EKS Pod Identity enabled, control-plane logging on.
ecr — one repository per service with image scanning on push.
iam — IRSA role for the Load Balancer Controller.
github-actions — OIDC trust policy and an IAM role so GitHub Actions can assume it without a long-lived access key.
Karpenter — IAM role, SQS interruption queue, and node role, via the EKS module's built-in Karpenter submodule using Pod Identity.

Terraform provisions the AWS platform. Everything above that — cluster add-ons, Argo CD, the app — is installed by scripts/addons.sh in the correct dependency order.

GitOps delivery

This is the piece I'm proudest of, because it's the difference between "I can run kubectl" and "I built a delivery pipeline."

The workflow:

Push to main triggers GitHub Actions.
GitHub Actions authenticates to AWS via an OIDC role — no AWS_ACCESS_KEY_ID in secrets, not anywhere.
All 7 services are built as Docker images and pushed to ECR, tagged with the git SHA.
The pipeline then bumps the image tag in helm/petclinic/values.yaml and commits it back to the repo.
Argo CD detects the change and syncs the Helm chart to the cluster.

The cluster never pulls credentials from CI. CI never holds cluster access. The audit trail for every deployment lives in git history. That's real GitOps, and it's meaningfully different from "run kubectl apply at the end of a pipeline."

Packaging with Helm

The app started with hand-written Kubernetes manifests with hardcoded image tags — one manifest per service, with the image version baked in. I converted everything into one values-driven Helm chart that renders all 7 services from a single config block.

That collapsed seven hardcoded image tags into one value that CI controls. It also eliminated hundreds of lines of duplicated YAML, made per-service configuration changes a one-line edit, and gave me a single versioned artifact I can promote, roll back, or diff. It also made Argo CD's diff view meaningful — you can actually see what changed per deployment.

Autoscaling at two layers

HPA (Horizontal Pod Autoscaler) is configured on the four stateless services — api-gateway, customers-service, vets-service, visits-service — with a minimum of 2 replicas, maximum of 4, scaling on CPU at 70%, fed by metrics-server.

Karpenter handles node autoscaling. When the HPA needs to schedule more pods than the current nodes can fit, Karpenter provisions a right-sized EC2 instance and decommissions it when idle. I didn't just configure this — I tested it under real load. Pending pods from HPA scaling triggered a t3a.medium provisioning event, and Karpenter had a node ready in approximately 90 seconds.

The choice to use Karpenter over the older cluster-autoscaler was deliberate. It bin-packs more efficiently, picks instance types dynamically, and it's the modern EKS approach. More setup, but a better result.

Observability

Prometheus scrapes every service via the /actuator/prometheus endpoint. Grafana visualises the metrics. Zipkin collects distributed traces, so you can follow a single user request as it travels from the api-gateway through customers-service and back.

Getting all three working together — and getting traces working end to end specifically — was one of the most instructive parts of the build. More on that in the debugging section.

Networking and TLS

An ALB Ingress (provisioned by the AWS Load Balancer Controller from a Kubernetes Ingress object) fronts the gateway. TLS is terminated at the ALB using an ACM certificate, with a real DNS record at petclinic.ralphnetwork.online. The cluster itself runs in private subnets. The only public entry point is the load balancer.

Reproducible from one command

A platform you can't rebuild from scratch isn't really infrastructure as code — it's a managed pet. So I encoded the full lifecycle in a Makefile:

# Provision the state backend once per AWS account
make state

# Provision the full platform + install add-ons + deploy the app
make up

# Tear everything down cleanly
make down

make up runs two phases in order: terraform apply to provision the AWS layer, then scripts/addons.sh to install add-ons in dependency order: AWS Load Balancer Controller → metrics-server → Karpenter → Argo CD → the PetClinic application.

make down is the part that trips people up. Terraform provisions the base infrastructure, but the in-cluster controllers create resources at runtime that Terraform doesn't know about — specifically the ALB and any Karpenter-provisioned EC2 nodes. A naive terraform destroy hangs waiting for a VPC it can't delete because the ALB is still attached. The teardown script deletes the Kubernetes layer first, waits for the ALB and extra nodes to actually drain, and then runs terraform destroy.

This means I can provision the full environment for a live demo and destroy it to near-zero cost afterward without leaving orphaned load balancers or a surprise AWS bill.

The bugs — the part that actually matters

I want to be honest about this: most of the learning in this project came from what broke. Here are the ones that taught me the most.

Zipkin showed no traces, even though all services were up

Tracing export is non-fatal — a failure to connect to Zipkin doesn't crash the application, it just silently drops spans. So the services appeared healthy while producing zero traces.

The root causes were two independent misconfigurations that had to be fixed together:

The tracing endpoint in the Spring Cloud Config pointed at tracing-server, which didn't match the Kubernetes Service name zipkin.
The endpoint was only set under one Spring profile, so most services never exported at all.

The fix: corrected the hostname, and had the Helm chart inject the Zipkin endpoint via an environment variable into every service — so tracing is now uniform and controlled at the platform level, not buried in per-service config files.

CI was building images that never actually deployed

The deploy step rewrote :latest tags, but the manifests had specific version pins (:4.0.1, :4.0.2). The substitution matched nothing — every "deployment" silently re-applied the old images. The cluster looked updated; it wasn't.

Migrating to Helm fixed this properly. Image tags became a single chart value that CI bumps to the git SHA, and Argo CD shows a visible diff when the value changes. There's no ambiguity about what's running.

Argo CD and the HPA fought over replica counts

With Argo's self-healing enabled, it kept resetting replicas to the chart value. The HPA simultaneously tried to scale based on CPU. They were in a tug of war that neither could win cleanly.

The fix is a standard but non-obvious GitOps pattern: omit replicas from the Deployment spec when an HPA controls the workload, and configure Argo to explicitly ignore differences on the replicas field. That way Argo reconciles everything except replica counts, and the HPA owns that field exclusively.

Karpenter's IAM policy exceeded AWS's size limit

The error was: LimitExceeded: Cannot exceed quota for PolicySize: 6144.

Karpenter's required IAM policy is large. The fix was to switch from a managed policy to an inline policy (10,240-character limit) using a flag built into the EKS Terraform module. One line change; the reason isn't obvious unless you've hit it.

A capacity planning decision that wasn't a bug

Enabling HPA would have scheduled more pods than the 2-node cluster could hold — they'd have sat in Pending indefinitely, which looks like a broken cluster. I had three options: add a third static node, use cluster-autoscaler, or add Karpenter.

I chose Karpenter: it scales on demand rather than requiring a fixed node count, bins-packs more efficiently, and it's the approach AWS recommends for EKS. The decision had a cost in setup time and complexity. The benefit is a cluster that genuinely scales rather than one that holds a fixed headroom.

Decisions and trade-offs

The interesting engineering questions in this project weren't "which tool" — they were "why this, versus that, given these constraints."

GitOps over direct kubectl apply in CI. More moving parts upfront. But: CI doesn't hold cluster credentials, every deployment is auditable in git, and Argo's self-healing means drift from the desired state gets corrected automatically. For any real team, this is non-negotiable.

Karpenter over cluster-autoscaler. Faster to respond, picks the right instance type for the pending workload, consolidates underutilised nodes. The trade-off is more setup. Worth it for the operational behaviour and the learning.

Kept Eureka + Spring Cloud Config, deliberately. On Kubernetes, native Service DNS and ConfigMaps overlap with what these frameworks provide — they're somewhat redundant. I kept them because rewriting all 7 services to drop them was out of scope, and doing it poorly would be worse than the overlap. Going fully Kubernetes-native is explicitly on the backlog as a next step, not an oversight I missed.

Single NAT gateway, single environment. A deliberate cost decision. Multi-AZ NAT gateways add ~$30/month per gateway, which adds up quickly for a demo project. I know exactly where the HA gap is and named it, rather than pretending it's production-grade multi-region when it isn't.

What this project demonstrates

Skill area	Evidence
Infrastructure as Code	Modular Terraform, remote state, full AWS platform from scratch
CI/CD	GitHub Actions, OIDC auth, build → push → tag bump → deploy automated
GitOps	Argo CD syncing a Helm chart; git as sole source of truth
Kubernetes	EKS, Helm, HPA, PDBs, ALB Ingress, Pod Identity / IRSA
Cloud (AWS)	EKS, ECR, IAM, VPC, ALB, ACM, SQS — provisioned end to end
Autoscaling	HPA + Karpenter, verified under real load, capacity reasoning documented
Observability	Prometheus, Grafana, distributed tracing with Zipkin
Security	OIDC over static keys, least-privilege IAM, TLS on all public traffic
Debugging	Real failures diagnosed and fixed; root causes explained
Engineering judgment	Trade-offs documented and defended, not assumed away

What I'd do next

I kept an honest backlog rather than claiming the project is "done":

NetworkPolicies — default-deny with explicit allow rules for each service-to-service path.
Secrets management — External Secrets Operator backed by AWS SSM Parameter Store or Secrets Manager, to replace env-var secrets.
kube-prometheus-stack — replace the hand-assembled Prometheus + Grafana setup with the community Helm chart.
Go Kubernetes-native — remove Eureka and Spring Cloud Config in favour of Kubernetes Service DNS and native ConfigMaps.

Closing

This project started as "deploy an app to Kubernetes" and became a study in what it actually means to build a platform. The delivery pipeline, the autoscaling, the tracing, the teardown ordering, the GitOps patterns — none of that comes from a tutorial. It comes from making deliberate choices, hitting real problems, and working through them.

That's the work I want to do professionally, and this project is my evidence that I can.

Repos:

petclinic-infrastructure — Terraform, Makefile, addon scripts
spring-petclinic-microservices — app code, Helm chart, GitHub Actions
spring-petclinic-microservices-config — Spring Cloud Config

Live app: petclinic.ralphnetwork.online

If you're hiring for DevOps or Platform Engineering roles — remote or Lagos on-site — I'd genuinely love to talk. Find me on LinkedIn.

From Learning DevOps to Deploying a Production Application: My DMI Cohort 2 Experience

Lanre Awe — Mon, 15 Jun 2026 07:07:39 +0000

Introduction

One of the biggest milestones in my DevOps journey was participating in the deployment of a production-ready microservices application as part of the DevOps Mentorship Initiative (DMI) Cohort 2.

Together with a team of talented engineers, we successfully deployed the Spring Petclinic Microservices application to Azure Kubernetes Service (AKS). This was not a simple demo project—it involved real-world tools, cloud infrastructure, CI/CD pipelines, observability, AI integration, and production deployment practices.

The experience challenged me technically and personally, but it also showed me what working on a real DevOps project looks like.

About the Project

Spring Petclinic is a cloud-native microservices application built with Spring Boot. The application consists of multiple services that communicate with each other and are deployed as containers.

Our technology stack included:

Spring Boot
Spring AI
Docker
Terraform
Azure Kubernetes Service (AKS)
Helm
Azure Pipelines
Azure OpenAI
Grafana

As part of the project, I worked as the GenAI Engineer, responsible for integrating the AI chatbot using Spring AI and Azure OpenAI.

My Role: Integrating AI into the Platform

My responsibility was to make the chatbot feature work seamlessly within the microservices environment.

This involved:

Connecting the application to Azure OpenAI
Integrating Spring AI into the backend services
Ensuring the chatbot worked correctly within a reactive Spring WebFlux environment
Troubleshooting deployment and runtime issues

Although it was a single project ticket, it required understanding multiple layers of the architecture—from application code to cloud infrastructure.

Challenges We Faced

No real-world deployment is without problems, and this project was no exception.

1. Docker Hub Access Issues

At one point, Docker Hub access became unreliable due to ISP restrictions.

To solve this, we imported images directly into Azure Container Registry (ACR) using:

az acr import

This removed our dependency on Docker Hub and improved reliability.

2. Azure OpenAI Quota Problems

Having Azure credits did not automatically mean we had access to AI model quotas.

We had to upgrade the subscription and request the required quota before deploying the chatbot successfully.

3. Spring WebFlux Blocking Calls

The AI service initially caused threading issues because the AI call was blocking.

The solution was to wrap the operation using:

Mono.fromCallable(...)
    .subscribeOn(Schedulers.boundedElastic())

Once implemented, the chatbot responded correctly without blocking the reactive application.

4. Secrets Accidentally Committed to Git

One of the most important lessons came from discovering that an API key had been committed to the repository.

The team immediately:

Rotated the compromised key
Removed it from Git history
Improved secret management practices

It was a valuable reminder that security starts from day one.

5. Empty Vector Store

The chatbot initially returned poor results because the vector store had been built with incompatible embeddings.

We regenerated the embeddings using the correct Azure OpenAI model and rebuilt the vector store using production data.

What I Learned

This project taught me lessons that go beyond certifications and tutorials.

Cloud Solves Real Problems

Some issues that are difficult to handle locally become much easier when using managed cloud services.

Reactive Programming Requires Discipline

Spring WebFlux works extremely well, but blocking operations can quickly cause problems if not handled correctly.

Secrets Management Is Critical

Security cannot be treated as an afterthought. Proper handling of credentials and sensitive information must be part of the development process from the beginning.

Deployment Order Matters

In a microservices environment, services depend on one another. Starting services in the wrong order can prevent the entire application from functioning correctly.

The Team Behind the Project

One of the most rewarding parts of this experience was working alongside an amazing team.

Special appreciation to:

Michael Ikedimma
Benjamin Akinteye
Gift Ukporo
Duru Juliet Chinenye
Pradeep Neelaboyina
Angela Chibuike
Oladayo Aremu
Ubani OnU Chukwu
Osman Farah Ali Farah
Kolawole Yinusa

Everyone brought unique skills and expertise that helped make the project successful.

Final Thoughts

Looking back, this project was one of the most impactful experiences of my career so far.

Beyond deploying applications and solving technical problems, I learned how real engineering teams collaborate, troubleshoot, communicate, and deliver production systems.

The journey was challenging, but every obstacle helped me grow as a DevOps engineer.

If you're looking for a practical way to learn DevOps by working on real-world projects, I highly recommend the DevOps Mentorship Initiative (DMI).

DMI Cohort 3 starts on 27 June.

Apply here:

https://docs.google.com/forms/d/e/1FAIpQLSel7ai7nyb0P1qLW4vEyfB_nEsD4lUF1XG88vmAaFGBOb6hPA/viewform

Connect With Me

GitHub: https://github.com/Ralphlarry

LinkedIn: www.linkedin.com/in/olanrewaju-awe-62761758

DMI #DevOps #CloudComputing #Azure #AKS #Terraform #Docker #Microservices #SpringBoot #AI #TheCloudAdvisory

Deploying Spring Petclinic Microservices Locally with Docker Compose

Lanre Awe — Sun, 14 Jun 2026 21:44:55 +0000

Introduction

As part of the DevOps Mentorship Initiative (DMI), I deployed the Spring Petclinic Microservices application locally using Docker Compose and explored how a modern microservices architecture operates in practice.

Spring Petclinic is a cloud-native sample application built with Spring Boot and Spring Cloud. Instead of running as a single application, it consists of multiple independent services that communicate with one another.

The deployment included:

Config Server
Discovery Server (Eureka)
API Gateway
Customers Service
Vets Service
Visits Service
GenAI Service
Admin Server

In addition, the observability stack included:

Prometheus
Grafana
Zipkin

This project provided hands-on experience with service discovery, centralized configuration, container orchestration, and observability.

Prerequisites

Before starting, I installed the following tools:
Docker engine for WSL.
Docker engine in WSL was used to run and manage all application containers.

Verify installation:
docker --version
docker compose version
Git

Git was used to clone the project repository.

Verify installation:
git --version

Step 1: Clone the Repository
Clone the Spring Petclinic Microservices repository:

https://github.com/Ralphlarry/spring-petclinic-microservices.git

Move into the project directory:
cd spring-petclinic-microservices

Step 2: Start the Entire Application
The most interesting part of the project was that the entire platform could be started with a single command:

docker compose up -d

Docker Compose automatically:
Pulled required images
Created containers
Connected services through a shared network
Applied startup dependencies
Started the monitoring stack

To confirm everything was running:
docker ps

Expected containers:
config-server
discovery-server
api-gateway
customers-service
vets-service
visits-service
genai-service
admin-server
prometheus-server
grafana-server
tracing-server

Step 3: Verify the Deployment
API Gateway

Check the gateway health endpoint:

curl http://localhost:8080/actuator/health
Expected response:
{"status":"UP"}

Eureka Dashboard
Open:
http://localhost:8761

All services should appear as registered instances.

Spring Boot Admin
Open:
http://localhost:9090

This dashboard provides visibility into application health and metrics.
Understanding the Startup Order

One important concept in this deployment is service startup dependency.
The Docker Compose file ensures that the Config Server and Discovery Server start before the other services.

Why Config Server Starts First
The Config Server stores centralized configuration for all services.
When services such as Customers Service or API Gateway start, they immediately request configuration from the Config Server.

Without it:
Services cannot retrieve configuration
Startup may fail
Environment settings become unavailable
Why Discovery Server Starts Second

The Discovery Server (Eureka) acts as a service registry.

Every microservice registers itself with Eureka when it starts.

Without Eureka:
Services cannot discover each other
API Gateway routing fails
Inter-service communication breaks

In short:

Config Server
↓
Discovery Server
↓
All Other Services

This startup sequence is critical for a healthy deployment.

Observability and Monitoring
One of the most valuable parts of this project was learning how observability tools provide visibility into distributed systems.

Prometheus
Prometheus continuously collects metrics from the Spring Boot Actuator endpoints.

Metrics include:
CPU usage
Memory usage
HTTP request counts
Application performance statistics

Access:
http://localhost:9091

Prometheus acts as the data collection layer for monitoring.

Grafana
Grafana visualizes metrics collected by Prometheus.

Access:
http://localhost:3000

Using Grafana dashboards, I could monitor:
Service health
JVM memory consumption
Request throughput
System performance trends

Instead of reading raw metrics, Grafana transforms them into easy-to-understand charts and dashboards.

Zipkin
Zipkin provides distributed tracing.

Access:
http://localhost:9411

Distributed tracing allows engineers to follow a request as it travels across multiple services.

For example:

Client
↓
API Gateway
↓
Customers Service
↓
Database

Zipkin records timing information for every step, helping identify bottlenecks and performance issues.

Although tracing required additional verification during testing, understanding how distributed tracing works was one of the most educational parts of the project.

Docker Compose Up vs Down
Start Everything
docker compose up -d

This command:
Creates containers
Creates networks
Starts services
Runs containers in the background
Stop Everything
docker compose down

This command:
Stops containers
Removes containers
Removes networks created by Compose

Using these two commands makes managing the entire environment simple and repeatable.

What I Learned
The biggest lesson from this deployment was that running microservices is much more than simply starting containers.

A successful deployment depends on:
Correct startup sequencing
Service discovery
Centralized configuration
Monitoring
Distributed tracing
Health checks

I also learned how observability tools such as Prometheus, Grafana, and Zipkin help engineers understand what is happening inside a distributed system.

These tools become increasingly important as systems grow larger and more complex.

Looking Ahead to Production

If deploying this architecture to AWS, I would replace local Docker Compose components with managed cloud services:

This would provide better scalability, availability, security, and operational reliability.

Conclusion
Deploying Spring Petclinic Microservices gave me practical experience with modern cloud-native architecture and DevOps practices.

From centralized configuration and service discovery to monitoring and tracing, this project demonstrated many of the concepts used in real-world production environments.

This project was completed as part of the DevOps Mentorship Initiative (DMI).

Interested in joining the next cohort?

DMI Cohort 3 Registration:
https://docs.google.com/forms/d/e/1FAIpQLSel7ai7nyb0P1qLW4vEyfB_nEsD4lUF1XG88vmAaFGBOb6hPA/viewform

Author: Olanrewaju Awe
GitHub: https://github.com/Ralphlarry
LinkedIn: www.linkedin.com/in/olanrewaju-awe-62761758