From Terraform Spaghetti to a Production-Grade GitOps Platform

Kalpit Swami — Sat, 06 Jun 2026 06:31:19 +0000

Section 1: Why I Built This (And Why I Then Tore It Apart)

Every good project has an origin story. Mine starts with a decision to take a real-world MERN e-commerce application and deploy it the way a proper engineering team would in production.

The application itself is ProShop v2, an open source e-commerce project built by Brad Traversy. I did not write the application code. That is an important distinction and I want to be upfront about it from the start. What I built was everything around it: the infrastructure, the CI/CD pipelines, the Dockerfiles, the GitOps setup, the secrets management, the security architecture, and the observability stack. The application was the workload. The infrastructure was the project.

Using a real application instead of a hello world example was a deliberate choice. A production e-commerce platform has a frontend, a backend API, an external database, payment integrations, authentication, and real secrets to manage. It forces you to solve real infrastructure problems rather than toy ones. If you can deploy this correctly, you can deploy almost anything.

The first version of the infrastructure worked. You could open it in a browser, browse products, add things to a cart, and check out with PayPal. From a user's perspective, everything was fine. But underneath the hood, the infrastructure was held together with the cloud equivalent of duct tape and optimism.

Here is what the setup actually looked like. Terraform was responsible for everything. And I mean everything. It provisioned the VPC, the EKS cluster, the IAM roles, AND it deployed the Kubernetes application pods, AND it injected secrets directly into the cluster by reading from AWS Secrets Manager and writing them as Kubernetes Secret resources. One tool, three completely different jobs, zero separation between them.

If you have been in the field for a while, you are already wincing. If you are newer to this, let me explain why this is a problem.

Terraform is an infrastructure provisioning tool. It is designed to create and manage cloud resources that change rarely, things like VPCs, clusters, and IAM roles. Your application pods, on the other hand, change multiple times a day every time you push new code. When both of these live inside the same Terraform state, every single application deployment carries the risk of accidentally modifying infrastructure. A failed pod update can corrupt your cluster state. A destroyed environment can get stuck
because Terraform has no idea that Kubernetes already cleaned something up on its own.

And the secrets situation was genuinely bad. When Terraform reads a value from AWS Secrets Manager and writes it into a Kubernetes Secret resource, that secret value gets stored inside the Terraform state file. The state file lives in an S3 bucket. So your MongoDB URI, your JWT secret, and your PayPal credentials are sitting in plaintext inside an S3 bucket. Anyone with read access to that bucket, including CI/CD pipelines and developers with broad IAM permissions, can read every single one of them. That is
not secrets management. That is secrets storage with extra steps.

On top of all this, there was a bug that took me a while to find. The IAM trust policy for the backend pods was referencing the wrong Kubernetes namespace. The service account was in the proshop namespace but the trust condition pointed at the default namespace. AWS validates this exact string when a pod tries to assume an IAM role. The mismatch meant the pods started up successfully, looked completely healthy, and then silently failed every single time they tried to read from Secrets Manager. No crash, no error on startup, just a quiet failure at runtime. Those are the worst kind of bugs.

So there I was, with a project that technically worked but was architecturally wrong in ways that would cause real problems the moment someone tried to maintain it, scale it, or destroy and redeploy it. And I had already run terraform destroy once and watched it hang for twenty minutes because the Application Load Balancer was stuck. Terraform
had no idea the ALB existed because Kubernetes had created it, not Terraform. AWS refused to delete the VPC because the ALB was still attached to the subnets. Classic.

That was the moment I decided to rebuild it properly.

The question I started with was simple: if I were a DevOps engineer at a real company and someone handed me this codebase, what would I actually change and why?

The answer to that question is what the rest of this blog is about.

Section 2: The One Insight That Changed Everything

Before writing a single line of new code, I spent time thinking about what was actually wrong. Not the symptoms, the bugs, the stuck destroys. The root cause underneath all of it.

Here is what I landed on. The original setup was failing because three completely different things, each with their own lifecycle, their own team, and their own rate of change, were all forced to live inside the same Terraform state. These three things are infrastructure, application deployment, and secret management.

Infrastructure changes rarely. You create a VPC once. You create an EKS cluster once. IAM roles, ECR repositories, VPC endpoints, these things might change once a month if that. Terraform is excellent at managing these because that is exactly what it was designed for.

Application deployments change constantly. Every time a developer pushes code, the application needs to be rebuilt, scanned, pushed to a container registry, and deployed to the cluster. This can happen multiple times a day. Terraform is terrible at this because it was never designed for it. It has no concept of rollback, no health checks for running pods, no progressive delivery, and no way to know if the new version of your application is actually working before it declares success.

Secret management is its own concern entirely. Secrets need to be rotated, audited, and accessed at runtime by the application. They should never be baked into infrastructure state, committed to Git, or stored in a deployment pipeline. They should exist only where they need to exist: inside the running application, at the moment they are needed.

Once I saw it this way, the solution became obvious. Each concern needed its own dedicated tool.

Terraform would own infrastructure only. The cluster, the networking, the IAM roles. Things that change once a month.

ArgoCD would own application deployment. It would watch a Git repository and make sure the cluster always matches what is declared there. Every deployment is a Git commit. Rollback is a git revert. Drift from the desired state gets corrected automatically.

External Secrets Operator would own secret management. It would reach into AWS Secrets Manager at runtime, pull the values the application needs, and create Kubernetes Secret objects inside the cluster. Secret values would never touch Terraform state, never appear in Git, and never live anywhere except inside the running cluster where they are needed.

This is the architecture that the rest of the project is built around.

That screenshot is the before. Those four files, app-backend.tf, app-frontend.tf, ingress.tf, and helm-alb.tf, all lived inside the Terraform EKS module. All of them were deleted in Phase 1. The EKS module became infrastructure only: cluster, node groups, IAM roles, OIDC provider, addons. Nothing else.

Here is what the separation looks like in practice after the rebuild.

Terraform runs once when you spin up the environment. It creates all the AWS resources and stops. It does not touch a single Kubernetes workload.

ArgoCD gets installed right after Terraform finishes, via a Helm install in the CI/CD pipeline. From that point on, ArgoCD takes over. It reads from the Git repository and deploys everything: the AWS Load Balancer Controller, the External Secrets Operator, Prometheus, Grafana, the backend, the frontend, the Ingress. Everything. If you manually change something in the cluster, ArgoCD detects the drift and corrects it. The Git repository is always the source of truth.

External Secrets Operator runs inside the cluster and continuously syncs secrets from AWS Secrets Manager. The application pods read environment variables from Kubernetes Secrets that ESO creates. Those secrets never appear in any file in this repository.

The result is three tools doing three jobs, each doing its job well, none of them stepping on each other. When the application needs a new deployment, you update a manifest in Git and ArgoCD handles it. Terraform never needs to run again. When a secret rotates in Secrets Manager, ESO picks it up and updates the Kubernetes Secret automatically. ArgoCD does not need to know it happened. When the infrastructure needs a change, Terraform runs. ArgoCD keeps doing its job the whole time.

That is separation of concerns applied at the infrastructure level. It sounds simple when you describe it like this. Getting there was not.

Section 3: The Tech Decisions and the Tradeoffs (The Honest Version)

Every architecture blog post eventually has a section where the author lists the tools they used and explains why each one is obviously the best choice. This is not that section. These decisions had real tradeoffs, some of them were genuinely difficult, and a few of them I got wrong the first time. Here is the honest version.

ArgoCD over Flux

When I decided to adopt GitOps, the two main options were ArgoCD and Flux. Flux is a perfectly good tool. But in February 2024, Weaveworks, the company that created Flux and literally coined the term GitOps, shut down. The community stepped in and Flux is still maintained as a CNCF project, but the commercial backing is gone and the momentum shifted.

More practically, in November 2025 AWS announced native managed ArgoCD support for EKS as a GA capability. When the cloud provider you are deploying on explicitly backs one tool over another, that is a signal worth paying attention to.

ArgoCD also has a visual web UI that shows you exactly what is deployed, what is synced, and what is drifting. When you are learning GitOps for the first time, being able to see the state of your cluster visually rather than running kubectl commands to figure out what ArgoCD is thinking is genuinely helpful. I am not too proud to admit that.

External Secrets Operator over Secrets Store CSI Driver

Both tools solve the same problem: getting secrets from AWS Secrets Manager into Kubernetes pods without storing them in Git or Terraform state. The difference is how they expose those secrets to your application.

The CSI Driver mounts secrets as files inside the pod filesystem. Your application reads them from file paths. This is actually more secure because secret values never touch Kubernetes etcd storage. But here is the thing: the ProShop backend reads configuration via process.env, standard Node.js environment variables. To use the CSI Driver properly, you would need to either modify the application code to read from file paths, or enable the CSI Driver's "sync as Kubernetes Secret" feature, which writes the values back to etcd anyway and defeats the main security advantage.

External Secrets Operator creates native Kubernetes Secrets from Secrets Manager. The application reads them as environment variables exactly as it already does. No code changes, no file path gymnastics, same security improvement where it actually matters: secrets out of Terraform state and out of Git. ESO was the right call for this specific situation.

S3 Native Locking over DynamoDB

This one is straightforward. In Terraform 1.11, released in February 2025, S3 native state locking became generally available. The DynamoDB locking approach was deprecated in the same release. Before this, you needed a separate DynamoDB table just to write a lock flag when Terraform was running. It was a two-resource solution to a one-resource problem.

With S3 native locking, you add use_lockfile = true to your backend config and you are done. The S3 bucket handles the locking itself using conditional writes at the API level. No DynamoDB table, no extra IAM permissions, and when a lock gets stuck, you delete a single file from S3 rather than hunting through the DynamoDB console. The original codebase was on Terraform 1.12.2 and still using DynamoDB locking. That was just an oversight waiting to be fixed.

Trivy for Container Scanning (And Why the SHA Pinning Matters)

Trivy is the industry standard for container vulnerability scanning. I almost switched away from it. In March 2026, Trivy's GitHub Actions infrastructure suffered two supply chain attacks in under a month. Attackers force-pushed malicious commits to 75 of 76 version tags in the official trivy-action repository. The malicious code was a credential stealer that ran silently before the real scanner, so pipelines appeared to work normally while AWS credentials, SSH keys, and Kubernetes tokens were being sent to attacker-controlled servers. Over 10,000 repositories were affected.

The root cause was not Trivy itself. The root cause was that pipelines were referencing Trivy using mutable version tags like @v0.30.0. Git tags can be force-pushed to point at any commit. They are not immutable references.

The fix is to pin every GitHub Action to a full 40-character commit SHA. A commit SHA cannot be rewritten. aquasecurity/trivy-action@a2f798cf does not mean the same thing tomorrow as it does today if someone pushes a new commit. aquasecurity/trivy-action@a2f798cf3f5b01eff4b7cf6f7a29f1d0e3c9b841 means exactly that commit, forever, no matter what happens to the tags.

I applied this to every single GitHub Action in both pipelines. Not just Trivy. Every action. Because the same attack vector exists for any mutable tag on any action from any vendor.

Why We Skipped SonarQube

SonarQube is a static analysis tool that scans your source code for bugs, code smells, and security patterns. It is a good tool. I did not use it, and the reason is worth explaining because it is actually a principle rather than a preference.

The application code in this project is ProShop v2, written by Brad Traversy. I did not write it and I cannot meaningfully explain its findings in a code review or an interview. Running SonarQube on third-party code you did not write and cannot speak to is security theater. It generates findings you cannot act on and gives a false sense of coverage.

The scanning we do covers what we actually own: Trivy scans the container images we build from our Dockerfiles, and Checkov scans the Terraform modules and Kubernetes manifests I wrote. Every finding from these tools is something wI can explain, act on, and fix. That is the bar for including a security tool in a pipeline.

VPC Endpoints Plus NAT Gateway

Private EKS nodes need to talk to AWS services: pulling images from ECR, reading secrets from Secrets Manager, exchanging OIDC tokens with STS, sending logs to CloudWatch. Without any special configuration, all of this traffic goes out through the NAT Gateway and back in through the public AWS endpoints.

VPC Interface Endpoints create private connections directly between your VPC and these AWS services. Traffic never leaves the AWS network. I added interface endpoints for ECR API, ECR DKR, Secrets Manager, STS, and CloudWatch Logs. I added an S3 Gateway Endpoint, which is free, to handle ECR image layer pulls since ECR stores image layers in S3.

The NAT Gateway stays in the architecture for one specific reason: MongoDB Atlas is an external service outside AWS. Backend pods need outbound internet access to reach the Atlas connection string. There is no VPC endpoint for Atlas, so NAT Gateway handles that traffic. Everything else goes through the private endpoints.

This is not the cheapest architecture for a low-traffic development environment. The five interface endpoints across two availability zones cost roughly $73 a month, which is the same as the EKS control plane fee. But the security tradeoff is real: AWS service traffic that never touches the public internet is a meaningful reduction in attack surface, and for a project demonstrating production practices, that matters more than saving $40 a month.

Section 4: How the System Actually Works

Most infrastructure blog posts end at "and then it deployed successfully." That is where the interesting part actually starts. Let me walk you through what is happening at every layer when this system runs, because understanding the full flow is what separates someone who set up a tool from someone who understands the system.

When a user loads the ProShop storefront

The user types the ALB DNS URL into their browser and hits Enter. That DNS name resolves to the Application Load Balancer sitting in the public subnets of the VPC. The ALB is internet-facing, meaning it has a public IP and accepts traffic from anywhere. It is the only thing in this entire architecture that does.

The request arrives at the ALB. The ALB evaluates its routing rules. The path is just /, which matches the frontend rule. The ALB forwards the request to the frontend Kubernetes Service on port 80. The Service load balances across the available frontend pods. A frontend pod receives the request, and Nginx serves the pre-built React application as static files back to the browser.

The browser renders the page. Then immediately, without the user doing anything, the React application fires an API call to /api/products to load the product listings.

That request goes back to the ALB. This time the path starts with /api/, which matches the backend routing rule. The ALB forwards it to the backend Kubernetes Service on port 5000. A backend pod receives it. Express routes it to the products controller. The controller calls Mongoose, which opens a connection to MongoDB Atlas using the MONGO_URI environment variable.

That connection travels outbound from the pod, through the node, out through the NAT Gateway, onto the public internet, and into MongoDB Atlas. Atlas returns the product data. The response travels back through the same path. The backend sends a JSON response. The React frontend renders the product grid.

The whole thing takes under a second if everything is healthy. That one round trip involves the ALB, two Kubernetes Services, a running pod, a NAT Gateway, and an external managed database. If any one of those is broken, the products do not load. Which is exactly why observability matters, but more on that later.

Where the secrets come from

The backend pod reads MONGO_URI, JWT_SECRET, PAYPAL_CLIENT_ID, and five other values as environment variables. These values are injected from a Kubernetes Secret named backend-secrets in the proshop namespace. But where does that Secret come from?

It does not come from Terraform. It does not come from the CI/CD pipeline. It does not live in Git anywhere. It comes from the External Secrets Operator.

ESO runs as a set of pods inside the cluster. It watches for ExternalSecret resources, which are Kubernetes manifests that describe what to fetch and where to put it. The ExternalSecret for the backend tells ESO: go to AWS Secrets Manager, read the secret at proshop/backend, and create a Kubernetes Secret named backend-secrets in the proshop namespace with these specific keys.

ESO uses IRSA to authenticate to AWS. IRSA stands for IAM Roles for Service Accounts. It works by linking a Kubernetes service account to an AWS IAM role through an OIDC trust relationship. The ESO service account in the cluster carries an annotation pointing to an IAM role ARN. When ESO calls the AWS API, it exchanges a signed Kubernetes token for temporary AWS credentials. Those credentials are valid only for that specific IAM role, which has permission to read only the secrets it needs.

No static credentials anywhere. No hardcoded access keys. No secrets in environment variables at the infrastructure level. ESO fetches them at runtime, keeps them fresh, and the application reads them as normal environment variables without knowing or caring where they came from.

How ArgoCD keeps everything running

ArgoCD runs a reconciliation loop. Every three minutes, it polls the Git repository and compares what is declared in the manifests against what is actually running in the cluster. If they match, nothing happens. If they differ, ArgoCD brings the cluster back into alignment with Git.

This means if someone manually runs kubectl delete deployment backend-deployment on the cluster, ArgoCD will notice within three minutes and recreate it. The Git repository always wins. This is what self-healing means in practice and it is one of those features that sounds nice on paper until the moment it saves you from a very awkward conversation about why the application suddenly stopped running.

How a new deployment actually happens

When I push code changes to the webapp repository, the GitHub Actions pipeline triggers. It builds new Docker images for the frontend and backend, runs Trivy to scan them for CVEs, and if they pass, pushes them to Amazon ECR with a commit SHA tag.

Then something slightly clever happens. The pipeline commits the new image tag directly back to the infrastructure repository, updating the image field in the backend and frontend deployment manifests from the old SHA to the new one. This commit lands in the infra repo on the main branch.

ArgoCD is watching that branch. Within three minutes it detects that the image tag in the manifest no longer matches what is running in the cluster. It triggers a rolling update. New pods start with the new image. Kubernetes waits for the new pods to pass their readiness probes before terminating the old ones. Zero downtime, assuming the new image actually works.

The entire deployment happened because of a Git commit. No kubectl commands, no Helm upgrades, no pipeline that talks directly to the cluster. The pipeline writes to Git, Git is the source of truth, ArgoCD reads from Git and acts. That is GitOps in practice and honestly, once you see it working, going back to direct kubectl deployments feels like driving without a seatbelt.

How the monitoring stack fits in

Prometheus runs inside the cluster and scrapes metrics from every pod at regular intervals. It knows which pods to scrape because of PodMonitor resources, which are small manifests that tell Prometheus where to look. ArgoCD deployed those PodMonitors, just like it deploys everything else.

Grafana connects to Prometheus as its data source and displays dashboards. The Grafana admin credentials come from AWS Secrets Manager via ESO, same pattern as the application secrets. Grafana persistent storage uses an EBS gp3 volume provisioned by the EBS CSI Driver, so metrics survive pod restarts and node replacements.

CloudWatch Container Insights runs independently as an EKS addon. It collects cluster and pod metrics and sends them to CloudWatch regardless of whether Prometheus is healthy. This is intentional redundancy. If the monitoring stack itself breaks, CloudWatch is still sending data and CloudWatch Alarms are still firing. Monitoring your monitoring infrastructure is one of those things that sounds paranoid until you need it.

Section 5: The Errors That Taught Me the Most

Every project has the errors you fix in five minutes and the errors that make you question your career choices. Here are two from this project that fall firmly in the second category.

The kube-prometheus-stack that refused to sync

The kube-prometheus-stack had been failing for over an hour and I was completely convinced the StorageClass was the problem. No PVCs were binding, Prometheus and Alertmanager were stuck in Pending, and every sync attempt in ArgoCD showed red. I added the gp3 StorageClass, forced refreshes, deleted and recreated the Application. Nothing worked.

The error kept changing on every retry, which made everything worse. First it was an annotation size limit on the CRDs. Then a structured merge diff error. Then the CRDs were partially installed but the critical ones were missing. When the error message changes on every attempt, you start questioning everything, including whether you even understand what you built.

The actual root cause turned out to be two separate problems stacked on top of each other, which is just a delightful thing to discover at hour two of debugging.

The first problem: Prometheus CRDs are enormous. They exceed Kubernetes' 262,144 byte annotation limit. When ArgoCD uses its default apply method, it tries to store the entire manifest as an annotation on the resource for tracking purposes. The manifest is too large. Kubernetes rejects it. The fix was enabling ServerSideApply=true in the ArgoCD Application sync options, which moves the field tracking to the Kubernetes API server instead of storing it in an annotation.
One line in the Application manifest.

The second problem: after hitting that error repeatedly, ArgoCD had accumulated a corrupted operation state. It was failing at the diff phase before it could even attempt a sync, which is why the error message kept changing. Deleting the Application entirely and letting the root app recreate it from scratch gave a clean slate.

What this taught me is something I now apply every time I am debugging: when the error message changes on every retry, you are not fixing the right thing. The real problem is one layer deeper than where you are looking. Stop patching symptoms and start asking why the symptoms exist in the first place.

The Trivy pipeline that lied to me

This one broke me in a different way. Not the frustration of changing errors, but the specific confusion of a tool that appeared to be doing one thing while quietly doing another.

The Trivy scan was failing with exit code 1 after I had already patched every CRITICAL vulnerability in the images. The GitHub Security tab showed zero criticals. Just a wall of HIGH and MEDIUM findings that I had explicitly decided should not block the pipeline. Trivy was clearly passing the severity check and still killing the build.

I assumed the SHA pin was wrong. I spent time verifying the commit hash, swapping to a different pinned version, re-running the pipeline. Nothing changed. The pipeline kept dying on exit code 1 with no CRITICAL findings anywhere.

The actual problem had nothing to do with the vulnerabilities. It had to do with how Trivy handles SARIF output format.

When you set format: sarif in the Trivy action, it writes all severity findings to the SARIF file regardless of your severity filter. The exit code then fires based on what is in the SARIF output, not based on your severity: CRITICAL setting. So Trivy was finding HIGH vulnerabilities, writing them into the SARIF file, and exiting with code 1 even though I never asked it to fail on HIGH findings. The scanner and the reporter were operating on different rules without telling anyone.

The fix was a single line: limit-severities-for-sarif: true. That forces Trivy to respect the severity filter when writing the SARIF output, so the exit code only fires for what you actually configured it to care about.

One line. After an entire afternoon.

What this taught me is something worth writing down: security tools have a difference between what they report and what they enforce, and assuming those two things are always the same will absolutely waste your afternoon. Read the documentation for the edge cases, not just the getting started guide. The getting started guide never mentions the edge cases.

Section 6: Okay, Does It Actually Work?

Alright. I have been talking for five sections straight about architecture, decisions, tradeoffs, and debugging sessions. At some point you are completely within your rights to say "okay man, cool story, but does this thing actually run or did you just write a very confident blog post about a broken cluster?"

Fair. Here is the proof.

The application running live

This is the ProShop storefront loaded via the ALB DNS URL. I already showed you this above, but you guys have trust issues :(
Products fetched from MongoDB Atlas, frontend served by Nginx inside a container, routed through an internet-facing Application Load Balancer that the AWS Load Balancer Controller provisioned automatically from a Kubernetes Ingress manifest. No manual AWS console clicking involved.

ArgoCD managing everything

11 applications, all Synced and Healthy. Now you will see the screenshot and say, "Hey.. I see 12, not 11. What a dumb man". No sir, the applications are 11, the 12th one is the root-app. :( :( :(
The root app sits at the top of the tree and every platform component, monitoring tool, and application workload branches underneath it. Everything you see here was deployed by ArgoCD reading from a Git repository. Nothing was applied manually to this cluster.

The full automated standup pipeline

This is the GitHub Actions workflow run that built the entire environment from scratch. Six jobs running in sequence: Terraform apply, trigger image build, bootstrap ArgoCD, validate deployment, apply CloudWatch alarms. Every step automated. The ALB DNS URL appears at the bottom of the validate-deployment job output so I know exactly where to open the application without checking the AWS console.

The monitoring stack with live data

Grafana loaded with the Node Exporter Full dashboard showing real cluster metrics. CPU, memory, disk, and network data flowing from the actual nodes. Prometheus is scraping, the data source is resolving correctly, and the dashboards are populated. Not placeholder data. Not a demo environment. A real running cluster.

CloudWatch Alarms doing their job

Three of the eight CloudWatch Alarms fired during validation on real cluster pressure. That is not a bad thing. That is the alarms being correctly tuned. An alarm that never fires is either monitoring something that never goes wrong or monitoring nothing at all. The ones that fired confirmed the thresholds are set at meaningful levels.

Prometheus scraping ArgoCD metrics

This one is a personal favourite because it was an entire Phase 5 gap that needed fixing. ArgoCD exposes metrics on its internal services but Prometheus does not discover them automatically. I had to write PodMonitor resources telling Prometheus exactly where to look. The targets page showing ArgoCD endpoints with status UP means the fix worked and the ArgoCD dashboard in Grafana has actual data behind it.

The Trivy scan gate working correctly

Every image push goes through Trivy first. This is the scan passing in the webapp CI pipeline after patching the SARIF severity filtering issue from Section 5. Zero CRITICAL findings, pipeline continues, image gets pushed to ECR. The security gate is not decoration.

Section 7: What I Would Do Differently and What Comes Next

No project is ever really finished. It either gets abandoned or it gets better. Here is what I would change and what I would add if I kept going.

HTTPS first, everything else second

The application runs on HTTP. For a project that handles user authentication and payment flows, that is the most glaring gap in the entire architecture. The fix is straightforward: cert-manager in the cluster to automate Let's Encrypt certificate provisioning, a Route 53 domain pointing to the ALB, and an ALB listener on port 443. The Ingress annotations change from HTTP to HTTPS with an automatic redirect. This is not a nice-to-have for an e-commerce application. It is the first thing I would add.

A custom domain also solves the problem of the ALB DNS name changing on every destroy and redeploy. Right now the URL looks like k8s-proshop-proshopi-b3bd903edf-191854457.us-east-1.elb.amazonaws.com which is not something you put on a business card. A Route 53 Alias record pointing to the ALB gives a stable URL that survives infrastructure rebuilds.

Canary deployments

The current deployment strategy is a rolling update. Kubernetes replaces old pods with new pods gradually, which is better than taking the application down completely, but a bad deployment still eventually reaches every user. Argo Rollouts adds canary deployment capability to the existing ArgoCD setup with minimal changes. Route five percent of traffic to the new version, watch the error rate and response time metrics for five minutes, then promote to full rollout if everything looks healthy. For an application that processes real payments, the ability to catch a bad deployment before it reaches everyone is worth the additional complexity.

Distributed tracing

Metrics tell you something is slow. Logs tell you what happened. Distributed tracing tells you exactly where in the request chain the slowness is coming from. If a product page is taking three seconds to load, right now I would have to cross-reference Prometheus metrics, CloudWatch logs, and my own intuition to figure out whether the bottleneck is the Nginx serving the frontend, the Express API, or the MongoDB query. OpenTelemetry with Jaeger or Tempo would show me the full request trace in a single view. It is the observability gap that would hurt the most in a real production debugging scenario.

Karpenter for smarter node scaling

The current node group scales by adding more instances of the same type. Karpenter replaces this with intelligent provisioning that picks the right instance type and size for the actual workload being scheduled. During low traffic periods it would provision smaller, cheaper instances. During spikes it would provision exactly what is needed rather than overshooting. The tradeoff is operational complexity, but for a production deployment with variable traffic, the cost savings over time would be significant.

Section 8: What Building This Actually Taught Me

I want to be honest about what changed from the beginning of this project to the end, because I think it is more useful than a list of technologies I learned.

At the start, I thought infrastructure was about provisioning resources. You write Terraform, things appear in AWS, you are done. That is not wrong exactly, but it is incomplete in a way that causes real problems. What I understand now is that infrastructure is about managing lifecycles. Different things change at different rates, and when you force them to share the same lifecycle, you create coupling that makes everything harder to change, harder to debug, and harder to destroy safely.

The terraform destroy hanging for twenty minutes because of an ALB that Terraform did not know about was not bad luck. It was a direct consequence of a design decision that mixed infrastructure and application concerns into the same tool. Once I understood that, the entire architecture of the rebuilt project became obvious. Not clever, not over-engineered, just the natural result of respecting the boundaries between things that have different jobs.

I also learned that documentation is not something you write at the end. Every architectural decision in this project was documented at the time it was made, including the decisions I rejected and why. That documentation is why I can explain the reasoning behind every line in this codebase in an interview. It is also why I can read this blog six months from now and remember why I made the choices I made, rather than staring at a values file wondering why I chose ArgoCD over Flux.

The errors taught me more than the successes. The IRSA namespace bug taught me that silent failures are more dangerous than loud ones because they let you believe everything is fine. The Trivy SARIF exit code issue taught me that tools have a difference between what they report and what they enforce, and that assuming those are the same is a mistake you only make once. The kube-prometheus-stack CRD annotation limit taught me that when the error keeps changing, you are debugging the symptom and not the cause.

If I could give one piece of advice to someone starting a project like this, it would be this: decide what each tool is responsible for before you start writing any code. Not what it is capable of. What it is responsible for. Terraform is capable of deploying Kubernetes pods. It should not be responsible for them. The clearer those boundaries are at the start, the fewer hours you spend at 2am watching a terraform destroy hang on a VPC that refuses to delete.

This project is not finished. It is at a point where I am proud of it and honest about what is missing. That feels like the right place to stop for now.

The full source code is available at cloud-native-ecommerce-platform. If you are building something similar or have questions about any of the decisions in this post, the comments are open.

DEV Community: Kalpit Swami