DEV Community: Bader Eddine El ouerghi

The 7 AWS charges nobody chose

Bader Eddine El ouerghi — Wed, 21 Oct 2026 19:52:32 +0000

Most AWS waste isn't a bad architecture decision. It's a charge that nobody decided on: a default, a leftover, a setting nobody revisited. I work across 170+ AWS accounts, and these seven show up in almost every one of them.

Each section has what it is, why it happens, how to find it (read-only commands) and the fix. Prices are us-east-1 list prices; check your region.

1. Unattached EBS volumes

What: an EC2 instance is terminated, but its data volume had DeleteOnTermination=false (the default for volumes attached after launch). The disk stays, billed every month: about US$0.08/GB-month for gp3, US$0.10 for gp2.

Find them:

aws ec2 describe-volumes --filters Name=status,Values=available \
  --query "Volumes[].[VolumeId,Size,VolumeType,CreateTime]" --output table

Fix: snapshot anything you're unsure about (snapshots are much cheaper per GB), then delete the volume. While you're there, convert remaining gp2 volumes to gp3: same or better performance, ~20% cheaper, no downtime.

2. Public IPv4 addresses

What: since February 2024, AWS charges US$0.005/hour for every public IPv4 address — attached or not. That's about US$3.60/month each. EC2 instances in public subnets, NAT Gateways, load balancers (one per AZ), and idle Elastic IPs all count.

Find them: the quickest view is the free Public IP Insights in VPC IP Address Manager. Or list Elastic IPs that aren't associated:

aws ec2 describe-addresses --query "Addresses[?AssociationId==null].PublicIp"

Fix: release idle Elastic IPs, turn off "auto-assign public IPv4" on subnets that don't need it, and put instances behind a load balancer or use SSM Session Manager instead of a public IP for access.

3. NAT Gateways in dev

What: a NAT Gateway costs about US$0.045/hour (≈ US$33/month) before a single byte goes through it, plus US$0.045 per GB processed. Many VPC templates create one per Availability Zone — three per VPC — and get copied into every dev and test account.

Find them:

aws ec2 describe-nat-gateways --filter Name=state,Values=available \
  --query "NatGateways[].[NatGatewayId,VpcId,SubnetId]" --output table

Fix: in non-production, one NAT Gateway per VPC is plenty (you accept that one AZ failure stops outbound traffic for dev — usually fine). For sandboxes that rarely need outbound internet, consider none at all.

4. CloudWatch Logs set to "Never expire"

What: every new log group defaults to keeping logs forever. Storage is about US$0.03/GB-month — small per GB, but debug logs from years ago add up quietly, and nobody reads them.

Find them:

aws logs describe-log-groups \
  --query "logGroups[?retentionInDays==null].[logGroupName,storedBytes]" --output table

Fix: set retention on every group (aws logs put-retention-policy --log-group-name X --retention-in-days 30), and set it in your Terraform/CloudFormation modules so new groups never default to forever. Keep long retention only where compliance requires it — and then consider exporting to S3 with a lifecycle rule.

5. S3 incomplete multipart uploads

What: when a large upload fails halfway, the parts already uploaded stay in the bucket. They don't appear in the console object list, but you pay for their storage — indefinitely.

Find them:

aws s3api list-multipart-uploads --bucket my-bucket \
  --query "Uploads[].[Key,Initiated]" --output table

S3 Storage Lens also shows "incomplete multipart upload bytes" across all buckets.

Fix: one lifecycle rule per bucket, ideally added by default in your bucket module:

{ "Rules": [{ "ID": "abort-mpu", "Status": "Enabled", "Filter": {},
  "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } }] }

6. EKS clusters on old Kubernetes versions

What: each Kubernetes version gets about 14 months of standard support on EKS. After that, AWS doesn't stop your cluster — it moves it to extended support, and the control plane goes from US$0.10/hour to US$0.60/hour. That's roughly US$73 → US$438 per cluster per month, 6× more, for the same cluster.

Find them:

for c in $(aws eks list-clusters --query "clusters[]" --output text); do
  aws eks describe-cluster --name "$c" \
    --query "cluster.[name,version,upgradePolicy.supportType]" --output text
done

Fix: delete clusters nobody uses (check workloads and owners first), and upgrade the rest one minor version at a time, non-prod first. The next post in this series walks through doing that safely.

7. S3 and DynamoDB traffic through a NAT Gateway

What: instances in private subnets talking to S3 or DynamoDB go through the NAT Gateway by default — at US$0.045 per GB processed. A gateway VPC endpoint for S3 or DynamoDB does the same job for free: no hourly charge, no per-GB charge.

Find it: check if your VPCs have gateway endpoints:

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-endpoint-type,Values=Gateway \
  --query "VpcEndpoints[].[VpcId,ServiceName]" --output table

Then compare with NAT Gateway "BytesOutToDestination" in CloudWatch. If a VPC has a busy NAT Gateway and no S3 endpoint, that's usually your answer.

Fix: add the gateway endpoints and associate them with the private route tables. In Terraform it's one aws_vpc_endpoint resource per service. Existing connections keep working; new traffic takes the free path.

Seeing them in Cost Explorer

The commands above work one account and region at a time. To see how much these seven cost across the whole organization, open Cost Explorer in the management (payer) account, group by Usage type, and filter for the last full month. Useful usage types to search for:

EBS:VolumeUsage.gp2 / .gp3 — compare with what's attached
PublicIPv4:InUseAddress and PublicIPv4:IdleAddress
NatGateway-Hours and NatGateway-Bytes
TimedStorage-ByteHrs under CloudWatch (log storage)
AmazonEKS usage with "extendedSupport" in the name

Then group the same view by Linked account. In my experience, a few accounts — often old sandboxes and abandoned projects — hold most of the waste. That tells you where to start, and who to ask before deleting anything.

The pattern

None of these need a re-architecture. Most take an afternoon, and the real fix is the same each time: change the default in the module or template that creates the resource, so the waste doesn't come back next quarter.

A simple order that works:

Run the seven read-only checks above in every account and region.
Fix the defaults in your IaC first (log retention, lifecycle rules, endpoints, gp3, subnet IP settings).
Clean up what already exists, snapshot first when in doubt.
Put a monthly reminder in the calendar to run the checks again.

What's the strangest AWS charge you've found? I'd love to hear it in the comments.

Badereddine is a cloud and DevOps engineer in Canada who operates 170+ AWS accounts and 500+ EKS clusters day to day, and runs TERRAX, a small AWS cost and platform engineering practice.

The 45 checks I run first, in the order that saves the most, are in a free checklist: AWS Cost Audit Checklist.

How We Built a Multi-Cloud Terraform Orchestrator: The TerraX Architecture

Bader Eddine El ouerghi — Tue, 23 Jun 2026 13:20:43 +0000

Beta is open. If you want to try TerraX before finishing this article: docs.terrax-cloud.com

Why We Built This

Managing infrastructure across AWS, GCP, Azure, and Cloudflare with a small team taught us one thing fast: Terraform itself is not the bottleneck. Everything around it is.

State backends scattered across providers. Credentials living on individual laptops. No shared record of who ran a plan and what changed. And a persistent gap between "infrastructure provisioned" and "things running on that infrastructure configured" — a gap that gets filled by Bash scripts nobody wants to maintain.

We wanted a platform that treated each cloud provider as a first-class object: centralized credentials, isolated state per provider, a sandboxed executor that runs terraform plan and terraform apply with full log streaming, and a workflow engine that could chain Terraform runs with script functions and pass outputs between them. That became TerraX.

This article walks through the key architectural decisions — what we chose, what we nearly chose, and where we got things wrong the first time.

System Overview

TerraX is composed of four main layers:

[ Angular Frontend ]
        |
[ Spring Boot Backend — "teraflare" ]
        |
[ Go Agent — "terrax-agent" ]
        |
[ Terraform / OpenTofu executor ]

Frontend (Angular) handles the UI layer: resource forms, plan/apply diff views, workflow builder, kubeconfig manager, compliance dashboards. It talks exclusively to the backend via REST.

Backend (Spring Boot / Java) is the core: provider credential storage, Terraform state management, RBAC enforcement, workflow orchestration, cost forecasting, compliance scanning. DB migrations run via Flyway.

Agent (Go) is the sandboxed executor. It receives jobs from the backend, bundles workspace files, runs Terraform/OpenTofu, and streams logs back in real time. It runs separately from the backend for isolation — a misbehaving Terraform run can't take down the API server.

Terraform / OpenTofu is the execution engine underneath everything. We default to OpenTofu.

The Go Agent: Why Go, and How the Sandbox Works

The decision to write the agent in Go rather than extending the Java backend was deliberate. Terraform execution is I/O-heavy and long-running — sometimes 10+ minutes for a full apply on a complex provider. We didn't want those goroutines competing with API request handling on the same JVM.

Go gave us lightweight concurrency for managing multiple concurrent jobs, straightforward process management for spawning tofu as a subprocess, and simple binary deployment into the Kubernetes pod that runs the executor.

When the backend enqueues a job, the agent:

Receives the job payload (provider credentials, workspace ID, operation type)
Bundles the workspace: pulls .tf files, variable files, and state from the backend
Writes them to a temporary working directory
Spawns tofu plan or tofu apply as a subprocess
Streams stdout/stderr back to the backend line by line
Reports the final exit code and cleans up

The key design decision here: state is never stored on the agent. The agent is stateless — it pulls what it needs per job and pushes results back. This means you can run multiple agent instances horizontally and jobs route to whichever is available.

We chose OpenTofu as the default because of the licensing clarity. The routing is configurable per job if you need Terraform for a specific provider version.

Cross-Cloud Kubeconfig Generation Without the Cloud CLIs

One of the most concrete problems we solved is kubeconfig generation across EKS, GKE, and AKS without depending on aws, gcloud, or az being installed anywhere.

Each provider has a completely different auth model:

EKS uses an exec-plugin model. The kubeconfig calls aws eks get-token at authentication time, which means the AWS CLI must be installed wherever the kubeconfig is used. We generate the kubeconfig server-side by calling eks:DescribeCluster via the AWS SDK to get the cluster endpoint and CA data, then embed the exec-plugin block pointing to the AWS SDK's token endpoint. Auth still flows through the SDK — we're not storing long-lived tokens.

GKE is simpler in some ways: we call the GCP Container REST API with a service account token, pull the cluster endpoint and CA certificate, and embed a short-lived Bearer token directly in the kubeconfig. The downside is that token expiry means regenerating the kubeconfig periodically, which we handle transparently.

AKS provides the cleanest result: listClusterAdminCredential from the Azure ARM API returns a ready-to-use kubeconfig with a cert-based auth block. No token rotation needed, though the cert-based approach means these are longer-lived credentials that need to be stored carefully.

All three are stored namespaced per cluster in the backend database, accessible based on the user's role, and selectable as the target context for any downstream operation — ArgoCD installs, Helm runs, raw kubectl operations via the agent.

The Workflow Engine: Passing Outputs Between Steps

The workflow engine is what ties infrastructure provisioning to operational automation. A workflow is a sequence of steps with conditional logic — each step is either a Terraform run or a Script Function (Python, Bash, or Docker container).

The interesting part is output passing. When a Terraform apply completes, its outputs (e.g., a cluster endpoint, a load balancer IP, a database hostname) need to be available to the next step without manual copy-paste. We built a dynamic variable resolution system using reference strings:

ref:var:VARIABLE_NAME — resolves a variable defined at the provider or workflow level
ref:provider:ATTRIBUTE — resolves an attribute from the connected provider
ref:function:OUTPUT_KEY — resolves an output captured from a previous Script Function step

At runtime, the backend resolves these references before passing the step payload to the agent. This means you can provision an EKS cluster in step 1, capture its endpoint as ref:var:CLUSTER_ENDPOINT, and pass it to a Helm install in step 2 — without hardcoding anything.

Script Functions run in isolated Docker containers or as Kubernetes jobs, with secrets injected as environment variables. Output is captured from stdout as structured key-value pairs and made available to subsequent steps via ref:function:.

State Isolation Per Provider

One of the early architectural decisions that saved us the most headaches: one state backend per provider, not one global state.

The alternative — a single shared state backend with workspaces — makes cross-provider references tempting and creates blast radius problems. A corrupted state file for your AWS provider shouldn't touch your GCP resources.

In practice, each provider object in TerraX has its own isolated state stored in the backend. When the agent runs a plan or apply, it pulls the state for that specific provider, executes against it, and pushes the updated state back. Providers never share state files.

The tradeoff: cross-provider data references have to go through the workflow engine's variable system, not Terraform's native data sources across workspaces. For most use cases this is fine. For deeply coupled multi-cloud architectures it adds some friction — something we're still working through.

Compliance Scanning: OPA + Checkov

Every Terraform configuration that passes through TerraX can be scored against compliance frameworks before apply. The scanning layer runs two tools:

Checkov handles static analysis of the .tf files — it understands Terraform's resource schema and checks for common misconfigurations (public S3 buckets, unencrypted RDS, overly permissive security groups, etc.).

OPA (Open Policy Agent) handles custom policy enforcement via Rego policies. This is where you encode organization-specific rules that Checkov doesn't cover — things like "all GCP resources must have a cost-center label" or "no EKS node groups with instance types larger than m5.xlarge without approval."

Results are scored against PCI DSS, CIS Benchmarks, NIST CSF, and SOC 2 controls, aggregated per provider, and surfaced as a compliance score in the dashboard. One-click "fix preview" shows the Terraform change that would resolve a finding.

The Browser Extension

The Chrome extension deserves a brief mention because it solved a real problem: existing Cloudflare resources.

When you have 200 DNS records, 30 WAF rules, and 15 Page Rules already configured in the Cloudflare dashboard, bringing them under Terraform management is painful. You need to write HCL for each one and run terraform import with the correct Cloudflare resource ID.

The extension runs on the Cloudflare dashboard pages, detects resource configurations as you browse, and sends them to TerraX where they're staged for import. It effectively automates the discovery + HCL generation step for resources you're already looking at in the UI.

What We Got Wrong

Abstracting too much from Terraform's output. In early versions we summarized the plan output into a "changes" view. Users hated it. They wanted to see the raw plan, exactly as Terraform prints it. We reverted to showing the full output with syntax highlighting.

Assuming exit codes were reliable. Terraform's exit code behavior across versions and providers is inconsistent enough that we had to parse stdout/stderr directly to determine plan/apply success. Lesson: don't trust exit codes alone for infrastructure tools.

Underestimating the import UX. The most-used feature after launch wasn't resource creation — it was state import. People have years of existing infra they want under management. We initially treated import as a secondary flow and had to rebuild it properly after the first wave of feedback.

Current Limitations

Drift detection is manual — no scheduled reconciliation against live cloud yet
Azure resource coverage is thinner than AWS and GCP
Generated HCL isn't always idiomatic for complex Terraform module structures
ArgoCD SSO not fully tested end-to-end against all identity providers

Try the Beta

TerraX is in open beta. 14-day free trial, no credit card required.

App: app.terrax-cloud.com
Docs: docs.terrax-cloud.com

If you have feedback — especially critical feedback on the architecture decisions above — I'm in the comments.