October 2026 | ~16 min read
The Line Item Nobody Checked
We were celebrating. Posts #7, #8, and #9 had taken our EKS compute costs from $14,200/month down to $3,800/month. Then finance called. Our overall AWS bill had only dropped by $6,000/month, not the $10,400 we expected. Where was the rest of the money going?
The answer was buried in a Cost Explorer category called "EC2-Other" — a catch-all that AWS uses for data transfer charges. We'd never drilled into it. When we did: $22,400/month. NAT Gateway alone was $9,400/month — our second-largest line item after EC2 compute, and nobody on the engineering team even knew it existed.
We spent four weeks fixing it. Data transfer dropped from $22,400/month to $8,300/month — a 63% reduction, $14,100/month in savings, $169,200 annualized. Here's everything we did.
The Numbers That Matter
┌──────────────────────────────────────────────────────────────────┐
│ DATA TRANSFER COST REDUCTION │
├──────────────────────────┬──────────┬──────────┬─────────────────┤
│ Category │ Before │ After │ Savings │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ NAT Gateway │ $9,400 │ $4,100 │ -$5,300 (-56%) │
│ Cross-AZ traffic │ $5,200 │ $1,400 │ -$3,800 (-73%) │
│ Internet egress │ $4,800 │ $1,600 │ -$3,200 (-67%) │
│ S3 data transfer │ $3,000 │ $1,200 │ -$1,800 (-60%) │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ TOTAL │ $22,400 │ $8,300 │ -$14,100 (-63%) │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ New infra cost │ │ $180/mo │ VPC endpoints │
│ Net monthly savings │ │ │ $13,920 │
│ Annualized │ │ │ $169,200 │
└──────────────────────────┴──────────┴──────────┴─────────────────┘
Table of Contents
- Understanding Your Data Transfer Bill
- NAT Gateway — The Silent Budget Killer
- Cross-AZ Traffic — The Per-Gigabyte Tax
- Internet Egress — CloudFront as a Cost Optimizer
- S3 Data Transfer
- Monitoring Data Transfer Costs
- Results and ROI
- Lessons Learned
- Action Items Checklist
Understanding Your Data Transfer Bill
The first problem: finding data transfer costs at all. AWS doesn't group them in one place. They're spread across multiple line items, and the biggest ones hide under "EC2-Other" in Cost Explorer.
Here's what we found when we finally broke it down:
┌──────────────────────────────────────────────────────────────────┐
│ 5 TYPES OF DATA TRANSFER CHARGES │
├────────────────────────┬──────────────┬───────────────────────────┤
│ Type │ Rate │ Where It Hides │
├────────────────────────┼──────────────┼───────────────────────────┤
│ NAT Gateway processing │ $0.045/GB │ EC2-Other → NAT Gateway │
│ NAT Gateway hourly │ $0.045/hr/AZ │ EC2-Other → NAT Gateway │
│ Cross-AZ transfer │ $0.01/GB/dir │ EC2-Other → Data Transfer │
│ Internet egress │ $0.09/GB* │ EC2-Other → Data Transfer │
│ S3 transfer out │ $0.09/GB* │ S3 → Data Transfer Out │
│ Inter-region │ $0.02/GB │ EC2-Other → Data Transfer │
├────────────────────────┴──────────────┴───────────────────────────┤
│ * First 10 TB/month. Tiered pricing after. │
└──────────────────────────────────────────────────────────────────┘
To actually see these numbers, you need to go beyond Cost Explorer's default view. We wrote a script that pulls data transfer costs from Cost and Usage Reports (CUR) via Athena:
#!/usr/bin/env python3
"""
Pull data transfer cost breakdown from CUR via Athena.
Requires: CUR export to S3, Athena table configured.
"""
import boto3
import time
import csv
import io
ATHENA_DATABASE = "cur_database"
ATHENA_TABLE = "cost_and_usage_report"
ATHENA_OUTPUT = "s3://your-athena-results-bucket/data-transfer/"
QUERY = """
SELECT
line_item_usage_type,
product_transfer_type,
CASE
WHEN line_item_usage_type LIKE '%NatGateway%' THEN 'NAT Gateway'
WHEN line_item_usage_type LIKE '%DataTransfer-Regional%'
OR line_item_usage_type LIKE '%IntraRegion%' THEN 'Cross-AZ'
WHEN line_item_usage_type LIKE '%DataTransfer-Out%' THEN 'Internet Egress'
WHEN line_item_usage_type LIKE '%InterRegion%' THEN 'Inter-Region'
WHEN product_product_name = 'Amazon Simple Storage Service'
AND line_item_usage_type LIKE '%DataTransfer%' THEN 'S3 Transfer'
ELSE 'Other'
END AS transfer_category,
SUM(line_item_unblended_cost) AS total_cost,
SUM(line_item_usage_amount) AS total_gb
FROM {database}.{table}
WHERE line_item_product_code IN ('AmazonEC2', 'AmazonS3')
AND line_item_usage_type LIKE '%DataTransfer%'
OR line_item_usage_type LIKE '%NatGateway%'
AND month = '{month}'
AND year = '{year}'
GROUP BY 1, 2,
CASE
WHEN line_item_usage_type LIKE '%NatGateway%' THEN 'NAT Gateway'
WHEN line_item_usage_type LIKE '%DataTransfer-Regional%'
OR line_item_usage_type LIKE '%IntraRegion%' THEN 'Cross-AZ'
WHEN line_item_usage_type LIKE '%DataTransfer-Out%' THEN 'Internet Egress'
WHEN line_item_usage_type LIKE '%InterRegion%' THEN 'Inter-Region'
WHEN product_product_name = 'Amazon Simple Storage Service'
AND line_item_usage_type LIKE '%DataTransfer%' THEN 'S3 Transfer'
ELSE 'Other'
END
ORDER BY total_cost DESC
""".format(
database=ATHENA_DATABASE,
table=ATHENA_TABLE,
month="09",
year="2026",
)
def run_query():
client = boto3.client("athena")
response = client.start_query_execution(
QueryString=QUERY,
QueryExecutionContext={"Database": ATHENA_DATABASE},
ResultConfiguration={"OutputLocation": ATHENA_OUTPUT},
)
query_id = response["QueryExecutionId"]
while True:
result = client.get_query_execution(QueryExecutionId=query_id)
state = result["QueryExecution"]["Status"]["State"]
if state in ("SUCCEEDED", "FAILED", "CANCELLED"):
break
time.sleep(2)
if state != "SUCCEEDED":
raise RuntimeError(f"Query {state}: {result}")
paginator = client.get_paginator("get_query_results")
rows = []
for page in paginator.paginate(QueryExecutionId=query_id):
for row in page["ResultSet"]["Rows"][1:]: # skip header
rows.append([col.get("VarCharValue", "") for col in row["Data"]])
# Print summary by category
categories = {}
for row in rows:
cat = row[2]
cost = float(row[3]) if row[3] else 0
gb = float(row[4]) if row[4] else 0
if cat not in categories:
categories[cat] = {"cost": 0, "gb": 0}
categories[cat]["cost"] += cost
categories[cat]["gb"] += gb
print(f"\n{'Category':<20} {'Cost':>12} {'GB':>12}")
print("─" * 48)
total = 0
for cat, data in sorted(categories.items(), key=lambda x: -x[1]["cost"]):
print(f"{cat:<20} ${data['cost']:>10,.2f} {data['gb']:>10,.1f}")
total += data["cost"]
print("─" * 48)
print(f"{'TOTAL':<20} ${total:>10,.2f}")
if __name__ == "__main__":
run_query()
When we ran this against September's bill, the output looked like this:
Category Cost GB
────────────────────────────────────────────────
NAT Gateway $9,413.28 209,184.0
Cross-AZ $5,218.40 260,920.0
Internet Egress $4,792.80 53,253.3
S3 Transfer $2,976.52 33,072.4
────────────────────────────────────────────────
TOTAL $22,401.00
Why don't most teams notice? Because "EC2-Other" is a single line in Cost Explorer. If you're only looking at service-level breakdown (EC2, RDS, S3), data transfer hides inside EC2 and S3 totals. You have to filter by usage type to see it. Most teams don't — until the bill is thousands per month.
NAT Gateway — The Silent Budget Killer
NAT Gateway is the most expensive data transfer line item for most AWS accounts, and the least understood. Here's why.
Every private subnet in your VPC routes internet-bound traffic through a NAT Gateway. That means every API call to an AWS service (ECR, S3, CloudWatch, STS, SSM) from your EKS nodes goes through NAT — and you pay $0.045 per GB processed, on top of $0.045/hour per NAT Gateway per AZ.
Our bill:
NAT Gateway cost breakdown:
Hourly charges: 3 AZs × $0.045/hr × 730 hrs/mo = $98.55/mo
Processing: 209 TB × $0.045/GB = $9,405/mo
──────────────────────────────────────────────────────────────
Total: $9,404/mo
209 TB per month through NAT Gateway. Where was it all going? We enabled VPC Flow Logs on the NAT Gateway ENI and analyzed the traffic:
NAT Gateway traffic breakdown (top destinations):
S3 (ECR image layers + log archival) 89 TB 42%
ECR API (image manifest pulls) 31 TB 15%
CloudWatch Logs (log ingestion) 28 TB 13%
CloudWatch Metrics (custom metrics) 19 TB 9%
STS (IRSA token refresh) 14 TB 7%
SSM (parameter store lookups) 11 TB 5%
Other AWS APIs 17 TB 8%
──────────────────────────────────────────────────────────
Total 209 TB 100%
89 TB to S3 alone — mostly ECR image layers (stored in S3 behind the scenes) and application log archival. Every container image pull, every log upload, every API call — all routing through NAT at $0.045/GB.
Fix 1: VPC Gateway Endpoints for S3 and DynamoDB
VPC gateway endpoints are free. They route traffic to S3 and DynamoDB directly through the AWS backbone instead of through NAT Gateway. There is no per-GB processing charge, no hourly charge, nothing. This should be the first thing you deploy in every VPC.
# modules/vpc-endpoints/gateway.tf
# S3 Gateway Endpoint — free, saves NAT processing costs
resource "aws_vpc_endpoint" "s3" {
vpc_id = var.vpc_id
service_name = "com.amazonaws.${var.region}.s3"
vpc_endpoint_type = "Gateway"
route_table_ids = var.private_route_table_ids
tags = {
Name = "${var.environment}-s3-gateway"
Environment = var.environment
Purpose = "cost-optimization"
}
}
# DynamoDB Gateway Endpoint — also free
resource "aws_vpc_endpoint" "dynamodb" {
vpc_id = var.vpc_id
service_name = "com.amazonaws.${var.region}.dynamodb"
vpc_endpoint_type = "Gateway"
route_table_ids = var.private_route_table_ids
tags = {
Name = "${var.environment}-dynamodb-gateway"
Environment = var.environment
Purpose = "cost-optimization"
}
}
variable "vpc_id" {
type = string
}
variable "region" {
type = string
default = "us-east-1"
}
variable "environment" {
type = string
default = "production"
}
variable "private_route_table_ids" {
type = list(string)
description = "Route table IDs for private subnets"
}
The key detail: you must associate the gateway endpoint with every private subnet's route table. If you miss one, traffic from that subnet still goes through NAT.
# Verify the endpoint is working — run from an EC2 instance in a private subnet
# Before: traffic routes through NAT Gateway
traceroute s3.us-east-1.amazonaws.com
# After gateway endpoint: traffic routes through VPC endpoint (no NAT hop)
traceroute s3.us-east-1.amazonaws.com
Result: -$2,800/month. All S3 traffic (ECR image layers + log archival) now bypasses NAT entirely. This single change took 10 minutes to deploy and saved more than most of our individual EKS optimizations.
Fix 2: VPC Interface Endpoints for High-Traffic AWS Services
Gateway endpoints only work for S3 and DynamoDB. For everything else (ECR, CloudWatch, STS, SSM), you need interface endpoints. These cost $0.01/GB processed + $7.20/month per AZ — but if you're pushing more than 160 GB/month per service through NAT, the interface endpoint is cheaper.
Our decision framework:
Interface endpoint break-even analysis:
NAT processing cost: $0.045/GB
Interface endpoint cost: $0.01/GB + $7.20/AZ/month
Break-even per AZ: $7.20 / ($0.045 - $0.01) = 205 GB/month
With 3 AZs: $21.60/month fixed + $0.01/GB
Rule: if a service pushes >205 GB/month through NAT, deploy an
interface endpoint. Below that, NAT is cheaper.
We deployed interface endpoints for four services:
# modules/vpc-endpoints/interface.tf
locals {
# Services where our NAT traffic exceeded the break-even threshold
interface_endpoints = {
ecr_api = {
service = "com.amazonaws.${var.region}.ecr.api"
name = "ecr-api"
monthly_gb = 31000 # 31 TB — well above break-even
}
ecr_dkr = {
service = "com.amazonaws.${var.region}.ecr.dkr"
name = "ecr-dkr"
monthly_gb = 31000
}
logs = {
service = "com.amazonaws.${var.region}.logs"
name = "cloudwatch-logs"
monthly_gb = 28000 # 28 TB
}
sts = {
service = "com.amazonaws.${var.region}.sts"
name = "sts"
monthly_gb = 14000 # 14 TB
}
ssm = {
service = "com.amazonaws.${var.region}.ssm"
name = "ssm"
monthly_gb = 11000 # 11 TB
}
}
}
resource "aws_vpc_endpoint" "interface" {
for_each = local.interface_endpoints
vpc_id = var.vpc_id
service_name = each.value.service
vpc_endpoint_type = "Interface"
private_dns_enabled = true
subnet_ids = var.private_subnet_ids
security_group_ids = [aws_security_group.vpc_endpoints.id]
tags = {
Name = "${var.environment}-${each.value.name}"
Environment = var.environment
Purpose = "cost-optimization"
MonthlyGB = each.value.monthly_gb
}
}
resource "aws_security_group" "vpc_endpoints" {
name_prefix = "${var.environment}-vpc-endpoints-"
vpc_id = var.vpc_id
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = [var.vpc_cidr]
description = "HTTPS from VPC"
}
tags = {
Name = "${var.environment}-vpc-endpoints"
Environment = var.environment
}
}
variable "private_subnet_ids" {
type = list(string)
description = "Subnet IDs for interface endpoints"
}
variable "vpc_cidr" {
type = string
description = "VPC CIDR block for security group rules"
}
Cost of interface endpoints: 5 endpoints × 3 AZs × $7.20/month = $108/month fixed, plus $0.01/GB processing. Against $0.045/GB through NAT for the same traffic, the savings are massive.
Result: -$1,900/month (after subtracting endpoint costs).
Fix 3: ECR Pull-Through Cache
Every time a pod starts, it pulls its container image. If you're running 200+ pods across 3 AZs with Karpenter scaling nodes up and down, that's a lot of redundant image pulls. ECR pull-through cache stores upstream images (Docker Hub, public ECR, GitHub Container Registry) in your private ECR registry, so pulls stay local.
# Create pull-through cache rules for Docker Hub and public ECR
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix docker-hub \
--upstream-registry-url registry-1.docker.io \
--region us-east-1
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix ecr-public \
--upstream-registry-url public.ecr.aws \
--region us-east-1
Update your Kubernetes deployments to use the cached path:
# Before: pulls from Docker Hub through NAT every time
# image: nginx:1.25
# After: pulls from ECR cache (stays in VPC via endpoint)
# image: <account-id>.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/nginx:1.25
We automated this with a mutating webhook that rewrites image references at admission time, so developers didn't need to change their manifests:
# ecr-cache-mutating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: ecr-pull-through-cache
webhooks:
- name: ecr-cache.k8s.io
admissionReviewVersions: ["v1"]
sideEffects: None
clientConfig:
service:
name: ecr-cache-webhook
namespace: kube-system
path: /mutate
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
namespaceSelector:
matchExpressions:
- key: ecr-cache
operator: NotIn
values: ["disabled"]
Result: -$600/month. Image pulls dropped by 78% because most pulls now hit the local ECR cache instead of traversing NAT to Docker Hub.
NAT Gateway Summary
NAT Gateway optimization results:
Before: $9,400/mo
After gateway endpoints (S3/DDB): -$2,800
After interface endpoints: -$1,900
After ECR pull-through cache: -$600
──────────────────────────────────────────────
After: $4,100/mo (-56%)
Cross-AZ Traffic — The Per-Gigabyte Tax
AWS charges $0.01/GB in each direction for traffic between Availability Zones — $0.02/GB round trip. This sounds negligible until you run microservices that make hundreds of thousands of cross-AZ calls per hour.
Our setup: 14 microservices running across 3 AZs, with Kubernetes service routing randomly selecting pods regardless of AZ. A request from a pod in us-east-1a to a pod in us-east-1b incurs a $0.02 round-trip charge. Multiply by millions of requests per day and our cross-AZ bill was $5,200/month.
Fix 1: Topology-Aware Routing in EKS
Kubernetes topology-aware routing (previously called topology hints) tells kube-proxy to prefer pods in the same AZ when routing service traffic. This keeps most service-to-service calls in-zone, avoiding cross-AZ charges.
# Enable topology-aware routing on high-traffic internal services
apiVersion: v1
kind: Service
metadata:
name: payments-api
namespace: payments
annotations:
service.kubernetes.io/topology-mode: Auto
spec:
selector:
app: payments-api
ports:
- port: 8080
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: user-service
namespace: platform
annotations:
service.kubernetes.io/topology-mode: Auto
spec:
selector:
app: user-service
ports:
- port: 8080
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: inventory-service
namespace: platform
annotations:
service.kubernetes.io/topology-mode: Auto
spec:
selector:
app: inventory-service
ports:
- port: 8080
targetPort: 8080
Important caveat: topology-aware routing requires at least 3 ready endpoints per zone to activate. If a service has fewer replicas than zones, Kubernetes falls back to cluster-wide routing. We ensured every high-traffic service had at least 3 replicas per AZ:
# Ensure enough replicas for topology hints to activate
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: payments
spec:
replicas: 9 # 3 per AZ minimum
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: payments-api
After enabling topology-aware routing on our top 8 services, 70% of service-to-service traffic stayed in-zone.
Fix 2: Single-AZ for Non-Critical Batch Workloads
Our data pipeline ran batch jobs across all 3 AZs by default. These jobs didn't need multi-AZ redundancy — they were retryable. We pinned them to a single AZ using a dedicated Karpenter NodePool:
# karpenter-nodepool-batch.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: batch-single-az
spec:
template:
spec:
requirements:
- key: topology.kubernetes.io/zone
operator: In
values: ["us-east-1a"] # Pin to single AZ
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
- key: karpenter.k8s.aws/instance-family
operator: In
values: ["m6i", "m6a", "m7i", "m7a"]
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: batch
limits:
cpu: 64
memory: 256Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: etl-worker
namespace: data-pipeline
spec:
replicas: 4
template:
spec:
nodeSelector:
karpenter.sh/nodepool: batch-single-az
tolerations:
- key: batch
operator: Equal
value: "true"
effect: NoSchedule
Cross-AZ traffic from the data pipeline dropped to near zero.
Fix 3: ElastiCache Read Replicas Per AZ
Our Redis cluster was in us-east-1a. Every cache read from pods in 1b and 1c was cross-AZ. We added read replicas to each AZ:
# modules/elasticache/main.tf
resource "aws_elasticache_replication_group" "redis" {
replication_group_id = "${var.environment}-redis"
description = "Redis cluster with per-AZ read replicas"
node_type = "cache.r6g.large"
num_cache_clusters = 3 # 1 primary + 2 replicas (one per AZ)
automatic_failover_enabled = true
multi_az_enabled = true
subnet_group_name = aws_elasticache_subnet_group.redis.name
security_group_ids = [aws_security_group.redis.id]
at_rest_encryption_enabled = true
transit_encryption_enabled = true
parameter_group_name = "default.redis7"
engine_version = "7.1"
tags = {
Environment = var.environment
Purpose = "cross-az-optimization"
}
}
resource "aws_elasticache_subnet_group" "redis" {
name = "${var.environment}-redis-subnets"
subnet_ids = var.private_subnet_ids # One subnet per AZ
}
Our application's Redis client was already configured with readFrom: REPLICA_PREFERRED, so reads automatically went to the nearest replica. Write traffic (about 15% of total) still goes cross-AZ to the primary, but that's a much smaller volume.
Cross-AZ Summary
Cross-AZ optimization results:
Before: $5,200/mo
After topology-aware routing: -$2,400
After single-AZ batch workloads: -$800
After ElastiCache per-AZ replicas: -$600
──────────────────────────────────────────────
After: $1,400/mo (-73%)
Internet Egress — CloudFront as a Cost Optimizer
AWS charges $0.09/GB for the first 10 TB of internet egress directly from EC2, ALB, or any other service. CloudFront charges $0.085/GB — slightly cheaper per GB — but the real savings come from caching: CloudFront serves cached responses without fetching from your origin, so total bytes transferred from AWS drops significantly.
Our setup: public API behind an ALB + static assets served from application pods. Total internet egress: $4,800/month.
Fix 1: CloudFront in Front of ALB for API Traffic
Not all API traffic is uncacheable. Health checks, configuration endpoints, public data listings, and even some authenticated GET requests with proper Cache-Control headers can be cached at the edge. We put CloudFront in front of our ALB and cached what we could:
# modules/cloudfront-api/main.tf
resource "aws_cloudfront_distribution" "api" {
enabled = true
is_ipv6_enabled = true
comment = "API distribution — cache eligible responses"
price_class = "PriceClass_100" # US, Canada, Europe only
origin {
domain_name = var.alb_dns_name
origin_id = "alb"
custom_origin_config {
http_port = 80
https_port = 443
origin_protocol_policy = "https-only"
origin_ssl_protocols = ["TLSv1.2"]
}
}
default_cache_behavior {
allowed_methods = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
cached_methods = ["GET", "HEAD", "OPTIONS"]
target_origin_id = "alb"
viewer_protocol_policy = "redirect-to-https"
compress = true # Enable automatic gzip/brotli
# Use managed cache policy — CachingOptimized
cache_policy_id = "658327ea-f89d-4fab-a63d-7e88639e58f6"
origin_request_policy_id = aws_cloudfront_origin_request_policy.api.id
}
# Cache static-like API responses longer
ordered_cache_behavior {
path_pattern = "/api/v1/config/*"
allowed_methods = ["GET", "HEAD"]
cached_methods = ["GET", "HEAD"]
target_origin_id = "alb"
viewer_protocol_policy = "redirect-to-https"
compress = true
forwarded_values {
query_string = false
cookies {
forward = "none"
}
}
min_ttl = 60
default_ttl = 300
max_ttl = 3600
}
restrictions {
geo_restriction {
restriction_type = "none"
}
}
viewer_certificate {
acm_certificate_arn = var.acm_certificate_arn
ssl_support_method = "sni-only"
minimum_protocol_version = "TLSv1.2_2021"
}
tags = {
Environment = var.environment
Purpose = "egress-cost-optimization"
}
}
resource "aws_cloudfront_origin_request_policy" "api" {
name = "${var.environment}-api-origin-request"
comment = "Forward necessary headers to ALB"
headers_config {
header_behavior = "whitelist"
headers {
items = ["Host", "Authorization", "Accept", "Accept-Language"]
}
}
cookies_config {
cookie_behavior = "all"
}
query_strings_config {
query_string_behavior = "all"
}
}
On the application side, we added Cache-Control headers to responses that could be cached:
# Example: Flask middleware for cache-control headers
from functools import wraps
from flask import request, make_response
CACHEABLE_PREFIXES = ["/api/v1/config/", "/api/v1/products", "/api/v1/categories"]
def cache_control(f):
@wraps(f)
def decorated(*args, **kwargs):
response = make_response(f(*args, **kwargs))
path = request.path
if request.method == "GET" and any(path.startswith(p) for p in CACHEABLE_PREFIXES):
response.headers["Cache-Control"] = "public, max-age=300, s-maxage=600"
else:
response.headers["Cache-Control"] = "no-store"
return response
return decorated
Fix 2: S3 + CloudFront for Static Assets
We were serving static assets (JS bundles, images, fonts) directly from our application pods via the ALB. Moving them to S3 + CloudFront with Origin Access Control (OAC) eliminated egress from EC2 entirely for those assets:
# modules/cloudfront-static/main.tf
resource "aws_s3_bucket" "static_assets" {
bucket = "${var.environment}-static-assets"
tags = {
Environment = var.environment
Purpose = "static-asset-hosting"
}
}
resource "aws_s3_bucket_public_access_block" "static_assets" {
bucket = aws_s3_bucket.static_assets.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_cloudfront_origin_access_control" "static" {
name = "${var.environment}-static-oac"
origin_access_control_origin_type = "s3"
signing_behavior = "always"
signing_protocol = "sigv4"
}
resource "aws_cloudfront_distribution" "static" {
enabled = true
is_ipv6_enabled = true
default_root_object = "index.html"
comment = "Static assets — S3 origin with OAC"
price_class = "PriceClass_100"
origin {
domain_name = aws_s3_bucket.static_assets.bucket_regional_domain_name
origin_id = "s3-static"
origin_access_control_id = aws_cloudfront_origin_access_control.static.id
}
default_cache_behavior {
allowed_methods = ["GET", "HEAD"]
cached_methods = ["GET", "HEAD"]
target_origin_id = "s3-static"
viewer_protocol_policy = "redirect-to-https"
compress = true
forwarded_values {
query_string = false
cookies {
forward = "none"
}
}
min_ttl = 3600
default_ttl = 86400
max_ttl = 31536000
}
restrictions {
geo_restriction {
restriction_type = "none"
}
}
viewer_certificate {
acm_certificate_arn = var.acm_certificate_arn
ssl_support_method = "sni-only"
minimum_protocol_version = "TLSv1.2_2021"
}
tags = {
Environment = var.environment
Purpose = "static-assets"
}
}
resource "aws_s3_bucket_policy" "static_assets" {
bucket = aws_s3_bucket.static_assets.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "AllowCloudFrontOAC"
Effect = "Allow"
Principal = { Service = "cloudfront.amazonaws.com" }
Action = "s3:GetObject"
Resource = "${aws_s3_bucket.static_assets.arn}/*"
Condition = {
StringEquals = {
"AWS:SourceArn" = aws_cloudfront_distribution.static.arn
}
}
}
]
})
}
Fix 3: Compression
This one's embarrassingly simple. Our ALB wasn't compressing API responses, and our application wasn't setting Content-Encoding headers. JSON API responses compress at roughly 85% ratio with gzip — so a 100 KB response becomes 15 KB on the wire.
CloudFront handles compression automatically when compress = true is set (which we did above). For the ALB itself, we ensured our application server was compressing responses for direct clients:
# nginx.conf snippet for API pods behind ALB
http {
gzip on;
gzip_types application/json text/plain application/xml text/css
application/javascript text/xml application/xhtml+xml;
gzip_min_length 256;
gzip_comp_level 6;
gzip_vary on;
gzip_proxied any;
}
Internet Egress Summary
Internet egress optimization results:
Before: $4,800/mo
After CloudFront for API: -$1,600
After S3+CloudFront for static: -$900
After compression: -$700
──────────────────────────────────────────────
After: $1,600/mo (-67%)
S3 Data Transfer
S3 data transfer costs are tricky because they overlap with other categories. S3-to-internet is the same pricing as EC2 egress ($0.09/GB). S3-to-EC2 in the same region is free — unless it goes through NAT Gateway (which we already fixed). The remaining S3-specific costs came from three sources.
The Problem
Our analytics pipeline was reading from an S3 bucket in us-west-2, but the compute (EMR cluster) was in us-east-1. Cross-region S3 transfer is $0.02/GB. The pipeline processed about 60 TB/month, costing $1,200/month in transfer alone. On top of that, daily CSV exports to a partner were 20 TB/month at $0.09/GB ($1,800/month).
Fix 1: Co-Locate S3 Buckets with Compute
We replicated the analytics data bucket to us-east-1 using S3 Cross-Region Replication. One-time sync cost, then the EMR cluster reads locally:
# S3 replication for analytics data
resource "aws_s3_bucket" "analytics_east" {
bucket = "${var.environment}-analytics-data-use1"
tags = {
Environment = var.environment
Purpose = "analytics-co-location"
}
}
resource "aws_s3_bucket_versioning" "analytics_east" {
bucket = aws_s3_bucket.analytics_east.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_replication_configuration" "analytics" {
bucket = var.analytics_source_bucket_id # us-west-2 bucket
role = aws_iam_role.replication.arn
rule {
id = "replicate-analytics"
status = "Enabled"
destination {
bucket = aws_s3_bucket.analytics_east.arn
storage_class = "STANDARD_IA" # Cheaper for read-heavy analytics
}
}
depends_on = [aws_s3_bucket_versioning.analytics_east]
}
After replication stabilized, we pointed EMR at the local bucket. Cross-region transfer dropped to just the replication traffic (write-once), instead of read-many through compute.
Result: -$900/month.
Fix 2: S3 Intelligent-Tiering for Infrequently Accessed Data
This doesn't reduce transfer directly, but it reduces the effective cost of storing data that gets accessed irregularly. For our analytics bucket, 80% of queries hit the last 30 days of data. Older partitions were rarely touched but stored in STANDARD:
resource "aws_s3_bucket_lifecycle_configuration" "analytics" {
bucket = aws_s3_bucket.analytics_east.id
rule {
id = "intelligent-tiering"
status = "Enabled"
transition {
days = 30
storage_class = "INTELLIGENT_TIERING"
}
}
}
resource "aws_s3_bucket_intelligent_tiering_configuration" "analytics" {
bucket = aws_s3_bucket.analytics_east.id
name = "analytics-tiering"
tiering {
access_tier = "ARCHIVE_ACCESS"
days = 90
}
tiering {
access_tier = "DEEP_ARCHIVE_ACCESS"
days = 180
}
}
Fix 3: S3 Select and Athena to Reduce Bytes Transferred
Our partner CSV exports were reading entire Parquet files from S3, converting to CSV in an EC2 instance, and uploading the results. We replaced the EC2-based pipeline with Athena + S3 Select, which pushes filtering down to S3 and only returns the columns and rows the partner needed:
-- Before: read entire partitions (20 TB/month)
-- SELECT * FROM analytics.events WHERE date >= '2026-09-01'
-- After: Athena query with column pruning and predicate pushdown
-- Only transfers the selected columns and matching rows
CREATE TABLE partner_export
WITH (
format = 'TEXTFILE',
field_delimiter = ',',
external_location = 's3://exports-bucket/partner/2026-10/'
) AS
SELECT
event_id,
event_type,
user_region,
event_timestamp,
revenue
FROM analytics.events
WHERE date >= '2026-09-01'
AND event_type IN ('purchase', 'refund', 'subscription')
AND user_region = 'US';
This reduced the bytes scanned and transferred from 20 TB/month to about 3 TB/month — Athena reads only the necessary Parquet row groups and columns.
S3 Transfer Summary
S3 transfer optimization results:
Before: $3,000/mo
After co-locating buckets: -$900
After Athena/S3 Select: -$900
──────────────────────────────────────────────
After: $1,200/mo (-60%)
Monitoring Data Transfer Costs
After cutting $14,100/month, we needed to make sure costs didn't creep back up. We built a monitoring layer that extends the EKS cost monitoring from post #9.
CloudWatch Dashboard for Data Transfer
# modules/data-transfer-monitoring/dashboard.tf
resource "aws_cloudwatch_dashboard" "data_transfer" {
dashboard_name = "${var.environment}-data-transfer-costs"
dashboard_body = jsonencode({
widgets = [
{
type = "metric"
x = 0
y = 0
width = 12
height = 6
properties = {
title = "NAT Gateway Bytes Processed (Daily)"
view = "timeSeries"
region = var.region
metrics = [
for nat_id in var.nat_gateway_ids :
["AWS/NATGateway", "BytesOutToDestination", "NatGatewayId", nat_id,
{ stat = "Sum", period = 86400 }]
]
}
},
{
type = "metric"
x = 12
y = 0
width = 12
height = 6
properties = {
title = "Cross-AZ Data Transfer (Daily)"
view = "timeSeries"
region = var.region
metrics = [
["AWS/EC2", "NetworkOut", { stat = "Sum", period = 86400 }]
]
}
},
{
type = "metric"
x = 0
y = 6
width = 24
height = 6
properties = {
title = "VPC Endpoint Bytes Processed (Daily)"
view = "timeSeries"
region = var.region
metrics = [
for endpoint_id in var.vpc_endpoint_ids :
["AWS/PrivateLinkEndpoints", "BytesProcessed",
"VPC Endpoint Id", endpoint_id,
{ stat = "Sum", period = 86400 }]
]
}
}
]
})
}
variable "nat_gateway_ids" {
type = list(string)
description = "NAT Gateway IDs to monitor"
}
variable "vpc_endpoint_ids" {
type = list(string)
description = "VPC Endpoint IDs to monitor"
}
variable "region" {
type = string
default = "us-east-1"
}
variable "environment" {
type = string
}
Daily NAT Gateway Cost Alarm
Alert when NAT Gateway daily processing cost exceeds a threshold:
# modules/data-transfer-monitoring/alarms.tf
resource "aws_cloudwatch_metric_alarm" "nat_daily_cost" {
for_each = toset(var.nat_gateway_ids)
alarm_name = "${var.environment}-nat-daily-cost-${each.key}"
alarm_description = "NAT Gateway daily bytes exceed threshold (est. >$${var.nat_daily_threshold}/day)"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 1
period = 86400
statistic = "Sum"
# Threshold in bytes: ($threshold / $0.045) * 1GB
# e.g., $100/day = 2,222 GB = 2,222,000,000,000 bytes
threshold = (var.nat_daily_threshold / 0.045) * 1000000000
namespace = "AWS/NATGateway"
metric_name = "BytesOutToDestination"
dimensions = {
NatGatewayId = each.key
}
alarm_actions = [var.sns_topic_arn]
ok_actions = [var.sns_topic_arn]
tags = {
Environment = var.environment
AlertType = "data-transfer-cost"
}
}
variable "nat_daily_threshold" {
type = number
default = 150
description = "Daily NAT Gateway cost threshold in USD"
}
variable "sns_topic_arn" {
type = string
description = "SNS topic ARN for alerts"
}
Weekly Data Transfer Cost Report
This script extends the weekly cost report from post #9 to include data transfer categories:
#!/usr/bin/env python3
"""
Weekly data transfer cost report — supplements the EKS cost report.
Pulls from Cost Explorer API and posts to Slack.
"""
import boto3
import json
import os
from datetime import datetime, timedelta
from urllib import request, error
SLACK_WEBHOOK_URL = os.environ["SLACK_WEBHOOK_URL"]
ENVIRONMENT = os.environ.get("ENVIRONMENT", "production")
def get_data_transfer_costs(start_date, end_date):
"""Pull data transfer costs from Cost Explorer by usage type."""
ce = boto3.client("ce")
response = ce.get_cost_and_usage(
TimePeriod={"Start": start_date, "End": end_date},
Granularity="MONTHLY",
Filter={
"And": [
{
"Dimensions": {
"Key": "SERVICE",
"Values": ["Amazon Elastic Compute Cloud - Compute",
"Amazon Simple Storage Service"],
}
},
{
"Or": [
{
"Dimensions": {
"Key": "USAGE_TYPE",
"Values": [],
"MatchOptions": ["CONTAINS"],
},
"CostCategories": {},
}
]
},
]
},
Metrics=["UnblendedCost"],
GroupBy=[{"Type": "DIMENSION", "Key": "USAGE_TYPE"}],
)
categories = {
"NAT Gateway": 0,
"Cross-AZ": 0,
"Internet Egress": 0,
"S3 Transfer": 0,
"Other": 0,
}
for group in response.get("ResultsByTime", [{}])[0].get("Groups", []):
usage_type = group["Keys"][0]
cost = float(group["Metrics"]["UnblendedCost"]["Amount"])
if "NatGateway" in usage_type:
categories["NAT Gateway"] += cost
elif "Regional" in usage_type or "IntraRegion" in usage_type:
categories["Cross-AZ"] += cost
elif "DataTransfer-Out" in usage_type:
categories["Internet Egress"] += cost
elif "S3" in usage_type and "DataTransfer" in usage_type:
categories["S3 Transfer"] += cost
else:
categories["Other"] += cost
return categories
def build_and_post_report():
"""Build weekly comparison and post to Slack."""
today = datetime.now()
this_week_start = (today - timedelta(days=7)).strftime("%Y-%m-%d")
this_week_end = today.strftime("%Y-%m-%d")
last_week_start = (today - timedelta(days=14)).strftime("%Y-%m-%d")
last_week_end = this_week_start
current = get_data_transfer_costs(this_week_start, this_week_end)
previous = get_data_transfer_costs(last_week_start, last_week_end)
lines = ["```
"]
lines.append(f"{'Category':<20} {'This Week':>10} {'Last Week':>10} {'Change':>8}")
lines.append("─" * 52)
total_curr = 0
total_prev = 0
for cat in ["NAT Gateway", "Cross-AZ", "Internet Egress", "S3 Transfer"]:
curr = current.get(cat, 0)
prev = previous.get(cat, 0)
total_curr += curr
total_prev += prev
change = ((curr - prev) / prev * 100) if prev > 0 else 0
lines.append(f"{cat:<20} ${curr:>8,.0f} ${prev:>8,.0f} {change:>+7.1f}%")
lines.append("─" * 52)
total_change = ((total_curr - total_prev) / total_prev * 100) if total_prev > 0 else 0
lines.append(f"{'TOTAL':<20} ${total_curr:>8,.0f} ${total_prev:>8,.0f} {total_change:>+7.1f}%")
lines.append("
```")
trend = ":large_green_circle:" if total_change <= 0 else ":red_circle:"
payload = {
"channel": "#cloud-costs",
"username": f"Data Transfer Report ({ENVIRONMENT})",
"icon_emoji": ":electric_plug:",
"blocks": [
{
"type": "header",
"text": {
"type": "plain_text",
"text": f":electric_plug: Weekly Data Transfer Report — {today.strftime('%B %d, %Y')}",
},
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": (
f"*Weekly Data Transfer:* ${total_curr:,.0f} "
f"({total_change:+.1f}% WoW) {trend}"
),
},
},
{
"type": "section",
"text": {"type": "mrkdwn", "text": "\n".join(lines)},
},
],
}
req = request.Request(
SLACK_WEBHOOK_URL,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
request.urlopen(req)
print("Data transfer report posted to #cloud-costs")
except error.HTTPError as e:
print(f"Slack post failed: {e.code} {e.read().decode()}")
raise
if __name__ == "__main__":
build_and_post_report()
Deploy alongside the EKS cost report CronJob from post #9 — same namespace, same Slack channel, different schedule (we run this on Wednesdays so it doesn't overlap with the Monday EKS report).
Results and ROI
Full Before/After Breakdown
┌──────────────────────────────────────────────────────────────────┐
│ COMPLETE DATA TRANSFER SAVINGS │
├──────────────────────────┬──────────┬──────────┬─────────────────┤
│ Category │ Before │ After │ Monthly Savings │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ NAT Gateway │ │ │ │
│ S3 gateway endpoint │ │ │ $2,800 │
│ Interface endpoints │ │ │ $1,900 │
│ ECR pull-through cache │ │ │ $600 │
│ Subtotal │ $9,400 │ $4,100 │ $5,300 │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ Cross-AZ │ │ │ │
│ Topology-aware routing │ │ │ $2,400 │
│ Single-AZ batch │ │ │ $800 │
│ ElastiCache replicas │ │ │ $600 │
│ Subtotal │ $5,200 │ $1,400 │ $3,800 │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ Internet Egress │ │ │ │
│ CloudFront for API │ │ │ $1,600 │
│ S3+CF for static │ │ │ $900 │
│ Compression │ │ │ $700 │
│ Subtotal │ $4,800 │ $1,600 │ $3,200 │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ S3 Transfer │ │ │ │
│ Co-locate buckets │ │ │ $900 │
│ Athena/S3 Select │ │ │ $900 │
│ Subtotal │ $3,000 │ $1,200 │ $1,800 │
├──────────────────────────┼──────────┼──────────┼─────────────────┤
│ TOTAL │ $22,400 │ $8,300 │ $14,100 │
└──────────────────────────┴──────────┴──────────┴─────────────────┘
Implementation Cost
Infrastructure cost of changes:
VPC interface endpoints (5 × 3 AZs × $7.20) $108/mo
ElastiCache additional replicas (2 × r6g.large) $0/mo (already budgeted)
CloudFront (included in AWS free tier initially) $52/mo
S3 replication (one-time + ongoing writes) $20/mo
──────────────────────────────────────────────────────
Total monthly infrastructure cost $180/mo
Engineer time: 2 engineers × 2 weeks ~$12,000 (one-time)
──────────────────────────────────────────────────────
Payback period: $12,000 / $13,920 = <1 month
Annual ROI: $167,040 / $12,000 = 1,392%
Combined Series Savings (Posts #1–#10)
┌──────────────────────────────────────────────────────────────────┐
│ AWS COST OPTIMIZATION SERIES — TOTAL IMPACT │
├─────┬─────────────────────────────────┬──────────────────────────┤
│ # │ Focus Area │ Monthly Savings │
├─────┼─────────────────────────────────┼──────────────────────────┤
│ 1 │ RDS → Aurora Serverless v2 │ $1,800 │
│ 2 │ Auto Scaling Groups │ $2,100 │
│ 3 │ Blue-Green Deployments │ $1,400 │
│ 4 │ Multi-Region Architecture │ $3,200 │
│ 5 │ Multi-Tenant Infrastructure │ $4,800 │
│ 6 │ Container Cold Starts │ $600 (perf, not cost) │
│ 7 │ Kubernetes/EKS Optimization │ $10,020 │
│ 8 │ Serverless vs Containers │ $2,400 │
│ 9 │ EKS Cost Monitoring Stack │ $2,400 │
│ 10 │ Data Transfer Costs │ $14,100 │
├─────┼─────────────────────────────────┼──────────────────────────┤
│ │ TOTAL │ $42,820/mo ($513,840/yr) │
└─────┴─────────────────────────────────┴──────────────────────────┘
Lessons Learned
1. VPC Gateway Endpoints are free money. If you're running workloads in private subnets and don't have S3 and DynamoDB gateway endpoints, you're paying NAT Gateway processing fees for no reason. Deploy them today — it takes 5 minutes in Terraform and saves immediately.
2. "EC2-Other" is not a small line item. For any account running workloads in private subnets with NAT Gateways, data transfer can easily be 20-40% of your total EC2 bill. Most teams never drill into it because Cost Explorer's default view hides it. Make a habit of filtering by usage type, not just service.
3. Cross-AZ costs are architecture decisions, not infrastructure ones. You can't fix cross-AZ costs with Terraform alone. Topology-aware routing, single-AZ batch pinning, and read replica placement are application-level decisions. They require coordination between platform and application teams.
4. CloudFront is a cost optimizer, not just a CDN. We deployed CloudFront primarily for cost reduction, not latency. The per-GB price is lower than direct egress, and caching multiplies the savings. Even if your API is "dynamic," there are usually cacheable endpoints you haven't considered.
Common Mistakes
- Deploying NAT Gateways in all AZs "for redundancy" without checking traffic volume. If most traffic routes through VPC endpoints, you may only need 1-2 NAT Gateways for the remaining internet-bound traffic.
- Ignoring cross-AZ costs for databases. Every read from a pod in AZ-b to RDS in AZ-a costs $0.02/GB round trip. Read replicas in each AZ aren't just for availability — they're for cost.
- Not enabling compression. JSON compresses at 85%+ ratios. If your API returns uncompressed JSON over the internet, you're paying 6-7x more in egress than you need to.
- Treating data transfer as fixed cost. Unlike compute (which you can right-size) or storage (which you can tier), teams assume network costs are "just what things cost." They're not — they're the most architecture-dependent cost category in AWS.
Action Items Checklist
Week 1: Quick Wins (NAT Gateway)
- [ ] Deploy S3 and DynamoDB VPC gateway endpoints (free, immediate savings)
- [ ] Run CUR/Athena query to find your actual data transfer costs
- [ ] Identify top NAT Gateway traffic destinations (VPC Flow Logs)
- [ ] Calculate interface endpoint break-even for each AWS service
Week 2: VPC Endpoints + ECR Cache
- [ ] Deploy interface endpoints for services above break-even threshold
- [ ] Set up ECR pull-through cache rules
- [ ] Deploy mutating webhook for image reference rewriting
- [ ] Verify NAT Gateway traffic reduction in CloudWatch
Week 3: Cross-AZ + Egress
- [ ] Enable topology-aware routing on high-traffic services
- [ ] Pin non-critical batch workloads to single AZ
- [ ] Deploy ElastiCache read replicas in each AZ
- [ ] Put ALB behind CloudFront with compression enabled
- [ ] Move static assets to S3 + CloudFront with OAC
Week 4: S3 + Monitoring
- [ ] Co-locate S3 buckets with compute (set up replication if needed)
- [ ] Replace full-scan exports with Athena/S3 Select queries
- [ ] Deploy CloudWatch dashboard for data transfer metrics
- [ ] Deploy NAT Gateway daily cost alarm
- [ ] Deploy weekly data transfer cost report
Ongoing
- [ ] Review data transfer costs monthly in architecture review
- [ ] Re-evaluate interface endpoint break-even quarterly (as traffic patterns change)
- [ ] Check for new AWS services that should use VPC endpoints
- [ ] Update CloudFront cache policies as API evolves
This is Part 10 of the AWS Cost Optimization Series. Part 9 covers the EKS cost monitoring stack. Part 7 covers Kubernetes compute optimization.
Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.