DEV Community: Sanket Patharkar

Real-World AWS Cost Optimization Strategies — A Technical Deep Dive

Sanket Patharkar — Fri, 25 Sep 2026 18:04:33 +0000

In a cloud-first world, keeping costs under control without losing performance, scalability or reliability matters for any business that wants to last. AWS gives you a lot of tools for measuring, managing and optimizing spend. The hard part is wiring them into an engineering system instead of doing a clean-up once a quarter.

I'm a Senior Cloud Operations Engineer with 6+ years across AWS, DevOps, CI/CD and enterprise cloud architecture, much of it in the automotive systems domain. The approaches below mix automation, governance and real-time monitoring, and each comes with the CLI commands, JSON/YAML configs and gotchas you need to use it in production.

Who this is for: Cloud Architects, DevOps/SRE engineers and FinOps practitioners who want measurable, automated, production-ready cost control, not just a checklist.

About the prices: Dollar amounts below are illustrative us-east-1 list prices at the time of writing. They vary by Region and change over time. Discount figures such as “up to 72%” or “up to 90%” are what AWS advertises as a maximum, not a saving you should expect on every workload. Check the current AWS pricing pages before you build a business case on any of them.

TL;DR — The Eight Levers

#	Strategy	Key AWS services	Where the savings come from
1	Measure before you optimize	Data Exports (CUR 2.0), Athena, Cost Explorer, Budgets, Cost Anomaly Detection	Knowing who spends what, and finding spikes within hours instead of at month-end
2	Right-size compute	Compute Optimizer, CloudWatch Agent, Auto Scaling, Graviton	Removing idle headroom; ~20% lower price per instance on Graviton
3	Commitments + Spot	Savings Plans, Reserved Instances, EC2 Fleet, Karpenter	AWS advertises up to 72% (Savings Plans) and up to 90% (Spot) off On-Demand
4	Storage tiering	S3 Lifecycle, Intelligent-Tiering, Storage Lens, EBS gp3, EFS IA	About 20% on gp2→gp3 in many Regions; AWS advertises up to ~95% for cold archive tiers
5	Network cost engineering	VPC Gateway/Interface Endpoints, CloudFront, AZ-aware routing	Avoiding NAT processing and cross-AZ charges
6	Non-prod shutdown	EventBridge Scheduler, Lambda, Instance Scheduler on AWS	60–70% of non-prod compute hours
7	Database optimization	Aurora Serverless v2, Aurora I/O-Optimized, DynamoDB modes, RDS on Graviton	Paying for actual load instead of peak provisioning
8	Continuous governance	Organizations SCPs, Tag Policies, AWS Config, Budgets Actions	Stopping expensive mistakes before they reach the bill

The Big Picture — A Closed FinOps Loop

Cost optimization isn't a project. It's a control loop: measure, detect, act, govern, then measure again. Every strategy in this article plugs into one of those four stages.

Figure: AWS cost optimization reference architecture — Measure, Detect & Alert, Act/Automate, and Govern.

Measure: Billing and usage data (CUR 2.0) lands in S3, is queried with Athena and visualized in QuickSight/Cost Explorer. Tags turn line items into team-level accountability.
Detect & Alert: Anomaly Detection, Budgets, Compute Optimizer and Cost Optimization Hub produce signals. SNS sends them to Slack/Teams/email.
Act / Automate: Schedulers, Lambda remediations, SSM runbooks and Karpenter make the change, ideally without a human opening the console.
Govern: SCPs, Tag Policies and Config rules stop the waste from coming back.

Production note: The examples below are reference implementations. Validate pricing, service availability, quotas, IAM permissions, and regional behavior against your AWS environment before applying them to production.

1. Measure Before You Optimize

You can't optimize what you can't see. Treat cost like any other telemetry signal: granular, tagged and close to real time.

1.1 Set up CUR 2.0 (Data Exports) + Athena

Cost Explorer is great for exploring, but a Cost and Usage Report (CUR 2.0) export, delivered through Billing and Cost Management → Data Exports, is the practical dataset for line-item analysis. What it contains depends on the export you configure. Hourly granularity and resource IDs are settings you turn on. They are not a promise that every charge has a resource identifier: some fees, credits, and tax lines do not. Treat the export as the working record for the charges it is configured to include, and confirm the columns before you build chargeback on it.

Recommended export settings:

Export type: Standard data export → CUR 2.0
Include resource IDs: ✅ (you can't chase waste without them)
Time granularity: Hourly
Format: Parquet, overwrite, delivered to a dedicated S3 bucket
Use the Athena/Glue integration (or a Glue crawler) to create the table

Query 1: top spend by service and environment

-- Partition/column names depend on how your Glue table was created
SELECT
  line_item_product_code                    AS service,
  resource_tags['user_environment']         AS environment,
  ROUND(SUM(line_item_unblended_cost), 2)   AS cost_usd
FROM cur2
WHERE billing_period = '2026-08'
GROUP BY 1, 2
ORDER BY cost_usd DESC
LIMIT 20;

Query 2: untagged spend (your accountability gap)

SELECT
  line_item_product_code AS service,
  ROUND(SUM(line_item_unblended_cost), 2) AS untagged_cost
FROM cur2
WHERE billing_period = '2026-08'
  AND line_item_line_item_type = 'Usage'
  AND (resource_tags['user_owner'] IS NULL OR resource_tags['user_owner'] = '')
GROUP BY 1
ORDER BY 2 DESC;

Query 3: the silent killers, NAT processing and cross-AZ transfer

SELECT
  line_item_usage_type,
  line_item_resource_id,
  ROUND(SUM(line_item_usage_amount), 0)  AS gb,
  ROUND(SUM(line_item_unblended_cost), 2) AS cost_usd
FROM cur2
WHERE billing_period = '2026-08'
  AND (line_item_usage_type LIKE '%NatGateway-Bytes%'
       OR line_item_usage_type LIKE '%DataTransfer-Regional-Bytes%')
GROUP BY 1, 2
ORDER BY cost_usd DESC
LIMIT 25;

💡 Insight: unblended vs amortized. If you've bought Savings Plans or RIs, unblended cost puts all the upfront or recurring fees on one line and makes teams look "free". For chargeback, use amortized cost (savings_plan_savings_plan_effective_cost and reservation_effective_cost in CUR), so every team carries its fair share of the commitment.

1.2 Activate cost allocation tags

Tags only show up in billing data after you activate them in the payer account:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status TagKey=Environment,Status=Active TagKey=Project,Status=Active TagKey=Owner,Status=Active

Required baseline: Environment, Project, Owner (plus CostCenter if finance charges back). Section 8 covers how to enforce them.

1.3 Cost Explorer API: query programmatically, but know it costs money

aws ce get-cost-and-usage \
  --time-period Start=2026-08-01,End=2026-09-01 \
  --granularity DAILY \
  --metrics UnblendedCost \
  --group-by Type=TAG,Key=Environment

⚠️ Gotcha: AWS currently charges $0.01 per paginated Cost Explorer API request against the primary billing view. A custom billing view is charged at $0.01 per source in that view. The Cost Explorer console is free. A dashboard polling the API every minute runs up a real bill. Hourly granularity is a separate opt-in charge and only covers the last 14 days. Confirm the current rate on the Cost Explorer pricing page before you automate it. For heavy analysis, query CUR in Athena instead.

1.4 Budgets with forecast alerts

Alert on forecasted spend, not just actual. By the time actual spend crosses 100%, the money is already gone.

budget.json

{
  "BudgetName": "platform-monthly",
  "BudgetLimit": { "Amount": "10000", "Unit": "USD" },
  "TimeUnit": "MONTHLY",
  "BudgetType": "COST",
  "CostFilters": { "TagKeyValue": ["user:Project$platform"] }
}

notifications.json

[
  {
    "Notification": { "NotificationType": "FORECASTED", "ComparisonOperator": "GREATER_THAN", "Threshold": 100, "ThresholdType": "PERCENTAGE" },
    "Subscribers": [{ "SubscriptionType": "SNS", "Address": "arn:aws:sns:us-east-1:111122223333:finops-alerts" }]
  },
  {
    "Notification": { "NotificationType": "ACTUAL", "ComparisonOperator": "GREATER_THAN", "Threshold": 80, "ThresholdType": "PERCENTAGE" },
    "Subscribers": [{ "SubscriptionType": "SNS", "Address": "arn:aws:sns:us-east-1:111122223333:finops-alerts" }]
  }
]

aws budgets create-budget \
  --account-id 111122223333 \
  --budget file://budget.json \
  --notifications-with-subscribers file://notifications.json

Send the SNS topic to Slack or Teams through Amazon Q Developer in chat applications (formerly AWS Chatbot).

1.5 Cost Anomaly Detection

A budget catches how much. Anomaly Detection catches something changed, like a runaway Lambda loop or a debug log level left on in production.

aws ce create-anomaly-monitor --anomaly-monitor '{
  "MonitorName": "per-service",
  "MonitorType": "DIMENSIONAL",
  "MonitorDimension": "SERVICE"
}'

aws ce create-anomaly-subscription --anomaly-subscription '{
  "SubscriptionName": "anomalies-over-100-usd",
  "MonitorArnList": ["arn:aws:ce::111122223333:anomalymonitor/abcd-1234"],
  "Subscribers": [{ "Type": "SNS", "Address": "arn:aws:sns:us-east-1:111122223333:finops-alerts" }],
  "Frequency": "IMMEDIATE",
  "ThresholdExpression": {
    "Dimensions": { "Key": "ANOMALY_TOTAL_IMPACT_ABSOLUTE", "Values": ["100"], "MatchOptions": ["GREATER_THAN_OR_EQUAL"] }
  }
}'

💡 Insight: track unit cost, not just total cost. A rising bill can be healthy growth. Divide spend by a business metric (cost per 1,000 API requests, per tenant, per CI build) and put it on the same dashboard as your latency SLOs. When unit cost goes up, that's a regression.

2. Right-Sizing Compute Resources

Right-sizing is more than CPU. You also have to look at memory, network bandwidth, EBS throughput and CPU credit behaviour against real performance baselines.

2.1 Feed Compute Optimizer the metric it can't see: memory

By default, EC2 doesn't publish memory metrics. Without them, Compute Optimizer can't safely recommend moving to a smaller or memory-optimized family. Install the CloudWatch Agent:

{
  "metrics": {
    "append_dimensions": { "InstanceId": "${aws:InstanceId}" },
    "metrics_collected": {
      "cpu":  { "measurement": ["cpu_usage_active"], "totalcpu": true, "resources": ["*"] },
      "mem":  { "measurement": ["mem_used_percent"] },
      "disk": { "measurement": ["used_percent"], "resources": ["/"] }
    }
  }
}

Compute Optimizer picks up mem_used_percent from the CWAgent namespace automatically. Then pull the over-provisioned list:

aws compute-optimizer get-ec2-instance-recommendations \
  --filters name=Finding,values=Overprovisioned \
  --query 'instanceRecommendations[].{id:instanceArn,current:currentInstanceType,best:recommendationOptions[0].instanceType,risk:recommendationOptions[0].performanceRisk}' \
  --output table

💡 Insight: size on p95/p99, not averages. A 12% average CPU can hide a 90% p99 spike during batch windows. Per-core metrics (resources: ["*"]) also show single-threaded apps pinning one core while the others sit idle. Those need a faster core, not more cores. Turn on enhanced infrastructure metrics in Compute Optimizer for a 3-month lookback on seasonal workloads.

⚠️ Gotcha: burstable (T-family) instances. t3/t4g default to unlimited mode. A T instance that burns through its CPU credits all day gets billed for surplus credits and can cost more than an m-family instance. Watch CPUSurplusCreditsCharged.

2.2 Graviton migration checklist

Graviton (arm64) instances usually cost ~20% less than comparable x86 instances, and AWS quotes up to 40% better price-performance.

Check compatibility: Interpreted and JVM languages (Java, Python, Node.js, Go, .NET 6+) usually just work. Check native dependencies and agents.
Build multi-arch images:

   docker buildx build --platform linux/amd64,linux/arm64 \
     -t 111122223333.dkr.ecr.us-east-1.amazonaws.com/api:1.4.0 --push .

Canary: Put a Graviton ASG or node pool behind the same ALB target group and shift traffic gradually while watching latency and error rates.
Go beyond EC2: Lambda on arm64 is ~20% cheaper per GB-second. RDS/Aurora (db.r7g, db.m7g), ElastiCache and OpenSearch all offer Graviton classes. These are often the easiest wins because there's no code to change.

2.3 Scale on the right signal

aws autoscaling put-scaling-policy \
  --auto-scaling-group-name web-asg \
  --policy-name cpu-target-50 \
  --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": { "PredefinedMetricType": "ASGAverageCPUUtilization" },
    "TargetValue": 50.0
  }'

For web tiers, ALBRequestCountPerTarget tracks real demand more closely than CPU.
Add predictive scaling for workloads with a daily pattern so capacity arrives before the morning ramp instead of sitting there all night.
Use attribute-based instance selection (InstanceRequirements) so the ASG isn't tied to one instance type. That matters again in section 3.

3. Savings Plans & Spot Fleet Engineering

The biggest single lever on compute is how you buy it. The rule is simple: layer your capacity.

Figure: Compute purchasing strategy — Savings Plans for the baseline, Spot for variable demand, On-Demand for short peaks.

3.1 Savings Plans: commit to the floor, not the average

Plan	Max discount	Flexibility	Best for
Compute Savings Plans	up to 66%	Any family, size, Region, OS; also covers Fargate & Lambda	Most organizations; containers and serverless
EC2 Instance Savings Plans	up to 72%	Locked to a family in a Region (size/OS flexible)	Very stable, known fleets
Reserved Instances (RDS, ElastiCache, OpenSearch, Redshift)	varies	Per service	Databases and data stores

How to size the commitment:

Pull the recommendation from AWS as a starting point:

   aws ce get-savings-plans-purchase-recommendation \
     --savings-plans-type COMPUTE_SP \
     --term-in-years ONE_YEAR \
     --payment-option NO_UPFRONT \
     --lookback-period-in-days THIRTY_DAYS

Commit to roughly 90–95% of your lowest sustained hourly usage, after right-sizing. Committing first and right-sizing afterwards leaves you paying for a commitment you no longer use.
Ladder your purchases, for example a small tranche every quarter. Commitments then expire on a rolling basis, and you're never locked into one big bet.
Track utilization (are you using what you bought?) and coverage (how much of your eligible spend is covered?). These are Cost Explorer APIs, ce:GetSavingsPlansUtilization and ce:GetSavingsPlansCoverage:

   aws ce get-savings-plans-utilization --time-period Start=2026-08-01,End=2026-09-01
   aws ce get-savings-plans-coverage   --time-period Start=2026-08-01,End=2026-09-01 --granularity MONTHLY

🎯 Targets I use: utilization > 95%, coverage of eligible compute 70–85%. Coverage near 100% usually means you over-committed and have no room left for Spot.

3.2 Spot: engineering for interruption

AWS advertises Spot discounts of up to 90% versus On-Demand. In exchange, the instance can be reclaimed with a 2-minute notice. Three things make it production-safe.

a) Diversify and use price-capacity-optimized. It's the recommended allocation strategy. It picks the deepest Spot pools and weighs price, so you get fewer interruptions than with lowest-price and better prices than with pure capacity-optimized.

mixed-instances.json (ASG):

{
  "LaunchTemplate": {
    "LaunchTemplateSpecification": { "LaunchTemplateName": "web-lt", "Version": "$Latest" },
    "Overrides": [
      {
        "InstanceRequirements": {
          "VCpuCount": { "Min": 2, "Max": 8 },
          "MemoryMiB": { "Min": 4096 },
          "CpuManufacturers": ["intel", "amd"],
          "InstanceGenerations": ["current"]
        }
      }
    ]
  },
  "InstancesDistribution": {
    "OnDemandBaseCapacity": 2,
    "OnDemandPercentageAboveBaseCapacity": 20,
    "SpotAllocationStrategy": "price-capacity-optimized"
  }
}

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name web-asg \
  --mixed-instances-policy file://mixed-instances.json \
  --min-size 2 --max-size 40 \
  --vpc-zone-identifier "subnet-aaa,subnet-bbb,subnet-ccc" \
  --capacity-rebalance

--capacity-rebalance launches a replacement as soon as AWS sends a rebalance recommendation, which often comes before the 2-minute notice.

b) Karpenter for EKS. Karpenter provisions nodes directly from pod requirements, prefers Spot when you allow it, and consolidates under-utilized nodes all the time.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64", "amd64"]       # multi-arch images required
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
  limits:
    cpu: "1000"

Point Karpenter at an SQS interruption queue (the settings.interruptionQueue Helm value) so it drains nodes gracefully on Spot interruption and rebalance events. If you run self-managed node groups without Karpenter, use the AWS Node Termination Handler.

c) Graceful shutdown on plain EC2. Poll instance metadata (IMDSv2) and drain in time:

#!/bin/bash
# /usr/local/bin/spot-watch.sh — run as a systemd service
while true; do
  TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
    -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
  CODE=$(curl -s -o /dev/null -w "%{http_code}" \
    -H "X-aws-ec2-metadata-token: $TOKEN" \
    http://169.254.169.254/latest/meta-data/spot/instance-action)
  if [ "$CODE" -eq 200 ]; then
    logger "Spot interruption notice received - draining"
    systemctl stop myapp        # finish in-flight work, deregister, flush buffers
    break
  fi
  sleep 5
done

💡 Insight: what should not run on Spot. Single-instance stateful services, long jobs without checkpoints, and anything with a license tied to a host. Everything else, including CI runners, batch/ETL, stateless APIs behind a load balancer and EKS workers with PodDisruptionBudgets, is a candidate.

4. Storage Cost Optimization

Storage costs grow quietly because nothing ever gets deleted.

4.1 S3: pick the class by access pattern

Storage class	~$/GB-month	Min. duration	Retrieval	Use for
S3 Standard	0.023	—	instant	Hot data
S3 Intelligent-Tiering	0.023 → auto-tiered	—	instant (frequent/IA/archive-instant tiers)	Unknown or changing access patterns
S3 Standard-IA	0.0125	30 days	instant, per-GB fee	Monthly-access data
S3 Glacier Instant Retrieval	0.004	90 days	milliseconds	Quarterly-access data, backups you might restore
S3 Glacier Flexible Retrieval	0.0036	90 days	minutes–hours	Archives
S3 Glacier Deep Archive	0.00099	180 days	~12–48 hours	Compliance retention

A production-grade lifecycle policy. Note that Filter replaces the legacy top-level Prefix, and there's a separate "hygiene" rule:

{
  "Rules": [
    {
      "ID": "logs-tiering",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 90,  "StorageClass": "GLACIER_IR" },
        { "Days": 180, "StorageClass": "DEEP_ARCHIVE" }
      ],
      "Expiration": { "Days": 730 }
    },
    {
      "ID": "bucket-hygiene",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 30, "NewerNoncurrentVersions": 3 }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-app-logs \
  --lifecycle-configuration file://lifecycle.json

⚠️ Gotchas that cost real money:

Incomplete multipart uploads are invisible in the console but billed. The AbortIncompleteMultipartUpload rule fixes that.

Versioned buckets keep every overwrite forever unless you expire noncurrent versions.

Small objects: by default, lifecycle doesn't transition objects under 128 KB. IA classes bill a 128 KB minimum, and Intelligent-Tiering doesn't auto-tier objects under 128 KB. Millions of tiny files belong in aggregated/compacted formats instead.

Transition requests cost money (per 1,000 objects). Moving 100M tiny objects to Glacier can cost more than it saves.

Use S3 Storage Lens (the free metrics are enough to start) to find buckets with no recent activity, large noncurrent-version bytes or incomplete multipart uploads. Use S3 Inventory for object-level audits.

4.2 EBS: gp2 → gp3 is often about a 20% storage saving

In many Regions, gp3 is about 20% cheaper per GB than gp2 and includes a 3,000 IOPS / 125 MiB/s baseline, independent of size. Confirm the current per-GB price before you migrate a large fleet. One catch: gp2 volumes above 1 TiB get more than 3,000 baseline IOPS (3 IOPS/GiB), so set IOPS explicitly or large volumes will lose performance:

#!/bin/bash
# Migrate all gp2 volumes to gp3, preserving the gp2 baseline IOPS (3 IOPS/GiB, capped at 16,000)
aws ec2 describe-volumes --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[].[VolumeId,Size]' --output text |
while read -r VOL SIZE; do
  IOPS=$(( SIZE * 3 )); (( IOPS < 3000 )) && IOPS=3000; (( IOPS > 16000 )) && IOPS=16000
  echo "Migrating $VOL (${SIZE}GiB) -> gp3 @ ${IOPS} IOPS"
  aws ec2 modify-volume --volume-id "$VOL" --volume-type gp3 --iops "$IOPS" --throughput 250
done

The migration is online, with no detach and no downtime. The same volume can't be modified again for 6 hours.

Hunt for orphans:

# Unattached EBS volumes
aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].{ID:VolumeId,GB:Size,Type:VolumeType,Created:CreateTime}' --output table

# Unassociated Elastic IPs (each public IPv4 address is billed hourly)
aws ec2 describe-addresses --query 'Addresses[?AssociationId==null].[PublicIp,AllocationId]' --output table

Move old snapshots you must keep to the EBS Snapshots Archive tier (about 75% cheaper, 90-day minimum, 24–72h restore):

  aws ec2 modify-snapshot-tier --snapshot-id snap-0abc123 --storage-tier archive

Automate snapshot retention with Amazon Data Lifecycle Manager or AWS Backup instead of hand-written cron jobs.

4.3 EFS and CloudWatch Logs, the ones people forget

EFS: Turn on lifecycle management to move cold files to Infrequent Access and Archive, with transition back to Standard on access for files that heat up again.
CloudWatch Logs: New log groups keep data forever by default, and ingestion (~$0.50/GB) is often a top-10 line item. Set retention on everything:

  for LG in $(aws logs describe-log-groups --query 'logGroups[?!retentionInDays].logGroupName' --output text); do
    aws logs put-retention-policy --log-group-name "$LG" --retention-in-days 30
  done

For high-volume, rarely queried logs, use the Infrequent Access log class (about half the ingestion price). Also drop DEBUG logging in production.

5. Network Traffic Cost Engineering

Data transfer is the silent killer of AWS budgets. It's spread across dozens of usage types, so it rarely looks like one big problem.

Figure: Network cost engineering — before-and-after paths for S3/DynamoDB, AWS APIs, service-to-service traffic, and content delivery.

5.1 Gateway VPC Endpoints: free, and frequently missing

Traffic from private subnets to S3 or DynamoDB through a NAT Gateway pays NAT data-processing charges. AWS has listed this at about $0.045/GB in us-east-1. Confirm the rate for your Region. A Gateway Endpoint costs nothing and takes one command:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0abc123 \
  --vpc-endpoint-type Gateway \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-0priv1a rtb-0priv1b rtb-0priv1c

Worked example (illustrative, using $0.045/GB): A data pipeline moves 20 TB/month from EC2 to S3 through NAT.
20,480 GB × $0.045 = $921.60/month. With a Gateway Endpoint, the NAT processing charge on that path becomes $0. The dollar result changes if your Region’s rate is different.

5.2 Interface Endpoints (PrivateLink): do the break-even math

For ECR, STS, CloudWatch Logs, SQS and similar services, an Interface Endpoint has often been listed at about $0.01/hr per AZ plus about $0.01/GB. Compared with NAT data processing at about $0.045/GB, the per-GB difference is about $0.035. Recalculate with your Region’s prices.

3 AZs × $0.01 × 730 h ≈ $21.90/month fixed
Break-even ≈ 21.90 / 0.035 ≈ ~625 GB/month through NAT to that service

EKS clusters pulling large container images from ECR through NAT usually pass that break-even quickly. Put ECR endpoints (ecr.api, ecr.dkr) in place, plus the S3 gateway endpoint, because ECR image layers are stored in S3.

5.3 Cross-AZ traffic: stay multi-AZ, route locally

Cross-AZ transfer has commonly been listed at about $0.01/GB in each direction. Chatty microservices, Kafka replication and read replicas in another AZ add up fast. The fix is not to give up multi-AZ. Keep the redundancy and keep the traffic local:

Kubernetes: topology-aware routing (service.kubernetes.io/topology-mode: Auto) or trafficDistribution: PreferClose on Services.
Kafka/MSK: configure client.rack so consumers fetch from the closest replica.
Databases: send read traffic to a replica in the same AZ as the caller.
ALB: check whether cross-zone load balancing is actually needed for each target group.

5.4 CloudFront in front of everything public

Data transfer from AWS origins to CloudFront is free, and CloudFront's egress rates are generally lower than EC2 internet egress at volume.
Every cache hit is a request your origin fleet doesn't serve, so the ALB/EC2 tier can be smaller.
Origin Shield adds a regional caching layer that collapses duplicate origin fetches from many edge locations.
Track cache hit ratio as a cost KPI. Tune Cache-Control headers and cache keys (drop unnecessary query strings and cookies).

5.5 Public IPv4 addresses are no longer free

AWS currently bills in-use and idle public IPv4 addresses. The published rate has been about $0.005 per hour (about $3.65 per month). Confirm it before you forecast a large cleanup. Hundreds of instances with public IPs "just in case" add up. Use private subnets and load balancers, and look at IPv6 where you can. VPC IP Address Manager (IPAM) shows public IP usage across accounts.

6. Automating Non-Prod Shutdown

A dev environment that runs only during office hours (12h × 5 days = 60 of 168 weekly hours) cuts that compute bill by ~64%. It's the highest-ROI automation in this article.

6.1 No-code option: EventBridge Scheduler universal targets

EventBridge Scheduler can call almost any AWS API directly, so you don't need a Lambda:

aws scheduler create-schedule \
  --name stop-dev-ec2-evening \
  --schedule-expression "cron(0 20 ? * MON-FRI *)" \
  --schedule-expression-timezone "Asia/Kolkata" \
  --flexible-time-window Mode=OFF \
  --target '{
    "Arn": "arn:aws:scheduler:::aws-sdk:ec2:stopInstances",
    "RoleArn": "arn:aws:iam::111122223333:role/scheduler-ec2-startstop",
    "Input": "{\"InstanceIds\":[\"i-0abc123\",\"i-0def456\"]}"
  }'

Create a matching startInstances schedule for the morning.

6.2 Tag-driven option: one Lambda for EC2 + RDS

Hard-coded instance IDs don't scale. Drive the schedule from tags (Schedule=OfficeHours) instead:

import boto3

ec2 = boto3.client("ec2")
rds = boto3.client("rds")

def lambda_handler(event, context):
    action = event.get("action", "stop")          # EventBridge passes {"action": "stop"|"start"}
    current_state = "running" if action == "stop" else "stopped"

    # EC2: find tagged instances in the right state
    paginator = ec2.get_paginator("describe_instances")
    ids = [
        i["InstanceId"]
        for page in paginator.paginate(Filters=[
            {"Name": "tag:Schedule", "Values": ["OfficeHours"]},
            {"Name": "instance-state-name", "Values": [current_state]},
        ])
        for r in page["Reservations"] for i in r["Instances"]
    ]
    if ids:
        (ec2.stop_instances if action == "stop" else ec2.start_instances)(InstanceIds=ids)

    # RDS: stop/start tagged DB instances (Aurora clusters need stop_db_cluster/start_db_cluster)
    for db in rds.describe_db_instances()["DBInstances"]:
        tags = {t["Key"]: t["Value"] for t in db.get("TagList", [])}
        if tags.get("Schedule") != "OfficeHours":
            continue
        if action == "stop" and db["DBInstanceStatus"] == "available":
            rds.stop_db_instance(DBInstanceIdentifier=db["DBInstanceIdentifier"])
        elif action == "start" and db["DBInstanceStatus"] == "stopped":
            rds.start_db_instance(DBInstanceIdentifier=db["DBInstanceIdentifier"])

    return {"action": action, "ec2": ids}

For many accounts with complex calendars, deploy the Instance Scheduler on AWS solution. It works off the Schedule tag and keeps schedule definitions in DynamoDB.

⚠️ Gotchas:

Auto Scaling groups replace stopped instances. For ASGs, scale to zero instead: aws autoscaling update-auto-scaling-group --auto-scaling-group-name dev-asg --min-size 0 --desired-capacity 0.

RDS restarts a stopped instance after 7 days. Your schedule has to stop it again. It's not a set-and-forget "off" switch.

EKS: scale node groups or Karpenter limits down, or scale deployments to zero after hours.

A stopped EC2 instance still pays for its EBS volumes and Elastic IPs.

7. Database Optimization

Databases are usually the second-largest line item, and the hardest to change later. Pick the right model early.

7.1 Aurora

Aurora Serverless v2 scales capacity (ACUs) in fine-grained steps, within seconds. It fits spiky or unpredictable workloads and dev/test databases, and it can now scale down to 0 ACUs with auto-pause, so idle non-prod databases cost almost nothing for compute.
Aurora I/O-Optimized: if I/O charges are more than ~25% of your Aurora bill, switching the cluster storage configuration removes per-I/O charges in exchange for higher instance and storage prices. You can switch back once every 30 days.
Use Graviton classes (db.r7g, db.r8g where available) for provisioned instances.

7.2 RDS

Right-size with Performance Insights / CloudWatch Database Insights (load by wait event, top SQL) instead of CPU alone. A slow query is cheaper to fix than a bigger instance.
Move RDS storage to gp3 and set IOPS independently of size.
Tune max_connections and use RDS Proxy when many Lambda functions or short-lived connections would otherwise force a larger instance.
⚠️ Extended Support surcharges: engine versions past end of standard support (e.g., older MySQL/PostgreSQL majors) are automatically enrolled in RDS Extended Support, billed per vCPU-hour. The rate goes up in later years. Upgrading is a cost optimization.
Buy Reserved Instances for steady production databases once they're right-sized.

7.3 DynamoDB

On-Demand vs Provisioned: On-Demand fits unpredictable or spiky traffic and got much cheaper after AWS's late-2024 price cut. Provisioned capacity with auto scaling (plus reserved capacity) is still cheaper for steady, high-throughput tables.
Standard-IA table class: about 60% cheaper storage and more expensive reads/writes. Use it for tables where storage dominates cost (history, audit, old orders).
TTL deletes expired items at no cost, which beats a scan-and-delete job.
DAX reduces read cost and latency for read-heavy hot keys, but it's a cluster you pay for by the hour. Measure before you add it.

7.4 Caching

ElastiCache for Valkey is priced lower than the Redis OSS engine (AWS quotes ~20% lower for nodes and ~33% lower for Serverless), and it's a drop-in replacement for most Redis workloads.

8. Continuous Cost Governance

Optimization without governance decays. Six months later, the waste is back. Make cost a DevOps responsibility backed by guardrails.

8.1 Service Control Policies: block expensive mistakes

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyExpensiveInstanceTypes",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringLike": {
          "ec2:InstanceType": ["p4d.*", "p5.*", "p5e.*", "u-*", "x2*", "*.metal*", "*.24xlarge", "*.32xlarge", "*.48xlarge"]
        },
        "ArnNotLike": { "aws:PrincipalArn": "arn:aws:iam::*:role/PlatformAdmin" }
      }
    },
    {
      "Sid": "DenyLaunchWithoutCostCenter",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": { "Null": { "aws:RequestTag/CostCenter": "true" } }
    }
  ]
}

Attach it to non-prod and sandbox OUs first. The PlatformAdmin exception gives a justified GPU or large-instance request a controlled path.

8.2 Tag Policies + AWS Config

Tag Policies (AWS Organizations) standardize tag keys and allowed values (Environment ∈ prod|staging|dev), so env, Env and ENV don't fragment your reports.
AWS Config required-tags flags non-compliant resources:

aws configservice put-config-rule --config-rule '{
  "ConfigRuleName": "required-tags",
  "Source": { "Owner": "AWS", "SourceIdentifier": "REQUIRED_TAGS" },
  "InputParameters": "{\"tag1Key\":\"Environment\",\"tag2Key\":\"Project\",\"tag3Key\":\"Owner\"}",
  "Scope": { "ComplianceResourceTypes": ["AWS::EC2::Instance","AWS::EC2::Volume","AWS::RDS::DBInstance","AWS::S3::Bucket"] }
}'

💡 Insight: Config has its own cost. Recording every configuration change of high-churn resources can be expensive. Use periodic recording for noisy resource types and record only what you actually govern.

8.3 Budgets Actions: guardrails that act

AWS Budgets can apply an IAM policy or SCP, or stop specific EC2/RDS instances, when a threshold is crossed, either automatically or after approval. A sandbox account that hits 120% of its budget can be locked down without anyone being paged at 2 a.m.

8.4 The monthly FinOps review

Run it from CUR + Athena (or QuickSight/CUDOS dashboards) and review these KPIs every month:

KPI	Target (starting point)
% of spend with valid owner tags	> 95%
Savings Plans / RI utilization	> 95%
Commitment coverage of eligible compute	70–85%
Spot share of stateless compute	growing quarter over quarter
Idle / orphaned resource spend	< 2%
Unit cost (e.g., $ per 1k requests)	flat or decreasing

Every item gets an owner, a date and a ticket. A review without owners is just a meeting.

Bonus: Hidden Cost Killers Checklist

These show up in almost every account I review:

[ ] NAT Gateway processing for S3/DynamoDB/ECR traffic (fix: endpoints)
[ ] CloudWatch Logs with no retention, and DEBUG logs in production
[ ] Idle load balancers and NAT Gateways in unused AZs
[ ] Unattached EBS volumes, old snapshots and unused AMIs
[ ] Public IPv4 addresses on instances that don't need them
[ ] gp2 volumes that were never migrated to gp3
[ ] EKS clusters on Kubernetes versions in extended support (AWS has published about $0.60/hr instead of about $0.10/hr per cluster, roughly $438 vs $73/month; confirm the current extended-support rate)
[ ] RDS engines in Extended Support
[ ] T-family instances in unlimited mode charging surplus credits
[ ] S3 incomplete multipart uploads and endless noncurrent versions
[ ] Cross-Region replication or backups nobody asked for
[ ] Dev/test environments running 24×7

AWS Services Reference: What Each One Does for Your Bill

Service	What it is	How it helps cost
Data Exports (CUR 2.0)	Line-item billing data delivered to S3	Working dataset for chargeback, anomaly hunting and unit cost, for the columns the export includes
Amazon Athena	Serverless SQL over S3	Query CUR and VPC Flow Logs without running a database
Amazon QuickSight (CUDOS)	BI dashboards	Self-service cost visibility per team
AWS Cost Explorer	Interactive cost analysis	Trends, forecasts, SP/RI coverage and utilization
Cost Categories	Rules that group costs	Map accounts and tags to business units
AWS Budgets / Budgets Actions	Thresholds + automated responses	Early warning; automatic lockdown of runaway accounts
Cost Anomaly Detection	ML-based spend monitoring	Catches spikes within hours, not at month-end
Cost Optimization Hub	Consolidated recommendations	One deduplicated, prioritized list of savings
AWS Compute Optimizer	ML right-sizing	EC2, ASG, EBS, Lambda, ECS on Fargate, RDS recommendations
AWS Trusted Advisor	Best-practice checks	Idle resources, low-utilization instances, unassociated EIPs
Savings Plans / Reserved Instances	Commitment discounts	AWS advertises up to 72% off steady compute; RIs for databases
EC2 Spot / EC2 Fleet / ASG mixed instances	Spare capacity at a discount	AWS advertises up to 90% off fault-tolerant compute
Karpenter	Kubernetes node autoscaler	Spot-first provisioning, bin-packing, consolidation
AWS Graviton	ARM-based instances	Lower price and better price-performance across EC2, Lambda, RDS, ElastiCache
EC2 Auto Scaling (target tracking, predictive)	Elastic capacity	Pay for demand instead of peak
S3 Lifecycle / Intelligent-Tiering / Storage Lens	Tiering and visibility	Move cold data to cheaper classes automatically
Amazon EBS gp3 / Snapshots Archive / Data Lifecycle Manager	Block storage controls	Often about 20% cheaper volumes, cheaper snapshot retention, automated cleanup
Amazon EFS lifecycle (IA/Archive)	File storage tiering	Cold files at a fraction of Standard price
VPC Gateway Endpoints	Private route to S3/DynamoDB	Removes NAT processing charges at no cost
AWS PrivateLink (Interface Endpoints)	Private access to AWS services	Cheaper than NAT above the break-even volume
Amazon CloudFront + Origin Shield	CDN	Lower egress, fewer origin requests, smaller origin fleet
Amazon EventBridge Scheduler	Serverless scheduler	Start/stop non-prod directly through API calls
AWS Lambda / SSM Automation	Serverless compute / runbooks	Automated remediation and right-sizing
Instance Scheduler on AWS	Tag-based scheduling solution	Org-wide office-hours schedules
Aurora Serverless v2 / I/O-Optimized	Elastic / I/O-flat database pricing	Pay for real load; predictable cost for I/O-heavy workloads
DynamoDB On-Demand / Standard-IA / TTL	Capacity modes and table classes	Match pricing to traffic and storage profile
ElastiCache for Valkey	Managed in-memory cache	Lower price than the Redis OSS engine
AWS Organizations (SCPs, Tag Policies)	Multi-account guardrails	Prevent costly launches; consistent tagging
AWS Config	Resource compliance	Enforce tags and flag non-compliant resources
Amazon Q Developer in chat apps	ChatOps (formerly AWS Chatbot)	Cost alerts where engineers already work

A 30-60-90 Day Rollout Plan

Days 0–30: visibility + quick wins

Enable CUR 2.0 → Athena, activate cost allocation tags, create Budgets and Anomaly Detection
Gateway Endpoints for S3/DynamoDB in every VPC
gp2 → gp3, delete orphaned volumes and EIPs, set CloudWatch Logs retention
S3 lifecycle "hygiene" rule on every bucket

Days 31–60: structural savings

Right-size from Compute Optimizer (with memory metrics)
Non-prod schedules through EventBridge Scheduler / Instance Scheduler
Spot for CI and batch; Karpenter with Spot on EKS
First Savings Plans tranche, sized to the post-right-sizing baseline

Days 61–90: make it stick

SCPs, Tag Policies, Config required-tags
Graviton canaries for stateless services and managed databases
Unit-cost dashboards and the monthly FinOps review, with owners

Conclusion

Real-world AWS cost optimization isn't one trick. It's an engineering discipline:

Measure with granular, tagged, near-real-time data.
Right-size on real percentiles, then commit to the floor and put the rest on Spot.
Tier storage and engineer network paths deliberately.
Automate the boring parts, like schedules, clean-ups and remediation.
Govern with guardrails so the waste doesn't come back.

Combine measurement, automation and governance, and you reduce waste, maximize savings and enforce accountability without giving up performance or reliability.

If this helped, drop a comment with the biggest hidden cost you've found in your AWS account. I'd love to compare notes. 👇

🧑‍💼 About the Author

Sanket Satish Patharkar is a Senior Cloud Operations Engineer with 6+ years of experience in AWS, DevOps, CI/CD and enterprise-grade cloud architecture, particularly in the automotive systems domain.

He specializes in building highly available, cost-efficient and automated infrastructure. He has designed cloud architectures that meet high availability (HA) and scalability requirements and keep applications running during component failures or traffic spikes. Sanket combines practical engineering, automation and governance to help organizations cut costs while keeping their cloud environments resilient and efficient.

For any issue or support: LinkedIn.

Dockerized Monitoring Stack

Sanket Patharkar — Fri, 25 Sep 2026 16:52:07 +0000

By Sanket Satish Patharkar – Senior Cloud Operation Engineer
Cloud Infrastructure & DevOps Specialist | 3× AWS Certified

If you've ever SSH'd into a box at 2 AM wondering "is it the database, the cache, or the load balancer?", you already know why a monitoring stack matters.

In this post I'll walk through how we dockerized our whole observability stack — Prometheus, Grafana, and a set of exporters for MongoDB, Redis, NGINX and TCP port checks — into one docker-compose.yml that can be deployed to multiple environments (UAT and Prod) with a single command.

You'll get:

🧱 A block diagram of the architecture
🐳 The Docker Compose setup, explained piece by piece
⚙️ Config for Prometheus, exporters and Grafana provisioning
💻 Every command you need to run, terminal-style
🚀 A Jenkins pipeline for deploy + automatic rollback
🩹 Lessons learned the hard way

All hostnames, IPs, and credentials below are placeholders. Replace them with your own.

1. The Architecture

Here's the big picture. Everything inside the dashed box runs as containers on one monitoring host (an EC2 instance in our case). The exporters reach out to the real infrastructure, Prometheus scrapes the exporters, and Grafana sits on top.

And this is how a change gets from Git to the server:

1.1 How the data actually flows (internals)

Understanding the internals makes every later config decision obvious.

Pull model. Prometheus pulls. Every scrape_interval it sends an HTTP GET /metrics to each target, parses the text exposition format, and appends samples to its TSDB. Exporters are stateless translators: on each scrape they query the backend (MongoDB serverStatus, Redis INFO, NGINX stub_status, a TCP handshake) and render the result as metrics. The only push path is Pushgateway, for short-lived jobs that finish before a scrape could happen.

A scrape, step by step:

relabel_configs run before the scrape, on target metadata (__address__, __meta_ec2_*). This is where the Blackbox trick happens.
metric_relabel_configs run after the scrape, on every sample. Use them to drop high-cardinality series you don't need.
Head block + WAL: the last ~2 hours live in memory, protected by a write-ahead log on the prometheus_data volume. That's why a named volume matters: without it, a container restart loses the WAL and the head block.
up metric: for every target Prometheus synthesises up{job,instance} = 1/0, plus scrape_duration_seconds and scrape_samples_scraped. These are your first debugging tools.

Two kinds of "down":

Metric	Meaning
`up{job="redis-exporter"} == 0`	Prometheus can't reach the exporter
`redis_up == 0`	The exporter is fine, but it can't reach Redis
`probe_success == 0`	Blackbox couldn't open a TCP connection to the target port

Alert on all three separately. They point to different fixes.

2. What's in the Stack

Service	Image	Port	What it does
`prometheus`	`prom/prometheus:v3.14.0`	9090	Scrapes & stores metrics (30 days)
`grafana`	`grafana/grafana:13.2.0`	3000	Dashboards + alerting
`mongodb-exporter`	`percona/mongodb_exporter:0.53.0`	9216	MongoDB server/replica-set metrics
`mongodb-query-exporter`	`ghcr.io/raffis/mongodb-query-exporter:v5.1.0`	9412	Business metrics from Mongo aggregations
`redis-exporter`	`oliver006/redis_exporter:v1.90.0`	9121	Redis memory, clients, keys, latency
`nginx-api-exporter`	`nginx/nginx-prometheus-exporter:1.5.3`	9113	NGINX `stub_status` connections/requests
`blackbox-exporter`	`prom/blackbox-exporter:v0.28.0`	9115	TCP "is the port up?" checks
`pushgateway`	`prom/pushgateway:v1.11.3`	9091	Receives metrics from short-lived jobs
`generate-hash`	`python:3-alpine`	–	Helper: bcrypt hash for Prometheus auth

💡 Pin your image versions. latest will eventually break your dashboards on a random Tuesday.

3. Project Layout

monitoring-stack/
├── docker-compose.yml
├── blackbox/
│   └── blackbox.yml
├── env/
│   ├── uat/            ├── .env  └── env.properties
│   └── prod/        ├── .env  └── env.properties
├── prometheus/
│   ├── web.yml                     # basic-auth users (bcrypt)
│   ├── uat/prometheus-tmpl.yml     # templates with ${VARS}
│   └── prod/prometheus-tmpl.yml
├── query-exporter/
│   └── <env>/config.yaml           # Mongo aggregation → metric
├── grafana/
│   ├── conf/grafana.ini
│   ├── provisioning/dashboards/<env>/dashboards.yml
│   └── dashboards/<env>/**.json    # folder structure = Grafana folders
├── scripts/
│   └── prometheus-healthcheck.sh
├── ssl-certs/
│   └── ca.pem                      # CA for Mongo TLS
└── Jenkinsfile (deploy.groovy)

The key idea: one compose file, many environments. Environment differences live in env/<env>/.env, prometheus/<env>/, query-exporter/<env>/ and grafana/dashboards/<env>/ — never in the compose file itself.

4. Prerequisites on the Monitoring Host

You need Docker Engine and the Compose v2 plugin (docker compose, not the old docker-compose).

Amazon Linux 2023:

ec2-user@monitoring-host:~$ sudo dnf install -y docker git gettext
ec2-user@monitoring-host:~$ sudo systemctl enable --now docker
ec2-user@monitoring-host:~$ sudo usermod -aG docker $USER && newgrp docker

# Compose v2 plugin
ec2-user@monitoring-host:~$ sudo mkdir -p /usr/local/lib/docker/cli-plugins
ec2-user@monitoring-host:~$ sudo curl -SL \
    https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 \
    -o /usr/local/lib/docker/cli-plugins/docker-compose
ec2-user@monitoring-host:~$ sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose

Ubuntu:

ubuntu@monitoring-host:~$ curl -fsSL https://get.docker.com | sudo sh
ubuntu@monitoring-host:~$ sudo apt-get install -y gettext-base git
ubuntu@monitoring-host:~$ sudo usermod -aG docker $USER && newgrp docker

Verify:

ec2-user@monitoring-host:~$ docker --version
Docker version 27.3.1, build ce12230
ec2-user@monitoring-host:~$ docker compose version
Docker Compose version v2.29.7
ec2-user@monitoring-host:~$ envsubst --version | head -1
envsubst (GNU gettext-runtime) 0.21

gettext gives us envsubst, which we use to render the Prometheus config from a template.

Network / security group checklist:

Direction	Port	Why
Inbound	3000	Grafana UI (ideally behind a reverse proxy / ALB)
Inbound	9090	Prometheus UI (restrict to VPN/office)
Outbound	27017 / 6379 / 80 / 8082 …	Exporters → targets
Outbound	443	Pull images, Slack/email webhooks

If Prometheus uses EC2 service discovery (below), attach an IAM instance role with ec2:DescribeInstances.

5. Dockerizing the Stack — the Compose File, Explained

Let's go through the interesting parts of docker-compose.yml.

5.1 Profiles = environments

Each service declares which environments it belongs to:

services:
  blackbox-exporter:
    image: prom/blackbox-exporter:v0.28.0
    container_name: blackbox-exporter
    restart: always
    profiles: [uat, prod]
    volumes:
      - ./blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
    command:
      - --config.file=/etc/blackbox_exporter/config.yml
    ports:
      - "9115:9115"

A service with no matching profile is simply not started. That lets UAT run extra QA exporters while prod runs a leaner set — from the same file.

5.2 Same container name, different config per environment

Our production MongoDB is big. Percona's --collect-all includes collstats, which on a large DB starved the connection pool so the important diagnosticdata collector never ran. The fix: two service definitions, same container_name, mutually exclusive profiles.

  # UAT: smaller DB tolerates collect-all
  mongodb-exporter:
    image: percona/mongodb_exporter:0.53.0
    container_name: mongodb-exporter
    restart: always
    profiles: [uat]
    command:
      - --collect-all
    environment:
      - MONGODB_URI=mongodb://${MONGO_DB_USERNAME}:${MONGO_DB_PASSWORD_ENCODED}@${MONGO_DB_HOST}:${MONGO_DB_PORT}/?directConnection=true&authSource=${MONGO_DB_NAME}&tls=true&tlsCAFile=/ssl-certs/ca.pem
    volumes:
      - ./ssl-certs:/ssl-certs:ro
    ports:
      - "9216:9216"
    extra_hosts:
      - "${MONGO_DB_HOST_DNS}"

  # Prod: only the collectors the dashboards need
  mongodb-exporter-prod:
    image: percona/mongodb_exporter:0.53.0
    container_name: mongodb-exporter
    restart: always
    profiles: [prod]
    command:
      - --collector.diagnosticdata
      - --collector.replicasetstatus
      - --collector.dbstats
      - --collector.topmetrics
    # ...same environment / volumes / ports as above

Two things worth copying:

extra_hosts injects hostname:ip into the container's /etc/hosts. Mongo TLS certs are issued for a hostname, so we connect by name even when there's no private DNS. The value comes from .env, e.g. MONGO_DB_HOST_DNS=mongo-primary.internal:x.x.x.x.
MONGO_DB_PASSWORD_ENCODED — URL-encode the password before putting it in a URI. A single @ or / in a password will ruin your afternoon.

5.3 Business metrics straight from MongoDB

Infrastructure metrics tell you the DB is healthy. They don't tell you "success rate for API calls dropped from 99% to 80%". For that we use mongodb-query-exporter, which turns an aggregation pipeline into a Prometheus gauge:

  mongodb-query-exporter:
    image: ghcr.io/raffis/mongodb-query-exporter:v5.1.0
    container_name: mongodb-query-exporter
    restart: always
    profiles: [uat, prod]
    environment:
      - MDBEXPORTER_MONGODB_URI=mongodb://${MONGO_DB_USERNAME}:${MONGO_DB_PASSWORD}@${MONGO_DB_HOST}:${MONGO_DB_PORT}/?authSource=${MONGO_DB_NAME}&tls=true&tlsCAFile=/ssl-certs/ca.pem&readPreference=secondaryPreferred
      - MDBEXPORTER_CONFIG=/config/config.yaml
    volumes:
      - ./query-exporter/${env}/config.yaml:/config/config.yaml:ro
      - ./ssl-certs:/ssl-certs:ro
    ports:
      - "9412:9412"

Note readPreference=secondaryPreferred — never run reporting aggregations on your primary.

query-exporter/<env>/config.yaml (simplified):

version: 3.0
bind: 0.0.0.0:9412
metricsPath: /metrics

global:
  queryTimeout: "120s"
  maxConnection: 1

servers:
  - name: main
    tls: true

aggregations:
  - server: main
    database: appdb
    collection: transactions
    cache: "5m"          # run the pipeline at most every 5 minutes
    mode: pull
    metrics:
      - name: app_total_calls
        type: gauge
        help: "Total calls per tenant (last hour)"
        value: total
        labels: [tenantId]
      - name: app_success_calls
        type: gauge
        help: "Successful calls per tenant (last hour)"
        value: success
        labels: [tenantId]
    pipeline: |
      [
        { "$match": { "createdAt": { "$gte": { "$date": { "$numberLong": "-3600000" } } } } },
        { "$group": {
            "_id": "$tenantId",
            "total":   { "$sum": 1 },
            "success": { "$sum": { "$cond": [{ "$eq": ["$status", "SUCCESS"] }, 1, 0] } }
        } },
        { "$project": { "_id": 0, "tenantId": "$_id", "total": 1, "success": 1 } }
      ]

Now app_success_calls / app_total_calls is a success-rate panel and an alert rule.

5.4 Redis & NGINX exporters

  redis-exporter:
    image: oliver006/redis_exporter:v1.90.0
    container_name: redis-exporter
    restart: always
    profiles: [uat, prod]
    command:
      - --redis.addr=redis://redis.internal:${REDIS_PORT}
      - --redis.password=${REDIS_DB_PASSWORD}
    ports:
      - "9121:9121"
    extra_hosts:
      - "${REDIS_HOST_DNS}"

  nginx-api-exporter:
    image: nginx/nginx-prometheus-exporter:1.5.3
    container_name: nginx-api-exporter
    restart: always
    profiles: [uat, prod]
    command:
      - --nginx.scrape-uri=http://nginx1:8082/stub_status
      - --nginx.scrape-uri=http://nginx2:8082/stub_status
    ports:
      - "9113:9113"
    extra_hosts:
      - "${NGINX_HOST1}"   # nginx1:x.x.x.x
      - "${NGINX_HOST2}"   # nginx2:x.x.x.x

On the NGINX side you need a stub_status location — keep it on an internal-only port:

server {
    listen 8082;
    allow 10.0.0.0/8;
    deny all;
    location /stub_status { stub_status; }
}

5.5 Prometheus — host networking, auth, healthcheck

  prometheus:
    image: prom/prometheus:v3.14.0
    container_name: prometheus
    restart: always
    profiles: [uat, prod]
    network_mode: host
    environment:
      - PROMETHEUS_PASSWORD=${PROMETHEUS_PASSWORD}
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/web.yml:/etc/prometheus/web.yml:ro
      - ./scripts/prometheus-healthcheck.sh:/usr/local/bin/prometheus-healthcheck:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.config.file=/etc/prometheus/web.yml'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
      - '--web.listen-address=0.0.0.0:9090'
      - '--web.external-url=${PROMETHEUS_WEB_EXTERNAL_URL}'
    healthcheck:
      test: ["CMD", "/bin/sh", "/usr/local/bin/prometheus-healthcheck"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s
    depends_on:
      - mongodb-query-exporter
      - blackbox-exporter
      - redis-exporter
      - nginx-api-exporter

Why these choices?

network_mode: host — every exporter publishes its port on the host, so Prometheus can scrape them all as localhost:<port>. Simple, and EC2 service discovery hits real private IPs without Docker NAT in the way.
--web.enable-lifecycle — lets you hot-reload config with POST /-/reload instead of restarting.
Named volume prometheus_data — metrics survive docker compose down.
web.yml — Prometheus' built-in basic auth, so the UI and API aren't wide open.

5.6 Grafana — plugins, provisioning, healthcheck

  grafana:
    image: grafana/grafana:13.2.0
    container_name: grafana
    restart: always
    profiles: [uat, prod]
    network_mode: host
    environment:
      - GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}
      - GF_SERVER_ROOT_URL=${GF_SERVER_ROOT_URL}
      - GF_SERVER_HTTP_PORT=3000
      - GF_SMTP_ENABLED=${GF_SMTP_ENABLED}
      - GF_SMTP_HOST=${GF_SMTP_HOST}
      - GF_SMTP_USER=${GF_SMTP_USER}
      - GF_SMTP_PASSWORD=${GF_SMTP_PASSWORD}
      - GF_INSTALL_PLUGINS=redis-datasource
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/conf/grafana.ini:/etc/grafana/grafana.ini:ro
      - ./grafana/provisioning/dashboards/${env}:/etc/grafana/provisioning/dashboards:ro
      - ./grafana/dashboards/${env}:/etc/grafana/dashboards:ro
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 2
    depends_on:
      - prometheus

volumes:
  grafana_data:
  prometheus_data:

Dashboards are file-provisioned — grafana/provisioning/dashboards/<env>/dashboards.yml:

apiVersion: 1
providers:
  - name: 'Default'
    orgId: 1
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/dashboards
      foldersFromFilesStructure: true   # sub-folders become Grafana folders

Drop a JSON file into grafana/dashboards/<env>/redis/cluster-health.json and it shows up in a redis folder within 10 seconds. Dashboards-as-code, no clicking.

5.7 A "tools" profile for one-off helpers

Need a bcrypt hash for prometheus/web.yml? Don't install Python on the server — run it in a throwaway container:

  generate-hash:
    image: python:3-alpine
    command: >
      sh -c "pip install --quiet bcrypt &&
      python3 -c 'import bcrypt, sys;
      print(bcrypt.hashpw(sys.argv[1].encode(), bcrypt.gensalt(rounds=10)).decode())'
      ${PROMETHEUS_PASSWORD}"
    profiles: [tools]

Because its only profile is tools, it never starts with the normal stack.

6. Configuration Files

6.1 The `.env` file

Non-secret, per-environment values are committed in env/<env>/.env. Secrets are not — they're injected at deploy time (see section 8).

# env/uat/.env  (example – placeholders only)
env=uat
COMPOSE_PROFILES=uat

MONGO_DB_HOST=mongo-primary.internal
MONGO_DB_PORT=27017
MONGO_DB_NAME=admin
MONGO_DB_HOST_DNS=mongo-primary.internal:x.x.x.x

REDIS_HOST_DNS=redis.internal:x.x.x.x
REDIS_PORT=6379

NGINX_HOST1=nginx1:x.x.x.x
NGINX_HOST2=nginx2:x.x.x.x

GF_SERVER_ROOT_URL=https://grafana.example.com
PROMETHEUS_WEB_EXTERNAL_URL=https://prometheus.example.com
PROMETHEUS_AWS_REGION=us-west-2

# Injected at deploy time – NOT committed:
# MONGO_DB_USERNAME, MONGO_DB_PASSWORD, MONGO_DB_PASSWORD_ENCODED,
# REDIS_DB_PASSWORD, PROMETHEUS_PASSWORD, GF_SECURITY_ADMIN_PASSWORD, GF_SMTP_PASSWORD

💡 Setting COMPOSE_PROFILES in .env means plain docker compose up -d automatically picks the right profile.

6.2 Prometheus template

prometheus/<env>/prometheus-tmpl.yml contains ${VARS} that get rendered with envsubst:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    basic_auth:
      username: admin
      password: "${PROMETHEUS_PASSWORD}"
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'grafana'
    static_configs:
      - targets: ['localhost:3000']

  - job_name: 'mongodb-exporter'
    scrape_interval: 30s
    static_configs:
      - targets: ['localhost:9216']

  - job_name: 'mongodb-query-exporter'
    scrape_interval: 60s
    scrape_timeout: 55s        # aggregations can be slow
    static_configs:
      - targets: ['localhost:9412']

  - job_name: 'redis-exporter'
    scrape_interval: 30s
    static_configs:
      - targets: ['localhost:9121']

  - job_name: 'nginx-api-exporter'
    static_configs:
      - targets: ['localhost:9113']

  # --- Blackbox TCP port checks via EC2 service discovery ---
  - job_name: 'app-port-check'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    ec2_sd_configs:
      - region: ${PROMETHEUS_AWS_REGION}
        port: 8080
        filters:
          - name: tag:cluster
            values: [APP-CLUSTER]
    relabel_configs:
      - source_labels: [__meta_ec2_private_ip]
        replacement: '${1}:8080'
        target_label: __param_target
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115   # send the probe to blackbox

The blackbox relabel dance, in plain English:

EC2 SD finds every instance tagged cluster=APP-CLUSTER.
Its private IP becomes the target query param.
The actual scrape goes to localhost:9115/probe?module=tcp_connect&target=x.x.x.x:8080.
You get probe_success{instance="app-1"} = 1 (up) or 0 (down).

New instances in the cluster are monitored automatically — no config change.

6.3 Blackbox module

# blackbox/blackbox.yml
modules:
  tcp_connect:
    prober: tcp
    timeout: 5s

6.4 Prometheus basic auth

# prometheus/web.yml
basic_auth_users:
  admin: "$2b$10$REPLACE_WITH_BCRYPT_HASH"

rounds=10 keeps bcrypt verification around ~50–80 ms. Prometheus checks the hash on every authenticated request (including Grafana's queries), so don't raise it to 14+ unless you enjoy slow dashboards.

6.5 Least-privilege credentials for exporters

Exporters should never connect as an admin. Create dedicated read-only identities.

MongoDB — the clusterMonitor role covers serverStatus, replSetGetStatus and dbStats. The query exporter additionally needs read on the databases it aggregates:

// mongosh, connected to the primary as an admin
use admin
db.createUser({
  user: "prom_exporter",
  pwd:  passwordPrompt(),
  roles: [
    { role: "clusterMonitor", db: "admin" },
    { role: "read",           db: "local" },   // oplog window metrics
    { role: "read",           db: "appdb" }    // query-exporter aggregations
  ]
})

Redis 6+ ACL — allow only the commands redis_exporter uses:

redis-cli ACL SETUSER prom_exporter on '>STRONG_PASSWORD' '~*' '&*' \
  -@all +@connection +memory +info +latency +slowlog +config|get +cluster|info \
  +client +scan +type +strlen +xinfo +xlen +pfcount +zcard +scard +llen +hlen +get +eval \
  -hello -echo -quit -auth -reset -readonly -readwrite -asking -wait

AWS IAM — the instance role only needs to describe instances for EC2 service discovery and read its own secret:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["ec2:DescribeInstances", "ec2:DescribeAvailabilityZones"], "Resource": "*" },
    { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:*:<ACCOUNT_ID>:secret:monitoring/*" }
  ]
}

7. Running the Stack — Step by Step

Here's the full manual flow, exactly as you'd type it.

Step 1 — Clone the repo

ec2-user@monitoring-host:~$ sudo mkdir -p /opt && cd /opt
ec2-user@monitoring-host:/opt$ git clone https://github.com/patharkar123/monitoring-stack.git monitoring
Cloning into 'monitoring'...
remote: Enumerating objects: 1432, done.
Receiving objects: 100% (1432/1432), 2.10 MiB | 8.40 MiB/s, done.
ec2-user@monitoring-host:/opt$ cd monitoring

Step 2 — Build the runtime `.env`

Start from the committed env file, then append secrets (from your secret manager, vault, or — for a lab — by hand):

ec2-user@monitoring-host:/opt/monitoring$ export ENV_NAME=uat
ec2-user@monitoring-host:/opt/monitoring$ cp env/$ENV_NAME/.env .env

ec2-user@monitoring-host:/opt/monitoring$ aws secretsmanager get-secret-value \
    --secret-id monitoring/$ENV_NAME --region us-west-2 \
    --query SecretString --output text \
  | jq -r 'to_entries[] | "\(.key)=\"\(.value)\""' >> .env

ec2-user@monitoring-host:/opt/monitoring$ chmod 600 .env
ec2-user@monitoring-host:/opt/monitoring$ grep -c = .env
58

Step 3 — Generate the Prometheus password hash

ec2-user@monitoring-host:/opt/monitoring$ docker compose --profile tools run --rm generate-hash
[+] Pulling 1/1
 ✔ generate-hash Pulled                                                     3.2s
$2b$10$Qm8x0l3d0Vb7gJ1tq5o1KeY8W2r3oNQ3m2xF1c9bZ0t4pF7yXh6uS

Paste that hash into prometheus/web.yml.

Step 4 — Render `prometheus.yml` from the template

ec2-user@monitoring-host:/opt/monitoring$ (set -a; source .env; set +a; \
    envsubst < prometheus/$ENV_NAME/prometheus-tmpl.yml > prometheus/prometheus.yml)

ec2-user@monitoring-host:/opt/monitoring$ docker run --rm --entrypoint promtool \
    -v $PWD/prometheus:/p prom/prometheus:v3.14.0 check config /p/prometheus.yml
Checking /p/prometheus.yml
 SUCCESS: /p/prometheus.yml is valid prometheus config file syntax

✅ Always run promtool check config before starting. A YAML typo otherwise shows up as a crash-looping container.

Step 5 — Validate the compose file

ec2-user@monitoring-host:/opt/monitoring$ docker compose --profile $ENV_NAME config --services
mongodb-exporter
mongodb-query-exporter
blackbox-exporter
redis-exporter
nginx-api-exporter
pushgateway
prometheus
grafana

This shows exactly which services the profile will start — a great sanity check.

Step 6 — Start everything

ec2-user@monitoring-host:/opt/monitoring$ docker compose -p monitoring --profile $ENV_NAME up -d
[+] Running 11/11
 ✔ Network monitoring_default            Created                            0.1s
 ✔ Volume "monitoring_prometheus_data"   Created                            0.0s
 ✔ Volume "monitoring_grafana_data"      Created                            0.0s
 ✔ Container blackbox-exporter           Started                            0.8s
 ✔ Container redis-exporter              Started                            0.8s
 ✔ Container mongodb-exporter            Started                            0.9s
 ✔ Container mongodb-query-exporter      Started                            0.9s
 ✔ Container nginx-api-exporter          Started                            0.8s
 ✔ Container pushgateway                 Started                            0.7s
 ✔ Container prometheus                  Started                            1.2s
 ✔ Container grafana                     Started                            1.6s

Step 7 — Check container health

ec2-user@monitoring-host:/opt/monitoring$ docker compose -p monitoring ps --format "table {{.Name}}\t{{.Status}}\t{{.Ports}}"
NAME                     STATUS                    PORTS
blackbox-exporter        Up 2 minutes              0.0.0.0:9115->9115/tcp
grafana                  Up 2 minutes (healthy)
mongodb-exporter         Up 2 minutes              0.0.0.0:9216->9216/tcp
mongodb-query-exporter   Up 2 minutes              0.0.0.0:9412->9412/tcp
nginx-api-exporter       Up 2 minutes              0.0.0.0:9113->9113/tcp
prometheus               Up 2 minutes (healthy)
pushgateway              Up 2 minutes              0.0.0.0:9091->9091/tcp
redis-exporter           Up 2 minutes              0.0.0.0:9121->9121/tcp

(Prometheus and Grafana show no ports because they use host networking.)

Step 8 — Verify each exporter

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9216/metrics | grep -m1 '^mongodb_up'
mongodb_up{cluster_role="mongod"} 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9121/metrics | grep '^redis_up'
redis_up 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9113/metrics | grep '^nginx_up'
nginx_up 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s "localhost:9115/probe?module=tcp_connect&target=x.x.x.x:27017" | grep '^probe_success'
probe_success 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9412/metrics | grep -m2 '^app_'
app_total_calls{tenantId="tenant-a"} 1824
app_success_calls{tenantId="tenant-a"} 1797

Step 9 — Verify Prometheus targets

ec2-user@monitoring-host:/opt/monitoring$ curl -s -u admin:$PROMETHEUS_PASSWORD \
    localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.health)\t\(.labels.job)"' | sort | uniq -c
      1 up  grafana
      4 up  app-port-check
      1 up  mongodb-exporter
      1 up  mongodb-query-exporter
      1 up  nginx-api-exporter
      1 up  prometheus
      1 up  redis-exporter

Anything showing down? Check .lastError on that target:

ec2-user@monitoring-host:/opt/monitoring$ curl -s -u admin:$PROMETHEUS_PASSWORD localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.health!="up") | "\(.labels.job): \(.lastError)"'

Step 10 — Open Grafana

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:3000/api/health
{
  "database": "ok",
  "version": "13.2.0"
}

Browse to http://<host>:3000, log in with GF_SECURITY_ADMIN_USER / GF_SECURITY_ADMIN_PASSWORD, add Prometheus (http://localhost:9090, basic auth) as a data source, and your provisioned dashboards are already waiting in their folders.

8. PromQL, Recording Rules & Alerting

Collecting metrics is half the job. These are the queries that actually power the dashboards and alerts.

8.1 Queries worth knowing

# Business success rate per tenant (%), from mongodb-query-exporter
100 * app_success_calls / clamp_min(app_total_calls, 1)

# MongoDB replication lag per secondary (seconds; optimeDate is in ms)
(scalar(max(mongodb_rs_members_optimeDate{member_state="PRIMARY"}))
  - mongodb_rs_members_optimeDate{member_state="SECONDARY"}) / 1000

# MongoDB connection saturation (%)
100 * mongodb_ss_connections{conn_type="current"}
    / (mongodb_ss_connections{conn_type="current"} + mongodb_ss_connections{conn_type="available"})

# Redis memory usage vs maxmemory (%)
100 * redis_memory_used_bytes / clamp_min(redis_memory_max_bytes, 1)

# Redis cache hit ratio (5m)
rate(redis_keyspace_hits_total[5m])
  / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))

# NGINX requests per second across the cluster
sum(rate(nginx_http_requests_total[1m]))

# Which ports are down right now?
probe_success == 0

# Exporter itself unreachable
up == 0

💡 clamp_min(x, 1) avoids divide-by-zero NaN when a tenant or instance has no traffic. NaN never triggers alerts, which silently hides real outages.

8.2 Recording rules

Heavy queries evaluated on every dashboard refresh waste CPU. Pre-compute them with recording rules, then point dashboards at the recorded series.

# prometheus/rules/recording.yml
groups:
  - name: app-sli
    interval: 1m
    rules:
      - record: tenant:success_ratio:pct
        expr: 100 * app_success_calls / clamp_min(app_total_calls, 1)

      - record: cluster:nginx_requests:rate1m
        expr: sum(rate(nginx_http_requests_total[1m]))

      - record: instance:redis_memory_used:pct
        expr: 100 * redis_memory_used_bytes / clamp_min(redis_memory_max_bytes, 1)

Load them by adding rule_files: ['/etc/prometheus/rules/*.yml'] to the template and mounting ./prometheus/rules:/etc/prometheus/rules:ro. The naming convention level:metric:operation tells readers at a glance what was aggregated.

8.3 Alert rules

We manage alerts in Grafana (provisioned from JSON via its API), but the logic is identical in native Prometheus rule syntax:

groups:
  - name: availability
    rules:
      - alert: ServicePortDown
        expr: probe_success == 0
        for: 2m                     # ignore single failed probes
        labels: { severity: critical }
        annotations:
          summary: "Port check failing on {{ $labels.instance }}"
          description: "TCP connect to {{ $labels.probe_target }} failed for 2 minutes."

      - alert: MongoReplicationLagHigh
        expr: |
          (scalar(max(mongodb_rs_members_optimeDate{member_state="PRIMARY"}))
            - mongodb_rs_members_optimeDate{member_state="SECONDARY"}) / 1000 > 30
        for: 5m
        labels: { severity: warning }

      - alert: TenantSuccessRateDrop
        expr: tenant:success_ratio:pct < 90 and app_total_calls > 50
        for: 10m
        labels: { severity: critical }
        annotations:
          summary: "Success rate for {{ $labels.tenantId }} is {{ $value | printf \"%.1f\" }}%"

Design notes:

for: is your noise filter. Port checks: 1–2 minutes. Capacity alerts: 10+ minutes.
and app_total_calls > 50 prevents alerting on a 1-of-2 failure at 3 AM when traffic is near zero.
Route severity=critical to on-call chat/pager and warning to email. Everyone ignores a channel that pages for everything.

9. Capacity Planning & Hardening

9.1 Sizing Prometheus storage

Prometheus stores roughly 1–2 bytes per sample after compression. Estimate disk with:

disk_bytes ≈ retention_seconds × ingested_samples_per_second × bytes_per_sample

Worked example for this stack:

Input	Value
Active series	~60,000
Average scrape interval	30 s
Samples/second	60,000 / 30 = 2,000
Retention	30 d = 2,592,000 s
Bytes/sample	~1.5
Disk	2,592,000 × 2,000 × 1.5 ≈ 7.8 GB (+ WAL & compaction headroom → provision 20 GB)

Check your real numbers instead of guessing:

ec2-user@monitoring-host:~$ curl -s -u admin:$PROMETHEUS_PASSWORD localhost:9090/api/v1/status/tsdb \
    | jq '.data.headStats, (.data.seriesCountByMetricName[:5])'
{
  "numSeries": 58312,
  "chunkCount": 121480,
  "minTime": 1790370000000,
  "maxTime": 1790377200000
}
[
  { "name": "mongodb_ss_wt_cache_bytes", "value": 4210 },
  { "name": "mongodb_collstats_storageStats_indexSizes", "value": 3877 },
  ...
]

ec2-user@monitoring-host:~$ curl -s -u admin:$PROMETHEUS_PASSWORD \
    'localhost:9090/api/v1/query?query=rate(prometheus_tsdb_head_samples_appended_total[5m])' \
    | jq -r '.data.result[0].value[1]'
1987.4

The seriesCountByMetricName list is your cardinality hit list. If one metric dominates, drop it with metric_relabel_configs:

  - job_name: 'mongodb-exporter'
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'mongodb_collstats_storageStats_indexSizes'
        action: drop

Memory rule of thumb: ~3–4 KB of RAM per active series in the head block. 60k series ≈ 250 MB, so a t3.medium (4 GB) runs this whole stack comfortably.

9.2 Container resource limits & log rotation

Without limits, one misbehaving exporter can starve Prometheus. Without log rotation, Docker's json-file logs will eventually fill the disk.

x-defaults: &defaults
  restart: always
  logging:
    driver: json-file
    options: { max-size: "10m", max-file: "3" }

services:
  redis-exporter:
    <<: *defaults
    image: oliver006/redis_exporter:v1.90.0
    mem_limit: 128m
    cpus: 0.25

  prometheus:
    <<: *defaults
    image: prom/prometheus:v3.14.0
    mem_limit: 2g
    cpus: 1.0

The x- extension field + YAML anchor (&defaults / <<: *defaults) keeps these settings in one place.

9.3 Network hardening

Prometheus and Grafana use host networking, so they reach exporters via localhost. That means exporter ports don't need to be exposed on the network interface at all. Bind them to loopback:

    ports:
      - "127.0.0.1:9216:9216"   # instead of "9216:9216"

Verify what's actually listening externally:

ec2-user@monitoring-host:~$ sudo ss -tlnp | awk 'NR==1 || /:(3000|9090|91[0-9][0-9]|92[0-9][0-9]|94[0-9][0-9])/'
State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
LISTEN  0       4096        127.0.0.1:9216        0.0.0.0:*      users:(("docker-proxy",pid=2211))
LISTEN  0       4096        127.0.0.1:9121        0.0.0.0:*      users:(("docker-proxy",pid=2240))
LISTEN  0       4096                *:9090              *:*      users:(("prometheus",pid=2302))
LISTEN  0       4096                *:3000              *:*      users:(("grafana",pid=2355))

Then put Grafana and Prometheus behind a TLS-terminating load balancer or reverse proxy, and restrict the security group so ports 3000/9090 accept traffic only from that proxy.

⚠️ Docker publishes ports by writing its own iptables rules, which bypass ufw/firewalld. Loopback binding or security groups are the reliable controls, not the host firewall.

10. Automating It — Jenkins Deploy with Rollback

Doing the above by hand is fine once. For every change we use a Jenkins pipeline with one parameter: which environment.

pipeline {
  agent { label "${whereTo}-jenkins-agent" }
  parameters {
    choice(name: 'whereTo', choices: ['uat', 'prod'], description: 'Which env?')
    string(name: 'branch',  defaultValue: 'main')
    booleanParam(name: 'skipDeploy', defaultValue: false)
  }
  stages {
    stage('checkout') {
      steps { dir('monitoring') { git url: env.REPO_URL, branch: params.branch, credentialsId: 'git-ssh' } }
    }
    stage('init') {
      steps { dir('monitoring') { script {
        def envVars = readFile("env/${whereTo}/.env")
        def props   = readProperties file: "env/${whereTo}/env.properties"   // hostIps=x.x.x.x,...
        hostIps     = props.hostIps.split(',')

        // secrets → appended to .env (never committed)
        def secret = readJSON text: getSecret("monitoring/${whereTo}")
        secret.each { k, v -> envVars += "\n${k}=\"${v}\"" }
        writeFile file: '.env', text: envVars

        sh """(set +x; set -a; source .env; set +a;
               envsubst < prometheus/${whereTo}/prometheus-tmpl.yml > prometheus/prometheus.yml)"""
      }}}
    }
    stage('deploy') {
      when { expression { !params.skipDeploy } }
      steps { script {
        hostIps.each { ip ->
          try { doRelease(ip) }
          catch (e) { echo "❌ Release failed on ${ip}: ${e} — rolling back"; doRelease(ip, true); error(e.message) }
        }
      }}
    }
  }
}

def doRelease(ip, rollback = false) {
  def live = '/opt/monitoring', bk = '/opt/monitoring-bk'
  if (rollback) {
    ssh(ip, "sudo rm -rf ${live} && sudo mv ${bk} ${live}")
  } else {
    ssh(ip, "sudo bash -c 'mkdir -p ${live} && rm -rf ${bk} /tmp/monitoring && mv ${live} ${bk}'")
    scp(ip, 'monitoring', '/tmp')                 // push the rendered workspace
    ssh(ip, "sudo mv /tmp/monitoring ${live}")
  }
  ssh(ip, "sudo bash -c 'cd ${live} && source .env && docker compose -p monitoring down || true && docker compose up -d'")
}

The pattern is simple but effective:

   live dir ──mv──► backup dir          (keep last good release)
   new files ──scp──► /tmp ──mv──► live dir
   docker compose down && up -d
        │
        ├─ success ─► done ✅
        └─ failure ─► rm live, mv backup → live, compose up ─► previous version running ↩️

Because the rendered .env and prometheus.yml travel with the release, the rollback restores both code and config.

11. Day-2 Operations Cheat Sheet

# Tail logs for one service
$ docker compose -p monitoring logs -f --tail=100 prometheus

# Hot-reload Prometheus after editing prometheus.yml (no restart)
$ curl -X POST -u admin:$PROMETHEUS_PASSWORD localhost:9090/-/reload

# Restart a single exporter
$ docker compose -p monitoring restart mongodb-query-exporter

# Upgrade an image: bump the tag in docker-compose.yml, then
$ docker compose -p monitoring pull grafana
$ docker compose -p monitoring up -d grafana

# See resource usage
$ docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

# How big is the TSDB?
$ docker system df -v | grep prometheus_data

# Push a metric from a cron/batch job
$ echo "backup_last_success_timestamp $(date +%s)" \
    | curl --data-binary @- localhost:9091/metrics/job/nightly_backup

# Stop everything (volumes – and your data – are kept)
$ docker compose -p monitoring down

# ⚠️ Stop AND delete all metrics & Grafana data
$ docker compose -p monitoring down -v

# Clean up old images after upgrades
$ docker image prune -f

12. Troubleshooting Guide

Symptom	Likely cause	How to confirm	Fix
`up{job="mongodb-query-exporter"} == 0` intermittently	Aggregation slower than `scrape_timeout`	`scrape_duration_seconds` ≈ timeout	Add indexes, raise `cache`, keep `scrape_timeout` < `scrape_interval`
`mongodb_up 0`, exporter `up 1`	TLS hostname mismatch / wrong `extra_hosts`	`docker logs mongodb-exporter` → `x509: certificate is valid for …`	Connect by the cert's hostname; fix `*_HOST_DNS`
`mongodb_ss_*` series missing	`collstats` starving the connection pool	Only `mongodb_collstats_*` present	Use explicit `--collector.*` flags
Compose: `service "x" depends on undefined service`	`depends_on` a service outside the active profile	`docker compose --profile <env> config`	Only depend on services present in every profile
Prometheus restarts in a loop	Invalid rendered `prometheus.yml` (empty `${VAR}`)	`promtool check config`	Ensure all template variables exist in `.env`
Grafana "401" on Prometheus datasource	Hash in `web.yml` ≠ password in datasource	`curl -u admin:<pw> localhost:9090/-/healthy`	Regenerate hash, restart Prometheus
Disk full on host	Unrotated container logs or TSDB growth	`docker system df -v`, `du -sh /var/lib/docker/containers/*`	Log rotation (9.2), drop high-cardinality metrics
`probe_success 0` but service works from your laptop	Security group blocks the monitoring host	`nc -vz <ip> <port>` from the host	Allow the monitoring host's SG as source

13. Lessons Learned (the Hard Way)

1. depends_on + profiles can break the whole project.
If Prometheus depends_on: mongodb-exporter-prod but you run the uat profile, Compose treats that service as undefined and refuses to load the project. Only depend on services that exist in every profile the dependent runs in.

2. --collect-all is not free.
On large MongoDB clusters, collstats monopolised the exporter's connection and critical metrics (mongodb_ss_*, replica-set state) went missing. Enable only the collectors your dashboards use.

3. Point business queries at secondaries and cache them.
readPreference=secondaryPreferred + cache: "5m" + a generous scrape_timeout kept heavy aggregations away from production traffic.

4. Use extra_hosts for TLS hostnames.
TLS verification needs the hostname on the certificate. Injecting host:ip via extra_hosts avoids hacking /etc/hosts on the server itself.

5. Keep secrets out of Git, render at deploy time.
Committed .env = non-secret config. Secrets are fetched from a secret manager and appended in the pipeline workspace. The Prometheus password is envsubst-ed into prometheus.yml only on the deploy host.

6. Validate before you restart.
docker compose config, promtool check config, and a --dry-run/skipDeploy flag in the pipeline catch 90% of mistakes before they take monitoring down — which is the one system you really need up when everything else is on fire.

7. Know who owns what in Grafana.
If you mix file provisioning, API provisioning and Grafana's Git Sync, make each one own a clearly separate set of resources (e.g. dashboards via files/Git, alerts & datasources via API). Overlap leads to "who deleted my datasource?" mysteries.

Wrapping Up

With one docker-compose.yml, a handful of config files and a small pipeline, we got:

✅ Infra metrics for MongoDB, Redis, NGINX
✅ Business KPIs straight from MongoDB aggregations
✅ Automatic port checks for every instance in a cluster via EC2 discovery
✅ Dashboards-as-code with folder structure
✅ Repeatable, multi-environment deploys with automatic rollback

The whole stack comes up in under a minute on a fresh host:

The full stack is in patharkar123/monitoring-stack:

$ git clone https://github.com/patharkar123/monitoring-stack.git monitoring
$ cd monitoring
$ cp env/uat/.env .env   # + secrets
$ envsubst < prometheus/uat/prometheus-tmpl.yml > prometheus/prometheus.yml
$ docker compose --profile uat up -d

If you found this useful, drop a ❤️ or a comment — and tell me what you monitor that I didn't cover here. For any issue or support, reach me on LinkedIn. Happy monitoring! 📈

🔜 Up Next

In the next article I'll cover alerting as code: provisioning Grafana alert rules, contact points and datasources through the Grafana HTTP API from Jenkins, with idempotent diffing and dry-run support.

👤 About the Author

Sanket Satish Patharkar is a Senior Cloud Operation Engineer with over 6 years of experience in AWS, DevOps, CI/CD, and cloud architecture for enterprise-grade systems. 3× AWS Certified, with a focus on cloud infrastructure, observability, automation, and GenAI.

📍 Pune, Maharashtra, India · 🎓 B.E. in Electronics & Telecommunication, Savitribai Phule Pune University

Repo: github.com/patharkar123/monitoring-stack · For any issue or support: LinkedIn

Designing a Highly Available and Scalable Architecture on AWS

Sanket Patharkar — Thu, 03 Jul 2025 08:41:04 +0000

By Sanket Satish Patharkar, Senior Cloud Operation Engineer

This document is a practical design for an application that must stay available when an instance, an Availability Zone, or a deployment fails, and that must scale with demand instead of with a fixed fleet. It is written so each section can be copied as-is.

1. Scope and design targets

High availability and disaster recovery are different problems. Multi-AZ keeps a single Region serving traffic through an instance or Availability Zone failure. Multi-Region is for the loss of a Region, and it is justified only after recovery objectives are explicit.

Set these before choosing services:

Availability target. 99.99% allows about 52 minutes of downtime per year. That target requires at least two Availability Zones, health-checked load balancing, and a data store that fails over without a manual restore.
Recovery point objective (RPO). How much data loss is acceptable. RDS Multi-AZ and Aurora use synchronous storage inside the Region, so the RPO for an Availability Zone failure is near zero. A cross-Region replica is asynchronous, so the RPO is the replication lag.
Recovery time objective (RTO). How long the service may be down. An Application Load Balancer removes a failed task in the health-check interval. An RDS Multi-AZ failover is typically one to two minutes. Aurora failover is usually under a minute. A Regional failover is measured in minutes only if the second Region is already warm.
Scale unit. What grows: requests per second, messages per second, or stored bytes. The Auto Scaling signal has to match that unit. ## 2. Reference architecture

One Virtual Private Cloud, three Availability Zones, three subnet tiers.

Public subnets hold only the load balancer nodes and the NAT gateways. Application subnets are private and hold Amazon EC2, Amazon ECS, or Amazon EKS. Data subnets are private and hold Amazon RDS or Amazon Aurora, and Amazon ElastiCache. There is no path from the internet to the data tier.

Request path for a synchronous API:

Amazon Route 53 resolves the name. A latency or failover record selects the Region. The record TTL for a failover name is 60 seconds or less.
Amazon CloudFront terminates viewer TLS with an AWS Certificate Manager certificate, caches what is safe to cache, and forwards dynamic requests to the Application Load Balancer. AWS WAF is attached here or on the load balancer.
The Application Load Balancer spans all three Availability Zones, checks a /health endpoint, and sends traffic only to healthy targets.
The application is stateless. Session and hot keys live in ElastiCache or Amazon DynamoDB. The instance disk is disposable.
Writes go to the RDS or Aurora writer. Reads that tolerate replication lag go to a reader endpoint. Connection spikes go through Amazon RDS Proxy so a scale-out does not exhaust database connections.
Work that does not need an immediate response is published to Amazon SQS or Amazon EventBridge. The API returns after the enqueue.

Outbound path from a private subnet: a NAT gateway in the same Availability Zone, or a VPC endpoint when the destination is an AWS API. Amazon S3 and DynamoDB use gateway endpoints. AWS Secrets Manager, Amazon KMS, Amazon CloudWatch Logs, Amazon ECR, and AWS Systems Manager use interface endpoints so a NAT failure does not block pulls, secrets, or log shipping.

Figure 1. Multi-AZ reference architecture.

3. Edge, DNS, and load balancing

Route 53 health checks the load balancer or a synthetic endpoint, not a single instance. Failover routing moves DNS to the standby Region. Latency routing is for active-active. Do not create the standby DNS record until the standby can actually serve.

Use an Application Load Balancer for HTTP and HTTPS. Use a Network Load Balancer only when you need a static IP, TCP or UDP, or millions of connections per second. Enable cross-zone load balancing on the Application Load Balancer. Set deregistration delay to match the longest request you must finish, commonly 30 to 60 seconds, so a scale-in does not cut in-flight work.

The target health check uses the application health path, an interval of 10 seconds, and a threshold of 2. A process that is up but cannot reach its database must fail that check.

Amazon CloudFront sits in front of the load balancer for TLS at the edge and to absorb bursts of static content. Amazon API Gateway is the front door when the client is a device or a partner calling a REST or WebSocket API, with throttling per API key. AWS Global Accelerator is the alternative when clients need static anycast IPs and fast Regional failover without waiting on DNS TTL.

AWS WAF on CloudFront or the Application Load Balancer blocks common web exploits. AWS Shield Standard is automatic. Shield Advanced is for workloads that are attacked and need cost protection and the response team, not for every design.

4. Compute and scaling

Amazon EC2 Auto Scaling uses a launch template, not a launch configuration. The template enforces IMDSv2, encrypts EBS with a customer managed KMS key, and attaches an instance profile. No long-lived access keys are placed on the instance.

Minimum capacity is at least two, spread across Availability Zones, and that minimum runs On-Demand. Spot capacity is extra scale above the minimum. If the minimum the availability target depends on is Spot, a capacity withdrawal is an outage.

Scaling policy is target tracking on the signal that matches the workload: ALBRequestCountPerTarget for a request-driven API, or CPU only when CPU is the real bottleneck. Scale-in is slower than scale-out. Health check type is ELB, so a target the load balancer has marked unhealthy is replaced.

A rolling deployment uses a minimum healthy percentage so both Availability Zones are never replaced in the same batch. For Amazon ECS on AWS Fargate, the service spans the private subnets, the deployment minimum healthy percent stays at 100 with a maximum of 200, and tasks are replaced only after the new task is healthy. Amazon EKS uses the same rule: a PodDisruptionBudget and topology spread so one zone cannot hold every replica.

AWS Lambda is for event-driven work with uneven load. Set reserved concurrency on functions that call the database so a spike cannot exhaust connections. Lambda destinations or an SQS dead-letter queue capture failed asynchronous invocations.

5. Data

Amazon RDS Multi-AZ keeps a synchronous standby in another Availability Zone and fails over the writer endpoint. Use it when the engine must be MySQL, PostgreSQL, SQL Server, or Oracle and failover in one to two minutes is acceptable. Amazon Aurora uses a shared storage volume across three Availability Zones and fails over faster. Use Aurora when the workload is MySQL- or PostgreSQL-compatible and the RTO is tighter. A read replica, and the Aurora reader endpoint, scale reads. They are not the failover mechanism for the writer.

Turn on automated backups with a retention that matches the RPO, and point-in-time recovery. Take snapshots before a schema change. Amazon RDS Proxy pools connections and makes a Multi-AZ failover less visible to the application. Store the master password in AWS Secrets Manager and rotate it. Do not put it in the task definition or user data.

Amazon DynamoDB is the choice when access is by key and the workload is spiky. It is Multi-AZ by default. Point-in-time recovery and on-demand or provisioned capacity with auto scaling belong in the table design. DynamoDB Accelerator (DAX) is a cache in front of DynamoDB, not a general application cache. Global Tables replicate across Regions asynchronously. That RPO is the replication lag, and writes must be idempotent because the same key can be written in two Regions.

Amazon ElastiCache for Redis uses a replication group with Multi-AZ and automatic failover. The application uses the primary endpoint for writes and the reader endpoint for cache reads. Memcached has no replication. Use it only for data that can be rebuilt. Do not store the only copy of a session in Memcached.

Amazon S3 holds objects, logs, and backups. Block public access at the account, enable versioning on buckets that hold data you cannot recreate, and encrypt with KMS. S3 Cross-Region Replication copies objects for a Regional loss. It is not a substitute for database backups. AWS Backup is the control plane for RDS, Aurora, EBS, and DynamoDB backup policies, copy to a second Region, and a vault lock when retention must be immutable.

6. Messaging and events

Amazon SQS decouples the API from slow work. Set the visibility timeout longer than the handler's worst case. Set a redrive policy to a dead-letter queue after a small max receive count, and alarm when that queue is non-empty. The consumer must be idempotent, because a timeout delivers the message again.

Use SQS standard queues for throughput. Use FIFO only when order per group and exactly-once processing matter, and accept the lower throughput. Amazon SNS fan-out is for one event and many independent consumers. Amazon EventBridge is for routing events between applications by rule, including a scheduled rule that replaces a single cron host.

Amazon Kinesis Data Streams is the ingest path for ordered, high-rate telemetry. Size shards for the write rate, and use enhanced fan-out when many consumers read the same stream. Amazon Data Firehose delivers the stream to S3 without operating consumers. A Lambda consumer on the stream needs a parallelization factor and a failure destination, or one bad record blocks the shard.

7. Security that the design depends on

Identity. Humans use IAM Identity Center. Workloads use IAM roles. Resource policies restrict who can assume those roles. There are no IAM users with access keys on servers.

Network. Security groups are stateful and reference other security groups, not wide CIDR ranges. The database security group allows the application security group on the database port only. Network ACLs stay close to the default unless a compliance rule requires them. A single NAT gateway is rejected in review, because its Availability Zone becomes a dependency for every private subnet.

Encryption and secrets. Customer managed KMS keys encrypt RDS, EBS, S3, and ElastiCache. Secrets Manager holds database and API credentials. ACM holds the public certificates for CloudFront and the load balancer. Private keys are not checked into Git.

Detection. Amazon GuardDuty on the account, AWS CloudTrail with log file validation writing to a dedicated S3 bucket, and AWS Config for the rules that drift breaks availability: a security group opened to the world, or a load balancer left in one subnet.

8. Observability

Amazon CloudWatch metrics, logs, and alarms are the minimum. Each alarm needs an owner and an action. The first alarms are:

UnHealthyHostCount and HTTPCode_Target_5XX_Count on the load balancer.
CPU, database connections, free storage, and replica lag on RDS or Aurora. Alarm on failover events, not only on saturation.
ApproximateAgeOfOldestMessage on each SQS queue, and depth of each dead-letter queue.
ElastiCache evictions and replication lag.
NAT gateway ErrorPortAllocation and packets dropped, per Availability Zone.
AWS X-Ray or OpenTelemetry traces a request from the load balancer through the application to the database. CloudTrail is the audit log of API calls, not an application log. A subscription filter ships application logs to the tooling you already operate, such as Prometheus and Grafana or an ELK stack, but the CloudWatch alarms above still exist so a third-party outage does not hide a production failure. ## 9. Delivery

Infrastructure is Terraform or AWS CloudFormation, reviewed like application code. Application delivery is GitHub Actions or AWS CodePipeline, with CodeBuild or the GitHub runner producing an immutable image in Amazon ECR. CodeDeploy or the ECS deployment controller rolls that image. A pipeline does not SSH to an instance and edit it.

Systems Manager Session Manager is the break-glass shell. Patch Manager or a rebuilt image handles OS patches. An instance is replaced, not patched in place, when the launch template is the source of truth.

A change that cannot roll back is not shipped on a Friday. Database migrations expand first and contract later, so the previous application version still runs during the rollout.

Figure 2. Failure boundaries: instance, Availability Zone, and Region.

10. What happens when something fails

Instance or task failure. The load balancer stops sending traffic after two failed health checks. Auto Scaling or the ECS service replaces the task. In-flight requests finish during deregistration delay. No session is lost, because no session lived on the task.

Bad deployment. The new tasks fail the health check and the rollout stops. The previous tasks keep the minimum healthy percentage. Both Availability Zones are never drained by the same batch.

Availability Zone failure. The load balancer, Auto Scaling, RDS Multi-AZ or Aurora, and ElastiCache failover continue in the remaining zones. NAT gateways in the surviving zones still have their own path out. Capacity planning assumes one zone is gone: the remaining zones must hold the steady-state load, not only the peak with all three zones up.

Dependency failure. A full database connection pool, a stuck queue consumer, or a poisoned message. RDS Proxy, a dead-letter queue, and idempotent consumers are what keep that from becoming an outage. Reserved concurrency on Lambda protects the database from the function.

Region failure. Route 53 failover sends traffic to the second Region only if that Region is running a pilot light or a warm standby: data replicated, image present, and a scale-up path tested. A DNS record with an empty Region behind it does not meet the RTO. Aurora Global Database or DynamoDB Global Tables set the data RPO. S3 Cross-Region Replication covers objects. AWS Elastic Disaster Recovery is the option when EC2 servers must be replicated at the block level rather than rebuilt from a launch template.

The test. On a schedule, stop one instance, fail over RDS, and shift the load balancer off one Availability Zone. Record the observed RTO. A design that has not been failed is still a diagram.

11. Worked example: automotive telematics

A telematics pipeline ingested vehicle signals at a rate that a single instance could not absorb and could not be allowed to drop.

Vehicles publish to Amazon API Gateway or directly to Amazon Kinesis Data Streams. Kinesis retains the stream so a slow consumer does not lose data. AWS Lambda, with reserved concurrency, validates and writes the hot record to DynamoDB and hands the bulk path to Amazon Data Firehose, which lands objects in S3. ElastiCache holds the latest position per vehicle for the read API. The read API is an Application Load Balancer in front of a stateless service across three Availability Zones, with Aurora or DynamoDB behind it depending on the query shape. Amazon QuickSight reads the curated S3 data for fleet dashboards. GitHub Actions builds the image and CodePipeline deploys it.

That system was operated at 99.99% availability and scaled with shard count and function concurrency instead of a fixed fleet, with about 60% less operational overhead than the host-based pipeline it replaced. The properties that made the number real were a stateless read path, a retained ingest stream, Multi-AZ data stores, and a deployment that could not take every zone down together.

12. Review checklist

Use this list before calling an environment production-ready.

Three Availability Zones. Private application and data subnets. One NAT gateway per zone. Gateway endpoints for S3 and DynamoDB. Interface endpoints for Secrets Manager, KMS, CloudWatch Logs, ECR, and Systems Manager.
Route 53 health check and an intentional TTL. CloudFront or API Gateway in front. ACM certificate. WAF attached. Application Load Balancer health check on a path that checks dependencies.
Launch template with IMDSv2, encrypted EBS, and an instance role. Auto Scaling minimum on On-Demand across zones. Target-tracking policy. Rolling deploy with a minimum healthy percentage.
RDS Multi-AZ or Aurora, backups and point-in-time recovery, RDS Proxy, credentials in Secrets Manager. ElastiCache replication group with Multi-AZ if the cache holds sessions. DynamoDB point-in-time recovery where DynamoDB is the system of record.
SQS redrive to a dead-letter queue. Idempotent consumers. Kinesis retention long enough to replay.
CloudWatch alarms on unhealthy hosts, 5xx, replica lag, queue age, and dead-letter depth. CloudTrail enabled. A tested instance failure, database failover, and single-zone evacuation, with the measured RTO written down. ## Closing

A highly available design is a small set of decisions applied consistently: no single Availability Zone on the critical path, no state on a node you intend to throw away, a data store that fails over inside its RTO, and a deployment and an alarm that you have already watched work. The service list is long. The design is those four rules.

Author: Sanket Satish Patharkar

Senior Cloud Operation Engineer. AWS, DevOps, and CI/CD for production systems.