By Sanket Satish Patharkar, Senior Cloud Operation Engineer
This document is a practical design for an application that must stay available when an instance, an Availability Zone, or a deployment fails, and that must scale with demand instead of with a fixed fleet. It is written so each section can be copied as-is.
1. Scope and design targets
High availability and disaster recovery are different problems. Multi-AZ keeps a single Region serving traffic through an instance or Availability Zone failure. Multi-Region is for the loss of a Region, and it is justified only after recovery objectives are explicit.
Set these before choosing services:
- Availability target. 99.99% allows about 52 minutes of downtime per year. That target requires at least two Availability Zones, health-checked load balancing, and a data store that fails over without a manual restore.
- Recovery point objective (RPO). How much data loss is acceptable. RDS Multi-AZ and Aurora use synchronous storage inside the Region, so the RPO for an Availability Zone failure is near zero. A cross-Region replica is asynchronous, so the RPO is the replication lag.
- Recovery time objective (RTO). How long the service may be down. An Application Load Balancer removes a failed task in the health-check interval. An RDS Multi-AZ failover is typically one to two minutes. Aurora failover is usually under a minute. A Regional failover is measured in minutes only if the second Region is already warm.
- Scale unit. What grows: requests per second, messages per second, or stored bytes. The Auto Scaling signal has to match that unit. ## 2. Reference architecture
One Virtual Private Cloud, three Availability Zones, three subnet tiers.
Public subnets hold only the load balancer nodes and the NAT gateways. Application subnets are private and hold Amazon EC2, Amazon ECS, or Amazon EKS. Data subnets are private and hold Amazon RDS or Amazon Aurora, and Amazon ElastiCache. There is no path from the internet to the data tier.
Request path for a synchronous API:
Amazon Route 53 resolves the name. A latency or failover record selects the Region. The record TTL for a failover name is 60 seconds or less.
Amazon CloudFront terminates viewer TLS with an AWS Certificate Manager certificate, caches what is safe to cache, and forwards dynamic requests to the Application Load Balancer. AWS WAF is attached here or on the load balancer.
The Application Load Balancer spans all three Availability Zones, checks a /health endpoint, and sends traffic only to healthy targets.
The application is stateless. Session and hot keys live in ElastiCache or Amazon DynamoDB. The instance disk is disposable.
Writes go to the RDS or Aurora writer. Reads that tolerate replication lag go to a reader endpoint. Connection spikes go through Amazon RDS Proxy so a scale-out does not exhaust database connections.
Work that does not need an immediate response is published to Amazon SQS or Amazon EventBridge. The API returns after the enqueue.
Outbound path from a private subnet: a NAT gateway in the same Availability Zone, or a VPC endpoint when the destination is an AWS API. Amazon S3 and DynamoDB use gateway endpoints. AWS Secrets Manager, Amazon KMS, Amazon CloudWatch Logs, Amazon ECR, and AWS Systems Manager use interface endpoints so a NAT failure does not block pulls, secrets, or log shipping.
Figure 1. Multi-AZ reference architecture.
3. Edge, DNS, and load balancing
Route 53 health checks the load balancer or a synthetic endpoint, not a single instance. Failover routing moves DNS to the standby Region. Latency routing is for active-active. Do not create the standby DNS record until the standby can actually serve.
Use an Application Load Balancer for HTTP and HTTPS. Use a Network Load Balancer only when you need a static IP, TCP or UDP, or millions of connections per second. Enable cross-zone load balancing on the Application Load Balancer. Set deregistration delay to match the longest request you must finish, commonly 30 to 60 seconds, so a scale-in does not cut in-flight work.
The target health check uses the application health path, an interval of 10 seconds, and a threshold of 2. A process that is up but cannot reach its database must fail that check.
Amazon CloudFront sits in front of the load balancer for TLS at the edge and to absorb bursts of static content. Amazon API Gateway is the front door when the client is a device or a partner calling a REST or WebSocket API, with throttling per API key. AWS Global Accelerator is the alternative when clients need static anycast IPs and fast Regional failover without waiting on DNS TTL.
AWS WAF on CloudFront or the Application Load Balancer blocks common web exploits. AWS Shield Standard is automatic. Shield Advanced is for workloads that are attacked and need cost protection and the response team, not for every design.
4. Compute and scaling
Amazon EC2 Auto Scaling uses a launch template, not a launch configuration. The template enforces IMDSv2, encrypts EBS with a customer managed KMS key, and attaches an instance profile. No long-lived access keys are placed on the instance.
Minimum capacity is at least two, spread across Availability Zones, and that minimum runs On-Demand. Spot capacity is extra scale above the minimum. If the minimum the availability target depends on is Spot, a capacity withdrawal is an outage.
Scaling policy is target tracking on the signal that matches the workload: ALBRequestCountPerTarget for a request-driven API, or CPU only when CPU is the real bottleneck. Scale-in is slower than scale-out. Health check type is ELB, so a target the load balancer has marked unhealthy is replaced.
A rolling deployment uses a minimum healthy percentage so both Availability Zones are never replaced in the same batch. For Amazon ECS on AWS Fargate, the service spans the private subnets, the deployment minimum healthy percent stays at 100 with a maximum of 200, and tasks are replaced only after the new task is healthy. Amazon EKS uses the same rule: a PodDisruptionBudget and topology spread so one zone cannot hold every replica.
AWS Lambda is for event-driven work with uneven load. Set reserved concurrency on functions that call the database so a spike cannot exhaust connections. Lambda destinations or an SQS dead-letter queue capture failed asynchronous invocations.
5. Data
Amazon RDS Multi-AZ keeps a synchronous standby in another Availability Zone and fails over the writer endpoint. Use it when the engine must be MySQL, PostgreSQL, SQL Server, or Oracle and failover in one to two minutes is acceptable. Amazon Aurora uses a shared storage volume across three Availability Zones and fails over faster. Use Aurora when the workload is MySQL- or PostgreSQL-compatible and the RTO is tighter. A read replica, and the Aurora reader endpoint, scale reads. They are not the failover mechanism for the writer.
Turn on automated backups with a retention that matches the RPO, and point-in-time recovery. Take snapshots before a schema change. Amazon RDS Proxy pools connections and makes a Multi-AZ failover less visible to the application. Store the master password in AWS Secrets Manager and rotate it. Do not put it in the task definition or user data.
Amazon DynamoDB is the choice when access is by key and the workload is spiky. It is Multi-AZ by default. Point-in-time recovery and on-demand or provisioned capacity with auto scaling belong in the table design. DynamoDB Accelerator (DAX) is a cache in front of DynamoDB, not a general application cache. Global Tables replicate across Regions asynchronously. That RPO is the replication lag, and writes must be idempotent because the same key can be written in two Regions.
Amazon ElastiCache for Redis uses a replication group with Multi-AZ and automatic failover. The application uses the primary endpoint for writes and the reader endpoint for cache reads. Memcached has no replication. Use it only for data that can be rebuilt. Do not store the only copy of a session in Memcached.
Amazon S3 holds objects, logs, and backups. Block public access at the account, enable versioning on buckets that hold data you cannot recreate, and encrypt with KMS. S3 Cross-Region Replication copies objects for a Regional loss. It is not a substitute for database backups. AWS Backup is the control plane for RDS, Aurora, EBS, and DynamoDB backup policies, copy to a second Region, and a vault lock when retention must be immutable.
6. Messaging and events
Amazon SQS decouples the API from slow work. Set the visibility timeout longer than the handler's worst case. Set a redrive policy to a dead-letter queue after a small max receive count, and alarm when that queue is non-empty. The consumer must be idempotent, because a timeout delivers the message again.
Use SQS standard queues for throughput. Use FIFO only when order per group and exactly-once processing matter, and accept the lower throughput. Amazon SNS fan-out is for one event and many independent consumers. Amazon EventBridge is for routing events between applications by rule, including a scheduled rule that replaces a single cron host.
Amazon Kinesis Data Streams is the ingest path for ordered, high-rate telemetry. Size shards for the write rate, and use enhanced fan-out when many consumers read the same stream. Amazon Data Firehose delivers the stream to S3 without operating consumers. A Lambda consumer on the stream needs a parallelization factor and a failure destination, or one bad record blocks the shard.
7. Security that the design depends on
Identity. Humans use IAM Identity Center. Workloads use IAM roles. Resource policies restrict who can assume those roles. There are no IAM users with access keys on servers.
Network. Security groups are stateful and reference other security groups, not wide CIDR ranges. The database security group allows the application security group on the database port only. Network ACLs stay close to the default unless a compliance rule requires them. A single NAT gateway is rejected in review, because its Availability Zone becomes a dependency for every private subnet.
Encryption and secrets. Customer managed KMS keys encrypt RDS, EBS, S3, and ElastiCache. Secrets Manager holds database and API credentials. ACM holds the public certificates for CloudFront and the load balancer. Private keys are not checked into Git.
Detection. Amazon GuardDuty on the account, AWS CloudTrail with log file validation writing to a dedicated S3 bucket, and AWS Config for the rules that drift breaks availability: a security group opened to the world, or a load balancer left in one subnet.
8. Observability
Amazon CloudWatch metrics, logs, and alarms are the minimum. Each alarm needs an owner and an action. The first alarms are:
- UnHealthyHostCount and HTTPCode_Target_5XX_Count on the load balancer.
- CPU, database connections, free storage, and replica lag on RDS or Aurora. Alarm on failover events, not only on saturation.
- ApproximateAgeOfOldestMessage on each SQS queue, and depth of each dead-letter queue.
- ElastiCache evictions and replication lag.
- NAT gateway ErrorPortAllocation and packets dropped, per Availability Zone.
- AWS X-Ray or OpenTelemetry traces a request from the load balancer through the application to the database. CloudTrail is the audit log of API calls, not an application log. A subscription filter ships application logs to the tooling you already operate, such as Prometheus and Grafana or an ELK stack, but the CloudWatch alarms above still exist so a third-party outage does not hide a production failure. ## 9. Delivery
Infrastructure is Terraform or AWS CloudFormation, reviewed like application code. Application delivery is GitHub Actions or AWS CodePipeline, with CodeBuild or the GitHub runner producing an immutable image in Amazon ECR. CodeDeploy or the ECS deployment controller rolls that image. A pipeline does not SSH to an instance and edit it.
Systems Manager Session Manager is the break-glass shell. Patch Manager or a rebuilt image handles OS patches. An instance is replaced, not patched in place, when the launch template is the source of truth.
A change that cannot roll back is not shipped on a Friday. Database migrations expand first and contract later, so the previous application version still runs during the rollout.
Figure 2. Failure boundaries: instance, Availability Zone, and Region.
10. What happens when something fails
Instance or task failure. The load balancer stops sending traffic after two failed health checks. Auto Scaling or the ECS service replaces the task. In-flight requests finish during deregistration delay. No session is lost, because no session lived on the task.
Bad deployment. The new tasks fail the health check and the rollout stops. The previous tasks keep the minimum healthy percentage. Both Availability Zones are never drained by the same batch.
Availability Zone failure. The load balancer, Auto Scaling, RDS Multi-AZ or Aurora, and ElastiCache failover continue in the remaining zones. NAT gateways in the surviving zones still have their own path out. Capacity planning assumes one zone is gone: the remaining zones must hold the steady-state load, not only the peak with all three zones up.
Dependency failure. A full database connection pool, a stuck queue consumer, or a poisoned message. RDS Proxy, a dead-letter queue, and idempotent consumers are what keep that from becoming an outage. Reserved concurrency on Lambda protects the database from the function.
Region failure. Route 53 failover sends traffic to the second Region only if that Region is running a pilot light or a warm standby: data replicated, image present, and a scale-up path tested. A DNS record with an empty Region behind it does not meet the RTO. Aurora Global Database or DynamoDB Global Tables set the data RPO. S3 Cross-Region Replication covers objects. AWS Elastic Disaster Recovery is the option when EC2 servers must be replicated at the block level rather than rebuilt from a launch template.
The test. On a schedule, stop one instance, fail over RDS, and shift the load balancer off one Availability Zone. Record the observed RTO. A design that has not been failed is still a diagram.
11. Worked example: automotive telematics
A telematics pipeline ingested vehicle signals at a rate that a single instance could not absorb and could not be allowed to drop.
Vehicles publish to Amazon API Gateway or directly to Amazon Kinesis Data Streams. Kinesis retains the stream so a slow consumer does not lose data. AWS Lambda, with reserved concurrency, validates and writes the hot record to DynamoDB and hands the bulk path to Amazon Data Firehose, which lands objects in S3. ElastiCache holds the latest position per vehicle for the read API. The read API is an Application Load Balancer in front of a stateless service across three Availability Zones, with Aurora or DynamoDB behind it depending on the query shape. Amazon QuickSight reads the curated S3 data for fleet dashboards. GitHub Actions builds the image and CodePipeline deploys it.
That system was operated at 99.99% availability and scaled with shard count and function concurrency instead of a fixed fleet, with about 60% less operational overhead than the host-based pipeline it replaced. The properties that made the number real were a stateless read path, a retained ingest stream, Multi-AZ data stores, and a deployment that could not take every zone down together.
12. Review checklist
Use this list before calling an environment production-ready.
- Three Availability Zones. Private application and data subnets. One NAT gateway per zone. Gateway endpoints for S3 and DynamoDB. Interface endpoints for Secrets Manager, KMS, CloudWatch Logs, ECR, and Systems Manager.
- Route 53 health check and an intentional TTL. CloudFront or API Gateway in front. ACM certificate. WAF attached. Application Load Balancer health check on a path that checks dependencies.
- Launch template with IMDSv2, encrypted EBS, and an instance role. Auto Scaling minimum on On-Demand across zones. Target-tracking policy. Rolling deploy with a minimum healthy percentage.
- RDS Multi-AZ or Aurora, backups and point-in-time recovery, RDS Proxy, credentials in Secrets Manager. ElastiCache replication group with Multi-AZ if the cache holds sessions. DynamoDB point-in-time recovery where DynamoDB is the system of record.
- SQS redrive to a dead-letter queue. Idempotent consumers. Kinesis retention long enough to replay.
- CloudWatch alarms on unhealthy hosts, 5xx, replica lag, queue age, and dead-letter depth. CloudTrail enabled. A tested instance failure, database failover, and single-zone evacuation, with the measured RTO written down. ## Closing
A highly available design is a small set of decisions applied consistently: no single Availability Zone on the critical path, no state on a node you intend to throw away, a data store that fails over inside its RTO, and a deployment and an alarm that you have already watched work. The service list is long. The design is those four rules.
Author: Sanket Satish Patharkar
Senior Cloud Operation Engineer. AWS, DevOps, and CI/CD for production systems.
For any issue or support: LinkedIn.
Top comments (0)